Skip to content

Implement SqlInfo in the IceGate Flight SQL server #178

Description

@s-prosvirnin

Problem

JDBC clients cannot browse the IceGate Flight SQL endpoint. Connecting works and SQL queries execute, but the client shows no catalogs, no schemas, and no tables.

The endpoint answers CommandGetCatalogs, CommandGetDbSchemas, and CommandGetTables, but it answers CommandGetSqlInfo with gRPC status Unimplemented. The Apache Arrow Flight SQL JDBC driver issues CommandGetSqlInfo while building its DatabaseMetaData, before it ever asks for catalogs. That call fails, so metadata construction aborts and the object tree stays empty.

In IntelliJ IDEA the failure is additionally disguised: the driver raises a FlightRuntimeException carrying a non-serializable CallStatus, and the IDE's RMI bridge reports NotSerializableException: ...flight.CallStatus instead of the real cause.

Context

  • The Flight SQL endpoint is hosted by the upstream datafusion-flight-sql-server service, wired up in crates/icegate-query/src/flight_sql/server.rs. The crate is pinned as an exact version in the workspace Cargo.toml.
  • The upstream service returns Unimplemented from both get_flight_info_sql_info and do_get_sql_info. Everything else the JDBC metadata path needs (catalogs, schemas, tables, table types) is already implemented upstream and returns correct data for IceGate: catalog iceberg, schema icegate, tables logs, spans, events, metrics.
  • Tenant isolation and the read-only guarantee are installed per request in crates/icegate-query/src/flight_sql/provider.rs: every request builds a session whose iceberg catalog is wrapped by a tenant-scoped decorator, and SQLOptions in crates/icegate-query/src/flight_sql/server.rs reject DDL and DML. Any new response path must not weaken either property.
  • crates/icegate-query/README.md currently lists the Arrow Flight SQL JDBC driver under "To verify" and documents the missing pieces under "Known caveats".

Term used throughout this task: SqlInfo response means the server's answer to the Flight SQL CommandGetSqlInfo command — a record batch of (info_name, value) pairs describing server identity and SQL capabilities, defined by the Flight SQL protocol. This is protocol vocabulary, not an IceGate-specific concept.

Target behavior

  1. CommandGetSqlInfo returns a valid SqlInfo response over both the GetFlightInfo and DoGet paths, instead of Unimplemented.
  2. The response describes the endpoint truthfully: server name and version, read-only server, catalog and schema support as actually exposed (catalog iceberg, schema icegate), identifier quoting and case behavior, and the SQL keyword/function information the driver asks for.
  3. A client that requests a subset of info codes receives exactly that subset; a client that requests none receives the full set.
  4. The Arrow Flight SQL JDBC driver builds DatabaseMetaData without error, and a JDBC-based client (IntelliJ IDEA / DataGrip / DBeaver) shows the tree iceberg → icegate → the four tables.
  5. Tenant isolation and read-only enforcement are unchanged: the SqlInfo response carries no tenant-derived data and opens no path around the tenant-scoped catalog wrapper.
  6. The four smoke-test steps already listed in crates/icegate-query/README.md under "Client compatibility" pass for the JDBC driver.

What to do

High level, in the order that makes sense to sequence the work:

  1. Introduce a server-side wrapper around the upstream Flight SQL service inside crates/icegate-query/src/flight_sql. It delegates every method to the upstream service and overrides only the two SqlInfo methods. Rationale for wrapping rather than patching: the upstream crate is an external pinned dependency, and a delegating wrapper keeps the override visible in our tree and survives version bumps. An upstream contribution can follow later, but is not part of this task.

  2. Define the SqlInfo value set the server advertises. Values must be derived from existing sources of truth — crate name and version from the package metadata, catalog and schema names from the engine configuration used in crates/icegate-query/src/flight_sql/provider.rs — never re-declared as literals. Read-only must be advertised consistently with the SQLOptions already applied.

  3. Handle info-code filtering so a request for specific codes returns only those, per the Flight SQL protocol.

  4. Cover the behavior with tests, following docs/tests.md: at minimum an integration test that starts the Flight SQL server and asserts the SqlInfo response is returned with the expected identity and capability entries, and that the existing catalog/schema/table metadata calls still work alongside it.

  5. Update crates/icegate-query/README.md: move the Arrow Flight SQL JDBC driver from "To verify" to "Verified" once the smoke test passes, and revise the "Known caveats" entries that this change invalidates.

No configuration surface changes are expected — no new config field, env var, port, or service URL — so the Helm and Docker Compose config pair stays untouched. If the implementation turns out to need a config field, both deployment sides must be updated together.

Out of scope

  • The other unimplemented metadata commands: primary keys, exported keys, imported keys, cross reference, XDBC type info.
  • BeginTransaction / EndTransaction, the handshake stub, and prepared-statement parameter binding.
  • The separate client-side defect where IntelliJ IDEA fails to render MAP(VARCHAR, VARCHAR) columns (attributes) because the driver returns a non-serializable Arrow Text value across the IDE's RMI bridge. This is a driver/IDE limitation, not a server defect; the documented workaround is projecting individual map keys.
  • Any change to the Iceberg schema or to the tenant isolation design.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions