Skip to content

Define BIGQUERY typed-result/export contract #790

Description

@apstndb

Problem

BIGQUERY preserves bigquery.Value rows and RowIterator.Schema while reading a query, but immediately converts every cell to display text through formatBigQueryValue. The returned Result therefore contains only Rows []Row.

Consequences for export formats include:

  • NULL, booleans, integers, and floats become JSON strings in JSONL output;
  • JSON, ARRAY, and RECORD values become quoted JSON text;
  • nested RECORD field names are lost because nested values are positional;
  • native type distinctions such as NUMERIC versus BIGNUMERIC do not survive downstream;
  • the display spelling may remain useful for STRING, BYTES, temporal, geography, and interval values, but their BigQuery type identity is absent.

This limitation is recorded in the internal/mycli/feature/bigquery package comment and was called out as tracked debt when BIGQUERY moved behind the feature seam in #781, originating from #778.

Issue #738 added the exclusive TypedRows payload for buffered results, but that payload intentionally contains Spanner ResultSetMetadata and []*spanner.Row. It is not a provider-neutral typed-result contract. Presentation rows without a TypedRows payload are currently replayed as synthetic Spanner STRING columns, so the existing machinery cannot preserve BigQuery values by itself.

Design choices

A. JSONL semantic fidelity with documented surrogate mappings

Recommended first step.

Add a feature-local conversion for the BigQuery types that can be represented faithfully enough by the existing Spanner-oriented typed pipeline. Preserve the current table-output bytes, keep SQL export disabled, and define explicit fallback behavior for unsupported or non-equivalent BigQuery types.

This option should not preclude option B later.

Potentially faithful mappings include STRING, BYTES, INT64, FLOAT64, BOOL, TIMESTAMP, DATE, NUMERIC, JSON, ARRAY, and STRUCT. The design must explicitly decide how to handle BIGNUMERIC, TIME, DATETIME, GEOGRAPHY, INTERVAL, RANGE, and any future BigQuery-only type without presenting a misleading native type.

Acceptance coverage must include:

  • null versus the literal string "NULL";
  • BOOL, INT64, and FLOAT64 values;
  • JSON values;
  • NUMERIC and BIGNUMERIC distinction;
  • repeated values;
  • nested RECORD values with field names;
  • the fallback policy for unsupported types;
  • empty-result headers;
  • byte-level CSV and JSONL output;
  • byte-identical existing table output.

B. Provider-neutral typed buffered results

Generalize TypedRows and its rendering hooks so buffered results can carry provider-specific schema and values without first translating them into Spanner rows.

This is the structural end state if multiple non-Spanner producers need typed buffered results, but BIGQUERY alone does not justify the abstraction and migration cost. The trigger for this option should be a second concrete consumer that needs provider-neutral typed results.

Constraints

This belongs near the table/output architecture tracked by the #29 issue family, but remains a standalone BIGQUERY design decision rather than a child issue.

Decision needed

Choose whether to pursue option A as a bounded first step, defer until option B has a second consumer, or retain display-only exports explicitly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    design-neededRequires architecture or design work before implementationenhancementNew feature or requestoutput-formattingOutput format and display improvements

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions