Skip to content

Fix row-order-dependent unit test comparisons - #13

Draft
fornwall wants to merge 1 commit into
mainfrom
order-independent-comparison
Draft

fornwall wants to merge 1 commit into
mainfrom
order-independent-comparison

Conversation

@fornwall

@fornwall fornwall commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Unit tests compare the model's actual rows with the expected fixture. The comparator previously matched rows by their returned positions, so identical results could fail when the warehouse returned them in different orders.

BigQuery ARRAY and STRUCT (and soon perhaps GEOGRAPHY and JSON) columns are excluded from the generated ORDER BY (as it's unsupported).

  • I'm not using Databricks or DuckDB, but they seem to have similar types that are not sortable from ORDER BY.

For example, this model returns 100 STRUCT rows:

-- models/struct_only.sql
SELECT STRUCT(id AS id) AS payload
FROM UNNEST(GENERATE_ARRAY(1, 100)) AS id;

A unit test supplies the same rows in reverse order:

# models/unit_tests.yml
unit_tests:
  - name: struct_rows_reversed
    model: struct_only
    given: []
    expect:
      format: sql
      rows: |
        SELECT STRUCT(id AS id) AS payload
        FROM UNNEST(GENERATE_ARRAY(100, 1, -1)) AS id

When BigQuery returns actual IDs 1…100 and expected IDs 100…1, the old comparator reports:

| payload/id |
|------------|
| 100 -> 1   |
| 99 -> 2    |
| ...        |
| 1 -> 100   |
100 row(s) differ.

With this fix, the same test passes. Complete typed rows are matched independently of their returned order, consuming each expected occurrence once. Adding a constant 1 AS sort_key to both the model and fixture also passes, even though the generated ORDER BY sort_key leaves all rows tied.

Duplicate counts still matter. If the expected fixture replaces ID 100 with another ID 99, there are still 100 rows on each side, but the test correctly fails with:

| payload/id |
|------------|
| 99 -> 100  |
1 row(s) differ.

Matching uses Arrow row encoding where supported and recursive typed comparison otherwise.

  • Maps containing the same key/value pairs match regardless of entry order, so {a: 1, b: 2} matches {b: 2, a: 1}.
  • Array element order remains significant: [1, 2] differs from [2, 1].

Related: dbt-labs#14110, dbt-labs#13988, dbt-labs#10167.

@fornwall
fornwall force-pushed the order-independent-comparison branch from 1d9f7fe to 4fe8efa Compare September 12, 2026 10:36
@fornwall
fornwall marked this pull request as draft September 12, 2026 10:58
@fornwall
fornwall force-pushed the order-independent-comparison branch from 4fe8efa to 0f20379 Compare September 12, 2026 11:22
@fornwall
fornwall force-pushed the order-independent-comparison branch from 0f20379 to 8f35396 Compare September 12, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant