Skip to content

Push down string_agg to pgduck_server#440

Merged
sfc-gh-okalaci merged 3 commits into
mainfrom
feat/pushdown-string-agg
Jul 10, 2026
Merged

Push down string_agg to pgduck_server#440
sfc-gh-okalaci merged 3 commits into
mainfrom
feat/pushdown-string-agg

Conversation

@sfc-gh-okalaci

Copy link
Copy Markdown
Collaborator

Summary

Adds string_agg(text, text) to the list of aggregates shipped to pgduck_server / DuckDB, gated on the delimiter being a non-NULL constant.

This is a customer feature request (Zendesk 11742, previously 10799). Investigation showed the delimiter is the only place string_agg diverges between PostgreSQL and DuckDB:

  • NULL delimiter — DuckDB returns NULL, PostgreSQL treats it as "no separator" and concatenates.
  • Non-constant (per-row) delimiter — DuckDB errors (Separator argument to StringAgg must be a constant), PostgreSQL evaluates it per row.

With a non-NULL constant delimiter, every other behavior is byte-for-byte identical between the engines: NULL/empty values (dropped/kept the same way), ORDER BY (incl. NULLS FIRST/LAST, multi-key), DISTINCT, FILTER, GROUP BY, unicode, and large inputs. This was verified with an extensive engine-vs-engine torture test (44+ scenarios).

Implementation notes

  • IsStringAggShippable receives an Aggref (not a FuncExpr), so it unwraps the delimiter from Aggref->args instead of using GetConstArg(). The delimiter is always the 2nd arg because sort-only junk TargetEntrys are appended after the real args.
  • The bytea variant is excluded automatically (DuckDB has no string_agg(BLOB, VARCHAR)), as is any non-constant delimiter.
  • Value casts that render differently between engines (e.g. float8::text1.0 vs 1) are governed by their own cast-pushdown behavior and are out of scope here.

Test plan

pg_lake_table/tests/pytests/test_string_agg_pushdown.py:

  • Pushed + parity vs heap (each with an internal ORDER BY for determinism): basic, ORDER BY DESC, separate sort key with NULLS LAST/FIRST, DISTINCT, FILTER, empty/unicode delimiter, GROUP BY.
  • Not pushed + still correct locally: NULL constant delimiter, per-row (column) delimiter, bytea variant.

Locally verified end-to-end via EXPLAIN (VERBOSE) on an Iceberg table: the constant-delimiter case becomes a Custom Scan (Query Pushdown) with string_agg(...) in the remote SQL, while the NULL/column-delimiter cases stay local. Relying on CI for the full pytest run.

Made with Cursor

Ship string_agg(text, text) to pgduck_server/DuckDB when the delimiter is a
non-NULL constant. That is the only condition under which the two engines
diverge: DuckDB returns NULL for a NULL separator (PostgreSQL concatenates)
and rejects a non-constant separator, whereas every other behavior
(NULL/empty values, ORDER BY, DISTINCT, FILTER) is identical.

The bytea variant is excluded automatically (DuckDB has no
string_agg(BLOB, VARCHAR)) as is a non-constant delimiter.

Aggregates reach the shippability callback as an Aggref, so IsStringAggShippable
unwraps the delimiter from Aggref->args rather than using GetConstArg (which
assumes a FuncExpr).

Adds test_string_agg_pushdown.py covering pushed cases (ORDER BY / DISTINCT /
FILTER / GROUP BY, unicode) and non-pushed cases (NULL delimiter, per-row
delimiter, bytea variant).

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Onder KALACI <oender.kalaci@snowflake.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@sfc-gh-okalaci sfc-gh-okalaci force-pushed the feat/pushdown-string-agg branch from d1eeee4 to b78c5b1 Compare July 10, 2026 09:22
The negative cases assert the pushed SQL does not contain "string_agg", but
the schema and S3 path were named "string_agg_pushdown", so the substring
matched the table/path name rather than the function. Rename to "sagg_pushdown".

Signed-off-by: Onder KALACI <oender.kalaci@snowflake.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
return false;

/* a NULL delimiter yields NULL in DuckDB but concatenation in PostgreSQL */
return !((Const *) delimNode)->constisnull;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should perhaps open an issue to add a new implementation

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #441 to track this. The NULL-constant-delimiter case is a straightforward transform (deparse a NULL delimiter as '', since that matches PostgreSQL's "no separator" semantics); the per-row/non-constant delimiter case is harder and captured there too, with examples.

Cover a single-codepoint emoji and a multi-codepoint emoji (with skin-tone
modifier) alongside the existing Japanese and accented-Latin values, since
these are 4-byte UTF-8 sequences worth exercising through the pushdown path.

Signed-off-by: Onder KALACI <oender.kalaci@snowflake.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@sfc-gh-okalaci sfc-gh-okalaci merged commit 890365c into main Jul 10, 2026
64 checks passed
@sfc-gh-okalaci sfc-gh-okalaci deleted the feat/pushdown-string-agg branch July 10, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants