Add Spark add_months and time travel; fix set-operation bindings - #18
Merged
Merged
Conversation
DuckDB lacks add_months, which appears in Spark SQL translated through lpts. Register add_months(DATE, INTEGER) -> DATE implementing full Spark semantics: day-of-month preserved, clamped to target month length, with the end-of-month rule (last day of source month maps to last day of target month). Reuses the DuckDB Date API for leap-year-correct month lengths. Registered in LoadInternal so any consumer loading lpts (or compiling its sources) gets the function. Adds test/sql/spark_add_months.test covering scalar correctness across month-end clamping, leap years, negative offsets, year rollover, and NULL propagation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mdrakiburrahman
added a commit
to mdrakiburrahman/openivm
that referenced
this pull request
Aug 16, 2026
Per review, the add_months scalar function belongs in lpts (the Spark-compat layer openivm already builds), not duplicated in openivm. Move the implementation to lpts (cwida/lpts#18) and consume it here via the pin: - delete src/functions/spark_scalar_functions.{cpp,hpp} (now in lpts) - compile ${LPTS_DIR}/src/spark_scalar_functions.cpp from the lpts submodule and include its header from ${LPTS_DIR}/src/include (already on the path) - keep the thin RegisterSparkScalarFunctions(loader) call in openivm's LoadInternal (openivm does not invoke lpts's LoadInternal, so it registers the lpts-provided function itself) - bump third_party/lpts 13786cb..642c762 (cwida/lpts main + add_months) - trim test/sql/spark_add_months.test to openivm's concern (add_months resolves + drives a real SIMPLE_PROJECTION delta / EXCEPT ALL parity); exhaustive scalar-correctness now lives in lpts Local openivm CI green (build + full sqllogictest: 10020 assertions / 83 cases). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CURRENT_TIMESTAMP folds to a TIMESTAMPTZ constant during lpts_query binding, which autoloads icu. Offline linux_amd64 CI cannot fetch icu and failed the whole file; `require icu` makes those runners skip instead (arm64 still runs it fully). Fixes the pre-existing dialect_spark.test amd64 failure surfaced on this add_months PR via the merge with main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mdrakiburrahman
added a commit
to mdrakiburrahman/openivm-spark
that referenced
this pull request
Aug 16, 2026
OPENIVM_COMMIT -> 1220ba4 (ila/openivm#10 head: sources add_months from lpts + bumps third_party/lpts) and LPTS_COMMIT -> 592d469 (cwida/lpts#18 head: native add_months + SPARK dialect icu test guard). Both upstream PRs are green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ila
pushed a commit
to mdrakiburrahman/openivm-spark
that referenced
this pull request
Aug 17, 2026
OPENIVM_COMMIT -> 1220ba4 (ila/openivm#10 head: sources add_months from lpts + bumps third_party/lpts) and LPTS_COMMIT -> 592d469 (cwida/lpts#18 head: native add_months + SPARK dialect icu test guard). Both upstream PRs are green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore the lhs column_map before traversing each set-op sibling so later UNION/EXCEPT/INTERSECT branches cannot leak bindings into the parent. Also resolve projection refs above set-ops against the set-op output binding when a rewrite leaves a stale child binding in place. Add focused regressions for downstream joins over N-ary UNION ALL and for duplicated UNION ALL key projections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Map only provably bounded HUGEINT values to Spark DECIMAL(38,0), and retain trailing rewritten UNION bindings as aliases of their physical multiplicity output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve Spark/Postgres merge conflicts while preserving the feature branch's Spark add_months, set-op, and bounded HUGEINT work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A Spark/Delta temporal clause (`FROM t VERSION AS OF 366`) reached DuckDB's
parser verbatim, so any pinned relation died with
`Parser Error: syntax error at or near "as"` before a plan was ever built.
Downstream that surfaced as a silent correctness fallback: OpenIVM's
`openivm_compile_with_facts` produced no result, the refresh span reported
`compile_refresh_type=COMPILE_FAILED effective_refresh_type=FULL_REFRESH
reason=compile_failed`, and the pinned snapshot the user asked for was lost.
Input side (`lpts_input_dialect = 'spark'`): the temporal clause now normalizes
to DuckDB's semantically equivalent `AT (...)` clause —
`[FOR] VERSION|SYSTEM_VERSION AS OF <n>` becomes `AT (VERSION => <n>)` and
`[FOR] TIMESTAMP|SYSTEM_TIME AS OF '<ts>'` becomes `AT (TIMESTAMP => '<ts>')`.
The pin is represented, never dropped: dropping it would silently promote every
pinned scan to "read latest" and change the meaning of the query. A match
requires the full `<keyword> AS OF <literal>` sequence, so a column or alias
merely named `version`/`timestamp` is untouched, as are string literals. A
timestamp pin given a bare number is refused rather than mis-pinned.
Output side: LPTS already carried a pin as a DuckDB `AT (...)` suffix on the
table name (DuckLake time travel), but emitted that DuckDB spelling into every
dialect. The suffix is now rendered per dialect — Spark gets
`VERSION AS OF <n>` / `TIMESTAMP AS OF '<ts>'`, DuckDB keeps `AT (...)`, and a
dialect with no verified time-travel syntax raises
`LPTS_UNSUPPORTED_TIME_TRAVEL` instead of emitting SQL the target cannot parse.
Unqualified renderings (Postgres/Redshift/Feldera, inline SQL) went through a
raw `table_name` path that also leaked the suffix and then mistook its
parentheses for a table-function argument list (`products AT (VERSION => 2)
_tf("1")`); those paths now split the pin off explicitly.
`test/sql/time_travel.test` pins the contract end to end: normalization
fixtures for both temporal forms and the negative cases, and a DuckLake table
whose version 2 holds 2 rows while the latest holds 3, so a dropped pin is
observable in the result under `lpts_check`, not just in the generated SQL.
Note for the pinned-scan compile path: a plain DuckDB catalog now answers
`Binder Error: Catalog type does not support time travel` instead of a parser
error. That is the honest outcome — LPTS represents the pin and the catalog
decides — and it is the downstream signal OpenIVM must handle when it registers
schema-only fact tables.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Spark spells a time-travel pin between the relation and its alias (`FROM t VERSION AS OF 2 p`), DuckDB spells it after the alias (`FROM t p AT (VERSION => 2)`). The Spark input normalizer emitted the qualifier in Spark's position, so every aliased pinned relation produced `FROM t AT (VERSION => 2) p`, which DuckDB refuses to parse -- the same class of failure the normalizer was added to remove. The normalizer now reads the optional `[AS] alias [(column, ...)]` that follows the temporal clause and emits it ahead of the rewritten qualifier, preserving the original quoting and case. A bare token is only taken as an alias when DuckDB's grammar allows it there (the `ColId` rule: reserved and type/function keywords are excluded), so `WHERE`, `GROUP BY`, `CROSS`/`NATURAL`/`ANTI JOIN`, a comma and a closing paren continue the query instead of being swallowed. test/sql/time_travel.test replaces the text-only expectation that asserted the unparseable form with executable DuckLake coverage: real pinned scans with a bare alias, an `AS` alias, a quoted alias, a column alias list, alias-qualified projections and filters, and two aliased relations pinned to two different non-latest versions -- all run under `lpts_check`, so the regenerated SQL is executed and its rows compared. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate the HUGEINT error expectation from the following interval query so the ICU-enabled runner parses them as distinct records. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match Spark 3.5 DateTimeUtils.dateAddMonths: clamp only when the target month is shorter. Cover scalar folding, vectors, negative shifts, leap dates, NULL months and year-zero rollover with round-trip checking. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the pinned DuckDB formatting configuration; the standalone checkout's DuckDB style symlink is not initialized during the shared native build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mdrakiburrahman
changed the title
mdrakiburrahman
marked this pull request as ready for review
September 13, 2026 22:37
ila
added a commit
to ila/openivm
that referenced
this pull request
Sep 22, 2026
* feat: native Spark add_months scalar function DuckDB lacks add_months, which appears in Spark SQL fed to the openivm compiler and caused COMPILE_FAILED -> silent FULL_REFRESH demotion. Register add_months(DATE, INTEGER) -> DATE implementing full Spark semantics: day-of-month preserved, clamped to target month length, with the end-of-month rule (last day of source month maps to last day of target month). Reuses DuckDB Date API for leap-year-correct month lengths. Adds test/sql/spark_add_months.test covering scalar correctness and incremental MV maintenance (SIMPLE_PROJECTION delta + EXCEPT ALL parity). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: source add_months from lpts instead of a native openivm copy Per review, the add_months scalar function belongs in lpts (the Spark-compat layer openivm already builds), not duplicated in openivm. Move the implementation to lpts (cwida/lpts#18) and consume it here via the pin: - delete src/functions/spark_scalar_functions.{cpp,hpp} (now in lpts) - compile ${LPTS_DIR}/src/spark_scalar_functions.cpp from the lpts submodule and include its header from ${LPTS_DIR}/src/include (already on the path) - keep the thin RegisterSparkScalarFunctions(loader) call in openivm's LoadInternal (openivm does not invoke lpts's LoadInternal, so it registers the lpts-provided function itself) - bump third_party/lpts 13786cb..642c762 (cwida/lpts main + add_months) - trim test/sql/spark_add_months.test to openivm's concern (add_months resolves + drives a real SIMPLE_PROJECTION delta / EXCEPT ALL parity); exhaustive scalar-correctness now lives in lpts Local openivm CI green (build + full sqllogictest: 10020 assertions / 83 cases). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: bump third_party/lpts to include the SPARK dialect icu test guard Points the lpts submodule at 592d469, which adds `require icu` to dialect_spark.test so offline linux_amd64 CI skips the TIMESTAMPTZ-binding assertions instead of failing. Keeps this PR's add_months source (already relocated into lpts) building against green lpts CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Linearize LEFT-JOIN IVM delta via N-term telescoping The compile-only regular N-term telescoping delta (openivm_regular_nterm) was gated to INNER-only joins; LEFT-join SIMPLE_PROJECTION views fell back to the inclusion-exclusion path, which enumerates 2^N-1 subset terms and copies the full plan per term. A 15-LEFT-JOIN star model therefore compiled in ~3.5h. Extend the linear telescoping path to LEFT joins, mirroring the shipping DuckLake N-term path (BuildDuckLakeJoinTerms/DemoteLeftJoinsForMask): - Add HasOnlyInnerOrLeftJoins() and gate the LEFT branch behind a new openivm_regular_nterm_left setting (default true). - In BuildRegularJoinTerms, per term demote only the outer join whose NULL-supplying subtree contains that term's single delta leaf; other LEFT joins stay LEFT, preserving their NULL-padded rows. - NULL<->match transition correctness is completed by the upsert layer's key-based partial recompute (BuildLeftJoinProjectionRefresh). - Propagate the new setting through PropagateRefreshPlanningSettings. Result: the 15-LEFT-JOIN model compiles in ~0.5s (statement/term count linear in join count) with identical results. FULL OUTER / RIGHT shapes and the inclusion-exclusion FK-pruning path are unchanged. Adds test/sql/left_join_regular_nterm.test: a 5-LEFT-JOIN star with a mixed DML batch (NULL->match, match->NULL, dim/fact updates, fact insert/delete) that compiles via the telescoping path and is bag-equal to the recomputed base query in both directions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test: fix sqllogictest header ordering for format-check duckdb/scripts/format.py requires the `# group:` tag immediately after a single-line `# description:`; the multi-line description tripped the Code Quality Check / Format Check CI leg. Collapse the description to one line and move the elaboration below the group tag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advance lpts pin to 77ed5e5 and add arc-machine/int-instance regressions Bump third_party/lpts to 77ed5e585f26e2702a7b2af3cc9e8623769256c8 (LPTS PR #18, "Fix set-op column binding remaps"). The fork URL/remote in .gitmodules is untouched -- only the pinned commit moves forward. This pin alone fixes a stale UNION ALL child-alias remap surfaced by the reduced "int_instance_status_transaction" shape (duplicated UNION ALL key projections/output alias remapping that could leave a stale reference to a child alias no longer present in the rewritten plan). Extend test/sql/compile_refresh.test, compile_spark_dialect_hardening.test, and cascade_simple_projection_join.test with reduced regressions covering both previously-failing benchmark shapes: - arc_machine_status_transaction: a CTE join (INNER JOIN + LEFT JOIN) feeding an outer query with two further chained LEFT JOINs -- downstream projection/join over an N-ary join whose lhs binding could be lost or clobbered by sibling traversal. - int_instance_status_transaction: a UNION ALL of two LEFT JOIN branches partitioned by an IS NOT NULL / IS NULL predicate on the joined side, exercising duplicated UNION ALL key projections/output alias remapping. Compile-only assertions (openivm_compile_with_facts with compile_only=true, force_view_delta_cascade=true) confirm both shapes compile to a real incremental SIMPLE_PROJECTION delta program -- never FULL_REFRESH -- including a batched multi-leaf delta variant of the arc-machine shape. The cascade_simple_projection_join.test additions add a real CREATE + batched INSERT/DELETE/UPDATE + one PRAGMA refresh integration case per shape, with bidirectional EXCEPT ALL bag-equality checks against a from-scratch recompute. The arc-machine integration case (batched deltas spread across all 5 base tables) requires the native fix in the following commit to pass; it is included here to document the exact regression that fix guards against. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix stale leaf table_index reuse in inclusion-exclusion delta substitution BuildInclusionExclusionTerms (src/delta/operators/join.cpp) substituted each per-mask delta leaf using leaves[i].get, a LogicalGet* captured once from the ORIGINAL, un-renumbered input.plan before that mask's own renumber_and_rebind_subtree pass. CreateDeltaGetNode/CompactDeltaNode reuse old_get->table_index verbatim for the replacement delta subtree, so the substituted node ended up at the STALE, pre-renumbering table_index while every other reference to that leaf within `term` (join conditions, transitioning-key guards, projection maps) had already been rebound to the FRESH, mask-specific index renumber_and_rebind_subtree assigned it. This mismatch was invisible for single-leaf-delta cases because the existing AppendMultiplicityToAncestorProjectionMaps "shift_stale_parent_indexes" patch fixes up the immediate parent join's own projection map to tolerate it. It was NOT masked once the substituted leaf feeds more than one join (e.g. msf join acd AND acd join hw) or a transitioning-key guard, since those other references are never patched and stay dangling -- surfacing as LPTS_UNSUPPORTED_COLUMN_REF once a batched, multi-table delta hit that shape. A single-row compile-only check cannot reach this: it takes a real CREATE + batched INSERT/DELETE/UPDATE + PRAGMA refresh across every base table to trigger more than one leaf changing at once. Fix: substitute using term's own, already-renumbered LogicalGet at leaves[i].path (via GetNodeAtPath), instead of the stale leaves[i].get, so the replacement delta node's table_index is always consistent with the rest of `term`. BuildRegularJoinTerms/CompileRegularLeafDelta and the transitioning-key-guard delta path were checked and already re-derive their leaf/get pointers from term's own renumbered tree (the former via its own post-renumbering CollectJoinLeaves call, the latter via FindGetInSubtree/GetNodeAtPath walks of term's live join tree), so they did not need the same fix. Verified against test/sql/cascade_simple_projection_join.test's cspj_arcm_mv integration case (real CREATE + batched multi-table INSERT/DELETE/UPDATE + PRAGMA refresh + bidirectional EXCEPT ALL), which failed with LPTS_UNSUPPORTED_COLUMN_REF before this fix and passes after. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advance lpts pin to 754c797 and add HUGEINT/left-deep UNION canaries Advance third_party/lpts from 77ed5e5 to 754c797781c9b56429e05c1781cea5ca99c628e8 ("Fix bounded Spark HUGEINT and UNION aliases", mdrakiburrahman/lpts branch dev/mdrrahman/spark-add-months), preserving the existing fork URL contract in .gitmodules. Extend test/sql/compile_spark_dialect_hardening.test with two new canary reductions exercising the OpenIVM/LPTS integration boundary via openivm_compile_with_facts(..., compile_only=true, force_view_delta_cascade=true): - sph_hugeint_*: a plain (non-aggregate) projection widening BIGINT to HUGEINT via COALESCE(CAST(amount AS HUGEINT), 0). Before this pin, ANY cast to HUGEINT under target_dialect=spark unconditionally raised LPTS_UNSUPPORTED_TYPE; the fix maps a provably-bounded HUGEINT cast to Spark DECIMAL(38,0). A literal COALESCE(SUM(bigint), 0) is classified GROUP_RECOMPUTE by OpenIVM and bypasses LPTS entirely, so this canary uses the equivalent non-aggregate shape that is classified SIMPLE_PROJECTION and does reach LPTS. Verified via A/B pin flip: the exact same view throws LPTS_UNSUPPORTED_TYPE on 77ed5e5 and compiles to refresh_type=2 SIMPLE_PROJECTION with SQL containing DECIMAL(38,0) on 754c797. - sph_ms_*: a literal three-way (left-deep) UNION ALL of LEFT JOIN branches partitioned by a mutually exclusive predicate on the joined side, matching the shape of upstream lpts commit 754c797's own test/sql/union.test regression (a left-deep "machine status" UNION ALL carrying duplicated key/multiplicity output columns). Asserts SIMPLE_PROJECTION classification and correct binding of the trailing openivm_left_key/openivm_multiplicity columns across the left-deep chain. Add the same sph_ms_* shape (as cspj_ms_*) to test/sql/cascade_simple_projection_join.test with a real CREATE + batched multi-table INSERT/DELETE/UPDATE + PRAGMA refresh, asserting bidirectional EXCEPT ALL bag-equality against the view definition, per the existing cspj_arcm_mv/cspj_inti_mv convention in that file. Investigation note: extensive probing (15+ constructions covering N-ary join-delta cascades at 2-5 simultaneous leaves, literal N-ary UNION ALL, union-of-unions nesting, column-pruning bait, and real batched-DML + PRAGMA refresh cycles, cross-checked with temporary debug instrumentation in third_party/lpts/src/lpts_ast_builder.cpp, since reverted) did not reproduce LPTS's internal "trailing binding" condition (GetColumnBindings().size() > types.size() on a LOGICAL_UNION node) via OpenIVM's current native construction code; AssembleJoinUnionAll and CompileUnionDelta were confirmed to keep types and bindings self-consistent at construction time for every topology tried. The added sph_ms_*/cspj_ms_* tests therefore verify the observable, task-required integration-boundary contract (SIMPLE_PROJECTION classification, never COMPILE_FAILED/ FULL_REFRESH, correct left-deep UNION column binding, bidirectional bag equality) for this shape family under the new pin, rather than proving the exact internal LPTS code path fires; no native OpenIVM defect was found or hidden. Verified: format-check clean; targeted suite (compile_refresh.test, compile_spark_dialect_hardening.test, cascade_group_recompute_delta.test, cascade_window_partition_delta.test, cascade_simple_projection_join.test) 352/352 assertions passing (up from the prior 320/320 baseline); full make test 10172/10172 assertions passing across 84 test cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Emit cascade view delta from unscopable WINDOW_PARTITION/GROUP_RECOMPUTE recomputes A 32-way local OpenIVM Spark canary demoted 36 of 58 incremental materialized views to FULL_REFRESH with reason `non_cascade_upstream:non_cascade:<parents>`; 54 nodes reported a non-cascade upstream. The dominant roots were nine views that the classifier kept as WINDOW_PARTITION (`window_partition_kept`) yet which reported `emits_cascade_view_delta='false'`: operating_system_dim, arc_sql_server_instance_{version,edition,type,mode}_dim, subscription_{offer,workload}_type_dim, infrastructure_dim and arc_sql_server_target_instances_mat_view — together ~39 of the non-cascade upstream references. Root cause: the partial-recompute compilers degrade to a plain full recompute whenever the affected partition/group key set cannot be scoped from the source deltas, and that fallback silently discarded a requested `CompileFacts::force_view_delta_cascade`: * refresh_compiler_aux.cpp CompileWindowRecompute — no affected keys and (no partition columns or no partition delta spec). This is the unpartitioned surrogate-key shape `CAST(ROW_NUMBER() OVER (ORDER BY ...) AS INT)` used by seven of the nine dims, and the computed-partition-key shape (`PARTITION BY lower(a) || '_' || lower(b)`) used by operating_system_dim. * refresh_window.cpp BuildWindowPartitionRefresh — DuckLake fallback, lineage UNSAFE fallback, and lineage-incomplete multi-source fallback (arc_sql_server_target_instances_mat_view). * refresh_compiler.cpp CompileGroupRecompute — degenerate no-group / no-delta-spec fallback. refresh_sql.cpp sets `recompute_handles_own_cascade_delta` for WINDOW_PARTITION/GROUP_RECOMPUTE when a cascade delta is requested, which suppresses the generic snapshot companion on the assumption that the recompute emits its own delta. For these fallbacks that assumption was false, so the compiled program contained no `openivm_delta_<view>` write at all and every downstream view had to fall back to a full refresh. Fix: add CompileFullRecomputeWithCascadeDelta, used only when the caller explicitly requested a cascade delta. It brackets the existing recompute with `openivm_old_<view>` / `openivm_new_<view>` temp snapshots and publishes the exact signed multiset delta (whole old content at multiplicity -1, whole new content at +1) via the existing BuildSignedMultisetDeltaInsertSQL, then drops the temps. That is exactly `new_bag - old_bag`: unchanged rows contribute cancelling -1/+1 pairs, so bag semantics and inclusion-exclusion behaviour are preserved. The statement shapes match the non-degenerate WINDOW_PARTITION/GROUP_RECOMPUTE cascade branches already emitted today. Nothing is relabeled and nothing is force-refreshed: the refresh type stays WINDOW_PARTITION / GROUP_RECOMPUTE, data-table maintenance is unchanged, unsupported plans still fail or classify as FULL_REFRESH explicitly, and `PRAGMA refresh` (default CompileFacts, cascade off) is bit-for-bit unaffected. Adds test/sql/cascade_window_unscopable_delta.test covering the three canary query shapes (global surrogate-key window, computed partition key, multi-source join with a computed partition key), the no-cascade program shape, an end-to-end cascade in which the emitted program is executed after batched insert/update/delete and a downstream view is refreshed incrementally off the delta, bag-multiplicity preservation on a duplicate-heavy view, and the preserved unsupported-plan behaviour. The test fails on the pre-fix build at the first cascade-shape assertion. Native gate: 170/170 test cases, 20494 assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: bump third_party/lpts to b3baf0b Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: bump third_party/lpts to dbac36d Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: preserve Spark time-travel pins through MV compilation Compiling a materialized view whose body pins a Delta relation (`FROM billing_meter_dim VERSION AS OF 366`) previously failed with `Parser Error: syntax error at or near "as"`, so OpenIvmCompiler fell back to COMPILE_FAILED/FULL_REFRESH and silently lost the pinned snapshot. Three gaps are closed. Input parsing. A new `openivm_input_dialect` setting records the dialect the caller writes MV bodies in. `ParseMaterializedViewStatement` routes the body through LPTS `NormalizeInputSqlToDuckDB(query, dialect)` before `Parser::ParseQuery`, so `VERSION AS OF n` becomes DuckDB's `AT (VERSION => n)` instead of being rejected. `SqlDialect::DUCKDB` (the default) keeps the previous code path byte for byte. The dialect is mirrored onto a `MaterializedViewParserExtensionInfo` because the parser extension entry points receive no `ClientContext`. Binding. OpenIvmCompiler registers fact stand-ins as plain in-memory tables, which honestly reject a pinned scan with `Catalog type does not support time travel`. `TimeTravelPins` peels the `AT (...)` qualifier off each `BaseTableRef` whose catalog does not implement time travel, binds against the stand-in, and restores the qualifier onto the matching `AstGetNode::table_name` after `LogicalPlanToAst`. Peeling is driven by `Catalog::SupportsTimeTravel()` rather than by dialect, so DuckLake pins still bind natively and are never touched. Pins are keyed by qualified relation name and tracked through aliases, backticks, repeated scans, joins, CTEs (with proper CTE-name shadowing) and several differently pinned relations in one body. Genuinely ambiguous bodies -- two pins on one relation, one pin naming two relations, a relation read both pinned and unpinned -- raise rather than guess. No path reads latest silently and no qualifier is stripped globally. Output. Rendering a stored view for a foreign dialect re-attaches the pins, so Spark full-recompute and delta SQL emit `VERSION AS OF n` on the correct relation. DuckDB output stays pin-free (the stand-in catalog has no snapshots). Every other dialect fails loudly through LPTS with `LPTS_UNSUPPORTED_TIME_TRAVEL`. Alias association is verified rather than trusted. Spark writes the pin between a relation and its alias (`FROM t VERSION AS OF 2 p`) where DuckDB wants it after both (`FROM t AS p AT (VERSION => 2)`), so normalizing has to carry the alias across the rewrite -- the one step where a pin could land on a neighbouring relation, attach to the wrong alias, or be dropped, each of which reads a different snapshot while still compiling cleanly. `CollectSourceSnapshotBindings` reads every `{relation, alias, qualifier}` triple off the source text before normalization and `VerifySnapshotBindings` re-checks them against the parse tree DuckDB produced, raising rather than compiling on. It is deliberately an independent derivation of the same facts, so a regression in the normalizer fails the compile instead of passing quietly. `test/sql/time_travel.test` covers input parsing and dialect validation, binding and initial load, version and timestamp pins, relation/pin and alias/pin association (bare and `AS` aliases, a column-alias list, a deliberate alias/relation name collision, two bare aliases on differently pinned relations), the ambiguity refusals, the emitted SQL for spark/duckdb/postgres, and `EXCEPT ALL` cross-checks of incremental against full recompute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: keep time-travel pins on the paths that never reach the AST Three ways a pin could be lost or misplaced after compilation: - `StripFrom` tracked only the most recent identifier, so a pin written after an alias (`FROM t p AT (VERSION => 366)`, the shape normalization produces from Spark's `FROM t VERSION AS OF 366 p`) was credited to the alias and left in place. The view then executed locally against a catalog that cannot bind the qualifier and failed at CREATE. - Refresh programs assembled as SQL text rather than rendered from the AST (min/max aggregates, group recompute, interrupted-refresh recovery) never passed through `RestoreInto`, so they shipped to the target engine reading the latest snapshot instead of the pinned one. `RestoreIntoSql` re-attaches every pin, in the target dialect's own spelling via LPTS, to FROM/JOIN scans that do not already carry it. - `Peel` skipped unpinned scans in catalogs that honour pins natively. Re-attachment is keyed by relation name, so a pin peeled off one relation could land on a same-named relation that was deliberately read unpinned; that ambiguity is now refused like any other. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: translate text-carried time-travel pins in place `RestoreIntoSql` treated every pinned scan as if it were AST-rendered. Bodies that never reach the AST keep their pin in the normalized text, where DuckDB spells it after the alias, so the qualifier was inserted a second time in front of that alias and the raw one was left standing: `FROM t VERSION AS OF 366 o AT (VERSION => 366)` is not valid in any dialect. It also armed relation scanning on FROM and JOIN only, so in a comma FROM list every relation after the first went out unpinned, silently reading the latest snapshot. The scanner now looks past an optional `[AS] alias` for a raw `AT (...)` belonging to the relation, replaces it with the target dialect's spelling directly behind the relation and re-emits the alias after it, and keeps the FROM list armed across commas at its own parenthesis depth so every implicit cross join is pinned. Names bound by a `WITH` clause are collected up front so a CTE reference that shadows a pinned relation is never handed a qualifier it cannot carry. Translating stays idempotent, comment- and string-safe, and unsupported dialects still refuse. DuckDB-dialect output now strips the pins from these text-assembled programs instead of passing them through: they run against a catalog that holds no snapshots, where a raw qualifier failed the refresh with `Catalog type does not support time travel`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: keep a parenthesized join list in table position A parenthesis in table position opens either a derived table or a join list. Treating every one as a derived table dropped the relation expectation, so the first scan of `FROM (a JOIN b)` kept its DuckDB `AT (...)` spelling when the refresh SQL was assembled as text for another dialect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: pin third_party/lpts to final PR #18 head (6980a13) Update LPTS submodule gitlink from dbac36d to 6980a13 which includes the sqllogictest separator fix for the Spark dialect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: update stale auto_refresh profile statement count expectation (6→7) The AGGREGATE_GROUP MERGE refresh path produces 7 profiled statements: 1. SET refresh_in_progress = true 2. INSERT INTO openivm_delta_<view> (delta computation) 3. MERGE INTO openivm_data_<view> (applies delta) 4. DELETE FROM openivm_delta_<view> (view delta cleanup) 5. DELETE FROM openivm_delta_<source> (source delta cleanup) 6. UPDATE openivm_delta_tables (metadata timestamp) 7. SET refresh_in_progress = false The expectation of 6 was stale since the transactional lifecycle hardening added the 7th statement. All statements are distinct and required — no duplicate. The old comment incorrectly referenced the "full-recompute path" but this view uses the MERGE path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make profiling test deterministic by disabling adaptive refresh The profiling statement-count assertion was flaky because Test 18 left openivm_adaptive_refresh = true. After 3+ history entries, the adaptive cost model could nondeterministically choose full recompute (6 stmts) vs incremental MERGE (7 stmts) depending on prior refresh durations. Fix: bracket the profiled refresh with SET openivm_adaptive_refresh = false / true, so the assertion always sees the 7-statement incremental path. Re-enable immediately after so subsequent refreshes still record history for Test 20 (expected count 6→5 since the profiled refresh no longer records a history entry). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(ivm): keep hidden window keys out of target predicates Build lineage from every nested window key while persisting only projected MV columns as legal refresh partition targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix join multiplicity through passthrough maps Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Format integrated native sources for upstream checks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pin LPTS to merged main commit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use merged LPTS snapshot metadata and remove duplicate parsing * Remove redundant snapshot walks and obsolete SQL pin parsing * Share source qualification walk and preserve explicit native snapshots * Fix snapshot identity and dialect rendering during recovery * Reuse parsed queries and shared refresh utilities --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: ila <ilaria.battiston@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
add_months(DATE, INTEGER)with day clamping, non-sticky month ends, negative offsets and null propagation.VERSION/TIMESTAMP AS OFsnapshot pins and aliases through DuckDB normalization and Spark SQL rendering; reject unsupported target dialects.UNIONoutput arity, including OpenIVM's hidden-key and multiplicity columns.