feat(host): duckdb-wasm storage write dispatch + CONTRACT_DIGEST rotation#2
Open
zacharywhitley wants to merge 40 commits into
Open
feat(host): duckdb-wasm storage write dispatch + CONTRACT_DIGEST rotation#2zacharywhitley wants to merge 40 commits into
zacharywhitley wants to merge 40 commits into
Conversation
added 30 commits
June 30, 2026 05:19
Previously pulled tvm-core from a sibling ~/git/tvm-wasm path checkout. Pin it to the upstream git rev (6f3bae38) so ducklink builds standalone (no sibling checkout) and so a build linking ducklink-host alongside another tvm consumer resolves a single tvm-core source. See chimeradb/PLAN-inprocess.md PR1.
Extract the CLI run path into run_cli_inner (shared by run_cli_with_stdio) and add run_cli_capture, which drives the CLI with in-memory stdio (MemoryInputPipe fed a script, MemoryOutputPipe captured) and returns stdout. Enables running DuckDB in-process and capturing results without a subprocess. See chimeradb/PLAN-inprocess.md PR3.
…tures Add an additive `functions` array to every loadable extension entry in registry/index.json, carrying full signatures (argument types + return type / output columns) extracted by actually loading each component. The `exports` field only carried bare function names; this fills the gap so `ducklink_extensions()` discovery in the native extension can surface typed signatures. Extraction is done by tools/catalog-sig-extract, a new workspace binary that loads each component with the shared ducklink-runtime loader (load_component_with_dynlink), drains the neutral reg::* registration model, and maps each reg::LogicalType to the DuckDB SQL type name the native extension's src/reg_duckdb.rs registers it as (Int64 -> BIGINT, Float64 -> DOUBLE, Text -> VARCHAR, etc.). It emits a name -> functions sidecar; merge-functions.py folds it into the catalog with the same serializer the catalog generator uses (ensure_ascii, 2-space indent), so the catalog is changed only by the additive `functions` key (byte-identical after stripping it). Coverage: 193/193 entries enriched (571 scalar, 36 table, 14 aggregate functions); 0 skipped, 0 failed. Catalog still passes verify-catalog.py. No artifacts re-published, no deterministic build digests touched.
…dows) Establishes the versioning rules before the @4.0.0 cutover: lockstep major = contract generation (implemented decoupled via providers[].abi), N/N-1 support window, the separate DuckDB-version axis and its stable-C-ABI portability lever, and the tier portability table.
…ld versions) - EOL window set to 6 months after deprecation (one DuckDB cycle). - Clarify old-DuckDB-version support is limited only for the community channel; self-publishing to a custom_extension_repository covers it. - Mark the stable-ABI upstream ask on hold; mechanics in progress.
Compat breadth verified in a real v1.5.4 CLI: a gen-4 host runs gen-2 scalar/table/aggregate blobs. Resolution selects the newest provider whose generation <= host, falling back to the top-level digest; select_provider + the runnable predicate are the two spots to revise if a future generation breaks backward-compat.
Common tier is stable-v1.2.0-clean except duckdb-rs's Arrow-based SQL execute path, reached only by two con.execute calls (the ducklink_version COMMENT and the ducklink.* view creation). A stable-portable common tier = a libduckdb-sys stable-build mode + re-plumbing those two calls onto duckdb_prepare/execute_prepared/fetch_chunk. Small and scoped.
Supersedes the backward-compat model: cross-major is rejected uniformly. The native extension is tightened to match the ducklink-runtime CLI + browser cores (within-major forward-compat only). Requires the extension's bundled snapshot to be gen-4; ships in v4.0.1.
…ompile) Designed .py authoring tier: zero-build Python extensions by driving the pylon Python-wasm runtime via py-offload + the #228 resident-provider machinery. Manifest-via-offload discovery, arrow-columnar dispatch, CPython default; run + compile modes off the same .py. Not built yet.
Standalone, wasm-independent authoring surface for the Python source tier: - @ducklink.scalar/table/aggregate decorators that register into a module-level registry and return the original callable/class unchanged (still directly callable; invocable by entry="module:name"). - Type-hint -> DuckDB/WIT mapping (str->VARCHAR, int->BIGINT, float->DOUBLE, bool->BOOLEAN, bytes/bytearray/memoryview->BLOB, Optional[T] nullable); unmapped/missing hints fail at decoration time. - ducklink.runtime.manifest() -> JSON-serializable list[dict] the host reads via offload.run(entry="ducklink.runtime:manifest"). - PEP 723 inline-deps parser (reference regex + tomllib). - example/text_tools.py and a 50-test pytest suite (all green in CPython).
…ion/sqlink - compose-core: `branch = "main"` -> `rev = 416226921c` (matches datafission's exact pin at HEAD of webassembly-component- orchestration main; sqlink now aligned via a parallel commit). - datalink-dynlink: bumped from 88782655 to d1e5984b (39 commits behind current datalink HEAD, matching datafission/sqlink). The d1e5984b commit is `feat(dynlink): async deep-reentrant + streaming compose:dynlink host (#225)` — the last major dynlink API landing before the peer hosts aligned. Aligns the three peer hosts (datafission + sqlink + ducklink) on the same compose:dynlink wire shape so a shared provider artifact like the forthcoming postgis-composed-provider.wasm loads interchangeably across all three consumers.
Previous commit 9d59c00 misaligned the pin to d1e5984b (datafission's older pin) instead of the 80010ad that ducklink main was already on. This restores alignment: the feature branch and main both target 80010ad, so the merge is a no-op for this dep. Alignment target across all three peer hosts: datalink-dynlink = 80010adf7c9ec3d07674bba12585c787328217f5 compose-core = 416226921cc166e98016f534a73bd6c9b7dcb33c
…to a WASI-only 'database' component Make ducklink-core wac-composable into a standalone component with no native host: the loader-stub provides all 16 of the core's host imports. - callback-dispatch: migrate the stub to the major-4 columnar ABI (call-scalar-batch-col / call-aggregate-col / call-cast-col over colvec; the old row-major batch methods are gone). - Add no-op providers for the eight *-host backend callbacks (storage, index, collation, pragma, parser, optimizer, files, table-stream): each *-list enumerator returns empty (the core registers nothing and its built-in local/in-memory functionality is unaffected) and per-handle ops are unreachable. Sync the seven missing *-host WITs from duckdb-wasm. - Extend world loader-stub with the eight exports. 'wac plug ducklink_core --plug ducklink_loader' now yields a valid component exporting duckdb:component/database with only type-only residual imports (duckdb:extension/types, column-types, tvm:memory/types).
…or compose:dynlink loads
Ports the per-sub-extension composed-provider materialization from
`datafission::df-plugin-loader::UnifiedPluginLoader::materialize_sub_ext_provider`
(loader.rs:337) to `ducklink-host` so a `LOAD postgis_core` (or any
sub-ext bridge that imports `compose:dynlink/linker`) resolves through
a lazily-composed provider registered in the process-global
`ProviderRegistry`.
Adds `crates/ducklink-host/src/sub_ext.rs` with `SubExtLoader` — owns
the `sub_ext -> {plan, bridge, derived-from}` maps plus the CAS/cache
roots, composes the plan on first LOAD, materializes bytes to
`<blob_cas>/composed-providers/<sub_ext>-composed.wasm`, and registers
under `sub_ext_provider_id(sub_ext)`. Idempotent; a re-issued LOAD
hits the guard and no-ops. Chain-resolves zero-digest derived
components through the same recursion the datafission loader uses.
`ExtensionManager` grows a `sub_ext_loader` field seeded from three env
vars parallel to `DUCKLINK_PROVIDERS`:
DUCKLINK_SUB_EXT_PLANS=<name>=<path>:<name>=<path>...
DUCKLINK_SUB_EXT_BRIDGES=<name>=<path>:<name>=<path>...
DUCKLINK_SUB_EXT_DERIVED=<component-id>=<upstream-sub-ext>:...
`ensure_extension_loaded` consults `sub_ext_loader.has_bridge(name)`
BEFORE the flat `<extensions-dir>/<name>.wasm` resolver shortcut; when
it matches, the bridge branch composes the provider and loads the
configured bridge wasm instead. The standard resolver path is
untouched for every non-sub-ext extension.
Verified via `tests/sub_ext_materialize.rs`: three cases covering the
compose+register+idempotence flow, the PlanMissing error, and the
has_bridge / bridge_path routing that `ensure_extension_loaded`
consults. Also `sub_ext::tests::` unit tests for the provider-id +
zero-digest predicate + env parser.
drain_pending was ignoring the pending_scalar_ex buffer, so every runtime-ext.register-scalar-ex call captured on the store never made it into PendingRegistrationsData and thus never reached the DuckDB sink. LOAD saw zero registrations from provider components that used the richer scalar-ex path (varargs / optionally named args / special NULL-handling), even though the guest was still successfully pushing 961 entries into pending_scalar_ex. Only the standalone take_pending_scalar_ex accessor could reach them, and nothing wired into direction-1 or direction-2 called it. Add pub scalar_ex: Vec<PendingScalarEx> to PendingRegistrationsData so a future direct sink can consume the raw records, and drain the buffer into it inside drain_pending(). Mirror the change in append() so multi-instance batching preserves scalar_ex too. INTERIM: until the wasm-core pending-registrations WIT record grows a matching scalar-ex arm and the C++ core-shim consumes it, every scalar-ex entry is ALSO projected down into scalars at drain time (dropping varargs / special_null but preserving name / arguments / returns / callback_handle / options). That projection is documented as temporary on the PendingRegistrationsData doc-comment and inside drain_pending; a follow-up must remove it once the direct sink lands to avoid double-registration. Extend the drain summary eprintln with the scalar_ex count so batch diagnostics reflect the drained buffer. Add a drain unit test that locks in both the raw entry preservation and the interim projection; extend the append merge test to cover the new field.
…ined monolithic providers The plan-driven per-sub-ext compose emits artifacts with unresolved external imports (postgis-core-composed had 19 imports from flatgeobuf/kml/geobuf/ marc21/gml/ttf/rustybuzz/gdal/sfcgal), so wasmtime instantiation traps at first `resolve-by-id`. DUCKLINK_SUB_EXT_PREBUILT lets an application register a self-contained composed wasm directly under `<sub-ext>-composed`, bypassing the plan-driven compose entirely. Population order in materialize_sub_ext_provider: 1. materialized guard (no-op on repeat LOAD) 2. prebuilt shortcut (new — file → register_provider → done) 3. plan-driven compose (existing normal path) Also populates composed_digests with sha256(prebuilt bytes) so chained sub-ext plans (postgis_raster depending on postgis_core-composed) can resolve the upstream through the existing invariant. Tested end-to-end with LOAD postgis_core + SELECT st_astext / st_area / st_distance / st_make_point / postgis_version returning correct results through the wasm-component pipeline.
…nsData Previously `PendingRegistrationsData` + `drain_pending()` + `append()` only covered ~9 of the ~27 `pending_*` buffers on `ExtensionStoreState`. The other ~18 (indexes, files, collations, pragmas, copy_handlers, secrets, settings, table_macros, modified_types, enum_types, conn_callbacks, coordinate_systems, arrow_tables, encodings, compressions, parsers, optimizers, filterable_tables) only surfaced through eager `take_pending_*` accessors that a direction- specific host had to remember to call. Buffers no direction eagerly drained were silently dropped at drain time -- the same class of bug as the pre-cc3eadb scalar_ex miss. Extend `PendingRegistrationsData` with a matching field per pending_* buffer; extend `append()` and `drain_pending()` to cover every field. Add a compile- time exhaustiveness lock: the new `PendingKind` enum's `as_str` match is exhaustive, and a const assert on `PendingKind::ALL.len()` fails compilation when a variant is added but ALL is not updated. Unit tests push a synthetic entry into every buffer and assert `drain_pending()` surfaces one entry per field.
`apply_function_prefixes` iterated scalars/tables/aggregates/macros but not
scalar_ex, so a `register-scalar-ex` registration only picked up prefixed
shadows through the interim projection into `scalars` performed by
`drain_pending`. That projection is temporary: the moment it is retired, the
prefixed-dispatch surface silently breaks for scalar_ex. Even today, any
downstream code that reads scalar_ex directly cannot see the prefixed shadows.
Add a SCALAR_EX block that mirrors SCALARS: for each scalar_ex entry, record
the collision + stage the __ducklink_prefix_function row, retain a new
`RetainedDef::ScalarEx` arm on prefix_retained so a pin resurrects the richer
shape, and append the `{prefix}__{name}` shadow to `data.scalar_ex`. Extend
`pin_macro_sql` to build the macro-alias parameter list from the ScalarEx
arm's args. Unit test locks in that both bare and prefixed names surface and
that the retained slot holds a ScalarEx arm (not a projection to Scalar).
The postgis-wasm and datafission READMEs advertise a pure-monolith setup that sets only `DUCKLINK_SUB_EXT_PREBUILT=...` — no `DUCKLINK_SUB_EXT_BRIDGES` — but `has_bridge()` only checked the bridges map. LOAD never took the sub-ext branch, materialization never ran, and the docs quietly did not work. Widen `has_bridge()` to also match prebuilt entries so the sub-ext branch runs, and fall back `bridge_path()` to the prebuilt path so the `has_bridge` ⇔ `bridge_path.is_some()` invariant the LOAD path relies on (via `.expect(...)`) still holds. An explicit BRIDGES entry keeps precedence, so the plan-driven case is unchanged.
The hand-maintained `PendingKind::ALL` array plus its literal-27 const assert did not catch a new variant added without extending `ALL`: the assert only fires on drift when a developer remembers to also bump the literal. Same bug, same failure mode. Swap the mirror array for a `strum::EnumIter` derive on the enum, expose the iterator through a new `PendingKind::all()`, and drop the const assert. The exhaustive `as_str` match is now the sole compile-time drift guard for the label surface — and the derived iterator makes an out-of-sync `ALL` structurally impossible. Callers migrate from `PendingKind::ALL.iter()` / `.len()` to `PendingKind::all()` / `.count()`. `strum` is stable Rust (no nightly needed) and pulls its derive macro in via the `derive` feature, so no separate `strum_macros` dependency.
The prebuilt-shortcut branch cached `sha256(prebuilt)` under `composed_digests` but never wrote the actual bytes into the compose CAS. In pure-monolith mode nothing else looked at that CAS entry, so the bug hid. Mixed mode surfaces it: postgis_core registered as a prebuilt monolith + postgis_raster driven by a plan whose derived-from chain references postgis_core. The raster compose walks the chain, splices the cached prebuilt digest into the raster plan, and hands off to compose-core — which calls `blobs.get(digest)` and fails with `BlobNotFound` because the shortcut never put the bytes in. Fix by reading the prebuilt once and calling `blobs.put(&bytes)` at the shortcut site (its returned digest matches what compose-core computes, so it is byte-identical to the previous cached value). Add a `mixed_mode_prebuilt_upstream_seeds_cas_for_derived_plan` integration test that composes a raster-shaped plan chained to a prebuilt-shortcut postgis_core and asserts the round-trip `blobs.get(digest)` succeeds. Also point the pre-existing `prebuilt_shortcut_registers_wasm_and_skips_compose` unit test at a tempdir CAS so the added `put` call doesn't drop a `.compose/blobs/` next to the crate root.
Same aba-core logic as the WASM aba-component, bound directly to DuckDB via duckdb-rs VScalar. Ships as its own community extension `aba`, independent of ducklink. Head-to-head on 1M-row aba_validate over random routing numbers: native aba 10-12 ms (~10 ns/row) WASM aba via ducklink 194-206 ms (~200 ns/row) ratio ~20x Same three functions: aba_validate(text) -> boolean aba_frb_district(text) -> int64 (NULL for unrecognized prefix) aba_fed_region(text) -> text (NULL for unrecognized prefix) Proves the dual-build pattern from ducklink-extension's docs/dual-build-native-and-wasm.md: same pure-logic crate, two bindings (WASM + native), two independent community extension distributions, users pick the tradeoff. Files: Cargo.toml cdylib + rlib, duckdb-rs loadable-extension Makefile ducklink-native's community-extensions build pattern description.yml manifest for the community-extensions PR src/lib.rs 3 VScalars, aba_init_c_api entry point test/sql/smoke.test 10 correctness assertions extension-ci-tools symlink to the ducklink-native's copy
Reorganizes the aba pilot into a single ducklink_native community extension. One PR, one release cycle, one description.yml for the whole curated set — as agreed in ducklink-extension's docs/dual-build-native-and-wasm.md. Bundled today: aba_* (3) — US ABA routing checks (aba-core) iban_* (3) — Intl bank account checks (iban-core) isbn_* (2) — ISBN check + normalize (isbn-core) luhn_* (2) — Luhn checksum + check-digit (luhn-core) cc_* (7) — Credit-card validate/brand/mask/.. (creditcard-core) Same pure-logic crates the ducklink WASM catalog uses — the only new code is the duckdb-rs binding. Two shared VScalar adapters (BoolAdapter, TextAdapter, IntAdapter) mean the boilerplate is one zero-sized-type impl per function. Adding a capability = drop a new src/scalars/*.rs file + add its -core dep to Cargo.toml + append register_* calls in loadable::init(). Also drops the standalone native-extension/aba/ from the earlier pilot — superseded by this bundle.
Uploads a .duckdb_extension to R2 at
native/sha256/<digest>/<platform>/<name>.duckdb_extension with
application/octet-stream + immutable cache-control, matching the layout
in src/catalog.rs::NATIVE_BLOB_BASE in ducklink-extension.
Idempotent (digest-keyed layout). Auto-detects platform from
`duckdb -c 'PRAGMA platform;'`. Reads creds from
${R2_ENV_FILE:-~/git/datalink/r2.env}.
Phase 9.1: sub-exts whose composed provider IS an upstream sub-ext's
provider (rather than a distinct shim) route through a new alias map
instead of registering their own provider id. Concrete case:
postgis_topology / _3d / _metadata / _clustering all share
postgis_core's composed wasm — one 30 MB shim serves all 5
postgis_core-family bridges instead of 4 duplicate ProviderRegistry
entries.
New field `SubExtLoader::sub_ext_shim_alias: BTreeMap<String,String>`,
env-var `DUCKLINK_SUB_EXT_ALIAS=<sub>=<upstream>:...`. New branch in
`materialize_sub_ext_provider` recurses into the upstream (before the
prebuilt shortcut and the plan-driven compose) and returns the
upstream's provider id. Both sides get marked materialized so a
repeat LOAD of either the aliased sub or the upstream is a no-op.
Aliased bridges are emitted with `--provider-id <upstream>-composed`
so their `compose:dynlink/linker.resolve-by-id` import lands on the
shared registration at instantiate time.
Adds `shim_alias_recurses_to_upstream_and_shares_provider_id` test
covering the postgis_topology / postgis_3d aliasing shape; asserts
resident_count("<sub>-composed") is 0 for the aliased subs and only
the upstream id gets registered.
Phase 9.1 aliased-only sub-exts skipped the sub-ext branch in ExtensionManager::ensure_extension_loaded because has_bridge() only consulted the direct prebuilt/bridge maps — it didn't fall through to the shim-alias entry. Aliased subs still need their own bridge_paths entry (the aliased bridge wasm), but the shim composed-provider comes from the upstream — so has_bridge for a sub-ext with only an alias but no direct maps must return true when the alias upstream has a prebuilt. Manifests as `LOAD postgis_topology` failing with "no plan registered for sub-extension" when only postgis_core has the prebuilt configured. After this fix, LOAD postgis_topology (alias→postgis_core) resolves the shared postgis_core-composed provider, reuses the ONCE-instantiated resident, and dedupes cleanly — verified live against ducklink CLI with the freshly self-contained postgis-core-provider-composed.wasm.
Adds `tests/write_boundary.rs` -- the mirror-image counterpart to the read-side `tests/boundary.rs`. Loads the synthetic ducklink writer bridge (companion to sqlink's `synthetic-mutating-vtab-bridge`) and drives its exported `duckdb:extension/storage-write-dispatch@2.0.0` interface end-to-end across the WIT component boundary via wasmtime, covering the full mutating path: begin-transaction, create-table, insert-rows, update-rows, delete-rows, commit-transaction, and the rollback path. The write world is `duckdb:extension@2.0.0`; the pre-existing wit/ tree uses the unversioned `duckdb:extension` package. Rather than version-mix, the write-side WIT lives in a sibling `wit-write/` dir with its own duckdb:extension@2.0.0 dependency, keeping the two bindgen! invocations independent. Bridge wasm is located via (1) $DUCKLINK_WRITER_BRIDGE_WASM, (2) `../../artifacts/extensions/synthetic_mutating_ducklink.wasm` (the staged copy convention), or (3) the sibling checkout at `~/git/synthetic-mutating-vtab-ducklink-bridge/target/wasm32-wasip2/ release/synthetic_mutating_ducklink_bridge_dynlink.wasm`. Test panics with a clear build-me message if none of those exist.
added 10 commits
July 13, 2026 21:52
The `wit/` tree used by the read-side `boundary` test had `callback-dispatch.wit` and the new `column-types.wit` pinned to `duckdb:extension@4.0.0` (major-4 columnar model) while every sibling WIT was still unversioned. wasmtime's WIT parser rejects a directory whose files disagree on package identity, so `cargo test -p storage-boundary-test --test boundary` failed at bindgen with: package identifier `duckdb:extension@4.0.0` does not match previous package name of `duckdb:extension` Harmonize to `@4.0.0` (matching `crates/ducklink-runtime/wit/deps/duckdb-extension` and `extensions/sqlitewasm-component/wit`, which the built `sqlitewasm.wasm` artifact was compiled against): bump the package header of every sibling WIT in `crates/storage-boundary-test/wit/` and add the trivial `column-types::Host` marker impl the boundary test's `State` needs now that the world transitively references the columnar type interface. The write-side `wit-write/` tree (pinned to `@2.0.0`, exercised by `write_boundary`) is untouched. Both tests now pass: - test pushdown_crosses_wit_boundary ... ok - test write_dispatch_crosses_wit_boundary ... ok - test write_dispatch_rollback_crosses_wit_boundary ... ok
Extend duckdb:extension@4.0.0 storage-host.wit with 7 host-import
callbacks that mirror the ExtensionInstance write trampolines
(storage_{begin,commit,rollback}_transaction,
storage_{create_table,insert_rows,update_rows,delete_rows}).
The core imports these; the host resolves the (extension, callback-
handle) pair via the same resolve_storage_backend picker the read side
uses and forwards to the writable component's storage-write-dispatch
export. Rotates CONTRACT_DIGEST (auto-computed by ducklink-runtime's
build.rs from the canonical WIT bytes).
Add 7 host-trait methods (storage_{begin,commit,rollback}_transaction,
storage_{create_table,insert_rows,update_rows,delete_rows}) on
`impl core_storage_host::Host for CoreStoreState`, each converting
core-WIT payloads to extension-WIT payloads and forwarding through
ExtensionManager.
Add 7 sibling `dispatch_storage_*` methods on ExtensionManager that
resolve the storage backend via the same resolve_storage_backend picker
the read-side scan uses and forward to the ExtensionInstance's write
trampolines (crates/ducklink-runtime/src/extension.rs:3022-3127).
Also adds the inverse conversion helpers
(convert_core_{columndef,logicaltype}_to_extension) mirroring the
extension_to_core helpers already used by the read side.
Sweeps the fleet's pinned witcanon@4.0.0 digest from
a2ad9764ac971345d6a650b92edbda034b160980acf148d354126f7e6f92ba40 to
3767a37476d93949d4fd0e724456ac1bed4608674e66214e019160698daab854 --
the digest ducklink-runtime's build.rs now computes over the extended
canonical WIT bytes.
Rotations:
* registry/index.json - 193 catalog entries (wit_contract field).
* docs/v4-columnar-abi.md + website/docs/architecture/columnar-abi.md
- documented current digest.
* website/docs/guides/distribution.md - documented current digest.
Verified with `python3 tooling/verify-catalog.py`: 192 entries pass the
identity check against the new digest.
Test fixtures in resolver.rs use a placeholder ("90fdc46a585c") that
does not need rotation -- it is a fake, not the runtime digest.
Matches the synthetic ducklink writer bridge's WIT bump (v0.1.0 -> v0.2.0)
so the write-boundary integration test continues to load the bridge:
* wit-write/deps/duckdb-extension/{types,storage-write-dispatch}.wit --
bumped from @2.0.0 to @4.0.0. Files are byte-identical to the ducklink
canonical set apart from the version line.
* wit-write/deps/duckdb-extension/storage.wit -- new. The bridge is now
full-stack, so guest.load() calls storage.register-storage; wasmtime
refuses to instantiate a component with an unsatisfied import, so the
test host now provides a no-op `storage` implementation. Copied from
the ducklink canonical.
* wit-write/boundary.wit -- world bumped to boundary-test:write@0.2.0,
imports duckdb:extension/storage@4.0.0 alongside the existing
storage-write-dispatch@4.0.0 export.
* tests/write_boundary.rs -- adds `types::Host` + `storage::Host` no-op
impls on `State` and wires them via WriteBoundaryTest::add_to_linker.
The test itself is unchanged; it still drives the write-side
trampolines directly and never actually calls guest.load().
The end-to-end LOAD + ATTACH + CRUD path is covered by the
`synthetic-mutating-ducklink` shim-bridge-smoke case.
propagate-wit.py now treats `${DUCKDB_WIT_ROOT}/wit/duckdb-extension/*.wit`
(default: sibling `~/git/duckdb-wit` checkout, overridable via env) as the
canonical source of truth for the pure `duckdb:extension` engine surface
(`*-host`, `callback-dispatch`, base types) and copies it byte-for-byte into
the local mirror.
The local mirror is a SUPERSET of upstream: ducklink layers its dispatch
WITs (`storage-dispatch`, `storage-write-dispatch`, `aggregate-incr-dispatch`,
`register-*`) plus the multi-world `worlds/duckdb-extension.wit` on top of
the pure-engine surface, all under the same `duckdb:extension@4.0.0`
package. The sync therefore:
- COPIES upstream files into the local mirror (byte-for-byte).
- LEAVES local-only files alone (no prune -- ducklink's dispatch WITs and
`deps.toml` are legit local additions, not upstream drift).
- SKIPS `worlds/duckdb-extension.wit` (local defines ~15 additional worlds
on top of upstream's single base world; overwriting would silently break
every extension that targets a superset world).
The pin phase (package + foreign-ref @<CONTRACT_VERSION>) is unchanged.
`--check` reports upstream drift + contract drift separately.
The REPL's line-join logic joined multi-line input with a single
space — `-- comment\nSELECT 1;` became `-- comment SELECT 1;`,
which DuckDB's parser reads as one giant line comment that
swallows the following statement. Any `.sql` case with `-- `
leading comments in the body was effectively broken (silent
statement drop, no error).
Fix: join with `\n` instead of ` `. DuckDB's SQL comment tokenizer
correctly terminates `--` at end-of-line, so the trailing SELECT
now parses as its own statement. Interior newlines are already
valid between SQL tokens, so multi-line statements like
`SELECT\n foo\nFROM bar;` remain unaffected.
Not fixed here: trailing-comment case (`SELECT 1; -- trailing`
where trimmed line doesn't end with `;` — the naive
`.ends_with(';')` check misses it and the statement never
dispatches). That's a separate pre-existing gap in the parser
that needs a `--`-aware line splitter to fix properly.
Verified manually against `~/git/ducklink/target/release/ducklink`
after rebuilding the CLI wasm component: piping
`-- comment\nSELECT 1;` now produces the `SELECT 1` result row.
…e-rows
Wire the additive storage-host@4.0.0 fields (upstream in duckdb-wit
c6d9646) through ducklink's local WIT mirror, dispatch layer, host
trampolines, and boundary tests so DuckDB's UPDATE/DELETE physical
plans (issued by duckdb-wasm's WasmPhysical{Update,Delete} Sink, see
duckdb-wasm e42db9e) round-trip end-to-end into a writable storage
component's storage-write-dispatch export.
WIT (canonical + propagated per propagate-wit.py managed globs):
* wit/duckdb-extension/storage-host.wit synced from duckdb-wit.
* wit/duckdb-extension/storage.wit scan-request.wants-rowid: bool --
the ducklink-local twin of the upstream storage-host field, so the
same semantic crosses the storage-dispatch layer to the guest.
* wit/duckdb-extension/storage-write-dispatch.wit update-rows
gains updated-columns: list<u32>. Each rows[i] is a PARTIAL row
carrying only the SET cells; updated-columns[c] is the schema
index of rows[i][c]. Mirrors DuckDB's LogicalUpdate::columns.
* Frozen copies under crates/ducklink-runtime/wit/, storage-boundary-
test/wit{,-write}/, table-stream-boundary-test/wit/, and every
storage-capable extension component (mysqlwasm, numstream,
postgreswasm, sqlitewasm, unityscan) synced in step.
Host wiring:
* CoreStoreState::storage_update_rows (crates/ducklink-host/src/lib.rs)
accepts updated_columns and forwards through
ExtensionManager::dispatch_storage_update_rows -> ExtensionInstance
(crates/ducklink-runtime/src/extension.rs storage_update_rows).
* convert_core_scan_request_to_storage propagates wants_rowid
verbatim from the core storage-host side to the storage-dispatch
ScanRequest handed to the backing component.
Boundary tests:
* crates/storage-boundary-test/tests/write_boundary.rs updates the
update_rows callsite: pass `updated_columns=[0, 1]` alongside the
full-width SET row (parity with the pre-M2c full-row semantic).
* crates/storage-boundary-test/tests/boundary.rs constructs
ScanRequest with wants_rowid: false on both pushdown + full-scan
paths (read-only tests never want rowids).
* Both tests pass (verified with a freshly-rebuilt sqlitewasm
component + synthetic writer bridge staged into artifacts/).
Extension components:
* sqlitewasm-component: src/bindings.rs regenerated by cargo
component build (new field on ScanRequest, new param on
update-rows). Its lone in-source ScanRequest construction in the
storage-dispatch test module gets wants_rowid: false.
* Every other storage-dispatch/storage-write-dispatch exporter that
is not test-driven this session (mysqlwasm, numstream,
postgreswasm, unityscan) has only its frozen WIT copy updated;
their generated bindings.rs will regenerate next time cargo
component build is run for the crate. They are not exercised by
the ducklink smoke matrix, so the deferred rebuild is safe.
Verified by:
* shim-bridge-smoke-tests: postgis-ducklink, mobilitydb-ducklink,
timescaledb-ducklink, synthetic-mutating-ducklink,
synthetic-mutating-provider, synthetic-mutating all pass.
* cargo test -p storage-boundary-test --release: 3 tests pass
(pushdown_crosses_wit_boundary, write_dispatch_crosses_wit_boundary,
write_dispatch_rollback_crosses_wit_boundary).
The CONTRACT_DIGEST rotation over the extended canonical WIT bytes
is a follow-up commit so this change stays reviewable.
…e extension
Sweeps the fleet's pinned witcanon@4.0.0 digest from
3767a37476d93949d4fd0e724456ac1bed4608674e66214e019160698daab854 to
b24dc901568e38e5b05fc0dc535ebaec3c93a9a44c772bb105686032e45e16b3 --
the digest ducklink-runtime's build.rs now computes over the
extended canonical WIT bytes (see the sibling commit that added
wants-rowid to scan-request + updated-columns to storage-update-rows).
Rotations:
* registry/index.json -- 215 hits across the 193 catalog entries
(some entries carry the digest in more than one wit_contract field).
* docs/v4-columnar-abi.md
* website/docs/architecture/columnar-abi.md
* website/docs/guides/distribution.md
Verified with `python3 tooling/verify-catalog.py --no-artifacts`:
192 entries pass the identity check against the new digest.
The catalog's 193 loadable components still carry frozen copies of
the storage.wit and storage-write-dispatch.wit files at their old
scan-request / update-rows shapes -- rebuilding each is a follow-up
per-component chore (only 5 exporters of storage-dispatch and 1 of
storage-write-dispatch actually consume the changed types; the rest
either use storage.wit for register-storage-only and are ABI-neutral,
or never got loaded at the changed types anyway). None of the
extension components exercised by the current smoke matrix
(postgis, mobilitydb, timescaledb, synthetic-mutating*) are storage
backends, so they are unaffected.
Test fixtures in resolver.rs use a placeholder ("90fdc46a585c") that
does not need rotation -- it is a fake, not the runtime digest.
…d storage WIT Regenerate wit-bindgen output + rebuild the four remaining storage-capable exporter components (mysqlwasm, postgreswasm, sqlitewasm, unityscan) against the additive storage-host@4.0.0 fields introduced in 828561b: `scan-request.wants-rowid: bool` and `storage-write-dispatch.update-rows. updated-columns: list<u32>`. Prior commit only refreshed the sqlitewasm.wasm artifact and its bindings (the one storage exporter that lives in the main cargo workspace). The three standalone-workspace crates (mysqlwasm/postgreswasm/unityscan) still shipped pre-built .wasm carrying the OLD ABI even though their frozen wit/ copies had been synced via propagate-wit.py. They loaded fine today because none of them are on the smoke matrix, but would have failed `LOAD <name>` under the new host once component instantiation type-checked the scan-request record. No guest code changes: all four crates use struct-init positional access on `ScanRequest`, so adding the additive `wants_rowid` field flows through bindgen with no source edits. `run_scan` intentionally ignores wants_rowid today -- these are read-only backends, DuckDB's UPDATE/DELETE physical plans don't route through them. Artifacts (all rebuilt at cargo component build --target wasm32-wasip2 --release, staged into artifacts/extensions/<name>.wasm per the existing convention): mysqlwasm.wasm 280299 bytes (was 352653) postgreswasm.wasm 278668 bytes (was 347619) sqlitewasm.wasm 1984317 bytes (was 1981537, same-day re-stamp) unityscan.wasm 1707817 bytes (was 1965937) registry/index.json: content_digest for sqlitewasm re-stamped by tooling/gen-catalog.py (the only exporter in the registry; the other three resolve via extensions_dir at load time). Digest rotates from 2343406b43bd… -> 7bee6176238e… Verification: * tooling/propagate-wit.py --check clean (2415 WIT files in sync) * tooling/verify-catalog.py --verify-content 192/192 verified * smoke matrix (from shim-bridge-smoke-tests): postgis-ducklink, mobilitydb-ducklink, timescaledb-ducklink, synthetic-mutating-ducklink -- all pass * sanity: `LOAD mysqlwasm/postgreswasm/sqlitewasm/unityscan;` under the current host contract (b24dc901568e) reports `extension '<name>' loaded successfully and ready for registrations` for all four -- no ABI/contract-mismatch on component instantiation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Host-side write dispatch for the mutating-vtab tier, bringing ducklink to parity with sqlink's mutating-vtab dispatch surface.
9e3c5ee— host impl + dispatch (+254 LOC)ae1e378— CONTRACT_DIGEST rotation across 4 files + 193 registry entries1aa7b88— write-side boundary WIT bump @2.0.0 -> @4.0.0267f31d— WIT extension (+40 LOC)Issue / motivation
Ducklink parity with sqlink's mutating-vtab dispatch tier. See:
tegmentum/sqlink@5cfc37dc— iVersion=2 fixtegmentum/sqlink@6ec38652— read/write unifyCoordination — all four PRs land together
Merging this alone leaves the tree in an intermediate broken state:
Sibling PRs