diff --git a/docs/rfcs/0033-blob-management.md b/docs/rfcs/0033-blob-management.md index e960864f5..1aee21623 100644 --- a/docs/rfcs/0033-blob-management.md +++ b/docs/rfcs/0033-blob-management.md @@ -7,7 +7,7 @@ implementation: partial authors: - OmniGraph maintainers created: 2026-08-09 -updated: 2026-10-10 +updated: 2026-10-11 discussion: null supersedes: [] superseded_by: [] @@ -27,10 +27,11 @@ and Lance 10.0.0 as the required implementation substrate. **Audience:** engine, compiler, server, CLI, security, storage, maintenance, and documentation maintainers. **Current rollout:** Phases 0, 1, 2A (HTTP read delivery), and 2B (CLI read -delivery) are implemented on the v0.10 development line. Phase 3 (mutation) is -planned against current main in §13, with its decisions in the 2026-10-08 -decision-log entry. Phase 4 (measured optimization) and the production delivery -telemetry named in §11 remain open. +delivery) are implemented on the v0.10 development line. Phase 3 (mutation: +put and clear through the engine, HTTP and the CLI) is implemented for v0.13.1, +with its decisions in the 2026-10-08 and 2026-10-10 decision-log entries. +Phase 4 (measured optimization) and the production delivery telemetry named in +§11 remain open. This RFC is the required successor to an earlier reverted blob-delivery experiment. The relevant evidence and decisions are restated here; no untracked @@ -1267,14 +1268,15 @@ They exist for three reasons: - a mutation or load holds its batches in memory until its one publication; - Lance writes a Blob value from an in-memory Arrow array. -The ceilings are not a setting. The evidence behind the figure is +A setting cannot raise the ceilings: `write_max_bytes` (§4.3) only lowers them, +for a session or a process, and 32 MiB is its maximum. The evidence behind the figure is [RFC 0023's cost gate](0023-key-conflict-fencing.md#114-cost-gate): paired peak-RSS overhead of bounded fenced transactions under the old combined 32 MiB accounting, for the workloads that gate names. It is not a workload-independent process-memory bound, and it does not qualify the split envelope of §4.3, up to 32 MiB of payload plus 32 MiB of framing; qualifying that remains open work for -the same gate. A setting without such a measured multiplier would promise a -bound it cannot keep. Going beyond the ceilings means keeping large data out of +the same gate. A setting that raised the ceilings without such a measured +multiplier would promise a bound it cannot keep. Going beyond the ceilings means keeping large data out of the write path's memory. There are three separate limits, each with its own dependency and trigger: @@ -1673,6 +1675,8 @@ correctness gate. ### Phase 3 — mutation +Implemented: 3A in #908, 3B in #922 and 3C in #928. + - **Served-read prerequisite:** met. The routed `GraphRequest` lease and the read observer are held by the GET/HEAD response through EOF; the bounded transport tests of §12.4 that assert it for Blob land with 3B. The schema-token @@ -1698,8 +1702,12 @@ correctness gate. - Batched reads (`read_blobs`) landed for materializing rewrites, export, the change-feed baseline, change images and entity reads (§4), and schema apply - stopped rewriting Blob columns (§8.4). Remaining: benchmark. Tuning is - optional and may not weaken the batched contract or change logical behavior. + stopped rewriting Blob columns (§8.4). Remaining: benchmark. The active + benchmark catalog runs GQ operations only, and Blob reads and export are not + GQ operations, so that benchmark waits for a harness that can drive them; + until then `export.rs::export_reads_each_batch_of_managed_blobs_in_one_read_per_column` + pins the read count. Tuning is optional and may not weaken the batched + contract or change logical behavior. - Retain the exact empty/null/neighbor compaction guard across every future Lance dependency bump. - Consider descriptor-preserving unchanged-cell updates only with an ownership @@ -1930,6 +1938,14 @@ publisher architecture. ## Decision log +- 2026-10-11: Phase 3 is implemented: the engine writes (#908), `PUT` and + `DELETE /blob` (#922) and `blob put` / `blob clear` (#928). The Phase 4 + benchmark waits for a harness that can drive Blob reads and export, which are + not GQ operations. Superseded: the header's "Phase 3 (mutation) is planned + against current main in §13"; and §10.1's "The ceilings are not a setting." + and "A setting without such a measured multiplier would promise a bound it + cannot keep.", since `write_max_bytes` now lowers the ceilings but cannot + raise them. - 2026-10-10: Step 3-pre landed as part of the `write_max_bytes` session setting rather than on its own. Row data and logical Blob payloads get independent allowances of `write_max_bytes` each, 32 MiB by default and at