Skip to content

fix(sqlite): retain fast COUNT and add D2 qualification checks - #4328

Merged
huangruiteng merged 5 commits into
loopx-project:mainfrom
LIHUA919:codex/sqlite-d2-batch1
Sep 14, 2026
Merged

huangruiteng merged 5 commits into
loopx-project:mainfrom
LIHUA919:codex/sqlite-d2-batch1

Conversation

@LIHUA919

@LIHUA919 LIHUA919 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

SQLite head continuity wrapped COUNT in CAST, disabling the engine's fast-count optimization and aggregating every retained operation row. Move the conversion outside the scalar subquery: counts remain exact SQL TEXT, all continuity/digest checks remain, and the connection lifetime is unchanged. A regression captures the production query and verifies the fast Count path plus empty/gap/int64 boundary behavior.

Also reject embedded SQLite releases affected by the WAL-reset concurrency bug, using Node 22.22.3 / SQLite 3.51.3 as the SQLite reference while keeping the public Node 22.18 minimum for File. Extend the existing capacity entrypoint with explicit rehearsal and matched 64 KiB 10k/100k profiles, cold CLI, separate byte metrics and a passed/failed/missing ledger, plus real-process recovery tests.

The no-argument capacity command intentionally changes from the old 4 KiB/100k run to the small rehearsal. After the narrow SQL fix, all 10 measured checks pass; 11 unmeasured D2 requirements remain explicit holds. The original failed 2.213x head-growth measurement is retained as the baseline.

窄 SQL 修复恢复快速计数,保留完整性与文本计数语义。正式 5:1 对照的 head p95 增长降至 1.642x,未改变预算;完整 D2 仍待后续资格证据。

Issue Or Task

Validation

  • Tested revision: 3400fab4231adf51c3b8137f7049940d29fd9b25. Current formal measurements, focused/full regressions and premerge cover the appended SQL fix. Original formal baseline: f6784e40845d36e7592c57c0b7620c5db744d6dc; the intermediate report metadata correction did not change workload or timers.
  • Run state: finished
  • Input classes: synthetic, public_fixture
Check kind Result Public-safe evidence / limitation
static passed npm run typecheck:control-plane, runtime-workflow smoke, CI classifier tests, diff checks and public/private scans. The current SQL fix and full PR passed premerge.
unit passed Full TypeScript suite: 1301 passed, 0 failed, 1 PostgreSQL integration skip. Focused real SQLite/provider/admission/capacity suite: 97 passed.
real_entrypoint passed tests/control_plane/test_sqlite_authority_cli.py: 5 passed. Capacity rehearsal and full profile exercise canonical status/quota reads, native mutation and isolated runtime shutdown.
real_backend passed Disposable real SQLite 3.51.3: 10k/100k matched groups completed, exact state/receipt/scan assertions passed, source fingerprint remained stable and both temporary databases were removed.
regression_parity passed Same synthetic request on baseline 34710d17a and candidate, using Node 22.18 / SQLite 3.50.2: baseline admits the vulnerable library; candidate rejects before authority directory creation. Default File head and receipts match. The fast-count regression fails on the prior production query and passes on the fix; existing corruption/recovery cases stay green. An injected vulnerable-version case traverses the actual provider entrypoint.
integration passed Local premerge: 4 direct checks + 18 selected checks, no failures/warnings. Real subprocess tests verify before/after-COMMIT interruption, exact committed proof, CAS/lost response and page-capacity failure atomicity.
manual passed Current formal head-read growth is 1.642x against <=2x. All 10 measured checks pass; the earlier 2.213x failure remains the regression baseline.
integration not_run Isolated real PostgreSQL integration was not run locally: this slice changes no PostgreSQL implementation, provider routing, projection or promotion contract. The full suite's one skip is retained above.
static passed Exact appended-head CI: 24 passed, 3 expected release/deployment skips; merge-gate passed. CI run.

Matched storage-axis summary (milliseconds; each group uses a new database). Host: macOS arm64, 18 logical CPUs and 64 GiB RAM; local temporary storage. The report does not claim hardware/OS qualification:

Operation Samples/group 10k p95 100k p95 Growth
Commit 1000 1.265 1.563 1.236x
Head load 3000 0.500 0.821 1.642x
Historical receipt 2000 0.756 0.885 1.170x
Scan 100 200 25.626 24.688 0.963x
Cold CLI mutation 20 451.224 446.191 —
Cold CLI status 20 1318.562 1312.158 —
Cold CLI quota 20 1471.448 1484.549 —

Full reports also contain p50/p99, Node/SQLite/source identifiers and sample counts. DB sizes at the target history, before the separate CLI mutations, are 696,594,432 / 6,966,067,200 bytes; sampled WAL is zero in both groups. These file sizes and application JSON bytes are not cumulative WAL or physical-write evidence. CLI mutation p95 difference is -5.033 ms against the <=200 ms increment budget.

  • Coverage and gaps: runtime admission and default-off isolation are tested through the real provider; recovery tests use real disposable SQLite and process termination. The current formal ledger has 10 passed, 0 failed and 11 missing rows. The original 9/1/11 baseline is retained; failure was addressed by the equivalent SQL rewrite, not by changing thresholds. Pure busy-wait, cumulative storage/WAL writes, stable-state RSS proof, full multi-agent/lease/capture workload, 1 MiB/300k, large-history recovery, consumer lag, restore/upgrade/rollback, OS/runtime coverage and elapsed soak remain explicit holds. A small database crash test is not power-loss or OS ENOSPC qualification. One matched pair supplies observations, not a confidence interval or profile promotion.
  • Future-facing pass: share one SQLite runtime probe between provider admission and reporting, and reuse the merged native projection fixture and journal-snapshot contract. The COUNT optimization also avoids a speculative connection pool: the existing lifecycle and exact checks remain. No duplicate business owner or checkpoint format was added.

Frontend / Visual Evidence

  • UI impact: none. No UI/settings or response-schema change; admission uses the existing typed provider-open failure path. The affected user entrypoints are the opt-in SQLite CLI/runtime and the developer qualification command.
  • Before: N/A
  • After: N/A
  • States and viewports shown: N/A; CLI acceptance/rejection/readback are covered above.
  • Source data: none

Type of Change

  • Bug fix
  • New feature
  • Breaking change — SQLite-specific runtime admission and the capacity command's no-argument default are intentionally changed and documented.
  • Refactoring (no functional changes)
  • Documentation update
  • Test update

LoopX Area

  • Control plane (goals, todos, quota, scheduler, registry, runtime)
  • Benchmark boundary (adapters, runners, verifiers, scoring, evidence)
  • Capability or extension (providers, adapters, skills)
  • Public docs or presentation surface (README, protocols, dashboard)
  • Build, packaging, installer, or CI
  • Host or runtime integration

Technical Direction

  • Core control-plane hardening

  • Long-horizon benchmark evidence

  • Operator surface and IM integration

  • Shared Goal Authority and cross-host coordination

  • Architecture and research incubator

  • Target base branch: main; implementation baseline 34710d17addbbdfd56e6ea1f03ec73519a4aa7fd.

  • Direction tracker or promotion unit: RFC section 7.2 / D2. Default provider changes, active-Goal migration, persistent-format redesign and soak launch remain separately gated.

Shared-authority RFC fixture impact

Boundary Checklist

  • Neither the diff nor this PR body/comments/attachments disclose private state, credentials, raw traces or verifier output, internal links, or local machine paths.
  • I did not duplicate maintainer-owned benchmark work unless a maintainer split out a public issue for it.
  • I kept the change scoped to the linked issue/task.
  • I completed the visual evidence section for UI changes, or marked UI impact none.
  • Every commit includes a DCO Signed-off-by trailer (git commit -s).

Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
@LIHUA919 LIHUA919 changed the title feat(sqlite): add D2 checks and require a WAL-fixed runtime fix(sqlite): retain fast COUNT and add D2 qualification checks Sep 13, 2026

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

评审目标是确认本 PR 在精确 head 3400fab4231adf51c3b8137f7049940d29fd9b25 上,是否真正解决 SQLite WAL-reset 风险、保持 authority 的原子性,并让容量报告不会把不完整的 D2 证据误报为通过。作者同时把 SQLite 资格 runtime、容量 profile 和 CI 分层写清楚;这比仅在旧 helper 中增加一个版本字符串更完整,但也扩大了示例与验证面。

改动思路

sqlite_runtime.ts 成为 SQLite runtime admission 的唯一决策者:先在 :memory: 中读取实际 sqlite_version()/sqlite_source_id(),验证已知修复版本线和同步 statement finalization,失败时在创建 authority 目录前返回协议失败。SqliteAuthorityStore 继续拥有 schema、identity、WAL、head/receipt 和事务不变量;只调用新的 admission helper,不改变 File provider。

容量路径由 sqlite-capacity.ts 驱动一次性 synthetic database,sqlite-capacity-report.ts 负责 typed passed/failed/missing ledger。默认命令改为小型 rehearsal,正式 matched-64k 明确运行 10k/100k 两条轴;CI 将 public-minimum Node 与 qualified SQLite Node 22.22.3 分开。文档说明了未覆盖的写放大、锁等待、OS/恢复、长 soak 和 promotion hold。

具体改动

  • sqliteAuthorityRuntime 严格解析 3.44.6+/3.50.7+/3.51.3+ 修复线并验证关闭后的 prepared statement;SqliteAuthorityStore.open 在任何 mkdir/open 前调用它。
  • current 将 scalar COUNT(*) 的 cast 移到子查询外,保留 SQLite Count fast opcode,同时继续检查 min=1, max=count=head 的连续性。
  • capacityLedger 只有完整 64 KiB、10k/100k、规定样本数且 cleanup 成功时才允许 formal pass;其余预算和 D2 维度保持 missing,失败保持 failed。
  • measureAxis 使用真实 authority store、local selector、legacy fence、CLI 及共享 fixture,记录 runtime/source identity、分位延迟、请求字节、文件大小、RSS 与 cleanup,并在失败时保留 phase/非零退出。
  • workflow、英文/中文 authority 文档、进程 crash/CAS/scan/容量测试和 todo continuation fixture 同步更新;没有改变默认 File provider 或持久 schema。

对主干的风险

我未发现阻塞性问题。风险主要是运行成本和环境边界:当前机器的 Node 25.5.0 内置 SQLite 3.51.2,按新契约被正确拒绝,因此本地无法直接跑完整 SQLite suite;远端 required CI 已在 qualified Node 22.22.3 lane 通过。正式 matched-64k/CLI profile 仍是昂贵的维护者操作,且报告明确不等价于十天 soak、OS ENOSPC、restore 或 promotion 证据。未来 SQLite minor 通过 minor > 51 规则被接受,若上游修复承诺发生变化,应更新版本表和测试。没有新增 scheduler/quota/authority 外部权限。

我的整体评价

APPROVE。该 PR 的 runtime gate、authority 复用、typed missing ledger 和默认行为披露形成了一个可审计的完整边界;正向/负向测试与远端 required checks 均覆盖了主要承诺。请继续把 missing/failed 当作 hold,不要把 rehearsal 结果用于 promotion。

English verdict: APPROVE for exact head 3400fab4231adf51c3b8137f7049940d29fd9b25. SQLite admission now fails closed before side effects, fast COUNT and authority invariants are preserved, and incomplete capacity evidence remains missing. The qualified Node 22.22.3 CI lane and required checks pass; this host’s Node 25.5.0/SQLite 3.51.2 rejection is expected.

@huangruiteng
huangruiteng merged commit 003cb16 into loopx-project:main Sep 14, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants