Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions .github/workflows/python-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ jobs:
- name: Set up the qualified SQLite runtime
uses: actions/setup-node@v6
with:
node-version: "22.18.0"
node-version: "22.22.3"

- name: Qualify the TypeScript Effect core
run: |
Expand Down Expand Up @@ -165,7 +165,7 @@ jobs:
- name: Set up the qualified Node.js runtime
uses: actions/setup-node@v6
with:
node-version: "22.18.0"
node-version: "22.22.3"
cache: npm
cache-dependency-path: |
package-lock.json
Expand Down Expand Up @@ -225,9 +225,17 @@ jobs:
run: |
npm ci --ignore-scripts
npm run typecheck:control-plane
# The public minimum is qualified for every TypeScript control-plane
# provider, including SQLite statement finalization.
node --no-warnings --experimental-sqlite --experimental-strip-types --test tests/control_plane_ts/*.test.ts
# Default File/control-plane compatibility stays on the public minimum.
# SQLite requires a WAL-fixed embedded driver; its admission rejection
# runs here, while full SQLite suites run on the qualified runtime.
tests=()
for test in tests/control_plane_ts/*.test.ts; do
case "$test" in
*/sqlite_authority_store.test.ts|*/local_authority_provider.test.ts|*/authority_provider_parity.test.ts|*/sqlite_capacity.test.ts) continue ;;
esac
tests+=("$test")
done
node --no-warnings --experimental-sqlite --experimental-strip-types --test "${tests[@]}"

node-forward-compatibility:
needs: changes
Expand Down Expand Up @@ -486,7 +494,7 @@ jobs:
- name: Set up the qualified SQLite runtime
uses: actions/setup-node@v6
with:
node-version: "22.18.0"
node-version: "22.22.3"

- name: Run SQLite CLI integration
run: python -m pytest -q tests/control_plane/test_sqlite_authority_cli.py
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1161,12 +1161,15 @@ acceptance; it is not completion of lane L. Its head pointer is bounded and
operation/cursor lookups are indexed, but it retains full historical projections
and counts a covering index for continuity. That count grows with history;
current/accessed-row digests are checked, not every historical payload per read.
The published fixed-4-KiB microbenchmark lacks the 64-KiB matched profile, p99,
RSS, logical-WAL-write, recovery and elapsed-soak evidence required above. It
must not be reported as meeting the <=2 history-growth ratio or the ten-day
target. Node 22.18 is the public minimum and current SQLite qualification runtime;
preserve the Node 24 primary lane and Node 26 non-blocking forward probe until
the supported profile changes explicitly.
The qualification entrypoint now separates a small rehearsal from an explicit
64-KiB 10k/100k storage axis, with p99/counts, cold CLI, RSS and a
passed/failed/missing ledger. Unavailable logical/WAL traffic, full-domain,
large-history recovery and elapsed-soak evidence remain holds; runner completion
cannot claim the <=2 growth budget or ten-day qualification. See the
[SQLite qualification commands](../../reference/sqlite-authority-store.md#reproduce-validation).
The public minimum remains Node 22.18 for File; SQLite additionally requires
synchronous finalization and the WAL-reset fix, with Node 22.22.3/SQLite 3.51.3
as the reference. Node 24 stays primary and Node 26 stays a non-blocking probe.

**Migration decision points.** Before the first existing-Goal cutover, freeze
one exact source lineage/revision under the authority writer fence, import a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -914,10 +914,14 @@ fencing/export 演练与 maintainer review 都通过才可晋升。发布紧凑
**当前证据位置。** #4121 对应第一个节点,仍待维护者接受,不代表 lane L 完成。
其 head pointer 有界,operation/cursor 查询有索引,但保留完整历史 projection,连续性
校验还会统计覆盖索引,因此该成本随历史增长。它验证当前及访问到的 row digest,
不是每次读取都审计全部历史 payload。已发布的固定 4 KiB 微基准尚缺上述 64 KiB 匹配
profile、p99、RSS、逻辑 WAL 写入、恢复及自然时间 soak 证据,不能宣称满足 <=2 的历史
增长比值或十天目标。Node 22.18 是公开最低版本,也是当前 SQLite 资格化 runtime;支持
profile 明确变化前,继续保留 Node 24 主 runtime 与 Node 26 非阻塞 forward probe。
不是每次读取都审计全部历史 payload。资格入口现在区分小型 rehearsal 与显式
64 KiB 10k/100k 存储轴,记录 p99/样本数、cold CLI、RSS 和 passed/failed/missing
账本。逻辑/WAL 流量、完整领域负载、大历史恢复和自然时间 soak 的缺口仍阻止晋升,
工具跑完不等于通过 <=2 增长预算或十天资格。参见
[SQLite 验证命令](../../reference/sqlite-authority-store.md#reproduce-validation)。
公开 Node 最低版本 22.18 继续用于 File;SQLite 另需同步 finalization 与 WAL-reset
修复,参考组合为 Node 22.22.3/SQLite 3.51.3。Node 24 主 runtime 与 Node 26
非阻塞 forward probe 保持原合同。

**迁移决策点。** 首次迁移已有 Goal 前,先在 authority writer fence 下冻结精确的源
lineage/revision,导入完整权威快照与保留证明,并独立比较原始 receipt 字段、operation/
Expand Down
116 changes: 100 additions & 16 deletions docs/reference/sqlite-authority-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,25 @@ outside this slice's qualification boundary.
## Explicit selection

Use an isolated qualification runtime and an empty, unpromoted goal. Set
`RUNTIME_ROOT` to that runtime's absolute directory. SQLite qualification uses the public minimum Node 22.18 runtime. The provider checks that
`DatabaseSync.close()` finalizes prepared statements synchronously before
opening an authority file; older experimental drivers are rejected. In
older Node 22 releases leave closed database handles alive until GC on Windows
and are not supported for this provider. The module loads SQLite only after opt-in.
`RUNTIME_ROOT` to that runtime's absolute directory. The SQLite qualification
reference is **Node 22.22.3 with SQLite 3.51.3**. The provider checks the actual
embedded SQLite version and synchronous prepared-statement finalization before
creating or opening an authority database. Record both `sqlite_version()` and
`sqlite_source_id()`; the Node version alone is insufficient.

SQLite 3.51.3 and later 3.x releases contain the
[WAL-reset concurrency fix](https://www.sqlite.org/wal.html#the_wal_reset_bug).
The fixed 3.44.x (3.44.6+) and 3.50.x (3.50.7+) backport lines are also admitted
when their driver finalizes statements on close. Unknown version strings or
unverified vendor backports fail closed. Passing these prerequisites does not
qualify the complete D2 profile.

This intentionally rejects SQLite runtimes previously accepted by the
statement-only probe, including vulnerable drivers shipped with older Node 22
releases. The public Node minimum remains 22.18 for the default File path;
SQLite requires the additional fix. No provider selection changes and no
fallback to File occur when an explicitly selected SQLite runtime is rejected.
The optional driver is still loaded only after opt-in.

From the repository checkout, preview selection:

Expand Down Expand Up @@ -170,23 +184,93 @@ external service access, cross-host synchronization or promotion authority.

## Reproduce validation

Use the qualified Node executable on PATH, including the Python CLI's managed
Effect runtime. The runner records the actual Node/SQLite/source identity.

```sh
npm ci --ignore-scripts
npm run typecheck:control-plane
node --no-warnings --experimental-sqlite --experimental-strip-types --test \
tests/control_plane_ts/sqlite_authority_store.test.ts \
tests/control_plane_ts/local_authority_provider.test.ts
tests/control_plane_ts/local_authority_provider.test.ts \
tests/control_plane_ts/sqlite_runtime_admission.test.ts \
tests/control_plane_ts/sqlite_capacity.test.ts
python -m pytest -q tests/control_plane/test_sqlite_authority_cli.py
node -e "require('node:fs').mkdirSync('.local', {recursive:true})"
node --no-warnings --experimental-sqlite --experimental-strip-types \
examples/coordination/sqlite-capacity.ts --profile rehearsal --cli \
--output .local/sqlite-rehearsal.json
node --no-warnings --experimental-sqlite --experimental-strip-types \
examples/coordination/sqlite-capacity.ts
examples/coordination/sqlite-capacity.ts --profile matched-64k --cli \
--output .local/sqlite-matched-64k.json
```

The tests exercise real SQLite, independent writer processes, CAS competition,
original-receipt replay, lost responses, interrupted head publication, schema
rejection, persistent selection, native CLI read/update and planning replay after
Markdown deletion, and archive acknowledgement against the selected store.
The capacity command uses a disposable database with a fixed 4 KiB live
payload, 10k/100k commits, and 100 samples per read workload. It emits measured
latency percentiles and database bytes, then deletes only its temporary
database. These accelerated measurements do not satisfy the separate ten-day
soak, retention, disk-exhaustion, restore or promotion gates.
The no-argument default intentionally replaces the former 4 KiB/100k run with
a small `rehearsal`; full capacity now requires an explicit profile. The default
`rehearsal` creates 100/1,000 commits and checks runner execution,
independent invariants and cleanup; it cannot satisfy formal performance
budgets. The explicit `matched-64k` profile creates separate 10k/100k databases,
serially, with exactly 64 KiB native synthetic projection JSON and at most 4 KiB
of new event/receipt JSON per commit. Each fill write is followed by three head
reads and two indexed historical receipt reads. Both formal groups sample the
last 1,000 commits and their corresponding reads, plus 200 scan-100 samples.
This one-Todo storage axis isolates history growth; it is not the complete
multi-agent/lease/capture workload.

`--cli` adds 20 formal samples (three in rehearsal) for complete CLI mutation,
status and quota, using fresh Python processes and a newly started managed
Effect runtime for each sample. Shutdown occurs outside the timed interval in
the isolated fixture. `--python` chooses the Python executable. These figures
include process startup but do not drop the OS file cache. Cold Node-only load
and warm actual-provider calls are separate. The provider's normal per-call
connection open/close remains inside warm timing. CLI mutations happen after
the fixed-history measurement; their extra commits are reported separately.

Reports carry p50/p95/p99 and counts, parent-process RSS, application request
JSON bytes and separate DB/WAL/SHM sizes at the target history. Resource-usage
peak RSS is process-lifetime across both groups; sampled axis RSS is separate,
and CLI child RSS is not measured. Application bytes, final files, SQLite
logical writes, cumulative WAL traffic and physical device writes are different
metrics. The unavailable write-traffic and pure busy-wait metrics remain
`missing`; a final WAL size of zero proves no cumulative-write bound.

Each axis reserves 5 GiB free space, caps its database at 16 GiB and checks a
2,400-second fill budget. All data are generated in a new temporary directory;
there is no flag to select an existing Goal/runtime for writes. Keep generated
reports in ignored local storage. Failure results survive in the report and
exit nonzero; omitted or incomplete evidence never becomes a pass. An exit
zero with `status=incomplete` means the requested measurements ran, not that
D2 qualified. Formal budget failures must remain visible without changing the
workload or thresholds to obtain a green report.

The real-process regressions exercise SIGKILL before and after business COMMIT,
lost-response receipt readback, exact head/event/receipt/scan equivalence and
SQLite `max_page_count` exhaustion. These are small disposable-database tests,
not power-loss, operating-system ENOSPC or large-history recovery qualification.
The source uses the shared retained-journal snapshot contract. No checkpoint,
retention deletion, restore-incarnation change or migration format is added.

### Qualification holds / 资格保留项

The report's `passed` rows apply only to their named axis and sample counts.
`failed` measurements remain failed; `missing` rows include cumulative storage
writes, pure lock wait, steady-state RSS proof, the full domain profile, 1 MiB
and 300k headroom, 24-hour consumer lag, large-history recovery, fenced
backup/restore, supported upgrades/rollback, OS/runtime coverage and a real
>=10-day soak. Those holds still block profile promotion. Accelerated volume
never substitutes for elapsed time, and running this command starts no soak.

SQLite 资格参考使用 Node 22.22.3/SQLite 3.51.3;打开前同时检查实际 WAL 修复版本
和 statement 关闭行为。公开 Node 最低版本 22.18 继续用于默认 File 路径。显式
SQLite 选择遇到不合格 runtime 会拒绝,不会改默认 provider 或静默回退。

默认无参数命令从旧的 4 KiB/100k 改为小型 `rehearsal`,只验证工具和不变量;
正式 64 KiB、10k/100k 对照必须显式选择
`matched-64k`。`--cli` 分开记录完整 CLI 冷启动与 warm store,返回分位数、样本数、
RSS 和文件大小;没有量到的累计 WAL/逻辑写入和纯锁等待保持 missing。
应用 JSON 字节不能替代底层写入量,WAL 最终归零不能证明没有写入放大。

进程中断与 SQLite 容量注入在一次性合成数据库上运行,不等于断电、真实文件系统
耗尽、长期 consumer backlog 或完整恢复验证。首批测量允许保留 failed/missing;
>=10 天自然时间 soak、迁移和晋升分别评审与授权。本入口不改变持久格式、Todo
语义、默认 provider 或任何活跃 Goal。
92 changes: 92 additions & 0 deletions examples/coordination/sqlite-capacity-report.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/** Measurement semantics for the disposable SQLite capacity entrypoint. */
export interface Latency {
n: number;
p50_ms: number;
p95_ms: number;
p99_ms: number;
}

export function latency(samples: readonly number[]): Latency {
if (!samples.length || samples.some(value => !Number.isFinite(value) || value < 0)) {
throw new Error("latency requires nonempty finite nonnegative samples");
}
const sorted = [...samples].sort((left, right) => left - right);
const at = (p: number) => sorted[Math.ceil(sorted.length * p) - 1]!;
return {n: sorted.length, p50_ms: at(.5), p95_ms: at(.95), p99_ms: at(.99)};
}

export interface CapacityAxis {
target_commits: number;
completed_commits: number;
projection_json_bytes: number;
sample_window: number;
status: "passed" | "failed";
warm: Record<"commit" | "head" | "receipt" | "scan_100", Latency> | null;
cold_node: Latency | null;
cold_cli: Record<"mutation" | "status" | "quota", Latency> | null;
application_request_json_bytes: number;
files_at_target: {database_bytes: number; wal_bytes: number; shm_bytes: number} | null;
sampled_peak_rss_bytes: number;
resource_peak_rss_bytes: number;
fill_seconds: number;
cli_commits: number;
cleanup_verified: boolean;
failure?: string;
}

export interface QualificationRow {
id: string;
status: "passed" | "failed" | "missing";
scope: string;
observed?: number;
budget?: number;
unit?: string;
}

/** Thresholds come from RFC 7.2; a rehearsal cannot qualify the full profile. */
export function capacityLedger(axes: readonly CapacityAxis[], formal: boolean): QualificationRow[] {
const rows: QualificationRow[] = [];
const valid = (value: Latency | undefined, samples: number): boolean => !!value && value.n === samples &&
[value.p50_ms, value.p95_ms, value.p99_ms].every(n => Number.isFinite(n) && n >= 0) &&
value.p50_ms <= value.p95_ms && value.p95_ms <= value.p99_ms;
const baseline = axes.find(axis => axis.target_commits === 10000);
const final = axes.find(axis => axis.target_commits === 100000);
const ready = formal && axes.length === 2 && baseline?.status === "passed" && final?.status === "passed" &&
[baseline, final].every(axis => axis.completed_commits === axis.target_commits &&
axis.projection_json_bytes === 65536 && axis.sample_window === 1000 && axis.cleanup_verified &&
valid(axis.warm?.commit, 1000) && valid(axis.warm?.head, 3000) &&
valid(axis.warm?.receipt, 2000) && valid(axis.warm?.scan_100, 200));
rows.push({id: "matched_profile_execution", status: axes.some(axis => axis.status === "failed") ? "failed" :
ready ? "passed" : "missing", scope: "complete 64 KiB 10k/100k runs and declared sample counts"});
const add = (id: string, value: number | undefined, budget: number, unit: "ms" | "ratio" | "delta_ms") => {
if (!ready || value === undefined || !Number.isFinite(value) || (value < 0 && unit !== "delta_ms")) {
rows.push({id, status: "missing", scope: "requires the complete matched 64 KiB 10k/100k profile"});
} else rows.push({id, status: value <= budget ? "passed" : "failed", scope: "fixed 64 KiB storage axis",
observed: value, budget, unit});
};
for (const [key, budget] of [["commit", 100], ["head", 50], ["receipt", 50], ["scan_100", 250]] as const) {
add(`${key}_p95`, final?.warm?.[key].p95_ms, budget, "ms");
}
for (const key of ["commit", "head", "receipt"] as const) {
const denominator = baseline?.warm?.[key].p95_ms;
add(`${key}_history_growth`, denominator && final?.warm ? final.warm[key].p95_ms / denominator : undefined, 2, "ratio");
}
add("cold_cli_status_p95", valid(final?.cold_cli?.status, 20) ? final?.cold_cli?.status.p95_ms : undefined, 2000, "ms");
add("cold_cli_mutation_increment_p95", valid(baseline?.cold_cli?.mutation, 20) && valid(final?.cold_cli?.mutation, 20) && baseline?.cold_cli && final?.cold_cli
? final.cold_cli.mutation.p95_ms - baseline.cold_cli.mutation.p95_ms : undefined, 200, "delta_ms");
const scope: Record<string, string> = {
domain_workload: "eight agents, four writers, leases/capture/archive and the production-scale fixture remain separate",
cumulative_storage_writes: "application input bytes and final files cannot qualify logical writes, WAL traffic or the <=15x budget",
lock_wait_distribution: "no pure busy-handler timing is exposed by this node:sqlite driver",
steady_state_rss: "sampled RSS and per-process peak are observations, not a proof across steady-state windows",
large_history_recovery: "small fault regressions do not qualify bounded recovery of a 100k history",
payload_and_headroom: "1 MiB, 300k and bursts are not launched by this profile",
consumer_lag: "24-hour logical consumer backlog requires its own persisted-cursor test",
restore_upgrade_rollback: "fenced restore lineage and supported upgrade/rollback are not implemented by this harness",
elapsed_soak: "at least ten actual days require a separately authorized recoverable synthetic soak",
os_runtime_matrix: "one local run cannot qualify every supported OS and installed runtime",
promotion: "provider defaults, live migration and D3 remain separately gated",
};
for (const [id, reason] of Object.entries(scope)) rows.push({id, status: "missing", scope: reason});
return rows;
}
Loading