Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1280,11 +1280,19 @@ budget, one checkpoint history read and 1,048,576 retained projection bytes plus
This is still not completion of lane L. File and NoKV continue to retain and
decode their complete journal on every load, so bounded recovery is a property
of the embedded candidate rather than cross-provider parity; the SQLite profile
also still retains receipts and events without pruning. Unavailable logical/WAL
traffic and the <=15x cumulative write-growth budget, 1 MiB and 300k headroom,
full-domain workload, large-history recovery, fenced backup/restore, supported
upgrade/rollback, OS/runtime coverage and the >=10-day elapsed soak remain
holds, and runner completion cannot claim them. See the
also still retains receipts and events without pruning. The split
storage-traffic measurements landed with the matched-capacity entrypoint
(#4224 batch 1): the formal 64 KiB 10k/100k profile on the reference runtime
(Node 22.22.3/SQLite 3.51.3, declared local host) measures logical writes at
70,326 vs 70,324 bytes per commit, WAL traffic at 22,611 vs 22,623 bytes per
commit (exact frame counts over pinned-read-mark windows), and an app-observed
lock-wait p95 of 244 ms under a 200 ms held write lock — cumulative growth
ratios of 10.00x and 10.01x against the <=15x budget, with whole-run WAL
totals, pure busy-handler time and physical device writes still unmeasured.
Remaining holds: 1 MiB and 300k headroom, full-domain workload,
large-history recovery, fenced backup/restore, supported upgrade/rollback,
OS/runtime coverage and the >=10-day elapsed soak; runner completion cannot
claim them. See the
[SQLite qualification commands](../../reference/sqlite-authority-store.md#reproduce-validation).
The public minimum remains Node 22.18 for File; SQLite additionally requires
synchronous finalization and the WAL-reset fix, with Node 22.22.3/SQLite 3.51.3
Expand Down
36 changes: 27 additions & 9 deletions docs/reference/sqlite-authority-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,15 +278,28 @@ the isolated fixture. `--python` chooses the Python executable. These figures
include process startup but do not drop the OS file cache. Cold Node-only load
and warm actual-provider calls are separate. The provider's normal per-call
connection open/close remains inside warm timing. CLI mutations happen after
the fixed-history measurement; their extra commits are reported separately.
the fixed-history measurement; CLI, traffic-window and lock-probe commits all
extend past the fill target and are counted separately, so target-state rows
keep their meaning.

Reports carry p50/p95/p99 and counts, parent-process RSS, application request
JSON bytes and separate DB/WAL/SHM sizes at the target history. Resource-usage
peak RSS is process-lifetime across both groups; sampled axis RSS is separate,
and CLI child RSS is not measured. Application bytes, final files, SQLite
logical writes, cumulative WAL traffic and physical device writes are different
metrics. The unavailable write-traffic and pure busy-wait metrics remain
`missing`; a final WAL size of zero proves no cumulative-write bound.
metrics and are never substituted for one another. Logical write volume is
measured from the filled database as the serialized bytes each commit hands to
SQLite (commits row plus the full-projection head rewrite plus amortized
checkpoint rows); page, index and compaction overhead belong to the other
columns. Cumulative WAL traffic is measured over one bounded commit window per
axis: read marks pinned by two observer connections make every WAL reset
impossible, so frame growth over the window is exact, and the per-commit
traffic at both depths carries the <=15x cumulative-growth budget. Lock wait is
app-observed: a probe process holds the write lock for a controlled interval
and the end-to-end store commit wait is reported against the uncontended
baseline. Whole-run WAL totals, pure busy-handler time and physical device
writes remain `missing`; a final WAL size of zero still proves no
cumulative-write bound.

Each axis reserves 5 GiB free space, caps its database at 16 GiB and checks a
2,400-second fill budget. All data are generated in a new temporary directory;
Expand Down Expand Up @@ -348,12 +361,17 @@ command, migration manifest and reverse export remain separate deliverables.
### Qualification holds / 资格保留项

The report's `passed` rows apply only to their named axis and sample counts.
`failed` measurements remain failed; `missing` rows include cumulative storage
writes, pure lock wait, steady-state RSS proof, the full domain profile, 1 MiB
and 300k headroom, 24-hour consumer lag, large-history recovery, fenced
backup/restore, supported upgrades/rollback, OS/runtime coverage and a real
>=10-day soak. Those holds still block profile promotion. Accelerated volume
never substitutes for elapsed time, and running this command starts no soak.
`failed` measurements remain failed. The split storage-write rows —
`logical_write_growth`, `wal_traffic_growth` and `lock_wait_observed` — carry
the <=15x cumulative-growth budget as per-commit traffic measured at both
depths, and an invalidated window or missing probe is missing evidence, never a
pass from the surviving columns. `missing` rows still include whole-run WAL
totals, pure busy-handler time, physical device writes, steady-state RSS proof,
the full domain profile, 1 MiB and 300k headroom, 24-hour consumer lag,
large-history recovery, fenced backup/restore, supported upgrades/rollback,
OS/runtime coverage and a real >=10-day soak. Those holds still block profile
promotion. Accelerated volume never substitutes for elapsed time, and running
this command starts no soak.

Retained state is measured where the formal profile runs: an axis reports its
checkpoint count, replay budget, recovery tail and retained projection/delta
Expand Down
74 changes: 72 additions & 2 deletions examples/coordination/sqlite-capacity-report.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,46 @@ export function latency(samples: readonly number[]): Latency {
return {n: sorted.length, p50_ms: at(.5), p95_ms: at(.95), p99_ms: at(.99)};
}

/**
* Exact WAL traffic over one bounded commit window. A held read mark blocks
* every WAL reset, so the file only appends and frame growth is exact. The
* window measures per-commit traffic at one history depth; it is not a
* whole-run total.
*/
export type WalTrafficWindow =
| {status: "measured"; warmup_commits: number; window_commits: number; page_size_bytes: number;
frame_bytes: number; wal_bytes: number; frames: number; wal_bytes_per_commit: number}
| {status: "invalid"; reason: string};

/**
* Logical write volume from the filled database itself: the serialized bytes
* each commit hands to SQLite (commits row plus the full-projection head
* rewrite, with checkpoint rows amortized). Page, index and compaction
* overhead are deliberately excluded; they belong to WAL traffic and file
* growth, which are reported separately.
*/
export interface LogicalWriteAccounting {
commits_rows_sampled: number;
commits_row_bytes_mean: number;
checkpoints: number;
checkpoint_row_bytes_mean: number;
head_projection_bytes: number;
per_commit_logical_bytes: number;
cumulative_logical_bytes: number;
formula: string;
}

/**
* App-observed lock wait: end-to-end store commit latency while a probe
* process holds the database write lock for a controlled interval. The
* node:sqlite driver does not expose busy-handler internals, so this is the
* application-observed wait, not pure busy time.
*/
export type LockWaitProbe =
| {status: "measured"; samples: number; held_write_lock_ms: number;
uncontended_commit_p50_ms: number; observed_wait: Latency}
| {status: "invalid"; reason: string};

export interface CapacityAxis {
target_commits: number;
completed_commits: number;
Expand All @@ -33,6 +73,9 @@ export interface CapacityAxis {
bounded_profile: SqliteAuthorityBoundedProfile | null;
/** Linear archive audit, only requested where its cost is affordable. */
history_audit: {status: string; commits: number; checkpoints: number} | null;
wal_traffic_window: WalTrafficWindow | null;
logical_writes: LogicalWriteAccounting | null;
lock_wait: LockWaitProbe | null;
sampled_peak_rss_bytes: number;
resource_peak_rss_bytes: number;
fill_seconds: number;
Expand Down Expand Up @@ -106,10 +149,37 @@ export function capacityLedger(axes: readonly CapacityAxis[], formal: boolean):
scope: "retained checkpoint and delta bytes against one full copy per retained commit",
observed: retained, budget: Math.floor(perCommitCopy / 8), unit: "bytes"});
}
// Logical writes, WAL traffic and final file size are three separate
// measurements; none may substitute for another. Each growth row is the
// cumulative 10k -> 100k growth implied by per-commit traffic measured at
// both depths under the identical matched workload, so a per-commit cost
// that grows with history depth fails the <=15x budget.
const perCommitGrowth = (id: string, baselinePerCommit: number | undefined,
finalPerCommit: number | undefined, method: string) => {
const ratio = baselinePerCommit !== undefined && finalPerCommit !== undefined &&
baselinePerCommit > 0 && finalPerCommit > 0 ? 10 * (finalPerCommit / baselinePerCommit) : undefined;
if (!ready || ratio === undefined || !Number.isFinite(ratio)) {
rows.push({id, status: "missing", scope: `requires the complete matched profile and a measured window at both depths (${method})`});
} else rows.push({id, status: ratio <= 15 ? "passed" : "failed",
scope: `cumulative ${method} growth from 10k to 100k commits at fixed live state and delta sizes`,
observed: ratio, budget: 15, unit: "ratio"});
};
perCommitGrowth("logical_write_growth",
baseline?.logical_writes?.per_commit_logical_bytes, final?.logical_writes?.per_commit_logical_bytes,
"logical write");
perCommitGrowth("wal_traffic_growth",
baseline?.wal_traffic_window?.status === "measured" ? baseline.wal_traffic_window.wal_bytes_per_commit : undefined,
final?.wal_traffic_window?.status === "measured" ? final.wal_traffic_window.wal_bytes_per_commit : undefined,
"WAL traffic");
const lock = final?.lock_wait;
if (!ready || lock?.status !== "measured" || lock.observed_wait.n !== (formal ? 12 : 3)) {
rows.push({id: "lock_wait_observed", status: "missing",
scope: "requires the matched profile's held-write-lock probe at the 100k axis"});
} else rows.push({id: "lock_wait_observed", status: "passed",
scope: "app-observed store commit wait while a probe process holds the write lock; driver busy-handler internals remain unexposed",
observed: lock.observed_wait.p95_ms, unit: "ms"});
const scope: Record<string, string> = {
domain_workload: "eight agents, four writers, leases/capture/archive and the production-scale fixture remain separate",
cumulative_storage_writes: "application input bytes and final files cannot qualify logical writes, WAL traffic or the <=15x budget",
lock_wait_distribution: "no pure busy-handler timing is exposed by this node:sqlite driver",
steady_state_rss: "sampled RSS and per-process peak are observations, not a proof across steady-state windows",
large_history_recovery: "small fault regressions do not qualify bounded recovery of a 100k history; the linear archive audit is only launched in the rehearsal profile",
payload_and_headroom: "1 MiB, 300k and bursts are not launched by this profile",
Expand Down
Loading
Loading