Skip to content

Commit f679911

Browse files
authored
perf(authority): reuse File proofs and bound quota observation output (#5222)
* fix(quota): bound plan observations with explicit full detail Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com> * perf(authority): retain a bounded File proof working set Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com> * test(quota): qualify compact observation and record read-cost limits Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com> --------- Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com> Co-authored-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
1 parent 00be577 commit f679911

15 files changed

Lines changed: 367 additions & 43 deletions

File tree

‎docs/architecture/rfcs/ledger/shared-goal-authority-state-provider-v0/2026-09-28-retirement-cadence.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,3 +166,41 @@ decision completeness and existing drill-down contracts at their shared typed
166166
owner; do not infer that backend switching alone fixes these costs. A/C still
167167
need integrated execution/adoption evidence, and no D2 elapsed soak starts or
168168
legacy-writer deletion is certified by this follow-up.
169+
170+
### Read-cost qualification update
171+
172+
After #4931 and #5215 integrated, matched detached File/SQLite copies retained
173+
379 original commits and the same final projection hash. On Node 24.21.0,
174+
three fresh processes per provider measured File head reads at 5.98–6.32 s
175+
versus SQLite at 34.5–36.0 ms; repeated reads were 9.1–10.2 ms and 25.7–28.2 ms
176+
respectively. This is process-cold, not OS-cache-cold: File proves its entire
177+
retained journal, whereas SQLite reads current state without making the same
178+
full-history proof. It is evidence for a long-history SQLite candidate, not
179+
equivalent integrity-work throughput or release-default acceptance.
180+
181+
Alternating two unchanged File stores exposed singleton proof-cache eviction:
182+
every read cost 6.30–6.49 s. A bounded four-store working set keeps the first
183+
proof for each store (6.15–6.16 s) and subsequent alternation at 9.8–11.2 ms,
184+
with identical cursors/hashes. Exact-byte and identity checks remain mandatory;
185+
eviction and corruption regressions cover the changed cache boundary.
186+
187+
Quota observation reused the existing should-run compactors: a captured single
188+
Goal row serialized from 1,252,747 to 78,688 UTF-8 bytes, with explicit full
189+
detail restoring the original row. This is a display measurement; collection,
190+
decision inputs and first-read verification are not reduced by it.
191+
192+
A separate 148-second isolated run appended 12 commits per provider through
193+
fresh processes, crossing a checkpoint and checking original-receipt replay,
194+
changed-intent rejection and projection/hash parity at every step. It qualifies
195+
that bounded storage journey, **not** Host execution, live Goal adoption or D2's
196+
ten-day soak. No active authority, release default or legacy-writer deletion
197+
decision changes. B still needs sustained workload/platform/capacity evidence;
198+
C still needs consumer/onboarding and supported upgrade acceptance.
199+
200+
The next B slice is the remaining whole-command cold path: profile history
201+
artifact lookup, active-contract validation and public-boundary scanning on
202+
the same retained inputs before selecting the owning repair. Separate provider
203+
head-read time from caller work; preserve freshness, full decision inputs and
204+
corruption rejection. Re-run installed CLI consumers after integration. Do not
205+
count this read optimization as closing A/C or use a fixed remaining-PR estimate;
206+
retire a writer only with its last supported caller and recovery acceptance.

‎docs/architecture/rfcs/ledger/shared-goal-authority-state-provider-v0/2026-09-28-retirement-cadence.zh-CN.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,3 +129,33 @@ settlement 链路验收。
129129
history,status/quota 仍可能生成数 MB 诊断包。在现有共享 typed owner 保留决策
130130
完整性与 drill-down 合同,不能推断换后端就能消除这些成本。A/C 仍需执行/采用集成
131131
证据;本轮没有启动 D2 自然时间 soak,也没有认证旧 writer 可以删除。
132+
133+
### 读取成本验收更新
134+
135+
#4931、#5215 集成后,配对的 File/SQLite 隔离副本保留 379 笔原始提交,最终
136+
projection hash 相同。Node 24.21.0 下,每个 provider 分别启动三个新进程,
137+
File 首次 head 读取为 5.98–6.32 秒,SQLite 为 34.5–36.0 毫秒;后续读取分别为
138+
9.1–10.2 毫秒、25.7–28.2 毫秒。这是进程冷读,没有清空 OS 文件缓存;File
139+
验证全部保留历史,SQLite 读取当前状态,不承担相同的全历史证明。这支持将 SQLite
140+
作为长历史候选,但不是同等完整性工作量的吞吐比较,也不构成发布默认值验收。
141+
142+
交替读取两个未变化的 File 存储,暴露了单份证明缓存互相淘汰的问题:每次都要
143+
6.30–6.49 秒。改为有总容量上限的四份缓存后,各存储首次验证仍为 6.15–6.16 秒,
144+
后续交替读取为 9.8–11.2 毫秒,cursor/hash 相同。每次仍检查实际字节摘要和存储
145+
身份;淘汰与损坏回归覆盖缓存边界。
146+
147+
Quota 观察复用既有 should-run 摘要:捕获的单 Goal 行序列化由 1,252,747 降至
148+
78,688 UTF-8 字节,显式明细恢复原行。这是展示体积测量,未减少采集、决策输入或
149+
首次读取的验证成本。
150+
151+
另一项 148 秒隔离演练通过新进程为两种 provider 各追加 12 笔提交,跨越 checkpoint,
152+
逐轮验证原回执重放、变更意图拒绝及 projection/hash 一致性。它证明这段有界存储
153+
流程,**不代表** Host 执行、活跃 Goal 采用或 D2 的十天 soak 已完成。本次不改变活跃
154+
authority、发布默认值或旧 writer 删除决定。B 仍缺持续负载/平台/容量证据;C 仍需
155+
consumer/新建入口及受支持升级验收。
156+
157+
B 的下一段是剩余整命令冷路径:在相同保留输入上分别分析历史 artifact 查找、
158+
active-contract 验证和公共边界扫描,再选择所属 owner 修复。区分 provider head
159+
读取与调用方工作,保留 freshness、完整决策输入和损坏拒绝;集成后重跑安装态 CLI
160+
消费者。不能把读取优化计为 A/C 完成,也不继续给固定的剩余 PR 数;只有最后受支持
161+
调用方退出且恢复验收通过,才能删除对应 writer。

‎docs/quota-allocation.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -890,6 +890,24 @@ The first screen should make it obvious why a project is quiet:
890890

891891
## CLI Surface
892892

893+
`quota status` and `quota plan` now default to bounded Todo summaries in JSON,
894+
reusing the summaries already used by `quota should-run`. Previously these two
895+
observation commands returned full Todo lists. Counts, quota decisions, ordering
896+
and health remain intact; `payload_compaction` identifies omitted lists and their
897+
detail command. Planning still consumes complete input before this CLI projection.
898+
Consumers that read individual Todo metadata or every item must opt into detail:
899+
900+
```bash
901+
loopx --format json quota status --include-detail all
902+
loopx --format json quota plan --include-detail agent-todos --include-detail user-todos
903+
```
904+
905+
Keep the original registry, runtime and Goal selection when following a detail
906+
command. `all` expands only the sections supported by that command. Detail reads
907+
do not acquire a Turn or spend quota. Markdown plan rendering and standalone
908+
`status`/`todo list` are unchanged. This bounds Todo-list display growth, not the
909+
cost of gathering and verifying the input or the total number of Goals returned.
910+
893911
The first read-only or preview commands are:
894912

895913
```bash

‎docs/reference/contracts/interface-budget-contract.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,11 +87,15 @@ removed without a separately validated caller migration.
8787
| `evidence-log --thin --limit 5` | explicit-limit cold path | returned-evidence bound | referenced run-history and rollout-event artifacts |
8888

8989
`quota should-run` uses one repeatable cold-path selector:
90-
`--include-detail scheduler`, `agent-todos`, `user-todos`, or
91-
`goal-boundary`; `--include-detail all` expands every section. Public docs,
90+
`--include-detail scheduler`, `agent-todos`, `user-todos`, `vision`, or
91+
`goal-boundary`. `quota status` and `quota plan` accept `agent-todos` and
92+
`user-todos`; `quota monitor-poll` accepts `decisions`.
93+
`--include-detail all` expands the selected command's sections. Public docs,
9294
emitted `detail_ref` commands, and internal callers use only this selector.
93-
Unknown sections and selectors attached to another quota command fail before
94-
status collection.
95+
Unknown or unsupported sections fail before status collection, including when
96+
combined with `all`. Status/plan summaries preserve counts and decisions and
97+
declare omitted lists; explicit detail preserves the full Todo metadata. These
98+
are CLI display projections after full planning, not truncated provider inputs.
9599

96100
The canonical emitted-output inventory and current characterization ceilings
97101
live in `loopx.control_plane.testing.cli_output_budget`. Those ceilings are

‎docs/reference/file-authority-state-log.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,13 @@ append-only even though File atomically replaces its physical envelope.
2222

2323
Cold reads verify every retained transaction and the final head; a valid head
2424
cannot hide a corrupt old delta or receipt. Verified pagination reconstructs at
25-
most 63 predecessor deltas plus the requested page. The exact-byte cache remains
26-
bounded. File still reads/hashes and rewrites one retained file: this reduces
25+
most 63 predecessor deltas plus the requested page. The exact-byte cache retains
26+
at most four store paths in least-recently-used order, with a shared 128 MiB
27+
serialized history/read-view budget. A large journal can retain only its head
28+
and receipt index (up to 16 MiB per read view); scans and writes still verify its
29+
history. Every cache hit requires matching file digest and store identity, not
30+
only file timestamps. These are encoded-data bounds, not a heap/RSS limit.
31+
File still reads/hashes and rewrites one retained file: this reduces
2732
repeated data, not asymptotic growth. Cold verification can be slower. Measure
2833
upgrade, cold verification, warm reads and steady writes separately.
2934

‎loopx/cli_commands/quota.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
from ..control_plane.capability_hooks import InteractionProjectionHookRegistration
2121
from ..control_plane.quota.cli_projection import (
2222
compact_quota_monitor_poll_cli_payload,
23+
compact_quota_plan_cli_payload,
2324
compact_quota_should_run_cli_payload,
2425
)
2526
from ..control_plane.quota.effective_action import EffectiveAction
@@ -318,6 +319,8 @@ def _project_quota_cli_payload(
318319
instead of masking it with a crash (issue #3687).
319320
"""
320321
if not bool(getattr(args, "turn_envelope", False)):
322+
if args.quota_command in {"status", "plan"}:
323+
return compact_quota_plan_cli_payload(payload, detail_sections=detail_sections)
321324
if args.quota_command == "should-run":
322325
return compact_quota_should_run_cli_payload(
323326
payload,

‎loopx/cli_commands/quota_context.py‎

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,7 @@
3030
from ..status import AUTONOMOUS_REPLAN_PERIODIC_LOOKBACK, collect_status
3131
from ..turn_identity import mint_turn_instance_id, normalize_turn_instance_id
3232
from .quota_request import (
33-
QUOTA_MONITOR_POLL_DETAIL_SECTIONS,
34-
QUOTA_SHOULD_RUN_DETAIL_SECTIONS,
33+
QUOTA_COMMAND_DETAIL_SECTIONS,
3534
quota_detail_sections_from_args,
3635
validate_quota_command_request,
3736
)
@@ -124,17 +123,13 @@ def validate_quota_command_context_request(
124123
"--turn-envelope is only valid with `quota should-run`"
125124
)
126125
requested_details = set(getattr(args, "include_details", None) or ())
127-
if requested_details and command not in {"should-run", "monitor-poll"}:
126+
if requested_details and command not in QUOTA_COMMAND_DETAIL_SECTIONS:
128127
raise QuotaCommandValidationError(
129-
"--include-detail is only valid with `quota should-run` or "
130-
"`quota monitor-poll`"
131-
)
132-
if requested_details and "all" not in requested_details:
133-
allowed_details = set(
134-
QUOTA_MONITOR_POLL_DETAIL_SECTIONS
135-
if command == "monitor-poll"
136-
else QUOTA_SHOULD_RUN_DETAIL_SECTIONS
128+
"--include-detail is only valid with `quota status`, `quota plan`, "
129+
"`quota should-run` or `quota monitor-poll`"
137130
)
131+
if requested_details:
132+
allowed_details = {*QUOTA_COMMAND_DETAIL_SECTIONS[command], "all"}
138133
unsupported_details = sorted(requested_details - allowed_details)
139134
if unsupported_details:
140135
raise QuotaCommandValidationError(

‎loopx/cli_commands/quota_registration.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,8 @@ def register_quota_command(
6969
action="append",
7070
choices=[*QUOTA_DETAIL_SECTIONS, "all"],
7171
help=(
72-
"Include one command-specific cold-path detail section. For `quota "
72+
"Include one command-specific cold-path detail section. Status/plan default "
73+
"to bounded Todo summaries; use agent-todos or user-todos for full lists. For `quota "
7374
"should-run`: scheduler, agent-todos, user-todos, goal-boundary, or "
7475
"vision. For `quota monitor-poll`: decisions. Repeat for multiple "
7576
"sections or use `all`."

‎loopx/cli_commands/quota_request.py‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,13 @@
1313
"vision",
1414
)
1515
QUOTA_MONITOR_POLL_DETAIL_SECTIONS = ("decisions",)
16+
QUOTA_PLAN_DETAIL_SECTIONS = ("agent-todos", "user-todos")
17+
QUOTA_COMMAND_DETAIL_SECTIONS = {
18+
"status": QUOTA_PLAN_DETAIL_SECTIONS,
19+
"plan": QUOTA_PLAN_DETAIL_SECTIONS,
20+
"should-run": QUOTA_SHOULD_RUN_DETAIL_SECTIONS,
21+
"monitor-poll": QUOTA_MONITOR_POLL_DETAIL_SECTIONS,
22+
}
1623
QUOTA_DETAIL_SECTIONS = (
1724
*QUOTA_SHOULD_RUN_DETAIL_SECTIONS,
1825
*QUOTA_MONITOR_POLL_DETAIL_SECTIONS,
@@ -178,9 +185,7 @@ def quota_detail_sections_from_args(args: argparse.Namespace) -> frozenset[str]:
178185
sections.add("scheduler")
179186
if "all" in sections:
180187
sections.update(
181-
QUOTA_MONITOR_POLL_DETAIL_SECTIONS
182-
if args.quota_command == "monitor-poll"
183-
else QUOTA_SHOULD_RUN_DETAIL_SECTIONS
188+
QUOTA_COMMAND_DETAIL_SECTIONS.get(args.quota_command, ())
184189
)
185190
sections.discard("all")
186191
return frozenset(sections)

‎loopx/control_plane/coordination/file_authority_store.ts‎

Lines changed: 28 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -28,13 +28,15 @@ import {AuthorityJournalScan} from "./authority_journal_scan.ts";
2828

2929
const STORE_IDENTITY_PATTERN = /^file:[0-9a-f]{32}$/;
3030
// File retains a checkpoint/delta journal in one durable envelope. A managed Effect server
31-
// opens a new store handle for each request. Keep one verified read view across
32-
// handles, keyed by exact bytes and store identity. Large journals retain only
31+
// opens a new store handle for each request. Retain a bounded working set across
32+
// handles so alternating Goals do not evict each other on every observation.
33+
// Every lookup still reads and hashes the full file and checks store identity. Large journals retain only
3334
// the head and receipt index in memory; commits and scans still load and verify
3435
// the complete history. This is a bounded read optimization, not a new source
3536
// of authority or a substitute for the SQLite long-goal profile.
3637
const MAX_CACHED_DOCUMENT_BYTES = 128 * 1024 * 1024;
3738
const MAX_CACHED_READ_VIEW_BYTES = 16 * 1024 * 1024;
39+
const MAX_CACHED_STORES = 4;
3840
interface VerifiedDocument {
3941
path: string;
4042
identity: string;
@@ -49,7 +51,7 @@ interface VerifiedDocument {
4951
}>;
5052
document?: FileAuthorityJournal;
5153
}
52-
let verifiedDocument: VerifiedDocument | null = null;
54+
const verifiedDocuments = new Map<string, {view: VerifiedDocument; bytes: number}>();
5355
// Only identical immutable input bytes share in-flight verification. Failed
5456
// proofs are removed too; neither a path nor a pending promise grants authority.
5557
const pendingVerification = new Map<string, Promise<FileAuthorityJournal>>();
@@ -76,10 +78,22 @@ function rememberVerifiedDocument(path: string, identity: string, raw: Uint8Arra
7678
const view: VerifiedDocument = {path, identity, digest, head: document.head,
7779
providerRevision: document.provider_revision, cursor: document.cursor,
7880
receipts, document};
79-
verifiedDocument = raw.byteLength <= maxDocumentBytes ? view
80-
: viewBytes <= MAX_CACHED_READ_VIEW_BYTES
81-
? {...view, document: undefined}
82-
: null;
81+
// Account for serialized history and the separate head/receipt index. This
82+
// is a retained-byte bound, not a claim about the JS heap or process RSS.
83+
const fullBytes = raw.byteLength + viewBytes;
84+
const retainHistory = raw.byteLength <= maxDocumentBytes && fullBytes <= MAX_CACHED_DOCUMENT_BYTES;
85+
const retained = retainHistory ? view : {...view, document: undefined};
86+
const bytes = retainHistory ? fullBytes : viewBytes;
87+
verifiedDocuments.delete(path);
88+
if (retainHistory || viewBytes <= MAX_CACHED_READ_VIEW_BYTES) {
89+
verifiedDocuments.set(path, {view: retained, bytes});
90+
let total = [...verifiedDocuments.values()].reduce((sum, entry) => sum + entry.bytes, 0);
91+
while (verifiedDocuments.size > MAX_CACHED_STORES || total > MAX_CACHED_DOCUMENT_BYTES) {
92+
const oldest = verifiedDocuments.keys().next().value!;
93+
total -= verifiedDocuments.get(oldest)!.bytes;
94+
verifiedDocuments.delete(oldest);
95+
}
96+
}
8397
return view;
8498
}
8599

@@ -274,10 +288,12 @@ export class FileAuthorityStore implements AuthorityStore {
274288
const identity = knownIdentity ?? await this.readStoreIdentity();
275289
try {
276290
const digest = documentDigest(raw);
277-
if (verifiedDocument?.path === this.path &&
278-
verifiedDocument.identity === identity && verifiedDocument.digest === digest &&
279-
(!requireHistory || verifiedDocument.document !== undefined)) {
280-
return verifiedDocument;
291+
const cached = verifiedDocuments.get(this.path);
292+
if (cached?.view.identity === identity && cached.view.digest === digest &&
293+
(!requireHistory || cached.view.document !== undefined)) {
294+
verifiedDocuments.delete(this.path);
295+
verifiedDocuments.set(this.path, cached);
296+
return cached.view;
281297
}
282298
const key = JSON.stringify([this.path, identity, digest]);
283299
let proof = pendingVerification.get(key);
@@ -420,7 +436,7 @@ export class FileAuthorityStore implements AuthorityStore {
420436
// A failure after rename may already have published the new bytes.
421437
// The next read must prove the actual file rather than reuse either
422438
// the previous or attempted document.
423-
verifiedDocument = null;
439+
verifiedDocuments.delete(this.path);
424440
return {
425441
status: "ambiguous",
426442
reason_code: "commit_outcome_unknown",

0 commit comments

Comments
 (0)