Repository navigation
hourly full run: red on main (CI) #22346
Description
Activity
- addedbugSomething isn't workingSomething isn't workingpriority:p1High: required for production / M2High: required for production / M2
on Oct 8, 2026 objectstack-fleet commented
on Oct 8, 2026 ContributorMore actionsTriage:
priority:p1stands ·domain:devx→domain:spec·pm:queue. Reading: one exhaustive case inpackages/lintran past vitest's 5 s default, and its lookup is quadratic. ⛔ never narrow, skip or delete the checkTriage seat (objectstack-wide, seat post #6015) ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-08T20:04Z. ⛔ Not a claim, ⛔ not a dispatch.Triage: the fix lands in
packages/lint's tests ⇒domain:spec. Rationale:packages/lintbelongs to that lane. The generator'sdomain:devxdefault does not fit here.- Where it is red: run 37828912468,
Test Core (4/6), job 113488905614, read from the job log. One case of 5,843 in@objectstack/lintfailed:src/build-access-matrix.test.ts:162, "exhaustively matches the fold over every declared bit combination" ([finding] the super-user permission fold is now stated three times — spec's objectPermissionGrants, plugin-security's PermissionEvaluator and lint's buildAccessMatrix — one rule, three implementations #18785).Error: Test timed out in 5000ms. No assertion failed.- The shard was loaded: the package's import phase took 473 s.
- Why it is slow, read in source:
- The case builds 3^8 = 6,561 permission sets.
- For each one, it calls
m.entries.find(...), and that callback rebuilds theps_NNNNname string on every comparison. - That comes to about 21.5 million callback calls and string builds, quadratic in the entry count.
- Not a regression from the range: no commit in
28bff18d0c..59fb299c54touches the case orbuild-access-matrix.ts; both were last changed in refactor(security): one permission fold — the enforcement door and the access matrix ASK the spec helper (#18785) #19512 (2026-09-21). The case sat close to the default and crossed it under load. - Direction:
- Index
m.entriesbypermissionSetonce, in aMap, so the lookup is linear. The assertions stay exactly as they are. - ⛔ Raising the timeout is not the fix, and the case is never skipped.
- Index
- Family: this is the second hourly red from a timeout today. The first was hourly full run: red on main (CI) #22292 (
@objectstack/spec, 60 s, closednot_planned). A third gets a family card that enumerates the heaviest cases.
- Where it is red: run 37828912468,
objectstack-fleet commented
on Oct 8, 2026 ContributorMore actionsTriage: the next hourly run, on
35afb15878, is red for a second, different reason, the shard-timing drift step. That is #22014's open half. The lint case still owes its fixTriage seat (objectstack-wide, seat post #6015) ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-08T20:53Z. ⛔ Not a claim, ⛔ not a dispatch. This adds to my grade6068065094. Labels unchanged.- What is red: run 37836262554,
Test Core (6/6), job 113514096795, read from the job log.- Every test passed:
check-test-completenessOK, 14 of 14 packages, 16,207 tests accounted for. - The step that failed is
shard-timing-drift: 2425.6 s measured against 1590.1 s predicted, 1.53x. The step warns past 1.3x and is red past 1.5x. - Heaviest overshoots:
runtime1.68x,plugin-security1.52x,driver-sql1.52x.
- Every test passed:
- This is not the diffs. The
domain:specseat measured the same shard at 1.52x–1.56x on three unrelated PRs today (6065689519on finding(ci): the shard-timings dataset rests on ONE scheduled run, and records @objectstack/spec at 1134.86 s against 1573–1651 s executed — #16468's 25%-headroom ceilings built on it would red every PR that runs spec #22014), one of them comments-only. A merge-group run was ejected by the same step. The dataset no longer describes the shard, which is finding(ci): the shard-timings dataset rests on ONE scheduled run, and records @objectstack/spec at 1134.86 s against 1573–1651 s executed — #16468's 25%-headroom ceilings built on it would red every PR that runs spec #22014's open half: the first multi-run refresh.- ⛔ Per the step's own text: no hand-edit of
scripts/test-shard-timings.json, and no raise of the bound. - finding(ci): the shard-timings dataset rests on ONE scheduled run, and records @objectstack/spec at 1134.86 s against 1573–1651 s executed — #16468's 25%-headroom ceilings built on it would red every PR that runs spec #22014 is raised to p1 in this act. The unblock is a maintainer
workflow_dispatchofShard Timings Refresh, thenupdate-branchon the bot's refresh PR. Otherwise the scheduled run on 2026-10-12 does it.
- ⛔ Per the step's own text: no hand-edit of
- The lint case is still owed.
build-access-matrix.test.ts:162passed in this run, but its quadratic lookup is unchanged onmain. The direction in6068065094stands: index the entries once, with no timeout raise. - This card closes when an hourly run is green, as its generator says. Both causes have to clear for that.
- What is red: run 37836262554,
objectstack-fleet commented
on Oct 8, 2026 ContributorMore actionsClaim: PM loop round 1 · 2026-10-08T21:13Z
Session:session_01DhTqaEHqPVSVnAkjG3jywn
Account:os-sales(the seat's linked user asGET /useranswers it; the card's assignee from this act)
Branch:claude/issue-22346-access-matrix-linear-lookup
Worktree:objectstack-issue-22346
Domain:domain:spec
Seat:domain:spec#2(seat post #18549)
File surface (atorigin/main54c3ce10cor later; stop on breach and explain in the report):packages/lint/src/build-access-matrix.test.ts, the case at about:162("exhaustively matches the fold over every declared bit combination"). Indexm.entriesbypermissionSetonce (aMap), so the per-entry lookup is linear. The assertions, the 3^8 enumeration and the cell count stay exactly as they are. ⛔ No timeout raise. ⛔ No skip. ⛔ No change tobuild-access-matrix.ts.- If the measurement shows a second quadratic step in the same case (for example the name string rebuilt per comparison elsewhere), it is fixed the same way in the same file.
- No changeset if the repo's changeset rules exempt a test-only diff in a published package (the dev reads them and applies the declared opt-out); otherwise as they require.
Container & model:Smechanical,mode:subagent,model: opus(dispatch-gates --tier --repo objectstack-ai/objectstackon this path: no path-derived mandate; the default tier). No contract review owed:Clause-②: no, a test-only change.
Clause-②: no
Responsibility: the test case's own quadratic lookup (m.entries.find(...)over 6,561 entries, rebuilding the name per comparison) | none: no platform path covers a test's own cost | the hourly full run onmain(CI), red on59fb299c54(hourly full run: red on main (CI) #22346) when a loaded shard pushed the case past vitest's 5 s default
Thread-read: 6068854002
Serial constraints cleared: none. No open PR touchesbuild-access-matrix.test.tsorbuild-access-matrix.ts(all 14 open PRs' file lists, read at this stamp). The second hourly red on this card (shard-timing drift) is finding(ci): the shard-timings dataset rests on ONE scheduled run, and records @objectstack/spec at 1134.86 s against 1573–1651 s executed — #16468's 25%-headroom ceilings built on it would red every PR that runs spec #22014's, not this claim's.
Generated by Claude Code
objectstack-fleet commented
on Oct 8, 2026 ContributorMore actionsos-dev-report
{ "issue": 22346, "status": "done", "branch": "claude/issue-22346-access-matrix-linear-lookup", "pr": "https://github.com/objectstack-ai/objectstack/pull/22358", "session": "session_01DhTqaEHqPVSVnAkjG3jywn (subagent: the parent PM session id, as the dispatch and the Claude-Session trailer carry it)", "premise_still_valid": true, "summary": "packages/lint/src/build-access-matrix.test.ts, case 'exhaustively matches the fold over every declared bit combination' (:162): the per-entry m.entries.find(...) lookup is replaced by a Map built once from m.entries, keyed by permissionSet, first match wins, so each lookup returns the row find returned. Test-only; build-access-matrix.ts untouched; no timeout raise; no skip; the 3^8 enumeration, the cell count and all 4 assertions unchanged. H1 confirmed: a counter in the find callback read 21,526,641 calls per run. H2 confirmed: lookup 780-951 ms vs buildAccessMatrix 3-25 ms, enumerate 1-4.5 ms, fold loop 1.7-5.6 ms, Map lookup 1.3-2.7 ms. H3 confirmed: all 8 cases green before, after and under the ablation. Case wall time 700/613/751 ms before -> 46/44/47 ms after (about 15x). Shared-box absolutes under the verify lock; read the ratios. PR body opens with 'Part of #22346' (the card's generator closes it on a green hourly run; the shard-timing drift half is #22014's and remains open) and carries 'Clause-②: no' at a line start.", "tests": "All at b447bfa07 unless stated, under scripts/pm/os-verify-lock.sh, VERDICT command-exit 0 each. (1) Case timing, 'pnpm --filter @objectstack/lint exec vitest run --maxWorkers=2 --reporter=verbose src/build-access-matrix.test.ts' x3: before at 54c3ce10c 700 ms / 613 ms / 751 ms ('Tests 8 passed (8)' x3); after at b447bfa07 46 ms / 44 ms / 47 ms ('Tests 8 passed (8)' x3). (2) H2 breakdown, scratch tsx script (not committed) against src at 54c3ce10c, 3 iterations: enumerate 4.5/1.0/0.9 ms; permissions map 1.6/1.9/0.6 ms; buildAccessMatrix 25.0/14.7/3.1 ms; per-entry find 950.8/779.8/845.7 ms (with a counter: 847.4/889.2/773.1 ms, find-callbacks=21526641); Map built once 2.7/1.3/1.7 ms; fold loop 5.6/1.7/1.7 ms; sameRows=true, cells=26244, mismatches=0. (3) Assertions unchanged: the case's expect( count 4 before and 4 after, matcher text cmp-identical (expect(entries.length).toBe(3 ** 8) / expect(m.entries.length).toBe(entries.length) / expect(cells).toBe(3 ** 8 * 4) / expect(mismatches).toEqual([])); whole file 28 expect( lines before and after, byte-identical as a sorted set; 8 it( cases before and after; the before/after diff of the case is in the PR body. (4) Ablation via scripts/ablation-replace.mjs (wrap mode, own trap plus an outer EXIT/INT/TERM trap with an absolute path), committed state HEAD b447bfa07: anchor 'const row = rowByPermissionSet.get(...)!;' x1 -> x0, replacement (the main-branch find line) 0 -> 1, blob 84866b0a6a8f -> 40761cc2c49a, x3 runs: case 834 / 803 / 696 ms, 'Tests 8 passed (8)' each (expected: the timing is the pin, no assertion added). Each restore leg: 'ok restored: blob == HEAD (84866b0a6a8f) and git diff HEAD is empty'; final read: anchor 1, replacement 0, blob 84866b0a6a8f, git diff HEAD empty. No build/dist leg: the subject is the test file vitest runs from src. (5) Package suite 'pnpm --filter @objectstack/lint exec vitest run --maxWorkers=2': 'Test Files 127 passed (127)', 'Tests 5838 passed | 5 skipped (5843)'. (6) 'pnpm --filter @objectstack/lint typecheck': exit 0; tsc --listFiles: the edited file is in tsconfig.test.json's program (1 hit) and excluded from tsconfig.json (0 hits); check:test-typecheck OK with this file not on the debt ledger, so it has 0 errors. (7) ESLint narrowed to the one changed file, 'eslint --no-inline-config --format json packages/lint/src/build-access-matrix.test.ts': files=1 errors=0 warnings=0; population = the diff's only path, which --print-config shows ESLint lints (5 rules); invariance: no type-aware linting for this file (parserOptions.project and projectService both null) and no config edited, so no untouched file's verdict can move. Repo-wide pnpm lint is CI's. (8) Dependency closure built first: pnpm --filter '@objectstack/lint^...' build (exit 0).", "mcp_calls": "0 — no MCP GitHub tool was called (reads went through gh api REST GETs; writes through the scripts/pm relay)", "api_writes": "3 relay dispatches (POST /repos/objectstack-ai/objectstack/dispatches, fleet-write, executed as objectstack-fleet[bot]), carrying 4 endpoint writes: (1) pr_create POST /repos/objectstack-ai/objectstack/pulls (draft, #22358); (2) label-write.mjs on #22358: labels_add POST /repos/objectstack-ai/objectstack/issues/22358/labels [skip-changeset] + assign POST /repos/objectstack-ai/objectstack/issues/22358/assignees [os-sales], read back MATCHES (size/s and tests were already on the PR from another actor, not this write's target); (3) post-stamped.mjs comment POST /repos/objectstack-ai/objectstack/issues/22346/comments (this os-dev-report). Two locally refused attempts (write-pace exit 10, zero requests) are in deviations. Not REST: 2 git pushes of claude/issue-22346-access-matrix-linear-lookup (the empty-branch probe at 54c3ce10c, then b447bfa07).", "open_questions": [], "out_of_scope_findings": [], "gates": { "derived": "node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack (no paths), at b447bfa07: 54 commands, identical to the dispatch-time list g22346.txt (sorted diff empty). Change set: 1 path vs merge base 54c3ce10c (committed 1, working tree 0, untracked 0).", "run": "54 run. 52 exit 0 on the first pass. pnpm check:lean-entry-closure exit 3 (PREREQUISITE NOT MET: packages/objectql/dist/core absent) -> built pnpm --filter '@objectstack/objectql...' under the lock -> rerun exit 0 (2 published conditions measured, admitted set held, 15 packages). pnpm check:dual-build-cjs-loads exit 3 (PREREQUISITE NOT MET: needs every package built, 83 dists absent) -> NOT MEASURED: dual-build-cjs-loads, reason: needs a full pnpm build of the workspace; the diff ships nothing (0 hits in lint dist), declared to CI.", "exit_codes": "all 0 except check:dual-build-cjs-loads 3 (NOT MEASURED); check:lean-entry-closure 3 then 0 after its prerequisite build", "ran_verdict": "✓ dispatch-gates --ran: 54 derived famil(ies) accounted for — 53 run, 1 NOT-MEASURED (1 DERIVED from a recorded exit 3)." }, "line_budget": "10 changed lines vs merge base 54c3ce10c (+9 / -1, 1 file); governed paths touched: 0", "deviations": [ "Changeset: none written; skip-changeset applied via label-write.mjs. Basis: AGENTS.md Post-Task Checklist step 3 (the label is for a diff that publishes nothing from any released package) and check-empty-changeset.mjs (an empty-frontmatter changeset is refused). Measured after pnpm --filter @objectstack/lint build: files[] = dist, README.md, CHANGELOG.md; 'rowByPermissionSet' 0 hits, the case title 0 hits, positive control 'buildAccessMatrix' 6 hits.", "Two writes were each refused once by write-pace before any request (exit 10: the container's shared 40-writes/hour token budget was spent by the fleet; nothing written): the pr_create dispatch and this report comment. Each was resent once after the window reopened (a foreground wait on the budget, no retry loop). Counted once each in api_writes because only the resend wrote.", "Commit trailers: the model-free pair AGENTS.md prescribes (Co-authored-by: Claude, Claude-Session), not the harness reminder's model-named Co-Authored-By line, per the role file's rule that the harness reminder yields to it." ], "files_changed": [ "packages/lint/src/build-access-matrix.test.ts" ] }
Generated by Claude Code
objectstack-fleet commented
on Oct 8, 2026 ContributorMore actionsSeat review of PR #22358 at
b447bfa07: ACCEPT. Lands when green (Part of: the drift half stays #22014's)domain:specseat 2 (#18549) ·os-sales· sessionsession_01DhTqaEHqPVSVnAkjG3jywn· 2026-10-08T22:08Z · holder of claim6069167724.The report is on this card (
os-dev-report). The seat read the net diff: 1 file, +9 / -1.- The change:
packages/lint/src/build-access-matrix.test.ts, the case at:162. It builds aMapfromm.entriesonce, keyed bypermissionSet, and the first match wins, so every lookup returns the rowfindreturned. The 3^8 enumeration, the cell count and the 4 assertions are byte-identical (28expect(lines in the file before and after).build-access-matrix.tsis untouched. No timeout raise, no skip. - Measured (triage's direction
6068065094confirmed):- The
findcallback ran 21,526,641 times per run, taking 780–951 ms. buildAccessMatrixitself takes 3–25 ms.- The case's wall time went from 613–751 ms to 44–47 ms (about 15x), against vitest's 5 s default that the loaded hourly shard crossed.
- The ablation (
findrestored) brings back 696–834 ms with all 8 tests passing. The timing is the pin here; no assertion was added, by design.
- The
- Level: test-only,
skip-changeset(the lint dist carries 0 hits of the new identifier).Clause-②: no. No contract review is owed. - Gates: 53 / 54 derived ran, all exit 0.
check:dual-build-cjs-loadsis NOT MEASURED: it needs a full workspace build, and this diff ships nothing. It is declared to CI.
Next: every check green → landing. The card stays open: its generator closes it on a green hourly run, which also needs #22014's dataset refresh.
Generated by Claude Code
- The change:
objectstack-fleet commented
on Oct 8, 2026 ContributorMore actionsLanded: PR #22358 →
b46015391(this card's lint half,Part of #22346). The card moves topm:blockedon #22014domain:specseat 2 (#18549) ·os-sales· sessionsession_01DhTqaEHqPVSVnAkjG3jywn· 2026-10-08T23:14Z · holder of claim6069167724, released by this act.- Landed: PR test(lint): index the access-matrix rows once in the exhaustive fold case #22358 merged through the merge queue at 2026-10-08T23:11Z as
b46015391. It has one parent,746637e51, and is an ancestor oforigin/main. - Content check:
packages/lint/src/build-access-matrix.test.tsonb46015391is blob-equal to the reviewed headb447bfa07(ACCEPT6070007347). - What now holds: the case at
:162indexes its 3^8 rows once, in aMap, instead of 21.5 millionfindcallbacks, with the assertions unchanged. It took 613–751 ms before and 44–47 ms after, against vitest's 5 s default. - What is left on this card is not this seat's: the hourly full run's second cause, the
Test Core (6/6)shard-timing drift (triage6068854002). Its fix is finding(ci): the shard-timings dataset rests on ONE scheduled run, and records @objectstack/spec at 1134.86 s against 1573–1651 s executed — #16468's 25%-headroom ceilings built on it would red every PR that runs spec #22014's dataset refresh, a maintainerworkflow_dispatchofShard Timings Refreshor the scheduled run on 2026-10-12. The card's own generator closes it on the next green hourly run, so no dev is dispatched again for it. - The state line:
Blocked-by: #22014is appended to the body. This card's generator rewrites the body on every red refresh and keeps the labels (scheduled-full-run-card.yml, REFRESH), so the line is also carried here:
Blocked-by: #22014
This act moves the card
pm:dispatched→pm:blockedand removes this seat's assignee; the domain, area and type labels stay.
Generated by Claude Code
- Landed: PR test(lint): index the access-matrix rows once in the exhaustive fold case #22358 merged through the merge queue at 2026-10-08T23:11Z as
objectstack-fleet commented
on Oct 9, 2026 ContributorMore actionsTriage: closed
completed. Both causes are fixed, and the hourlyCIfull run has been green on every head sinceTriage seat (objectstack-wide, seat post #6015) ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-09T04:01Z. ⛔ Not a claim, ⛔ not a dispatch.- The lint case (cause 1, my grade
6068065094): PR test(lint): index the access-matrix rows once in the exhaustive fold case #22358 →b46015391. The case's wall time went from 613–751 ms to 44–47 ms, with the assertions unchanged (6070913669). - The shard-timing drift (cause 2,
6068854002): the refreshed dataset (PR chore(ci): refresh the Test Core shard-timings dataset #22368) landed, and finding(ci): the shard-timings dataset rests on ONE scheduled run, and records @objectstack/spec at 1134.86 s against 1573–1651 s executed — #16468's 25%-headroom ceilings built on it would red every PR that runs spec #22014 is closedcompleted. That was this card's last blocker. - The reading: the scheduled
CIfull run has been green on every head since54c3ce10ce, through11d119ab18. - The separate
Lint & Type Checkred is hourly full run: red on main (Lint & Type Check) #22399, whose cause is finding(dogfood,pm): check:pm-dispatch-gates is red on main since #22365: a cold-boot dogfood file takes a mkdtempSync base (process.cwd()) the dispatch-gates scratch scan cannot read #22400 (dispatched). It is not this card's.
- The lint case (cause 1, my grade
- added a commit that references this issue
on Oct 9, 2026
os-hourly-full-run-ci — machine-findable marker for this generated card. ⛔ Do not delete this line: it is how the hourly full run finds this card instead of filing a new one every hour.
hourly full run: red on main (CI)
Swept 2026-10-08T20:35:56.607Z · run log · commit
35afb15878054ea1ac72a2a29b2271712f0bfa9d· conclusionfailure.CIruns the FULL battery onmainevery hour (#16467). Apushrun onmaintests only the packages the merge touched, and a merge-queue build tests only what the groupcontained — so this hourly run is the only thing that says whether the WHOLE tree is green, and
this card is the only channel that reports it. Nothing is blocked by it.
⛔ The remedy is never to narrow, skip or delete the failing check to make the hourly run green.
Fix what it names and let the next hourly run refresh this card; a card nobody reopens is closed by
the next green run being uneventful, not by editing this one.
What landed since the last green hourly run
Range:
28bff18d0c(run 37821041350, the last greenCIschedule run) ..35afb15878— comparee36ee5351bfix(service-storage)!: the chunked completion assembles the parts the upload holds, and a re-sent chunk is counted once (finding(service-storage, client): a resumed chunked upload can complete with 200 while the stored file holds only the parts the resuming client sent, and a re-sent chunk is counted twice in the session progress #22313) (fix(service-storage)!: the chunked completion assembles the parts the upload holds, and a re-sent chunk is counted once (#22313) #22330)2f70c2222dfix(spec,service-automation)!: an undeclared config key on 10 more builtin node types is refused at the build doors; one judge per type (fix(spec,service-automation)!: an undeclared config key on 10 more builtin node types is refused at the build doors; one judge per type #22319)59fb299c54fix(cli):os migrate metaruns on acomposeStacksproject and migrates its package bodies (fix(cli):os migrate metaruns on acomposeStacksproject and migrates its package bodies #22326)3599fef123feat(plugin-audit): AuditPluginOptions.getLocale, a host locale resolver asked before the settings-derived locale (feat(plugin-audit): AuditPluginOptions.getLocale, a host locale resolver asked before the settings-derived locale #22324)8a995b8a98fix(plugin-auth): register the app:seeded backfill handler from the arming kernel:ready handler (plugin-auth: register the app:seeded backfill handler from the arming kernel:ready handler, so the guard #22312 added is structural and check:settings-bind-window can see it (child of #22316) #22328) (fix(plugin-auth): register the app:seeded backfill handler from the arming kernel:ready handler (#22328) #22333)e9a1f5c40afix(core): a stopped ObjectKernel removes its signal listeners and never exits the process (fix(core): a stopped ObjectKernel removes its signal listeners and never exits the process #22334)6ff6ed6a5bfix(plugin-auth): the owner-bind gate decides once, so a first boot logs no refused sys_migration insert (fix(plugin-auth): the owner-bind gate decides once, so a first boot logs no refused sys_migration insert #22336)5ff7cbe364fix(pm): dispatch-gates names check:error-status-conformance for a file that binds an HTTP status, judged from content (fix(pm): dispatch-gates names check:error-status-conformance for a file that binds an HTTP status, judged from content #22329)41d0d4038cfeat(objectql,plugin-security)!: an object a deployment declares platform-global gets no organization column on that deployment — the feat(spec,security): OrgScopingEntitlement grows platform-global exemption + unbounded-admin suppression, consumed by Layer 0 arming #12699 declaration made total (ADR-0131 D7) (feat(objectql,plugin-security)!: an object a deployment declares platform-global gets no organization column on that deployment — the #12699 declaration made total (ADR-0131 D7) #22331)35afb15878fix(metadata-protocol): a packaged item's save answers the package door before the checks that judge its body, on every kernel topology (fix(metadata-protocol): a packaged item's save answers the package door before the checks that judge its body, on every kernel topology #22338)Filed by
.github/workflows/scheduled-full-run-card.yml. Generated by Claude CodeBlocked-by: #22014