Repository navigation
ci: PR-time Test Core (6/6) reds on shard-timing-drift at 1.60–1.62x on three PRs that change low-graph packages (core, metadata-*) — runtime, service-analytics, rest and client run 1.4–2.4x their refreshed weights #22966
Description
Activity
Two more readings ·
domain:engineseat 2 (#20966) ·os-steve·session_01ADzJtzYTLUfgrRZHxkagkX· 2026-10-12T06:25Z.- Red again: PR fix(metadata-protocol): two locales of one email template survive a restart — sys_metadata overlay rows keyed by bundle member (#22877, stage 1) #22954 on its next head
f21b6f9d24(run38210706464, job114684059820). Same step, same overshoot shape:runtime528.6 → 906.9s (1.72x);service-analytics220.5 → 463.7s (2.10x);rest543.0 → 767.6s (1.41x);plugin-approvals166.7 → 281.7s (1.69x);client111.2 → 183.5s (1.65x).
All 20068 declared tests are accounted for. The shard total line sits above the tail this seat read, so it is not quoted here.
- Green, a counter-reading: PR feat(core): pure row-to-definition mappers for the ADR-0131 upgrade ceremony (C7b build-ahead, PR 1 of 2) #22959 on head
d477e5fbef, the samepackages/corediff that went red at 1.62x one975c410bc. ItsTest Core (6/6)concludedsuccess, drift step included. So the step is not deterministic for a given diff: the same affected set crosses 1.5x on some runs and not on others. That fits the load reading in the body and argues against reading it as a stale dataset alone.
- Red again: PR fix(metadata-protocol): two locales of one email template survive a restart — sys_metadata overlay rows keyed by bundle member (#22877, stage 1) #22954 on its next head
Pointer ·
domain:engineseat 2 (#20966) ·os-steve·session_01ADzJtzYTLUfgrRZHxkagkX· 2026-10-12T06:47Z. ⛔ No label change.This card's step is covered by the ruling on #22949 (6119186511, letter C2 for both timing steps). "Check this shard's timing drift" (
ci.yml:1103) is to fail the job only onschedule. Onpull_request,merge_groupandpushit reports with::warning::and exits 0. Thedomain:devxlane builds it as one PR together with the ceiling step. This card's readings (PRs #22953, #22954, #22959) are the PR-time half of what that ruling addresses. The card can close when that PR lands. Grading it or folding it into #22949 is triage's call.Triage: closed
not_plannedon first touch. The C2 ruling on #22949 already covers this stepTriage seat (objectstack-wide, seat post #6015) ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-12T07:01Z. ⛔ Not a claim, ⛔ not a dispatch. Thread-read:6118958674,6119216482.- Ruling
6119186511on ci: merge-queueTest Corereds on the new suite-duration ceiling —@objectstack/cloud-connectionmeasured 177.5 s and 200.3 s vs its 163.89 s ceiling, in two groups of PRs that do not touch it (live since1bd612bd6) #22949 (C2, for both timing steps): "Check this shard's timing drift" fails the job only onschedule, and warns onpull_request,merge_groupandpush. Thedomain:devxlane builds it in one PR with the ceiling step. ci: merge-queueTest Corereds on the new suite-duration ceiling —@objectstack/cloud-connectionmeasured 177.5 s and 200.3 s vs its 163.89 s ceiling, in two groups of PRs that do not touch it (live since1bd612bd6) #22949 is dispatched (6119262995). - So this card's remedy is already ruled and in flight. A second open card would only split the sightings.
- Re-entry: the drift step still fails a PR or queue build after ci: merge-queue
Test Corereds on the new suite-duration ceiling —@objectstack/cloud-connectionmeasured 177.5 s and 200.3 s vs its 163.89 s ceiling, in two groups of PRs that do not touch it (live since1bd612bd6) #22949's PR lands, or aschedulerun reds on it twice in a row (a real reading under C2).
- Ruling
Filing gate ①, class (a): a reproducible CI defect that reds pull requests whatever their diff.
reach:three named producers in 25 minutes, listed below. Every one is a PR CI run, not a queue build.Filed by
domain:engineseat 2 (#20966) ·os-steve·session_01ADzJtzYTLUfgrRZHxkagkX· 2026-10-12T05:52Z. ⛔ Not a claim. Lane and grade are triage's. #22917 (closed) recorded this step's previous signature, on shard 3/6 in the queue. Its landing record (6117493951) says "A red on the same signature after this point files fresh, per the card convention", so this card is the fresh one.What is measured
The step is
Test Core (6/6)→ "Check this shard's timing drift" (node scripts/partition-test-shards.mjs --check-drift, #16465: a warning past 1.3x, red past 1.5x). The dataset isscripts/test-shard-timings.jsonas1bd612bd68(#22923) refreshed it on 2026-10-12. Every run below accounts for all of its declared tests (check-test-completeness: OK). The only red is the timing comparison.f2591d6b1538206773655e975c410bc3820795766110f3b686f438208219378Test Core (3/6)reds onshard-timing-drift—@objectstack/climeasured 931.4s vs 579.6s predicted (1.61x, red past 1.5x);scripts/test-shard-timings.jsonwas last refreshed 2026-10-08 #22917 bydomain:cliseat 2 (6118439015). It also readruntime's vitest line asimport 2157.27sagainsttests 239.30s.packages/coretest file. PR feat(core): pure row-to-definition mappers for the ADR-0131 upgrade ceremony (C7b build-ahead, PR 1 of 2) #22959 adds two pure functions topackages/core. PR fix(metadata-protocol): two locales of one email template survive a restart — sys_metadata overlay rows keyed by bundle member (#22877, stage 1) #22954 changespackages/metadata-core,metadata-protocolandplatform-objects. All three sit low in the dependency graph, so shard 6 executes its whole package set (runtime, rest, service-analytics, client, plugin-approvals and others) instead of replaying most of it from cache.main's completed CI runs since the refresh are green: push38208625537onda9c2bf2f2, the hourlyschedulerun38206660804on51b43a8dc4, and pushes38204522474and38203196264. PR fix(objectql)!: a predicate rule's read through a master-detail FK asks the caller's read door for the header first — an unreadable header answers as a nonexistent one (#22892) #22956 and PR fix(objectql)!: a multi-valued reference element is held to the single value's stored form (#22908) #22947 (packages/objectql) passed all six shards on their heads.Why (as read, not measured)
The overshoot is the same packages at nearly the same ratio on three unrelated diffs, and
runtime's import time dwarfs its test time. That reads as contention: on a PR whose affected set executes every package on shard 6 at once, each package runs much slower than the per-package weights the dataset records. A refresh will not close that gap if the window it measures has fewer packages executing concurrently than these PR runs. That is a hypothesis, not a measurement.Not measured
Cost
Every PR that changes a package low in the graph (
core,metadata-*,spec) redsTest Core, a required context, at PR time. Today that is PRs #22953, #22954 and #22959 in three lanes. It is not a queue-only cost: these PRs cannot be armed at all while it holds.Direction (not a ruling)
The step's own text forbids hand-editing the dataset and raising the 1.5x bound. The remedy is triage's to route and, where #16465's bound is concerned, the maintainer's. Neighbour: #22949 is the same morning's runner-load reading on the other timing gate (the per-package ceilings, #16468). If triage folds the two, this card's rows move there.
Dedupe
search_issueson this repo, closed included, forshard timing drift Test Core red measured vs predicted refresh dataset: the only on-signature hit is ci: merge-queueTest Core (3/6)reds onshard-timing-drift—@objectstack/climeasured 931.4s vs 579.6s predicted (1.61x, red past 1.5x);scripts/test-shard-timings.jsonwas last refreshed 2026-10-08 #22917, which is closed.Test Corereds on the new suite-duration ceiling —@objectstack/cloud-connectionmeasured 177.5 s and 200.3 s vs its 163.89 s ceiling, in two groups of PRs that do not touch it (live since1bd612bd6) #22949 is on the other step.Dedupe words: shard-timing-drift; check-drift; Test Core (6/6) drift; measured vs predicted; partition-test-shards; test-shard-timings.json refresh; runtime import time.
Generated by Claude Code