chore(scripts): check.sh runs the browser suites, and the cycle suite is repeatable (#314) - #322
Merged
Merged
Conversation
…each works `scripts/cycle_server.sh` rebuilds the workspace once per *server start* and `--repeat-each` reuses that one server, so a project created from a fixed literal made the flag useless: repeat 2 onward died on `POST /projects -> 409`, a wall standing in front of everything the suite is about. Repetition meant N whole invocations at about ninety seconds of rebuild each. Unique names rather than teardown, because teardown that has to survive a mid-run crash is a second bug surface, and there is nothing to tear down when nothing collides. The project is the only name that had to move, and the spec now says so, so the next collision is looked for rather than assumed: a release tag is unique per dataset, a batch name is not unique at all, and a source's idempotency key (project, kind, path, fps) leads with the project. Verified rather than reasoned about: --repeat-each=3 gives three `POST /projects -> 201`, zero 409 anywhere in the run, and the only 4xx at all is the expected schema-less-project 404, once per repetition. Refs #314
…ings check.sh ran no browser suite while calling itself the canonical "before you say it works" invocation. Two of the three suites were invisible to it, and the real-server cycle run was three separate times the *only* one to catch a regression during the 2026-08 remediation run (#306, #308, #309) -- one of which shipped on a green run of this script and went red in CI. The full run is the default and --fast is the exception. That inversion is the point: the previous default was silently the fast one. - `browser` is addressable as a group and is a complete run on its own, because each Playwright config's webServer.command builds the engine and the design system first. - CI=1 is set by the script, for the Playwright steps, from frontend/app, in a subshell so the cd cannot leak into a later group. It is load-bearing: reuseExistingServer: !CI means a stale vite server on :5273 answers instead of the build under test. The lesson lived in a skill and in memories; it lives in the tooling now. - Skipping prints a banner, *after* the verdict -- it exists to qualify "All checks passed", so it has to be the thing still on screen once that line has scrolled off. It prints whenever the browser suites did not run, not only under --fast: a partial run is the same lie however it was asked for. - Per-step timings with a total, so the cost is visible and a step that quietly doubles gets noticed. - --fast together with an explicit `browser` group is refused rather than resolved. Either answer would guess at which half of a contradictory command line was meant. The timing table is length-checked before expanding, and the reason is written down: macOS still ships bash 3.2, where "${arr[@]}" on an empty array is an unbound variable under set -u and kills the script. The neighbouring `failed` block had always been written that way and never said why. No CI or ruleset change. The two jobs already run these suites and stay separate -- each is a required check with its own cache and report artifact -- and no job name moved. Refs #314
JArmandoAnaya
enabled auto-merge (squash)
August 5, 2026 02:15
JArmandoAnaya
added a commit
that referenced
this pull request
Aug 21, 2026
… is repeatable (#314) (#322) * test(app): the cycle suite's project name is run-scoped, so --repeat-each works `scripts/cycle_server.sh` rebuilds the workspace once per *server start* and `--repeat-each` reuses that one server, so a project created from a fixed literal made the flag useless: repeat 2 onward died on `POST /projects -> 409`, a wall standing in front of everything the suite is about. Repetition meant N whole invocations at about ninety seconds of rebuild each. Unique names rather than teardown, because teardown that has to survive a mid-run crash is a second bug surface, and there is nothing to tear down when nothing collides. The project is the only name that had to move, and the spec now says so, so the next collision is looked for rather than assumed: a release tag is unique per dataset, a batch name is not unique at all, and a source's idempotency key (project, kind, path, fps) leads with the project. Verified rather than reasoned about: --repeat-each=3 gives three `POST /projects -> 201`, zero 409 anywhere in the run, and the only 4xx at all is the expected schema-less-project 404, once per repetition. Refs #314 * chore(scripts): check.sh runs the browser suites, with --fast and timings check.sh ran no browser suite while calling itself the canonical "before you say it works" invocation. Two of the three suites were invisible to it, and the real-server cycle run was three separate times the *only* one to catch a regression during the 2026-08 remediation run (#306, #308, #309) -- one of which shipped on a green run of this script and went red in CI. The full run is the default and --fast is the exception. That inversion is the point: the previous default was silently the fast one. - `browser` is addressable as a group and is a complete run on its own, because each Playwright config's webServer.command builds the engine and the design system first. - CI=1 is set by the script, for the Playwright steps, from frontend/app, in a subshell so the cd cannot leak into a later group. It is load-bearing: reuseExistingServer: !CI means a stale vite server on :5273 answers instead of the build under test. The lesson lived in a skill and in memories; it lives in the tooling now. - Skipping prints a banner, *after* the verdict -- it exists to qualify "All checks passed", so it has to be the thing still on screen once that line has scrolled off. It prints whenever the browser suites did not run, not only under --fast: a partial run is the same lie however it was asked for. - Per-step timings with a total, so the cost is visible and a step that quietly doubles gets noticed. - --fast together with an explicit `browser` group is refused rather than resolved. Either answer would guess at which half of a contradictory command line was meant. The timing table is length-checked before expanding, and the reason is written down: macOS still ships bash 3.2, where "${arr[@]}" on an empty array is an unbound variable under set -u and kills the script. The neighbouring `failed` block had always been written that way and never said why. No CI or ruleset change. The two jobs already run these suites and stay separate -- each is a required check with its own cache and report artifact -- and no job name moved. Refs #314
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #314.
bash scripts/check.shran no browser suite while calling itself the canonical"before you say it works" invocation. Two of the three suites were invisible to it, and
the real-server cycle run was three separate times the only one to catch a regression
during the 2026-08 remediation run (#306, #308, #309) — one of which shipped on a green
run of this script and went red in CI.
Two parts, because the second was blocked on the first.
Part 1 — the cycle suite is repeatable now
--repeat-eachcould not amplify the cycle suite.scripts/cycle_server.shrebuilds theworkspace once per server start,
--repeat-eachreuses that one server, and the speccreated its project from a fixed literal — so repeat 2 onward died on
POST /projects → 409, a wall standing in front of everything the suite is about.Repetition meant N whole invocations at about ninety seconds of rebuild each. (Found while
proving #319's fix; recorded in that PR's body as "found, not fixed".)
cycle/cycle.spec.tsderives the project name fromtestInfo.repeatEachIndex. Uniquenames rather than teardown: teardown that has to survive a mid-run crash is a second bug
surface, and there is nothing to tear down when nothing collides.
The project is the only name that had to move, and the spec now says so, so the next
collision is looked for rather than assumed — a release tag is unique per dataset, a batch
name is not unique at all, and a source's idempotency key
(project, kind, path, fps)leads with the project. All three are already scoped by a project that is new. Verified
rather than reasoned about: three
POST /projects → 201, zero 409 anywhere in the run,and the only 4xx at all is the expected
GET /projects/{id}/schema → 404on a schema-lessproject, once per repetition.
Part 2 —
check.shruns all three suites--fastis the exception. That inversion is the point:the previous default was silently the fast one.
browseris addressable as a group (bash scripts/check.sh browser), and it is acomplete run on its own — each Playwright config's
webServer.commandbuilds@visionset/annotatorand@visionset/ui-corefirst, becausefrontend/appresolvesthem through their
dist/.CI=1is set by the script, for the Playwright steps, fromfrontend/app, in asubshell so the
cdcannot leak into a later group. It is load-bearing:reuseExistingServer: !process.env.CImeans a stale vite server on :5273 answers insteadof the build under test. That lesson lived in a skill and in three people's memories; it
lives in the tooling now.
passed", so it has to be the thing still on screen once that line has scrolled off. It
names both suites, both CI jobs, the 3×-only-detector fact, and the command that runs
them. It prints whenever the browser suites did not run, not only under
--fast— apartial run is the same lie however it was asked for.
so it is worth seeing where they go, and worth noticing the day a step quietly doubles.
--fasttogether with an explicitbrowsergroup is refused, not resolved. Eitheranswer would be a guess about which half of a contradictory command line was meant.
The script header now documents the three suites, the
CI=1requirement and the stale-5273failure mode it prevents, and the
git ls-filescaveat (gates read the index, so anunstaged new file is invisible to them and passes locally while failing in CI).
Found, not fixed
"${arr[@]}"on an empty array is an unboundvariable under
set -uand kills the script. Nothing reaches the new timing table emptyarray today, but it is one refactor away, so it is length-checked with the reason written
down — the existing
failedblock has always been written that way and never said why.No CI or ruleset change
CI's
annotator e2e (chromium)andbrowser cycle (chromium)jobs already run thesesuites, and they stay separate jobs — each is a required check with its own Playwright
cache and its own report artifact. No job name moved, so the
mainruleset's twelvecontexts are untouched.
annotator bench (chromium, manual)stays out of the default herefor the same reason it stays out of the ruleset: it is
workflow_dispatch-only.Test plan
Acceptance for Part 1 —
--repeat-each=3against a single persistent workspace:51 s for three repetitions, against ~90 s of rebuild per repetition before.
Acceptance for Part 2 — one full
bash scripts/check.sh, and one--fastshowing thebanner. Both below.
Full
bash scripts/check.sh— all three suites, exit 0:Worth reading off that table: the browser suites are 24% of the wall clock and
python testsis 61% of it. The cost of honesty here is smaller than the cost ofpytest, which nobody was proposing to skip.
Edge cases, all exit 2 with the reason named:
--nope(unknown flag),nope(unknowngroup),
browser --fast(contradiction).bash scripts/check.sh --fast— browser suites skipped, exit 0, banner last: