Reduce duplicate release QA work with coverage-aware orchestration

[janicduplessis]

Problem

Stable-release QA repeats expensive native preparation, cold builds, warm launches, and cleanup across template lifecycle/cache suites and real-repository manual/lifecycle/cache passes. This makes the release's critical path unnecessarily long and hard to track. The 1.0.0 release retains its existing checklist; the release owner requested this follow-up for implementation after publication.

Evidence

• RELEASE.md section 3 requires the full major-release matrix on two representative real repositories. docs/field-test-protocol.md also requires manual launch/error/recovery and cache evidence.
• In 1.0.0 QA, the real Expo iOS lifecycle and cache suites took approximately 10 and 20 minutes; Android took approximately 7 and 20 minutes. Real bare iOS took approximately 5 and 10 minutes, in addition to native preparation and manual interaction.
• Both lifecycle and cache suites independently exercise cold launch, cross-worktree artifact reuse, and cleanup. Manual verification adds another warm launch, although UI interaction and deliberate error/recovery are distinct valuable checks.
• The campaign also found genuine correctness/remedy issues (Preserve shared-build results across prebuild cache-key changes #576, iOS simulator launch hits a 10-second timeout during concurrent cold boots #578, Resolve shared app schemes when the workspace name differs #589, Bare iOS reload remedy assumes a Reload control that may not exist #596) and a needless populated-Metro-cache wait (Avoid labeling settled nonempty Metro QA caches as EMPTY #583). Reducing QA must preserve the assertions that caught those failures.
• Heavy native work runs sequentially on the 16 GB QA host. Reviews, report assembly, packaging inspection, and remote CI need not all sit on that same critical path.

Cause

The runners own separate fixtures and repeat overlapping setup. Release orchestration lacks a shared coverage/evidence manifest, a dependency-aware schedule, and a clear policy for retaining unaffected evidence after a narrow candidate fix. Preparation/tooling friction and real regressions also consume time; not all elapsed time is removable test duplication.

Fix idea

• Combine compatible lifecycle and cache execution so one cold/warm pair satisfies both sets of assertions; add manual UI/error/recovery at the appropriate lifecycle stage.
• Preserve deliberate fresh-cache boundaries, single-flight races, final-key correctness, cache engagement/storage/reuse, launch verification, process/device ownership, and all cleanup assertions. Never let setup preseed a cache that a test claims is cold.
• Define which candidate changes invalidate which evidence. Rerun affected rows after a fix, retain unaffected results with their exact commit and a documented applicability reason, and keep final packaging/version/exact-commit CI gates.
• Schedule low-impact review/report/package inspection and remote CI in parallel where safe. Keep benchmark measurements isolated and heavy native builds sequential on this host unless resource measurements justify another policy.
• Produce a concise current-state report plus stage timings; distinguish setup, active test time, regressions, retries, and review/CI wait. Preserve private failed-attempt evidence.
• Update release documentation and actual orchestration together. Validate equivalent coverage and measure total critical-path reduction on representative Expo and bare apps, including a seeded failure that must still fail.

Do not shorten the active 1.0.0 checklist as part of this ticket. Claim and implement after that release is published.

[janicduplessis]

Testing improvement plan — 2026-09-11

Keep the current host-based approach. No VMs. Improve isolation, synchronization, prerequisites, and reuse of expensive native QA. The first batch is merged in PRs #725, #732, #734, #735, and #736 after clear agent reviews and passing PR CI; later stages below remain planned work.

1. Make the reported failures reproducible, then fix their narrow causes. Each test owns its temporary state and sockets, races wait for an observable event, and real-I/O cases get explicit bounded timeouts. Preserve real process contention and real-tool argument checks.
2. Complete the remaining isolation audit in Give remaining real-I/O tests isolated ports and explicit time budgets #733. Separate fast deterministic tests, real Git/process/socket integration tests, and explicit hardware/native compatibility checks. A disconnected personal phone must not decide the ordinary unit result. Verify with concurrent worktree test runs and unrelated pre-existing listeners/state; measure worker limits before changing defaults.
3. Implement Reduce duplicate release QA work with coverage-aware orchestration #598 incrementally: inventory the assertions and their required fixture/cache state, then share compatible lifecycle/cache setup and cold/warm builds. Record candidate SHA, fixture/tool versions, environment, applicable assertions, results, artifacts, and stage timings. Missing evidence fails the gate. Preserve fresh-cache boundaries, one-builder races, final cache keys, cache engagement/storage/reuse, verified launches, ownership, cleanup, and manual UI/error/recovery. Validate a combined run against the existing suites on Expo and bare apps, including injected failures that must still fail. Measure critical-path reduction before replacing the existing release procedure.
4. Finish platform-specific compatibility and interrupted-cleanup work separately. Keep heavy native builds sequential on this host and benchmark measurements isolated. Carry prior QA evidence across a candidate fix only with an explicit applicability reason; packaging and exact-commit CI remain required.

Issue	Coverage and completion criterion
#714 — shared temporary-directory assertions	PR #725 verifies cleanup only for the invocation's own directory and preserves seeded unrelated state. Focused tests and CI pass. The separate interrupted-process residue report remains unconfirmed; production already uses finally.
#719 — managed-start race flakes	PR #732 uses OS-assigned ports, listener readiness, explicit lock-contention barriers, and bounded timeouts. Occupied-port before/after proof, a missing-handoff fault, 20 repeated runs, and CI pass. The exact original Node 24 CI cause is still unconfirmed.
#702 — native harness prerequisites and reused names	PR #734 checks SDK/locale before setup and gives runs unique device names. Four regressions fail on the old harness; all 84 end-to-end checks pass. No full native build rerun yet. Interrupted Gradle cleanup still needs a controlled failure reproduction and proof that owned daemons exit before their home is removed.
#730 — partial readiness JSON	PR #735 publishes the completed file by rename. A paused partial write fails before and passes after; 12 race runs with four concurrent runners pass.
#731 — large Git fixture timeout	PR #736 gives this real-I/O case 30 seconds, retaining the actual greater-than-1MB Git output and copy assertions. This fixes the reported budget mismatch, not arbitrary overload.
#733 — remaining real-I/O isolation	Newly recorded follow-up for other fixed-port cases, load-sensitive process tests, and connected-phone dependencies observed during this work. Completion requires concurrent suite proof and explicit placement of hardware checks without losing real-tool coverage.
#598 — duplicated release QA	Covered by stage 3. No orchestration rewrite has been implemented. Accept only after equivalent assertions, seeded-failure detection, and measured runtime savings on representative Expo and bare apps.
#469 — benchmark sandbox compatibility	Retain host isolation. Existing detection is not build compatibility proof. Evaluate the existing toolchain adapter with untimed native build and recording checks under the exact policy, including forbidden-path denial and owned-device cleanup, before accepting timed runs. No frozen campaign change or weakened outer sandbox.
#575 — Expo/RN prebuilt mismatch	This is upstream build compatibility, not a test-runner bug. Search upstream first, record the fixture's source/prebuilt settings, and retest default and supported fallback paths with real native launch evidence. Do not turn the documented workaround into a default-path pass.
#604 — generated Test App fingerprint inputs	The scoped disposable-fixture fix already has real lifecycle/cache proof in the issue. Remaining work is project/upstream guidance; keep parity across worktrees and sensitivity to real native input changes.

#701 is closed as an invalid false-pass report: the shell pipeline reported tail's exit status. Preserve runner status and missing-check assertions, but do not redesign around that diagnosis.

#710 and #720 are production AVD recovery/teardown defects found through QA. Unique test names prevent a fresh suite from reusing an earlier name; they do not repair or reclaim orphaned AVDs. Those issues still need their own fixes and real Android-tool verification. Other product work, including #709 and #696, also remains outside what a test-harness change can resolve.

The plan accounts for all currently identified testing-related issues, but the first batch does not close them all. In particular, interrupted cleanup, remaining shared resources, QA deduplication, and native compatibility still require the evidence listed above. The five first-batch PRs are merged; no release has been performed.

[janicduplessis]

Current triage: complete #733 test isolation before changing release orchestration. This remains a separate campaign: a shared lifecycle/cache run must preserve the existing cold-cache and single-flight assertions, detect seeded failures, and demonstrate equivalent Expo/bare coverage with measured runtime savings. The current release gate is unchanged; passing individual bug-fix PRs does not satisfy this ticket.

[janicduplessis]

Claimed: @janic/qa-orchestration-598