ci(webui): the Activity performance gate fails on runner noise #1925

Problem / Background

The WebUI installed artifact job (.github/workflows/ci.yml:503) fails at random in the step "Run Activity performance gate with the hidden acceptance deferred" on changes that cannot affect decode or the Activity polling path. The job runs on the single self-hosted GB10 runner lablup-dgxspark21, one job at a time, so every rerun costs 10 to 40 minutes. Context: measured on 2026-09-19 while landing the WebUI PRs of epic #1910 (this issue is not a child of it), which still has several WebUI PRs queued.

Every failure below was the per-mode verdict investigate. Per-sample logs are in each run's webui-activity-performance-diagnostics-Linux artifact (activity-performance.log, one JSON line per request).

A faster result (-6.76) and a flat one (+0.20) both fail on baseline CV alone, and flat changes cross the 2 percent median threshold. Five pairs cannot resolve a 2 percent threshold against a per-pair spread of 6 to 21 points.

Current Behavior

• webui/scripts/activity-performance.mjs:126 sends one warmup request; :128-139 runs, per mode, 5 alternating off/on pairs (:130, order flipped each pair at :132), computes the paired degradation (off - on) / off * 100 (:133), the population CV of the 5 off rates (:135-136) and the median of 5 (:137, helper at :121), and sets status: degradation > 2 || cv > 5 ? 'investigate' : 'within-target' (:138). Each request (:111) decodes 256 tokens at about 140 to 172 tok/s, roughly 1.6 s.
• The baseline CV mostly measures slow drift across the whole block, which the alternating pairs are designed to cancel: in run 35425050365 attempt 1 the off rate fell from 171.4 to 141.7 tok/s over the two-visible block, and the mode failed on CV 7.45 although the Activity arm measured 6.76 percent faster.
• The same rule is enforced two more times: scripts/webui/verify_activity_performance.py:258-260 fails on any non-within-target summary, :269-270 requires paired_runs == 5, :275-276 re-applies degradation > 2 or cv > 5, and :39-44 hard-codes the sample counts (5 per mode). scripts/webui/summarize_activity_evidence.py:64-68 repeats the status, paired_runs == 5, <= 2 and <= 5 checks, and :77-82 emits that schema.
• webui/scripts/activity-performance-config.mjs:16-27 (performanceCompletion) aggregates only investigate; under --perf-mode visible-only-headed the top-level status is incomplete regardless, so the per-mode status is what fails the job.
• The CI step (.github/workflows/ci.yml:683-731) has a 45 minute timeout and --activity-timeout 1800; it currently finishes in about 2 minutes (run 35427681676: 06:58:02 to 06:59:58). make verify-webui-activity-performance (Makefile:880-890) drives the same harness in full mode.

Proposed Solution

Keep the intent (fail when visible Activity pages slow decode by more than the 2 percent budget) and make the verdict statistically meaningful. The properties below are required; the statistic, pair counts and retry budget are the implementer's call, sized from the measured noise in the artifacts above.

• Interval, not a bare median. Judge the paired degradations with a confidence interval (bootstrap or t-interval of the per-pair differences, the implementer's choice). Decision criterion for the rule: with the per-pair noise observed on this runner, show (by resampling the artifact data, with the calculation in the PR body) a false-failure rate of at most 1 percent on a true 0 percent change and at least 95 percent detection of the injected slowdown used in the acceptance test. Recorded tradeoff so it is not re-derived: failing only when the interval lies wholly above 2 meets both targets with a few tens of pairs at this noise, while passing only when it lies wholly below 2 needs far more pairs to keep false failures down.
• Precision gate replaces the CV verdict. Baseline CV stops being a failure condition. When the interval is too wide to decide, the harness adds pairs up to a cap; only when the cap is exhausted does the mode end as a distinct status such as inconclusive, which fails the job with a message that names measurement noise, not a slowdown. Keep baseline_cv_percent and paired_range_percent in the summary as diagnostics.
• Warmup. Discard leading pairs only if the artifact data shows a first-pair bias; state the finding either way.
• One rule, checked consistently. The Python validators must not keep an independent copy of the thresholds. Either they re-derive the verdict from the raw samples array with a port of the JS function, or they validate structure and internal consistency only (a within-target summary whose interval violates the rule is rejected). Whichever is chosen, JS and Python are tested against one shared fixture file of sample vectors with expected verdicts.
• Rejected: raising the 2 percent or 5 percent thresholds. That hides noise and real regressions alike.

Scope

In scope: webui/scripts/activity-performance.mjs (sampling loop and verdict), webui/scripts/activity-performance-config.mjs (pure verdict function and performanceCompletion handling of the new status), scripts/webui/verify_activity_performance.py and scripts/webui/summarize_activity_evidence.py (variable paired_runs, sample counts derived from it, new summary fields), their tests (webui/scripts/activity-stream.spec.mjs:41-72, scripts/webui/verify_activity_performance_tests.py, scripts/webui/summarize_activity_evidence_tests.py), .github/workflows/ci.yml only if the timeout needs adjusting, and the Activity row of docs/webui-integration-matrix.md:17 where it describes the budget.

Out of scope: the deferred native hidden acceptance (--perf-mode full on a browser that propagates document.hidden), runner provisioning or a second GB10 runner, making the job non-required, and any product code in webui/src or src/server.

Implementation Notes

• Reuse: keep performanceMode, assertGeometry, the clients() visibility checks, and the fail-closed GPU-process check at .github/workflows/ci.yml:692-697. Seed the shared fixture from the per-sample logs of the six runs above, copied into the repository now, because artifacts expire.
• Constraints: the step must stay well inside the 45 minute timeout; target under 15 minutes worst case including re-measurement. full mode (with hidden) uses the same verdict. Keep n_predict, seed, temperature: 0 and cache_prompt: false unless the PR shows why a change is needed.
• Edge cases: a negative degradation (Activity arm faster) never fails by itself; a failed or non-finite request still throws as today (:112-115); sample counts must equal 2 * paired_runs per mode plus warmup, and the validators reject any mismatch.
• Error handling: each failing mode reports its status, point estimate, interval, pair count and the rule that fired, so a reader can tell investigate (slowdown) from inconclusive (noise) in the job log without downloading the artifact.

Acceptance Criteria

• The verdict is a pure function covered by unit tests, and the JS and Python sides are tested against the same fixture of recorded sample vectors, including all six runs above, none of which returns investigate.
• Baseline CV above 5 percent alone no longer fails a mode; an undecidable interval triggers re-measurement and, at the cap, fails with a status distinct from a slowdown.
• The PR body gives the resampling estimate of false-failure rate and detection power for the chosen rule and pair counts.
• With an injected slowdown whose true effect is measured beforehand at 4 to 8 percent (for example a per-poll busy loop in the Activity page; if a browser-side loop does not reach decode, inject in the ui_runtime handler at src/server/router_server.rs:708, which serves the Activity page's /ui-api/v1/runtime poll in the router mode the harness launches), the gate fails with investigate on lablup-dgxspark21. The injection is reverted, never merged, and its run ID is in the PR body.
• The job passes on at least five consecutive runs on lablup-dgxspark21 of a tree whose only change is the gate, with run IDs in the PR body.
• The gate still runs inside the real WebUI installed artifact job on every WebUI or Rust change.

Verification

pnpm --dir webui run unit
make verify-webui-helper-tests
python3 scripts/webui/verify_activity_performance_tests.py
python3 scripts/webui/summarize_activity_evidence_tests.py
# On GB10: rerun the Activity job and record run IDs
gh run rerun <run-id> --job <job-id>

A pass is all unit tests green, five consecutive green WebUI installed artifact runs on the unchanged tree, and one red run with investigate under the injected slowdown.