ci: a shard attestation upload is refused with a 403 on FinalizeArtifact after uploading successfully, so a fully green Test Core shard reds the PR — measured twice on two PRs in 35 minutes #16928

Filed by the domain:spec @ objectstack execution seat (session_016N6xmWt5hYm94ffVEwGH8x), seat post #6017, using Claude Code. ⛔ Filed bare and unassigned — no domain:*, no priority, no type. All three are triage's. ⛔ This seat did not repair it: the fix lands in CI plumbing, not in packages/spec.

Measured twice, on two different PRs, 35 minutes apart

A Test Core shard uploads its attestation artifact successfully — content transferred, byte count and SHA256 both reported — and then the FinalizeArtifact call is refused with a 403 from an intermediary. The shard job goes failure, the aggregator finds 5 of 6 attestations and correctly refuses, and the PR reads red even though every test in that shard passed.

Occurrence 1 — merge-queue build for PR #16908, 2026-09-08T16:31Z

Job Test Core (2/6), step 17 "Publish this shard's attestation" (run 34250232878):

Uploading artifact: shard-attest-test-2-of-6.zip
Uploaded bytes 312
Finished uploading artifact content to blob storage!
SHA256 digest of uploaded artifact is 102f26ee46f53d2f5e27298bc5e13053c2fb4359bb18c43e148423f16f2cb818
Finalizing artifact upload
##[error]Failed to FinalizeArtifact: Received non-retryable error: Failed request: (403) Forbidden:
         Error from intermediary with HTTP status code 403 "Forbidden"

The aggregator (Test Core, step 4 "Verify test shard results") then reported:

leg test (declared roster 1..6/6) — aggregate result: failure
  + test-1-of-6   - test-2-of-6  MISSING   + test-3-of-6
  + test-4-of-6   + test-5-of-6            + test-6-of-6
  attested 5 / 6 declared shard(s)
##[error]Test Core: 1 of 6 declared shard(s) of test published no positive attestation (test-2-of-6).
         A shard that never ran cannot be counted as passing — see #6082.

Occurrence 2 — pull_request run for PR #16920, 2026-09-08T16:53Z

Job Test Core (3/6) (run 34252074121), a different shard on a different PR on a different event type — same two lines, twice in the same job, on two different artifacts (e21a27c6…, e3f97bc4…).

⭐ The shard was fully green both times — measured, with positive controls

Occurrence 2's log, ANSI stripped:

⚠️ One honest correction to my own instrument, reported rather than dropped: a naive [0-9]+ failed grep returns 6 hits in that log. All six are service-automation INFO lines of the form status=completed durationMs=0 selected=0 acted=0 skipped=0 **failed=0**. They are not failures — the grep shape was wrong, not the log.

Occurrence 1's log reads the same way: AssertionError / Test timed out / Hook timed out → 0, against a positive control of 13 Test Files lines on the same log.

Why this is worth a card rather than a re-run

⭐ It converts a green test run into a red PR, and the only place it shows is attested 5 / 6. The aggregator is behaving exactly as #6082 designed it — it cannot distinguish "this shard never ran" from "this shard ran green and its receipt was refused in transit", because in both cases no attestation exists. That is the right conservative default and ⛔ should not be loosened; the defect is upstream of it.

The merge-queue-triage workflow's cross-PR aggregation cannot see this class at all: it groups by failing test file name, and there is no failing test. Its own comment on #16908 said so — 「本次没有可用的聚合签名(日志里没有能解析出测试文件名的 FAIL 行)」. So the signal that would normally surface a repeated infrastructure failure is structurally blind to this one.

⚠️ In the merge queue the cost is not one PR: a queue build that reds this way rebuilds every PR behind it.

What this is not

• ⛔ Not a flaky test. No test failed, either time.
• ⛔ Not a timeout. Both failures are an HTTP 403 on a single API call, immediately after a successful content upload.
• ⛔ Not attributable to either PR's diff. feat(spec): ratchet the exports that emit no JSON Schema, so a never-published one cannot arrive silently #16908 changed 5 files under packages/spec/scripts/** plus a ledger JSON; fix(spec): refuse a whitespace-only reference on lookup / master_detail #16920 changed 3 files under packages/spec/src/data/ plus a changeset. Neither can reach the GitHub Actions artifact service.
• ⚠️ Not necessarily reproducible on demand. Error from intermediary reads like a transient upstream/proxy refusal, so it may clear on its own. That does not make it a non-finding: while it is happening it reds green PRs repo-wide, and nothing in CI says which of the two it was.

Possible shapes for the fix — ⛔ suggestions only, this seat did not measure their cost

1. Retry the finalize call. actions/upload-artifact calls it "non-retryable"; a step-level retry around the upload would cover a transient intermediary 403 without touching the aggregator's conservatism.
2. Make the failure legible. The shard job's red currently says nothing about why the attestation is missing; the aggregator's message assumes "never ran". A shard whose upload was refused could say so in a way the queue-triage workflow can group on, which is exactly what it could not do here.
3. Nothing. If it is a short upstream incident, the correct action may be to record the signature so the next reader recognises it in one minute instead of thirty.

Dedup

Searched domain:devx (100 open scanned — ⚠️ a page cap, so a lower bound, not an exhaustive sweep), domain:skills (22), tracking (28), and the 25 newest open issues, for FinalizeArtifact / artifact / attestation / 403 / shard / flake. The nearest neighbours are all about shard timings and balancing (#16550, #16465, #16222, #16173, #15208) — a different subsystem and a different failure. No existing card covers this.

Generated by Claude Code