ClawSweeper re-review fails with Codex transient_transport on moderate fork PRs (3/3 retries, then falls back to off-meta tidepool)

[wangwllu]

Symptom

@clawsweeper re-review on a fork PR consistently fails with reason=transient_transport from Codex retries, then falls back to a placeholder review:

• decision=keep_open confidence=low action=kept_open
• review_summary: "Review failed before ClawSweeper could summarize the requested change."
• routed verdict label: 🌊 off-meta tidepool / [P1] Review did not complete (retryable codex transport failure)

The run itself reports conclusion: success (all GitHub Actions steps succeed), so the failure is only visible by reading step 13 "Review exact event item" logs.

Repro

PR: openclaw/openclaw#92181 (fork PR, +135 src / +103 tests, 4 files, body 4243 B, diff 21460 B — moderate size).

Three independent @clawsweeper re-review attempts:

Time (UTC)	Run	Outcome
2026-06-11 14:45Z	https://github.com/openclaw/clawsweeper/actions/runs/27355240940	Failed (transient_transport)
2026-06-11 16:16Z	(status comment shows same pattern)	Failed (transient_transport)
2026-06-12 02:10Z	https://github.com/openclaw/clawsweeper/actions/runs/27389971002	Failed (transient_transport)

Run 27389971002 step 13 log excerpt

[review] 2026-06-12T02:11:27Z shard=0/1 selected=1 scanned_pages=0
[review] 2026-06-12T02:11:27Z shard=0/1 start #92181 (1/1)
[review] 2026-06-12T02:11:35Z shard=0/1 start-comment=existing #92181
[review] 2026-06-12T02:12:28Z codex-retry #92181 attempt=2/3 delay_ms=15000 reason=transient_transport
[review] 2026-06-12T02:13:30Z codex-retry #92181 attempt=3/3 delay_ms=30000 reason=transient_transport
[review] 2026-06-12T02:14:10Z shard=0/1 done #92181 (1/1) decision=keep_open confidence=low action=kept_open
[review] 2026-06-12T02:14:10Z shard=0/1 complete reviewed=1
Error: Codex failed for 1 item; review artifacts were written and the workflow recovery lane can requeue the planned set.
ELIFECYCLE  Command failed with exit code 1.

Each attempt completed in well under the configured --codex-timeout-ms 600000 and timeout 12m outer limits — Codex returned the transport error early rather than hitting the time budget.

Codex CLI invocation

From the same job log:

timeout --kill-after=30s 12m pnpm run review -- \
  --target-repo openclaw/openclaw \
  --target-dir openclaw \
  --artifact-dir artifacts/event \
  --batch-size 1 \
  --max-pages 1 \
  --codex-model internal \
  --codex-reasoning-effort high \
  --codex-sandbox danger-full-access \
  --codex-timeout-ms 600000 \
  --item-numbers 92181 \
  --readonly-openclaw \
  --shard-index 0 \
  --shard-count 1

codex-cli 0.139.0 is what was installed in the cache hit step.

Why this looks like a backend issue, not a content issue

• Three independent attempts with identical retry pattern across two days → not flake
• All three exhausted the 3-retry budget without ever returning a non-transport error → suggests the failure is at connection / streaming layer rather than model output
• PR diff and body are moderate; this isn't a giant patch
• Other PRs reviewed by the same workflow during the same window completed normally (e.g. recent runs against unrelated openclaw/openclaw PRs in the same dispatch event window succeeded), so the runner / setup / token plumbing is fine

Possible directions

The failing attempts all use --codex-reasoning-effort high, which produces long thinking streams. If the internal Codex backend or any intermediate proxy has a stream / idle timeout shorter than the long-tail thinking time on certain prompts, the connection would close and surface as transient_transport regardless of the explicit per-call timeout. That would also explain why specifically this PR keeps reproducing while other PRs in the same workflow complete.

Possible mitigations on the ClawSweeper side, in case the upstream Codex fix is not quick:

• Per-PR fallback path that retries a failed review at lower reasoning effort instead of marking 🌊 off-meta tidepool / Review did not complete
• Surface transient_transport in the verdict comment (currently it just says did not finish cleanly, which reads like a queue / dispatcher problem)
• Optional additional-prompt knob to let an author request a smaller-context retry without changing the underlying PR

Affected user-visible state

• PR carries 🌊 off-meta tidepool rating with [P1] Review did not complete (retryable codex transport failure)
• Per-PR clawsweeper-command-progress:end block shows State: Failed / Detail: The targeted re-review did not finish cleanly. Check the workflow run for details.
• No path forward for the contributor short of pinging a maintainer, since further @clawsweeper re-review calls reproduce the same outcome

Happy to provide more logs, retry traces, or attempt a smaller-diff variant of the same PR if it helps narrow down whether prompt size, reasoning effort, or backend capacity is the dominant factor.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed June 12, 2026, 9:03 AM ET / 13:03 UTC.

Summary
This remains a valid core automation bug: three live targeted re-reviews exhausted the current transient-transport retry path, and current main still replaces the requested review with a low-confidence placeholder rather than offering a successful recovery path. Same-day main changes improve failure classification but do not resolve the central contributor problem.

Reproducibility: yes. Three live exact re-review runs exercised the current retry path, exhausted all three attempts with transient_transport, and produced the same placeholder result; current source inspection confirms that outcome after exhaustion.

Next step
The failure is reproduced, bounded to identifiable review and workflow paths, and can be addressed without a product, security, or architectural decision.

Review details

Best possible solution:

Add one bounded, visibly marked lower-reasoning attempt after retryable transport exhaustion when sufficient time remains, and carry the classified capacity or network reason into both the durable placeholder review and exact-command status if every attempt fails.

Do we have a high-confidence way to reproduce the issue?

Yes. Three live exact re-review runs exercised the current retry path, exhausted all three attempts with transient_transport, and produced the same placeholder result; current source inspection confirms that outcome after exhaustion.

Is this the best way to solve the issue?

Yes, with a narrow boundary: a single explicitly marked lower-effort fallback plus accurate final status is preferable to exposing a contributor-controlled prompt or reasoning configuration knob.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• A lower-reasoning fallback can produce a less thorough review, so the resulting review should explicitly disclose degraded mode and retain the original transport diagnostic.
• The upstream transport fault may remain backend-specific and intermittent; regression tests should validate deterministic fallback routing rather than claim to reproduce the external service failure locally.
• Fallback attempts must respect the existing overall timeout budget so targeted runs do not become stalled workflow jobs.

Codex review notes: model internal, reasoning high; reviewed against 6d78394e75bc.

Label changes

Label changes:

• add P1: Repeated failures leave affected contributors unable to obtain a requested review or recover through the documented self-service command.
• add impact:other: This is a concrete user-visible review automation failure outside the owned data, message, session, authentication, security, and crash categories.
• add issue-rating: 🦀 challenger crab: Current issue advisory state selects this label.
• add clawsweeper:current-main-repro: Current issue advisory state selects this label.
• add clawsweeper:queueable-fix: Current issue advisory state selects this label.
• add clawsweeper:fix-shape-clear: Current issue advisory state selects this label.

Label justifications:

• P1: Repeated failures leave affected contributors unable to obtain a requested review or recover through the documented self-service command.
• impact:other: This is a concrete user-visible review automation failure outside the owned data, message, session, authentication, security, and crash categories.

Evidence reviewed

Acceptance criteria:

• pnpm run check
• node --test test/clawsweeper.test.ts --test-name-pattern 'Codex|transient|retry|fallback|failure'

What I checked:

• Live reproduction matches current path: The issue provides three independent workflow attempts that each exhausted all three transient-transport retries before producing the failed placeholder review; current source contains the matching retry log and exhaustion path. (src/clawsweeper.ts:6729, 76b696750df6)
• Retry implementation predates the failures: Commit 76b6967 introduced transient Codex review classification and bounded retries before the reported June 11–12 runs, demonstrating that the existing mitigation was active but insufficient. (src/clawsweeper.ts:6729, 76b696750df6)
• No degraded fallback exists: After retry exhaustion, current main throws CodexReviewError and generates the failed placeholder decision; it does not retry with lower reasoning effort or another bounded strategy. (src/clawsweeper.ts:6737, 6d78394e75bc)
• Contributor status remains generic: The exact-event workflow reports only that the targeted re-review did not finish cleanly when the review step fails, leaving the classified transport failure in workflow logs rather than the command status. (.github/workflows/sweep.yml:474, 71e4da1fa1a5)
• Latest main change is diagnostic only: Current-main commit 6d78394 distinguishes capacity failures from network failures in the placeholder summary but adds no successful recovery mechanism. (src/clawsweeper.ts:6361, 6d78394e75bc)
• Checkout remained clean: Read-only inspection left the target checkout unchanged. (6d78394e75bc)

Likely related people:

• steipete: Authored the commit that added transient Codex review detection, retry timing, review-path integration, and focused regression coverage. (role: introduced current retry behavior; confidence: high; commits: 76b696750df6; files: src/clawsweeper.ts, src/codex-transient.ts, test/clawsweeper.test.ts)
• joshavant: Recently changed exact-event review capacity/status handling and authored the current-main capacity-versus-network failure wording. (role: recent exact-review and diagnostics contributor; confidence: high; commits: 24e86607bd5a, 9f6394783b87, 4e6f4135e291; files: .github/workflows/sweep.yml, src/clawsweeper.ts, test/clawsweeper.test.ts)
• Momo: Authored the workflow-status stabilization work that currently classifies and publishes failed Codex review outcomes. (role: recent workflow-status contributor; confidence: medium; commits: 429a73be116e; files: src/clawsweeper.ts, test/clawsweeper.test.ts)
• Tak Hoffman: Git history traces the generic exact re-review completion status and surrounding workflow structure to the proof-lane workflow introduction. (role: introduced exact re-review status surface; confidence: medium; commits: 71e4da1fa1a5; files: .github/workflows/sweep.yml, src/clawsweeper.ts)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[wangwllu]

Cross-references after triage search

Searched the repo for prior reports of this symptom — transient_transport keyword has no earlier hits, so the specific failure mode in this issue is new. A few related-but-distinct issues are worth linking:

• Adaptive codex timeout based on PR size and media proofs — large PRs with videos consistently time out #272 Adaptive codex timeout based on PR size and media proofs — overlaps in that both fail inside the Codex call layer for fork PRs and both point at the same architectural seam (dispatchItemReview() client_payload not surfacing any codex_* knobs). But distinct: Adaptive codex timeout based on PR size and media proofs — large PRs with videos consistently time out #272 is about the 10-minute outer --codex-timeout-ms actually being hit on large/video PRs. The runs in this issue all return transient_transport after only ~2:46 of wall-clock per attempt — they never reach the timeout. The PR is also moderate size (~21 KB diff, 4 KB body, no video proofs), so the size-based mitigations in Adaptive codex timeout based on PR size and media proofs — large PRs with videos consistently time out #272 would not trigger here.
• re-review workflow reports "Complete" but durable verdict marker not refreshed to new head SHA #170 re-review reports "Complete" but durable verdict marker not refreshed — same surface (@clawsweeper re-review workflow) but a different leg: re-review workflow reports "Complete" but durable verdict marker not refreshed to new head SHA #170 is in the verdict sync path, this issue is in the upstream review-execution path.
• clawsweeper re-review fails for repos with default branch of 'master' instead of 'main' #252 clawsweeper re-review fails for repos with default branch of 'master' — another re-review failure mode (workflow can't fetch repo) that is unrelated to Codex transport but useful as a sibling failure category.
• ClawSweeper command status comment edits can re-enter activity handling and churn review labels #266 ClawSweeper command status comment edits can re-enter activity handling and churn review labels — text-search collision on "off-meta tidepool" but the underlying bug is label churn, not review failure.

Suggested coupling with #272

If #272's Option A is implemented (push codex_timeout_ms into the dispatchItemReview() payload), it would be a small additional step to also surface codex_reasoning_effort and a codex_retry_lower_effort_on_transient_transport flag in the same payload. That would give the fallback path proposed here ("retry the failing review at lower effort instead of dropping to 🌊 off-meta tidepool") a clean place to live without expanding the workflow surface.

[yetval]

#283 solves this: after the transient_transport retries are spent, runCodex runs one bounded lower-effort fallback pass (high -> low) while time remains, so you get a real review instead of the off-meta tidepool placeholder. It discloses degraded mode, caps confidence so it can't auto-close, and preserves the transport classification if the fallback still fails. Before/after proof and tests are in the PR.

[wangwllu]

Update on root-cause investigation, with new evidence that narrows the scope of #282.

Re-classification of the failure mode

The clawsweeper review comment on the affected PR (openclaw/openclaw#92181) already classified the failure via codexFailureReason():

failure reason: retryable codex transport failure (capacity)

That is the capacity branch at src/clawsweeper.ts:6357 (matches rate limit reached / TPM / RPM / 429 / overloaded / temporarily unavailable / try again in Ns). So the failure is Codex backend capacity rejection, not a network/transport flake and not a timeout.

Timing rules out timeout

From clawsweeper run 27355240940, the three in-process retry attempts died this fast:

attempt 1: 14:48:25 → 14:48:37   12s
attempt 2: 14:48:52 → 14:49:23   31s
attempt 3: 14:49:53 → 14:50:04   11s

Each attempt died in 11-31 seconds, far below the 600s --codex-timeout-ms budget. The codex CLI never reached model thinking; it was rejected at the request-acceptance layer.

New evidence: this is PR-specific, not a fleet-wide capacity event

I checked the 14:35Z-15:00Z window for Review event item runs in clawsweeper. Aside from #92181, every other event-item run in that window completed successfully:

Time (UTC)	Run	Result
14:48:25	Review event item openclaw/openclaw#92181	failed (this issue)
14:50:47	Review event item steipete/CodexBar#1442	success
14:53:31	Review event item steipete/CodexBar#1436	success
14:57:34	Review event item steipete/CodexBar#1433	success
14:57:41	Review event item openclaw/openclaw#91632	success (same target_repo)
14:58:52	Review event item openclaw/openclaw#92196	success (same target_repo)
14:59:49	Review event item steipete/CodexBar#1434	success

steipete/CodexBar#1442 succeeded on the same backend ~2 minutes after #92181 failed; openclaw/openclaw#91632 (same target_repo) succeeded ~9 minutes later. So the capacity rejection was not a fleet-wide event — it was triggered by something specific to the #92181 request.

Implication for the fix direction

This changes the calculus for what would actually fix #282 (and unblock #92181):

• Longer in-process backoff in codex-transient.ts would help on transient global capacity dips, but it does not help when each retry sends the same large request that gets rejected at TPM admission. Each of the three attempts on #92181 sends the same prompt; backoff alone won't change the outcome.
• Reducing per-request token cost is the only path that actually helps a PR-specific TPM rejection. The two adjacent levers are:
  • --codex-reasoning-effort medium (or low) → fewer reasoning tokens per request.
  • Surface-aware prompt trimming (cf. Adaptive codex timeout based on PR size and media proofs — large PRs with videos consistently time out #272's adaptive selection idea).

Suggested next step (no code yet)

Before opening any PR, I'd like to align on direction. The candidate I'd lean toward — and which seems closest to #260's intent — is:

Have retry-failed-reviews (the daily cron lane added in #260) automatically lower --codex-reasoning-effort to medium when the failed report's classified failure_reason is capacity. The first-pass event-review path stays at high; only the recovery lane pays the smaller-attention cost.

That keeps the user-facing surface unchanged, scopes the change to a single lane, and is consistent with #260's "automatic recovery from infrastructure errors" framing. It would also unblock #92181 on the next cron tick (next at 08:13Z / 20:13Z UTC).

Happy to put this together as a Tier-1 PR, but wanted to share the evidence and proposed direction here first in case you'd prefer #272's adaptive lane to handle this end-to-end. Let me know which direction you'd like, and I'll align the PR scope before opening it.

cc @vincentkoc