Skip to content

fix(runway): ISS-004 retry transient Git failures - #678

Merged
sbalabanov merged 1 commit into
mainfrom
sergeyb/fix-iss-004-git-classifier
Sep 10, 2026
Merged

fix(runway): ISS-004 retry transient Git failures#678
sbalabanov merged 1 commit into
mainfrom
sergeyb/fix-iss-004-git-classifier

Conversation

@sbalabanov

@sbalabanov sbalabanov commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Intent:

  • Prevent temporary Git process and remote failures from being dead-lettered on their first delivery.
  • Preserve terminal merge-result behavior and fail fast for recognizable permanent Git configuration failures.

Changes:

  • Add structured Git command errors and a centralized Git classifier for retryability and dependency attribution.
  • Preserve Git process causes through Runway merger wrapping and wire the classifier into the primary consumer.
  • Cover classifier decisions and verify Runway nacks transient Git failures while rejecting unknown failures.

Reproduction:

  • A merge delivery runs git fetch origin or git push origin ... while the remote temporarily resets the connection, producing a wrapped *exec.ExitError.
  • Previously Runway registered only generic and MySQL classifiers, so the error stayed non-retryable and the consumer rejected it to the DLQ after one attempt.
  • With this change the structured Git error is classified as a retryable dependency failure, so the consumer nacks it for redelivery; authentication, invalid repository/configuration, process-start, and unknown command failures remain non-retryable.

Generated by the 🪄 pr-create skill in devexp-agent-marketplace

Test Plan

Issues

T3-ISS-004

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@sbalabanov sbalabanov left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed for classifier idiomaticity against platform/errs/README.md.

What's right

  • CommandError is a clean carrier: it records provenance without assigning retry policy, which is exactly what "Extensions return plain errors" asks for.
  • Classify type-asserts a single node and never calls errors.Is/errors.As — the contract the README is emphatic about.
  • Terminal merge outcomes still short-circuit in runway/controller/merge (merger.IsTerminal → FAILED result + ack), so conflicts and invalid requests never reach the processor. That's the load-bearing bit and it survives the change.
  • Dependency attribution (InfraDependency* for the remote subcommands) is worth having on its own — errs.Attribution feeds the failure record regardless of retryability.
  • Extracting newPrimaryErrorProcessor so the wiring is testable is a nice touch.

The classification policy is inverted

The README states the contract as: "Non-retryable by default … Retryability must be explicitly opted into. This prevents accidental infinite retry loops from unclassified errors."

This classifier does the reverse. Any of 14 allowlisted subcommands that exits non-zero is retryable unless its stderr happens to contain one of 9 English substrings. So the axis is the subcommand — but a subcommand carries no information about whether its failure is transient. git fetch fails transiently on a connection reset and permanently on a deleted branch; both land in the same bucket.

I ran the classifier against real git failures, constructed through the same runAs shape the merger uses (git 2.x, throwaway repo):

command git's stderr verdict
rev-parse origin/main (branch absent) fatal: ambiguous argument 'origin/main': unknown revision or path not in the working tree. InfraRetryable
cat-file -e <missing sha> (empty) InfraRetryable
commit -m … (nothing to commit) (empty) InfraRetryable
merge-base --is-ancestor HEAD HEAD (unborn HEAD) fatal: Not a valid object name HEAD InfraRetryable
clean -fdx -- /etc fatal: '/etc' is outside repository at … InfraRetryable
push origin main (non-fast-forward) ! [rejected] main -> main (fetch first) InfraDependencyRetryable

All six are deterministic and fail identically on every redelivery. Row 1 is not hypothetical: both resetToRemote and refetchTipSHA run rev-parse <remote>/<target>, so a misconfigured or deleted target branch lands there.

Blast radius, stated honestly: Runway's primary subscriptions use DefaultSubscriptionConfig (Retry.MaxAttempts = 3), so this is 3 attempts instead of 1 before the DLQ — bounded, not an infinite loop. But each redelivery re-runs the whole merge (fetch, reset --hard, clean -fdx, the cherry-picks, and in promote up to MaxPushAttempts pushes against the remote), so it is 3× the git and remote work for something that can never succeed, and it delays the FAILED signal the client is waiting on by the backoff. The reason I'd still call it blocking is the direction of the default rather than today's cost: a classifier is a platform component, and this one makes unrecognised git failures retryable, which is the specific thing the README's default exists to prevent.

Suggested direction

Key on the failure, not the subcommand — an allowlist of known-transient signals, everything else Unknown:

  1. Killed by a signal (ExitError.ProcessState, Signaled() / ExitCode() == -1) — OOM-kill, SIGTERM on drain, context-cancel kill. This is the one genuinely transient case, and it's currently caught only as a side effect of the blanket default.
  2. Known-transient remote diagnostics on fetch/push/ls-remote: connection reset by peer, could not resolve host, the remote end hung up unexpectedly, early eof, rpc failed, operation timed out, connection refused, 502/503, ssh_exchange_identification, remote end hung up.
  3. Local contention: index.lock / unable to create ... File exists, which resolves on retry.

Everything else returns Unknown and dead-letters on attempt 1 as it does today. That inverts the failure mode: a transient case you forgot to list costs one lost retry, instead of a permanent case you forgot to list costing three full merge runs. It also shrinks the list you have to maintain — the transient set is short and stable, the permanent set is unbounded.

Details inline.


🤖 [posted by agent] — automated review by Claude Code, requested by @sbalabanov.

Comment thread platform/errs/git/git.go Outdated
Comment thread platform/errs/git/git.go Outdated
Comment thread platform/errs/git/git.go
Comment thread platform/git/exec/command_error.go Outdated
Comment thread platform/git/exec/command_error.go Outdated
Comment thread runway/extension/merger/git/git_merger.go Outdated
Comment thread platform/errs/README.md
Comment thread platform/errs/git/git_test.go Outdated
Comment thread service/runway/server/main_test.go
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-004-git-classifier branch from 623ae95 to f3a437d Compare September 8, 2026 18:27
@sbalabanov
sbalabanov marked this pull request as ready for review September 8, 2026 18:27
@sbalabanov
sbalabanov requested review from a team and behinddwalls as code owners September 8, 2026 18:27
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-004-git-classifier branch from f3a437d to e70bb3a Compare September 8, 2026 18:36
@sbalabanov
sbalabanov marked this pull request as draft September 8, 2026 18:41
@sbalabanov
sbalabanov marked this pull request as ready for review September 8, 2026 19:51

@behinddwalls behinddwalls left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline comments on the remaining high-signal issues after the allowlist revision:

  1. rpc failed / remote end hung up unexpectedly / unexpected disconnect while reading sideband packet over-match permanent git-http 4xx.
  2. classifyMergeFailure's doc comment still says those non-conflict failures should be retried.

Architecture otherwise looks right (CommandError as carrier, fail-closed pair rule, cancellation left to generic).


Generated by Cursor. Posted on behalf of @preetam_UBER.

Comment thread platform/errs/git/git.go Outdated
Comment thread runway/extension/merger/git/git_merger.go Outdated
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-004-git-classifier branch from e70bb3a to a72710f Compare September 10, 2026 17:12
Summary:
Intent:
- Prevent temporary Git remote and checkout failures from being dead-lettered on their first delivery.
- Keep every other Git failure fast-failing, so a deterministic error is not replayed through the retry budget.

Changes:
- Add structured Git command errors and a Git classifier that opts a failure into retryability only on a known diagnostic/operation pair.
- Surface a cancelled context at the Git execution boundary, so cancellation reaches the generic classifier instead of dying as an opaque "signal: killed".
- Derive the Git subcommand through one guarded helper and wire the classifier into the Runway primary consumer.

Reproduction:
- A merge delivery runs `git fetch origin` or `git push origin ...` while the remote temporarily resets the connection, producing a wrapped `*exec.ExitError`.
- Previously Runway registered only generic and MySQL classifiers, so the error stayed non-retryable and the consumer rejected it to the DLQ after one attempt.
- With this change the structured Git error is classified as a retryable dependency failure, so the consumer nacks it for redelivery.

Retryability is an allowlist. Git has no typed status to read, so the
classifier pairs the subcommand with the diagnostic: a transport fragment
counts only against a command that talks to the remote, and a lock fragment
counts against any command that writes to the checkout. Only a recognised pair
is retryable. Every other Git failure, including a diagnostic the package has
never seen, is a permanent infrastructure failure attributed to the remote or
to this service, so a deleted target branch, an empty squash commit or a
rejected push still dead-letters on the first delivery rather than re-running
the fetch, reset and cherry-picks behind it on every attempt.

`os/exec` reports a context-killed child as a bare `*exec.ExitError` reading
"signal: killed", with neither `context.Canceled` nor
`context.DeadlineExceeded` anywhere in the chain. `gitexec.CommandFailure`
reads `ctx.Err()` and surfaces it, which is what lets the generic classifier
recognise a cancelled merge rather than seeing an unexplained Git failure.

---

<sub>Generated by the 🪄 [pr-create](https://sg.uberinternal.com/code.uber.internal/uber-code/devexp-agent-marketplace/-/blob/claude-code/plugins/dev/uber-dev/skills/pr-create/SKILL.md) skill in devexp-agent-marketplace</sub>
@sbalabanov
sbalabanov force-pushed the sergeyb/fix-iss-004-git-classifier branch from a72710f to 250eabc Compare September 10, 2026 17:28
@sbalabanov
sbalabanov added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit 39d91a4 Sep 10, 2026
14 of 15 checks passed
@behinddwalls
behinddwalls deleted the sergeyb/fix-iss-004-git-classifier branch September 10, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants