123 · T108 — the Kubernetes node runner and the two retry budgets

[athvin]

Ticket: T108 — docs/implementation/123-T108-kubernetes-node-runner.md

Summary

K8sNodeRunner implements NodeRunner and drives the whole remote attempt —
record → submit → await → read shard → replay → TerminalState — through the
unchanged driver. Submission is a bare Pod, write-ahead-logged before creation via a
SubmissionLog that is the run's sequence authority, with the new attempt-submitted
record landing in dagr-artifact for the @1.3 kind. The two retry budgets are kept
distinct: pre-start failures spend --dagr.pod-launch-retries and consume no
user-visible attempt, while post-start failures spend NodePolicy::retries with real
backoff. --dagr.executor=k8s still refuses at bootstrap — lifting that is T112's.

Tests-first

Confirmed — failing tests committed first in 94ee107.

Definition of done

• K8sNodeRunner implements NodeRunner and drives record → submit → await → read shard → replay → TerminalState, reporting through the existing hooks; no driver change — crates/cli/src/driver.rs and all of crates/core/ are absent from this diff, and the seam is proven by a_mixed_pipeline_runs_end_to_end_through_the_unchanged_driver, which drives a real drive() with one placed and one local runner.
• The attempt-submitted writer and record type land in dagr-artifact for the @1.3 kind, with a fixture-corpus artifact parsed in CI.
• The record is durably flushed before pod creation (ordering asserted), and carries the identity triple, ordered { uri, content_hash } inputs, both fingerprints, tool version, image digest, and intended pod name; observed name, UID, and host are recorded after creation.
• inputs is an empty array for a consume-nothing source and has exactly the declared arity in positional order; an arity mismatch fails before launching.
• No credential-bearing reference is ever recorded.
• A local run emits no attempt-submitted record and stays byte-identical; the fold is unperturbed by streams that contain them.
• Replayed streams have gapless seq and fold cleanly; the orchestrator is the single writer.
• Pre-start failures use --dagr.pod-launch-retries and consume no user-visible attempt; post-start failures consume NodePolicy::retries with real backoff.
• A pre-start failure is detected as Pending + a known-fatal waiting.reason under the runner's own 60s bound — not by awaiting a terminal phase (T101 proved none arrives for an unpullable image).
• Terminal state comes from pod status; OOMKilled/Evicted/Unschedulable/ImagePullBackOff are diagnostics; the terminal taxonomy still has nine members.
• Cluster-side retry is refused at bootstrap with the duplicate-execution rationale.
• A hung pod is bounded by the node timeout and deleted; a terminal pod with no readable shard, or a fingerprint-mismatched shard, is a classified failure.
• Submission is idempotent on (run_id, node, attempt); cancellation deletes pods inside the shutdown budget.
• A placed node with non-Payload payloads is a compile error (trybuild).
• --dagr.executor=local remains byte-identical to a pre-M10 run.
• scripts/check-metastore-acceptance-boundary.sh passes.
• Tests pass on ubuntu-latest and macos-latest — pending, confirmed by this PR's CI.
• CI is green on the ticket branch — pending. The local gate is GATE=PASS (35 PASS, 1 SKIP: coverage-matrix, already run as criteria-matrix, 0 FAIL), verified by the orchestrator on the committed tree, but CI on the PR is the authoritative verdict.

Open questions resolved

The ticket's two, plus eight decisions the implementation forced, are recorded in the
ticket's ## Open questions — resolved section (docs/tasks.md enumerates M0–M4 only
and carries no T108 entry):

1. Bare Pod or Job with backoffLimit=0? → a bare Pod.
2. How a pre-start failure is detected → not by the shard and not by a terminal phase, but by two pod-status surfaces under the runner's own bound.
3. Where the write-ahead record becomes durable, given AttemptEventSink buffers → a SubmissionLog that is the run's sequence authority.
4. Intent and reality in one record or two? → two records; the schema already allows it.
5. What re-arms the observer after a launch retry → object identity, not the attempt key.
6. Does --dagr.executor=k8s stop refusing? → no; that is T112's, not this ticket's.
7. Where remote eligibility is enforced → at the placement-taking registrars, as a Payload bound.
8. The pre-start bound, and is it a knob? → 60s, and no.
9. How a remote attempt honours NodePolicy::timeout → the cancellable half of C14, because a remote wait really is cancellable.
10. Does PodLifecycle get a real backend here or in T112? → here.

Deviations

None.

Notes

The runtime-placement boundary established by T107 holds: dagr-k8s still carries no
tokio (or any async-runtime) edge, and dagr-core remains zero-runtime-dependency. No
scripts/check-*.sh allowlist was widened and no assertion weakened — the only script
changes add assertions.

One rustdoc lint (redundant-explicit-links at crates/k8s/src/lib.rs:19) was caught
by the local gate and fixed in 82b1a8e before this PR opened. The bare
[PodLifecycle] shorthand is safe in every feature combination because
PodLifecycle is re-exported unconditionally at the crate root and mod api carries
no #[cfg] — unlike the genuinely gated client module, which is deliberately left
unlinked.