Skip to content

docs(handoff): the rank-7 ownership probe ran — fresh PVC and fresh hostPath are indistinguishable, and the failure did NOT reproduce - #1182

Open
ZacxDev wants to merge 2 commits into
mainfrom
docs/handoff-ci-speedup-rank7
Open

docs(handoff): the rank-7 ownership probe ran — fresh PVC and fresh hostPath are indistinguishable, and the failure did NOT reproduce#1182
ZacxDev wants to merge 2 commits into
mainfrom
docs/handoff-ci-speedup-rank7

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Aug 31, 2026

Copy link
Copy Markdown
Member

Continues ci-speedup rank 7 (retry the devrc-ci node unpin). Two things landed: the perf baseline was re-taken, and the blocking ownership question was probed. Docs only.

1. The baseline was re-taken at requests.cpu: 2

claude/skills/tekton/reference/pipelines.md states the unpin's quoted wins are not re-derivable — pruned runs, measured at cpu: 4 when live is 2. Re-measured, n=31 gate TaskRuns:

metric value
gate pod-start latency p50 / p90 / max 101s / 748s / 1043s
wall clock median / p90 23.4m / 34.2m (45m budget)
nodes used by gate pods 1 of 4 (talos-xr6-r7p, 24/24)

The upside survives the re-take — ~12.5 min at p90 is pure scheduling wait from pinning four nodes' demand onto one.

PipelineRun creation→startTime is median 0.0s and is a decoy: it reads as "there is no queue". The wait is entirely TaskRun→pod.

2. The ownership probe — a NEGATIVE result

A hand-applied scratch Pod (not GitOps: no Flux, no eventlistener.yaml edit), ci-bulk priority so it could preempt nothing, and it never mounted the live nix-store-cache. Two arms — fresh local-path PVC vs fresh hostPath — both seeded with the gate's own cp -a /nix/. <vol>/.

Every dimension came back identical: 600 root:root big-lock, 755 throughout, build-users-group=nixbld, sandbox=false, /build absent, builds ok, positive control ok.

🔴 The failure did not reproduce on either arm. This extends the reference's already-REFUTED candidate 1 from "the volume ROOT's mode" to the whole tree — neither volume kind nor fresh-seed ownership is the variable, and 7839ef54's stated cause is less supported than before.

Two blind spots, stated rather than folded in

  • The unprivileged-client test never rannixos/nix:2.24.15 ships no su/setpriv/runuser/doas, so NIXBLD_CLIENT=NOT_MEASURED on both arms.
  • The builder's uid came back empty — the build's /bin/sh is static busybox with no id.

v1 of the probe reported NIXBLD_CLIENT=FAILED, which looked like a finding and was su: command not found. Caught only by reading stderr rather than the label; v2 carries an explicit positive control and reports NOT_MEASURED instead of a verdict.

3. A correction to this doc's own earlier claim

It said the privileged namespace label "grants nothing". Half wrong: it grants nothing to the current gate, but hostPath admission depends on it--dry-run=server admitted the probe's hostPath only because the namespace enforces privileged. Ranked items 5 and 7 are therefore coupled; reverting the label would block a hostPath retry at admission.

Cleanup

Both probe pods, the cleanup pod and the probe PVC deleted; nix-store-cache verified Bound 30Gi afterwards. One empty dir remains at /var/lib/mnt/disk-1/nixlock-probe-hostpath on talos-xr6-r7p — recorded rather than left silent; removing it needs a pod mounting a shared disk's parent.

devrc-ci was never touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant