The release publishes on a fixed sleep, and v0.15.0 half-published when crates.io was slow

[vladimirrott]

The v0.15.0 release failed halfway on the first attempt, published nothing wrong,
and needed a manual re-run. The cause is a race with the crates.io index, and it
will recur.

What happened

sysknife-daemon v0.15.0 uploaded successfully. Cargo then waited for it to
appear in the index and gave up:

15:27:29  Uploaded sysknife-daemon v0.15.0 to registry `crates-io`
15:27:29  note: waiting for sysknife-daemon v0.15.0 to be available at registry `crates-io`
15:28:29  warning: timed out waiting for sysknife-daemon v0.15.0 to be available in registry `crates-io`
          note: the registry may have a backlog that is delaying making the crate available.

Thirty seconds later the next crate in the loop tried to package against an
index that still did not have it:

15:29:00  error: failed to prepare local package for uploading
Caused by:
  failed to select a version for the requirement `sysknife-daemon = "^0.15.0"`
  candidate versions found which didn't match: 0.14.0, 0.13.1, 0.13.0, ...
  required by package `sysknife-cli v0.15.0`

Five of six crates were on crates.io, sysknife-cli was not, and
sysknife-setup was already on npm. So the ecosystem showed a half-published
release: npm i sysknife-setup would fetch 0.15.0 and cargo install sysknife-cli would fetch 0.14.0.

Re-running the same job on the same commit fixed it, because the step is
idempotent and skips what already exists. That part of the design worked.

Why the current pacing cannot be right

.github/workflows/release.yml publishes in dependency order and sleeps a fixed
30 seconds between crates. Cargo adds its own 60-second wait. Ninety seconds was
not enough here, and no fixed number can be: the delay is crates.io's backlog,
which is not ours to predict. The failure lands on whichever crate happens to be
slow that day, which is why this has not bitten before.

The blast radius is the part that matters. A partial publish is visible to
everyone immediately, and the two registries disagree until a human notices and
re-runs the job by hand.

Scope

• Replace the fixed sleep 30 with a poll: after each upload, query
  https://crates.io/api/v1/crates/<crate>/<version> until it returns 200, with
  a timeout and a clear failure message naming the crate that never appeared.
  The step already makes exactly that call for its skip check, so the code is
  there.
• Consider cargo publish --no-verify for the dependents, or publishing with a
  workspace-aware tool that does not re-resolve each crate against the public
  index. The re-resolution is what actually fails; the archive was already fine.
• Say in docs/release.md what a partial publish looks like and that the fix is
  re-running the job rather than moving the tag. That is written down in the
  general case; the specific symptom is not, and the error message points at a
  version mismatch that does not exist.

Difficulty

Easy, and the polling loop is the whole job. Worth doing before the next release
rather than after, because the next one has the same coin flip.

Tests first

A release workflow is awkward to test, which is the reason this shape survived.
Make the polling a shell function in a script the suite can call, rather than
inline run: steps, and then both directions are cheap.

Break what the guard protects. Point the poll at a stub that answers 404
three times and then 200, and require it to return success without a fixed
sleep. Then point it at a stub that answers 404 forever and require a non-zero
exit whose message names the crate and the version that never appeared. That
second case is the one that turns the v0.15.0 failure from a cryptic
failed to select a version into a sentence an operator can act on.

Break the guard's own input. Assert the loop actually polled. A poll that
returns success because its URL was malformed, or because the stub was never
reached, passes every test above while doing nothing: count the requests and
require at least one. Pin the timeout too, so a future edit cannot turn a bounded
wait into an unbounded one without a test going red.

Then assert the fixed sleep is gone, so it cannot come back:

$ grep -n 'sleep' .github/workflows/release.yml
210:        # sleep 30 lets crates.io index each crate before its dependents publish.
225:              sleep 30

Getting started

CONTRIBUTING.md
has the build and test commands. Everything here is one workflow file plus a
shell script and its test; no Rust, no VM, no daemon, and no crates.io
credentials. You never need to publish anything to work on this: the skip check
at .github/workflows/release.yml:221 already makes the exact crates.io API
call the poll needs, so the request shape is in the file.

Do not test this by cutting a release. No CLA and no copyright waiver. The
project is MIT.

[Humerous]

I'd like to work on this.

I can take the polling/release-workflow side, and I may have another contributor work with me on the failure-path validation and release documentation so the final PR covers both implementation and recovery behaviour properly.

Is a collaborative PR acceptable for this issue?

[vladimirrott]

@mikevillari offering you this one, on the strength of #398 and #399.

The release workflow publishes six crates in dependency order and waits between
them with a fixed sleep 30:

$ git rev-parse --short=8 HEAD
61b3a878
$ sed -n '219,227p' .github/workflows/release.yml
            if curl --fail --silent --retry 3 --retry-connrefused \
              --user-agent 'sysknife-release/1.0 (https://github.com/lacs-project/sysknife)' \
              "https://crates.io/api/v1/crates/${crate}/${VERSION}" >/dev/null; then
              echo "${crate} ${VERSION} already exists; skipping"
            else
              cargo publish -p "$crate" --locked --token "$CARGO_REGISTRY_TOKEN"
              sleep 30
            fi
          done

On v0.15.0 that was not enough. sysknife-cli tried to resolve
sysknife-daemon = "^0.15.0" against an index that did not have it yet, the job
stopped with five crates on crates.io and one missing, and sysknife-setup was
already on npm. npm i sysknife-setup fetched 0.15.0 while cargo install sysknife-cli fetched 0.14.0 until somebody re-ran the job by hand. The full
error output is in the issue above.

Why you. #399 was a step reporting success over a set it never counted, and #398
was a release script accepting an argument it never validated. This is the same
defect one layer out: sleep 30 asserts a wait and never asserts the thing it
waits for. No fixed number can be right, because the delay belongs to crates.io's
backlog rather than to us.

The call you need is already in the loop, six lines above the sleep, doing the
skip check. Turning it into a bounded poll is most of the work.

One trap worth naming before you start, and it is the #399 distinction again.
That if has two branches, so "crates.io says no such version" and "I could not
reach crates.io" both land in the same one today. A poll built on it inherits
that. It will burn its whole timeout during an outage and then fail with a
message about indexing, which sends the next person to the wrong place. Separate
them, and name the crate in the failure either way.

Neither of us can run the workflow end to end without cutting a release, so the
testable surface is the shell: a poll with the HTTP call injectable, and a test
that proves it gives up rather than looping forever. That is the assertion I
would want to see go red first.

Say the word and I will label and assign it. No is a complete answer, and there
is nothing attached to it either way.

[vladimirrott]

Two corrections on this thread and both are mine.

@Humerous asked for this issue on 10 September and I never answered. Nothing on
the issue recorded the request either: no label, no assignee. So when I came back
to it today it read as unclaimed, and I offered it to somebody else on this same
thread. That is a double booking, and the reason it happened is that I checked
the labels on this issue and not the conversation.

@mikevillari I am withdrawing the offer above. Humerous asked first and it is
theirs. Nothing about the fit changed and I am sorry for the whiplash. Two
issues went up today and #431 is nearer to your #399 than this one ever was: a
contract test that reads its pass condition out of a fixed /tmp path, so a run
that could not write its output reports as the refusal it was supposed to
observe. I have put the detail on that thread rather than here.

@Humerous the issue is labelled and assigned to you, and yes to a collaborative
PR. One mechanical constraint is worth knowing before you split the work.

The merge gate refuses a pull request that closes a claimed issue when the PR
author is not among the issue's assignees. It is there because a contributor was
promised an issue in August and a different person's PR closed it ten hours
later. So whoever opens the PR has to be assigned here. Two straightforward ways
through it:

• You open the PR and your collaborator lands in Co-authored-by: trailers.
  Nothing else changes.
• Or they comment here and I assign you both, and then either of you can open it.

Pick whichever suits how you want to divide it. Tell me which and I will set it
up before you have anything to push.

On the split itself: the polling side and the failure-path side are genuinely
separable, and the failure path is the half I care more about. A poll that gives
up has to say which crate never appeared and has to distinguish "crates.io says
no such version" from "I could not reach crates.io". Those land in the same
branch of the current if, and a poll built on it inherits that.

Two smaller PRs are welcome if that fits better than one. I would rather review
the poll on its own than wait for the documentation to be ready.

[vladimirrott]

@Humerous added Tests first and Getting started to the body. Nothing about the defect changed; the issue was missing both sections and you should not have to infer how to prove a polling loop works.

The short version: make the poll a shell function the suite can call, drive it from a stub that answers 404 a few times then 200, and require the timeout case to name the crate that never appeared. Then assert the loop really polled, because a poll that never reaches its stub passes every other test while doing nothing.

Rechecked the two anchors at b2c823e6 and both still land:

$ grep -n 'sleep' .github/workflows/release.yml
210:        # sleep 30 lets crates.io index each crate before its dependents publish.
225:              sleep 30
$ sed -n '221p' .github/workflows/release.yml
              "https://crates.io/api/v1/crates/${crate}/${VERSION}" >/dev/null; then

You never need to publish anything to work on this, and please do not test it by cutting a release.