test(hu): verify the release against its own documentation - #313
Merged
Merged
Conversation
|
YuanYuYuan
force-pushed
the
feat/hu-release-pipeline
branch
from
August 21, 2026 08:18
feb9a5c to
d4f172d
Compare
YuanYuYuan
force-pushed
the
test/hu-release-coverage
branch
from
August 21, 2026 08:18
52e307f to
156f373
Compare
YuanYuYuan
force-pushed
the
feat/hu-release-pipeline
branch
from
August 21, 2026 19:34
d4f172d to
099c423
Compare
Every defect the release work uncovered lived in a path no test executed, and none would have failed a build. This is the coverage that would have caught them. Runs every documented hu command against an installed binary in a scratch HOME with the plugin path unset, so the suite measures what a download gives a reader rather than what a build tree gives a developer. Encodes the tag-to-asset-name rule as assertions, because the release workflow never runs on a pull request. Adds the installer's own refusal tests. Also repairs two gaps in existing CI: hiroz-union's tests never ran, because --lib matches nothing on a binary-only crate, and the web-plugins feature was never compiled. Both run now.
The release workflow gained this step in the distribution branch, which did not carry the script it calls. That branch would have failed every `v*` tag: build, promote, fail here, withdraw the release, skip crates.io. The step was removed there and belongs here, with the script. Also restores the RELEASING.md claim about `scripts/test-release-version-semantics.sh`. The distribution branch promised a script it did not contain, so the sentence was corrected to say nothing enforced the invariant. This branch adds the script and runs it from ci.yml, so the original claim is true again.
YuanYuYuan
force-pushed
the
test/hu-release-coverage
branch
from
August 22, 2026 03:53
156f373 to
f7a4bbf
Compare
There was a problem hiding this comment.
Pull request overview
Adds release-focused validation to ensure installed hu artifacts reproduce documented behavior and release version semantics.
Changes:
- Adds documentation, installer, and release-semantics test suites.
- Expands CI coverage across plugins, platforms, and features.
- Updates release and user documentation.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/test-release-workflow.nu |
Derives smoke-test tags from the crate version. |
scripts/test-release-version-semantics.sh |
Tests tag, asset, and workflow version rules. |
scripts/test-pure-rust.nu |
Runs binary tests and checks web-plugin targets. |
scripts/test-install-hu.sh |
Tests installer refusal and lifecycle paths. |
scripts/test-hu-docs-repro.nu |
Executes documented commands against an installed release. |
scripts/ci/hu-tests.sh |
Tests successful plugin installation and removal. |
RELEASING.md |
Documents workspace-version enforcement. |
flake.nix |
Adds an aarch64 cross-compilation shell. |
docs/user-guide/examples.md |
Documents type-description behavior in examples. |
docs/tools/why-hu.md |
Classifies a non-reproducible example. |
docs/tools/hu.md |
Documents released artifacts and reproducible commands. |
docs/tools/hu-vs-ros2cli.md |
Adds reproduction classifications and schema guidance. |
docs/getting-started/quick-start.md |
Explains type-description service usage. |
.github/workflows/test.yml |
Isolates Rust caches by distribution. |
.github/workflows/release.yml |
Verifies published releases against documentation. |
.github/workflows/ci.yml |
Adds release, installer, platform, and docs checks. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
`install-readme` interpolated `($target)`, and it takes no such parameter. Nushell raised "Variable not found", so `build-hu-release.nu` failed outright on the macOS packaging leg and on the aarch64 cross-compile leg. The line is a hint for a reader checking one file, so it now names no variable at all. That defect entered with the release-pipeline slice, which had no job that runs the script -- the docs-reproduction suite this branch adds is what executes it, and it failed on the first run. `test-release-version-semantics.sh` pinned the withdraw job to `result == 'failure'`. Review found that wrong: GitHub reports a cancelled job as `cancelled` and a skipped one as `skipped`, so equality on 'failure' leaves an unverified release public. The workflow now tests `!= 'success'`, and this asserts that property in both directions.
The docs-reproduction extractor classified a line by `starts-with "hu "`.
Three documented commands do not begin with that token:
HU_WEB_BIND=0.0.0.0 hu web
rate=$(hu meter hz /camera/image_raw --duration 5 --json | jq ...)
The extractor dropped them, so their `repro:` directives never reached
the report either. The suite covered less than it claimed, which is the
failure this harness exists to remove -- and the comment directly above
the test asserted that nothing is ever dropped.
`strip-shell-wrappers` now removes leading environment assignments, a
command substitution, and a pipeline tail before the `hu` test runs. An
unrecognised form is returned unchanged, so it still reaches the test and
is reported rather than dropped. All three lines already carry
`repro: skip`, so they are now reported as skips.
`test-release-version-semantics.sh` derived the version with `grep -m1`,
which reads only the first derivation block. `release.yml` has one per
release producer, `build-binaries` and `build-hu-plugins`, and they are
independent copies. The second could regress to `HU_CORE=$V` unchecked,
publishing a binary and a plugin set that disagree about the version.
It now evaluates every block and asserts the count is at least one, so a
producer added without a check fails rather than passing silently.
Both directions measured. As committed: 56 passed, 0 failed. Regressing
only the second derivation block: 55 passed, 1 failed, naming
`[block 2/2]` -- the case the previous form could not see.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last of the four slices that replaced #301. This targets
main, because #310 and #312 have merged. #315 is the remaining sibling, and it is independent of this one.It tests what #312 builds, and it carries two things #312 could not:
v*tag. The step was removed there and lands here, with the script.scripts/test-release-version-semantics.sh, whichRELEASING.mddescribes andci.ymlruns on every pull request.What this does
Every defect the release work uncovered lived in a path no test executed, and none would have failed a build. This is the coverage that would have caught them.
scripts/test-hu-docs-repro.nuhucommand against an installed binary in a scratchHOMEwith the plugin path unset — what a download gives a reader, not what a build tree gives a developer. It reads through the shell forms the docs use, so an environment prefix or a command substitution is reported rather than skipped in silencescripts/test-release-version-semantics.shrelease.ymlnever runs on a pull request. It evaluates every derivation block, so the plugin leg cannot regress behind a green binary legscripts/test-install-hu.shSHA256SUMS, no credentialWhat fails without this
Two pre-existing CI gaps, both of which let a real defect ship:
hiroz-union's tests never ran —--libmatches nothing on a binary-only crateweb-pluginsfeature was never compiledhu webpanicked on startup against axum 0.8, and nothing caught itThe docs-reproduction suite was observed RED before the release work landed, and mutation-checked afterwards: truncating a plugin in the install directory turns it red again.
Breaking Changes
None. New scripts, new CI jobs, and two existing jobs widened to compile and run what they always claimed to.