Skip to content

Adopt cargo-nextest as the default test runner (#463) - #473

Merged
leynos merged 8 commits into
mainfrom
issue-463-adopt-cargo-nextest-as-the-default-test-runner
Jul 30, 2026
Merged

Adopt cargo-nextest as the default test runner (#463)#473
leynos merged 8 commits into
mainfrom
issue-463-adopt-cargo-nextest-as-the-default-test-runner

Conversation

@leynos

@leynos leynos commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Closes #463.

make test now runs the whole non-doctest Rust suite through cargo-nextest and gates doctests in a companion pass. make test remains the single canonical entry point for both local development and CI.

What changed

Runner

  • make test is now a composite of two targets, mirroring the existing lint: lint-clippy lint-whitaker pattern:
    • make test-nextestRUSTFLAGS="-D warnings" cargo nextest run --all-targets --all-features
    • make doctestRUSTFLAGS="-D warnings" cargo test --doc --all-features
  • nextest cannot execute doctests. Worth noting for reviewers: the previous cargo test --all-targets did not run them either, so documentation examples were not gated at all before this change. They are now, and all 44 pass.
  • Cargo spells build parallelism -j; nextest reserves -j for test concurrency and spells build parallelism --build-jobs. The Makefile therefore keeps BUILD_JOBS and NEXTEST_BUILD_JOBS as separate variables rather than silently reinterpreting one as the other.

Configuration

.config/nextest.toml is deliberately small:

  • One narrow serial-env test group (max-threads = 1) scoped by filterset to exactly three binaries — manifest_env_tests, ninja_env_tests, env_path_tests — which mutate PATH, NINJA_ENV, and ad hoc NETSUKE_* variables. Everything else stays fully parallel.
  • No blanket retries. A flaky test is a defect to diagnose, not to paper over.
  • A conservative slow-timeout (warn at 60s, terminate after five periods) to surface hangs without failing the legitimately slow documentation end-to-end suites, which shell out to real Ninja.

One thing worth flagging honestly, and which the config comment and the developers' guide both state rather than obscure: because nextest runs every test in its own process, the serial-env group is not load-bearing for the tests that exist today — process isolation already prevents env and cwd interference. It is retained because it states the serialization contract once so both runners agree, and so the constraint is not silently lost if a future test in those binaries reaches for genuinely shared state such as a fixed on-disk path. The #[serial] markers and the serial_test dependency are untouched, since the coverage workflow still drives an in-process runner where they are load-bearing.

CI

  • NEXTEST_VERSION (0.9.133) is a job-level env var in build-test, the single source of truth for the pin.
  • taiki-e/install-action runs unconditionally after Setup Rust and before Test, because every matrix leg runs make test.
  • Coverage and mutation testing are unchanged and deliberately so: coverage-main.yml and the coverage step in ci.yml delegate to the generate-coverage shared action (its own cargo llvm-cov run, never make test), and mutation-testing.yml keeps --all-features parity with the new make test.

Contract tests

  • tests/workflow_ci.rs gains coverage of the nextest install wiring: NEXTEST_VERSION is an exact major.minor.patch pin, the installer resolves its tool spec from it, the step carries no if: guard, and it is ordered before the Test step, which still runs make test.
  • tests/makefile_test_target.rs is new and pins the Makefile contract itself: the test composite, both recipes, warnings denied on each, and the serial-env group in .config/nextest.toml.

One deliberate deviation from the plan, called out for review: the plan asked the CI contract test to assert the literal taiki-e/install-action SHA. It asserts the shape instead — a 40-character lowercase-hex commit SHA. Asserting the literal value would make every Dependabot bump fail the test until a human edited the constant, which is exactly the lockstep problem "Workflow pins and Dependabot" in docs/developers-guide.md already rules out for shared-actions callers. The pin is still enforced; only the manual-edit treadmill is avoided. Happy to switch to a literal assertion if you would rather have the tighter pin.

Documentation

  • AGENTS.md (and CRUSH.md, which is a symlink to it, not a byte-identical copy) claimed make test was cargo test --workspace. Corrected, with the runner, its config file, and the local install command named in the Testing section.
  • docs/developers-guide.md gains a ## Test execution section covering both passes, the config's scope, the serial-env group, the retry and timeout policy, how it relates to the retained #[serial] markers and the EnvLock/CwdGuard utilities, and why the coverage and mutation runners are untouched. Local install guidance is added under ## Quality gates.
  • Behavioural-testing guidance now describes nextest's process-per-test model as reinforcing the existing per-scenario isolation policy, and cargo test --test bdd_tests becomes cargo nextest run --test bdd_tests.
  • The snapshot guide points at the nextest-aware cargo insta test --test-runner nextest flow.

Capability-scoped filesystem access

Review feedback: the contract test read repository files through std::fs/std::path, bypassing the repository's capability-oriented filesystem policy. It now uses a cap_std::fs_utf8::Dir opened at the repository root, with camino paths, so it cannot reach outside the checkout — and consequently needs no no_std_fs_operations exclusion at all.

Whitaker has since gained an excluded_paths option that scopes an exemption to a module and its descendants, and its no_std_fs_operations lint now honours in-source #[allow]. The user's guide is re-imported and dylint.toml rewritten around it:

  • The netsuke application crate was exempt wholesale. Only netsuke::stdlib::which::lookup (PATH probing and cross-directory symlink canonicalization, which cap_std cannot express) and netsuke::runner::process::file_io (temporary-file sync) actually need ambient access. The rest of the application crate is now under the capability policy.
  • The behavioural step definitions, CLI integration tests, and the shared workflow-reading helper are scoped the same way, retiring the crate-level entries for bdd_tests, cli_tests, workflow_ci, and workflow_release.
  • dependabot_config_tests and logging_stderr_tests no longer trip the lint at all; their entries are removed as stale.
  • Crate-level entries remain only where the ambient I/O lives in the crate root, where a path entry would be no narrower.

The developers' guide also had a stale claim that the lint ignores in-source allow/expect attributes; corrected.

Verification

All gates green:

  • make check-fmt — pass
  • make lint (Clippy + Whitaker Dylint) — pass
  • make test — 1153 nextest tests passed, 0 skipped; 44 doctests passed, 20 ignored
  • make markdownlint (including spelling) — pass
  • make nixie — pass
  • make test-workflow-contracts — 6 passed

coderabbit review --agent returned 0 findings.

References

🤖 Generated with Claude Code

Summary by Sourcery

Adopt cargo-nextest as the default non-doctest test runner while keeping doctests in a separate pass, and formalize this contract across CI, Makefile targets, configuration, tests, and documentation.

New Features:

  • Introduce a cargo-nextest configuration file to control timeouts and a narrow serial-env test group for environment-mutating integration binaries.
  • Add dedicated Makefile targets for running non-doctest tests via cargo-nextest and doctests via cargo test, composed under the canonical make test entry point.

Enhancements:

  • Extend CI to install a pinned version of cargo-nextest and ensure the Test step continues to run make test with nextest installed beforehand.
  • Add behavioural and unit tests to validate the CI workflow wiring for cargo-nextest and to pin the Makefile test contract, including nextest usage and serial-env coverage.
  • Clarify and expand developer and agent documentation around test execution, nextest installation and configuration, behavioural tests under nextest, and snapshot-testing workflows to align with the new runner.

CI:

  • Update the main CI workflow to set a NEXTEST_VERSION environment variable and install cargo-nextest via a pinned taiki-e/install-action step before running tests.

Documentation:

  • Document the new nextest-based test execution model, including separate doctest handling, configuration scope, and isolation contracts, and update existing guides and agent docs to reference cargo-nextest commands and usage.

Tests:

  • Add workflow contract tests to assert that CI installs cargo-nextest from the pinned version and that the Test step runs make test with correct step ordering, and introduce Makefile contract tests to enforce the structure and semantics of the make test, test-nextest, and doctest targets.

Chores:

  • Exclude the new Makefile contract test binary from the Dylint configuration to keep linting focused on relevant crates.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Summary

  • Adopted cargo-nextest for non-doctest Rust tests while retaining a separate, failure-reporting cargo test --doc pass through make test.
  • Added checked-in nextest configuration with targeted environment-test serialization, conservative timeouts, and no blanket retries.
  • Pinned nextest installation in CI and documented local setup and test workflows.
  • Added Makefile and CI workflow contract tests.
  • Narrowed no_std_fs_operations filesystem exclusions to capability-scoped paths and updated related documentation.
  • Preserved existing coverage and mutation-testing runners.

Walkthrough

Adopt cargo-nextest for non-doctest tests, retain a separate doctest pass, pin CI installation, add contract coverage, document the workflow, and narrow no_std_fs_operations exemptions from crate-level to path-level configuration.

Changes

Cargo-nextest test adoption

Layer / File(s) Summary
Split test execution and nextest configuration
.config/nextest.toml, Makefile
Run nextest and doctests separately, serialize selected environment-mutating binaries, and apply timeout and retry policies.
Pinned CI provisioning and validation
.github/workflows/ci.yml, tests/workflow_ci.rs
Pin and install nextest before make test; validate version, action pinning, ordering, and command usage.
Makefile and nextest contract tests
tests/makefile_test_target.rs
Parse Makefile rules and verify the two-pass workflow and serial-env configuration.
Contributor and test-runner guidance
AGENTS.md, docs/developers-guide.md, docs/snapshot-testing-in-netsuke-using-insta.md
Document nextest execution, doctests, isolation, installation, snapshot testing, and runner scope.

Whitaker filesystem-lint exemptions

Layer / File(s) Summary
Path-scoped lint configuration and guidance
dylint.toml, docs/whitaker-users-guide.md
Add excluded_paths, remove broader crate exclusions, and document path matching and local exemptions.

Sequence Diagram(s)

sequenceDiagram
  participant Developer
  participant Makefile
  participant Nextest
  participant CargoTest
  Developer->>Makefile: make test
  Makefile->>Nextest: run non-doctest tests
  Nextest->>Nextest: serialize serial-env binaries
  Makefile->>CargoTest: run doctests with --doc
Loading

Possibly related issues

  • leynos/dear-diary#63 — Covers the same cargo-nextest adoption, split doctest workflow, CI integration, and configuration.
  • leynos/netsuke#467 — Covers the same two-phase make test workflow with an explicit doctest pass.
  • leynos/netsuke#462 — Covers the related narrowing of no_std_fs_operations exemptions.

Possibly related PRs

  • leynos/theoremc#44 — Also splits nextest execution from a separate doctest pass.
  • leynos/netsuke#472 — Also modifies Makefile test invocation and warning-related environment handling.

Suggested labels: Issue

Suggested reviewers: codescene-access

Poem

Nextest runs, the doctests follow,
Pinned tools keep CI bright;
Serial env tests move in line,
Lint paths narrow just right.
Make test guides the way.

🚥 Pre-merge checks | ✅ 19 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The PR also changes Whitaker and capability-scoped filesystem policy, which is not required by issue #463. Move the lint-policy and filesystem-scope edits into a separate PR, or extend the linked issue to cover them explicitly.
✅ Passed checks (19 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the main change and includes the linked issue reference.
Description check ✅ Passed The description tracks the nextest runner swap and doctest split in the changeset.
Linked Issues check ✅ Passed The PR covers the nextest migration, doctest split, CI pinning, contract tests, docs, and the required narrow serial group.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Testing (Overall) ✅ Passed Real-file contract tests assert the Makefile nextest/doctest split, pinned CI install, and serial-env TOML wiring, so the obvious regressions would fail.
User-Facing Documentation ✅ Passed The PR only changes test/CI/developer workflow; docs/users-guide.md is untouched and contains no nextest/test-runner content, so no user-facing guide update is required.
Developer Documentation ✅ Passed Confirm docs/developers-guide.md documents nextest, doctests, build-job pinning, and lint policy; no ADR/execplan or locale gap was left.
Module-Level Documentation ✅ Passed PASS: The added Rust modules and shared helper all begin with //! docs, and a repo scan found no Rust modules missing top-level documentation.
Testing (Unit And Behavioural) ✅ Passed PASS: The new unit tests cover parser and pin-validation edge cases, and the behavioural tests exercise the Makefile and CI workflow contracts at the file boundary.
Testing (Property / Proof) ✅ Passed Runner/config/docs wiring only; no new range-based invariant or lemma proof obligation was introduced, so property/proof testing is not required.
Testing (Compile-Time / Ui) ✅ Passed PASS: The PR adds text-contract tests for Makefile/CI files only; no compile-time or UI-adjacent behaviour appears, so trybuild or snapshots are not required.
Unit Architecture ✅ Passed PASS: The patch keeps responsibilities separate; it adds read-only contract tests and runner/config wiring, with explicit capability-scoped file reads and no new hidden side effects.
Domain Architecture ✅ Passed Only docs, CI, Makefile, config, and test-contract files changed; no src/ domain modules were edited, so domain boundaries stay intact.
Observability ✅ Passed Only test runner, CI, docs and contract tests changed; no production runtime behaviour was altered, so no new logs, metrics, traces or alerts were needed.
Security And Privacy ✅ Passed Accept the change: it adds no secrets, auth bypasses, or widened permissions; the new env pin is a fixed version string, not sensitive data.
Performance And Resource Use ✅ Passed Pass: keep this change out of hot paths; the only loops are linear scans over small Makefile/CI files, and nextest stays bounded with no blanket retries.
Concurrency And State ✅ Passed PASS: env-mutating suites use EnvLock/RAII guards, the new serial-env nextest group is documented and tested, and re-entrant lock behaviour is covered.
Architectural Complexity And Maintainability ✅ Passed PASS: The PR stays explicit and local—Makefile, config, docs, and file-scoped contract tests add no new framework, service layer, or hidden orchestration.
Rust Compiler Lint Integrity ✅ Passed The new Rust test modules add no allow(dead_code|unused*)/expect(dead_code) suppressions, no fake-use anchors, and only cheap, justified string ownership in YAML lookups.
📋 Issue Planner

Built with CodeRabbit's Coding Plans for faster development and fewer bugs.

View plan used: #463

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-463-adopt-cargo-nextest-as-the-default-test-runner

Comment @coderabbitai help to get the list of available commands.

codescene-access[bot]

This comment was marked as outdated.

@sourcery-ai

sourcery-ai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Adopts cargo-nextest as the default non-doctest test runner, refactors make test into a nextest + doctest composite, wires a pinned cargo-nextest install into CI, introduces a minimal .config/nextest.toml, and adds workflow/Makefile contract tests plus documentation updates to describe the new testing model.

Flow diagram for make test using cargo-nextest and doctest runners

flowchart TD
  A[make_test] --> B[test-nextest]
  A --> C[doctest]

  B --> D[cargo_nextest_run_all_targets_all_features]
  C --> E[cargo_test_doc_all_features]

  D --> F[.config_nextest_toml_serial-env_group]
  F --> D

  subgraph Env
    G[BUILD_JOBS]
    H[NEXTEST_BUILD_JOBS]
  end

  G --> C
  H --> D
Loading

File-Level Changes

Change Details Files
Make make test a composite nextest+doctest runner with distinct build-parallelism knobs.
  • Refactor the test target to depend on new test-nextest and doctest targets instead of invoking cargo test --all-targets directly.
  • Add NEXTEST_BUILD_JOBS alongside BUILD_JOBS so cargo-nextest build parallelism (--build-jobs) is configured independently of Cargo’s -j.
  • Have test-nextest run RUSTFLAGS="-D warnings" cargo nextest run --all-targets --all-features and doctest run RUSTFLAGS="-D warnings" cargo test --doc --all-features.
  • Mark test-nextest and doctest as phony targets in the Makefile.
Makefile
Introduce nextest configuration to encode isolation and timeout policies for the non-doctest suite.
  • Add .config/nextest.toml with a default profile slow-timeout (warn after 60s, terminate after five periods) to surface hangs without breaking slow suites.
  • Define a serial-env test group (max-threads = 1) and scope it via an override filter to manifest_env_tests, ninja_env_tests, and env_path_tests binaries.
  • Document in comments that there are no blanket retries and that the serial-env group expresses a shared serialization contract even though nextest is process-per-test.
.config/nextest.toml
Wire cargo-nextest installation and usage into CI with a version pin and contract tests.
  • Add NEXTEST_VERSION to build-test job env in ci.yml and install taiki-e/install-action as Install cargo-nextest using tool: nextest@${{ env.NEXTEST_VERSION }}.
  • Ensure the CI Test step still runs make test, with nextest installation happening before it.
  • Add helpers in tests/workflow_ci.rs to locate steps, read job env, and parse action refs/versions.
  • Add is_pinned_action_ref and is_exact_version helpers with unit tests to assert action ref shape and semantic version pinning.
  • Add behavioural tests that assert NEXTEST_VERSION is an exact major.minor.patch, that the Install cargo-nextest step uses a pinned taiki-e/install-action@<40-char lowercase-hex SHA>, resolves its tool spec from NEXTEST_VERSION, is unguarded by if:, and precedes the Test step which runs make test.
  • Exclude the new Rust contract test crate from Dylint via dylint.toml.
.github/workflows/ci.yml
tests/workflow_ci.rs
dylint.toml
Add Rust-side contract tests to pin the Makefile test entrypoint and nextest config shape.
  • Introduce tests/makefile_test_target.rs to validate that make test depends on test-nextest and doctest, that test-nextest uses cargo-nextest with --all-features and RUSTFLAGS="-D warnings", and that doctest runs cargo test --doc (not nextest) with warnings denied.
  • Implement small Makefile parsing helpers (parse_rule, target_prerequisites, target_recipe) with unit tests to extract targets, prerequisites, and recipes while ignoring help comments and non-rule lines.
  • Add a behavioural test that checks .config/nextest.toml declares the serial-env test-group with max-threads = 1 and that it covers manifest_env_tests, ninja_env_tests, and env_path_tests.
  • Use filesystem helpers to read repo-local files from tests via CARGO_MANIFEST_DIR-relative paths.
tests/makefile_test_target.rs
.config/nextest.toml
Update developer and agent documentation to describe nextest as the canonical runner and document configuration scope.
  • In docs/developers-guide.md, document that make test runs cargo-nextest for non-doctest tests and a separate doctest pass; add a ## Test execution section describing the composed targets, the separation of BUILD_JOBS and NEXTEST_BUILD_JOBS, the nextest config’s serial-env group, no-retry policy, slow-timeout, and how this interacts with #[serial] and isolation utilities; and note that coverage and mutation workflows keep their own runners.
  • Update behavioural testing guidance to state that BDD tests run under cargo-nextest and that nextest’s process-per-scenario model reinforces isolation; change the recommended command from cargo test --test bdd_tests to cargo nextest run --test bdd_tests.
  • Adjust the snapshot testing guide to note that the canonical runner is cargo-nextest, recommend make test or cargo nextest run --test ninja_snapshot_tests, and instruct using cargo insta test --test-runner nextest instead of the default runner.
  • Update AGENTS.md (and CRUSH.md via symlink) so the make test description shows the two-pass nextest+doctest commands, notes that both deny warnings, names cargo-nextest as the runner, and adds a Testing section bullet describing local nextest installation and pointing at the developers guide’s Test execution section.
docs/developers-guide.md
AGENTS.md
docs/snapshot-testing-in-netsuke-using-insta.md

Assessment against linked issues

Issue Objective Addressed Explanation
#463 Update the test runner so that make test executes all non-doctest Rust tests via cargo-nextest, runs doctests explicitly, and CI installs a pinned cargo-nextest version while invoking the same canonical Make target.
#463 Introduce a checked-in cargo-nextest configuration (including narrow, documented serialization groups without blanket retries) and keep coverage/mutation-testing workflows using their existing purpose-built runners.
#463 Update developer and contributor documentation and workflow contract tests so that make test is documented as the canonical entry point with the new nextest-based runner and doctest pass, and ensure these contracts are enforced by tests.

Possibly linked issues

  • Adopt cargo-nextest as the default test runner #463: They are the same change: adopting cargo-nextest for make test, adding nextest config, doctest pass, CI pin, and docs.
  • #0: PR fully implements the issue: adds cargo test --doc to make test, keeps it separate, updates docs and workflow contracts.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@leynos
leynos marked this pull request as ready for review July 29, 2026 00:10

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @leynos, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7ab32bdbf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/makefile_test_target.rs Outdated
codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

@pandalump

Copy link
Copy Markdown
Collaborator

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot added the Issue label Jul 29, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/developers-guide.md`:
- Around line 87-96: Update the Nextest installation guidance in
docs/developers-guide.md lines 87-96 and AGENTS.md lines 247-250 to use the
version pinned by NEXTEST_VERSION in .github/workflows/ci.yml for both cargo
install and cargo binstall examples, preserving the claim that local runs match
CI.
- Around line 125-128: Update the lint guidance in the developers guide to
remove the in-source #[allow(no_std_fs_operations)] exemption path. Direct
permanent exceptions to narrowly scoped configuration and mention item-level
#[expect(...)] only for permitted temporary cases, while preserving the
preference for migrating to cap_std.

In `@docs/whitaker-users-guide.md`:
- Around line 646-648: Remove the tip’s guidance to use
#[allow(no_std_fs_operations)] and replace it with the repository-approved
exemption mechanism, preserving the single-site exemption context without
recommending forbidden lint attributes.

In `@tests/makefile_test_target.rs`:
- Around line 111-124: The test-nextest assertions in the makefile test should
validate the complete runner configuration rather than isolated substrings.
Extend the checks around target_recipe and the related assertions near the
additional range to require --all-targets and verify that the override
containing all three binary filters assigns test-group = 'serial-env',
preserving the existing runner, feature, and warning requirements.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ac5de7a5-594a-4cd5-9e8a-27fdf23f0aed

📥 Commits

Reviewing files that changed from the base of the PR and between 23591d0 and 22306da.

📒 Files selected for processing (10)
  • .config/nextest.toml
  • .github/workflows/ci.yml
  • AGENTS.md
  • Makefile
  • docs/developers-guide.md
  • docs/snapshot-testing-in-netsuke-using-insta.md
  • docs/whitaker-users-guide.md
  • dylint.toml
  • tests/makefile_test_target.rs
  • tests/workflow_ci.rs
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • leynos/rstest-bdd (auto-detected)
  • leynos/ortho-config (auto-detected)
  • leynos/shared-actions (auto-detected)

Comment thread docs/developers-guide.md
Comment thread docs/developers-guide.md Outdated
Comment thread docs/whitaker-users-guide.md Outdated
Comment thread tests/makefile_test_target.rs
codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

@leynos

leynos commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai resume

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Reviews resumed.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/developers-guide.md`:
- Around line 518-523: Update the documentation sentence describing serial
execution to refer to “the #[serial] markers on tests in the three binaries”
rather than implying the markers are applied to binaries, while preserving the
surrounding explanation and references.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 87da3732-0404-46e1-95a6-591acab8b4c3

📥 Commits

Reviewing files that changed from the base of the PR and between 23591d0 and b87b8dd.

📒 Files selected for processing (10)
  • .config/nextest.toml
  • .github/workflows/ci.yml
  • AGENTS.md
  • Makefile
  • docs/developers-guide.md
  • docs/snapshot-testing-in-netsuke-using-insta.md
  • docs/whitaker-users-guide.md
  • dylint.toml
  • tests/makefile_test_target.rs
  • tests/workflow_ci.rs
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • leynos/rstest-bdd (auto-detected)
  • leynos/ortho-config (auto-detected)
  • leynos/shared-actions (auto-detected)

Comment thread docs/developers-guide.md Outdated
Repository owner deleted a comment from coderabbitai Bot Jul 29, 2026
codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

@pandalump

Copy link
Copy Markdown
Collaborator

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/developers-guide.md (1)

492-493: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Fix the malformed failure sentence.

Replace “Either pass failing fails make test” with “If either pass fails, make test fails.”

Triage: [type:grammar]

As per path instructions, grammatical findings require a Triage: paragraph with the grammar tag.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/developers-guide.md` around lines 492 - 493, Correct the malformed
sentence in the developer guide to state that if either pass fails, make test
fails, and retain the required Triage paragraph with the grammar tag.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/developers-guide.md`:
- Around line 492-493: Correct the malformed sentence in the developer guide to
state that if either pass fails, make test fails, and retain the required Triage
paragraph with the grammar tag.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 2d3f4e8b-6869-4d35-80ef-e47630c9fc2b

📥 Commits

Reviewing files that changed from the base of the PR and between b87b8dd and e117313.

📒 Files selected for processing (1)
  • docs/developers-guide.md
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • leynos/rstest-bdd (auto-detected)
  • leynos/ortho-config (auto-detected)
  • leynos/shared-actions (auto-detected)

leynos and others added 6 commits July 30, 2026 11:40
Route `make test` through `cargo nextest run` and add a companion
`doctest` pass, so `make test` remains the single canonical entry point
while gaining per-test process isolation.

`cargo test --all-targets` never ran doctests, so documentation examples
were previously ungated; the new `doctest` target restores that coverage.

Add `.config/nextest.toml` with a single narrow `serial-env` test group
covering the three environment-mutating integration binaries. No blanket
retries; a conservative slow-timeout surfaces hangs without failing
legitimately slow suites.

nextest spells build parallelism `--build-jobs` and reserves `-j` for
test concurrency, so `NEXTEST_BUILD_JOBS` is kept separate from
`BUILD_JOBS` to avoid silently reinterpreting a `-j` value.
Install cargo-nextest unconditionally in the `build-test` job, before the
`Test` step, so every matrix leg can run `make test`. `NEXTEST_VERSION`
is the single source of truth for the pin.

Extend the CI contract tests with the nextest install wiring and add a
Makefile contract test covering the `test` composite, the nextest and
doctest recipes, and the `serial-env` group in `.config/nextest.toml`.

The install-action reference is asserted by shape (a 40-hex commit SHA)
rather than a literal commit. Asserting the exact SHA would make every
Dependabot bump fail until a human edited the constant, which is the
lockstep problem "Workflow pins and Dependabot" in the developers' guide
already rules out for shared-actions callers.

Coverage keeps its own llvm-cov runner and the mutation workflow keeps
`--all-features` parity, so neither changes implicitly.
AGENTS.md (and CRUSH.md, a symlink to it) claimed `make test` was
`cargo test --workspace`; replace that with the actual nextest and
doctest passes, and name the runner, its configuration file, and the
local install command in the Testing section.

Add a "Test execution" section to the developers' guide covering the two
passes, the `.config/nextest.toml` scope, the `serial-env` group and why
it is narrow, the no-blanket-retry and timeout policy, how the group
relates to the retained `#[serial]` markers and the EnvLock/CwdGuard
utilities, and why the coverage and mutation runners are unchanged.

Update the behavioural testing strategy to describe nextest's
process-per-test model as reinforcing the per-scenario isolation policy,
and point the snapshot guide at the nextest-aware `cargo insta` flags.
Read the Makefile and nextest configuration through a capability-scoped
`cap_std::fs_utf8::Dir` rooted at the repository, with `camino` paths,
instead of `std::fs` and `std::path`. The contract test now needs no
`no_std_fs_operations` exclusion at all.

Re-import the Whitaker user's guide, which documents the new
`excluded_paths` option and the fact that the lint now honours in-source
`#[allow]`. Correct the developers' guide, which still claimed the lint
ignores lint-level attributes.

Rewrite dylint.toml around `excluded_paths`. The application crate was
exempt wholesale; only `netsuke::stdlib::which::lookup` and
`netsuke::runner::process::file_io` actually need ambient access, so the
rest of `netsuke` is now under the capability policy. The behavioural
step definitions, CLI integration tests, and the shared workflow-reading
helper are scoped the same way, which retires the crate-level entries
for `bdd_tests`, `cli_tests`, `workflow_ci`, and `workflow_release`.

`dependabot_config_tests` and `logging_stderr_tests` no longer trip the
lint at all; their entries are removed. Crate-level entries are kept only
where the ambient I/O lives in the crate root, where a path entry would
be no narrower.
Point the local install guidance at the CI pin. The docs claimed local
runs matched CI while installing whatever version was current; they now
read `NEXTEST_VERSION` out of `.github/workflows/ci.yml` rather than
copying the number, so the two cannot drift.

Stop recommending `#[allow(no_std_fs_operations)]`. This repository
denies `clippy::allow_attributes`, so that attribute will not compile
here; permanent exceptions belong in `dylint.toml` and an in-source
exemption must be a temporary `#[expect(..., reason = "…")]`. The
vendored Whitaker guide carries the same correction, flagged as a local
deviation so a future re-import does not silently drop it.

Assert `--all-targets` in the nextest recipe, and parse
`.config/nextest.toml` with the `toml` crate instead of matching
substrings: the test now checks that `serial-env` serializes its members
and that one override actually binds all three binaries to that group.
Verified it fails when the binding is broken.
`#[serial]` annotates test functions; the guide implied it was applied to
the binaries themselves.
`make markdownlint` is not one of the CI steps, so this MD012 violation
reached main unnoticed. It is not in a region this branch authored, but
the branch's gate run trips over it, so fix it here.
@lodyai
lodyai Bot force-pushed the issue-463-adopt-cargo-nextest-as-the-default-test-runner branch from e117313 to 93e6b25 Compare July 30, 2026 09:45
codescene-access[bot]

This comment was marked as outdated.

The commands passed `--version "$NEXTEST_VERSION"` but nothing in the
file defined that variable; it deferred to the developers' guide for the
extraction. A reader following AGENTS.md alone would have run
`--version ""`. Inline the same read-the-pin-from-the-workflow form the
developers' guide uses, so both files obtain the version from CI's
`NEXTEST_VERSION` and pass it explicitly to both install commands.
codescene-access[bot]

This comment was marked as outdated.

codescene-access[bot]

This comment was marked as outdated.

@codescene-access codescene-access Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No quality gates enabled for this code.

@leynos

leynos commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai Have the following now been resolved?

docs/developers-guide.md (1)

492-493: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix the malformed failure sentence.
Replace “Either pass failing fails make test” with “If either pass fails, make test fails.”
Triage: [type:grammar]
As per path instructions, grammatical findings require a Triage: paragraph with the grammar tag.

🤖 Detailed instructions

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @docs/developers-guide.md around lines 492 - 493, Correct the malformed
sentence in the developer guide to state that if either pass fails, make test
fails, and retain the required Triage paragraph with the grammar tag.

Source: Path instructions

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- Relevant sentence matches ---'
rg -n -C 3 -e 'Either pass failing fails' -e 'If either pass fails' docs/developers-guide.md || true

printf '%s\n' '--- Lines 480-505 ---'
nl -ba docs/developers-guide.md | sed -n '480,505p'

Length of output: 1019


@leynos Mark this finding as resolved. docs/developers-guide.md Line 492 now states exactly: “If either pass fails, make test fails.” The malformed wording is no longer present.

The line-display utility was unavailable in the sandbox, but rg returned the numbered matching context and conclusively verified the correction.

You are interacting with an AI system.

@leynos
leynos merged commit 695727d into main Jul 30, 2026
20 checks passed
@leynos
leynos deleted the issue-463-adopt-cargo-nextest-as-the-default-test-runner branch July 30, 2026 23:51
lodyai Bot pushed a commit that referenced this pull request Jul 31, 2026
`main` adopted cargo-nextest as the default runner in `#463`/`#473`,
splitting `make test` into a `test-nextest` pass and a separate
`doctest` pass. That left this branch stating, as a composition rule,
that nextest "is not used in this repository" — now false — and left
`make dev-test` on `cargo test` while the gate it stands in for had
moved to a runner with different process isolation.

Point `dev-test` at `cargo nextest run --all-targets --all-features`,
so the accelerated loop and the gate exercise the same runner under the
same `.config/nextest.toml`, including its `serial-env` group. The
acceleration is unaffected: `RUSTUP_TOOLCHAIN` and `cargo --config` are
Cargo-level rather than runner-level, so they compose with nextest
unchanged. Verified end to end — 1188 tests pass under Cranelift and
mold.

Use `NEXTEST_BUILD_JOBS` rather than `BUILD_JOBS`, for the reason main
records when it introduced the split: nextest reserves `-j` for test
concurrency, so a Cargo-shaped `-j` would silently become a thread
count.

`dev-test` deliberately omits the doctest pass, `cargo test --doc`
being a separate and comparatively quick runner. The guide says so
rather than leaving the omission to be discovered, and continues to
direct developers at `make test` before proposing a change.

Update the Make-target test to assert the new command line, so the
contract stays checked rather than assumed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt cargo-nextest as the default test runner

2 participants