Skip to content

Add BoltSpec plan-testing suite (#76) - #77

Merged
silug merged 3 commits into
simp:mainfrom
silug:issue-76-boltspec
Jul 31, 2026
Merged

Add BoltSpec plan-testing suite (#76)#77
silug merged 3 commits into
simp:mainfrom
silug:issue-76-boltspec

Conversation

@silug

@silug silug commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Closes #76

Adds the BoltSpec plan-testing suite planned in the issue: unit coverage for the layer nothing tested before — plan logic, run with stubbed tasks/commands. No clones, no network; the whole suite runs in ~1 second.

What's covered (13 examples)

  • puppetsync::pipeline_stage (spec/plans/pipeline_stage_spec.rb + three fixture plans): stage-list gating, the Full idempotency: repos that need no changes must pass through cleanly #49 skip_unchanged_targets contract — expect_command('true').with_targets(['changed-repo']) makes BoltSpec require that the stubbed command ran only against the changed target — held-back-after-failure filtering (via a return-block building a mixed-exit-code ResultSet), and the Fail fast on unrecordable stage results instead of binding.pry #64 fatal error on non-Bolt-result stage returns.
  • The puppetsync plan's dynamic-inventory branch (Build inventory dynamically from the GitHub API instead of hand-maintained repolists #55): with list_github_repos stubbed, asserts the snapshot lands as a loadable repolist in a tmpdir project, static repos_config wins over generated entries, list_pipeline_stages mode never calls the listing task (no stub declared — a call would raise), and the plain static path writes no snapshot.
  • puppetsync::approve_github_prs / merge_github_prs — previously zero coverage of any kind: per-repo task parameters captured and asserted (target repo/branch from the repolist, fork user/branch, approval message), failure summarization fails the plan with the errors, and stage-list opt-out runs nothing.
  • puppetsync::batch: one puppetsync run per repolist, in order, via expect_plan stubs.

Harness & wiring

  • spec/plans/plan_spec_helper.rb — shared context: modulepath (fixtures + dist + modules + .modules), BoltSpec::Plans.init, env defaults for the token/Jira params, and the conventions from the probe (catch-all allow_out_message before specific expects; stub matching is last-defined-wins).
  • Fixture plans live in spec/fixtures/modules/puppetsync_test/not on the real bolt-project modulepath, so bolt plan show is unaffected (verified).
  • Plan specs need Bolt's Ruby (bolt_spec ships with openbolt), so .rspec excludes spec/plans/ from the plain-Ruby run, and the CI bolt job gets a step that installs rspec into .gems and runs the suite with -O /dev/null.

BoltSpec quirks encountered (documented in the helper / worth knowing for future specs)

  • Task parameters marked "sensitive": true in task metadata arrive wrapped in Sensitive at the stub, and metaparams arrive as symbols — exact with_params matching fights this, so the suite uses return-blocks that capture and unwrap params, then asserts on the captures.
  • return_for_targets only supports success shapes for commands; per-target failures need a return-block constructing a ResultSet with nonzero exit_code results.

Verification

  • Plan suite: 13 examples, 0 failures under Bolt's Ruby
  • Plain suite: 195 examples, 0 failures, with --dry-run confirming zero plan specs leak into the default run
  • bolt plan show unaffected by the fixture module; workflow YAML valid

The existing layers stay in place per the issue: subprocess specs own task internals, the managed_file contract script owns profile behavior (BoltSpec's apply() support is stub-only), and the e2e scripts own the real-pipeline paths.

🤖 Generated with Claude Code

silug and others added 3 commits July 31, 2026 15:14
Unit coverage for the layer nothing tested before: plan logic, run
with stubbed tasks/commands — no clones, no network, milliseconds per
example. 13 examples across:

- puppetsync::pipeline_stage: stage-list gating, skip_unchanged_targets
  (the simp#49 contract, via expect_command().with_targets), held-back-
  after-failure filtering, and the fatal non-Result error (simp#64)
- The puppetsync plan's dynamic-inventory branch (simp#55): stubbed
  list_github_repos, snapshot written as a loadable repolist,
  static-over-generated merge precedence, list_pipeline_stages skips
  generation, and the no-repos_source passthrough
- puppetsync::approve_github_prs / merge_github_prs (previously zero
  coverage of any kind): per-repo task params captured and asserted
  (unwrapping metadata-sensitive params), failure summarization fails
  the plan, stage-list opt-out
- puppetsync::batch: one sync run per repolist, in order, via plan
  stubs

Harness: spec/plans/plan_spec_helper.rb (shared context: modulepath
incl. .modules, BoltSpec init, env-var defaults for token/Jira params,
and the stub conventions learned during simp#76's probe — catch-all
allow_out_message before specific expects, last-defined-wins). Fixture
plans live in spec/fixtures/modules/puppetsync_test/, which is not on
the real bolt-project modulepath.

Wiring: plan specs require Bolt's Ruby (bolt_spec ships with
openbolt), so .rspec excludes spec/plans from the plain-Ruby run and
the CI bolt job runs them with -O /dev/null after installing rspec
into .gems. Closes simp#76.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI failed because `ruby -S rspec` only works when some rspec is
already on PATH — locally it silently found the system-Ruby binstub
from an earlier user install and then loaded libs from GEM_HOME, which
is why the suite passed here. Invoke the .gems binstub explicitly
instead (all docs/comments updated).

Re-testing under a hostile environment (env -i, no PATH rspec, no PWD)
also exposed that the approve/merge/batch specs relied on ambient PWD
for the plans' project_dir default; they now pass project_dir
explicitly. 13 examples green under `env -i PATH=/usr/bin:/bin`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bundler isolation hides the packaged openbolt gem, and declaring
openbolt in a Gemfile makes bundler vendor a second bolt from rubygems
and test that copy instead of the OS package that runs real syncs
(verified empirically). Record the rationale in the helper header so
the question doesn't need re-deriving.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@silug
silug merged commit 56c9003 into simp:main Jul 31, 2026
2 checks passed
@silug
silug deleted the issue-76-boltspec branch July 31, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a BoltSpec plan-testing suite (unit coverage for plan logic)

2 participants