Skip to content

Add merge_gha_workflows task and pipeline stage (#50 slice 3) - #73

Merged
silug merged 2 commits into
simp:mainfrom
silug:issue-50-workflow-merge
Jul 31, 2026
Merged

Add merge_gha_workflows task and pipeline stage (#50 slice 3)#73
silug merged 2 commits into
simp:mainfrom
silug:issue-50-workflow-merge

Conversation

@silug

@silug silug commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Final slice of #50: refresh GHA workflow files from their baseline templates while preserving every Renovate-managed value already in the repo. Independent of #70/#72 (disjoint files; the stage anchors next to configure_renovate (immediately before it) specifically to avoid overlapping #72's diff) — but the three together complete the issue.

Cleaner than planned: no psych-pure

The issue proposed a psych-pure comment-preserving round-trip. This implementation avoids re-serialization entirely: since downstream workflow files are stamped copies of the templates, the template text is canonical — structure, comments, formatting, byte-for-byte — and the Renovate-managed scalars are grafted back onto it using stdlib Psych's node line/column coordinates (surgical rest-of-line replacement). Comment/format preservation holds by construction, and there's no new gem to install or trust.

Semantics

  • preserve_keys (default: uses, image, container, ruby-version, runs-on; extensible via the session config's merge_github_workflows.preserve_keys) selects which mapping keys carry Renovate-managed values. The defaults cover the real-world cases from pupmod-simp-pupmod: dd98340 (ruby-version: 3.2 → 3.4.9) and 837fb5e (runs-on: ubuntu-20.04 → ubuntu-24.04), plus action refs and container images.
  • Pairing is by [mapping path, key, identity], where identity is the part Renovate never changes (actions/checkout@v5actions/checkout; ghcr.io/x/y:8ghcr.io/x/y; bare versions → none). So: distinct versions per job survive independently; trailing comments ride along (pinned-digest conventions keep the human-readable version there); steps genuinely new to — or restored by — the template take the template's value, per the policy in the issue.
  • Scope discipline: the merge_github_workflows stage merges only files present in both the repo and the template chain (same per-module override resolution as profile::github_actions). Creating and removing workflow files stays the profile's job, and repo-specific workflows with no template are never touched.

Verification

  • 13 specs, several driven by the real pr_tests.yml template, including byte-identity assertions ("output equals template except the grafted lines") and cases for pinned-digest comments, per-job version divergence, restored jobs, multi-file invocations, and idempotency. 167 examples total, 0 failures.
  • End-to-end against a file:// fixture whose pr_tests.yml had Renovate bumps (checkout@v7, ruby-version: 3.4.9, ubuntu-24.04) and structural drift, plus an untemplated custom.yml: the merge restored template structure, preserved all 6+5+6 bumped values, left custom.yml untouched, and run 2 reported 0 ok / 1 unchanged / 0 failed. (A deliberately bumped value inside a commented-out template job was correctly reverted to template text — comments belong to the template.)
  • puppet parser validate --tasks, bolt plan show, stage-list dry run, CI idempotency e2e all green. No CHANGELOG edit per the freeze.

Completing #50

Once this and #70 merge, the closing move is a one-line Hiera flip (profile::github_actions::strategy: bootstrap per project_type) plus closing the issue — I'd fold that into whichever merges last, or a tiny follow-up.

Refs #50

🤖 Generated with Claude Code

Final slice of the Renovate-resilient template machinery: refresh GHA
workflow files from their baseline templates while preserving every
Renovate-managed value already in the repo.

Approach: the template text is canonical — structure, comments, and
formatting come from it byte-for-byte — and the Renovate-managed
scalar values are grafted back on using Psych's node line/column
coordinates. No YAML re-serialization happens at all, so comment/
format preservation holds by construction and no new gem (psych-pure)
is needed.

- preserve_keys (default: uses, image, container, ruby-version,
  runs-on; extensible per session config) selects which mapping keys'
  values are Renovate's
- Values pair by [mapping path, key, identity], where identity is the
  part Renovate never changes (action before @, image before :, none
  for bare versions) — so distinct versions per job survive, trailing
  comments (pinned-digest conventions) ride along, and steps new to
  the template take the template's value
- The merge_github_workflows stage merges only files that exist in
  BOTH the repo and the template chain (same per-module override
  resolution as profile::github_actions); creating/removing workflow
  files remains the profile's job, and repo-specific workflows with
  no template are never touched

Verified with 13 specs (several driven by the real pr_tests.yml
template, including byte-identity assertions) and an e2e run against
a fixture with Renovate-bumped checkout@v7 / ruby-version 3.4.9 /
ubuntu-24.04 plus structural drift: the merge restored template
structure, preserved all bumped values (including per-job ones), left
an untemplated custom workflow untouched, and a second run reported
"1 unchanged". Real-world cases from pupmod-simp-pupmod dd98340
(ruby-version) and 837fb5e (runs-on) are covered by the defaults.

After this and simp#70 merge, profile::github_actions can flip to
bootstrap mode per project_type in Hiera, completing simp#50.

Refs simp#50

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@nick-markowski

Copy link
Copy Markdown
Member

Really like this approach — grafting managed values onto canonical template text via Psych node coordinates is clean, and using Psych.parse (AST only, not load/unsafe_load) means there's no object-instantiation surface from untrusted workflow YAML. The pairing by [path, key, identity] with sequence indices excluded is sound: steps reorder safely, per-job version divergence survives, and new/restored steps correctly fall through to the template value. Rest-of-line capture carries pinned-digest trailing comments verbatim, it's idempotent, and the byte-identity-except-grafts invariant is asserted directly. Approving, with two things:

1. Drop the warn stdin debug line (same as #72) — it dumps the full JSON input, including every workflow template, to stderr on each run.

2. Make the Hiera-flip dependency explicit — the stage is inert without it. It runs after apply_puppet_role (line 183), and while profile::github_actions is still enforce, that stage overwrites each workflow with pure template first — so this stage then reads template-equals-existing and no-ops, with the Renovate values already enforced away upstream. So enabling merge_github_workflows before the profile::github_actions::strategy: bootstrap flip protects nothing; the stage and the flip have to land together. Worth tying them explicitly (config docs / a note) so nobody enables it pre-flip expecting preservation.

Minor: identity() returns nil for a ported registry (registry:5000/foo:8) since [\w.-]+ excludes //:, so it'd pair per-key positionally instead of by image — unlikely in GHA, fine to leave. And the body says the stage anchors "after configure_renovate" but it's inserted just before it — immaterial, just a note.

Everything else — coordinate grafting, multi-line skip, malformed/empty handling, plan scoping, template resolution — checks out.

- Remove the `warn stdin` debug line (it echoed every workflow template
  to stderr on each run)
- identity() now handles ported registries (registry:5000/foo:8 pairs
  by everything before the tag colon), with a spec
- Ship the profile::github_actions::strategy: bootstrap Hiera flip for
  pupmod/pupmod_skeleton WITH this stage: while the profile enforces,
  the apply stage overwrites workflows with pure template before this
  stage runs, so enabling merge_github_workflows pre-flip protects
  nothing. The key is inert until the managed_file strategy support
  merges, and the stage comment now documents the coupling. rubygem and
  unknown_with_ci project types stay enforce, since the merge stage
  does not cover them

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@silug

silug commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

All addressed in 144716f:

  1. warn stdin dropped.
  2. The flip now ships with the stage — you're right that they're inseparable: with the profile still enforcing, apply_puppet_role overwrites each workflow with pure template before this stage runs, so pre-flip enablement protects nothing. profile::github_actions::strategy: bootstrap is now set for pupmod/pupmod_skeleton in data/project_types/ (the exact scope the merge stage covers — rubygem/unknown_with_ci stay enforce since the stage doesn't run for them). The key is inert until Add bootstrap mode for managed baseline files (#50 slice 1) #70's strategy support merges (Hiera automatic lookup only consults declared params), and it activates the moment both are in — so merge order doesn't matter. The stage comment documents the coupling.
  3. Minor Initial hacks to get pdk convert --force running #1 taken too since it was cheap: identity() now handles ported registries (registry:5000/foo:8 pairs by everything before the tag colon), with a spec. Minor Rollup add_gce and de-jirafication #2: correct — the stage sits immediately before configure_renovate, not after; PR body fixed.

12 examples, 0 failures.

🤖 Generated with Claude Code

@silug
silug merged commit 4c6be60 into simp:main Jul 31, 2026
2 checks passed
@silug
silug deleted the issue-50-workflow-merge branch July 31, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants