Skip to content

feat(scaffold): per-repo pre-commit tools registry (L2 additive merge) - #81

Closed
guyoron1 wants to merge 541 commits into
mainfrom
mirror-2663-precommit-registry
Closed

feat(scaffold): per-repo pre-commit tools registry (L2 additive merge)#81
guyoron1 wants to merge 541 commits into
mainfrom
mirror-2663-precommit-registry

Conversation

@guyoron1

Copy link
Copy Markdown
Owner

Mirror of fullsend-ai#2663 for QualityFlow integration testing.

Original PR Summary

Extends the pre-commit tools registry with per-repo additive merge (L2). Target repos can place .pre-commit-tools.yaml at repo root to extend the upstream/org registry. Includes supply-chain security (reads from base branch, not PR head), uv match fix, and 13 unit tests.

QualityFlow

This PR will be used to demonstrate QualityFlow's automated test planning and generation pipeline.

Benkapner and others added 30 commits June 22, 2026 21:07
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Benjamin Kapner <bkapner@redhat.com>
…egrity

docs(security): add audit log integrity to threat model
…egrity

docs(security): add audit log integrity to threat model
Add a functional test framework for agent pipelines using
agent-eval-harness lifecycle hooks. The harness drives case iteration
with before_each/after_each hooks for ephemeral repo management,
while fullsend runs inside openshell sandboxes.

Key components:
- eval/scripts/setup-fixture.sh: before_each hook creates ephemeral
  GitHub repos and fixtures (issues/PRs) from input.yaml
- eval/scripts/run-fullsend.sh: CLI runner invokes fullsend run
- eval/scripts/capture-fixture.sh: after_each hook snapshots fixture
  state for judges
- eval/scripts/teardown-fixture.sh: after_each hook deletes repos
- eval/run-functional.sh: orchestrator calling workspace.py,
  execute.py, and score.py with behavioral threshold checks
- eval/triage/: first eval suite with LLM judge and label checks

Also includes CI workflow, behavioral thresholds (max_turns,
max_cost_usd), metrics capture from Claude Code stream events,
ADRs, and documentation.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Add a functional test framework for agent pipelines using
agent-eval-harness lifecycle hooks. The harness drives case iteration
with before_each/after_each hooks for ephemeral repo management,
while fullsend runs inside openshell sandboxes.

Key components:
- eval/scripts/setup-fixture.sh: before_each hook creates ephemeral
  GitHub repos and fixtures (issues/PRs) from input.yaml
- eval/scripts/run-fullsend.sh: CLI runner invokes fullsend run
- eval/scripts/capture-fixture.sh: after_each hook snapshots fixture
  state for judges
- eval/scripts/teardown-fixture.sh: after_each hook deletes repos
- eval/run-functional.sh: orchestrator calling workspace.py,
  execute.py, and score.py with behavioral threshold checks
- eval/triage/: first eval suite with LLM judge and label checks

Also includes CI workflow, behavioral thresholds (max_turns,
max_cost_usd), metrics capture from Claude Code stream events,
ADRs, and documentation.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
- Missing metrics.json is now a FAIL (not a warning), ensuring
  behavioral thresholds cannot be silently bypassed when the agent
  crashes or fullsend run fails.
- Validate that jq output is numeric before threshold comparison,
  preventing null/malformed values from silently passing as 0.
- Add shellcheck SC2317 disable directives for trap handler commands
  that shellcheck incorrectly flags as unreachable.

Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
- Missing metrics.json is now a FAIL (not a warning), ensuring
  behavioral thresholds cannot be silently bypassed when the agent
  crashes or fullsend run fails.
- Validate that jq output is numeric before threshold comparison,
  preventing null/malformed values from silently passing as 0.
- Add shellcheck SC2317 disable directives for trap handler commands
  that shellcheck incorrectly flags as unreachable.

Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The harness was referenced twice: once as a git submodule and once as a
pip install from the same git URL. Install from the already-checked-out
submodule so the fork URL only appears in .gitmodules.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The harness was referenced twice: once as a git submodule and once as a
pip install from the same git URL. Install from the already-checked-out
submodule so the fork URL only appears in .gitmodules.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Move max_turns and max_cost_usd checks from custom shell code in
run-functional.sh into deterministic check judges in eval.yaml. The
harness's score.py now enforces these via min_pass_rate: 1.0 thresholds.

Extract the pre-flight annotation validation into a standalone
eval/lint-cases.sh linter, wired up as `make lint-eval-cases` and
included in `make test`. This runs cheaply without executing agents.

Net effect: ~90 lines removed from run-functional.sh.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Move max_turns and max_cost_usd checks from custom shell code in
run-functional.sh into deterministic check judges in eval.yaml. The
harness's score.py now enforces these via min_pass_rate: 1.0 thresholds.

Extract the pre-flight annotation validation into a standalone
eval/lint-cases.sh linter, wired up as `make lint-eval-cases` and
included in `make test`. This runs cheaply without executing agents.

Net effect: ~90 lines removed from run-functional.sh.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Extend lint-cases.sh to verify that eval.yaml declares max_turns and
max_cost judges, not just that annotations.yaml declares the thresholds.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Extend lint-cases.sh to verify that eval.yaml declares max_turns and
max_cost judges, not just that annotations.yaml declares the thresholds.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The CLI runner receives {output_dir} which is workspace/output. Writing
metrics.json to $OUTPUT_DIR/output/ created a double-nested path that
score.py couldn't find. Write to $OUTPUT_DIR/metrics.json instead so
the file appears at the expected output/metrics.json key.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The CLI runner receives {output_dir} which is workspace/output. Writing
metrics.json to $OUTPUT_DIR/output/ created a double-nested path that
score.py couldn't find. Write to $OUTPUT_DIR/metrics.json instead so
the file appears at the expected output/metrics.json key.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The triage agent consistently takes ~23 turns on this case. The
previous threshold of 15 was too tight.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The triage agent consistently takes ~23 turns on this case. The
previous threshold of 15 was too tight.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Add three new triage eval cases covering the remaining major outcomes:

- 002-needs-info-vague-crash: vague issue with no repro steps, expects
  action "insufficient" and needs-info label
- 003-feature-request: clear feature request, expects action "sufficient"
  with category "feature" and triaged+feature labels (not ready-to-code)
- 004-duplicate-issue: issue duplicating a seed issue, expects action
  "duplicate" with duplicate label

Supporting changes:
- setup-fixture.sh: support seed_issues in input.yaml for pre-populating
  issues before the main fixture (needed by duplicate test)
- eval.yaml: add forbidden_labels judge to verify wrong labels are NOT
  applied (needs-info must not get ready-to-code, etc.)
- lint-cases.sh: check for forbidden_labels judge presence

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Add three new triage eval cases covering the remaining major outcomes:

- 002-needs-info-vague-crash: vague issue with no repro steps, expects
  action "insufficient" and needs-info label
- 003-feature-request: clear feature request, expects action "sufficient"
  with category "feature" and triaged+feature labels (not ready-to-code)
- 004-duplicate-issue: issue duplicating a seed issue, expects action
  "duplicate" with duplicate label

Supporting changes:
- setup-fixture.sh: support seed_issues in input.yaml for pre-populating
  issues before the main fixture (needed by duplicate test)
- eval.yaml: add forbidden_labels judge to verify wrong labels are NOT
  applied (needs-info must not get ready-to-code, etc.)
- lint-cases.sh: check for forbidden_labels judge presence

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Cases use isolated ephemeral repos with UUID suffixes, so there's no
shared state. Sequential execution took ~15 min of agent time across
4 cases; parallel should bring wall-clock down to ~6 min.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Cases use isolated ephemeral repos with UUID suffixes, so there's no
shared state. Sequential execution took ~15 min of agent time across
4 cases; parallel should bring wall-clock down to ~6 min.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0045 was taken on main by forge-portable-harness-schema while this
branch was out of date. Renumber both branch ADRs and update all
cross-references.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0045 was taken on main by forge-portable-harness-schema while this
branch was out of date. Renumber both branch ADRs and update all
cross-references.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0046-host-side-api-server-design was merged to main while this
branch was open, creating a duplicate number. Renumber
0046-functional-tests-for-agent-pipelines to 0048 and update all
cross-references. Also fix stale heading numbers in ADRs 0047 and 0048.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0046-host-side-api-server-design was merged to main while this
branch was open, creating a duplicate number. Renumber
0046-functional-tests-for-agent-pipelines to 0048 and update all
cross-references. Also fix stale heading numbers in ADRs 0047 and 0048.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
- Write partial metrics.json on agent failure so downstream judges can
  inspect behavioral data even when the agent errors out
- Add CLOUD_ML_REGION to runner.env in eval.yaml (was only in
  execution.env)
- Let teardown-fixture.sh fail visibly instead of swallowing errors
  with 2>/dev/null || true — the harness on_failure: continue already
  handles non-fatal teardown
- Distinguish infrastructure failures from agent failures in
  run-functional.sh by checking for case output before proceeding to
  scoring
- Fix stale reference to eval/fullsend-runner.sh in ADR 0047 (actual
  path is eval/scripts/run-fullsend.sh)

Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
- Write partial metrics.json on agent failure so downstream judges can
  inspect behavioral data even when the agent errors out
- Add CLOUD_ML_REGION to runner.env in eval.yaml (was only in
  execution.env)
- Let teardown-fixture.sh fail visibly instead of swallowing errors
  with 2>/dev/null || true — the harness on_failure: continue already
  handles non-fatal teardown
- Distinguish infrastructure failures from agent failures in
  run-functional.sh by checking for case output before proceeding to
  scoring
- Fix stale reference to eval/fullsend-runner.sh in ADR 0047 (actual
  path is eval/scripts/run-fullsend.sh)

Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Main acquired ADR numbers 0047-0050 while this branch was open.
Renumber the PR's ADRs and update all cross-references.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
waynesun09 and others added 27 commits June 24, 2026 12:10
When the CLI scaffolds workflow files during `fullsend github setup`,
pin the `uses:` directive and `fullsend_ai_ref` parameter to the
binary's build-time commit SHA instead of the mutable `@v0` tag.
Dev builds fall back to `@v0`.

Closes fullsend-ai#1933

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
…pr-commits

fix(ci): lint individual commits on PRs to catch invalid prefixes early
…levant

The merge_group trigger had no path filter and the relevance-check step
only ran for pull_request_target events, so every PR entering the merge
queue ran the full e2e and functional test suites regardless of what
files changed.

Extend the relevance check to also run on merge_group events, using the
compare API with the merge group base/head SHAs to determine changed
files.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
…mmit SHAs

PR fullsend-ai#2508 pinned actions in .github/workflows/ and
.github/actions/setup-gcp/ but missed the root action.yml composite
action. Repos with strict SHA-pinning policies (e.g. openkaiden/kaiden)
reject the unpinned tag refs, failing the Triage job with:

  "actions/setup-go@v6, actions/cache/restore@v4, and
   actions/upload-artifact@v7 are not allowed because all actions
   must be pinned to a full-length commit SHA"

Pin all five remaining tag refs to match the SHAs already used in the
workflow files:
- actions/setup-go@v6 → v6.4.0 SHA (×2)
- actions/cache/restore@v4 → v4.3.0 SHA
- actions/cache/save@v4 → v4.3.0 SHA
- actions/upload-artifact@v7 → v7.0.1 SHA

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
…affold-sha

feat(scaffold): pin workflow refs to release commit SHA
…ion-yml

fix(ci): pin third-party actions in root action.yml to commit SHAs
Switch GoReleaser changelog to `use: github` so release notes list
merged PRs rather than individual commits. The existing group and
filter rules still apply — they now match against PR titles.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Replace hardcoded @v0 with __FULLSEND_AI_REF__ placeholder for the
mint-token composite action in prioritize-scheduler and
repo-maintenance scaffold templates. At scaffold time the placeholder
is replaced with a SHA pin, matching the pattern already used by
reusable workflow refs.

Without this, downstream .fullsend repos that enable
sha_pinning_required reject mint-token@v0 as an unpinned action ref.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
… display

Remote skills loaded via URL were logged as "tree" during sandbox
bootstrap because filepath.Base() on the cache path returns the
last segment of GitHub's /tree/ URL convention. Add
resolveSkillDisplayName() that reads SKILL.md frontmatter via
skill.ParseFrontmatter() and uses the name field when available,
falling back to filepath.Base() for local skills or when no
frontmatter is present.

Note: pre-commit could not run in sandbox (shellcheck network
error). go vet and go test passed.

Closes fullsend-ai#2632
The GitHub compare API caps at 300 files with no pagination support.
Add a check after fetching merge group files: if the count is >= 300,
assume possible truncation and run tests as a precaution, matching
the existing fail-open pattern for API errors.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The pinact config and pre-commit hook only covered .github/workflows/,
.github/actions/, and the scaffold workflow directory. The root
action.yml was never scanned, which allowed unpinned tag refs to slip
through in PR fullsend-ai#2508 (fixed by fullsend-ai#2621).

Add action.yml to both .pinact.yaml file patterns and the pre-commit
hook's file regex so unpinned refs are caught automatically.

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The repo contains .github/workflows/fullsend.yaml which was not matched
by the *.yml-only patterns. Add *.yaml patterns for both the root and
scaffold workflow directories so pinact scans all workflow files
regardless of extension.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
…-display-name

fix(fullsend-ai#2632): use SKILL.md frontmatter name for remote skill display
…-root-action-yml

fix(ci): include root action.yml in pinact and pre-commit checks
Update scaffold_test.go assertions to expect
mint-token@__FULLSEND_AI_REF__ instead of mint-token@v0, matching
the template change.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
…en-scaffold

fix(scaffold): use __FULLSEND_AI_REF__ for mint-token action refs
Scope automerge to patch bumps and pin updates via packageRules.
Combined with repo-level allow_auto_merge and the ruleset bypass
for renovate-fullsend, these low-risk updates merge automatically
after CI passes.

Closes fullsend-ai#2506

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Hector Martinez <hemartin@redhat.com>
…te-automerge

feat(renovate): enable automerge for low-risk PRs
Add a dedicated macOS runner in lint.yml to exercise darwin-specific
sandbox behavior. Add TestUploadDir_SuppressesAppleDoubleInTarball on
darwin: verifies COPYFILE_DISABLE=1 prevents ._* files in tarballs using
python3 tarfile inspection, with a negative control to confirm xattr
application actually triggers AppleDouble generation without the flag.

Signed-off-by: Hector Martinez <hemartin@redhat.com>
…test

ci(sandbox): add macOS runner for darwin-specific tar behavior test
…ools

feat(scaffold): auto-detect and install pre-commit tool dependencies
…r-based-changelog

fix(ci): use GitHub PR titles for release notes
…path-filter

fix(ci): skip e2e/functional tests in merge queue for irrelevant paths
Documents the L2 additive merge design: per-repo .pre-commit-tools.yaml
at repo root extends upstream/org defaults. Covers three-layer
resolution order, merge semantics (extend/override/exclude), and
base-branch-only reads for supply-chain security.

Relates: fullsend-ai#1270

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
…erge

Refactor resolve() to accept a parsed dict instead of a file path.
Add merge_registries() that merges a per-repo .pre-commit-tools.yaml
with upstream/org defaults: new entries extend, matching (repo, hook_id)
entries override, and exclude: true suppresses.

Add --local-registry CLI arg. Caller scripts extract the base-branch
registry via git show (not the PR head) for supply-chain safety and
pass it to the resolver.

Also fixes fullsend-ai#1270 P1: add uv match_entry so hooks with entry "uv run ..."
are recognized alongside the existing "uvx" match.

Closes: fullsend-ai#1270

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
Add "Customizing Pre-commit Tool Dependencies" section to the
customizing-agents guide. Covers three-layer resolution, examples for
adding and suppressing entries, and the base-branch security model.

Add unit tests for merge_registries() and resolve() covering extend,
override, exclude, dedup, uv/uvx match, malformed input, and
end-to-end merged resolution.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 25, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:01 PM UTC · Completed 4:04 PM UTC
Commit: db171e5 · View workflow run →

@guyoron1 guyoron1 closed this Jun 25, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Review skipped — this PR is already closed.

The /fs-review command only reviews open pull requests.

Posted by fullsend post-review check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants