feat(scaffold): per-repo pre-commit tools registry (L2 additive merge) - #81
Closed
guyoron1 wants to merge 541 commits into
Closed
feat(scaffold): per-repo pre-commit tools registry (L2 additive merge)#81guyoron1 wants to merge 541 commits into
guyoron1 wants to merge 541 commits into
Conversation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Benjamin Kapner <bkapner@redhat.com>
…egrity docs(security): add audit log integrity to threat model
…egrity docs(security): add audit log integrity to threat model
Add a functional test framework for agent pipelines using agent-eval-harness lifecycle hooks. The harness drives case iteration with before_each/after_each hooks for ephemeral repo management, while fullsend runs inside openshell sandboxes. Key components: - eval/scripts/setup-fixture.sh: before_each hook creates ephemeral GitHub repos and fixtures (issues/PRs) from input.yaml - eval/scripts/run-fullsend.sh: CLI runner invokes fullsend run - eval/scripts/capture-fixture.sh: after_each hook snapshots fixture state for judges - eval/scripts/teardown-fixture.sh: after_each hook deletes repos - eval/run-functional.sh: orchestrator calling workspace.py, execute.py, and score.py with behavioral threshold checks - eval/triage/: first eval suite with LLM judge and label checks Also includes CI workflow, behavioral thresholds (max_turns, max_cost_usd), metrics capture from Claude Code stream events, ADRs, and documentation. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Add a functional test framework for agent pipelines using agent-eval-harness lifecycle hooks. The harness drives case iteration with before_each/after_each hooks for ephemeral repo management, while fullsend runs inside openshell sandboxes. Key components: - eval/scripts/setup-fixture.sh: before_each hook creates ephemeral GitHub repos and fixtures (issues/PRs) from input.yaml - eval/scripts/run-fullsend.sh: CLI runner invokes fullsend run - eval/scripts/capture-fixture.sh: after_each hook snapshots fixture state for judges - eval/scripts/teardown-fixture.sh: after_each hook deletes repos - eval/run-functional.sh: orchestrator calling workspace.py, execute.py, and score.py with behavioral threshold checks - eval/triage/: first eval suite with LLM judge and label checks Also includes CI workflow, behavioral thresholds (max_turns, max_cost_usd), metrics capture from Claude Code stream events, ADRs, and documentation. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
- Missing metrics.json is now a FAIL (not a warning), ensuring behavioral thresholds cannot be silently bypassed when the agent crashes or fullsend run fails. - Validate that jq output is numeric before threshold comparison, preventing null/malformed values from silently passing as 0. - Add shellcheck SC2317 disable directives for trap handler commands that shellcheck incorrectly flags as unreachable. Signed-off-by: Ralph Bean <rbean@redhat.com> Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
- Missing metrics.json is now a FAIL (not a warning), ensuring behavioral thresholds cannot be silently bypassed when the agent crashes or fullsend run fails. - Validate that jq output is numeric before threshold comparison, preventing null/malformed values from silently passing as 0. - Add shellcheck SC2317 disable directives for trap handler commands that shellcheck incorrectly flags as unreachable. Signed-off-by: Ralph Bean <rbean@redhat.com> Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
The harness was referenced twice: once as a git submodule and once as a pip install from the same git URL. Install from the already-checked-out submodule so the fork URL only appears in .gitmodules. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
The harness was referenced twice: once as a git submodule and once as a pip install from the same git URL. Install from the already-checked-out submodule so the fork URL only appears in .gitmodules. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Move max_turns and max_cost_usd checks from custom shell code in run-functional.sh into deterministic check judges in eval.yaml. The harness's score.py now enforces these via min_pass_rate: 1.0 thresholds. Extract the pre-flight annotation validation into a standalone eval/lint-cases.sh linter, wired up as `make lint-eval-cases` and included in `make test`. This runs cheaply without executing agents. Net effect: ~90 lines removed from run-functional.sh. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Move max_turns and max_cost_usd checks from custom shell code in run-functional.sh into deterministic check judges in eval.yaml. The harness's score.py now enforces these via min_pass_rate: 1.0 thresholds. Extract the pre-flight annotation validation into a standalone eval/lint-cases.sh linter, wired up as `make lint-eval-cases` and included in `make test`. This runs cheaply without executing agents. Net effect: ~90 lines removed from run-functional.sh. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Extend lint-cases.sh to verify that eval.yaml declares max_turns and max_cost judges, not just that annotations.yaml declares the thresholds. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Extend lint-cases.sh to verify that eval.yaml declares max_turns and max_cost judges, not just that annotations.yaml declares the thresholds. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
The CLI runner receives {output_dir} which is workspace/output. Writing
metrics.json to $OUTPUT_DIR/output/ created a double-nested path that
score.py couldn't find. Write to $OUTPUT_DIR/metrics.json instead so
the file appears at the expected output/metrics.json key.
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The CLI runner receives {output_dir} which is workspace/output. Writing
metrics.json to $OUTPUT_DIR/output/ created a double-nested path that
score.py couldn't find. Write to $OUTPUT_DIR/metrics.json instead so
the file appears at the expected output/metrics.json key.
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
The triage agent consistently takes ~23 turns on this case. The previous threshold of 15 was too tight. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
The triage agent consistently takes ~23 turns on this case. The previous threshold of 15 was too tight. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Add three new triage eval cases covering the remaining major outcomes: - 002-needs-info-vague-crash: vague issue with no repro steps, expects action "insufficient" and needs-info label - 003-feature-request: clear feature request, expects action "sufficient" with category "feature" and triaged+feature labels (not ready-to-code) - 004-duplicate-issue: issue duplicating a seed issue, expects action "duplicate" with duplicate label Supporting changes: - setup-fixture.sh: support seed_issues in input.yaml for pre-populating issues before the main fixture (needed by duplicate test) - eval.yaml: add forbidden_labels judge to verify wrong labels are NOT applied (needs-info must not get ready-to-code, etc.) - lint-cases.sh: check for forbidden_labels judge presence Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Add three new triage eval cases covering the remaining major outcomes: - 002-needs-info-vague-crash: vague issue with no repro steps, expects action "insufficient" and needs-info label - 003-feature-request: clear feature request, expects action "sufficient" with category "feature" and triaged+feature labels (not ready-to-code) - 004-duplicate-issue: issue duplicating a seed issue, expects action "duplicate" with duplicate label Supporting changes: - setup-fixture.sh: support seed_issues in input.yaml for pre-populating issues before the main fixture (needed by duplicate test) - eval.yaml: add forbidden_labels judge to verify wrong labels are NOT applied (needs-info must not get ready-to-code, etc.) - lint-cases.sh: check for forbidden_labels judge presence Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Cases use isolated ephemeral repos with UUID suffixes, so there's no shared state. Sequential execution took ~15 min of agent time across 4 cases; parallel should bring wall-clock down to ~6 min. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Cases use isolated ephemeral repos with UUID suffixes, so there's no shared state. Sequential execution took ~15 min of agent time across 4 cases; parallel should bring wall-clock down to ~6 min. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0045 was taken on main by forge-portable-harness-schema while this branch was out of date. Renumber both branch ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0045 was taken on main by forge-portable-harness-schema while this branch was out of date. Renumber both branch ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0046-host-side-api-server-design was merged to main while this branch was open, creating a duplicate number. Renumber 0046-functional-tests-for-agent-pipelines to 0048 and update all cross-references. Also fix stale heading numbers in ADRs 0047 and 0048. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
ADR 0046-host-side-api-server-design was merged to main while this branch was open, creating a duplicate number. Renumber 0046-functional-tests-for-agent-pipelines to 0048 and update all cross-references. Also fix stale heading numbers in ADRs 0047 and 0048. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
- Write partial metrics.json on agent failure so downstream judges can inspect behavioral data even when the agent errors out - Add CLOUD_ML_REGION to runner.env in eval.yaml (was only in execution.env) - Let teardown-fixture.sh fail visibly instead of swallowing errors with 2>/dev/null || true — the harness on_failure: continue already handles non-fatal teardown - Distinguish infrastructure failures from agent failures in run-functional.sh by checking for case output before proceeding to scoring - Fix stale reference to eval/fullsend-runner.sh in ADR 0047 (actual path is eval/scripts/run-fullsend.sh) Signed-off-by: Ralph Bean <rbean@redhat.com> Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
- Write partial metrics.json on agent failure so downstream judges can inspect behavioral data even when the agent errors out - Add CLOUD_ML_REGION to runner.env in eval.yaml (was only in execution.env) - Let teardown-fixture.sh fail visibly instead of swallowing errors with 2>/dev/null || true — the harness on_failure: continue already handles non-fatal teardown - Distinguish infrastructure failures from agent failures in run-functional.sh by checking for case output before proceeding to scoring - Fix stale reference to eval/fullsend-runner.sh in ADR 0047 (actual path is eval/scripts/run-fullsend.sh) Signed-off-by: Ralph Bean <rbean@redhat.com> Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Main acquired ADR numbers 0047-0050 while this branch was open. Renumber the PR's ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
When the CLI scaffolds workflow files during `fullsend github setup`, pin the `uses:` directive and `fullsend_ai_ref` parameter to the binary's build-time commit SHA instead of the mutable `@v0` tag. Dev builds fall back to `@v0`. Closes fullsend-ai#1933 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
…pr-commits fix(ci): lint individual commits on PRs to catch invalid prefixes early
…levant The merge_group trigger had no path filter and the relevance-check step only ran for pull_request_target events, so every PR entering the merge queue ran the full e2e and functional test suites regardless of what files changed. Extend the relevance check to also run on merge_group events, using the compare API with the merge group base/head SHAs to determine changed files. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
…mmit SHAs PR fullsend-ai#2508 pinned actions in .github/workflows/ and .github/actions/setup-gcp/ but missed the root action.yml composite action. Repos with strict SHA-pinning policies (e.g. openkaiden/kaiden) reject the unpinned tag refs, failing the Triage job with: "actions/setup-go@v6, actions/cache/restore@v4, and actions/upload-artifact@v7 are not allowed because all actions must be pinned to a full-length commit SHA" Pin all five remaining tag refs to match the SHAs already used in the workflow files: - actions/setup-go@v6 → v6.4.0 SHA (×2) - actions/cache/restore@v4 → v4.3.0 SHA - actions/cache/save@v4 → v4.3.0 SHA - actions/upload-artifact@v7 → v7.0.1 SHA Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
…affold-sha feat(scaffold): pin workflow refs to release commit SHA
…ion-yml fix(ci): pin third-party actions in root action.yml to commit SHAs
Switch GoReleaser changelog to `use: github` so release notes list merged PRs rather than individual commits. The existing group and filter rules still apply — they now match against PR titles. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
Replace hardcoded @v0 with __FULLSEND_AI_REF__ placeholder for the mint-token composite action in prioritize-scheduler and repo-maintenance scaffold templates. At scaffold time the placeholder is replaced with a SHA pin, matching the pattern already used by reusable workflow refs. Without this, downstream .fullsend repos that enable sha_pinning_required reject mint-token@v0 as an unpinned action ref. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
… display Remote skills loaded via URL were logged as "tree" during sandbox bootstrap because filepath.Base() on the cache path returns the last segment of GitHub's /tree/ URL convention. Add resolveSkillDisplayName() that reads SKILL.md frontmatter via skill.ParseFrontmatter() and uses the name field when available, falling back to filepath.Base() for local skills or when no frontmatter is present. Note: pre-commit could not run in sandbox (shellcheck network error). go vet and go test passed. Closes fullsend-ai#2632
The GitHub compare API caps at 300 files with no pagination support. Add a check after fetching merge group files: if the count is >= 300, assume possible truncation and run tests as a precaution, matching the existing fail-open pattern for API errors. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
The pinact config and pre-commit hook only covered .github/workflows/, .github/actions/, and the scaffold workflow directory. The root action.yml was never scanned, which allowed unpinned tag refs to slip through in PR fullsend-ai#2508 (fixed by fullsend-ai#2621). Add action.yml to both .pinact.yaml file patterns and the pre-commit hook's file regex so unpinned refs are caught automatically. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
The repo contains .github/workflows/fullsend.yaml which was not matched by the *.yml-only patterns. Add *.yaml patterns for both the root and scaffold workflow directories so pinact scans all workflow files regardless of extension. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
…-display-name fix(fullsend-ai#2632): use SKILL.md frontmatter name for remote skill display
…-root-action-yml fix(ci): include root action.yml in pinact and pre-commit checks
Update scaffold_test.go assertions to expect mint-token@__FULLSEND_AI_REF__ instead of mint-token@v0, matching the template change. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
…en-scaffold fix(scaffold): use __FULLSEND_AI_REF__ for mint-token action refs
Scope automerge to patch bumps and pin updates via packageRules. Combined with repo-level allow_auto_merge and the ruleset bypass for renovate-fullsend, these low-risk updates merge automatically after CI passes. Closes fullsend-ai#2506 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>
…te-automerge feat(renovate): enable automerge for low-risk PRs
Add a dedicated macOS runner in lint.yml to exercise darwin-specific sandbox behavior. Add TestUploadDir_SuppressesAppleDoubleInTarball on darwin: verifies COPYFILE_DISABLE=1 prevents ._* files in tarballs using python3 tarfile inspection, with a negative control to confirm xattr application actually triggers AppleDouble generation without the flag. Signed-off-by: Hector Martinez <hemartin@redhat.com>
…test ci(sandbox): add macOS runner for darwin-specific tar behavior test
…ools feat(scaffold): auto-detect and install pre-commit tool dependencies
…r-based-changelog fix(ci): use GitHub PR titles for release notes
…path-filter fix(ci): skip e2e/functional tests in merge queue for irrelevant paths
Documents the L2 additive merge design: per-repo .pre-commit-tools.yaml at repo root extends upstream/org defaults. Covers three-layer resolution order, merge semantics (extend/override/exclude), and base-branch-only reads for supply-chain security. Relates: fullsend-ai#1270 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
…erge Refactor resolve() to accept a parsed dict instead of a file path. Add merge_registries() that merges a per-repo .pre-commit-tools.yaml with upstream/org defaults: new entries extend, matching (repo, hook_id) entries override, and exclude: true suppresses. Add --local-registry CLI arg. Caller scripts extract the base-branch registry via git show (not the PR head) for supply-chain safety and pass it to the resolver. Also fixes fullsend-ai#1270 P1: add uv match_entry so hooks with entry "uv run ..." are recognized alongside the existing "uvx" match. Closes: fullsend-ai#1270 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
Add "Customizing Pre-commit Tool Dependencies" section to the customizing-agents guide. Covers three-layer resolution, examples for adding and suppressing entries, and the base-branch security model. Add unit tests for merge_registries() and resolve() covering extend, override, exclude, dedup, uv/uvx match, malformed input, and end-to-end merged resolution. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 4:01 PM UTC · Completed 4:04 PM UTC |
|
Review skipped — this PR is already closed. The Posted by fullsend post-review check |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror of fullsend-ai#2663 for QualityFlow integration testing.
Original PR Summary
Extends the pre-commit tools registry with per-repo additive merge (L2). Target repos can place
.pre-commit-tools.yamlat repo root to extend the upstream/org registry. Includes supply-chain security (reads from base branch, not PR head),uvmatch fix, and 13 unit tests.QualityFlow
This PR will be used to demonstrate QualityFlow's automated test planning and generation pipeline.