Skip to content

ci: add pin-sync CI + regression guards for MCP install-script drift#63

Merged
geobio merged 2 commits into
mainfrom
ci/mcp-pin-sync-guard
Jul 22, 2026
Merged

ci: add pin-sync CI + regression guards for MCP install-script drift#63
geobio merged 2 commits into
mainfrom
ci/mcp-pin-sync-guard

Conversation

@geobio

@geobio geobio commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Credit to @Abhishek21g in #57

Why

The template→install-script pin drift that broke directory_tree (#34) shipped to Docker images because the repo has no CItests/test_mcp_config_sync.py existed and does catch version mismatches, but nothing ever ran it. A perfect sync test prevents nothing if it isn't wired up.

This closes that gap and hardens the tests, superseding the remainder of #57 (its version bumps already landed via 60ae87d; template↔script are fully in sync on main today, verified both directions).

Changes

  • .github/workflows/agent-environment-tests.yml — runs the pin-sync tests on every PR (and push to main) that touches the template, install script, or tests. This is the piece that actually prevents recurrence.
  • tests/test_filesystem_pin.py (from fix(agent-environment): sync MCP install script with template pins (#34) #57) — a minimum-CalVer floor for the filesystem pin. The sync test only enforces script == template; if both get bumped back to the broken @2025.11.25 the bug returns silently. This guard rejects anything below the upstream fix (2025.12.18). Cleaned up the two Greptile nits (redundant assertion removed, upstream issue ref aligned to #3110).
  • tests/test_mcp_config_sync.py — closed a reverse-drift blind spot: the docstring claimed to catch stale packages left in the script, but the code only checked template→script. Now also asserts script − template is empty (bloat / stale removals). Verified it fires with a negative test.
  • README.md (from fix(agent-environment): sync MCP install script with template pins (#34) #57) — documents the pin policy.

Verification

cd services/agent-environment && python3 -m pytest tests/ -v   # 2 passed

Both new guards confirmed to fire on regression (stale-package injection; pin rolled back to 2025.11.25).

Follow-up

#57 can be closed — its install-script sync already merged, and the useful remainder (test + docs) is carried here rebased and cleaned.

🤖 Generated with Claude Code

Greptile Summary

This PR wires up CI for the agent-environment service and hardens the existing pin-sync tests to close the gap that let the directory_tree regression (#34) reach Docker images undetected.

  • .github/workflows/agent-environment-tests.yml: New workflow runs pytest tests/ on every PR or push to main that touches the template, install script, tests, or the workflow file itself — path filters are tight and correct.
  • test_filesystem_pin.py: Adds a CalVer floor guard (>= 2025.12.18) so that even a perfectly-synced template+script pair can't silently regress to the broken @2025.11.25 pin.
  • test_mcp_config_sync.py: Closes the reverse-drift blind spot by also asserting script − template = ∅, preventing stale packages from accumulating in the install script after they're removed from the template.

Confidence Score: 5/5

Safe to merge — adds CI and test-only files with no changes to production code paths.

All three changed files are either a new GitHub Actions workflow or test code. The reverse-direction sync check is logically correct, and no production code is modified.

No files require special attention.

Important Files Changed

Filename Overview
.github/workflows/agent-environment-tests.yml New CI workflow that runs pin-sync tests on PRs and pushes to main touching the template, install script, or tests; path filters are correct and minimal.
services/agent-environment/tests/test_filesystem_pin.py New regression guard enforcing a minimum CalVer floor (2025.12.18) for the filesystem MCP pin; logic is correct and well-scoped.
services/agent-environment/tests/test_mcp_config_sync.py Adds a reverse-direction check (script − template) to catch stale packages left in the install script; set arithmetic and early-exit ordering are correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    PR[PR or push to main] -->|paths filter matches| TRIGGER[Workflow triggered]
    TRIGGER --> CHECKOUT[checkout v4]
    CHECKOUT --> PYSETUP[setup-python 3.12]
    PYSETUP --> INSTALL[pip install pytest]
    INSTALL --> RUN[pytest tests]
    RUN --> T1[test_mcp_config_sync]
    RUN --> T2[test_filesystem_pin]
    T1 --> FWD[Forward: template pkgs present in script]
    FWD -->|missing| FAIL1[AssertionError: missing packages]
    FWD -->|ok| REV[Reverse: script pkgs present in template]
    REV -->|extra| FAIL2[AssertionError: stale packages in script]
    REV -->|ok| PASS1[Sync OK]
    T2 --> READPIN[Read filesystem pin from template json]
    READPIN --> PARSEPIN[Parse CalVer tuple]
    PARSEPIN -->|below 2025.12.18| FAIL3[AssertionError: pin too old]
    PARSEPIN -->|meets minimum| PASS2[Pin OK]
Loading

Reviews (2): Last reviewed commit: "Update README.md" | Re-trigger Greptile

geobio and others added 2 commits July 21, 2026 16:00
The template->install-script pin drift that broke directory_tree (#34) shipped
because the sync test was never wired to run — the repo had no CI at all.

- Add .github/workflows/agent-environment-tests.yml to run the pin-sync tests
  on every PR touching the template, install script, or tests.
- Add tests/test_filesystem_pin.py: minimum-CalVer floor so the filesystem pin
  can't regress below the upstream fix (2025.12.18), which the sync test alone
  would miss when template and script are bumped down together. (from #57)
- Close the reverse-drift blind spot in test_package_sync: the docstring
  claimed to catch stale packages left in the script, but only checked the
  template->script direction. Now asserts script-template is empty too.
- Document the pin policy in services/agent-environment/README.md. (from #57)

Supersedes the remainder of #57 (its version bumps already landed via 60ae87d).

Co-Authored-By: Abhishek Enaguthi <enaguthiabhishek@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@geobio
geobio merged commit f401756 into main Jul 22, 2026
4 checks passed
@geobio
geobio deleted the ci/mcp-pin-sync-guard branch July 22, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant