test: contract tests against real hermes-agent (CI + local staging clone) - #2
Merged
Merged
Conversation
Keeps the kit from silently drifting from upstream. tests/test_hermes_contract.py
registers a kit-built tool into the REAL hermes registry and asserts:
- registry.get_definitions() yields a function with populated parameters (the
empty-{} failure mode, checked against the actual conversion);
- registry.dispatch() runs the handler and the kit's validation in-band;
- register_all's calls bind to the real PluginContext.register_tool signature.
Skip-guarded, so it's green standalone. Runs against upstream via a staging
clone: 'make test-contract' (clones NousResearch/hermes-agent into .hermes-agent)
and a CI 'hermes-contract' job that checks out upstream main. PyYAML added as a
dev-only dep (runtime stays dependency-free).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers "does the kit have contract tests against actual hermes-agent source so it stays relevant?" — now yes.
What
tests/test_hermes_contract.pyregisters a kit-built@toolinto the real hermesregistryand asserts the kit's output still satisfies the runtime contract:registry.get_definitions()(the actual{**schema, "name": ...}conversion the model receives) yields a function with populatedparameters— the empty-{}failure mode this kit prevents, checked against real source.registry.dispatch()runs the handler and the kit's required-arg validation in-band.register_all's calls bind to the realinspect.signature(PluginContext.register_tool)— fails if upstream renames/removes a param the kit uses.How it runs against upstream (CI + local)
hermes-contractjob checks outNousResearch/hermes-agentinto.hermes-agentand runs the contract module against upstreammain.make test-contractclones hermes-agent into the same staging dir and runs it — identical to CI.make teststays green standalone.Verified locally against a fresh upstream clone: 3 contract tests pass. PyYAML is dev-only (runtime stays dependency-free).