Concept from opencode:
To test LLM integrations reliably and cheaply, opencode built a custom HTTP VCR tool. It executes real requests on the first run and caches the payloads as JSON "cassettes", including strict automatic redaction to prevent secret leaks.
How decafclaw could implement this:
decafclaw's current eval suite either hits live LLMs (costly, slow, flaky) or relies on manual mocking that easily drifts from reality.
Proposed Implementation:
- Integrate
vcrpy (or build a lightweight equivalent) into the pytest suite for the llm client.
- Add a
make record-evals command to hit real models and commit sanitized cassettes.
- Allow
make test and make eval-tools to run deterministically and instantly in CI based on the recorded cassettes.
Concept from opencode:
To test LLM integrations reliably and cheaply,
opencodebuilt a custom HTTP VCR tool. It executes real requests on the first run and caches the payloads as JSON "cassettes", including strict automatic redaction to prevent secret leaks.How
decafclawcould implement this:decafclaw's current eval suite either hits live LLMs (costly, slow, flaky) or relies on manual mocking that easily drifts from reality.Proposed Implementation:
vcrpy(or build a lightweight equivalent) into thepytestsuite for thellmclient.make record-evalscommand to hit real models and commit sanitized cassettes.make testandmake eval-toolsto run deterministically and instantly in CI based on the recorded cassettes.