Skip to content

test: add provider compatibility evals#105

Merged
RNT56 merged 1 commit into
mainfrom
task/P15-T13
Jul 12, 2026
Merged

test: add provider compatibility evals#105
RNT56 merged 1 commit into
mainfrom
task/P15-T13

Conversation

@RNT56

@RNT56 RNT56 commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • add hermetic golden transcripts for generic OpenAI-compatible, reasoning, structured-output, and OpenRouter request/response contracts
  • exercise Anthropic native web-search rendering and prove hostile raw server-tool results are dropped before loop re-entry
  • prove the client fallback flags suspicious content, fences it as untrusted data, and escapes embedded fence markers
  • fail if either token-cap key duplication or fixture-secret leakage reaches the transcripts

Verification

  • go build ./...
    go vet ./...

WARNING: golangci-lint not installed — skipping local lint (CI still enforces it).
Install the pinned version for local/CI parity (see .github/workflows/ci.yml).
go test ./...
ok nilcore/cmd/nilcore (cached)
ok nilcore/cmd/tools/nilcore-browser (cached)
ok nilcore/cmd/tools/nilcore-desktop (cached)
ok nilcore/cmd/tools/nilcore-desktop-darwin (cached)
ok nilcore/eval (cached)
ok nilcore/eval/browse (cached)
ok nilcore/eval/desktop (cached)
ok nilcore/eval/provider-compat 0.733s
ok nilcore/eval/self (cached)
ok nilcore/internal/advisor (cached)
ok nilcore/internal/agent (cached)
ok nilcore/internal/agent/bus (cached)
ok nilcore/internal/agenticflows (cached)
ok nilcore/internal/artifact (cached)
ok nilcore/internal/artifact/evverify (cached)
ok nilcore/internal/artifact/packs (cached)
ok nilcore/internal/artifact/packs/audit (cached)
ok nilcore/internal/artifact/packs/benchmark (cached)
ok nilcore/internal/artifact/packs/code (cached)
ok nilcore/internal/artifact/packs/finance (cached)
ok nilcore/internal/artifact/packs/software (cached)
ok nilcore/internal/artifact/packs/ui (cached)
ok nilcore/internal/artifact/packs/web (cached)
ok nilcore/internal/artifact/schema (cached)
ok nilcore/internal/ask (cached)
ok nilcore/internal/autosrc (cached)
ok nilcore/internal/backend (cached)
ok nilcore/internal/blastbudget (cached)
ok nilcore/internal/browseragent (cached)
? nilcore/internal/browseragent/plan [no test files]
ok nilcore/internal/browsersession (cached)
ok nilcore/internal/browserwire (cached)
ok nilcore/internal/budget (cached)
ok nilcore/internal/capability (cached)
ok nilcore/internal/capguard (cached)
ok nilcore/internal/cdp (cached)
ok nilcore/internal/channel (cached)
ok nilcore/internal/channel/slack (cached)
ok nilcore/internal/channel/telegram (cached)
ok nilcore/internal/codeintel/ast (cached)
ok nilcore/internal/codeintel/graph (cached)
ok nilcore/internal/codeintel/impact (cached)
ok nilcore/internal/codeintel/live (cached)
ok nilcore/internal/codeintel/lsp (cached)
ok nilcore/internal/codeintel/repomap (cached)
ok nilcore/internal/codeintel/retrieve (cached)
ok nilcore/internal/codeintel/semantic (cached)
ok nilcore/internal/cron (cached)
ok nilcore/internal/desktop (cached)
ok nilcore/internal/desktopagent (cached)
ok nilcore/internal/desktopsession (cached)
ok nilcore/internal/desktopwire (cached)
ok nilcore/internal/egressprofile (cached)
ok nilcore/internal/embed (cached)
ok nilcore/internal/emit (cached)
ok nilcore/internal/eventlog (cached)
ok nilcore/internal/experience (cached)
ok nilcore/internal/flywheel/distiller (cached)
ok nilcore/internal/flywheel/loop (cached)
ok nilcore/internal/flywheel/measure (cached)
ok nilcore/internal/flywheel/selfeval (cached)
ok nilcore/internal/forge (cached)
ok nilcore/internal/graapprove (cached)
ok nilcore/internal/guard (cached)
ok nilcore/internal/inbox (cached)
ok nilcore/internal/inspect (cached)
ok nilcore/internal/integrate (cached)
ok nilcore/internal/kernel (cached)
ok nilcore/internal/loopctl (cached)
ok nilcore/internal/maint (cached)
ok nilcore/internal/mcp (cached)
ok nilcore/internal/memory (cached)
ok nilcore/internal/memory/lessons (cached)
ok nilcore/internal/meter (cached)
ok nilcore/internal/model (cached)
ok nilcore/internal/objective (cached)
ok nilcore/internal/onboard (cached)
ok nilcore/internal/paths (cached)
ok nilcore/internal/planner (cached)
ok nilcore/internal/policy (cached)
ok nilcore/internal/pool (cached)
ok nilcore/internal/project (cached)
ok nilcore/internal/provider (cached)
ok nilcore/internal/registry (cached)
ok nilcore/internal/report (cached)
ok nilcore/internal/report/render (cached)
ok nilcore/internal/requeue (cached)
ok nilcore/internal/roster (cached)
ok nilcore/internal/route (cached)
ok nilcore/internal/router (cached)
ok nilcore/internal/sandbox (cached)
ok nilcore/internal/scheduler (cached)
ok nilcore/internal/scmhook (cached)
ok nilcore/internal/secrets (cached)
ok nilcore/internal/selfimprove (cached)
ok nilcore/internal/server (cached)
ok nilcore/internal/session (cached)
ok nilcore/internal/skills (cached)
ok nilcore/internal/som (cached)
ok nilcore/internal/spawn (cached)
ok nilcore/internal/steering (cached)
ok nilcore/internal/store (cached)
ok nilcore/internal/strongcap (cached)
ok nilcore/internal/summarize (cached)
ok nilcore/internal/super (cached)
ok nilcore/internal/swarm (cached)
ok nilcore/internal/swarm/board (cached)
ok nilcore/internal/swarm/preset (cached)
ok nilcore/internal/termui (cached)
ok nilcore/internal/tools (cached)
ok nilcore/internal/trace (cached)
ok nilcore/internal/trigger (cached)
ok nilcore/internal/trust (cached)
ok nilcore/internal/verb (cached)
ok nilcore/internal/verify (cached)
ok nilcore/internal/verify/selfacc (cached)
ok nilcore/internal/verify/vcache (cached)
ok nilcore/internal/wake (cached)
ok nilcore/internal/worktree (cached)
ok nilcore/internal/worktreefs (cached)
ok nilcore/test/smoke (cached)

  • go test -race
    ./internal/session/...
    ./internal/swarm/...
    ./internal/agent/...
    ./internal/super/...
    ./internal/scheduler/...
    ./internal/pool/...
    ./internal/channel/...
    ./internal/blastbudget/...
    ./internal/model/...
    ./internal/meter/...
    ./internal/mcp/...
    ./internal/trace/...
    ./internal/trust/...
    ./internal/graapprove/...
    ok nilcore/internal/session (cached)
    ok nilcore/internal/swarm (cached)
    ok nilcore/internal/swarm/board (cached)
    ok nilcore/internal/swarm/preset (cached)
    ok nilcore/internal/agent (cached)
    ok nilcore/internal/agent/bus (cached)
    ok nilcore/internal/super (cached)
    ok nilcore/internal/scheduler (cached)
    ok nilcore/internal/pool (cached)
    ok nilcore/internal/channel (cached)
    ok nilcore/internal/channel/slack (cached)
    ok nilcore/internal/channel/telegram (cached)
    ok nilcore/internal/blastbudget (cached)
    ok nilcore/internal/model (cached)
    ok nilcore/internal/meter (cached)
    ok nilcore/internal/mcp (cached)
    ok nilcore/internal/trace (cached)
    ok nilcore/internal/trust (cached)
    ok nilcore/internal/graapprove (cached)
  • go build -tags tui ./...
    && go vet -tags tui ./cmd/nilcore ./internal/swarm/board ./internal/trace
    && go test -tags tui ./cmd/nilcore ./internal/swarm/board ./internal/trace
    ok nilcore/cmd/nilcore (cached)
    ok nilcore/internal/swarm/board (cached)
    ok nilcore/internal/trace (cached)
  • 0 issues.
  • No vulnerabilities found.
  • ok nilcore/eval/provider-compat 1.698s

Closes P15-T13.

Add hermetic golden transcripts for generic compatibility, reasoning models, structured output, OpenRouter extras, and both native/client web-search safety boundaries.\n\nProve the token-cap replacement, secret redaction, native drop-on-decode behavior, and client-side injection fencing without live provider access.
@RNT56 RNT56 marked this pull request as ready for review July 12, 2026 02:48
@RNT56 RNT56 merged commit a68c22f into main Jul 12, 2026
9 checks passed
@RNT56 RNT56 deleted the task/P15-T13 branch July 12, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant