Skip to content

Add local agent trace DAG and deterministic evals - #219

Merged
NarwhalChen merged 2 commits into
mainfrom
kobe/convera-trace-eval-jd34fd
Aug 2, 2026
Merged

Add local agent trace DAG and deterministic evals#219
NarwhalChen merged 2 commits into
mainfrom
kobe/convera-trace-eval-jd34fd

Conversation

@NarwhalChen

Copy link
Copy Markdown
Collaborator

Summary

  • replace the diagnostic turn JSONL with a versioned, local-first Trace DAG covering Mission, Task, Run, Turn, Model, Tool, and Handoff spans
  • persist append-only, redacted P0 metadata with correlation IDs, causal links, crash-tail repair, idempotent writes, and whole-trace retention
  • add a deterministic local Eval CLI for graph integrity, canonical hierarchy, complete runtime evidence, process counts, token usage, and repeated tool calls
  • integrate Agent Host lifecycle and provider/tool streaming into production and the development bridge without making trace writes part of the execution success path
  • recognize the structured delegation and handoff provenance introduced by Add structured task delegation and handoff #218: delegated Task parentage, source-run causal edges, explicit Handoff spans, task depth, result receipts, and collaboration metrics

PR #218 compatibility

This PR remains independently mergeable into main. Its recorder reads the forward-compatible parentTaskId, collaboration, and outputMessageIds job shape only when those fields exist. Once #218 lands, delegated children attach beneath their source Task and ownership transfers produce an explicit Handoff span without another adapter change.

A temporary integration worktree overlaid this branch onto the exact #218 head (16420ad): TypeScript passed and 29 focused Trace, Agent Host, and structured-task tool tests passed.

Verification

  • pre-push Prettier check passed
  • ESLint passed with 10 existing warnings and no errors
  • 542 tests passed, 1 skipped
  • tsc --noEmit --skipLibCheck passed
  • built-in Trace Eval: 5/5 scenarios passed with mean score 1.0, including structured delegation and handoff
  • real Electron automation used an isolated profile and real codex-cli / gpt-5.6-sol authenticated through ChatGPT subscription
  • the real provider completed list_channels, read_channel, and send_message; the UI displayed TRACE_TOOL_OK
  • the resulting production JSONL contained 22 events across 8 spans and passed all 5 DAG checks with no orphan, cycle, hierarchy, or incomplete-runtime failures

Privacy and scope

Trace payloads contain identifiers, statuses, provider/model names, timing, and numeric usage metrics. They do not persist prompts, model text, tool arguments, tool output, raw error text, delegation briefs, or memory content. Trace failures are diagnostic-only and cannot fail an agent turn.

@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
foxyfox-docs Error Error Aug 2, 2026 12:01pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
foxyfox-website Skipped Skipped Aug 2, 2026 12:01pm

Request Review

@NarwhalChen
NarwhalChen force-pushed the kobe/convera-trace-eval-jd34fd branch from f52d770 to 4068419 Compare August 2, 2026 12:00
@vercel
vercel Bot temporarily deployed to Preview – foxyfox-website August 2, 2026 12:00 Inactive
@NarwhalChen
NarwhalChen merged commit fd9e32a into main Aug 2, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant