Skip to content

feat(agent-bench): add fixture-free tau3, DABStep, and FinResearch adapters #449

Description

@drewstone

Problem

agent-lab R258/R259 shows the external finance/τ lane is not ready for policy-scored research cells:

  • tau3-banking: catalog only, no runnable adapter, 0 readiness cells.
  • DABStep: adapter exists, but fixture-free official checkout inspection found 5/6 required paths and 7/7 resource files present; missing dataset.csv leaves 0 official task rows.
  • FinResearchBench: catalog only, no official dataset/judge path in agent-bench.

Why this belongs in agent-runtime / agent-bench

agent-lab should consume benchmark adapters through @tangle-network/agent-bench. It should not own benchmark-specific loaders, official data acquisition, or grader semantics.

Acceptance criteria

  • tau3-banking adapter exists in agent-bench and fails loudly without official data/simulator/state scorer.
  • tau3-banking no-spend load/preflight can return at least one official task with simulator/version metadata pinned.
  • DABStep fixture-free path documents where dataset.csv comes from or changes the adapter to the official available task source without fabricating scores.
  • DABStep preflight without DABSTEP_FIXTURES loads at least one official task and exposes files/ as AgentProfile resources.
  • FinResearchBench adapter owns official dataset loading and the logic-tree judge path, with judge model/cost captured or explicit fail-loud status.
  • All three adapters preserve the rule: fixtures test plumbing only; fixtures never count as benchmark scores.

Evidence from agent-lab

Research unblock

Once these adapters/data paths are real, agent-lab can run no-spend readiness, then calibrated one-cell smokes, then policy-vs-baseline cells for M3 external validity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions