Grader is blind to non-DB requirements (task 47) → one field on StructuredUserInstructions + paired re-scoring

[borisdev]

Summary

τ³'s grade is a low-dimensional projection of the trajectory (final DB state + required substrings). Any user requirement that doesn't touch the database is invisible to it. Concrete instance: airline task 47 scores PASS while the agent violates an explicit, stated requirement ("you don't want to be transferred to another agent") that lives only in the free-text task_instructions and never enters the grader's structured criteria.

The fix is a paired re-scoring experiment: make the same task's requirements typed and checkable, preserve the user-simulator prose byte-for-byte, and score the same recorded trajectory with two graders — τ³'s existing grader and a new structured-requirements grader. Any verdict difference is attributable to what the grader can represent, not to a changed conversation.

Evidence (task 47, verified)

• The requirement is in data/tau2/domains/airline/tasks.json (task 47, user_scenario.instructions.task_instructions): "you don't want to be transferred to another agent."
• The grade is DB-only: reward_basis = [DB, COMMUNICATE], communicate_info = [] → the score reduces to "did the DB change?"
• What happened: the agent correctly refused the (uncovered) refund, then fired transfer_to_human_agents anyway. DB unchanged → τ³ scores it PASS.
• Trace + provenance: poc/CASE_STUDY.md, poc/traces/task_47.md, machine-verified by poc/verify_findings.py.

Root cause

The requirement lives in a free-text task_instructions string, so it is neither a structured gradeable criterion nor visible to the grader. This is the "revealed but missed" failure pattern (Phase 1).

Proposed fix — Phase 1: one field on StructuredUserInstructions + paired re-scoring

No StructuredUserInstructionsV2 wrapper. We add one optional field directly to τ³'s own StructuredUserInstructions — backward compatible (default None; every existing task still loads), prose untouched:

class StructuredUserInstructions(BaseModel):   # τ³'s own type
    domain; reason_for_call; known_info; unknown_info
    task_instructions: str                                                   # unchanged — byte-for-byte
+   user_preflight_requirements: UserPreflightRequirements | None = None     # NEW — grader-visible, typed

UserPreflightRequirements (and ActionPrecondition) live in src/tau2/data_model/preflight_requirements.py. Task 47's requirements, with the correct semantics + provenance:

UserPreflightRequirements(
    action_preconditions=[ActionPrecondition(
        id="task47.no_unwanted_transfer", action="transfer_to_human_agents",
        rule="must not transfer — the user explicitly refused",
        source_field="task_instructions",
        source_quote="You don't want to be transferred to another agent.")])

Every requirement carries a source_quote — making an existing stated rule gradeable, not inventing one.

Plan (Phase 1)

• preflight_requirements.py: UserPreflightRequirements, ActionPrecondition, verify_provenance
• Add optional user_preflight_requirements field to StructuredUserInstructions (prose rendering unchanged)
• Task-47 fixture (real StructuredUserInstructions with the field set; task_instructions preserved by construction)
• PreflightRequirementsEvaluator reads instructions.user_preflight_requirements
• poc/compare_graders.py — paired re-scoring → JSON + Markdown; task 47: τ³ PASS, structured FAIL
• Measurements: verdict-flip rate, score delta by action; (multi-model, later) model-ranking stability

Deferred — later phases

• Agent belief tracking — does the agent resolve each requirement before acting? A belief-state layer; the paired re-scoring experiment needs only the grader's view.
• PreflightPolicyPack — SME-authored reusable per-action rules ("should-exist but omitted", Phase 2).

Guardrails

• task_instructions byte-for-byte unchanged; do not regenerate simulator prose (isolate the grader-visibility variable).
• Do not hand user_preflight_requirements to the agent (no leakage).
• Every rule traces to a source_quote verified verbatim in the real task; reject invented rules.
• A prohibition is an explicit refusal, not merely "the action was not requested". Conditional (world-state) authorizations are future work.
• The new field is a local addition to vendored τ³ code (tasks.py) — pull upstream that file carefully.