tasks: propose an ops domain — five ops-altitude tasks (policy-grounded detection, governed drafting, evidence discipline)

[shyan-ai]

Summary

Following up on the call with Nimit, Jeff, and Ahmed — the tasks half of the setup I described, packaged per CONTRIBUTING.md and docs/task-authoring.md.

Five tasks under a proposed new ops domain. Where the existing domains measure retrieval and analysis Q&A, these measure the operations altitude — the seam right below the L3 roadmap:

• ops-l1-a — negative evidence: a fake account + a fake MSA tier; the correct answer is a grounded "does not exist" with the searches shown.
• ops-l1-b — deterministic cross-system revenue-at-risk rollup via the part hierarchy (L1 per the authoring guide's deterministic-join rule).
• ops-l2-a — SLA breach detection under per-tier MSA thresholds, with mandated exclusions (no-SLA Starter tickets; post-reference-date tickets).
• ops-l2-b — governed drafting: client-facing escalation drafts grounded in real tickets, bounded by the MSA, framed for human review.
• ops-l2-c — a decision brief where every claim carries a citation and silence in the data must be stated, not filled.

All ground truth computed from the dataset (reference date 2026-04-13); criteria are required/weighted per the format, ID-agnostic; canary embedded in every file; no data changes. Trajectories are semantic reference answers per the authoring guide — happy to swap in recorded run trajectories if preferred.

Type of contribution

• Task addition or update
• Dataset addition or update
• Agent result submission
• Documentation
• Bug fix / setup improvement
• Other

Validation

• make validate passes
• I ran at least one affected task, or explained why not below
• Dataset changes are synthetic and safe to publish
• New/changed task criteria avoid answer leakage
• Documentation is updated

All five tasks were run extensively against this dataset via my own A/B harness (multi-arm, multi-model — findings summarized in the PR comment). I have not run make run-task in this checkout or verified with Harbor; dataset.toml digests were computed locally over each task dir's contents, so please regenerate via harbor add if the publish flow needs canonical ones. No dataset changes in this PR (the two checked dataset/doc boxes are vacuously true: no data touched, no behavior-changing docs required).

Commands run

make validate   # Enterprise-Bench validation passed: 19 task(s), 19 dataset entry(s)
uv run ruff check .   # All checks passed

Notes for reviewers

• Review feedback applied (daed27b): all five instruction.md files now carry the standard submission protocol + ## Initial user message format.
• Open question 1: is ops the domain name you want, or should these fold into support/sales with new letters?
• Open question 2: the context-tier A/B harness + findings from reproducing the interface-vs-model result are packaged and ready as a second PR (summary in the PR comment thread; narrative: https://shyanming.substack.com/p/same-model-different-floor) — glad to shape it as a findings/ section, contrib/ dir, or condensed README addition.
• Happy to split, rename, or trim per review.

[jonmaaku-rev]

@shyan-ai

The instruction.md should include instruction on how the agent should submit the response:

This is a conversational task where you respond to a user message via an API endpoint.

The container exposes: POST http://localhost:8000/submit_agent_response

Request format:

{
  "assistant": "Your response text"
}

Response: {"status": "accepted", "task_finished": true}

Example curl:

curl -X POST http://localhost:8000/submit_agent_response \
  -H "Content-Type: application/json" \
  -d "{\"assistant\": \"Your response here\"}"

---

Initial user message

[shyan-ai]

@jonmaaku-rev — done: all five instruction.md files now open with the standard submission preamble (POST /submit_agent_response + example curl) and present the task query under ## Initial user message, matching the eng-* format. Pushed as daed27b; make validate passes (19 tasks / 19 entries).

---

On the findings (per the Slack thread — happy to shape this into whatever section fits):

We ran an interface A/B over this dataset's central claim — that retrieval architecture dominates model choice. Same model, same tasks, same judge. Arm A: protocol-realistic raw MCP surfaces (SOQL / Jira v3 / Drive v3). Arm B: a "governed context" tier — pre-joined views with policy thresholds applied at the data layer and a context fingerprint (records/documents/exclusions consumed) on every response.

Four findings, honestly caveated (n=1 per cell, two mid-difficulty tasks — directional, not statistical):

1. Efficiency effect reproduces, strongly. Governed tier: 3.1–4.8x faster, 2.7–3.5x cheaper, 3–5 agent turns vs 16–20. The turn counts expose the mechanism: the raw agent spends its loop reconstructing joins and filters the curated tier had precomputed.
2. Model-downgrade recovery. Governed + Haiku passed both tasks at $0.13 / 64s total; raw + Opus passed at $4.02 / 804s. Same accuracy on these tasks, ~31x cheaper, ~13x faster — operationally consistent with your "interface effect ≫ model-version effect" claim.
3. The unplanned one — and the most benchmark-relevant: the raw-arm agent caught a curation error in the governed tier. A mis-transcribed MSA threshold (which the task criteria shared) meant the curated arm "passed" on a wrong constant, while the raw arm — reading the binding document — applied the correct value and was initially failed by the wrong criteria. The lesson generalizes to any curated tier and to task criteria themselves: curated values need derivation checks against their cited sources, or curation becomes a new hallucination surface.
4. Provenance rides free on the governed arm only — judge-verifiable context fingerprints, the auditability axis the bench names but doesn't currently instrument.

The runnable pieces are packaged and ready as a second PR: the ops-context MCP server implementing the governed tier, the A/B harness, and the full findings doc (four rounds — the two above plus a linkage-isolation round and a skill-arm delivery-form round). Glad to format it as a findings/ section, a contrib/ dir, or fold a condensed version into the ops-domain README — whichever fits the repo's shape. Narrative version: https://shyanming.substack.com/p/same-model-different-floor