Skip to content

data: clarify operative-vs-baseline SLA values in the Growth/Enterprise MSA schedules - #30

Open
shyan-ai wants to merge 1 commit into
devrev:mainfrom
shyan-ai:fix/msa-operative-terms-convention
Open

data: clarify operative-vs-baseline SLA values in the Growth/Enterprise MSA schedules#30
shyan-ai wants to merge 1 commit into
devrev:mainfrom
shyan-ai:fix/msa-operative-terms-convention

Conversation

@shyan-ai

Copy link
Copy Markdown

Summary

Clarifies the meaning of the bold tier-specific values and parenthetical (std: …) values in the
Growth and Enterprise MSA schedules.

The documents currently show values such as:

  • Growth P1 response: 2 hours (std: 1 hr).
  • Enterprise P1 response: 30 minutes (std: 1 hr).

Because std: is not defined, a reader or agent can reasonably interpret the one-hour standard value
as the operative commitment. This PR makes the intended convention explicit:

Values shown in bold are the tier-specific commitments negotiated for and operative under the
agreement. Values marked (std: …) are Maple's standard published baseline values, included for
comparison only. Where the two differ, the bold tier-specific value governs.

Changes

The MSA source documents are packaged in artifacts/data.zip (internal_docs/), so the edits are made
inside the archive:

  • Add the operative-terms convention to internal_docs/msa_growth_tier.md and
    internal_docs/msa_enterprise_tier.md, immediately before §3.2 "Response & Resolution Targets".
  • Cross-reference the convention from the response/resolution (§3.2) and billing-dispute (§4.1)
    schedules, where the ambiguity affects benchmark retrieval most directly.
  • Only these two files change inside the archive; the other 77 dataset files are byte-identical, and
    the CANARY.md sidecar is preserved.

Intentionally unchanged

  • Benchmark tasks, prompts, grading criteria, and gold answers.
  • Ticket/account fixture data and account tiers.
  • Operative SLA thresholds:
    • Enterprise P1 = 30 minutes / 0.5 hours.
    • Growth P1 = 2 hours.
    • Starter = no formal SLA.
  • The existing support-l2-a answer set: 27 qualifying breaches, including the current
    16 Enterprise / 11 Growth split.

Validation

  • Both revised documents unzip cleanly and are retrievable through the benchmark file/Drive interface;
    the convention text is present in each, with §3.2 and §4.1 cross-references.
  • Recomputed support-l2-a with the existing tier thresholds (Enterprise 0.5 h, Growth 2.0 h,
    reference date 2026-04-13): 27 breaches, 16 Enterprise / 11 Growth — unchanged.

Context

Addresses a source-document clarity issue identified during independent benchmark validation. Some
agent/harness traces distinguished the tier-specific values correctly, while others treated the generic
Standard one-hour value as a contractual account tier and mapped Growth accounts to it. The
clarification removes that interpretation variance without changing benchmark semantics.


Exact text added (immediately before §3.2 in each of the two MSA docs):

Operative terms convention

Values shown in bold are the tier-specific commitments negotiated for and operative under this
agreement. Values marked (std: …) are Maple's standard published baseline values, included for
comparison only. Where the two differ, the bold tier-specific value governs this agreement.

…se MSA schedules

Adds an 'Operative terms convention' note to internal_docs/msa_growth_tier.md and
msa_enterprise_tier.md (inside artifacts/data.zip): bold tier values are the operative commitments;
(std: ...) values are Maple's standard baselines, for comparison only; where they differ the bold
tier value governs. Cross-referenced from the response/resolution (3.2) and billing-dispute (4.1)
schedules, where the ambiguity most affects retrieval.

Enterprise-Bench v1.1 source-document clarity fix. Operative thresholds, tasks, scoring, and the
support-l2-a answer set are UNCHANGED: 27 breaches (16 Enterprise / 11 Growth), independently
recomputed. Only the 2 MSA docs change; the other 77 dataset files are byte-identical; canary preserved.
@nimit2801

Copy link
Copy Markdown
Contributor

First-touch impression: dataset/source-document clarity change. This updates only artifacts/data.zip, clarifying that bold Growth/Enterprise MSA SLA values are operative while (std: ...) values are comparison baselines.

Checks: no GitHub status checks are reported on this branch yet, so maintainers should run the repo validation locally before merge.

Risk: low semantic intent, but zip-only dataset edits can hide canary drift, answer leakage, or reproducibility changes if not independently inspected.

Maintainer verification points:

  1. Unzip and confirm only the two intended MSA docs changed and CANARY.md is preserved.
  2. Check synthetic-data safety and confirm the added text does not leak benchmark answers beyond clarifying source terms.
  3. Run make validate and recompute the affected support task to confirm the expected 27 breaches / 16 Enterprise / 11 Growth result is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants