feat(verify): add per-citation self-check for anchor brevity#17
feat(verify): add per-citation self-check for anchor brevity#17bensonwong merged 3 commits intomainfrom
Conversation
Haiku stochastically violates the ≤4-word rule on 2/3 runs (iter 22b
R3: 80% long, iter 47 R1: 26%, R2: 44%) despite the existing "HARD
LIMIT" language. The failure pattern is consistent: leading
quantifiers and size adjectives pull phrases over 4 words
("six ground floor commercial units", "fifty-nine interior
underground parking units").
Add a 3-step per-citation SELF-CHECK right after the hard rules that:
1. Tells the model to count words explicitly before committing
2. Calls out the specific "drop the leading quantifier/adjective"
pattern with real failing examples from iter 20
3. Adds a ctrl+F uniqueness test
This mirrors the SELF-CHECK block in COMPACT_CITATION_SCENARIO2_PROMPT
which has shown stronger anchor-brevity compliance.
|
PR Review: feat(verify) - per-citation self-check for anchor brevity The motivation is clear and the approach is well-grounded - mirroring the existing SELF-CHECK pattern from COMPACT_CITATION_SCENARIO2_PROMPT is a sensible consistency move, and explicit word-counting instructions are a known technique for improving compliance in smaller models like Haiku. A few things worth addressing before merge: Step 2 - rule title vs. example are inconsistent The step heading says "Drop the leading quantifier or filler adjective" (singular drop), but the second example drops two words: "fifty-nine interior" from "fifty-nine interior underground parking units". "interior" is not a quantifier - it is a descriptive adjective. The right instinct is there but the mismatch between the rule label and the example could confuse a model trying to generalize. Consider either updating the title to "Drop leading quantifiers and filler adjectives" or splitting into sub-bullets so the model understands both operations are valid. Step 3 - uniqueness vs. brevity conflict is unresolved The instruction "If it takes 5+ words to be unique, you're probably grabbing too much context - pick the noun head, not the whole phrase" could cause regressions in uniqueness compliance. In dense legal or construction documents (exactly the domain this targets), two concepts can legitimately share the same 3-4 word noun phrase. Silently picking brevity over uniqueness is worse than a 5-word anchor. A safer framing: "If it takes 5+ words to be unique, prefer the most specific noun phrase; in rare cases where true uniqueness requires 5 words, that is acceptable." The goal is to prevent padding, not to hard-cap at 4 words when the document itself forces specificity. Minor: first example may lose the noun head "six ground floor commercial units" (5w) -> drop "six" -> "ground floor commercial" (3w) drops both "six" and the trailing noun "units". The resulting anchor reads as an adjective phrase rather than a noun phrase. Retaining the noun head - e.g. "ground floor units" (3w) or "commercial units" (2w) - might be more faithful and consistent with step 3's uniqueness goal. Pre-existing issue (not introduced here, but surfaced by this PR) skills/verify/rules/auth.md references "deepcitation@latest login --browser" and "login --key", while SKILL.md uses "deepcitation@latest auth" (with pseudo-TTY via script) and "auth --key". These are divergent command interfaces. Since SKILL.md states it is authoritative for auth, auth.md should either be updated to match or removed to avoid confusion. Summary Motivation: Strong - addresses a real, observed compliance failure Overall this is a useful addition. The step 2 title fix is a quick change; the step 3 framing is the more substantive concern worth discussing. |
…erative guidance, failure table
- Reference deep-citation-standards.md for canonical hard rules and field defs
- Add Rule 4 (Verbatim): k must be contiguous substring, no paraphrase/ellipsis
- Add ≤40 chars constraint to Rule 2 alongside word limit
- Add "Avoid heading-only anchors" section with BAD/GOOD examples
- Add "Prefer operative phrases over category labels" with BAD/GOOD examples
- Add four-failure-modes table (word count, bold≠k, multi-value, ellipsis)
- Trim truncation examples from 13 to 4; defer canonical list to standards doc
- Restore inline p-field format reminder ("pageNumber_arrayIndex")
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ll text
Iter 48 result: stronger before/after word-count self-check made things
worse (R2/R3: 47-48% long vs iter47's 26-44%). Root cause: counting
instructions don't prevent clause-level over-quoting. Model quotes full
operative clauses ("will automatically convert into the number of shares")
because each word feels necessary; the "drop leading quantifier" rule
doesn't fire since there IS no leading quantifier.
Iter 49 fix:
- Added 6 clause-truncation worked examples targeting mechanism verbs
and liquidation tier markers (the exact patterns failing in iter48)
- Updated sub-agent word-count gate with inline SAFE examples
- Retained iter48 before/after counting steps (harmless, possibly helpful)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Summary
SELF-CHECKblock inCOMPACT_CITATION_SCENARIO2_PROMPTwhich has shown stronger complianceTest plan