Skip to content

feat(verify): add per-citation self-check for anchor brevity#17

Merged
bensonwong merged 3 commits intomainfrom
feat/verify-anchor-brevity-self-check
Apr 10, 2026
Merged

feat(verify): add per-citation self-check for anchor brevity#17
bensonwong merged 3 commits intomainfrom
feat/verify-anchor-brevity-self-check

Conversation

@bensonwong
Copy link
Copy Markdown
Contributor

Summary

  • Adds a 3-step SELF-CHECK block to SKILL.md, placed immediately after the ≤4-word hard limit rule
  • Haiku stochastically violates the brevity rule on 2/3 runs (iter 22b R3: 80% long, iter 47 R1: 26%, R2: 44%) despite existing "HARD LIMIT" language
  • Failure pattern: leading quantifiers/size adjectives push anchors over 4 words (e.g. "six ground floor commercial units", "fifty-nine interior underground parking units")
  • Self-check instructs the model to: (1) count words explicitly, (2) drop leading quantifiers with concrete before/after examples, (3) verify ctrl+F uniqueness
  • Mirrors the SELF-CHECK block in COMPACT_CITATION_SCENARIO2_PROMPT which has shown stronger compliance

Test plan

  • Run verify skill on a condo/construction doc and confirm anchor brevity — anchors should be ≤4 words
  • Check that leading-quantifier phrases like "six ground floor units" get truncated to the noun phrase
  • Confirm no regression in citation uniqueness (ctrl+F test still passes)

Haiku stochastically violates the ≤4-word rule on 2/3 runs (iter 22b
R3: 80% long, iter 47 R1: 26%, R2: 44%) despite the existing "HARD
LIMIT" language. The failure pattern is consistent: leading
quantifiers and size adjectives pull phrases over 4 words
("six ground floor commercial units", "fifty-nine interior
underground parking units").

Add a 3-step per-citation SELF-CHECK right after the hard rules that:
1. Tells the model to count words explicitly before committing
2. Calls out the specific "drop the leading quantifier/adjective"
   pattern with real failing examples from iter 20
3. Adds a ctrl+F uniqueness test

This mirrors the SELF-CHECK block in COMPACT_CITATION_SCENARIO2_PROMPT
which has shown stronger anchor-brevity compliance.
@claude
Copy link
Copy Markdown

claude bot commented Apr 10, 2026

PR Review: feat(verify) - per-citation self-check for anchor brevity

The motivation is clear and the approach is well-grounded - mirroring the existing SELF-CHECK pattern from COMPACT_CITATION_SCENARIO2_PROMPT is a sensible consistency move, and explicit word-counting instructions are a known technique for improving compliance in smaller models like Haiku.

A few things worth addressing before merge:

Step 2 - rule title vs. example are inconsistent

The step heading says "Drop the leading quantifier or filler adjective" (singular drop), but the second example drops two words: "fifty-nine interior" from "fifty-nine interior underground parking units". "interior" is not a quantifier - it is a descriptive adjective. The right instinct is there but the mismatch between the rule label and the example could confuse a model trying to generalize. Consider either updating the title to "Drop leading quantifiers and filler adjectives" or splitting into sub-bullets so the model understands both operations are valid.

Step 3 - uniqueness vs. brevity conflict is unresolved

The instruction "If it takes 5+ words to be unique, you're probably grabbing too much context - pick the noun head, not the whole phrase" could cause regressions in uniqueness compliance. In dense legal or construction documents (exactly the domain this targets), two concepts can legitimately share the same 3-4 word noun phrase. Silently picking brevity over uniqueness is worse than a 5-word anchor. A safer framing: "If it takes 5+ words to be unique, prefer the most specific noun phrase; in rare cases where true uniqueness requires 5 words, that is acceptable." The goal is to prevent padding, not to hard-cap at 4 words when the document itself forces specificity.

Minor: first example may lose the noun head

"six ground floor commercial units" (5w) -> drop "six" -> "ground floor commercial" (3w) drops both "six" and the trailing noun "units". The resulting anchor reads as an adjective phrase rather than a noun phrase. Retaining the noun head - e.g. "ground floor units" (3w) or "commercial units" (2w) - might be more faithful and consistent with step 3's uniqueness goal.

Pre-existing issue (not introduced here, but surfaced by this PR)

skills/verify/rules/auth.md references "deepcitation@latest login --browser" and "login --key", while SKILL.md uses "deepcitation@latest auth" (with pseudo-TTY via script) and "auth --key". These are divergent command interfaces. Since SKILL.md states it is authoritative for auth, auth.md should either be updated to match or removed to avoid confusion.

Summary

Motivation: Strong - addresses a real, observed compliance failure
Approach: Sound - mirrors existing patterns that work
Step 1: Good
Step 2: Title/example inconsistency - easy fix
Step 3: Uniqueness conflict needs a clearer resolution
Example anchors: First example may lose the noun head unintentionally
auth.md drift: Pre-existing, but worth a follow-up ticket

Overall this is a useful addition. The step 2 title fix is a quick change; the step 3 framing is the more substantive concern worth discussing.

bensonwong and others added 2 commits April 10, 2026 08:30
…erative guidance, failure table

- Reference deep-citation-standards.md for canonical hard rules and field defs
- Add Rule 4 (Verbatim): k must be contiguous substring, no paraphrase/ellipsis
- Add ≤40 chars constraint to Rule 2 alongside word limit
- Add "Avoid heading-only anchors" section with BAD/GOOD examples
- Add "Prefer operative phrases over category labels" with BAD/GOOD examples
- Add four-failure-modes table (word count, bold≠k, multi-value, ellipsis)
- Trim truncation examples from 13 to 4; defer canonical list to standards doc
- Restore inline p-field format reminder ("pageNumber_arrayIndex")

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ll text

Iter 48 result: stronger before/after word-count self-check made things
worse (R2/R3: 47-48% long vs iter47's 26-44%). Root cause: counting
instructions don't prevent clause-level over-quoting. Model quotes full
operative clauses ("will automatically convert into the number of shares")
because each word feels necessary; the "drop leading quantifier" rule
doesn't fire since there IS no leading quantifier.

Iter 49 fix:
- Added 6 clause-truncation worked examples targeting mechanism verbs
  and liquidation tier markers (the exact patterns failing in iter48)
- Updated sub-agent word-count gate with inline SAFE examples
- Retained iter48 before/after counting steps (harmless, possibly helpful)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@bensonwong bensonwong merged commit edcf6df into main Apr 10, 2026
@bensonwong bensonwong deleted the feat/verify-anchor-brevity-self-check branch April 10, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant