A fact quotes the words that name each side

[WaylandYang]

Closes #582.

What was wrong

#559, #578 and #581 are one failure with three faces: the model attaches a fact to the wrong participant. "Former OpenAI personnel have founded … Anthropic" became OpenAI —founded→ Anthropic; "companies using OpenAI contacted Anthropic" became OpenAI —contacted→ Anthropic. Each face got a rule in the prompt and a guard with a word list. The guards measured that the rules get about half compliance, and a word list only knows the shapes it has already seen: #579's list caught former X personnel and X's investors, mistook partners for a group noun, and would have let the company's board through.

What changed

The model quotes; the machine judges. Two fields on every fact, subject_span and object_span: the verbatim words in quote that name each side. Copying is a task the model does reliably; deciding whether a description "counts" as the entity is the task it does badly. Rule 8d shrinks from a paragraph with two JSON examples (732 characters) to one sentence about copying the spans (340).

At ingest, verify_span compares each span with the name of the entity the fact is bound to (the entity its subject_ref / object_ref points at, or the written name when there is no reference) and sorts it into a tier, no model call:

1. In the quote? (case and whitespace aside). Otherwise the fact is handled as if it had no span and span_not_in_quote is recorded, so the rate reads per model.
2. The bound name itself: the same recall keys as resolution (same name, same stem, generic suffix), or the span's words are all in the name (Altman for Sam Altman, Anthropic for Anthropic, PBC), after stripping punctuation, possessives and a leading article. → kept.
3. Another declared entity's name (exact, or the span's words all inside it: OpenAI's board → OpenAI's board of directors) → the fact is rebound to it (span_rebound).
4. The name with words after it, or with a possessive (Former OpenAI personnel, The Verge reporter, Anthropic's safeguards) → the name is a modifier inside a larger phrase: a description. A described subject drops the fact (subject_described); a described object keeps the fact with the span as its literal value (object_described), the path An undeclared name does not become a node #568 uses for undeclared objects. Words after the name that all look like proper names (Anthropic PBC, OpenAI Global, LLC) are the same thing and are kept.
5. The name with words only before it (entrepreneur Tasha McCauley, companies using OpenAI) → a title or a different thing, and the machine cannot tell which without a parser. The binding is kept and span_prefixed is recorded with the phrase. A span that ends with the fact's other side (CEO of Applications: Fidji Simo as an object span bound to OpenAI) is a misplaced copy and is only counted (span_misplaced). A name followed by a comma is an apposition and is kept.
6. No declared name at all (him, the company) → the model resolved a coreference; nothing to check against. Kept, recorded as span_coreference.

The same tiers are then applied to the written name against its reference: a fact that writes OpenAI employees and points object_ref at OpenAI has contradicted itself, and the written name is judged like a span (tier 3 rebinds it, tier 4 makes it a description). This is what catches a description whose span carries no name at all (eleven employees, below).

Only tiers 3 and 4 change a fact. The first cut of this change treated every mismatch as a description and the half-run ledger showed why that is wrong: Altman, Sutskever, Anthropic (for Anthropic, PBC) were being dropped, entrepreneur Tasha McCauley and him were becoming literals. The tiers above are what survived that ledger; the two measured-only tiers say how much the mechanical check still cannot decide.

"Former OpenAI personnel" is tier 4 against OpenAI, and the fact is dropped with the phrase in the ledger. No word list is involved; the #579 guard and its list of group nouns are gone. "companies using OpenAI" is tier 5: kept, and counted.

Facts without spans (an older model, or a response that omits them) behave exactly as before.

Measured

openai.txt, removal-of-sam-altman-from-openai.txt and anthropic.txt re-extracted on this branch (schema.org pack, DeepSeek-V3), three runs, the same 123 chunks each time; the #579 run of the same three is the reference.

run	entities	untyped	facts	OpenAI–Anthropic one-hop edges
#579 (word list)	604	2%	1,288	1 (declined_merger, true)
v2: span checked against the written name	512	2%	1,008	2, both false (founded, proposes_merger)
v3: span checked against the bound entity	484	2%	1,178	0
v4: + "a span ending with a declared name is that name"	522	1%	1,025	1 false (companies using OpenAI rebound to OpenAI by that rule)
v5: the rule constrained (bound name declared, precedes, no comma)	546	1%	1,240	1 false (Anthropic —founded_by→ OpenAI from "Eleven employees left OpenAI … to establish Anthropic")
v6: written name checked against its reference; the head rule removed	511	2%	1,001	0

The fact counts move with the model: no chunk was skipped in any run, and v3, v5 and v6 are the same prompt at 1,178, 1,240 and 1,001, so the entity and fact totals are within run-to-run variance of #579. The untyped share holds at 1–2%. This branch is the v6 build.

The sentence, and why v2 missed it. With debug logging on, the raw reply for the chunk reads:

{"subject": "Former OpenAI personnel", "subject_ref": "k1", "subject_span": "Former OpenAI personnel",
 "predicate": "founded", "object": "Anthropic", "object_ref": "e1", "object_span": "Anthropic"}

The model copies the span faithfully and writes the subject as the description, then points subject_ref at k1, the base's OpenAI. The error is in the reference. v2 compared the span with the written subject (identical, so it passed) and the reference bound the fact to OpenAI: OpenAI —founded→ Anthropic came back and sorted first in paths_between all three times. v3 compares the span with the name of the entity the fact is bound to; the ledger reads subject_described | founded ×4 | Former OpenAI personnel (OpenAI) and the edge is gone. The second sentence came out of the model as subject companies using OpenAI with no reference and was dropped as undeclared (#568's path) in every run.

paths_between("OpenAI", "Anthropic") on v3, asked three times through the chat: one call each, all real edges; ranked first the DoD contracts (OpenAI ←awarded— DoD; DoD ←refused_to_authorize— Anthropic), then OpenAI ←worksFor— Dario Amodei (→ 2021); Dario Amodei ←has_ceo— Anthropic, the path #581 is about, which the model wrote this time.

What the tiers caught in v3 (1,178 facts):

tier	facts	reading of the examples
span_not_in_quote	281 (24%)	almost all Anthropic (Anthropic, PBC), Altman (Sam Altman), the company: the quoted sentence names the side by pronoun or not at all, and the model writes the entity's name instead. Nothing to check; the fact is kept.
span_prefixed	82 (7%)	titles kept correctly (Microsoft CEO Satya Nadella, Chief Scientist Ilya Sutskever, entrepreneur Tasha McCauley); about a dozen are phrases bound to a real entity that is not the phrase (OpenAI's relationship with Microsoft (Microsoft), an international watchdog organization similar to IAEA (IAEA)); the rest are verb phrases bound to concept entities the base would be better without. Kept, counted.
span_coreference	60 (5%)	him, the board's decision, whole clauses with no name. Kept.
object_described	58	descriptions stored as literals: a holding company owned by employees and other investors, Palantir's data platform, his removal from Y Combinator (Sam Altman). Two losses: Helen Toner, strategy director for … (Helen Toner) ×2 (an apposition) and CEO of Applications: Fidji Simo (OpenAI) ×11 (the object span copied the subject's org-chart line).
subject_described	14	Former OpenAI personnel (OpenAI) ×4, OpenAI's competitors (OpenAI), OpenAI's tools (OpenAI), About 738 of OpenAI's 770 employees (OpenAI employees), Allies of Altman (employees). All read as descriptions. One loss: OpenAI's board (OpenAI), where the board was not declared in that response and could not be rebound.
span_rebound	8	Claude (was Anthropic, PBC), GPT-3 (was GPT series), ChatGPT Plus (was ChatGPT), United States Department of Defense (was US Department of Defense): right. One wrong: the company → for-profit company (was OpenAI).

v4 and v5 were the method meeting more shapes. v4 added "a span that ends with a declared name is rebound to that name" for the Nadella/Ive losses; it fired 22 times and was wrong 9 times (his vested equity in OpenAI → OpenAI, TBPN, a media company in California → California, and Over one hundred companies using OpenAI → OpenAI, which brought the second sentence's false edge back). A name after a preposition or in a list is a complement, not the head. v5 constrained the rule (the bound name must itself be declared, precede the trailing name, with no comma between); it then fired twice, both on coordinations (Swisher and The Verge reporter Alex Heath → Alex Heath, Claude Fable 5 and Mythos 5 → Mythos 5), both wrong. The rule is gone; those spans are descriptions now and the tests pin all six shapes.

v5 also produced the last false edge in a shape no span can catch:

{"subject": "Anthropic", "subject_ref": "k18", "subject_span": "Anthropic",
 "predicate": "founded_by", "object": "OpenAI employees", "object_ref": "k1", "object_span": "eleven employees"}

The object span eleven employees is verbatim and names nothing, so it is a coreference and the binding stood: Anthropic —founded_by→ OpenAI, first in paths_between on v5. But the model's own written object, OpenAI employees, contradicts its reference: the same tier 4 applied to the written name against the entity it references makes it a description. That check is in the v6 build (written_verdict); across the v5 replies it would have touched 12 of 913 response-entity references, all of them contradictions of the same kind (Palantir's data platform → Palantir, AMD shares → AMD, GPT-4 training → GPT-4).

What the model does with the fields (v5 raw replies, 123 chunks, 1,125 facts): both spans on 69% of facts, one on 28% (value facts have no object), none on 2%. Subject spans are verbatim in the quote 75% of the time; of the rest, 215 of 266 are the entity's own name written where the quoted sentence uses a pronoun. Object spans are verbatim 91% of the time.

The commit after v3 acts on the three losses the v3 ledger names, with tests: a name followed by a comma is an apposition (Toner); a span that ends with another declared name is rebound to it (Microsoft chief executive Satya Nadella → Nadella, former Apple designer Jony Ive → Ive) unless that name is the fact's other side, which is a misplaced copy and is only counted (span_misplaced, the Simo line); a single common word inside a declared name is a coreference, not a rebind (the company). v6, the build in this branch. Neither sentence yields an edge; there is no one-hop edge between OpenAI and Anthropic at all. paths_between("OpenAI", "Anthropic") asked three times: one call each, all real (Microsoft Azure, the shared investors Coatue and Altimeter, Reddit, Jan Leike's move). The written-name check fired where the v5 edge came from: several OpenAI employees (OpenAI), OpenAI employees (OpenAI) ×3, Microsoft executives (Y Combinator) and Satya Nadella (Google Meet), the last two being references the model simply got wrong. Its one loss is OpenAI nonprofit (OpenAI Foundation) ×3, the same organisation under another wording. Rebinds: none in this run; span_misplaced: 9.

Tests

• a_span_that_names_a_description_is_not_the_entity: the two real sentences (tier 4 and tier 5); possessive; a phrase around a third entity; rebind by full name, by surname, by prefix of a declared name; a pronoun is a coreference; no span is Ok; a span absent from the quote is NotInQuote.
• a_name_continued_in_capitals_is_the_same_thing: Anthropic PBC, OpenAI Global, LLC kept; OpenAI employees described.
• a_written_name_that_describes_its_reference_is_a_description: OpenAI employees → OpenAI, Former OpenAI personnel → OpenAI are descriptions; OpenAI's board of directors → OpenAI rebinds; a suffix, a surname, no reference, a coreference are left alone.
• a_slot_matches_its_name_by_stem_and_suffix: OpenAI / OpenAI, Inc., Acme / Acme Corp., case, possessive, surname, Anthropic / Anthropic, PBC, a leading article; a description does not match its modifier.
• a_span_is_found_in_its_quote_regardless_of_case_and_spacing.
• spans_parse_and_default_to_none in utopia-extract; the prompt test asserts the one-sentence rule.

🤖 Generated with Claude Code

[WaylandYang]

Recall bench after the merge, same truth set as before (52 items over four NVIDIA filings, DeepSeek-V3.2, round.mjs: clear, re-extract, score; one standard deviation ≈ 2.7 items):

build	items	edges	literals	facts written
833fc30 (dev before this PR)	50/52	12/12	37/39	561
this PR, round 1	47/52	9/12	37/39	502
this PR, round 2	50/52	12/12	37/39	606

Round 1's three misses were one sentence (the Vera Rubin partners); the ledger had no row for them and the graph had no fact about them, so the model did not write them that time. Round 2 has all three. The same sentence on its own yields the five edges on both builds (this PR's build picked is_similar_to for the predicate, which the domain check refuses, and the edge lands with the proposed predicate). No recall cost is visible above the noise, and the fact totals bracket the baseline (502, 606 vs 561).

What the span tiers touched in round 2: object_described 11 (all read as descriptions: applications based on them, non-GAAP measures, Section 21E of the Securities Exchange Act of 1934, as amended), subject_described 2, span_rebound 5 (one doubtful: a subject span Stockholders rebound from NVIDIA to the annual-meeting entity because the single capitalised word is inside that name), span_misplaced 1. Round 1 had one wrong description: the coordinated subject span SB Energy and SoftBank bound to SB Energy. Both are candidates for a follow-up (a conjunction as a grammar word; a single-word rebind wants more than capitalisation).