You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#559, #578 and #581 are one failure with three faces: the model attaches a fact to the wrong participant. It writes OpenAI for "former OpenAI personnel", mints a node for "lawsuit against OpenAI", keeps "Vice President of Research" and drops the OpenAI it belonged to. Each got a rule in the prompt and a guard with a word list; the guards measured that the rules get about half compliance, and the word lists only know the shapes we have already seen.
The general fix is to stop asking the model to obey a rule and ask it to quote. A fact already carries quote, the sentence it rests on. Add two fields: subject_span and object_span, the verbatim words in the quote that name each side. Copying is a task models do reliably; deciding whether a description "counts" as the entity is the task they do badly. Then the check is mechanical and needs no model call:
The span must be in the quote. Otherwise the fact is treated as if it had no span (span_not_in_quote signal, so the rate reads per model).
The span must name the entity it is bound to. Same name after normalisation, or the same stem (Acme / Acme Corporation), or an alias. When it does not:
the span names another entity declared in this response or known to the base → rebind to that one (span_rebound);
the span names nothing → it is a description. A described subject drops the fact (subject_described); a described object keeps the fact and stores the span as the literal value (object_described), as An undeclared name does not become a node #568 does for undeclared objects.
"Former OpenAI personnel" fails check 2 against OpenAI and matches nothing → dropped, with the phrase in the ledger. "companies using OpenAI" likewise. "the company's board" or "he" as a subject span: the same path, with no word list at all. The #579 guard and its list of group nouns retire.
What this does not cover: a fact the model never wrote (#581). That is a recall problem and stays separate.
Cost
Two short fields per fact in the output, no extra call. The prompt loses rule 8d's paragraph and example; the rule becomes one sentence about the spans.
Acceptance
Re-extract openai.txt, removal-of-sam-altman-from-openai.txt and anthropic.txt and compare with the #579 run of the same three (604 entities, 1,288 facts, no OpenAI–Anthropic edge from either sentence):
the two sentences still yield no OpenAI–Anthropic edge, now caught by the span check rather than the word list;
span_not_in_quote rate per fact, and what the model does with the fields (are they filled, are they verbatim);
subject_described / object_described examples read as descriptions; span_rebound examples read as correct rebinds;
#559, #578 and #581 are one failure with three faces: the model attaches a fact to the wrong participant. It writes
OpenAIfor "former OpenAI personnel", mints a node for "lawsuit against OpenAI", keeps "Vice President of Research" and drops the OpenAI it belonged to. Each got a rule in the prompt and a guard with a word list; the guards measured that the rules get about half compliance, and the word lists only know the shapes we have already seen.The general fix is to stop asking the model to obey a rule and ask it to quote. A fact already carries
quote, the sentence it rests on. Add two fields:subject_spanandobject_span, the verbatim words in the quote that name each side. Copying is a task models do reliably; deciding whether a description "counts" as the entity is the task they do badly. Then the check is mechanical and needs no model call:span_not_in_quotesignal, so the rate reads per model).Acme/Acme Corporation), or an alias. When it does not:span_rebound);subject_described); a described object keeps the fact and stores the span as the literal value (object_described), as An undeclared name does not become a node #568 does for undeclared objects."Former OpenAI personnel" fails check 2 against
OpenAIand matches nothing → dropped, with the phrase in the ledger. "companies using OpenAI" likewise. "the company's board" or "he" as a subject span: the same path, with no word list at all. The #579 guard and its list of group nouns retire.What this does not cover: a fact the model never wrote (#581). That is a recall problem and stays separate.
Cost
Two short fields per fact in the output, no extra call. The prompt loses rule 8d's paragraph and example; the rule becomes one sentence about the spans.
Acceptance
Re-extract
openai.txt,removal-of-sam-altman-from-openai.txtandanthropic.txtand compare with the #579 run of the same three (604 entities, 1,288 facts, no OpenAI–Anthropic edge from either sentence):span_not_in_quoterate per fact, and what the model does with the fields (are they filled, are they verbatim);subject_described/object_describedexamples read as descriptions;span_reboundexamples read as correct rebinds;paths_between(OpenAI, Anthropic)still rests on real edges only;Related: #559 / #568, #578 / #579, #581, #558 / #565.