Dario Amodei, the latter of whom was OpenAI's Vice President of Research
came out as Dario Amodei —job_title→ "Vice President of Research" and nothing else. The role is kept; the organisation it names is gone. On the graph Dario is a founder of Anthropic and nothing at OpenAI, so paths_between(OpenAI, Anthropic) cannot find the path the corpus most obviously states, and the answer to "how are the two related" leads with John Schulman instead.
Counted on the three-document re-extraction from #578: 52 role values; 9 of them name an organisation in their own quote (OpenAI's Vice President of Research, chief executive of Quora, the president of OpenAI, appointed … as OpenAI's chief technology officer of applications); 5 of those 9 have no edge between the person and that organisation. Rule 8b of the prompt already says to write both facts, the tie and the role, when a person is introduced by role; the possessive and the of/at forms, and the past tense, are where the model stops at the role.
What should come out
Dario Amodei —works_for→ OpenAI, valid_to: "unknown" (the sentence says was), plus the role value, both on the same quote.Greg Brockman —works_for→ OpenAI from "the president of OpenAI"; Adam D'Angelo —works_for→ Quora from "chief executive of Quora".- "appointed Statsig's founding CEO Vijaye Raji as OpenAI's chief technology officer of applications": the tie to OpenAI; the Statsig role is a different fact, and its tie is the model's to state with the end the sentence implies.
Where the fix goes
Two layers, the same shape as #578:
- The extractor's contract. Extend rule 8b with the forms it misses:
Y's <role>, <role> of Y, <role> at Y, was Y's <role>, former <role> of Y; the tie is a relation fact to Y, the role is a value on the person, and a past-tense or former sentence gives the tie valid_to: "unknown".
- A derivation at ingest, with a signal. A role value fact whose quote contains this fact's own value right after
<Org>'s or right before of|at <Org>, where <Org> is an entity declared in the same response and not a person, and where the response asserts no relation between the person and <Org>, gets the tie derived: works_for (or the base's employment relation by signature), confidence 0.7, the same quote as evidence, valid_to: "unknown" when the forty characters before the match contain was, former, then or previously. Recorded under role_tie_derived with the quote, so the rate reads per base and per model, and so the count of derived ties measures how often layer 1 still fails. Only the organisation that possesses this fact's value is tied; a second organisation in the same sentence is not.
Not this: deriving a tie from any organisation mentioned anywhere in the quote, or tying to a person (Sam Altman's chief of staff).
Acceptance
Re-extract the same three documents: the five missing ties above exist (four written by the model or derived, with role_tie_derived saying which); the Amodei path appears in paths_between(OpenAI, Anthropic); no tie is derived for the second organisation in the Raji sentence; the tests pin the possessive, of, at and past-tense shapes and leave a role with no organisation in its quote alone.
Related: #578 / #579 (the subject-side sibling), #558 / #565 (where the missing path shows), #559.
came out as
Dario Amodei —job_title→ "Vice President of Research"and nothing else. The role is kept; the organisation it names is gone. On the graph Dario is a founder of Anthropic and nothing at OpenAI, sopaths_between(OpenAI, Anthropic)cannot find the path the corpus most obviously states, and the answer to "how are the two related" leads with John Schulman instead.Counted on the three-document re-extraction from #578: 52 role values; 9 of them name an organisation in their own quote (
OpenAI's Vice President of Research,chief executive of Quora,the president of OpenAI,appointed … as OpenAI's chief technology officer of applications); 5 of those 9 have no edge between the person and that organisation. Rule 8b of the prompt already says to write both facts, the tie and the role, when a person is introduced by role; the possessive and theof/atforms, and the past tense, are where the model stops at the role.What should come out
Dario Amodei —works_for→ OpenAI,valid_to: "unknown"(the sentence says was), plus the role value, both on the same quote.Greg Brockman —works_for→ OpenAIfrom "the president of OpenAI";Adam D'Angelo —works_for→ Quorafrom "chief executive of Quora".Where the fix goes
Two layers, the same shape as #578:
Y's <role>,<role> of Y,<role> at Y,was Y's <role>,former <role> of Y; the tie is a relation fact to Y, the role is a value on the person, and a past-tense orformersentence gives the tievalid_to: "unknown".<Org>'sor right beforeof|at <Org>, where<Org>is an entity declared in the same response and not a person, and where the response asserts no relation between the person and<Org>, gets the tie derived:works_for(or the base's employment relation by signature), confidence 0.7, the same quote as evidence,valid_to: "unknown"when the forty characters before the match containwas,former,thenorpreviously. Recorded underrole_tie_derivedwith the quote, so the rate reads per base and per model, and so the count of derived ties measures how often layer 1 still fails. Only the organisation that possesses this fact's value is tied; a second organisation in the same sentence is not.Not this: deriving a tie from any organisation mentioned anywhere in the quote, or tying to a person (
Sam Altman's chief of staff).Acceptance
Re-extract the same three documents: the five missing ties above exist (four written by the model or derived, with
role_tie_derivedsaying which); the Amodei path appears inpaths_between(OpenAI, Anthropic); no tie is derived for the second organisation in the Raji sentence; the tests pin the possessive,of,atand past-tense shapes and leave a role with no organisation in its quote alone.Related: #578 / #579 (the subject-side sibling), #558 / #565 (where the missing path shows), #559.