Repository navigation
feat(citations): resolve and validate [n] markers - #114
Merged
Merged
Conversation
…nswer The prompt numbered its sources but threw the provenance away: by the time an answer existed there was no page, block or span to point back at. `SearchResult` already carries a `locator` (#74), so assemble it into a shared `Citation` snapshot — document, page range, block id, char span and the retrieved quote — and persist it under `chat_messages.metadata.citations`. The citation is a snapshot, not a reference: re-indexing or deleting the source later must not erase where the answer was grounded. A defensive reader (`parseCitations`) drops malformed rows individually so one bad entry cannot hide the rest. The renderer's in-memory message never sees the DB row written before streaming, so the persisted metadata also rides along on the `finish` event; otherwise an answer would stay un-citable until the session was reloaded. Refs #69
…rounded one An answer asked to mark its sources with `[n]` can mark a source that was never retrieved, or attribute a passage to the wrong span. A plausible link to the wrong paragraph is worse than no link, so both are treated as first-class failure modes. `resolveCitations(answer, contexts)` is pure and deterministic — no model call — and returns every marker split into `resolved` / `unresolved` / `misattributed`, so citation precision is derivable for the eval harness (#75) and the UI can choose to render a bad marker as plain text. Validation is case/whitespace-insensitive containment of the quote in the cited span and nothing more: fuzzy matching would let a fabricated quote through. A missing span cannot refute the quote, so it is not treated as an error. The chat handler resolves markers against the locator's block text once the answer is final, and persists only the grounded citations when the answer used markers at all. With no markers, the full evidence set is kept, because the model simply did not use the convention. Refs #70
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a deterministic, model-free
resolveCitations(answer, contexts)service that maps an answer's[n]markers back to the citations produced during retrieval, and splits every marker intoresolved/unresolved/misattributed. The chat path now persists only the grounded citations when the answer actually used markers.Why?
Once the model is asked to emit
[n], it can emit a marker with no corresponding source or attribute a quote to the wrong span. A research-grade citation feature has to treat both as failures rather than render a plausible-looking link. This is also the measurement point for citation precision in the eval harness (#75).Related issue
Fixes #70
What changed?
src/shared/utils/citationResolution.ts— pureresolveCitations+normalizeForComparison. Returnsmatches,resolved,unresolved,misattributedandprecision(resolved / total).src/shared/types/citation.ts—CitationContext(citation + the span text it covers),CitationMatch,CitationResolution.src/main/ipc/chatHandlers.ts— builds contexts from the locator's block text, resolves against the final answer, and persists the grounded subset inmetadata.citationswhen markers are present. No markers → keep the full evidence set.test/citationResolution.test.ts— grounded marker, fabricated[9], mis-attributed quote, plus case/whitespace normalisation, non-fuzzy matching, missing-span behaviour, precision and repeated markers.Design notes
unresolvedandmisattributedare returned, not silently dropped, so precision is computable and the renderer ([Feat] Citation click → open source at page/block with highlight #72) can show the marker as plain text.How was this tested?
npm run typecheck— passes.npm test— 147 tests pass (9 new).npm run build— passes.Screenshots / recordings
Not applicable (service + persistence; chips land in #72).
Checklist
npm run typecheckpasses.npm run buildpasses.Desktop / build changes