Skip to content

feat: detect wiki source references invalidated by a file move - #636

Open
Jithin (JM) Mohandas (DecentralizedJM) wants to merge 4 commits into
langchain-ai:mainfrom
DecentralizedJM:feat/source-citation-validator
Open

feat: detect wiki source references invalidated by a file move#636
Jithin (JM) Mohandas (DecentralizedJM) wants to merge 4 commits into
langchain-ai:mainfrom
DecentralizedJM:feat/source-citation-validator

Conversation

@DecentralizedJM

Copy link
Copy Markdown
Contributor

Problem

A wiki's most frequent factual claim is "this behavior lives in this file", and a refactor invalidates every one of them without touching the wiki. Readers follow paths that no longer exist, and an agent using the wiki as memory falls back to crawling the repo -- the expensive behavior the wiki was meant to replace. The generated Markdown writes these claims as inline code spans, not links, so validateWikiInternalLinks cannot see them.

This is not hypothetical. Reconstructing this repository at 9a415b5~1 -- after the domain-directory reorg in #513 and before the regeneration in #617 -- and running the new check finds 115 stale source references across all 7 pages, each with the correct destination:

Scanned 7 pages and 381 source citations in openwiki/.

Stale source references (115)
  openwiki/architecture/overview.md
    line 13: src/commands.ts -> src/cli/commands.ts
    line 23: src/ingestion.ts -> src/ingestion/ingestion.ts
    line 126: src/diagnostics.ts -> src/platform/diagnostics.ts
    ...

Every page on main pointed at files that did not exist for 9 commits and 4 days, and it took a full regeneration to correct.

What this adds

Finalize-pass validator (src/agent/source-citation-validator.ts), alongside the mermaid and link validators. It stamps each stale citation in place:

<!-- openwiki: stale source reference [src/commands.ts] source file "src/commands.ts" does not exist; a file with that name is now at "src/cli/commands.ts". Read the replacement file, correct the path and any claim that depended on it, then delete this comment. -->

The run never fails, and the prompt teaches the agent to repair from that comment, so the wiki self-corrects on the next update.

openwiki doctor reports the same findings without writing, adds the pages whose cited files changed since the commit recorded in .last-update.json, and exits non-zero when a reference is stale so CI can gate on it. It reads only wiki text and git, so it is fast and makes no model calls.

Precision over recall

A citation is reported only when its file is missing and a file of the same name exists elsewhere. A missing file with no such twin is ambiguous: it may be a path the reader creates in their own repository (openwiki/.langsmith.json), a runtime artifact, or a genuine deletion, and only the last is a defect. An earlier version without this gate produced exactly that false positive against the checked-in wiki. A move is self-evidencing and arrives with the correct path in hand, so every report is actionable.

Candidate detection is correspondingly conservative: a code span must be slash-separated, carry a letter-initial extension, avoid traversal segments, sit outside fenced code blocks, and begin with a directory that exists at the repository root. That last gate is what excludes the model id z-ai/glm-5.2, the repo slug langchain-ai/openwiki, and the media type application/json. Personal-mode runs are skipped entirely, since a personal wiki has no source tree.

Drive-by fix

test/agent/wiki-link-validator-dogfood.test.ts resolved rootDir to test/ instead of the repository root, so it scanned zero files and passed vacuously. It now points at the real repository (and passes), with a comment explaining why that matters.

Test plan

  • pnpm test green (2347 passed)
  • pnpm lint:check and pnpm format clean
  • Unit coverage for the extractor's precision rules, stamping, stamp clearing, idempotency across repeated passes, dot-file resolution, ambiguous rename candidates, reserved-file skipping, and personal-mode skip
  • Middleware integration test asserting the stamp lands via afterAgent without failing the run
  • openwiki doctor tests for clean and stale wikis, missing-wiki guidance, read-only behavior, and graceful degradation outside a git repository
  • Dogfood tests asserting the checked-in wiki has zero broken links and zero stale source references

A wiki's most frequent claim is that some behavior lives in a given file.
A repository reorganization invalidates every one of those claims without
touching the wiki, and because the generated Markdown writes them as inline
code spans rather than links, internal-link validation cannot see them.

Resolve the repository paths each page cites and report the ones a move
explains, naming where the file now lives. A finalize-pass validator stamps
them in place so a later update run self-corrects, matching the existing
mermaid and link validators. `openwiki doctor` reports the same findings
without writing, adds the pages whose cited files changed since the commit
the wiki last documented, and exits non-zero when a reference is stale.
Both are deterministic and make no model calls.

Only citations with a same-named file elsewhere in the repository are
reported. A missing file with no such twin is ambiguous -- it may be a path
the reader creates in their own repository, a runtime artifact, or a real
deletion -- and flagging those costs more trust than it returns.

Also fix the internal-link dogfood test, which resolved its root to test/
rather than the repository root and so scanned nothing and passed vacuously.
@changeset-bot

changeset-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 88b6ac3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
openwiki Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@DecentralizedJM

Copy link
Copy Markdown
Contributor Author

Colin Francis (@colifran) Please have a look when you have time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant