Internal bookkeeping paths are never corpus content

[yellowman]

The upload manifest was a document. Measured, the indexed chunk reads

{" ferrothorn. txt":{" checksum":" 527c75bd8631…"," contexts":[]}}

— a user's own filenames and checksums, sitting in the corpus where a retrieval can answer out of it. Nobody uploaded that.

The rule already existed, in one place

The Files API draws this line and says so in its own comment: hidden components are internal bookkeeping, uploads and extraction strip leading dots, so a user can never own such a name. Listings omit them; download and delete treat them as absent.

Ingestion never learned it. .json is in the default extension list, so a directory source rooted at files/ — the obvious source to add, being everything the user has uploaded — walked straight into the manifest.

Where the rule lived is worth noting: spelled twice inside routes.py alone, once inline in the listing and once as _is_hidden_relpath, and a third time nowhere. A predicate with two copies and one missing caller is the shape this defect is made of.

Components, not basenames — and position, not spelling

The obvious patch is if file_path.name == ".checksums.json": continue, which fixes the sighting rather than the class. bundle/.internal/secret.md is internal for the same reason and would stay indexed.

Two predicates, because two different questions are being asked:

• is_internal_path(relative) — the component rule, over a path already expressed relative to a namespace.
• is_internal_under(base, path) — the same question asked of an absolute path, by first putting it in the frame that gives "internal" a meaning. It sits between two opposite errors: the absolute path must not be scanned, because whether a deployment lives under /srv/.storage is its own spelling and would refuse an entire installation; and the basename alone is not enough, because bundle/.internal/secret.md has an ordinary basename and an internal position.

Every ingestion route, not only the walk

route	before	after
directory walk	no check	each entry classified relative to the source root
source named outright	path.name only	classified against the base it was authorized within
durable ingest queue	never asked	classified against fs_root, job closed superseded
Files API listing	inline copy of the rule	calls the shared predicate
_is_hidden_relpath	second copy of the rule	calls the shared predicate

The queue was the seam that mattered most. Re-indexing calls rag.ingest_file directly, so every refusal added to the walk was invisible to it — and the queue is the durable machinery a replacement actually runs through, so an internal path reaching ingest_job would be chunked on a schedule long after whoever created it stopped watching. The job is closed rather than failed: nothing is owed now or later, and a failure would be retried five times to reach the same conclusion.

Authorization and classification stay separate, which is why the direct-source case exists at all. authorize_path grants authority over anything under the caller's own users/{id} directory and says nothing about bookkeeping. A caller is entitled to read their own manifest, and it is still not a document.

What the file budget actually depends on

A tree full of bookkeeping must not exhaust max_files before reaching anything a user wrote. It does not — but the reason is narrower than where the check sits. files_processed is incremented only after a successful ingest, so a path that continues before that leaves the budget untouched wherever the test for it appears. What makes the property hold is that internal entries are refused at all.

That distinction is recorded because it was nearly mis-stated as a claim about ordering, and a code comment asserting it had already been written before the claim was checked. The witness for the budget says plainly that it pins a property rather than reproducing a failure — it passes against the unfixed code whenever the walk happens to yield a real document first.

Verification

Nine witnesses, six failing before the fix. Two exist to hold the fix at the right altitude: one drives the Files API and the corpus walker over one tree and requires that what the listing omits is exactly what ingestion refuses; one drives POST /v1/contexts/{id}/sources, parameterized over a file named beneath .internal and a directory rooted there.

Six mutations, all applied, all killed, and the two narrowest land exactly where they should:

mutation	kills
walker stops refusing internal paths	4
rule weakened to basename	nested case
rule narrowed to the manifest's name	nested case
source never classified	3
source classified by basename, not position	the 2 direct-source cases only
queue stops asking	the queue case only

Lanes: make test-xdist — 2884 passed, 27 skipped, 0 failed. CI's lint selection clean.

Two corrections made during the work

A mutation reported SURVIVED while measuring nothing: its replacement ended in an escaped quote inside a raw string, producing an unterminated literal, so pytest never ran. The driver now treats a run with no summary line as a broken build and reads stderr as well as stdout.

The queue witness was vacuous as first written. It enqueued a placeholder generation, so the job was declined as stale before ingestion was attempted, and "no chunks were written" passed for a reason unrelated to the path — the detail column said on-disk generation 2faf4dced1fb, which is what that column was added for. It now enqueues the generation the bytes actually have, and the unfixed queue does index the internal file.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DQtPsg9YSUXaStGXyUjozA

---

Generated by Claude Code

---

Note

Overview
Fixes a disclosure where .checksums.json (and other dot-component paths) could be indexed as RAG corpus—e.g. after ingesting a user's entire files/ tree—even though the Files API already hides them from listings and downloads.

Centralizes the rule in service.fs as is_internal_path (any path component starting with .) and is_internal_under (same check for absolute paths relative to an authorized base). The Files API listing and _is_hidden_relpath now call the shared predicate instead of duplicating inline logic.

Corpus ingestion applies it everywhere: directory walks skip internal entries; ingest_path classifies named sources against their authorized base (not basename-only); the durable ingest queue closes internal jobs as superseded instead of calling ingest_file directly. SPEC now states internal paths are never content and must be refused by any route.

Adds tests/test_internal_paths_are_not_corpus.py to lock listing/walker agreement, nested hidden dirs, direct sources, and queue behavior.

^{Reviewed by Cursor Bugbot for commit 86bb25c. Bugbot is set up for automated code reviews on this repo. Configure here.}