Release 0.6.0 — freshness a timestamp cannot fake, a search that stays in your project - #35
Merged
Conversation
Four read and write paths accepted a cache entry as current when its recorded mtime was at least the file's, with no hash check. Any rewrite that preserves timestamps satisfies that gate over different bytes, so `write append`, `edit` and `batch_edit` could build on superseded text and write it back over the file. Freshness is now exact equality, via one shared predicate: an mtime that moved backwards is the strongest staleness signal a stat can give, not evidence of freshness. Every mutation reads and hashes disk first, and uses the cached text only when it provably matches, so an incremental re-store still reuses the chunk manifest. The base is read as bytes and decoded rather than through read_text, whose universal-newline translation would convert a CRLF file to LF on every edit and never match the stored hash.
…action get_content joined a file's child rows and returned them with no comparison to the content_hash they claimed, so a missing or duplicated row produced wrong content under a hash saying otherwise. The rewrite that produces those rows was three separate store calls. Interrupted between insert and re-tag it left a chunk set that looks healthy — one probe found 80 children numbered 0..79 with no gaps or duplicates, totalling 166,000 bytes for a 164,000-byte file, matching neither version. reconcile_chunks now does all three in one transaction, which also costs one IO-thread hop instead of three. The locked helpers exist because `with conn:` does not nest. The join is checked against the hash and raises ContentIntegrityError; the cache catches that alone — not the store's unrelated ValueErrors — drops the rows and re-reads from disk. Queries also take a path scope: grep a `within` root, and keyword_search a SQL path prefix. The prefix belongs in SQL because LIMIT applies after WHERE, so filtering afterwards spends result slots on files the caller cannot use. It is escaped for LIKE, since a directory may contain % or _.
…read The docstore is one file shared by every project the client has ever opened. grep with a relative pattern and search with no directory both ranked across all of it: in this repo, grep for "^##" in CHANGELOG.md returned two other projects' changelogs and never reached this one before the match cap. Both are now anchored to the client root by default, the rule grep's path filter already implied; an absolute path or explicit directory still reaches anywhere. The root is resolved before use, since cached paths are stored resolved and a symlinked checkout would otherwise match nothing, and it is carried across the worker boundary rather than re-derived there. batch_read joins the multi-file responses that name their shared directory once as root and report paths relative to it, debug summary included, so one response no longer mixes two path conventions. Tool docstrings drop the hash mechanics they restated from the server instructions, which are sent once, and keep only their own exceptions: 9,436 advertised tokens to 8,791, paid on every request.
mypy was advisory on pre-push, and ruff ran with an --ignore list that existed only in the hook, so the configuration the hook enforced was not the one in pyproject.toml. A hook that passes what CI will fail is worse than no hook. The ignores moved into pyproject.toml, mypy blocks, and the pre-commit/pre-push split is by cost, not by strictness. The token-savings floors were up to 19 points below the published claim, so they defended nothing; raised to the figures the README states.
The root is resolved now, because cached document paths are stored resolved and the prefix test would otherwise miss every one of them. The test still asserted the literal URI path, which passes on Linux and fails on macOS, where /home is a firmlink to /System/Volumes/Data/home.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two assumptions in this server were wrong in the same way: both trusted a signal that is cheap to read instead of the one that is true.
Behaviour changes to look at first
searchwith nodirectorynow means the current project, not every project in the store.grepwith a relativepath, or none, is anchored to the client root. An absolute path still reaches anywhere.The mtime assumption
Four read and write paths accepted a cache entry as current when its recorded
mtimewas at least the file's, with no hash check — and one of those was the base text forwrite(append=true),editandbatch_edit. Any rewrite that preserves timestamps satisfies that gate over different bytes:cp -p,rsync -t,tar -x,touch -d, a checkout that restores mtimes. An append then built on superseded text and wrote it back over the real file. That is data loss, and it needed no race to happen.Freshness is now exact equality — an mtime that moved backwards was rewritten by exactly those tools, so a backwards jump is the strongest staleness signal a
statcan give, not evidence of freshness — and every mutation reads and hashes disk before it writes.The shape of the store
The docstore is one file shared by every project the client has ever opened. In this repo,
grep(pattern="^##", path="CHANGELOG.md")returned two other projects' changelogs and never reached this one before the match cap. Both tools are now anchored to the client root, which is the rulegrep's path filter already implied.For
searchthe directory reaches SQL as a path-prefix clause. That placement is the point:LIMITapplies afterWHERE, so filtering after the ranking would spend result slots on files the caller cannot use and return a handful of matches, or none, while plenty existed. The prefix is escaped forLIKE, since a directory may contain%or_.Chunk integrity
get_contentjoined a file's child rows and returned them with no comparison to thecontent_hashthey claimed. The rewrite producing those rows was three separate store calls; interrupted between insert and re-tag it left a chunk set that looks healthy — one probe found 80 children numbered 0..79 with no gaps or duplicates, totalling 166,000 bytes for a 164,000-byte file, matching neither version. The join is now verified, and the reconcile is one transaction (also one IO-thread hop instead of three).Also
batch_readjoins the responses that name their shared directory once asroot.mypyblocks, and the ruff ignores live inpyproject.tomlrather than only in the hook.No cache-format change and no migration. See the CHANGELOG entry for the full list, including six smaller fixes.
Verification
uv run pytest -q→ 5144 passed, 6 skippeduv run mypy src/→ no issues in 45 source filesuv run ruff check src/ tests/andruff format --check→ cleanuv run bandit -c pyproject.toml -r src/→ 0 issuesLatency was not measured: the machine was too loaded to produce a number worth publishing (the unrelated edit path swung 3.24–11.42 ms across consecutive runs). The atomic reconcile reduces IO-thread hops from three to one, so it is expected neutral-or-faster, but that is reasoning, not a measurement.