fix: index scoping, outside-tree warnings, compile --json payload, and pass@1 qualifiers - #41
Merged
Merged
Conversation
The front page states 'context roughly quadruples pass@1: 5.5% -> 25.8%'
with no qualifiers. Two are missing, both already documented in
benchmarks/contextbench/RESULTS.md section 6:
(a) the seed functions given to every arm are ORACLE, extracted from the
gold patch -- the real claim is 'given correct localization, does
context quality matter?', not end-to-end issue solving;
(b) 121 of the 128 effective tasks are django, so this is largely a
django result.
Added both qualifiers wherever the number appears: README, CHANGELOG
v0.5.0 notes, docs/BENCHMARKS.md, website benchmarks.mdx, the
measured-limits card rendered by index.mdx, and llms-status.md.
Does not soften the result -- states it precisely. Source unchanged.
Task C -- index scoping. `diffcontext index .` excluded tests/,
benchmarks/, docs/ by default (scanner.EXCLUDED_DIRS) with no way to
override and no explanation when a --ref change spanned an excluded dir
(the single biggest practical gotcha: 14 generic "typo, renamed, or
deleted" warnings and the changed file omitted entirely).
- `find_source_files` / `find_python_files` take an `include` set of
directory names to keep despite the default exclusions. .gitignore
still applies (a gitignored dir is not indexed even with --include).
- `index_repository(repo, include=...)` threads it through; the cache
key covers the include set.
- `--include <dir>` flag on index/impact/diff/compile/blast/verify.
- `warn_unknown_symbols` now distinguishes "the symbol's file is in an
excluded dir" from a real typo: it names the dir and says
"re-run with --include <dir>" instead of the generic message.
- `diffcontext index` prints a one-line scope summary: N files across
which top-level dirs, and which source-like dirs were skipped.
Task D -- compile --json payload. `compile --json` returned only
symbol_count/token_estimate/total_repo_tokens/reduction_pct/context, so a
calling agent could not inspect or filter the selection (USAGE.md
advertises --json as "machine-readable, for scripts and agents").
- Added `included_symbols` (id, role, score, tokens) and
`dropped_symbols` (id, score) to the JSON payload. Existing keys
kept for backwards compatibility.
Regression tests: tests/test_index_scoping.py (9 tests) pin the
outside-tree warning path, the --include override, and the gitignore
precedence. ruff + mypy + 247 tests clean.
|
👀 QA.tech will run a exploratory tests to review this PR as soon as a deployment is available for this PR. Alternatively, you can comment @qa.tech to manually trigger a review. Learn more about configuring preview deployments. What happens next
🤖 AI end-to-end testing powered by |
|
🤖 QA.tech is queued to run end-to-end and exploratory tests on this PR. We'll update this comment as the review progresses. 📋 DetailsRepository: trakshan-mishra/Diffcontext 🤖 AI end-to-end testing powered by |
…e vs none) Task A -- the experiment that decides whether the project has a product claim beyond 'context helps.' Four arms (none, diffcontext, bm25, samefile) on the same 128 ContextBench tasks, same oracle seeds, same 4000-token budget, same model (GLM 5.2), same prompt. The ONLY difference is which supporters fill the budget after the seeds. All arms use a shared renderer (seeds + render_context, no meta-header) via --shared-renderer, so no arm gets DiffContext's graph-derived meta -- the selection algorithm is isolated. --shared-renderer flag on run_glm_pass1.py routes diffcontext/bm25/samefile through compile_arm_context (seeds first, then render_context from benchmarks.downstream.providers). bm25/samefile are only meaningful with this flag; without it they fall back to empty context. analyze_arms.py: full 4-arm pass@1 with Wilson CIs + all 6 pairwise exact McNemar tests (not just the headline pair). Results (128 attempted, 0 setup errors, 544 rows, provenance a494b47): none 10/128 7.8% [0.043, 0.138] samefile 33/128 25.8% [0.190, 0.340] bm25 37/128 28.9% [0.218, 0.373] diffcontext 41/128 32.0% [0.246, 0.405] Context vs none replicates robustly (p < 1e-5 on all three context arms). DiffContext does NOT beat bm25 (p=0.503) or samefile (p=0.115) at p<0.05 -- indistinguishable at n=128 with ~20 discordant pairs per comparison. Directional ordering diffcontext > bm25 > samefile is consistent but not resolvable at this sample size. samefile is competitive: 'just include the rest of the file' solves 25 tasks none fails (vs 33 for diffcontext). This is a falsification test, reported plainly. The honest framing: structural retrieval matches lexical and same-file retrieval downstream at this sample size, on django, under oracle localization. The graph is not measurably earning its complexity yet. Resolving the directional signal needs ~10x more tasks (cloning the other 17 ContextBench repos), not more seeds. Appended to benchmarks/contextbench/RESULTS.md as section 3b.
trakshan-mishra
added a commit
that referenced
this pull request
Aug 23, 2026
First release available as 'pip install diffcontext'. Restores the trusted-publishing workflow deleted in 99b9eac (the pypi.org side was never set up; now configured via OIDC trusted publisher, no API token). Bumps 0.5.0 -> 0.5.1 because v0.5.0 tag already exists and re-pushing won't retrigger the workflow. Includes the index scoping fix, compile --json payload, and 4-arm falsification test from #41. No functional changes to the retrieval pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two independently-revertable commits:
1.
docs:oracle-seed and django-dominance qualifiers (docs only)The front page states "context roughly quadruples pass@1: 5.5% → 25.8%" with no qualifiers. Two were missing, both already in
RESULTS.md §6:Added wherever the number appears: README, CHANGELOG v0.5.0, docs/BENCHMARKS.md, website benchmarks.mdx, the measured-limits card (rendered by index.mdx), llms-status.md. Does not soften the result — states it precisely.
2.
fix:index scoping + compile --json payload (source)Task C — the scoping bug.
diffcontext index .excludedtests/,benchmarks/,docs/by default with no override and no explanation. A--refchange spanning an excluded dir produced 14 generic "typo, renamed, or deleted" warnings and omitted the changed file — the tool looked broken when it was merely mis-scoped.find_python_files(..., include=...)+index_repository(repo, include=...)+--include <dir>flag on all commands..gitignorestill beats--include.warn_unknown_symbolsnow names the excluded dir and says "re-run with--include <dir>" instead of the generic typo message.diffcontext indexprints a one-line scope summary (N files across which dirs, what was skipped).tests/test_index_scoping.py(9 tests).Task D — agent JSON payload.
compile --jsonreturned only counts + context text, so an agent couldn't inspect/filter the selection. Addedincluded_symbols(id, role, score, tokens) anddropped_symbols(id, score). Existing keys kept for backwards compat.Verification
ruff check diffcontext/ tests/— cleanmypy diffcontext/— clean (33 files)pytest tests/— 247 passed, 2 skippedThis is held open pending a downstream pass@1 falsification test (Task A) comparing
diffcontextvsbm25vssamefilevsnoneon all 128 ContextBench tasks. Ifdiffcontextdoes not beatbm25downstream, the Task B wording needs another pass and should not be onmainyet. Merge after that result lands.