fix: strip false A/B claim, add query_text signal, --version - #42
Conversation
The 'A/B test showed full meta costs ~10pp pass@1' and 'compact is the measured middle ground' claims were never run — the numbers came from a cross-run comparison already flagged as a confound in RESULTS.md 3b. Replaced with 'the pass@1 effect of meta level is UNMEASURED.' The compact/off modes remain (useful for tight budgets) but are not benchmarked. The MCP server default is reverted to 'full' in a follow-up commit.
query_text: the one signal that ranks by relevance to the *described problem* rather than structural nearness. BM25 scores the bug report / issue text against every symbol's source code and adds query_weight * normalized_score to each candidate. Default query_weight=0.0 = no effect (backwards compatible). Threaded through analyze_impact() -> _blend_hybrid() -> compile --query-text CLI flag -> MCP compile_context task_description. When only task_description is given (no changed_symbols or git_ref), changes are auto-detected from HEAD. 3 tests in test_core.py. Also: strip remaining false A/B claim instances from pipeline.py, cli, mcp_server.py, CHANGELOG, docs/MCP.md. Revert MCP default to 'full'. Add diffcontext --version. Fix hardcoded MCP server version to use diffcontext.__version__. Bump to 0.5.3.
|
👀 QA.tech will run a exploratory tests to review this PR as soon as a deployment is available for this PR. Alternatively, you can comment @qa.tech to manually trigger a review. Learn more about configuring preview deployments. What happens next
🤖 AI end-to-end testing powered by |
3 tests for query_text (the biggest new feature had zero coverage):
test_none_reproduces_baseline_byte_identically — regression guard:
query_text=None must not change ANY score, not even by float epsilon.
test_query_text_reorders_results — a non-None query_text must measurably
reorder the ranking (is_valid_email moves up with 'email validation
check').
test_query_weight_zero_is_noop — query_weight=0.0 is a no-op even
with non-None query_text.
The query_weight=0.3 magic number in the CLI and MCP server now carries
an explicit comment: 'Untuned default; not derived from any sweep. Do
not cite this as a measured value.'
277 tests, ruff, mypy clean.
|
Note Unable to verify through end-to-end testing The new commit only adds internal unit tests for the query_text BM25 signal and a code comment clarifying the query_weight default is untuned — it doesn't change the nature of the PR. This remains entirely a change to the diffcontext Python library, CLI (argparse flags), and MCP server, with no browser-based UI. The only associated deployment is a PyPI publish GitHub Actions job log, not a running web application, so there is nothing QA.tech can click through or exercise end-to-end. All functional changes (query_text signal, --version flag, false-claim docstring fix) are only reachable via Python calls, the CLI, or an MCP tool call. 📜 Previous reviews (1)
🤖 AI end-to-end testing powered by |
1 participant
What
Three fixes in two commits:
1. URGENT — false claim retraction
Stripped every instance of "the A/B test showed full meta costs ~10pp pass@1" and "compact is the measured middle ground" from 6 files. That A/B has never been run. The numbers came from a cross-run comparison already flagged as a confound in RESULTS.md §3b. Replaced with "the pass@1 effect of meta level is UNMEASURED." The compact/off modes remain (useful for tight budgets) but the MCP server default is reverted to
fulluntil measured.2. query_text signal (step 3 of the distribution plan)
compile_contexthad notask_descriptionparam — the MCP server couldn't accept "fix the timezone bug in date parsing," only a list of already-changed symbols. Addedquery_texttoanalyze_impact()→_blend_hybrid(): BM25 scores the bug report text against every symbol's source code and addsquery_weight * normalized_scoreto each candidate. This is the one signal that ranks by relevance to the described problem rather than structural nearness — the "thematic siblings" blind spot. Defaultquery_weight=0.0= no effect (backwards compatible). Surfaced astask_descriptionon the MCPcompile_contexttool and--query-texton the CLI. When onlytask_descriptionis given, changes are auto-detected fromHEAD.3. Minor
diffcontext --versionnow works (was missing)diffcontext.__version__Verification