mcp: byte-budget response limiter, projection, and pagination for token efficiency - #102
Merged
Conversation
Byte ceiling on serialized MCP tool responses, default 100_000 (~25k tokens), following the existing CODE_SEARCH_ env-prefixed tunable pattern. Foundation for the MCP response-shaping work; not yet wired into app/main.py. Co-authored-by: Isaac
…etry (Steps 2-6)
Implements the byte-budgeted MCP response pipeline entirely in app/main.py
per the consensus plan (.omc/plans/ralplan-token-limiter.md):
- project_for_mcp() drops webui-only fields (duration_ns, content_sha,
byte_ranges) from the MCP wire payload; service.py/webui builders are
untouched.
- _effective_budget/_fit_list/_shape_response implement the serialize ->
measure -> tail-trim -> re-verify pipeline against the exact json.dumps
wire string, never an estimate.
- Per-tool truncators: static tail-trim table for list_repos/
find_references/list_imports/semantic_search; a search_code truncator
that synthesizes next_cursor via a bounded repo name->id SELECT,
degrading to a flagged handle-less truncation on a no-row/fault; a
dedicated get_file line-paging shaper (start_line/next_start_line,
split("\n") congruent with grep.py:436).
- _dispatch gains max_bytes threading, response_bytes_pre/response_bytes
telemetry, and duration_ns/cursor_invalid in _signals().
- search_code always runs in pagination mode (cursor param, CursorError
caught into a structured cursor_invalid payload); get_file gains
start_line; all six tools gain max_bytes.
Migrated the two search_code wrapper tests whose fakes needed the new
cursor kwarg; added response_bytes_pre/response_bytes and
duration_ns-before-projection assertions to the dispatch log tests.
Per-tool truncator/shaping unit tests land in a follow-up commit
(tests/unit/test_mcp_shaping.py).
Co-authored-by: Isaac
New tests/unit/test_mcp_shaping.py: pure-function pins for
_effective_budget's clamp matrix, _fit_list/_fit_lines exactness,
project_for_mcp's drop-list (and None-safety on skeletal payloads),
each static truncator's flag/recompute behavior, the pinned
post-projection MCP search_code envelope shape, and the
cursor_invalid structured payload.
Fake-engine integration tests cover the search_code byte-budget
truncation path end to end: cursor-traversal losslessness for content
matches, the no-row and lookup-fault cursor-synthesis degrade paths
(never raising), and the documented mixed grep+symbol carve-out (a
page-1-only symbol match in a truncated tail is lost and flagged,
while content matches for that same file remain recoverable).
get_file gets a parametrized multi-page reassembly property test
(plain/CRLF/no-trailing-newline/form-feed/unicode-line-separator
fixtures), a line-numbering-congruence pin against split("\n") vs
splitlines(), and the single-oversized-line documented edge.
test_all_tools_respect_budget parametrizes all six tools (incl. a
200-site x 32-candidate find_references worst case) over the env
default and a small max_bytes (AC2/AC6). test_lane3_fixture_reduction
pins projection alone at >=25% serialized-byte reduction on a
synthetic 200-file/6-match/6-range search_code fixture (AC4).
Co-authored-by: Isaac
…n change Update the MCP tools table (cursor/start_line/max_bytes params) and add a "Response size limits" section covering CODE_SEARCH_MCP_MAX_RESPONSE_BYTES (default 100000, ~4 bytes/token heuristic), per-request max_bytes clamping down, the truncated/truncation_reason="token_budget" signal, and the search_code/get_file resume handles (next_cursor/next_start_line). Release-notes the search_code behavior change: because the MCP tool now always runs in pagination mode, a plain row-cap fill reports truncated=false + next_cursor instead of the old truncated=true/truncation_reason="row_cap". Co-authored-by: Isaac
test_all_six_tools_thread_max_bytes_to_dispatch monkeypatches main._dispatch to capture the max_bytes kwarg each tool wrapper passes through, closing the per-tool threading gap the plan called out alongside test_effective_budget_clamp_matrix. Co-authored-by: Isaac
Blocking defect from code review: when the first file's own serialized size alone exceeded the byte budget, _fit_list returned keep=0, and the `if keep > 0` guard around cursor synthesis meant the response came back as `files: [], next_cursor: null, truncated: true` -- a silent, unrecoverable dead end under default config (confirmed: one file with 2000 matches serializes past 300KB > the 100_000 default budget). Fix mirrors get_file's existing single-oversized-line edge: floor `keep` at 1 whenever the untruncated payload had at least one file (that one response may still exceed the budget -- the same documented trade-off get_file already makes), and always attempt cursor synthesis from whatever file was kept. The safety re-serialize/shrink loop is now floored at keep=1 rather than allowed to reach 0, so the guarantee can never be undone there either. Also, per review: - Removed dangling "Principle 4"/"AC6"/"D2"/"D3" plan-doc references in app/main.py comments/docstrings, replaced with the actual constraint spelled out in words (app/main.py, README.md). - Deduped test_signals_log_includes_duration_ns_before_projection, which existed verbatim in both test_main.py and test_mcp_shaping.py; kept the copy in test_mcp_shaping.py (its charter is the shaping pipeline). - Added test_search_code_single_oversized_file_progress_guarantee: a single file whose own serialized size exceeds max_bytes still comes back alone, flagged, with a next_cursor that lets a caller traverse past it to the remaining files -- exercising the exact scenario the prior test only asserted in a comment without ever constructing. Co-authored-by: Isaac
tannerwendland-db
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MCP tool responses previously had no size governance:
get_filecould return ~250k tokens in a single call, andfind_referencescould exceed 300k tokens at default parameters. This PR bounds every MCP response with a byte-denominated budget and strips webui-only fields from the MCP surface — without touching the shared payload builders or the webui contract (app/service.py,app/search/*, andwebui/are byte-for-byte unchanged).What's included
CODE_SEARCH_MCP_MAX_RESPONSE_BYTES(default100000, ≈25k tokens at ~4 bytes/token) enforced against the exact emitted JSON at the_dispatchchoke-point; per-requestmax_byteson all six tools, clamped down to the server ceiling with a 1024-byte floor.truncated: true/truncation_reason: "token_budget"— never a hard error. Resume handles where plumbing exists:search_codesynthesizes anext_cursor(contiguous-prefix rule over the(repo_id, path, content_sha)sort order),get_filegainsstart_line/next_start_lineline paging with byte-exact reassembly congruent with grep line numbering.byte_ranges,duration_ns, andcontent_shafrom MCP responses only (webui keeps them) — 51.1% serialized-byte reduction on a representativesearch_codefixture.search_codeMCP tool (cursorparam,next_cursoralways present). Behavior note: row-cap fills now reporttruncated: false+next_cursor(matching webui semantics); garbled cursors return a structuredcursor_invalidpayload._dispatchlog line now carries pre/post-projection response byte sizes;duration_nsobservability preserved via_signals().Provenance
Built via a traced investigation (per-field token measurement across all six tools) → consensus-reviewed plan (
.omc/plans/ralplan-token-limiter.md) → implementation with three independent validation reviews (functional, security, code quality). The quality review caught and fixed one real defect pre-merge: thesearch_codetruncator lacked a progress guarantee when a single file exceeded the budget (a0d014a).Testing
make test: 1138 passed (baseline was 1082; +53-testtests/unit/test_mcp_shaping.pysuite plus wrapper-test migrations)ruff check/ruff format --check/mypy: cleangit diff --name-only master... | grep -E '^(app/service\.py|app/search/|webui/)'→ no hitsget_filereassembly is byte-exact (CRLF, no-trailing-newline, form-feed, U+2028/U+2029 fixtures);search_codecursor traversal recovers all content matches across truncated pages; documented edges (oversized single file/line) flagged and resumableThis pull request and its description were written by Isaac.