Skip to content

Trim prescriptive guidance from the Freebuff prompt harness - #1276

Open
thomasboom wants to merge 3 commits into
CodebuffAI:mainfrom
thomasboom:trim-prompt-guidance
Open

Trim prescriptive guidance from the Freebuff prompt harness#1276
thomasboom wants to merge 3 commits into
CodebuffAI:mainfrom
thomasboom:trim-prompt-guidance

Conversation

@thomasboom

@thomasboom thomasboom commented Sep 4, 2026

Copy link
Copy Markdown

Three layers of guidance were shipping on every model turn, eating into both latency and per-task cost without buying any model behavior, while boating the context window and therefore making model performance worse:

  1. base3 system prompt + CLI appendix dropped from ~40 lines of prescriptive bullets to a 3-line system prompt and a one-line identity string. The destructive-command safety rail stays; the "match conventions / verify libs / use write_todos" bullets were product-guidance dressed up as engineering judgement.

  2. Eight tool descriptions trimmed to what the model actually needs to call the tool: code-search lost a 100-line ripgrep flag essay plus result-limiting notes; write-todos lost a 4-bullet manual; suggest-followups lost its "Good suggestions include" / "Don't include" lists; web-search / list-directory / glob lost their Purpose/Use-cases essays; run-terminal-command lost the 6-bullet "Stick to these use cases" and the 9-bullet <commit_analysis> checklist (the destructive-command rail is already in base3's system prompt). ask-user lost its redundant options list but kept the IMPORTANT no-Custom-options rule. Schema fields untouched - the model still knows each parameter.

  3. Deleted the unused getToolsInstructions / fullToolList / getShortToolInstructions exports in packages/agent-runtime/src/tools/prompts.ts (the [CRITICAL] Formatting Requirements, DO NOT narrate tool names, AS LITTLE TEXT AS POSSIBLE optimization) and the unused knowledgeFilesPrompt in system-prompt/prompts.ts. None had a live caller; the AI SDK reads each tool's description directly via getToolSet.

Per-turn hot cost: ~4,400 tokens saved on every model call (8 tool descriptions: ~13 KB / ~3,800 tokens; base3 system prompt: ~2 KB / ~625 tokens, cache-warm). On a 20-step task that's ~88k tokens not billed.

base2, base-chat, and base-deep untouched - different surfaces. base2 is the legacy/free-fallback harness and the eval control arm; base-chat is the freebuff.com/chat surface; base-deep is the deprecated deep-reasoning path.

Three layers of guidance were shipping on every model turn, eating into
both latency and per-task cost without buying any model behavior:

1. base3 system prompt + CLI appendix dropped from ~40 lines of
   prescriptive bullets to a 3-line system prompt and a one-line
   identity string. The destructive-command safety rail stays; the
   "match conventions / verify libs / use write_todos" bullets were
   product-guidance dressed up as engineering judgement.

2. Eight tool descriptions trimmed to what the model actually needs
   to call the tool: code-search lost a 100-line ripgrep flag essay
   plus result-limiting notes; write-todos lost a 4-bullet manual;
   suggest-followups lost its "Good suggestions include" / "Don't
   include" lists; web-search / list-directory / glob lost their
   Purpose/Use-cases essays; run-terminal-command lost the
   6-bullet "Stick to these use cases" and the 9-bullet
   <commit_analysis> checklist (the destructive-command rail is
   already in base3's system prompt). ask-user lost its redundant
   options list but kept the IMPORTANT no-Custom-options rule.
   Schema fields untouched - the model still knows each parameter.

3. Deleted the unused getToolsInstructions / fullToolList /
   getShortToolInstructions exports in packages/agent-runtime/src/tools/prompts.ts
   (the [CRITICAL] Formatting Requirements, DO NOT narrate tool
   names, AS LITTLE TEXT AS POSSIBLE optimization) and the unused
   knowledgeFilesPrompt in system-prompt/prompts.ts. None had a
   live caller; the AI SDK reads each tool's description directly
   via getToolSet.

Per-turn hot cost: ~4,400 tokens saved on every model call (8 tool
descriptions: ~13 KB / ~3,800 tokens; base3 system prompt: ~2 KB /
~625 tokens, cache-warm). On a 20-step task that's ~88k tokens not
billed.

base2, base-chat, and base-deep untouched - different surfaces.
base2 is the legacy/free-fallback harness and the eval control
arm; base-chat is the freebuff.com/chat surface; base-deep is the
deprecated deep-reasoning path.
@codebuff-team

Copy link
Copy Markdown
Contributor

The dead-code removal is clean and verifiable: getToolsInstructions, fullToolList, getShortToolInstructions in packages/agent-runtime/src/tools/prompts.ts and knowledgeFilesPrompt in system-prompt/prompts.ts genuinely look unused if getToolSet reads descriptions directly — that part alone would be an easy, welcome port.

The rest is a much bigger claim than the diff can support. Prompt wording for a coding agent (especially the base3 harness that's presumably the primary path) is usually tuned against eval suites, not just token-cost math. The PR asserts "buying no model behavior" for things like "match conventions," "verify libs before using them," and "use write_todos," but offers no eval numbers, only a token-savings estimate. That's a real product-behavior change dressed as a cleanup.

More concretely risky: run-terminal-command.ts's GIT_COMMIT_GUIDE_TAIL and the six-item "DO NOT" list (no git push, no global installs, no unsupervised prod-affecting commands, right package manager, etc.) got collapsed into one line: Don't push, never alter git config, don't use interactive flags, don't create empty commits. Dropping the explicit "ask for permission before running scripts / anything touching prod" guidance and the package-manager-matching rule is exactly the kind of safety rail that's stated more than once because single mentions get missed by models under context pressure — that's a functional regression candidate, not dead weight.

Also telling: the test changes (base3.test.ts, run-terminal-command-attribution.test.ts) were rewritten to assert the new strings exist rather than asserting old safety-relevant behavior is preserved — they validate the diff, not the claim.

Split this into (1) the dead-export removal, which is a clean, small, verifiable PR, and (2) the prompt-content trims, which need eval evidence or at minimum a narrower, one-tool-at-a-time PR so a maintainer can judge each trim on its own.

@codebuff-team codebuff-team added bot:triaged Classified by the community triage bot pr:needs-work Right idea, not mergeable as written labels Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:triaged Classified by the community triage bot pr:needs-work Right idea, not mergeable as written

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants