fix: guard against binary output context exhaustion

[Salil Das (sadlilas)]

Summary

When a bash command emits binary data — cat on a binary file, head -c N /dev/urandom, a stray blob on stderr — that data is decoded with errors="replace" and flows straight into the model's context as noise it cannot use.

_truncate_output() bounds a single call at ~100KB, but it does not bound a session. A 500KB binary payload still costs tens of thousands of tokens after truncation, so a few dozen such calls exhaust even a 1M-token context.

This PR withholds binary streams entirely rather than truncating them, and replaces each with a short, actionable placeholder.

The Fix

_guard_binary_output() runs per-stream, before truncation, on the raw bytes from the subprocess:

1. If the bytes decode as strict UTF-8, it's text. Returned unchanged. This covers ASCII, UTF-8 (accents, CJK, emoji), and the NUL-delimiter idioms (find -print0, grep -z, xargs -0) — NUL is valid UTF-8.
2. Otherwise it's binary or text in a legacy 8-bit encoding. These separate on the proportion of C0/C1 control bytes (excluding tab/LF/CR), which are pervasive in binary and absent from text. At or above 5%, the stream is withheld.

The placeholder carries the byte count, the control-byte percentage, and a file-redirect workflow (command > out.bin, then file out.bin or xxd out.bin | head). Output gains a binary_output_withheld: True flag when either stream is withheld.

Also fixed: stdout_bytes / stderr_bytes now report the true subprocess byte counts. They were previously derived by re-encoding the decoded string, which inflated them — U+FFFD re-encodes to 3 bytes for what was a 1-byte input.

Design Notes

Why not key off NUL bytes? It's the obvious binary marker and what web-content tools use, but in shell output NUL is a legitimate delimiter. find -print0, grep -z, and xargs -0 emit it on purpose. A NUL-keyed guard would break the standard safe-filename idiom; test coverage pins this.

Why detect on raw bytes, not the decoded string's U+FFFD ratio? The U+FFFD approach was implemented first, then measured and rejected — it is inverted on both sides:

Sample	U+FFFD ratio	Control-byte ratio
/bin/cat (real binary)	3.6%	≥ 6.0%
python3 (real binary)	2.3%	≥ 6.0%
Russian text, cp1251	80.9%	0.0%
Japanese text, shift_jis	64.8%	0.0%

Real executables carry large ASCII string tables and NUL padding that decode cleanly, so they would slip past a U+FFFD threshold. Legitimate text in legacy encodings is high-bit on nearly every character, so it would be destroyed. On the control-byte measure the two populations separate cleanly: every real binary tested scored ≥ 6.0%, every text sample scored 0.0%.

No configuration knob is exposed. The threshold sits in the middle of a wide empirical gap, and a knob would be a speculative surface with no caller asking for it.

Verification

Full suite: 141 passed, 12 skipped — no regressions.

New coverage (tests/test_binary_output_guard.py): 27 tests

• Binary withheld; placeholder contents; per-stream naming
• Real-binary bytes (ELF/Mach-O headers, NUL padding) withheld
• Legacy-encoded text (cp1251, shift_jis, latin-1) passes through undestroyed
• find -print0 / grep -z / xargs -0 pass through untouched
• ASCII, UTF-8 accents, CJK, emoji, JSON, ANSI-coloured output untouched
• Short-output and empty-output floors
• Byte-count reporting reflects true subprocess output size

End-to-end against real BashTool.execute():

• head -c 500000 /dev/urandom → withheld; ~906KB of decoded noise reduced to a ~217-char placeholder
• find . -name '*.txt' -print0 → passes through, NUL delimiters intact
• find . -name '*.txt' -print0 | xargs -0 -n1 basename → full idiom works
• printf 'café 日本語 🚀 ok' → untouched
• Binary redirected to stderr → withheld, placeholder names stderr

Code quality: ruff check clean on all changed files. (ruff format reports one pre-existing deviation in __init__.py unrelated to this change; left untouched to keep the diff scoped.)