Bind stdio to UTF-8, make paraId addressable, and have assessment verify what it accepts

[Misha-42]

What

Seven defects, all found while editing a real 4.6 MB Russian research document (1181 paragraphs, 26 tables, 126 fields, scientific symbols). Five of them make the tool report a confident, wrong answer — or report no answer — rather than fail, which is the worst shape this class of bug takes for a tool whose whole promise is verified surgical editing.

Three are on the Windows/cp1251 path. The other four were found while verifying those, and are the ones an agent meets first, since they are the documented anchor, the documented pre-flight step, and the verdict fields it reads afterwards.

1. stdio used the process locale instead of UTF-8

run_stdio() read sys.stdin and wrote sys.stdout with whatever encoding the process locale assigned them. Python defaults those streams to the locale encoding: on Windows that is the ANSI codepage (cp1251 on a Russian host). MCP stdio is UTF-8 by specification, and Claude Code, OpenCode and Codex all send UTF-8 — so every non-ASCII argument was decoded as cp1251.

The failure was silent, not loud:

• docx_search_text with a Russian query returned count: 0 and no error at all, so it looked like a missing-text problem in the document rather than a transport problem.
• A write operation stored mojibake in the document and returned a normal-looking success envelope.
• In the other direction the server crashed: a response containing a character outside cp1251 (the document uses ₃ ² → − ≥ Δ; 221 occurrences) died with UnicodeEncodeError while encoding the reply, so the document could not be read at all.

This reconfigures both streams to UTF-8 before the first read.

2. docx_validate could report an unverified pass

The isolation checks (protected_*) only run when touched scopes are known, and they were entirely the caller's to supply:

• Supplying nothing, with no sibling .audit.json, skipped the deep checks outright — metrics.protected_object_checks said "skipped" while the report still returned ok: true.
• Supplying an explicit touched_* replaced the audit-derived set rather than merging with it. The checks use two axes — paragraphs are matched by w14:paraId, body blocks by paragraph index — so a caller who passed one axis made the other check compare against an empty set and reported protected_paragraph_changed or protected_body_block_changed for an edit that changed nothing else.

Measured on a one-paragraph edit, correct values paraId=1810201C, index=236:

Supplied	Before	After
nothing, audit present	pass	pass
touched_para_ids only	protected_body_block_changed	pass (merged)
touched_paragraph_indices only	protected_paragraph_changed	pass (merged)
both	pass	pass
nothing, no audit	ok: true, nothing checked	refused, with the reason
allow_skipped_isolation_checks: true	—	isolation_checks: "skipped" + warning

Explicit scopes now merge into the audit instead of replacing it, validating with neither an audit nor a claim is refused unless the caller explicitly opts into an inconclusive result, and every report carries isolation_checks: performed|skipped plus touched_scopes.

3. Two defects found while verifying the above

_canonical_hash used inclusive c14n. Inclusive canonicalisation renders every in-scope namespace declaration, so a part that merely redeclares a namespace changes the hash of every element beneath it. The .NET backend hoists drawing namespaces (xmlns:a, xmlns:a14, xmlns:pic) into ancestor scope, so validating a .NET-edited document with the Python backend reported every paragraph and table as changed: protected_paragraph_changed + protected_table_changed + protected_body_block_changed on a correct single-paragraph edit. Same paragraph, inclusive c14n: 3215 bytes vs 3016; exclusive c14n: 675 vs 675, byte-identical. Exclusive c14n renders only the namespaces an element actually uses.

normalize_patchset rejected a patchset sent as a JSON string, which is what a client sends when it does not expand the "$ref" in the tool schema. Every assess / dry_run / apply call from such a client failed with patchset must be an object even for a valid payload. It now parses the string first, and still refuses junk with a clear message.

4. paraId was a documented anchor that could not be written to

paraId is declared throughout schemas/patchset.schema.json, and SKILL.md ranks it above paragraph_index in anchor priority — so the documented first choice was the one that did not work. On the .NET backend, addressing a paragraph by its w14:paraId failed with target_resolution_failed: Target paragraph not found, while paragraph_index resolved to that very same id and the engine reported the id back in touched.para_ids. The Python backend resolved both.

The .NET engine ships prebuilt and cannot be rebuilt here, so the facade translates the anchor instead of the caller having to know which backend is in play:

	paragraph_index	paraId
before	works	target_resolution_failed
after	works	works

paraId stays the more precise anchor — an index shifts when content is inserted earlier in the body — so it is resolved against the source document rather than discarded, it wins when an operation carries both axes (as it already did in Python), and paragraphs inside tables resolve too. An id that is not in the document is left untouched so the engine, not the facade, reports the missing target.

5. assess could bless a PatchSet the write path then refused

Assessment resolved every target and then checked only that a precondition was present (is None), never that it held. It answered ok: true, risks: [] — on both backends — for an expected_old_sha256 of all zeroes, and for a table's xml_sha256, while apply refused that same edit. Verified on a one-table edit of the 4.6 MB document:

expected_old_sha256	before assess	after assess	apply
the table's text_sha256	ok: true	ok: true	applies
the table's xml_sha256	ok: true	precondition_mismatch	refuses
"0" * 64	ok: true	precondition_mismatch	refuses

The table case is a genuine trap rather than carelessness on the caller's part: docx_list_tables offers text_sha256 and xml_sha256 side by side, expected_old_sha256 hashes the target's text, and nothing said which was which — the schema declared the field as a bare "type": "string". Assessment now compares through _assert_expected_text, the very call the write path makes, so the two cannot drift apart again; the .NET result is corrected in the facade, since that backend cannot be rebuilt; and the schema now names the hash a precondition needs.

6. A dry run reported no verdict at the top level

docx_assess_patchset, docx_validate and docx_compare_structure each answer with a top-level ok. docx_dry_run_patchset did not — its verdict lived only inside safety_assessment.ok and validation.ok, on both backends. A client checking result["ok"], exactly as it does for the three siblings, met a KeyError; one that reads the missing key as falsy calls a successful dry run a failure. The verdict is now the conjunction of the two sub-verdicts, and an absent sub-verdict counts as not-ok, since an unproven pass is the one answer this tool must never give.

7. A missing argument was answered with a bare KeyError

Tools read their required arguments as args["docx_path"], so a caller who guesses the parameter name — path, say — was answered with KeyError: 'docx_path' and a Python traceback, which says nothing about what the tool accepts. The dispatcher now names the missing argument together with the tool's own required list:

docx_search_text: missing required argument 'docx_path'; this tool requires: docx_path, query

A KeyError that is not a declared argument — a tool's own internal data lookup — keeps its original traceback, since that is not the caller's mistake to fix.

Test

Four scripts, added to CONTRIBUTING.md and to the Python checks in ci.yml:

• scripts/run_validation_isolation_regression.py — drives the server-level validation path directly, so it needs no .NET backend and no fixture document. On the previous commit it fails at the first assertion: AssertionError: validation without an isolation claim should have been refused.
• scripts/run_paraid_target_regression.py — 15 assertions pinning the translation: the id resolves to the right index, a stale index yields to it, table paragraphs are covered, an unknown id is left for the engine, and a patchset that needed no translation is returned unchanged.
• scripts/run_precondition_assessment_regression.py — 9 cases over paragraphs, table rows, table cells and content-control-free anchors, asserting assess and apply agree in both directions, since apply_patchset refuses whenever assess_patchset is not ok. It also drives the .NET merge through a stand-in backend, so it stays free of .NET and of the prebuilt engine.
• scripts/run_tool_verdict_regression.py — the dry-run verdict is the conjunction of both sub-verdicts and an absent sub-verdict is not a pass; a real dry run reports a top-level ok; and a missing argument is asserted through handle, not through the helper, so the test notices the helper going unwired.

All four fail on the pre-fix commit. The stdio test substitutes loopback streams wrapped in explicit cp1251 rather than relying on the host locale, so it is deterministic on any platform — including a UTF-8 CI runner, where a locale-dependent test would pass even without the fix.

A fifth script on this branch, scripts/run_e2e_stdio_utf8_regression.py, was written by a second contributor working alongside me. It passes; it is not wired into ci.yml or CONTRIBUTING.md, which is theirs to decide.

Checks

compileall, run_smoke_test, run_structure_regression, run_outline_regression, run_stdio_encoding_regression, run_validation_isolation_regression, run_paraid_target_regression, run_precondition_assessment_regression, run_tool_verdict_regression, run_word_session_smoke and validate_word_ai_skill all pass, and the tool surface is still 63. run_patchset_alias_regression and run_engine_selection_regression fail on this machine for an unrelated reason — no .NET SDK installed (global.json SDK resolution failure); I confirmed the same failure on a clean main checkout before opening this.

Note on verification

The .NET backend re-serialises the whole word/document.xml part, even for a one-paragraph edit: on the test document it grew from 982 912 to 994 014 bytes and all 1160 paragraph fragments differ — attribute order (w:rsid*), <w:spacing ... /> spacing, and hoisted namespace declarations. Nothing semantic changes, and the package-level guarantee holds (every other part is byte-identical, which is what preserves images, styles, headers and numbering), but a byte-level diff inside document.xml is not local. It may be worth documenting, since "only your paragraph changed" cannot be shown that way.

Scope

Two things are deliberately not in this PR:

• assess couples allow_paragraph_count_change and allow_table_dimension_change: a table row insertion needs both, so a caller who set only the table flag is refused for the paragraph count. I left it alone. Splitting the two is a guard-rail semantics question — the coupling protects against an edit that changes a dimension nobody declared — and loosening it to save callers one flag is the wrong trade for a tool whose value is the refusal. It is worth a maintainer's judgement, not a silent widening here.

• word-ai http was reported to accept TCP connections and then never answer (GET /, /health, /mcp all hang to timeout, log file empty). I could not reproduce it — GET / 200, GET /health 200, POST /mcp 200 with all 63 tools, GET /mcp 404 — on either the prebuilt engine or a source checkout. Two things that look like the report are not it: an empty log file is stdout block buffering when the stream is redirected, and two processes apparently bound to one port is Windows SO_REUSEADDR. Happy to take it further with the reporter's word-ai --version, host and exact command.

Happy to file any of these separately or fold them in on request.

🤖 Generated with Claude Code