feat: agent self-test protocol and the diagnostic fixes it surfaced#11
Merged
Conversation
Add docs/agent-testing.md: a manual protocol for exercising the specification's authorability and this runtime's diagnostics by having an LLM agent author packs with the validator as its only feedback. Client-neutral (Claude Code, Cline, Continue are examples); the agent drives the existing CLI, so it needs no new code -- the MCP surface remains future work per ADR-0003. The output shape, exit classes, and the sample diagnostic are taken from a real run, not invented. Complements judgment-pack-spec/TESTING.md (the schema-validator protocol) and the automated contributor checks, and feeds the specification roadmap's authoring-ergonomics gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Brian Jin <35789537+kikashy@users.noreply.github.com>
Improve the structural and semantic diagnostics an author sees, without changing any conformance outcome (all 47 bundled cases still pass) or the machine contract (only messages and provisional codes change; outputVersion, envelope, and exit classes are untouched). - Type errors name the expected and actual JSON type instead of the bare "Value has the wrong JSON type." - A numeric operand on an ordered comparison reports JPS-STRUCTURE-DECIMAL-OPERAND with a concrete example, gated by comparisonOperandValue so a source locator value or a literal boolean is never misreported as an operand. - Unresolved reference messages name the offending id and list the declared ids of that kind. - A oneOf condition branch is rejected only when its "op" discriminator did not match, so a composite condition with more than one invalid child reports a specific, well-located diagnostic per child rather than collapsing to a single generic JPS-STRUCTURE-CONDITION-SHAPE. - Add a --pretty flag that indents JSON output; default output stays compact. Verified by two no-context authoring probes and an adversarial sweep across every layer: no document is wrongly accepted or rejected, and no instancePath is mislocated. Focused tests are added for each change and the existing condition-diagnostic guard test still passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Brian Jin <35789537+kikashy@users.noreply.github.com>
The self-test protocol no longer centers on one client. It names Claude Code, Cline, Cursor, and Continue as examples, and presents the instruction block as something any client can take -- a rules file such as .clinerules or .cursorrules, a custom-instructions setting, or simply the agent's first message. Add installing a tagged release from GitHub Releases alongside building from source, so a reader does not need a Go toolchain to run the protocol. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Brian Jin <35789537+kikashy@users.noreply.github.com>
kikashy
changed the title
An input that exceeds the byte limit -- the hard 10 MiB cap or a smaller
--max-bytes -- reported the generic JPS-INPUT-READ code, identical to a missing
file or a directory, and was the only resource limit without a dedicated
JPS-RESOURCE-* code. The adversarial diagnostics sweep surfaced this.
fssecure.ReadRegular and the stdin reader now return a shared ErrTooLarge
sentinel, and "spec validate" maps it to JPS-RESOURCE-INPUT-BYTE-LIMIT with a
message that names the limit ("Input exceeds the N-byte limit."). Genuine read
failures keep JPS-INPUT-READ. The exit class is unchanged (ExitIO), codes are
provisional, and no conformance outcome changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Brian Jin <35789537+kikashy@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an agent-driven self-test protocol for the runtime, then applies the diagnostic improvements that running it surfaced.
The protocol (docs)
docs/agent-testing.md— a manual, client-neutral protocol for exercising the specification's authorability and this runtime's diagnostics with any agent client (Claude Code, Cline, Cursor, Continue are examples), bringing your own key. Covers setup (install a tagged release or build from source), the validation loop and output contract, a client-neutral instruction block, a create/update/delete/adversarial task list, guardrails, and how to read the result. Complementsjudgment-pack-spec/TESTING.mdand ADR-0003.The diagnostic fixes (code)
Two no-context authoring probes and an adversarial QA sweep surfaced these. All are message/flag/code-level: no conformance outcome changes (47/47 pass) and no machine-contract change (only messages and
provisionalcodes;outputVersion, envelope, and exit classes untouched).JPS-STRUCTURE-DECIMAL-OPERANDwith a concrete example, precisely gated so a sourcelocator.valueor aliteralboolean is never misrouted.oneOfbranch is rejected only on anop-discriminator mismatch, so multiple invalid children each report a specific, well-located diagnostic instead of one genericJPS-STRUCTURE-CONDITION-SHAPE.--max-bytes-exceeded input →JPS-RESOURCE-INPUT-BYTE-LIMIT(naming the limit), instead of the genericJPS-INPUT-READit shared with a missing file; genuine read failures keepJPS-INPUT-READ.--prettyindents JSON output (default stays compact).Each change has a focused test; the existing condition-diagnostic guard test and all 47 conformance cases still pass.
Verification
Two no-context authoring probes plus an adversarial QA sweep (attacking every layer) found no wrong-accept, wrong-reject, mislocated path, or crash. The one defect the sweep flagged — the input byte-limit code above — is fixed in this PR.
🤖 Generated with Claude Code