feat: agent self-test protocol and the diagnostic fixes it surfaced

docs/agent-testing.md

@@ -0,0 +1,167 @@
+# Exercising the runtime with an agent
+
+This runtime validates Judgment Pack Specification (JPS) documents; it does not author them. A
+practical way to test both the specification's *authorability* and this runtime's *diagnostics* is
+to let an LLM agent author packs with the validator as its only feedback, then report where the
+diagnostics helped and where they did not.
+
+This is a manual, exploratory protocol, distinct from the automated contributor checks in
+[CONTRIBUTING.md](../CONTRIBUTING.md) and the bundled `spec test-conformance` suite. It complements
+the specification's schema-validator protocol (`judgment-pack-spec`'s `TESTING.md`), which drives a
+generic Draft 2020-12 validator rather than an agent. Why the runtime, rather than a bespoke UI, is
+the integration and testing surface is recorded in
+[ADR-0003](adr/0003-mcp-integration-and-testing-surface.md).
+
+Today the agent drives the command-line binary directly. A first-class MCP surface is future work
+(ADR-0003); the loop below needs only the CLI.
+
+## What you need
+
+- A `judgment-pack` binary (installed or built below).
+- Any agent client that can run a shell command and read and write files, configured with **your
+  own** model API key. The runtime holds no key, opens no network connection, and evaluates nothing,
+  so the key lives entirely in the client. Any of the many such clients works — for example Claude
+  Code, Cline, Cursor, or Continue.
+
+## Get the validator
+
+Either install a released binary or build from source.
+
+**Install a tagged release.** Download the archive for your operating system and architecture from
+the repository's [GitHub Releases](https://github.com/Judgment-Pack/judgment-pack-runtime/releases)
+and put the binary on your `PATH`. Each archive also ships a `jpack` short alias.
+
+```bash
+tar -xzf judgment-pack_<version>_<os>_<arch>.tar.gz   # on Windows, expand the .zip instead
+install -m 0755 judgment-pack "$HOME/.local/bin/judgment-pack"
+judgment-pack version
+```
+
+**Build from source.** With a currently supported Go toolchain, from a checkout of this repository.
+If `go env GO111MODULE` reports `off`, force module mode as shown:
+
+```bash
+env GO111MODULE=on go build -o ~/.local/bin/judgment-pack ./cmd/judgment-pack
+judgment-pack version
+```
+
+## How the loop works
+
+The agent edits a candidate document and validates it after every change:
+
+```bash
+judgment-pack spec validate <path-or-> --format json
+```
+
+- `status` is `valid`, `invalid`, or `unsupported`; the loop stops at `valid`.
+- The process exit code is `0` for a valid document and non-zero otherwise.
+- `layers[]` reports `carrier`, `structural`, and `semantic`; a later layer runs only after the
+  earlier ones pass.
+- `diagnostics[]` carries one entry per problem, each naming a `code`, its `layer`, a `severity`, an
+  `instancePath` (a JSON Pointer to the exact location), and a `message`:
+
+```json
+{
+  "code": "JPS-SEMANTIC-UNRESOLVED-OUTCOME",
+  "codeStability": "provisional",
+  "layer": "semantic",
+  "severity": "error",
+  "instancePath": "/rules/0/outcome",
+  "message": "Outcome reference does not resolve."
+}
+```
+
+The agent fixes each diagnostic at its `instancePath` and re-runs. `--format human` prints the same
+result as a scannable summary; `spec schema 0.1.0-draft --write -` prints the exact bundled schema;
+`--through carrier|structural|semantic` stops after a chosen layer.
+
+## Brief the agent
+
+Give the agent these instructions however your client accepts project instructions -- a rules file
+(for example Cline's `.clinerules` or Cursor's `.cursorrules`), the client's custom-instructions
+setting, or simply the agent's first message. Replace `<spec>` with the path to a
+`judgment-pack-spec` checkout.
+
+```markdown
+# JPS pack authoring -- validation loop
+
+You author and revise Judgment Pack Specification (JPS) documents. The judgment-pack CLI is your
+validation oracle. It checks document conformance only -- carrier (bytes/JSON), structural (schema),
+and semantic (references) -- for spec version 0.1.0-draft. It does not evaluate rules, choose an
+outcome, fetch sources, or judge whether a pack is correct, authorized, safe, or fit. Structural
+validity means well-formed, nothing more.
+
+## The loop
+After every edit, validate and fix what it reports:
+
+    judgment-pack spec validate <path-or-> --format json
+
+- status: "valid" | "invalid" | "unsupported". Stop at "valid".
+- exit code: 0 = valid, non-zero = not valid.
+- diagnostics[]: each has code, layer (carrier|structural|semantic), severity, instancePath (a JSON
+  Pointer), and message. Fix by instancePath and re-run.
+- layers[]: a later layer runs only after earlier layers pass.
+
+To see the full schema:  judgment-pack spec schema 0.1.0-draft --write -
+
+## Where to start
+Read one example first to learn the shape, then build your own -- do not copy it verbatim:
+- <spec>/examples/minimal-expense-approval.json   (smallest)
+- <spec>/examples/supplier-invoice-approval.json
+
+## Guardrails
+- Work on a scratch copy. Never modify the checked-in examples.
+- Never put real, personal, confidential, regulated, or production data in a pack. Invented,
+  low-risk data only.
+- Never add or fetch a real URL in a source locator. Validation stays offline.
+
+## Report as you go
+The goal is to judge whether these diagnostics are enough to author by. Call out every diagnostic
+that was unclear, misleading, wrongly located, or missing when you expected one. That list is the
+deliverable.
+```
+
+## Tasks
+
+Give the agent one or more of these, in rough order of difficulty:
+
+1. **Create** -- author a new pack for an everyday approval scenario (travel reimbursement, time-off,
+   ...), seeded by reading the minimal example. Iterate to `status:valid`.
+2. **Update, then break a reference** -- add a rule or exception to a scratch copy of a larger
+   example and keep it valid; then point that rule at an outcome id that exists nowhere, and confirm
+   the semantic layer reports the dangling reference.
+3. **Delete** -- remove an outcome or rule and its now-orphaned references; confirm the diagnostics.
+4. **Adversarial** -- break a valid pack three structural ways and predict each diagnostic's
+   `instancePath` before running.
+5. **Report** -- list which diagnostics were clear, which confused, and any case where an error was
+   expected but none appeared.
+
+## Guardrails
+
+The same rules the agent is given apply to whoever runs the protocol: work on scratch copies, never
+edit the checked-in examples, use only invented low-risk data, and keep validation offline -- never
+add or fetch a real URL. This mirrors the data-handling warning in `judgment-pack-spec`'s
+`TESTING.md`.
+
+## Reading the result
+
+The deliverable is the agent's list of diagnostics that confused it, were mislocated, or were missing
+when it expected one. Judge each diagnostic by whether a first-time author could act on it *without*
+reading the schema or an example:
+
+- A strong diagnostic names the exact location and the specific problem -- for instance, distinct
+  semantic codes per reference kind (`JPS-SEMANTIC-UNRESOLVED-OUTCOME`,
+  `JPS-SEMANTIC-UNRESOLVED-EVIDENCE`), each pointing at the offending array element.
+- A weak diagnostic states only that something is wrong -- for instance, a structural type error that
+  does not name the expected type -- leaving the author to guess or to copy an example.
+
+Findings of the second kind are the point: they are the backlog for improving the runtime's
+diagnostics or the specification's clarity.
+
+## See also
+
+- `judgment-pack-spec`'s `TESTING.md` -- the schema-validator protocol for a generic Draft 2020-12
+  validator.
+- [ADR-0003](adr/0003-mcp-integration-and-testing-surface.md) -- why MCP is the integration and
+  testing surface.
+- [CONTRIBUTING.md](../CONTRIBUTING.md) -- the automated checks expected before a pull request.

internal/cli/app.go

@@ -26,6 +26,7 @@ type App struct {
 	errOut io.Writer
 	engine *validation.Engine
 	runner *conformance.Runner
+	pretty bool
 }
 
 type handledExit struct {
@@ -40,7 +41,7 @@ func Run(args []string, in io.Reader, out, errOut io.Writer) int {
 	if err != nil {
 		message := "Bundled JPS artifacts failed their integrity check."
 		if requestedFormat(args) == "json" {
-			if writeJSON(out, result.NewOperationalError(requestedCommand(args), "JPS-ARTIFACT-INTEGRITY", message)) != nil {
+			if writeJSON(out, result.NewOperationalError(requestedCommand(args), "JPS-ARTIFACT-INTEGRITY", message), false) != nil {
 				return result.ExitIO
 			}
 		} else {
@@ -58,7 +59,7 @@ func Run(args []string, in io.Reader, out, errOut io.Writer) int {
 		}
 		message := display.Sanitize(err.Error())
 		if requestedFormat(args) == "json" {
-			if writeJSON(out, result.NewOperationalError(requestedCommand(args), "JPR-INVOCATION-ARGUMENTS", message)) != nil {
+			if writeJSON(out, result.NewOperationalError(requestedCommand(args), "JPR-INVOCATION-ARGUMENTS", message), false) != nil {
 				return result.ExitIO
 			}
 		} else {
@@ -87,6 +88,7 @@ func (a *App) rootCommand() *cobra.Command {
 	root.SetErr(a.errOut)
 	root.SetVersionTemplate("judgment-pack {{.Version}}\n")
 	root.CompletionOptions.DisableDefaultCmd = true
+	root.PersistentFlags().BoolVar(&a.pretty, "pretty", false, "indent JSON output")
 	root.AddCommand(a.versionCommand(), a.specCommand())
 	return root
 }
@@ -121,7 +123,7 @@ func (a *App) versionCommand() *cobra.Command {
 			}
 			output := describe.Runtime(set, "version")
 			if format == "json" {
-				if err := writeJSON(a.out, output); err != nil {
+				if err := a.writeJSON(output); err != nil {
 					return &handledExit{code: result.ExitIO}
 				}
 			} else {
@@ -160,6 +162,9 @@ func (a *App) validateCommand() *cobra.Command {
 			}
 			data, err := a.readPack(args[0], maxBytes)
 			if err != nil {
+				if errors.Is(err, fssecure.ErrTooLarge) {
+					return a.operational("spec validate", format, result.ExitIO, "JPS-RESOURCE-INPUT-BYTE-LIMIT", fmt.Sprintf("Input exceeds the %d-byte limit.", maxBytes))
+				}
 				return a.operational("spec validate", format, result.ExitIO, "JPS-INPUT-READ", "Input could not be read as one bounded regular file or standard input stream.")
 			}
 			output, operational := a.engine.Validate(data, validation.Options{Through: through, Limits: carrier.DefaultLimits()})
@@ -291,7 +296,7 @@ func readBounded(reader io.Reader, limit int64) ([]byte, error) {
 		return nil, err
 	}
 	if int64(len(data)) > limit {
-		return nil, errors.New("input exceeds byte limit")
+		return nil, fssecure.ErrTooLarge
 	}
 	return data, nil
 }

internal/cli/app_test.go

@@ -194,3 +194,77 @@ func first(value string, count int) string {
 	}
 	return value[:count]
 }
+
+func TestPrettyFlagIndentsJSONOutput(t *testing.T) {
+	set, err := artifacts.Load(artifacts.DraftVersion)
+	if err != nil {
+		t.Fatal(err)
+	}
+	valid, err := set.Case("valid/minimal-literal.json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	validPath := writeFixture(t, valid)
+	code, stdout, stderr := runTest(t, []string{"spec", "validate", validPath, "--format", "json", "--pretty"}, "")
+	if code != 0 || stderr != "" {
+		t.Fatalf("exit=%d stderr=%q", code, stderr)
+	}
+	if strings.Count(stdout, "\n") < 2 {
+		t.Fatalf("pretty JSON should span multiple lines: %q", stdout)
+	}
+	var output map[string]any
+	if err := json.Unmarshal([]byte(stdout), &output); err != nil {
+		t.Fatal(err)
+	}
+	if output["status"] != "valid" {
+		t.Fatalf("unexpected result: %#v", output)
+	}
+}
+
+func TestInputSizeAndReadErrorsUseDistinctCodes(t *testing.T) {
+	set, err := artifacts.Load(artifacts.DraftVersion)
+	if err != nil {
+		t.Fatal(err)
+	}
+	valid, err := set.Case("valid/minimal-literal.json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	validPath := writeFixture(t, valid)
+
+	// A byte limit below the document size reports a dedicated resource code,
+	// distinct from a generic unreadable-input failure.
+	code, stdout, stderr := runTest(t, []string{"spec", "validate", validPath, "--max-bytes", "10", "--format", "json"}, "")
+	if code != 4 || stderr != "" {
+		t.Fatalf("oversized: exit=%d stderr=%q", code, stderr)
+	}
+	if got := diagnosticCode(t, stdout); got != "JPS-RESOURCE-INPUT-BYTE-LIMIT" {
+		t.Fatalf("oversized input should report the byte-limit code, got %q", got)
+	}
+
+	// A genuinely unreadable input keeps the generic read code.
+	missing := filepath.Join(t.TempDir(), "absent.json")
+	code, stdout, stderr = runTest(t, []string{"spec", "validate", missing, "--format", "json"}, "")
+	if code != 4 || stderr != "" {
+		t.Fatalf("missing: exit=%d stderr=%q", code, stderr)
+	}
+	if got := diagnosticCode(t, stdout); got != "JPS-INPUT-READ" {
+		t.Fatalf("missing input should report the generic read code, got %q", got)
+	}
+}
+
+func diagnosticCode(t *testing.T, jsonOutput string) string {
+	t.Helper()
+	var output struct {
+		Diagnostics []struct {
+			Code string `json:"code"`
+		} `json:"diagnostics"`
+	}
+	if err := json.Unmarshal([]byte(jsonOutput), &output); err != nil {
+		t.Fatalf("decode: %v (output %q)", err, jsonOutput)
+	}
+	if len(output.Diagnostics) != 1 {
+		t.Fatalf("expected exactly one diagnostic, got %d: %q", len(output.Diagnostics), jsonOutput)
+	}
+	return output.Diagnostics[0].Code
+}

internal/cli/render.go

@@ -10,19 +10,26 @@ import (
 	"github.com/Judgment-Pack/judgment-pack-runtime/internal/result"
 )
 
-func writeJSON(writer io.Writer, value any) error {
+func writeJSON(writer io.Writer, value any, pretty bool) error {
 	encoder := json.NewEncoder(writer)
 	encoder.SetEscapeHTML(false)
+	if pretty {
+		encoder.SetIndent("", "  ")
+	}
 	return encoder.Encode(value)
 }
 
+func (a *App) writeJSON(value any) error {
+	return writeJSON(a.out, value, a.pretty)
+}
+
 func (a *App) operational(command, format string, exitCode int, code, message string) error {
 	status := "error"
 	if exitCode == result.ExitUnsupported {
 		status = "unsupported"
 	}
 	if format == "json" {
-		if err := writeJSON(a.out, result.NewOperationalResult(command, status, code, message)); err != nil {
+		if err := a.writeJSON(result.NewOperationalResult(command, status, code, message)); err != nil {
 			return &handledExit{code: result.ExitIO}
 		}
 	} else if status == "unsupported" {
@@ -35,7 +42,7 @@ func (a *App) operational(command, format string, exitCode int, code, message st
 
 func (a *App) renderValidation(format string, output result.Validation) error {
 	if format == "json" {
-		return writeJSON(a.out, output)
+		return a.writeJSON(output)
 	}
 	if output.Status == "valid" {
 		if output.ValidationScope.FullDocumentConformance {
@@ -61,7 +68,7 @@ func (a *App) renderValidation(format string, output result.Validation) error {
 
 func (a *App) renderSuite(format string, output result.Suite) error {
 	if format == "json" {
-		return writeJSON(a.out, output)
+		return a.writeJSON(output)
 	}
 	if output.Status == "valid" {
 		fmt.Fprintf(a.out, "passed: %d/%d conformance cases matched expectations\n", output.Summary.Passed, output.Summary.Total)
@@ -83,7 +90,7 @@ func (a *App) renderSuite(format string, output result.Suite) error {
 
 func (a *App) renderSchema(format string, output result.Schema) error {
 	if format == "json" {
-		return writeJSON(a.out, output)
+		return a.writeJSON(output)
 	}
 	fmt.Fprintf(a.out, "JPS schema %s\n", display.Sanitize(output.SpecVersion))
 	fmt.Fprintf(a.out, "id: %s\n", display.Sanitize(output.SchemaID))

internal/fssecure/read.go

@@ -5,6 +5,11 @@ import (
 	"io"
 )
 
+// ErrTooLarge reports that a read stopped because the input exceeded the
+// caller's byte limit, rather than being missing or not a regular file. It lets
+// a caller distinguish a size-limit failure from a generic read failure.
+var ErrTooLarge = errors.New("input exceeds the byte limit")
+
 func ReadRegular(filePath string, limit int64) ([]byte, error) {
 	file, err := openRegular(filePath)
 	if err != nil {
@@ -16,14 +21,14 @@ func ReadRegular(filePath string, limit int64) ([]byte, error) {
 		return nil, errors.New("path is not a regular file")
 	}
 	if info.Size() > limit {
-		return nil, errors.New("file exceeds limit")
+		return nil, ErrTooLarge
 	}
 	data, err := io.ReadAll(io.LimitReader(file, limit+1))
 	if err != nil {
 		return nil, err
 	}
 	if int64(len(data)) > limit {
-		return nil, errors.New("file exceeds limit")
+		return nil, ErrTooLarge
 	}
 	return data, nil
 }