diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 4ae9eb4..0b0b7e3 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -56,7 +56,7 @@ - [ ] KPI updates include version bumps plus math/clamp/rationale updates in `internal_docs/schema/kpi*.yaml` - [ ] Relevant entry added to `MIGRATIONS.schema.md` or `MIGRATIONS.kpi.md` - [ ] `python scripts/schema_guard.py --strict --json` passes locally -- [ ] Docs under `docs/app/reference/*` updated when new fields/KPIs surface to users +- [ ] Docs under `docs/reference/*` updated when new fields/KPIs surface to users ## Observability & Safety diff --git a/README.md b/README.md index c2a5a1f..9a7e8d5 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,7 @@ episode_id = ns.solve( # Governance vetoes the risky write → # run automatically emits interrupt + checkpoint # UI shows: run is paused, waiting on you +``` ## How it works @@ -158,9 +159,14 @@ except NoesisVeto as veto: ## Workspace verification -## Artifact contract +Verification compares pre/post workspace snapshots and records the result in the episode artifacts. -Every episode writes a sealed artifact pack: +```python +verify = ( + ns.file_exists("canary-rollout.json"), + ns.file_contains("canary-rollout.json", "canary: true"), + ns.only_modified(["canary-rollout.json"]), +) episode_id = ns.solve( "Update config", @@ -170,6 +176,12 @@ episode_id = ns.solve( ) ``` +Verification produces `snapshots/pre.json` and `snapshots/post.json` when a workspace is provided. + +## Artifact contract + +Terminal episodes are sealed with `final.json` and `manifest.json`. Paused runs keep the mutable lifecycle open and may only expose `events.jsonl`, `learn.jsonl`, and checkpoint metadata until continuation completes. + ## Pause, checkpoint, and continue ```python @@ -185,11 +197,6 @@ episode_id = ns.solve( ns.only_modified(["canary-rollout.json"]), ), ) -``` - -Verification produces `snapshots/pre.json` and `snapshots/post.json` and records verification state in the episode artifacts. - ---- interrupt_id = ns.interrupt(episode_id, reason="awaiting approval") checkpoint = ns.checkpoint(episode_id, caused_by=interrupt_id) @@ -201,9 +208,6 @@ episode_id = ns.resume_run( checkpoint_id=checkpoint["checkpoint_id"], using=my_graph, ) - -episode_id = session.solve(task="...", using=lambda: my_agent()) -restore() ``` Rule of thumb: @@ -240,10 +244,11 @@ uv run python examples/demo.py ## Docs and links -- Artifacts guide: `docs/artifacts/state.md` -- Schema index: `docs/app/reference/schema-index.mdx` -- CLI reference: `docs/app/reference/cli/page.mdx` -- Quickstart guide: `docs/app/guides/quickstart/page.mdx` +- Artifacts guide: `docs/explanation/artifacts.mdx` +- State schema: `docs/reference/state.mdx` +- Events schema: `docs/reference/events.mdx` +- CLI reference: `docs/reference/cli.mdx` +- Quickstart guide: `docs/quickstart.mdx` - Examples: `examples/README.md` ## Status diff --git a/docs/docs.json b/docs/docs.json index 955c7ba..d978f68 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -48,7 +48,11 @@ { "group": "Tutorials", "pages": [ + "tutorials/first-episode", + "tutorials/first-policy", + "tutorials/guarded-langgraph-agent", "tutorials/hello-episode", + "tutorials/incident-triage", "tutorials/langgraph-episode", "tutorials/governed-side-effects", "tutorials/trace-based-evals" diff --git a/docs/explanation/determinism.mdx b/docs/explanation/determinism.mdx index 749844d..be12989 100644 --- a/docs/explanation/determinism.mdx +++ b/docs/explanation/determinism.mdx @@ -49,38 +49,43 @@ This locks timestamps, random numbers, and episode timestamps; the configuration ## Replay and comparison -Compare two episodes to check for drift. Ignore timing fields to focus on behavior: +Use the shipped diagnostics command to compare two episode directories: -```python -import noesis as ns +```bash +noesis diagnostics replay \ + .noesis/episodes/ \ + .noesis/episodes/ +``` +The command is backed by `noesis.diagnostics.compare_runs()` and reports `NO DRIFT` or `DRIFT detected`. -def compare_episodes(ep_a: str, ep_b: str) -> dict: - events_a = list(ns.events.read(ep_a)) - events_b = list(ns.events.read(ep_b)) +Use JSON output for CI logs or custom gates: - diffs = { - "event_count_match": len(events_a) == len(events_b), - "phase_sequence_match": True, - "payload_diffs": [], - } +```bash +noesis diagnostics replay \ + .noesis/episodes/ \ + .noesis/episodes/ \ + --json +``` - for e1, e2 in zip(events_a, events_b): - if e1["phase"] != e2["phase"]: - diffs["phase_sequence_match"] = False - break +Exit codes: - # Ignore timing differences; compare payloads to catch behavioral drift - p1 = dict(e1.get("payload", {})) - p2 = dict(e2.get("payload", {})) +| Code | Meaning | +| --- | --- | +| `0` | No drift found. | +| `1` | Drift found. Inspect `mismatches[].artifact` and `mismatches[].detail`. | +| `2` | Usage error, usually a missing run directory argument. | - if p1 != p2: - diffs["payload_diffs"].append( - {"phase": e1["phase"], "diff": {"expected": p1, "actual": p2}} - ) +What replay compares: - return diffs -``` +- file set parity between the two episode directories +- byte-identical `summary.json` +- normalized `state.json`, `manifest.json`, and `events.jsonl` +- action-candidate invariants on both event logs + +Timestamps, generated event IDs, snapshot payloads, and manifest hashes are normalized where appropriate so replay focuses on behavioral drift instead of observational noise. + +Run replay after changes to planners, governance, action gating, event emission, state projection, artifact sealing, or deterministic clocks/IDs. ## Golden tests (pytest) @@ -166,26 +171,19 @@ Avoid deterministic mode for: ## CI validation -Add deterministic tests and replay checks to CI: - -```yaml -# .github/workflows/determinism.yml -name: Determinism Validation +The repository replay gate runs in `.github/workflows/determinism-replay.yml`: -on: [push, pull_request] +```bash +uv run python scripts/replay_gate.py +``` -jobs: - golden-tests: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v4 +The gate currently checks golden pairs under: - - name: Run golden tests - run: uv run pytest tests/golden/ -v +- `tests/golden/deterministic_run` +- `tests/golden/veto_enforce` +- `tests/golden/llm_real` +- `tests/golden/adr_008/*` - - name: Check replay stability - run: uv run python scripts/validate_replay.py -``` +The veto golden ensures governance veto semantics remain deterministic: no `act` event is emitted, terminal status stays `vetoed`, and action-candidate/governance lineage remains intact. -The replay gate includes multiple goldens; one is an enforce-veto run (`tests/golden/veto_enforce/run_{a,b}`) to ensure governance veto semantics remain deterministic (no Act events, terminate status `vetoed`, direction_blocked + governance lineage intact). +Set `NOESIS_REPLAY_GATE_DEBUG=1` when you need full fixture-hygiene errors instead of the compact CI failure message. diff --git a/docs/reference/cli.mdx b/docs/reference/cli.mdx index 0305b08..0393f21 100644 --- a/docs/reference/cli.mdx +++ b/docs/reference/cli.mdx @@ -145,6 +145,23 @@ noesis diagnostics replay .noesis/episodes/ep_a .noesis/episodes/ep_b noesis validate-ports ``` +`diagnostics replay` compares two episode directories using the same drift logic as the CI replay gate: + +```bash +noesis diagnostics replay .noesis/episodes/ep_a .noesis/episodes/ep_b --json +``` + +Replay returns `0` for `NO_DRIFT`, `1` for behavioral drift, and `2` for missing replay arguments. JSON mode emits: + +```json +{ + "status": "DRIFT", + "mismatches": [ + {"artifact": "events.jsonl", "detail": "structural drift"} + ] +} +``` + Trace integrity is fail-closed. Commands that read `events.jsonl` (for example `noesis events` and `noesis view`) return a non-zero exit code when the event log is corrupted, rather than skipping bad lines. ### Other utility commands @@ -227,4 +244,20 @@ Fix: install UI dependencies, then re-run `noesis browse`. Cause: unsupported `--cli-version` value. -Fix: use `cli/1.0` or omit `--cli-version`. +Fix: use a `cli/1.x` value such as `cli/1.1`, or omit `--cli-version`. + +### `DRIFT detected` from `diagnostics replay` + +Cause: the compared episode directories differ after replay normalization, or an action-candidate invariant failed. + +Fix: + +- inspect each reported `artifact` / `detail` pair +- compare `events.jsonl` phase order and payloads first +- re-run `uv run python scripts/replay_gate.py` if the drift affects checked-in golden fixtures + +### `usage: noesis diagnostics replay ` + +Cause: replay mode needs two episode directories. + +Fix: pass concrete run directories, not only episode IDs. diff --git a/docs/reference/llms-txt.mdx b/docs/reference/llms-txt.mdx new file mode 100644 index 0000000..7ab15de --- /dev/null +++ b/docs/reference/llms-txt.mdx @@ -0,0 +1,43 @@ +--- +title: "llms.txt" +description: "How Noesis exposes LLM-friendly documentation indexes." +--- + +Noesis publishes a root `llms.txt` index so assistants and search tools can discover the most useful documentation pages without crawling the whole site. + +## Files + +| File | Purpose | +| --- | --- | +| `llms.txt` | Curated index of public documentation URLs and short descriptions. | +| `llms-full.txt` | Optional expanded documentation bundle. This repository currently ships the curated `llms.txt`; generate a full bundle from the docs site if your publishing pipeline needs one. | + +## Source of truth + +The checked-in `llms.txt` file is maintained by hand alongside the Mintlify pages under `docs/`. + +When changing docs navigation, keep these files aligned: + +- `docs/docs.json`: sidebar and site navigation. +- `llms.txt`: assistant-friendly public index. +- `docs/reference/llms-txt.mdx`: this reference page, linked from both navigation and `llms.txt`. + +## Update checklist + +1. Add or rename the MDX page under `docs/`. +2. Update `docs/docs.json` if the page should appear in navigation. +3. Update `llms.txt` only for public pages that should be easy for assistants to find. +4. Remove links for deleted pages instead of leaving placeholders. +5. Run the docs link checker when Mintlify is available: + +```bash +cd docs +mintlify broken-links +``` + +## Constraints + +- Use canonical public URLs rooted at `https://docs.noesis.systems`. +- Keep descriptions short enough to scan. +- Do not include private run artifacts, unpublished internal paths, API keys, or workspace-specific data. +- Verify every linked page exists under `docs/` before adding it to `llms.txt`. diff --git a/llms.txt b/llms.txt index df1f502..67dd406 100644 --- a/llms.txt +++ b/llms.txt @@ -21,8 +21,6 @@ - [Human-in-the-loop](https://docs.noesis.systems/guides/human-in-the-loop): How to add human approval workflows to your Noēsis episodes. - [Integrate adapters](https://docs.noesis.systems/guides/integrate-adapters): How to connect Noēsis to LangGraph, CrewAI, or your own agent runtime. - [Write intuition policies](https://docs.noesis.systems/guides/write-policies): How to create, test, and deploy policies that guide and guard your agent episodes. -- [Use Noēsis in Cursor](https://docs.noesis.systems/guides/use-noesis-in-cursor): Run episodes, add guardrails, and inspect traces without leaving your editor. -- [Use GitHub Copilot with Noēsis](https://docs.noesis.systems/guides/use-copilot): Pasteable Noēsis cheat sheet for Copilot to guide users through the API and CLI without hallucinations. ## Explanation - [Core concepts](https://docs.noesis.systems/explanation/core-concepts): Understand the fundamental building blocks of Noēsis: episodes, faculties, and artifacts.