Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
- [ ] KPI updates include version bumps plus math/clamp/rationale updates in `internal_docs/schema/kpi*.yaml`
- [ ] Relevant entry added to `MIGRATIONS.schema.md` or `MIGRATIONS.kpi.md`
- [ ] `python scripts/schema_guard.py --strict --json` passes locally
- [ ] Docs under `docs/app/reference/*` updated when new fields/KPIs surface to users
- [ ] Docs under `docs/reference/*` updated when new fields/KPIs surface to users

## Observability & Safety

Expand Down
33 changes: 19 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ episode_id = ns.solve(
# Governance vetoes the risky write →
# run automatically emits interrupt + checkpoint
# UI shows: run is paused, waiting on you
```

## How it works

Expand Down Expand Up @@ -158,9 +159,14 @@ except NoesisVeto as veto:

## Workspace verification

## Artifact contract
Verification compares pre/post workspace snapshots and records the result in the episode artifacts.

Every episode writes a sealed artifact pack:
```python
verify = (
ns.file_exists("canary-rollout.json"),
ns.file_contains("canary-rollout.json", "canary: true"),
ns.only_modified(["canary-rollout.json"]),
)

episode_id = ns.solve(
"Update config",
Expand All @@ -170,6 +176,12 @@ episode_id = ns.solve(
)
```

Verification produces `snapshots/pre.json` and `snapshots/post.json` when a workspace is provided.

## Artifact contract

Terminal episodes are sealed with `final.json` and `manifest.json`. Paused runs keep the mutable lifecycle open and may only expose `events.jsonl`, `learn.jsonl`, and checkpoint metadata until continuation completes.

## Pause, checkpoint, and continue

```python
Expand All @@ -185,11 +197,6 @@ episode_id = ns.solve(
ns.only_modified(["canary-rollout.json"]),
),
)
```

Verification produces `snapshots/pre.json` and `snapshots/post.json` and records verification state in the episode artifacts.

---

interrupt_id = ns.interrupt(episode_id, reason="awaiting approval")
checkpoint = ns.checkpoint(episode_id, caused_by=interrupt_id)
Expand All @@ -201,9 +208,6 @@ episode_id = ns.resume_run(
checkpoint_id=checkpoint["checkpoint_id"],
using=my_graph,
)

episode_id = session.solve(task="...", using=lambda: my_agent())
restore()
```

Rule of thumb:
Expand Down Expand Up @@ -240,10 +244,11 @@ uv run python examples/demo.py

## Docs and links

- Artifacts guide: `docs/artifacts/state.md`
- Schema index: `docs/app/reference/schema-index.mdx`
- CLI reference: `docs/app/reference/cli/page.mdx`
- Quickstart guide: `docs/app/guides/quickstart/page.mdx`
- Artifacts guide: `docs/explanation/artifacts.mdx`
- State schema: `docs/reference/state.mdx`
- Events schema: `docs/reference/events.mdx`
- CLI reference: `docs/reference/cli.mdx`
- Quickstart guide: `docs/quickstart.mdx`
- Examples: `examples/README.md`

## Status
Expand Down
4 changes: 4 additions & 0 deletions docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,11 @@
{
"group": "Tutorials",
"pages": [
"tutorials/first-episode",
"tutorials/first-policy",
"tutorials/guarded-langgraph-agent",
"tutorials/hello-episode",
"tutorials/incident-triage",
"tutorials/langgraph-episode",
"tutorials/governed-side-effects",
"tutorials/trace-based-evals"
Expand Down
82 changes: 40 additions & 42 deletions docs/explanation/determinism.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,38 +49,43 @@ This locks timestamps, random numbers, and episode timestamps; the configuration

## Replay and comparison

Compare two episodes to check for drift. Ignore timing fields to focus on behavior:
Use the shipped diagnostics command to compare two episode directories:

```python
import noesis as ns
```bash
noesis diagnostics replay \
.noesis/episodes/<episode-a> \
.noesis/episodes/<episode-b>
```

The command is backed by `noesis.diagnostics.compare_runs()` and reports `NO DRIFT` or `DRIFT detected`.

def compare_episodes(ep_a: str, ep_b: str) -> dict:
events_a = list(ns.events.read(ep_a))
events_b = list(ns.events.read(ep_b))
Use JSON output for CI logs or custom gates:

diffs = {
"event_count_match": len(events_a) == len(events_b),
"phase_sequence_match": True,
"payload_diffs": [],
}
```bash
noesis diagnostics replay \
.noesis/episodes/<episode-a> \
.noesis/episodes/<episode-b> \
--json
```

for e1, e2 in zip(events_a, events_b):
if e1["phase"] != e2["phase"]:
diffs["phase_sequence_match"] = False
break
Exit codes:

# Ignore timing differences; compare payloads to catch behavioral drift
p1 = dict(e1.get("payload", {}))
p2 = dict(e2.get("payload", {}))
| Code | Meaning |
| --- | --- |
| `0` | No drift found. |
| `1` | Drift found. Inspect `mismatches[].artifact` and `mismatches[].detail`. |
| `2` | Usage error, usually a missing run directory argument. |

if p1 != p2:
diffs["payload_diffs"].append(
{"phase": e1["phase"], "diff": {"expected": p1, "actual": p2}}
)
What replay compares:

return diffs
```
- file set parity between the two episode directories
- byte-identical `summary.json`
- normalized `state.json`, `manifest.json`, and `events.jsonl`
- action-candidate invariants on both event logs

Timestamps, generated event IDs, snapshot payloads, and manifest hashes are normalized where appropriate so replay focuses on behavioral drift instead of observational noise.

Run replay after changes to planners, governance, action gating, event emission, state projection, artifact sealing, or deterministic clocks/IDs.

## Golden tests (pytest)

Expand Down Expand Up @@ -166,26 +171,19 @@ Avoid deterministic mode for:

## CI validation

Add deterministic tests and replay checks to CI:

```yaml
# .github/workflows/determinism.yml
name: Determinism Validation
The repository replay gate runs in `.github/workflows/determinism-replay.yml`:

on: [push, pull_request]
```bash
uv run python scripts/replay_gate.py
```

jobs:
golden-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
The gate currently checks golden pairs under:

- name: Run golden tests
run: uv run pytest tests/golden/ -v
- `tests/golden/deterministic_run`
- `tests/golden/veto_enforce`
- `tests/golden/llm_real`
- `tests/golden/adr_008/*`

- name: Check replay stability
run: uv run python scripts/validate_replay.py
```
The veto golden ensures governance veto semantics remain deterministic: no `act` event is emitted, terminal status stays `vetoed`, and action-candidate/governance lineage remains intact.

The replay gate includes multiple goldens; one is an enforce-veto run (`tests/golden/veto_enforce/run_{a,b}`) to ensure governance veto semantics remain deterministic (no Act events, terminate status `vetoed`, direction_blocked + governance lineage intact).
Set `NOESIS_REPLAY_GATE_DEBUG=1` when you need full fixture-hygiene errors instead of the compact CI failure message.
35 changes: 34 additions & 1 deletion docs/reference/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,23 @@ noesis diagnostics replay .noesis/episodes/ep_a .noesis/episodes/ep_b
noesis validate-ports
```

`diagnostics replay` compares two episode directories using the same drift logic as the CI replay gate:

```bash
noesis diagnostics replay .noesis/episodes/ep_a .noesis/episodes/ep_b --json
```

Replay returns `0` for `NO_DRIFT`, `1` for behavioral drift, and `2` for missing replay arguments. JSON mode emits:

```json
{
"status": "DRIFT",
"mismatches": [
{"artifact": "events.jsonl", "detail": "structural drift"}
]
}
```

Trace integrity is fail-closed. Commands that read `events.jsonl` (for example `noesis events` and `noesis view`) return a non-zero exit code when the event log is corrupted, rather than skipping bad lines.

### Other utility commands
Expand Down Expand Up @@ -227,4 +244,20 @@ Fix: install UI dependencies, then re-run `noesis browse`.

Cause: unsupported `--cli-version` value.

Fix: use `cli/1.0` or omit `--cli-version`.
Fix: use a `cli/1.x` value such as `cli/1.1`, or omit `--cli-version`.

### `DRIFT detected` from `diagnostics replay`

Cause: the compared episode directories differ after replay normalization, or an action-candidate invariant failed.

Fix:

- inspect each reported `artifact` / `detail` pair
- compare `events.jsonl` phase order and payloads first
- re-run `uv run python scripts/replay_gate.py` if the drift affects checked-in golden fixtures

### `usage: noesis diagnostics replay <run_a_dir> <run_b_dir>`

Cause: replay mode needs two episode directories.

Fix: pass concrete run directories, not only episode IDs.
43 changes: 43 additions & 0 deletions docs/reference/llms-txt.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: "llms.txt"
description: "How Noesis exposes LLM-friendly documentation indexes."
---

Noesis publishes a root `llms.txt` index so assistants and search tools can discover the most useful documentation pages without crawling the whole site.

## Files

| File | Purpose |
| --- | --- |
| `llms.txt` | Curated index of public documentation URLs and short descriptions. |
| `llms-full.txt` | Optional expanded documentation bundle. This repository currently ships the curated `llms.txt`; generate a full bundle from the docs site if your publishing pipeline needs one. |

## Source of truth

The checked-in `llms.txt` file is maintained by hand alongside the Mintlify pages under `docs/`.

When changing docs navigation, keep these files aligned:

- `docs/docs.json`: sidebar and site navigation.
- `llms.txt`: assistant-friendly public index.
- `docs/reference/llms-txt.mdx`: this reference page, linked from both navigation and `llms.txt`.

## Update checklist

1. Add or rename the MDX page under `docs/`.
2. Update `docs/docs.json` if the page should appear in navigation.
3. Update `llms.txt` only for public pages that should be easy for assistants to find.
4. Remove links for deleted pages instead of leaving placeholders.
5. Run the docs link checker when Mintlify is available:

```bash
cd docs
mintlify broken-links
```

## Constraints

- Use canonical public URLs rooted at `https://docs.noesis.systems`.
- Keep descriptions short enough to scan.
- Do not include private run artifacts, unpublished internal paths, API keys, or workspace-specific data.
- Verify every linked page exists under `docs/` before adding it to `llms.txt`.
2 changes: 0 additions & 2 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@
- [Human-in-the-loop](https://docs.noesis.systems/guides/human-in-the-loop): How to add human approval workflows to your Noēsis episodes.
- [Integrate adapters](https://docs.noesis.systems/guides/integrate-adapters): How to connect Noēsis to LangGraph, CrewAI, or your own agent runtime.
- [Write intuition policies](https://docs.noesis.systems/guides/write-policies): How to create, test, and deploy policies that guide and guard your agent episodes.
- [Use Noēsis in Cursor](https://docs.noesis.systems/guides/use-noesis-in-cursor): Run episodes, add guardrails, and inspect traces without leaving your editor.
- [Use GitHub Copilot with Noēsis](https://docs.noesis.systems/guides/use-copilot): Pasteable Noēsis cheat sheet for Copilot to guide users through the API and CLI without hallucinations.

## Explanation
- [Core concepts](https://docs.noesis.systems/explanation/core-concepts): Understand the fundamental building blocks of Noēsis: episodes, faculties, and artifacts.
Expand Down
Loading