From 3b056e85615aa47c97ca704677f2cbc291a4e9ce Mon Sep 17 00:00:00 2001 From: Rita Agafonova <36133358+rita-aga@users.noreply.github.com> Date: Wed, 26 Aug 2026 12:40:10 -0400 Subject: [PATCH 1/5] feat: repo contract - distilled AGENTS.md, REVIEW.md, .agents/ neutral layout, verify-temper skill, cloud wiring - AGENTS.md distilled to kernel-specific substance (global boilerplate now arrives from the stack layer); stale PM-app workflow, .progress/ references, and the outdated two-Codex review bar removed - CLAUDE.md -> symlink to AGENTS.md (was a hand-mirrored 203-line copy) - REVIEW.md: five kernel passes (DST, invariants/spec contract, authz fail-closed, TigerStyle bounds, dependency discipline) - .agents/ is the neutral source of truth: agents/ (3 reviewers), commands/, skills/ (verify-temper new; desloppify + temper-agent.md moved as-is, dedup vs stack pending); .claude/* and .cursor/skills are symlinks - .claude/settings.json + hooks/global-context.sh + global.md: cloud sessions get the global layer only when ~/.claude/CLAUDE.md is absent Proof run of verify-temper follows on this branch before the PR leaves draft. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VLPhB9kjLeE48kLUyAXXq2 --- .../agents/alignment-reviewer.md | 0 {.claude => .agents}/agents/code-reviewer.md | 0 {.claude => .agents}/agents/dst-reviewer.md | 0 {.claude => .agents}/commands/temper-user.md | 0 .../skills/desloppify/SKILL.md | 0 {.claude => .agents}/skills/temper-agent.md | 0 .agents/skills/verify-temper/SKILL.md | 41 ++++ .../skills/verify-temper/features/README.md | 7 + .../verify-temper/features/dst-proof.md | 16 ++ .../verify-temper/features/serve-and-odata.md | 18 ++ .../verify-temper/features/spec-cascade.md | 16 ++ .claude/agents | 1 + .claude/commands | 1 + .claude/global.md | 165 +++++++++++++ .claude/hooks/global-context.sh | 7 + .claude/settings.json | 22 +- .claude/skills | 1 + .cursor/skills | 1 + AGENTS.md | 229 ++++-------------- CLAUDE.md | 204 +--------------- REVIEW.md | 26 ++ 21 files changed, 371 insertions(+), 384 deletions(-) rename {.claude => .agents}/agents/alignment-reviewer.md (100%) rename {.claude => .agents}/agents/code-reviewer.md (100%) rename {.claude => .agents}/agents/dst-reviewer.md (100%) rename {.claude => .agents}/commands/temper-user.md (100%) rename {.claude => .agents}/skills/desloppify/SKILL.md (100%) rename {.claude => .agents}/skills/temper-agent.md (100%) create mode 100644 .agents/skills/verify-temper/SKILL.md create mode 100644 .agents/skills/verify-temper/features/README.md create mode 100644 .agents/skills/verify-temper/features/dst-proof.md create mode 100644 .agents/skills/verify-temper/features/serve-and-odata.md create mode 100644 .agents/skills/verify-temper/features/spec-cascade.md create mode 120000 .claude/agents create mode 120000 .claude/commands create mode 100644 .claude/global.md create mode 100755 .claude/hooks/global-context.sh create mode 120000 .claude/skills create mode 120000 .cursor/skills mode change 100644 => 120000 CLAUDE.md create mode 100644 REVIEW.md diff --git a/.claude/agents/alignment-reviewer.md b/.agents/agents/alignment-reviewer.md similarity index 100% rename from .claude/agents/alignment-reviewer.md rename to .agents/agents/alignment-reviewer.md diff --git a/.claude/agents/code-reviewer.md b/.agents/agents/code-reviewer.md similarity index 100% rename from .claude/agents/code-reviewer.md rename to .agents/agents/code-reviewer.md diff --git a/.claude/agents/dst-reviewer.md b/.agents/agents/dst-reviewer.md similarity index 100% rename from .claude/agents/dst-reviewer.md rename to .agents/agents/dst-reviewer.md diff --git a/.claude/commands/temper-user.md b/.agents/commands/temper-user.md similarity index 100% rename from .claude/commands/temper-user.md rename to .agents/commands/temper-user.md diff --git a/.claude/skills/desloppify/SKILL.md b/.agents/skills/desloppify/SKILL.md similarity index 100% rename from .claude/skills/desloppify/SKILL.md rename to .agents/skills/desloppify/SKILL.md diff --git a/.claude/skills/temper-agent.md b/.agents/skills/temper-agent.md similarity index 100% rename from .claude/skills/temper-agent.md rename to .agents/skills/temper-agent.md diff --git a/.agents/skills/verify-temper/SKILL.md b/.agents/skills/verify-temper/SKILL.md new file mode 100644 index 000000000..a05bb3a51 --- /dev/null +++ b/.agents/skills/verify-temper/SKILL.md @@ -0,0 +1,41 @@ +--- +name: verify-temper +description: Launch the Temper kernel locally and verify a change end to end - build, serve, drive the OData surface, run the verification cascade and DST proof. Use before calling any temper change done. +--- + +# Verify temper + +## Launch + +```bash +cargo build -p temper-cli # first build is long (~29 crates) +cargo run -p temper-cli -- serve --port 3600 # pick a free port; capture the PID +``` + +Ready when `GET http://localhost:3600/observe/health` returns 200 and `GET /tdata/$metadata` returns CSDL XML. No env vars are required for a local scratch serve. + +**ISOLATE**: run from your worktree so state lands in the worktree, not in a shared checkout. Never point at another session's data directory. + +## Doctor + +- Build fails on `edition 2024`: rustup update; rust-version is 1.85. +- Port in use: pick another; read the real port from the serve log, not the flag you passed. +- Serve exits immediately: read the log bottom-up; a spec that fails the L0-L3 cascade at bootstrap names itself. + +## Verify a change + +Pick the feature file matching what changed (see `features/`): + +- `features/serve-and-odata.md` - boot, health, CSDL metadata, entity reads +- `features/spec-cascade.md` - L0-L3 verification of `.ioa.toml` changes +- `features/dst-proof.md` - deterministic simulation, seeded reproduction + +Always finish with the suite for the crates you touched (`cargo test -p `), then `cargo test --workspace` before push (the pre-push hook runs it anyway). + +## Evidence + +Capture into `/tmp/verify-temper//`: the health response, the metadata head, cascade output, and the DST test result. Hand commands + outputs to the PR, do not assert. + +## Teardown + +Kill only the serve PID you captured at spawn. Never kill by pattern - other temper worktrees run servers on this machine. diff --git a/.agents/skills/verify-temper/features/README.md b/.agents/skills/verify-temper/features/README.md new file mode 100644 index 000000000..a70a56d93 --- /dev/null +++ b/.agents/skills/verify-temper/features/README.md @@ -0,0 +1,7 @@ +# Feature map + +| Feature | File | Drive when you changed | +|---|---|---| +| Serve + OData | serve-and-odata.md | server, routes, stores, platform bootstrap | +| Spec cascade | spec-cascade.md | any `.ioa.toml`, temper-spec, temper-verify | +| DST proof | dst-proof.md | temper-runtime, temper-jit, temper-server sim paths | diff --git a/.agents/skills/verify-temper/features/dst-proof.md b/.agents/skills/verify-temper/features/dst-proof.md new file mode 100644 index 000000000..822768af5 --- /dev/null +++ b/.agents/skills/verify-temper/features/dst-proof.md @@ -0,0 +1,16 @@ +# Deterministic simulation proof + +## Sub-features +Seeded runs, fault injection, invariant checking, reproduction. + +## Driving it +```bash +cargo test -p temper-platform --test platform_e2e_dst # E2E shared-registry proof +cargo test -p temper-runtime # sim runtime suite +``` + +## What proves it +The DST suite passes, and a failure reproduces under the same seed (the failing output names the seed; rerunning with it must fail identically). For changed sim-visible code, the determinism guard (`scripts/check-determinism.sh`) reports no new violations. + +## Gotchas +Code that passes tests can still break determinism (wall clock, HashMap order) - the guard and the DST reviewer ruleset in `.agents/agents/dst-reviewer.md` are the check, not the test suite alone. diff --git a/.agents/skills/verify-temper/features/serve-and-odata.md b/.agents/skills/verify-temper/features/serve-and-odata.md new file mode 100644 index 000000000..c60261372 --- /dev/null +++ b/.agents/skills/verify-temper/features/serve-and-odata.md @@ -0,0 +1,18 @@ +# Serve and the OData surface + +## Sub-features +Boot, health, CSDL metadata, entity-set reads, action dispatch. + +## Driving it +```bash +cargo run -p temper-cli -- serve --port 3600 # capture PID +curl -sf http://localhost:3600/observe/health +curl -sf http://localhost:3600/tdata/$metadata | head -c 400 # CSDL XML +``` +Read an entity set named in the metadata; dispatch an action via `POST /tdata/('')/Temper.` with `X-Tenant-Id`. + +## What proves it +Health 200 with a live process; metadata is CSDL XML listing the bootstrapped entity types; an entity read returns `@odata.context`. A dispatch is proven by reading the entity back and seeing the state move - a 200 on dispatch alone is not a transition. + +## Gotchas +The serve log at bootstrap lists every spec that loaded; a missing entity set means its spec failed the cascade - read the log, not the route. diff --git a/.agents/skills/verify-temper/features/spec-cascade.md b/.agents/skills/verify-temper/features/spec-cascade.md new file mode 100644 index 000000000..62cd34248 --- /dev/null +++ b/.agents/skills/verify-temper/features/spec-cascade.md @@ -0,0 +1,16 @@ +# Spec verification cascade (L0-L3) + +## Sub-features +IOA parse, TransitionTable build, model checking, DST invariants. + +## Driving it +```bash +cargo run -p temper-cli -- verify # single spec +scripts/verify-cascade.sh # all spec dirs, results in .cascade-results/ +``` + +## What proves it +The cascade reports each level passed for the changed spec. An edit that adds a state or action must show the new element in the pass output. A deliberately broken guard must FAIL the cascade - if it passes, that is a finding in the verifier, not a success. + +## Gotchas +The `.claude` hook runs this automatically on `.ioa.toml` edits and BLOCKS on failure; running it yourself first avoids losing the edit loop. `.cascade-results/` is local state, never committed. diff --git a/.claude/agents b/.claude/agents new file mode 120000 index 000000000..4c8a5fc93 --- /dev/null +++ b/.claude/agents @@ -0,0 +1 @@ +../.agents/agents \ No newline at end of file diff --git a/.claude/commands b/.claude/commands new file mode 120000 index 000000000..1ea3574a1 --- /dev/null +++ b/.claude/commands @@ -0,0 +1 @@ +../.agents/commands \ No newline at end of file diff --git a/.claude/global.md b/.claude/global.md new file mode 100644 index 000000000..3ae1b835f --- /dev/null +++ b/.claude/global.md @@ -0,0 +1,165 @@ +# Global Agent Instructions + +> **Source of truth:** `arni-labs/stack` **- edit** `AGENTS.md` **here and only here.** `~/AGENTS.md`, `~/.claude/CLAUDE.md`, and `~/.grok/AGENTS.md` are symlinks maintained by `sync.sh`. Distilled from recurring corrections across ~136 working sessions - don't make Rita repeat them. + +## A note from Rita + +- Ambitious ideas, simple systems, software that feels obvious. +- **Do not preserve complexity just because it already exists. Do not introduce machinery because it looks architecturally impressive.** Find the real constraint, then fight for the smallest design that makes correct behavior unsurprising. +- Fight scope creep. Prefer less code, better code, readable code. Deletion is a feature. +- If a rule here fights the task in front of you, say so loudly and get sign-off before breaking it. + +## Encode lessons - corrections become constraints + +- Nothing here updates automatically. When Rita corrects you, or something happens that was not supposed to happen: fix the instance, then **encode the class - as part of the task, not as an optional follow-up**. +- Climb the ladder; encode at the strongest rung that fits: + 1. Make it impossible - types, state machine, permissions, removed capability. + 2. A lint or test in CI that fails loudly. + 3. A hook or script. + 4. A rule in prose - routed to its source of truth, never only to your local context: + - Global behavior or a fact about Rita's environment -> THIS file, in the `arni-labs/stack` repo on GitHub. The repo is the source of truth, not any local path. On Rita's laptop it is checked out at `~/Development/stack` and the deployed copies are symlinks into it, so edit there, commit, push. Anywhere that checkout does not exist (cloud sessions, other machines), clone `arni-labs/stack` and push, or open a PR against it - editing only a synced or vendored copy changes nothing. Vendored copies in repos (e.g. `.claude/global.md`) update on the next sync pass; note in the completion report when one is stale. + - How to use a specific tool, CLI, or service -> that tool's skill in `stack/skills/` (create a small one if none exists). + - True only in one repo -> that repo's AGENTS.md. + Prose is the weakest rung; prefer moving existing rules UP the ladder over adding more text. +- A correction that dies in the session is the failure mode this section exists to prevent. "I was wrong about a capability" (e.g. "X full-archive search is blocked" when the bearer token supports it) is a lesson like any other: route it per the list above. +- Threshold: the same mistake twice MUST be encoded. Once is a judgment call. +- Edits to this file are their own commit and are announced in the completion report - never silent. (Planned: eval-gated via `evals/`.) + +## Terminology + +- **TBD** + +## Repo map + +- **temper** - the Temper kernel (Rust). Kernel code ONLY. +- **temperpaw** - the agent OS on Temper (OpenPaw rebranded - same project). Agents, os-apps, skills. +- **genesis** - formerly "temper-git". Never invent names. +- **katagami** - the design commons. Lives on Genesis (`katagami/katagami-commons`, `katagami-curation`), GitHub mirrors. +- Crucible, Paw, etc. are components, not brands. Unsure what something is called? Ask - do not coin names. +- App/agent logic found inside the kernel (or vice versa): flag it, don't silently relocate it. + +## Working discipline + +- Never edit a primary checkout. If one is dirty, detached, or off its default branch: leave it, say so. +- All work in a worktree off up-to-date main, branch `/` (`claude/...`, `codex/...`). +- Before mutating anything, state repo, worktree, and branch. Never assume `origin` = GitHub; name the remote host. +- Draft PR as soon as changes begin. Exactly ONE PR per repo per effort. Multi-repo merge order: Genesis -> Temper -> TemperPaw -> Katagami. +- Genesis is source of truth for temperpaw/katagami apps: merging to GitHub is not done - publish to Genesis and verify the installed pinned ref (`owner/app@hash`). On divergence, Genesis wins. +- Conventional commits (`feat:`/`fix:`/`refactor:`/`docs:`/`chore:`); professional language everywhere; commit and push after implementing planned changes. +- GitHub ops run as **rita-aga** (`gh auth switch -u rita-aga`); if a push is denied for the wrong user, set `git config credential.helper '!gh auth git-credential'` in that repo. +- Clean up worktrees older than ~3 days. New repos default private. + +## Plan first + +- Brainstorm and align BEFORE implementing, even with a detailed spec. Intent unclear? Interview Rita - ask, don't bake assumptions in. +- Plan the implementation itself, not meta-work. A plan keeps its "what we are addressing" and "expected end state" through revisions - never shrinks to just review comments. +- The artifact chain: intent -> spec (a significant effort's spec IS its RFC - one per effort, outsider-readable) -> plan -> decision log -> PR. Intent arrives from anywhere - chat, Slack, a dictated note; it is SYNCED to a Linear issue for tracking, but Linear is where intents are tracked, not where they come from. ADRs per repo, written when a decision outlives the effort. Deferred ideas recorded in the RFC, not dropped. +- Core or invariant-bearing systems (kernel, platform, backend state machines): the spec has three expressions of ONE contract: a readable spec.md, a formal model (TLA+-class or a state model), and the DST harness's invariants. All three say the same thing; keep them aligned the derived-artifact way (same pattern as rendering a Temper app's policy sheet from its spec - ARN-404; temper already enforces `from_tla_source()` for this). +- Agent suggests, human decides. Present options and tradeoffs in plain language. +- Use your intelligence, not a canned skill, for judgment work. + +## Decision log - record calls as you make them + +- You cannot know everything ahead of time. Every non-obvious mid-implementation call is recorded AT THE MOMENT it is made in your working notes for the task, never reconstructed at report time. The plan is what you intended before building; the decision log is what reality forced during it. A plan deviation is always an entry, written when you deviate. +- Each entry self-contained for a reader with zero session context: **Decision** (one plain sentence) / **Came up because** / **Options** (incl. the rejected default) / **Chose A over B because** (what was gained, what was given up) / **Where** (file:line, commit, PR). +- The durable home is the PR body's `## Decisions & Tradeoffs` section - the completion report and PR carry the list verbatim, copied, not re-summarized. No decisions? Say exactly that; silence is not an answer. A decision whose consequences outlive the effort becomes an ADR in the repo's `docs/adrs/`. + +## Pull requests + +- Titles: Conventional Commits in plain language - `fix(web): new threads no longer spike CPU`. A human reads the title and knows what changed. +- Body opens with **the problem in a sentence or two, then how you fixed it** - upfront, concise, legible to someone arriving cold. No boilerplate sections, no restating the diff. +- Then `## Decisions & Tradeoffs` (verbatim from the decision log) and evidence per the Definition of Done (UI changes: before/after images). +- End the body with the model and harness that did the work - the review panel rule depends on knowing the author. +- **One PR per repo per effort stays the rule.** Do not split PRs finer for its own sake. + +## Definition of Done + +"Tests pass" is not done. Done means: + +1. Full objective implemented - no deferred core, no punted hard part. +2. **Live local e2e run - gates EVERY prod deploy.** Build it, start it, drive the changed flow yourself: open the link, click through, curl and read the response. Green suites do not substitute. Prod is never the testing ground. Test the production shape - real policy/config, generic verbs too (PATCH/PUT/DELETE, not only the named action). +3. Merge + deploy, then verify live in the deployed system. Find out exactly what is deployed - never guess. Use **Datadog** for prod verification and diagnosis. +4. **Three fresh-context reviews - one per harness, panel FIXED regardless of author** (the authoring harness gets no extra vote): + - **Grok** - grok CLI, Grok 4.6, highest effort it exposes + - **Codex** - `codex exec --model gpt-5.6-sol -c model_reasoning_effort="xhigh" --sandbox read-only` + - **Claude Code** - fresh Fable subagent, effort xhigh + Plus **Greptile** on every PR (`@greptile review`). Each reviewer gets diff + goal + permission to be adversarial; wants severity, `file:line`, concrete failure scenario. FIX EVERYTHING found - including critiques of your fixes - re-verify, report per reviewer. Two agreeing never excuses the third. The `interrogate` skill orchestrates this panel and the synthesis - use it where available; these rules bind either way. *(OPEN: trio is laptop-bound; off-laptop path unsolved - ARN-405.)* +5. Hand over evidence: PR links, merge commits, deploy links, live test commands + results, residual risks. + +Never report something as deployed, running, or visible without verifying it yourself in that environment - a handed-off link that doesn't open is a failed task. When iterating on a deployed artifact, deploy the latest and re-verify. + +## Root cause & scope integrity + +- **DO NOT PUNT.** No narrowed scope, no deferred hard part, no "phase 0" that skips the objective. Address ALL review feedback, not a subset. +- **No band-aids.** No temporary fixes, no fallbacks kept "for compatibility", no backward compatibility for bad implementations. Fix fundamentally. Never remove capabilities that currently work. +- "Keeps happening" or "didn't use to happen": find WHAT CHANGED, fix the root cause, explain the causal story. Consecutive patches that each break something else = stop and diagnose. +- Fix classes of problems, not the two instances in front of you. Prefer event-driven over polling. + +## Groundedness & communication + +- **Never invent.** Verify every factual claim against code or source before writing it. Show derivations for numbers. Treat year-old knowledge as stale until re-verified. +- Answer the question actually asked - concise, simple words, why-it-matters. If Rita re-asks, your previous answer missed: address the question, don't rephrase the miss. +- Show the real artifact (code, page, link) - open results in the browser instead of dumping code in chat. +- **Surface every error, failure, and policy denial to the human channel.** Silent failure is itself a bug. +- Rita often dictates: parse typos charitably; her directive usually follows "`--`" after a pasted transcript. +- Docs voice: down to earth, no drama, no literary devices, no vanity metrics, don't lead with jargon. Content removed in an earlier edit stays removed. +- Skills and any adopted material follow the same voice: highly readable, concise, no metaphors, no invented vocabulary or personas. LLM-flavored writing is a defect. + +## Models - route by characteristic failure, not just smarts + + +| model | smarts | characteristic failure | strongest at | +| -------- | ------- | ------------------------------------ | --------------------------------------------------------------- | +| Fable 5 | highest | token-hungry at high effort | judgment, taste, hard problems unsupervised, final review | +| Opus 5 | high | sloppy - quality drifts | breadth and volume; never the last set of hands | +| Sol 5.6 | high | rigid; scope-creeps, overcomplicates | precise well-specced implementation; adversarial review | +| Grok 4.6 | high | underdoes - may stop short | subtraction: deleting, simplifying, "should this exist at all?" | + + +- Judge the output, not the price tag: if a cheaper model's output misses the bar, redo with a smarter one without asking. +- Simplification and removal passes: prefer Grok - it is the best enforcer of "do not preserve complexity". +- Pair Sol with the no-machinery rule; pair Opus with a review; anything user-facing ends with Fable-grade taste. + +## Design output + +Any styled artifact (HTML reports, dashboards, pages) follows the Katagami way: + +- Use the linked design language's DESIGN.md tokens exactly. Standing default: [Galley](https://katagami.ai/language/en-019f2e78-8711-7072-b72d-200b095d9a51). The linked language always wins. +- Bright and clean; <=3 accents; no grey borders; no emoji on buttons; light default; body 17px+; high contrast; responsive; titles never stuck to container tops. +- Diagrams are real architecture diagrams (inline SVG, explainer under each), not tables or charts. +- One canonical living HTML deliverable; preserve the previous version before restyling; verify the rendered page yourself before handover. + +## Temper operating layer + +> *UNDER CONSTRUCTION - likely outdated; will be rewritten as the universal setup lands. Treat as directional, not gospel.* + +- When the Temper MCP is connected, stateful work management belongs in Temper. Read the `temper-agent` skill for the API. Probe once (`temper.specs`); if no PM app (403/404), say so once and fall back to `.progress/` - do not retry-loop. +- Fallback planning (no Temper PM): the Linear issue carries the plan summary; write the working plan in your notes before code. (Older repos have `.vision/` and `.progress/` folders - read them for history, do not extend them.) +- Cedar denial -> surface the pending decision to the human; they approve in Observe; you retry. +- WASM integration rules (every Temper app): a module fired by a transition never dispatches transitions itself - sequencing belongs to the state machine; and one integration means one concern - a module doing several things in sequence gets broken into transitions with one module each. +- During execution: update progress each phase; re-read the plan before major decisions; save findings every ~2 significant operations. Before done: verify quality (no hacks or placeholders), check the Definition of Done, update status, commit and push. + +## Testing + +- Write tests for new code - features and fixes alike. Red-green TDD. +- Run locally before pushing; failing tests get fixed before push, not leaned on CI. +- Tests alone never satisfy the Definition of Done - live e2e and QA is also required. + +## Session journaling (second brain) + +> *UNDER CONSTRUCTION - the sync path is laptop-local and will not survive the universal setup; rules below hold until replaced.* + +- Any real work is a goal. One OKF file per goal in `~/Development/aya/brain/journal/` (local-only, never committed). Frontmatter: `type: work_summary, id, domain, goal, objective, key_result, runtime, started_at, state`. `objective`/`key_result` are OKR CODES (`O9`, `KR9.2`) or `none` - never sentences. +- Record milestones AS THEY HAPPEN, in numbers - counts, %, pass/fail, links - never "made good progress". Real challenges only; none is a valid answer. +- ALWAYS close: set `state` done/abandoned + `ended_at`; close the current goal before switching to different work. Then push: `python3 ~/.config/aya/sync-journal.py ` (offline fallback: leave the file). +- Honesty over flourish - this is real visibility, including for Rita's partner. + +## Linear sync (always) + +Linear is the source of truth for what is being worked on (team Arni-build, prefix ARN, `mcp__linear__*`). Every harness syncs at three moments - consulting Linear is part of starting, not a follow-up: + +- **Discovering** -> search first; update the existing issue or create one in the right project. +- **Starting** -> find/create the issue, comment, assign, move to In Progress. +- **Completing** -> Done only with every artifact attached: commits, PRs, proof, deploy links; residual risks as linked issues. +Non-negotiables: no duplicate issues; integrate additively; completion always links commits and PRs; Linear never drifts from reality. Tools unavailable? Say so - never pretend the sync happened. + diff --git a/.claude/hooks/global-context.sh b/.claude/hooks/global-context.sh new file mode 100755 index 000000000..0ca063cba --- /dev/null +++ b/.claude/hooks/global-context.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# Injects the global instruction layer ONLY where the user-level file is absent +# (cloud VMs). Locally ~/.claude/CLAUDE.md exists, so this emits nothing and +# the global layer loads once, from home. No duplication either way. +if [ ! -f "$HOME/.claude/CLAUDE.md" ]; then + cat "$(dirname "$0")/../global.md" +fi diff --git a/.claude/settings.json b/.claude/settings.json index 334102a49..f24c2ca8f 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -10,6 +10,15 @@ "timeout": 5 } ] + }, + { + "matcher": "startup|resume|clear|compact", + "hooks": [ + { + "type": "command", + "command": "bash \"$CLAUDE_PROJECT_DIR\"/.claude/hooks/global-context.sh" + } + ] } ], "PreToolUse": [ @@ -92,5 +101,16 @@ ] } ] + }, + "extraKnownMarketplaces": { + "arni-stack": { + "source": { + "source": "github", + "repo": "arni-labs/stack" + } + } + }, + "enabledPlugins": { + "stack@arni-stack": true } -} +} \ No newline at end of file diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 000000000..2b7a412b8 --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../.agents/skills \ No newline at end of file diff --git a/.cursor/skills b/.cursor/skills new file mode 120000 index 000000000..2b7a412b8 --- /dev/null +++ b/.cursor/skills @@ -0,0 +1 @@ +../.agents/skills \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 66154240a..a3c204741 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,203 +1,72 @@ -# Temper — Codex Project Guide +# Temper -> Synchronized with `CLAUDE.md` (Claude Code) — same rules, agent-specific paths. When you change one, mirror the other. Global rules live in `~/AGENTS.md` / `~/.claude/CLAUDE.md`; this file adds what is temper-specific. +The Temper kernel: specs, verification, runtime, server, stores, observe, evolution. **Kernel code only** - app and agent logic belongs in temperpaw; if you find it mixed in here, flag it, do not silently relocate it. Global rules (worktrees, PRs, reviews, Definition of Done) come from the stack layer; this file is what is temper-specific. -## Scope & Naming +Worktrees live under `~/Development/temper-worktrees/` (the primary at `~/Development/temper` is bare on purpose). -- This repo is the **Temper kernel only**: specs, verification, runtime, server, stores, observe, evolution. App and agent logic belongs in **temperpaw** (the agent OS — OpenPaw rebranded, same project); the Temper-native git server is **genesis** (formerly temper-git); the design commons is **katagami**. If you find code mixing those concerns into the kernel, **flag it — don't silently relocate it**. -- Never invent product names. When unsure what something is called, ask. -- Professional language everywhere — branch names, commits, PR text, docs. No internal slang ("deslop" etc.). +## Repo map -## Working Discipline +- `crates/temper-spec` - how defined: IOA + CSDL parsers +- `crates/temper-jit` - what is defined: `TransitionTable`, `Effect` +- `crates/temper-verify` - how verified: L0-L3 cascade (own `ModelEffect`) +- `crates/temper-runtime` - runtime (replaceable): mailbox, sim, `EventStore` trait; WASM host in `temper-wasm` + `temper-wasm-sdk` +- `crates/temper-store-turso` - default store; `-sim`, `-postgres`, `-redis` (journal only) are plugs +- `crates/temper-server` - the mix, read last: HTTP + EntityActor + `apply_effects` + registry +- Control plane: `temper-odata`, `temper-authz`, `temper-observe`, `temper-evolution`, `temper-ots`, `temper-platform`, `temper-cli`, `temper-mcp`, `temper-sdk`, `temper-sandbox` +- Default serve is own-Rust actors; `--actor-runtime postgres` is an adapter, not a second kernel. `apply_effect` must become one function; today it is three. +- Verification: `.agents/skills/verify-temper/` - the verification skill and feature map +- Architecture Decision Records: `docs/adrs/` (template at `docs/adrs/TEMPLATE.md`). A material architecture change gets its ADR before code - required for new features, new integrations, multi-crate changes, new patterns; not for bug fixes, single-file refactors, docs, or test additions. -- **The primary checkout at `~/Development/temper` is bare on purpose** — you cannot and must not work there. Create a worktree branched from up-to-date `main` (`codex/`), state which repo/worktree/branch you're on before mutating anything, open a **draft PR as soon as changes begin**, and keep it to **one PR per repo per effort**. -- **DO NOT PUNT.** No "phase 0" that defers the hard part; no band-aid or local/temporary fixes; no backward compatibility for bad implementations (but never drop working capabilities). Address ALL review findings, then re-run review until clean. -- **Definition of done**: implement fully → run live end-to-end locally → merge + deploy → verify live in the deployed system, using **Datadog** for verification and diagnosis → hand over PR links, merge commits, deployment links, live test results, residual risks. Tests passing alone is not done. -- Brainstorm and align before implementing, even from a detailed handoff. Rita's handoffs follow: Objective / Why This Matters / Relevant PRs / Hot Files / Acceptance Criteria / Must Start With ADR / Suggested Validation / Copy-Paste Prompt — honor that structure, including the ADR-first step. -- Explain tradeoffs in plain language before merge; merged-but-unexplained tradeoffs get reopened. +## Specs -## IMPORTANT: Use Temper for Your Own Work +- IOA TOML (`.ioa.toml`) is the spec format; `TransitionTable::from_ioa_source()` in production. TLA+ is legacy: `from_tla_source()` is `#[cfg(test)]` only. +- Specs are generated from conversation, never hand-written; code derived from specs is regenerable. +- Framework code must not hardcode entity-specific state names. Domain invariants come from the spec's `[[invariant]]` sections. The verification cascade gates every spec change. +- Data contract is CSDL (`*.csdl.xml`); a running server publishes it at `GET /tdata/$metadata`. -**You are an agent running on Temper. This is your operating layer — not an optional tool.** When a Temper server is running (`mcp__temper__execute` is available), ALL your stateful work — planning, task tracking, progress, knowledge — MUST flow through Temper entities, not markdown files. +## Deterministic simulation (DST) -**Specifically:** -- **Planning and task tracking** → Use the PM OS app (`await temper.install_app("{tenant}", "project-management")` if not installed). Create Issues, use `BeginPlanning` → `WritePlan` → `ApprovePlan` → `StartWork` flow. Do NOT use `.progress/` files when Temper is available. -- **Building apps for users** → Use the Temper App Builder skill (`.claude/skills/temper-developer.md`). Workflow: interview → generate IOA specs + CSDL → verify → deploy. Use `/temper-developer`. -- **Any MCP tool call** → ALWAYS read `.claude/skills/temper-agent.md` first. It has the exact Python API, spec format, and governance flow. +In simulation-visible crates (`temper-runtime`, `temper-jit`, `temper-server`): -**Cedar governance applies to everything.** If denied, surface the pending decision to the human. They approve via the Observe UI. You poll and retry. +- `sim_now()` / `sim_uuid()`, never wall clock or random UUIDs +- `BTreeMap`/`BTreeSet`, never `HashMap`/`HashSet` (deterministic iteration) +- No `std::thread::spawn`, `rayon`, multi-threaded `tokio::spawn` - single-threaded actor model +- No `std::fs`, `std::net`, `std::env::var` - I/O behind traits +- No `static mut`, `lazy_static!`, `thread_local!` - state through actor context +- No `chrono::Utc::now()`, `std::thread::sleep()`, `OsRng`, `getrandom` - simulated time, seeded PRNG +- `SimActorHandler::spec_invariants()` auto-checks `[[invariant]]` sections +- `// determinism-ok` suppresses guard false positives +- Full ruleset: `.agents/agents/dst-reviewer.md` -Use `/temper-agent` or read `.claude/skills/temper-agent.md` for the full API reference. +## Multi-tenancy and identity -## PM App: Planning/Planned Workflow +- `SpecRegistry` maps (TenantId, EntityType) to specs + TransitionTable; single-tenant is `TenantId::default()` = "default". Pass the tenant explicitly; never assume a hardcoded one. +- Agent identity registry: the platform verifies the `agent_type` claim and sets `agentTypeVerified` on the Cedar principal. Policies treat unverified types as untrusted. -Issues in the Project Management OS App use a **Planning → Planned** phase with role separation: +## Rust conventions -1. **Supervisor** triages issue, assigns a planner (`AssignPlanner`) and implementer (`Assign`) -2. **Planner** calls `BeginPlanning` → drafts plan via `WritePlan` (plan + acceptance_criteria) -3. **Supervisor/Human** reviews and calls `ApprovePlan` → issue moves to `Planned` -4. **Implementer** calls `StartWork` (requires approved plan + assignee) → implements → `SubmitForReview` +- Edition 2024, rust-version 1.85. `gen` is a reserved keyword. +- Files over 500 lines split into directory modules. All pub items documented. +- TigerStyle: bounded mailboxes, pre/post assertions at function entry and return, budgets not limits, fail fast on invariant violation, no silent failures. +- `temper-jit` must not depend on `temper-verify` in `[dependencies]`. Production binaries must not pull in `stateright` or `proptest`. -**Role separation is Cedar-enforced:** -- Planner cannot approve their own plan (`resource.PlannerId != principal.id`) -- Implementer cannot approve their own review (`resource.AssigneeId != principal.id`) -- Only supervisors/humans can triage, approve plans, and approve reviews +## Commands -**Agent API:** Read `.claude/skills/temper-agent.md` for the full Temper Python API including planning methods (`begin_planning`, `write_plan`, `approve_plan`). - -## What is Temper? -A conversational application platform. Developers describe what they want through conversation — the system generates specs, verifies them, and deploys entity actors. End users interact through a separate production chat. Unmet user intents feed back through the Evolution Engine for developer approval. - -## The Vision -``` -Developer Chat: "I want a project management tool" - → System interviews developer about entities, states, actions, guards - → Generates IOA specs + CSDL + Cedar from conversation - → Runs 3-level verification cascade - → Hot-deploys entity actors + OData API - -Production Chat: end users operate the app - → Unmet intents → trajectory spans → ClickHouse → Sentinel - → O-Record → I-Record → Developer reviews → D-Record → spec change -``` - -Two separated contexts: Developer Chat (design-time, can modify specs) and Production Chat (runtime, operates within specs). The developer holds the approval gate for all behavioral changes. - -## Architecture - -Six seams. ★ = replaceable plug. `apply_effect` must stay one function; today it is three. - -- **How defined** — `temper-spec` (IOA + CSDL parsers) -- **What is defined** — `temper-jit` (`TransitionTable`, `Effect`). Apply is *not* here yet. -- **How verified** — `temper-verify` (L0–L3). Has its own `ModelEffect`. -- **Control plane** — `temper-odata`, `temper-authz`, `temper-observe`, `temper-evolution`, `temper-ots`, `temper-platform`, `temper-cli`, `temper-mcp`, `temper-sdk`, `temper-sandbox` -- **Runtime ★** — `temper-runtime` (mailbox, sim, `EventStore` trait). Optional PG adapter: `temper-actor-runtime` (not default serve). WASM host: `temper-wasm` + `temper-wasm-sdk`. `temper-agents` is Gabriele’s PG-path agent chain; not default serve. -- **Store ★** — `temper-store-turso` (default), `temper-store-sim`, `temper-store-postgres`, `temper-store-redis` (event journal only) -- **Mix — read last** — `temper-server` (HTTP + EntityActor + `apply_effects` + registry) - -Default serve is own-Rust actors. `--actor-runtime postgres` is an adapter, not a second kernel. - -## Architecture Decision Records (ADRs) - -**Every significant implementation MUST start with an ADR as the first step.** Before writing any code, create `docs/adrs/NNNN-short-title.md` following the template at `docs/adrs/TEMPLATE.md`. Required for new features, architectural changes, new integrations, multi-crate changes, or new patterns. Not required for bug fixes, single-file refactors, doc changes, or test additions. - -## Agent Identity Registry - -Agent types are registered in the platform's identity registry. When an agent connects, the platform verifies its `agent_type` claim against the registry and sets the `agentTypeVerified` attribute on the Cedar principal: - -- **`agentTypeVerified: true`** — the agent's claimed type matches a registered entry; Cedar policies can trust scope decisions based on `agent_type` -- **`agentTypeVerified: false`** — self-asserted type with no registry match; Cedar policies should treat as untrusted - -Cedar policies reference this attribute to distinguish verified agents from unverified ones (e.g., only verified agents can approve plans). - -## Issue Pickup Before Work - -**You MUST pick up or create a Temper issue before making code changes.** The `check-issue-pickup.sh` hook (advisory, PreToolUse on Write|Edit) checks for a session marker at `/tmp/temper-harness/{project_hash}/{session_id}/issue-active`. The marker is created when you transition an issue to Planning or InProgress via `BeginPlanning` or `StartWork`. If you see the advisory warning, stop and pick up an issue first. - -**Reality check**: probe the connected server once (`await temper.specs("")`). If it returns 403/404 or the PM app isn't deployed, say so once, fall back to a `.progress/` plan file, and continue — do not retry-loop or block the task on Temper plumbing. Long-running plans can also be recorded as Temper **goals** when the goals system is deployed. - -## Key Rules - -### Platform Philosophy -- Specs are generated from conversation, never hand-written by developers -- Code is derived from specs and is regenerable -- Framework code must NOT hardcode entity-specific state names -- Domain invariants come from the spec's [[invariant]] sections -- Trajectory intelligence captures every unmet intent -- The verification cascade gates every spec change - -### Spec Format -- I/O Automaton TOML (`.ioa.toml`) is the primary spec format -- Use `TransitionTable::from_ioa_source()` in production -- TLA+ is legacy — `from_tla_source()` is `#[cfg(test)]` only - -### Multi-Tenancy -- SpecRegistry maps (TenantId, EntityType) → specs + TransitionTable -- Postgres/Redis are tenant-scoped -- Single-tenant uses TenantId::default() = "default" -- **Active agent tenant:** use the tenant configured for the current project or server. Pass it explicitly to agent MCP calls instead of assuming a hardcoded repository tenant. - -### Deterministic Simulation (FoundationDB/TigerBeetle Standards) -In simulation-visible crates (temper-runtime, temper-jit, temper-server): -- Use `sim_now()` / `sim_uuid()` instead of wall clock / random UUIDs -- Use `BTreeMap`/`BTreeSet` not `HashMap`/`HashSet` — deterministic iteration order -- No `std::thread::spawn`, `rayon`, or multi-threaded `tokio::spawn` — single-threaded actor model -- No `std::fs`, `std::net`, `std::env::var` — abstract all I/O behind traits -- No `static mut`, `lazy_static!`, `thread_local!` — pass state through actor context -- No `chrono::Utc::now()`, `std::thread::sleep()` — use simulated time -- No `OsRng`, `getrandom` — use seeded PRNG -- `SimActorHandler::spec_invariants()` auto-checks [[invariant]] sections -- Add `// determinism-ok` to suppress false positives in the determinism guard -- See `.claude/agents/dst-reviewer.md` for the full DST compliance ruleset - -### Dependency Discipline -- `temper-jit` must NOT depend on `temper-verify` in `[dependencies]` -- Production binaries must not pull in `stateright` or `proptest` - -### Rust Conventions -- Edition 2024, rust-version 1.85 -- `gen` is a reserved keyword — never use as variable name -- Files > 500 lines must be split into directory modules -- All pub items must have doc comments -- TigerStyle: bounded mailboxes, pre/post assertions, budgets not limits - -## Testing ```bash -cargo test --workspace # Full workspace (430+ tests) -cargo test -p temper-server # Including multi-tenant integration -cargo test -p temper-platform # Platform unit + deploy pipeline -cargo test -p temper-platform --test platform_e2e_dst # E2E shared registry proof +cargo test --workspace # full suite +cargo test -p temper-platform --test platform_e2e_dst # E2E shared-registry proof +cargo run -p temper-cli -- serve --port 3000 # HTTP server, OData API, Observe UI +scripts/setup-hooks.sh # install git hooks (pre-commit integrity, pre-push 4-gate) ``` -## Development Harness - -See `docs/HARNESS.md` for the full harness reference with diagrams. +## Enforcement hooks -### Automated Enforcement (Codex Hooks) -- **Plan Reminder** (advisory): Reminds to create a plan (Temper issue or `.progress/` fallback) before edits -- **Issue Pickup** (advisory): Warns if no active Temper issue for the session before Write|Edit -- **Spec Verification** (BLOCKING): L0-L3 cascade on every `.ioa.toml` edit -- **Determinism Guard** (BLOCKING): 25-pattern DST scan on `.rs` edits in sim-visible crates -- **Pre-Commit Review Gate** (BLOCKING): Blocks `git commit` without DST review + code review markers -- **Post-Push Marker** (advisory): Records push for session tracking -- **Session Exit Gate** (DISABLED): Can be re-enabled in settings.json Stop hook +`.claude/settings.json` wires blocking hooks: L0-L3 spec verification on every `.ioa.toml` edit, a 25-pattern determinism guard on `.rs` edits in sim-visible crates, and a pre-commit gate requiring DST-review and code-review markers (`.agents/agents/dst-reviewer.md`, `code-reviewer.md` write them on PASS). Tests run at push time, not commit time. -### Mandatory Reviews Before Commit -**You MUST run both reviews before committing any code changes:** +## Deploying spec changes -1. **DST Compliance Review** (for simulation-visible code in temper-runtime, temper-jit, temper-server): - - Invoke the DST reviewer agent (`.claude/agents/dst-reviewer.md`) - - Writes a marker file on PASS — the pre-commit gate checks for it - -2. **Code Quality Review** (for all significant changes): - - Invoke the code-reviewer agent - - Writes a marker file on PASS — the pre-commit gate checks for it - -The pre-commit gate BLOCKS `git commit` if either marker is missing. Markers are session-scoped (`/tmp/temper-harness/{project_hash}/{session_id}/`) so multiple agent sessions don't conflict. - -### Git Hooks (installed via `scripts/setup-hooks.sh`) -- **Pre-commit**: Integrity check (no TODO/unwrap), spec syntax, dep audit -- **Pre-push**: 4-gate pipeline — rustfmt, clippy, readability ratchet, full test suite - -Tests run once at push time only — not at commit or post-push. This keeps commits fast. - -## Error Handling Standards (TigerStyle) -- **Bounded mailboxes**: Every actor mailbox has a capacity limit -- **Pre-assertions**: Validate inputs at function entry (`assert!` or `debug_assert!`) -- **Post-assertions**: Validate outputs before return -- **Budgets not limits**: Express constraints as budgets that get consumed, not arbitrary limits -- **Fail fast**: If an invariant is violated, panic immediately rather than propagating corrupt state -- **No silent failures**: Every error path must be logged or propagated - -## Deployment Verification Steps -Before deploying any spec change: -1. Spec passes all 5 verification cascade levels (L0-L3) -2. TransitionTable builds successfully from verified spec +1. Spec passes the verification cascade (L0-L3) +2. TransitionTable builds from the verified spec 3. Entity actors hot-deploy without dropping existing state -4. OData endpoints respond correctly for all entity types +4. OData endpoints respond for all entity types 5. Telemetry emits WideEvents for all transitions - -## Definition of done — review bar - -- **Three independent fresh-context reviews before anything is "done".** Nothing counts as fully implemented until all three have reviewed it, each with NO prior context on the work: **Codex** (`codex exec --model gpt-5.6-sol -c model_reasoning_effort="high" --sandbox read-only`), **a second independent Codex Sol session**, and **an independent Fable** (a fresh Claude subagent, `model: fable`). Run **Greptile** on every PR too (`@greptile review` as a PR comment). Ask each for severity, `file:line`, and a concrete failure scenario, then fix everything they find — including findings that criticise your own fixes — and re-verify. They catch different classes: one finds bypass surface, one finds fail-open behaviour, one finds whether it actually runs. Two agreeing does not excuse skipping the third. -- **Test the production shape, not a convenient one.** A gate proven against a permissive or mock engine, and an end-to-end that only exercises the happy verb, are not proof — verify against the real policy/config set and probe the generic paths too (PATCH/PUT/DELETE, not only the named action). diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 695394c6a..000000000 --- a/CLAUDE.md +++ /dev/null @@ -1,203 +0,0 @@ -# Temper — Claude Code Project Guide - -> Synchronized with `AGENTS.md` (Codex) — same rules, agent-specific paths. When you change one, mirror the other. Global rules live in `~/.claude/CLAUDE.md` / `~/AGENTS.md`; this file adds what is temper-specific. - -## Scope & Naming - -- This repo is the **Temper kernel only**: specs, verification, runtime, server, stores, observe, evolution. App and agent logic belongs in **temperpaw** (the agent OS — OpenPaw rebranded, same project); the Temper-native git server is **genesis** (formerly temper-git); the design commons is **katagami**. If you find code mixing those concerns into the kernel, **flag it — don't silently relocate it**. -- Never invent product names. When unsure what something is called, ask. -- Professional language everywhere — branch names, commits, PR text, docs. No internal slang ("deslop" etc.). - -## Working Discipline - -- **The primary checkout at `~/Development/temper` is bare on purpose** — you cannot and must not work there. Create a worktree branched from up-to-date `main` (`codex/`), state which repo/worktree/branch you're on before mutating anything, open a **draft PR as soon as changes begin**, and keep it to **one PR per repo per effort**. -- **DO NOT PUNT.** No "phase 0" that defers the hard part; no band-aid or local/temporary fixes; no backward compatibility for bad implementations (but never drop working capabilities). Address ALL review findings, then re-run review until clean. -- **Definition of done**: implement fully → run live end-to-end locally → merge + deploy → verify live in the deployed system, using **Datadog** for verification and diagnosis → hand over PR links, merge commits, deployment links, live test results, residual risks. Tests passing alone is not done. -- Brainstorm and align before implementing, even from a detailed handoff. Rita's handoffs follow: Objective / Why This Matters / Relevant PRs / Hot Files / Acceptance Criteria / Must Start With ADR / Suggested Validation / Copy-Paste Prompt — honor that structure, including the ADR-first step. -- Explain tradeoffs in plain language before merge; merged-but-unexplained tradeoffs get reopened. - -## IMPORTANT: Use Temper for Your Own Work - -**You are an agent running on Temper. This is your operating layer — not an optional tool.** When a Temper server is running (`mcp__temper__execute` is available), ALL your stateful work — planning, task tracking, progress, knowledge — MUST flow through Temper entities, not markdown files. - -**Specifically:** -- **Planning and task tracking** → Use the PM OS app (`await temper.install_app("{tenant}", "project-management")` if not installed). Create Issues, use `BeginPlanning` → `WritePlan` → `ApprovePlan` → `StartWork` flow. Do NOT use `.progress/` files when Temper is available. -- **Building apps for users** → Use the Temper App Builder skill (`.claude/skills/temper-developer.md`). Workflow: interview → generate IOA specs + CSDL → verify → deploy. Use `/temper-developer`. -- **Any MCP tool call** → ALWAYS read `.claude/skills/temper-agent.md` first. It has the exact Python API, spec format, and governance flow. - -**Cedar governance applies to everything.** If denied, surface the pending decision to the human. They approve via the Observe UI. You poll and retry. - -Use `/temper-agent` or read `.claude/skills/temper-agent.md` for the full API reference. - -## PM App: Planning/Planned Workflow - -Issues in the Project Management OS App use a **Planning → Planned** phase with role separation: - -1. **Supervisor** triages issue, assigns a planner (`AssignPlanner`) and implementer (`Assign`) -2. **Planner** calls `BeginPlanning` → drafts plan via `WritePlan` (plan + acceptance_criteria) -3. **Supervisor/Human** reviews and calls `ApprovePlan` → issue moves to `Planned` -4. **Implementer** calls `StartWork` (requires approved plan + assignee) → implements → `SubmitForReview` - -**Role separation is Cedar-enforced:** -- Planner cannot approve their own plan (`resource.PlannerId != principal.id`) -- Implementer cannot approve their own review (`resource.AssigneeId != principal.id`) -- Only supervisors/humans can triage, approve plans, and approve reviews - -**Agent API:** Read `.claude/skills/temper-agent.md` for the full Temper Python API including planning methods (`begin_planning`, `write_plan`, `approve_plan`). - -## What is Temper? -A conversational application platform. Developers describe what they want through conversation — the system generates specs, verifies them, and deploys entity actors. End users interact through a separate production chat. Unmet user intents feed back through the Evolution Engine for developer approval. - -## The Vision -``` -Developer Chat: "I want a project management tool" - → System interviews developer about entities, states, actions, guards - → Generates IOA specs + CSDL + Cedar from conversation - → Runs 3-level verification cascade - → Hot-deploys entity actors + OData API - -Production Chat: end users operate the app - → Unmet intents → trajectory spans → ClickHouse → Sentinel - → O-Record → I-Record → Developer reviews → D-Record → spec change -``` - -Two separated contexts: Developer Chat (design-time, can modify specs) and Production Chat (runtime, operates within specs). The developer holds the approval gate for all behavioral changes. - -## Architecture - -Six seams. ★ = replaceable plug. `apply_effect` must stay one function; today it is three. - -- **How defined** — `temper-spec` (IOA + CSDL parsers) -- **What is defined** — `temper-jit` (`TransitionTable`, `Effect`). Apply is *not* here yet. -- **How verified** — `temper-verify` (L0–L3). Has its own `ModelEffect`. -- **Control plane** — `temper-odata`, `temper-authz`, `temper-observe`, `temper-evolution`, `temper-ots`, `temper-platform`, `temper-cli`, `temper-mcp`, `temper-sdk`, `temper-sandbox` -- **Runtime ★** — `temper-runtime` (mailbox, sim, `EventStore` trait). Optional PG adapter: `temper-actor-runtime` (not default serve). WASM host: `temper-wasm` + `temper-wasm-sdk`. `temper-agents` is Gabriele’s PG-path agent chain; not default serve. -- **Store ★** — `temper-store-turso` (default), `temper-store-sim`, `temper-store-postgres`, `temper-store-redis` (event journal only) -- **Mix — read last** — `temper-server` (HTTP + EntityActor + `apply_effects` + registry) - -Default serve is own-Rust actors. `--actor-runtime postgres` is an adapter, not a second kernel. - -## Architecture Decision Records (ADRs) - -**Every significant implementation MUST start with an ADR as the first step.** Before writing any code, create `docs/adrs/NNNN-short-title.md` following the template at `docs/adrs/TEMPLATE.md`. Required for new features, architectural changes, new integrations, multi-crate changes, or new patterns. Not required for bug fixes, single-file refactors, doc changes, or test additions. - -## Agent Identity Registry - -Agent types are registered in the platform's identity registry. When an agent connects, the platform verifies its `agent_type` claim against the registry and sets the `agentTypeVerified` attribute on the Cedar principal: - -- **`agentTypeVerified: true`** — the agent's claimed type matches a registered entry; Cedar policies can trust scope decisions based on `agent_type` -- **`agentTypeVerified: false`** — self-asserted type with no registry match; Cedar policies should treat as untrusted - -Cedar policies reference this attribute to distinguish verified agents from unverified ones (e.g., only verified `claude-code` agents can approve plans). - -## Issue Pickup Before Work - -**You MUST pick up or create a Temper issue before making code changes.** The `check-issue-pickup.sh` hook (advisory, PreToolUse on Write|Edit) checks for a session marker at `/tmp/temper-harness/{project_hash}/{session_id}/issue-active`. The marker is created when you transition an issue to Planning or InProgress via `BeginPlanning` or `StartWork`. If you see the advisory warning, stop and pick up an issue first. - -**Reality check**: probe the connected server once (`await temper.specs("")`). If it returns 403/404 or the PM app isn't deployed, say so once, fall back to a `.progress/` plan file, and continue — do not retry-loop or block the task on Temper plumbing. Long-running plans can also be recorded as Temper **goals** when the goals system is deployed. - -## Key Rules - -### Platform Philosophy -- Specs are generated from conversation, never hand-written by developers -- Code is derived from specs and is regenerable -- Framework code must NOT hardcode entity-specific state names -- Domain invariants come from the spec's [[invariant]] sections -- Trajectory intelligence captures every unmet intent -- The verification cascade gates every spec change - -### Spec Format -- I/O Automaton TOML (`.ioa.toml`) is the primary spec format -- Use `TransitionTable::from_ioa_source()` in production -- TLA+ is legacy — `from_tla_source()` is `#[cfg(test)]` only - -### Multi-Tenancy -- SpecRegistry maps (TenantId, EntityType) → specs + TransitionTable -- Postgres/Redis are tenant-scoped -- Single-tenant uses TenantId::default() = "default" -- **Active agent tenant:** use the tenant configured for the current project or server. Pass it explicitly to agent MCP calls instead of assuming a hardcoded repository tenant. - -### Deterministic Simulation (FoundationDB/TigerBeetle Standards) -In simulation-visible crates (temper-runtime, temper-jit, temper-server): -- Use `sim_now()` / `sim_uuid()` instead of wall clock / random UUIDs -- Use `BTreeMap`/`BTreeSet` not `HashMap`/`HashSet` — deterministic iteration order -- No `std::thread::spawn`, `rayon`, or multi-threaded `tokio::spawn` — single-threaded actor model -- No `std::fs`, `std::net`, `std::env::var` — abstract all I/O behind traits -- No `static mut`, `lazy_static!`, `thread_local!` — pass state through actor context -- No `chrono::Utc::now()`, `std::thread::sleep()` — use simulated time -- No `OsRng`, `getrandom` — use seeded PRNG -- `SimActorHandler::spec_invariants()` auto-checks [[invariant]] sections -- Add `// determinism-ok` to suppress false positives in the determinism guard -- See `.claude/agents/dst-reviewer.md` for the full DST compliance ruleset - -### Dependency Discipline -- `temper-jit` must NOT depend on `temper-verify` in `[dependencies]` -- Production binaries must not pull in `stateright` or `proptest` - -### Rust Conventions -- Edition 2024, rust-version 1.85 -- `gen` is a reserved keyword — never use as variable name -- Files > 500 lines must be split into directory modules -- All pub items must have doc comments -- TigerStyle: bounded mailboxes, pre/post assertions, budgets not limits - -## Testing -```bash -cargo test --workspace # Full workspace (430+ tests) -cargo test -p temper-server # Including multi-tenant integration -cargo test -p temper-platform # Platform unit + deploy pipeline -cargo test -p temper-platform --test platform_e2e_dst # E2E shared registry proof -``` - -## Development Harness - -See `docs/HARNESS.md` for the full harness reference with diagrams. - -### Automated Enforcement (Claude Code Hooks) -- **Plan Reminder** (advisory): Reminds to create a plan (Temper issue or `.progress/` fallback) before edits -- **Issue Pickup** (advisory): Warns if no active Temper issue for the session before Write|Edit -- **Spec Verification** (BLOCKING): L0-L3 cascade on every `.ioa.toml` edit -- **Determinism Guard** (BLOCKING): 25-pattern DST scan on `.rs` edits in sim-visible crates -- **Pre-Commit Review Gate** (BLOCKING): Blocks `git commit` without DST review + code review markers -- **Post-Push Marker** (advisory): Records push for session tracking -- **Session Exit Gate** (DISABLED): Can be re-enabled in settings.json Stop hook - -### Mandatory Reviews Before Commit -**You MUST run both reviews before committing any code changes:** - -1. **DST Compliance Review** (for simulation-visible code in temper-runtime, temper-jit, temper-server): - - Invoke the DST reviewer agent (`.claude/agents/dst-reviewer.md`) - - Writes a marker file on PASS — the pre-commit gate checks for it - -2. **Code Quality Review** (for all significant changes): - - Invoke the code-reviewer agent - - Writes a marker file on PASS — the pre-commit gate checks for it - -The pre-commit gate BLOCKS `git commit` if either marker is missing. Markers are session-scoped (`/tmp/temper-harness/{project_hash}/{session_id}/`) so multiple Claude sessions don't conflict. - -### Git Hooks (installed via `scripts/setup-hooks.sh`) -- **Pre-commit**: Integrity check (no TODO/unwrap), spec syntax, dep audit -- **Pre-push**: 4-gate pipeline — rustfmt, clippy, readability ratchet, full test suite - -Tests run once at push time only — not at commit or post-push. This keeps commits fast. - -## Error Handling Standards (TigerStyle) -- **Bounded mailboxes**: Every actor mailbox has a capacity limit -- **Pre-assertions**: Validate inputs at function entry (`assert!` or `debug_assert!`) -- **Post-assertions**: Validate outputs before return -- **Budgets not limits**: Express constraints as budgets that get consumed, not arbitrary limits -- **Fail fast**: If an invariant is violated, panic immediately rather than propagating corrupt state -- **No silent failures**: Every error path must be logged or propagated - -## Deployment Verification Steps -Before deploying any spec change: -1. Spec passes all 5 verification cascade levels (L0-L3) -2. TransitionTable builds successfully from verified spec -3. Entity actors hot-deploy without dropping existing state -4. OData endpoints respond correctly for all entity types -5. Telemetry emits WideEvents for all transitions - -## Definition of done — review bar - -- **Three independent fresh-context reviews before anything is "done".** Nothing counts as fully implemented until all three have reviewed it, each with NO prior context on the work: **Codex** (`codex exec --model gpt-5.6-sol -c model_reasoning_effort="high" --sandbox read-only`), **a second independent Codex Sol session**, and **an independent Fable** (a fresh Claude subagent, `model: fable`). Run **Greptile** on every PR too (`@greptile review` as a PR comment). Ask each for severity, `file:line`, and a concrete failure scenario, then fix everything they find — including findings that criticise your own fixes — and re-verify. They catch different classes: one finds bypass surface, one finds fail-open behaviour, one finds whether it actually runs. Two agreeing does not excuse skipping the third. -- **Test the production shape, not a convenient one.** A gate proven against a permissive or mock engine, and an end-to-end that only exercises the happy verb, are not proof — verify against the real policy/config set and probe the generic paths too (PATCH/PUT/DELETE, not only the named action). diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 000000000..47dc3e3d8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/REVIEW.md b/REVIEW.md new file mode 100644 index 000000000..c3f3229c8 --- /dev/null +++ b/REVIEW.md @@ -0,0 +1,26 @@ +# Reviewing temper + +Repo-specific passes on top of the global review bar. Severity + `file:line` + concrete failure scenario for every finding. + +## Pass 1: Determinism (DST) + +Any touched code in `temper-runtime`, `temper-jit`, `temper-server`: scan against the DST ruleset in `.agents/agents/dst-reviewer.md`. Wall clock, random UUIDs, HashMap iteration, thread spawns, direct I/O, and global state are all findings even when tests pass - they break seeded reproduction. Check that new `// determinism-ok` suppressions are justified, not convenient. + +## Pass 2: Invariants and the spec contract + +- A spec change and its TransitionTable behavior must say the same thing; look for code paths that bypass the table. +- New states or actions without `[[invariant]]` coverage are a finding. +- Framework code hardcoding entity-specific state names is a finding. +- `from_tla_source()` outside `#[cfg(test)]` is a finding. + +## Pass 3: Authorization fail-closed + +Every new route, action dispatch, and effect path goes through Cedar with an explicit principal. Look for: `is_system` -style bypasses, handlers that default-allow on policy load failure, tenant id taken from request data instead of authenticated context, `agentTypeVerified: false` principals granted trust. + +## Pass 4: TigerStyle bounds + +Unbounded mailboxes, queues, or retries; missing pre/post assertions on new pub functions; limits where budgets belong; error paths that swallow instead of log-or-propagate; `unwrap`/`expect` on external input. + +## Pass 5: Dependency discipline + +`temper-jit` gaining a `temper-verify` dependency, or production binaries pulling `stateright`/`proptest`, fails the review outright. From 8b7e3b2443f07abd215367c6790d5ae89c6f9943 Mon Sep 17 00:00:00 2001 From: Rita Agafonova <36133358+rita-aga@users.noreply.github.com> Date: Wed, 26 Aug 2026 12:49:13 -0400 Subject: [PATCH 2/5] fix: verify-temper skill corrected from proof run (healthz route, X-Tenant-Id on metadata, TEMPER_API_KEY before serve, verify --specs-dir) Proof: healthz 200, CSDL metadata 200 (117KB), cascade PASS on os-apps/project-management, platform_e2e_dst 6/6, keyless entity read 401 (fail-closed). Evidence: /tmp/verify-temper/2026-08-26/. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VLPhB9kjLeE48kLUyAXXq2 --- .agents/skills/verify-temper/SKILL.md | 4 +++- .agents/skills/verify-temper/features/serve-and-odata.md | 4 ++-- .agents/skills/verify-temper/features/spec-cascade.md | 2 +- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/.agents/skills/verify-temper/SKILL.md b/.agents/skills/verify-temper/SKILL.md index a05bb3a51..244a22251 100644 --- a/.agents/skills/verify-temper/SKILL.md +++ b/.agents/skills/verify-temper/SKILL.md @@ -12,7 +12,9 @@ cargo build -p temper-cli # first build is lo cargo run -p temper-cli -- serve --port 3600 # pick a free port; capture the PID ``` -Ready when `GET http://localhost:3600/observe/health` returns 200 and `GET /tdata/$metadata` returns CSDL XML. No env vars are required for a local scratch serve. +Ready when `GET http://localhost:3600/healthz` returns 200 (unauthenticated liveness; `/observe/health` is behind Observe auth and 401s) and `GET /tdata/$metadata` with `X-Tenant-Id: default` returns CSDL XML. + +For authenticated entity reads and dispatches, set `TEMPER_API_KEY=` in the environment BEFORE serve - the platform bootstraps a tenant credential from it at startup, and a keyless boot serves 401 on every governed route (that 401 is itself the fail-closed proof). **ISOLATE**: run from your worktree so state lands in the worktree, not in a shared checkout. Never point at another session's data directory. diff --git a/.agents/skills/verify-temper/features/serve-and-odata.md b/.agents/skills/verify-temper/features/serve-and-odata.md index c60261372..94c78e07f 100644 --- a/.agents/skills/verify-temper/features/serve-and-odata.md +++ b/.agents/skills/verify-temper/features/serve-and-odata.md @@ -6,8 +6,8 @@ Boot, health, CSDL metadata, entity-set reads, action dispatch. ## Driving it ```bash cargo run -p temper-cli -- serve --port 3600 # capture PID -curl -sf http://localhost:3600/observe/health -curl -sf http://localhost:3600/tdata/$metadata | head -c 400 # CSDL XML +curl -sf http://localhost:3600/healthz +curl -sf -H 'X-Tenant-Id: default' 'http://localhost:3600/tdata/$metadata' | head -c 400 # CSDL XML ``` Read an entity set named in the metadata; dispatch an action via `POST /tdata/('')/Temper.` with `X-Tenant-Id`. diff --git a/.agents/skills/verify-temper/features/spec-cascade.md b/.agents/skills/verify-temper/features/spec-cascade.md index 62cd34248..cf02d1b61 100644 --- a/.agents/skills/verify-temper/features/spec-cascade.md +++ b/.agents/skills/verify-temper/features/spec-cascade.md @@ -5,7 +5,7 @@ IOA parse, TransitionTable build, model checking, DST invariants. ## Driving it ```bash -cargo run -p temper-cli -- verify # single spec +cargo run -p temper-cli -- verify --specs-dir # takes a directory, not a file scripts/verify-cascade.sh # all spec dirs, results in .cascade-results/ ``` From 0899fec0533edbad4c9be8be041b25ae6329d3b2 Mon Sep 17 00:00:00 2001 From: Rita Agafonova <36133358+rita-aga@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:02:59 -0400 Subject: [PATCH 3/5] chore: delete unused scripts and outdated docs; vendor deterministic-simulation skill; DST-driven development in AGENTS.md and REVIEW.md - scripts deleted (no references anywhere): bench.sh, demo.sh, e2e-trusted-issuer.sh, set-branch-protection.sh, verify-all.sh - latency-observability package deleted (completed May effort): verify-latency-observability-package.sh, temper_agent_e2e_proof.py, the HTML report, and its release runbook; the operational DBM runbook and its SQL stay - docs deleted: internal/crate-refactor-plan.md and internal/GAP_TRACKER.md (stale trackers; Linear is the tracker of record), docs/proofs/, ui/landing/*.bak junk - .agents/skills/deterministic-simulation/ vendored from stack, with the wrong 'temper-dst' crate name corrected to the real locations (temper-runtime sim, temper-store-sim, platform DST suites) - AGENTS.md: DST-driven development loop (harness first, failing invariant, many seeds, regression seeds, root-cause fixes); REVIEW.md pass 1 extended with the matching findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VLPhB9kjLeE48kLUyAXXq2 --- .agents/skills/desloppify/SKILL.md | 151 - .../skills/deterministic-simulation/SKILL.md | 56 + .../010_20260225_140000_end-to-end-tryable.md | 39 - .progress/010_20260225_self_contained_mcp.md | 15 - ...11_20260226_governed_external_api_calls.md | 56 - .progress/012_20260226_agent_governance_ux.md | 34 - ...023_20260223_codebase_audit_and_cleanup.md | 134 - .../024_20260223_split_oversized_files.md | 50 - .progress/024_20260223_wasm_integration.md | 55 - .progress/026_20260223_fix_6_audit_issues.md | 15 - ..._20260224_move_mutations_out_of_observe.md | 51 - ...028_20260224_agent_backend_capabilities.md | 48 - .progress/028_20260224_wasm_persistence.md | 30 - ...20260224_cedar_authorization_for_agents.md | 65 - ...0260225_140739_agent-policy-audit-layer.md | 34 - .../031_20260225_code-mode-agent-interface.md | 41 - ...260226_122604_wasm-developer-experience.md | 31 - ...001_gepa_temperagent_failure_reflection.md | 55 - ...-24-action-triggers-merge-main-live-e2e.md | 62 - .proof/GEPA_E2E_PROOF.md | 1625 -------- .../gepa-real-claude-live-proof-2026-03-19.md | 199 - .proof/golden-soaring-cerf.md | 321 -- .proof/pawfs-context-read-plane-batch-read.md | 45 - .proof/pawfs-context-read-plane-live-e2e.md | 76 - .proof/temper-agent-e2e-proof.md | 929 ----- ...ed-evolution-repair-aware-variant-lanes.md | 55 - .proofs/2026-04-27-delta-os-app-reconcile.md | 107 - .../full-trace-and-session-perf-2026-04-23.md | 57 - .proofs/knuth-postgres-migration-full.md | 164 - .proofs/session-stall-remediation.md | 79 - AGENTS.md | 12 +- REVIEW.md | 5 + docs/internal/GAP_TRACKER.md | 68 - docs/internal/crate-refactor-plan.md | 209 -- ...8-idempotent-spec-persistence-local-e2e.md | 63 - .../runbooks/latency-observability-release.md | 377 -- docs/temper-latency-observability-report.html | 3283 ----------------- scripts/bench.sh | 50 - scripts/demo.sh | 184 - scripts/e2e-trusted-issuer.sh | 129 - scripts/set-branch-protection.sh | 52 - scripts/temper_agent_e2e_proof.py | 1424 ------- scripts/verify-all.sh | 7 - .../verify-latency-observability-package.sh | 152 - ui/landing/index.html.bak | 947 ----- ui/landing/index.html.bak2 | 930 ----- 46 files changed, 72 insertions(+), 12499 deletions(-) delete mode 100644 .agents/skills/desloppify/SKILL.md create mode 100644 .agents/skills/deterministic-simulation/SKILL.md delete mode 100644 .progress/010_20260225_140000_end-to-end-tryable.md delete mode 100644 .progress/010_20260225_self_contained_mcp.md delete mode 100644 .progress/011_20260226_governed_external_api_calls.md delete mode 100644 .progress/012_20260226_agent_governance_ux.md delete mode 100644 .progress/023_20260223_codebase_audit_and_cleanup.md delete mode 100644 .progress/024_20260223_split_oversized_files.md delete mode 100644 .progress/024_20260223_wasm_integration.md delete mode 100644 .progress/026_20260223_fix_6_audit_issues.md delete mode 100644 .progress/027_20260224_move_mutations_out_of_observe.md delete mode 100644 .progress/028_20260224_agent_backend_capabilities.md delete mode 100644 .progress/028_20260224_wasm_persistence.md delete mode 100644 .progress/029_20260224_cedar_authorization_for_agents.md delete mode 100644 .progress/030_20260225_140739_agent-policy-audit-layer.md delete mode 100644 .progress/031_20260225_code-mode-agent-interface.md delete mode 100644 .progress/032_20260226_122604_wasm-developer-experience.md delete mode 100644 .proof/001_gepa_temperagent_failure_reflection.md delete mode 100644 .proof/2026-04-24-action-triggers-merge-main-live-e2e.md delete mode 100644 .proof/GEPA_E2E_PROOF.md delete mode 100644 .proof/gepa-real-claude-live-proof-2026-03-19.md delete mode 100644 .proof/golden-soaring-cerf.md delete mode 100644 .proof/pawfs-context-read-plane-batch-read.md delete mode 100644 .proof/pawfs-context-read-plane-live-e2e.md delete mode 100644 .proof/temper-agent-e2e-proof.md delete mode 100644 .proofs/0129-directed-evolution-repair-aware-variant-lanes.md delete mode 100644 .proofs/2026-04-27-delta-os-app-reconcile.md delete mode 100644 .proofs/full-trace-and-session-perf-2026-04-23.md delete mode 100644 .proofs/knuth-postgres-migration-full.md delete mode 100644 .proofs/session-stall-remediation.md delete mode 100644 docs/internal/GAP_TRACKER.md delete mode 100644 docs/internal/crate-refactor-plan.md delete mode 100644 docs/proofs/2026-04-28-idempotent-spec-persistence-local-e2e.md delete mode 100644 docs/runbooks/latency-observability-release.md delete mode 100644 docs/temper-latency-observability-report.html delete mode 100755 scripts/bench.sh delete mode 100755 scripts/demo.sh delete mode 100755 scripts/e2e-trusted-issuer.sh delete mode 100755 scripts/set-branch-protection.sh delete mode 100644 scripts/temper_agent_e2e_proof.py delete mode 100755 scripts/verify-all.sh delete mode 100755 scripts/verify-latency-observability-package.sh delete mode 100644 ui/landing/index.html.bak delete mode 100644 ui/landing/index.html.bak2 diff --git a/.agents/skills/desloppify/SKILL.md b/.agents/skills/desloppify/SKILL.md deleted file mode 100644 index 1828136be..000000000 --- a/.agents/skills/desloppify/SKILL.md +++ /dev/null @@ -1,151 +0,0 @@ - - ---- -name: desloppify -description: > - Codebase health scanner and technical debt tracker. Use when the user asks - about code quality, technical debt, dead code, large files, god classes, - duplicate functions, code smells, naming issues, import cycles, or coupling - problems. Also use when asked for a health score, what to fix next, or to - create a cleanup plan. Supports 28 languages. -allowed-tools: Bash(desloppify *) ---- - -# Desloppify - -## 1. Your Job - -Improve code quality by maximising the **strict score** honestly. - -**The main thing you do is run `desloppify next`** — it tells you exactly what to fix and how. Fix it, resolve it, run `next` again. Keep going. - -Follow the scan output's **INSTRUCTIONS FOR AGENTS** — don't substitute your own analysis. - -## 2. The Workflow - -Two loops. The **outer loop** rescans periodically to measure progress. -The **inner loop** is where you spend most of your time: fixing issues one by one. - -### Outer loop — scan and check - -```bash -desloppify scan --path . # analyse the codebase -desloppify status # check scores — are we at target? -``` -If not at target, work the inner loop. Rescan periodically — especially after clearing a cluster or batch of related fixes. Issues cascade-resolve and new ones may surface. - -### Inner loop — fix issues - -Repeat until the queue is clear: - -``` -1. desloppify next ← tells you exactly what to fix next -2. Fix the issue in code -3. Resolve it (next shows you the exact command including required attestation) -``` - -Score may temporarily drop after fixes — cascade effects are normal, keep going. -If `next` suggests an auto-fixer, run `desloppify fix --dry-run` to preview, then apply. - -**To be strategic**, use `plan` to shape what `next` gives you: -```bash -desloppify plan # see the full ordered queue -desloppify plan move top # reorder — what unblocks the most? -desloppify plan cluster create # group related issues to batch-fix -desloppify plan focus # scope next to one cluster -desloppify plan defer # push low-value items aside -desloppify plan skip # hide from next -desloppify plan done # mark complete -desloppify plan reopen # reopen -``` - -### Subjective reviews - -The scan will prompt you when a subjective review is needed — just follow its instructions. -If you need to trigger one manually: -```bash -desloppify review --run-batches --runner codex --parallel --scan-after-import -``` - -### Other useful commands - -```bash -desloppify next --count 5 # top 5 priorities -desloppify next --cluster # drill into a cluster -desloppify show # filter by file/detector/ID -desloppify show --status open # all open findings -desloppify plan skip --permanent "" --note "reason" # accept debt (lowers strict score) -desloppify scan --path . --reset-subjective # reset subjective baseline to 0 -``` - -## 3. Reference - -### How scoring works - -Overall score = **40% mechanical** + **60% subjective**. - -- **Mechanical (40%)**: auto-detected issues — duplication, dead code, smells, unused imports, security. Fixed by changing code and rescanning. -- **Subjective (60%)**: design quality review — naming, error handling, abstractions, clarity. Starts at **0%** until reviewed. The scan will prompt you when a review is needed. -- **Strict score** is the north star: wontfix items count as open. The gap between overall and strict is your wontfix debt. -- **Score types**: overall (lenient), strict (wontfix counts), objective (mechanical only), verified (confirmed fixes only). - -### Subjective reviews in detail - -- **Preferred**: `desloppify review --run-batches --runner codex --parallel --scan-after-import` — does everything in one command. -- **Manual path**: `desloppify review --prepare` → review per dimension → `desloppify review --import file.json`. -- Import first, fix after — import creates tracked state entries for correlation. -- Integrity: reviewers score from evidence only. Scores hitting exact targets trigger auto-reset. -- Even moderate scores (60-80) dramatically improve overall health. -- Stale dimensions auto-surface in `next` — just follow the queue. - -### Key concepts - -- **Tiers**: T1 auto-fix → T2 quick manual → T3 judgment call → T4 major refactor. -- **Auto-clusters**: related findings are auto-grouped in `next`. Drill in with `next --cluster `. -- **Zones**: production/script (scored), test/config/generated/vendor (not scored). Fix with `zone set`. -- **Wontfix cost**: widens the lenient↔strict gap. Challenge past decisions when the gap grows. -- Score can temporarily drop after fixes (cascade effects are normal). - -## 4. Escalate Tool Issues Upstream - -When desloppify itself appears wrong or inconsistent: - -1. Capture a minimal repro (`command`, `path`, `expected`, `actual`). -2. Open a GitHub issue in `peteromallet/desloppify`. -3. If you can fix it safely, open a PR linked to that issue. -4. If unsure whether it is tool bug vs user workflow, issue first, PR second. - -## Prerequisite - -`command -v desloppify >/dev/null 2>&1 && echo "desloppify: installed" || echo "NOT INSTALLED — run: pip install --upgrade git+https://github.com/peteromallet/desloppify.git"` - - - -## Claude Code Overlay - -Use Claude subagents for subjective scoring work that should be context-isolated. - -### Subjective review - -1. **Preferred**: `desloppify review --run-batches --runner codex --parallel --scan-after-import` — does everything in one command. -2. **Claude cloud path**: `desloppify review --external-start --external-runner claude` → use generated `claude_launch_prompt.md` + `review_result.template.json` → run printed `desloppify review --external-submit --session-id --import `. -3. **Manual path**: split dimensions across N subagents (one message, multiple Task calls), merge outputs, then `desloppify review --import findings.json`. - -For the manual path: -- Read `dimension_prompts` from `query.json` for dimension definitions. -- Give each agent the codebase path, dimensions, and output format. Let agents decide what to read. -- Each agent writes output to a separate file. Merge assessments (average overlaps) and findings. -- Import first, fix after — import creates tracked state for correlation. - -### Subagent rules - -1. Prefer delegating review tasks to a project subagent in `.claude/agents/`. -2. Set `context: fork` so prior chat context does not leak into scoring. -3. For blind reviews, consume `.desloppify/review_packet_blind.json` instead of full `query.json`. -4. Score from evidence only; do not anchor to target thresholds. When mixed, score lower. -5. Return machine-readable JSON matching the format in the base skill doc. For `--external-submit`, include `session` from the generated template. -6. `findings` MUST match `query.system_prompt` exactly. Use `"findings": []` when no defects found. -7. Import is fail-closed: invalid findings abort unless `--allow-partial` is passed. - - - diff --git a/.agents/skills/deterministic-simulation/SKILL.md b/.agents/skills/deterministic-simulation/SKILL.md new file mode 100644 index 000000000..574420a0d --- /dev/null +++ b/.agents/skills/deterministic-simulation/SKILL.md @@ -0,0 +1,56 @@ +--- +name: deterministic-simulation +description: Deterministic simulation testing (DST), TigerBeetle/FoundationDB style, and DST-driven development. Use when building or testing a stateful system (Temper and anything like it - databases, queues, engines, protocol code), when a task mentions simulation, seeds, fault injection, or invariants, or before writing tests for concurrent/distributed logic. Not for frontend apps. +--- + +# Deterministic simulation testing + +## What it is + +The whole system runs inside a simulator that owns every source of nondeterminism: time, randomness, scheduling, network, disk. One seed drives one execution. The same seed replays the exact same execution, byte for byte. The simulator injects faults (crashes, partitions, delayed and dropped messages, disk errors) while invariants are checked continuously. Different seeds explore different executions; a failing seed is a permanent, replayable reproduction of a bug. + +Reference implementations and what each proved: +- **FoundationDB**: a deterministic simulator running the whole cluster in one process; `BUGGIFY` markers in production code cooperatively inject faults with some probability when simulating; "swizzle-clogging" (clog a random subset of nodes' networks one by one, unclog in random order) finds the deep interleavings. Their bar: if production hits a bug the simulator could have expressed, that is a simulator gap to fix. +- **TigerBeetle (VOPR)**: an entire cluster of real code under network, storage, and process faults at ~1000x real-time (a virtual clock means simulated time runs as fast as the CPU allows); runs continuously across many cores and seeds; assumes the disk WILL fail - corruption and misdirected reads/writes are in the fault model, not just crashes. +- **Antithesis**: DST as a service over unmodified systems. + +In this repo: the simulator lives in `temper-runtime` (sim module) with `temper-store-sim` as the simulated store; DST suites are `platform_e2e_dst` and `system_entity_dst` in `crates/temper-platform/tests/`. + +## What it is NOT - the mistakes agents make + +- **Not an integration test.** An integration test runs the system against real dependencies on real time and passes or fails once. A simulation runs thousands of seeded executions against simulated dependencies with faults injected. +- **Not a mock-based unit test.** Mocks replace the system's parts to isolate one piece. In DST the PRODUCTION CODE runs - all of it, unmodified. Only the environment (clock, network, disk, scheduler, entropy) is simulated. +- **Not a parallel reimplementation.** You do not write a second version of the logic and compare. The one real implementation runs in the simulator. A simplified MODEL may exist as an oracle to check results against, but the thing under test is always the production code. +- **Not "tests that use a seed."** If any nondeterminism leaks (a real clock read, an unseeded RNG, thread timing, iteration order of an unordered map), replay breaks and the whole method is void. Determinism is the load-bearing property. + +## Requirements on the code under test + +- All time via an injected clock. Never read the wall clock directly. +- All randomness from one seeded source the simulator provides. +- Single logical thread of execution, or scheduling fully controlled by the simulator. +- All I/O (network, disk, external services) behind interfaces the simulator can implement. +- No dependency on unordered iteration, real timers, or ambient environment. + +If the code cannot meet these, that is an architecture finding to raise, not a reason to fall back to integration tests. + +## DST-driven development + +For systems like Temper this replaces test-driven development. The loop: + +1. **Define the harness first.** Before implementing, extend the simulator with the scenario: the workload, the faults, and the invariants - the things that must never happen (lost write, double apply, stuck state machine, divergent replicas). Run it. **The invariant must fail now** - a harness that cannot catch the missing behavior proves nothing. +2. **Implement.** Production code, running inside the simulation. +3. **Run seeds until the invariants hold.** Not one seed - many. A green run on one seed is one execution, not correctness. +4. **A failing seed found later is committed as a regression case** and stays in the suite forever. +5. Fix by root cause. Never fix by weakening the invariant or narrowing the workload. + +## Writing good simulations + +- Coverage lives in the workload and fault schedule, not the framework. It is easy to build a simulator that explores almost nothing. Vary operation mixes, timings, fault frequencies; check that interesting states are actually reached. +- Invariants are properties, not examples: "no acknowledged write is ever lost", not "this call returns 3". +- Keep seeds cheap. A virtual clock costs nothing to advance - simulated hours run in wall-clock seconds. Fast executions buy more seeds per CI run, and more seeds are more coverage. +- Put cooperative fault points in production code (FoundationDB's BUGGIFY pattern): rare branches - a timeout firing early, a message reordered - taken with small probability only under simulation. The code helps the simulator find its own weaknesses. +- Report failures as: seed, invariant violated, minimal event trace. The seed IS the bug report. + +## Limits - say them, do not hide them + +DST cannot catch: bugs in the simulator's model of the environment, behavior of real external systems, real-clock/performance issues, and nondeterminism the harness failed to capture. Code changes invalidate old seeds' meaning (the seed replays a different execution). DST complements live verification; it does not replace the Definition of Done's live run. diff --git a/.progress/010_20260225_140000_end-to-end-tryable.md b/.progress/010_20260225_140000_end-to-end-tryable.md deleted file mode 100644 index 6baa12454..000000000 --- a/.progress/010_20260225_140000_end-to-end-tryable.md +++ /dev/null @@ -1,39 +0,0 @@ -# 010: Make Temper End-to-End Tryable - -**Goal**: By end of day, be able to start Temper, load specs, interact as an agent, see Cedar deny→approve→succeed, and see it all in the observe dashboard. - -## Phases - -### Phase 1: Infrastructure & Server ✅ -- [x] Verify CLI compiles -- [x] Docker is available -- [x] Observe dashboard deps installed -- [x] Start Postgres (local already running) -- [x] Start Temper server with ecommerce specs on port 3333 -- [x] Verify OData endpoints respond (GET /tdata shows Orders, Payments, Shipments) - -### Phase 2: Observe Dashboard ✅ -- [x] Dashboard already running on port 3000 -- [x] Connected to server on 3333 - -### Phase 3: Agent Governance Demo ✅ -- [x] Enable Cedar default-deny (PUT policies with empty permits) -- [x] Agent bound action → HTTP 403 (Cedar deny) -- [x] Pending decision created and visible -- [x] Approve with broad scope → Cedar policy auto-generated -- [x] Agent retries → HTTP 200 (success) -- [x] Full lifecycle: AddItem → SubmitOrder → ConfirmOrder -- [x] Agent audit trail: 4 actions, 3 success, 1 denied, 75% rate - -### Phase 4: Bug Fixes ✅ -- [x] Fix: extract_agent_context() falls back to X-Temper-Principal-Id header -- [x] Fix: trajectory log always pushes to in-memory (not conditional on persist success) -- [x] All tests pass (temper-server + ecommerce-reference) - -### Phase 5: Demo Script ✅ -- [x] Created `scripts/demo.sh` with full curl workflow - -## Findings -- Server can start in-memory (no DATABASE_URL → no persistence, but works) -- Default port is 3000 (serve command), dashboard expects server on 3333 -- Docker compose provisions: Postgres, Redis, ClickHouse, OTEL collector diff --git a/.progress/010_20260225_self_contained_mcp.md b/.progress/010_20260225_self_contained_mcp.md deleted file mode 100644 index 96610f19c..000000000 --- a/.progress/010_20260225_self_contained_mcp.md +++ /dev/null @@ -1,15 +0,0 @@ -# Self-Contained MCP Server Implementation - -## Status: COMPLETE - -## Steps -1. [x] Read all files -2. [x] Port 0 support in serve (actual port in Listening message) -3. [x] Make --port optional in McpConfig + CLI -4. [x] Add server_port OnceLock to RuntimeContext -5. [x] Guard execute methods (temper_request, temper_request_text) -6. [x] Implement start_server in tools.rs -7. [x] Update protocol.rs tool descriptions -8. [x] Update .mcp.json -9. [x] Update tests (lib_tests.rs, main.rs tests) — 2 new tests added -10. [x] Build and test — all 20 MCP tests + 29 CLI tests pass diff --git a/.progress/011_20260226_governed_external_api_calls.md b/.progress/011_20260226_governed_external_api_calls.md deleted file mode 100644 index 9557bbb52..000000000 --- a/.progress/011_20260226_governed_external_api_calls.md +++ /dev/null @@ -1,56 +0,0 @@ -# 011: Governed External API Calls Through MCP - -**ADR:** docs/adrs/0005-governed-external-api-calls-through-mcp.md -**Date:** 2026-02-26 -**Status:** COMPLETE - -## Phases - -### Phase 1: Integration Config Fields (Parts 1a-1c) -- [x] Add `config: BTreeMap` to Integration struct (types.rs) -- [x] Update TOML parser to store unknown keys in config (parser.rs) -- [x] Add `integration_config` to WasmInvocationContext (wasm types.rs) -- [x] Add test for config parsing (`test_integration_config_captures_unknown_keys`) - -### Phase 2: Fix trigger_params + Pass Config (Parts 1d + 4) -- [x] Thread action params through dispatch_wasm_integrations -- [x] Thread action params through dispatch_wasm_integrations_blocking -- [x] Pass integration.config to WasmInvocationContext -- [x] Pass actual trigger_params (not null) - -### Phase 3: upload_wasm MCP Method (Part 3) -- [x] Add `upload_wasm` method to tools.rs -- [x] Add `temper_request_bytes` helper to runtime context -- [x] Update protocol.rs tool descriptions (added upload_wasm + http_fetch mention) -- [x] Add upload_wasm to unknown-method error message - -### Phase 4: Generic http_fetch WASM Module (Part 2) -- [x] Create http-fetch WASM module (examples/wasm-modules/http-fetch/) -- [x] Build and copy to crates/temper-wasm/modules/http_fetch.wasm -- [x] Add `builtins` support to WasmModuleRegistry -- [x] Pre-register http_fetch at startup via `register_builtin_wasm_modules()` - -### Phase 5: Tests + Verification -- [x] cargo test -p temper-spec (42/42 pass) -- [x] cargo test -p temper-wasm (4/4 pass) -- [x] cargo test -p temper-server (121/121 pass) -- [x] cargo test -p temper-mcp (20/20 pass) -- [x] cargo check --workspace (clean) - -## Files Changed - -| File | Change | -|------|--------| -| `crates/temper-spec/src/automaton/types.rs` | Added `config: BTreeMap` to Integration | -| `crates/temper-spec/src/automaton/parser.rs` | Store unknown keys in config, init config in constructor, added test | -| `crates/temper-wasm/src/types.rs` | Added `integration_config` to WasmInvocationContext | -| `crates/temper-wasm/tests/e2e_invoke.rs` | Added `integration_config` to test context | -| `crates/temper-server/src/state/dispatch.rs` | Pass config + actual trigger_params, added action_params param | -| `crates/temper-server/src/state/dispatch_blocking.rs` | Same as dispatch.rs | -| `crates/temper-server/src/wasm_registry.rs` | Added `builtins` field + `register_builtin()` + fallback in `get_hash()` | -| `crates/temper-server/src/state/mod.rs` | Pre-register http_fetch WASM module at startup | -| `crates/temper-mcp/src/tools.rs` | Added `upload_wasm` method + `temper_request_bytes` helper | -| `crates/temper-mcp/src/protocol.rs` | Updated execute tool description with upload_wasm + http_fetch | -| `examples/wasm-modules/http-fetch/` | New generic HTTP fetch WASM module | -| `crates/temper-wasm/modules/http_fetch.wasm` | Compiled WASM binary (48KB) | -| `Cargo.toml` | Added http-fetch to workspace exclude | diff --git a/.progress/012_20260226_agent_governance_ux.md b/.progress/012_20260226_agent_governance_ux.md deleted file mode 100644 index 055b7c76e..000000000 --- a/.progress/012_20260226_agent_governance_ux.md +++ /dev/null @@ -1,34 +0,0 @@ -# 012: Agent Governance UX — Default-Deny with Human Approval - -## Status: In Progress - -## Phases - -### Phase 0: ADR-0008 [DONE] -- [x] Write ADR-0008 - -### Phase 1: Fix Immediate Bugs [DONE] -- [x] 1a. WASM engine result reading fix (engine.rs) — host_set_result checked first -- [x] 1b. Fix set_policy content type (tools.rs) — removed entirely (governance write) -- [x] 1c. Default Turso storage (tools.rs) — start_server passes --storage turso - -### Phase 2: Remove Governance Write Methods [DONE] -- [x] 2a. Remove approve_decision, deny_decision, set_policy from dispatch (tools.rs) -- [x] 2b. Update tool description (protocol.rs) -- [x] 2c. Enhance authz denied error (sandbox.rs) — includes poll_decision guidance - -### Phase 3: Observe UI Auto-Start [DONE] -- [x] 3a. --observe flag for temper serve (serve/mod.rs) -- [x] 3b. MCP start_server starts Observe (tools.rs) — passes --observe - -### Phase 4: Claude Code Hook + temper decide CLI [DONE] -- [x] 4a. PostToolUse hook (postToolUse-temper-decide.sh) -- [x] 4b. temper decide CLI subcommand (decide/mod.rs) - -### Phase 5: http_fetch Hardening [DONE] -- [x] Harden http_fetch module — includes status_code in response -- [x] Rebuild WASM binary - -### Phase 6: Verification [DONE] -- [x] Build passes (temper-wasm, temper-mcp, temper-cli all compile) -- [x] Tests pass (temper-wasm: 20, temper-mcp: 20, temper-cli: 29, temper-server: 2) diff --git a/.progress/023_20260223_codebase_audit_and_cleanup.md b/.progress/023_20260223_codebase_audit_and_cleanup.md deleted file mode 100644 index 4ccb3f225..000000000 --- a/.progress/023_20260223_codebase_audit_and_cleanup.md +++ /dev/null @@ -1,134 +0,0 @@ -# 023 — Codebase Audit & Cleanup - -> Date: 2026-02-23 -> Goal: Understand entirety, clean up bloat, eliminate gaps/inconsistencies, make presentable - -## Phase 1: Assessment (COMPLETE) - -### Compilation Status: CLEAN -- Initial test run showed 4 errors — caused by stale incremental build cache -- Fresh `cargo check --workspace` and `cargo test --workspace` both pass with **0 errors, 0 warnings** -- The methods `upsert_tenant_constraints` / `delete_tenant_constraints` DO exist in TursoEventStore (lines 278-311) -- All ~311 tests pass across all crates - -### Bloat Identified - -#### Files to Delete -1. `generated_imgs/` — 11 AI-generated PNGs, not source code, not gitignored -2. `temper-demo/` — contains only `node_modules/`, zero source files -3. `TEMPER_BUILDER.skill.md` — duplicate of `.claude/skills/temper.md` -4. `TEMPER_USER.skill.md` — duplicate of `.claude/skills/temper-user.md` -5. `.z3-trace` — debugging artifact (already in .gitignore) -6. `.progress/` — 18 deleted but untracked plan files showing in git status - -#### Files Violating 500-Line Rule (.vision/CONSTRAINTS.md) -1. `temper-server/src/observe_routes.rs` — **4,121 lines** (8x limit) -2. `temper-cli/src/serve/mod.rs` — **1,228 lines** (2.5x limit) -3. `temper-mcp/src/lib.rs` — **1,153 lines** (2.3x limit) -4. `temper-server/src/dispatch.rs` — **918 lines** (1.8x limit) -5. `temper-server/src/state.rs` — **1,211 lines** (2.4x limit) -6. `temper-spec/src/automaton/parser.rs` — **753 lines** (1.5x limit) -7. `temper-spec/src/csdl/parser.rs` — **676 lines** (1.4x limit) -8. `temper-spec/src/tlaplus/extractor.rs` — **664 lines** (1.3x limit) -9. `temper-runtime/src/scheduler/mod.rs` — **650 lines** (1.3x limit) -10. `temper-store-redis/src/event_store.rs` — **605 lines** (1.2x limit) -11. `temper-odata/src/types.rs` — **599 lines** (1.2x limit) -12. `temper-verify/src/simulation.rs` — **545 lines** (1.1x limit) -13. `temper-verify/src/smt.rs` — **565 lines** (1.1x limit) - -#### Duplicate/Redundant Documentation -- `docs/PAPER.md` (73KB) — Research paper, massive -- `docs/AGENT_GUIDE.md` (54KB) — Agentic development guide -- `docs/AUDIT.md` (26KB) — Previous audit results, may be stale -- `docs/USE_CASES.md` (29KB) — Use case analysis -- Multiple overlap between docs/, CLAUDE.md, .vision/, CODING_GUIDELINES.md, README.md - -### Gaps & Inconsistencies - -#### Compilation Gaps -1. **Missing Turso methods** — `state.rs` calls `upsert_tenant_constraints` / `delete_tenant_constraints` on TursoEventStore but they're not implemented - -#### Architectural Inconsistencies -1. **Cargo.toml says rust-version 1.85** but CONSTRAINTS.md also says 1.85, while actual workspace Cargo.toml may differ — need to verify consistency -2. **temper-macros** is a 30-line stub — either implement or remove from workspace -3. **temper-codegen** is partially stubbed — generator structure exists but output is limited -4. **temper-optimize** — framework present but tuning algorithms are basic/placeholder - -#### Open P2 Gaps from GAP_TRACKER.md (14 remaining) -- #20: MaxCount guard never parsed -- #21: Hand-rolled TOML parser fragility -- #22: Shadow testing uses legacy API only -- #24: No batch request support -- #25: No $search support -- #26: No $apply aggregation support -- #27: init template uses relative path -- #28: No graceful shutdown -- #29: Optimization recommendations not applied -- #30: No observability providers beyond ClickHouse -- #31: Legacy TLA+ extractor is brittle -- #32: Generated code not validated -- #34: Proc macros limited to marker traits -- #36: IncrementItems/DecrementItems legacy aliases - -### Verification & Testing Summary - -**Total Tests**: ~594 across workspace (can't run due to compile error) - -**Verification Cascade** (temper-verify): -- L0: SMT symbolic verification via Z3 — checks guard satisfiability -- L1: Stateright exhaustive model checking — dead guards, unreachable states, inductive invariants -- L2: Deterministic simulation — fault injection, message drops/reordering/delays -- L2b: Multi-actor simulation — cross-entity consistency -- L3: Property-based tests via proptest — boundary values, edge cases - -**Test Types Present**: -- Unit tests: inline `#[test]` in most modules -- Integration tests: `tests/` directories in temper-server, temper-platform -- E2E tests: platform_e2e_dst, system_entity_dst, compile_first_e2e -- Benchmark tests: agent_checkout, agent_triage in reference apps -- DST compliance: determinism guard hook runs on every edit - -**Automated Enforcement** (hooks): -- Pre-edit: plan reminder, spec verification cascade, dep isolation, determinism guard -- Pre-commit: integrity check (no TODO/unwrap), spec syntax, dep audit -- Pre-push: 3-gate pipeline (integrity, determinism, full test suite) -- Session exit: unverified push check - ---- - -## Phase 2: Fix Compilation (Priority 1) — NOT NEEDED - -- [x] Compilation is clean — initial errors were from stale incremental cache -- [x] `cargo test --workspace` passes: 311 tests, 0 failures, 0 warnings - -## Phase 3: Remove Bloat (Priority 2) - -- [ ] Delete `generated_imgs/` directory -- [ ] Delete `temper-demo/` directory (empty except node_modules) -- [ ] Delete `TEMPER_BUILDER.skill.md` (duplicate) -- [ ] Delete `TEMPER_USER.skill.md` (duplicate) -- [ ] Delete `.z3-trace` if present -- [ ] Add `generated_imgs/`, `temper-demo/` to .gitignore -- [ ] Clean up stale `.progress/` files from git status - -## Phase 4: File Size Violations (Priority 3) - -- [ ] Split `observe_routes.rs` (4,121 lines) into directory module -- [ ] Split `state.rs` (1,211 lines) into directory module -- [ ] Split `serve/mod.rs` (1,228 lines) into submodules -- [ ] Split `temper-mcp/lib.rs` (1,153 lines) into modules -- [ ] Split `dispatch.rs` (918 lines) into submodules - -## Phase 5: Consistency & Presentability (Priority 4) - -- [ ] Evaluate temper-macros — stub or remove -- [ ] Clean up duplicate docs surface (reconcile docs/, CLAUDE.md, README.md overlap) -- [ ] Ensure README.md is presentable for external viewing -- [ ] Verify all `pub` items have doc comments in core crates - ---- - -## Instance Log -| Instance | Phase | Status | -|----------|-------|--------| -| Main | Assessment | COMPLETE | diff --git a/.progress/024_20260223_split_oversized_files.md b/.progress/024_20260223_split_oversized_files.md deleted file mode 100644 index e43fc9256..000000000 --- a/.progress/024_20260223_split_oversized_files.md +++ /dev/null @@ -1,50 +0,0 @@ -# Split Top 3 Oversized Files - -## Status: COMPLETE - -All 3 oversized files split into directory modules, committed, and pushed to main. - -## Commits -- `6b1d54a` refactor: split 3 oversized files into directory modules (20 files, +7215/-6764) -- `9551c75` fix: remove temper_wasm references from split files (3 files, +27/-104) -- `277e0d8` chore: remove dead observe/wasm.rs (unreferenced after split) - -## Files Split -1. `temper-server/src/observe_routes.rs` (4,247 lines) → `observe/` directory module (7 files) -2. `temper-cli/src/serve/mod.rs` (1,370 lines) → submodules (loader.rs, storage.rs) -3. `temper-server/src/state.rs` (1,666 lines) → `state/` directory module (6 files) - -## Phase 1: observe_routes.rs → observe/ [DONE] -- [x] Create observe/ directory -- [x] Extract mod.rs (router, shared types, skills) -- [x] Extract specs.rs -- [x] Extract entities.rs -- [x] Extract verification.rs -- [x] Extract metrics.rs -- [x] Extract evolution.rs -- [x] Update lib.rs -- [x] cargo check - -## Phase 2: serve/mod.rs → submodules [DONE] -- [x] Extract storage.rs -- [x] Extract loader.rs -- [x] Update mod.rs -- [x] cargo check - -## Phase 3: state.rs → state/ [DONE] -- [x] Create state/ directory -- [x] Extract mod.rs (struct def, new(), builders) -- [x] Extract metrics.rs -- [x] Extract trajectory.rs -- [x] Extract persistence.rs -- [x] Extract entity_ops.rs -- [x] Extract dispatch.rs -- [x] Update lib.rs -- [x] cargo check + cargo test - -## Verification -- All tests pass (`cargo test --workspace`) -- HEAD == origin/main at `3bba3ab` -- DST review: PASS -- Code review: PASS -- Alignment review: PASS diff --git a/.progress/024_20260223_wasm_integration.md b/.progress/024_20260223_wasm_integration.md deleted file mode 100644 index 24e4c62fa..000000000 --- a/.progress/024_20260223_wasm_integration.md +++ /dev/null @@ -1,55 +0,0 @@ -# WASM Integration for Agent-Generated API Calls - -**Date**: 2026-02-23 -**Status**: Complete (pending commit) - -## Phases - -- [x] Phase 0: ADR document (`docs/adrs/0002-wasm-integration-for-agent-generated-api-calls.md`) -- [x] Phase 1: `temper-wasm` crate (engine, host trait, types) — 4 tests passing -- [x] Phase 2: Spec format changes (Effect::Trigger, Integration fields, parser, builder) — 4 new tests -- [x] Phase 3: Storage (wasm_modules table in Turso + Postgres) — schema + CRUD methods -- [x] Phase 4: Dispatch wiring (WasmModuleRegistry, ServerState integration) — non-async fire-and-forget -- [x] Phase 5: DST support (SimIntegrationResponses, callback scheduling in SimActorSystem) -- [x] Phase 6: Platform & deploy pipeline (WASM module validation, DeployInput.wasm_modules) - -## Key Decisions -- Async post-transition model: WASM runs AFTER transition succeeds, feeds back as input action -- Separate `wasm_modules` table for module storage -- Wasmtime v29 with fuel metering (TigerStyle budgets) -- DST: WASM not executed in simulation, callbacks injected by scheduler -- `dispatch_wasm_integrations()` is non-async — spawns tokio tasks for each callback (avoids recursive async) -- `Effect::Trigger { name }` at spec level maps to `Effect::Custom(name)` at JIT level - -## Verification -- `cargo check --workspace` — clean -- `cargo test --workspace` — 622 tests, 0 failures -- DST review: running -- Code review: running - -## Files Created -- `docs/adrs/0002-wasm-integration-for-agent-generated-api-calls.md` -- `crates/temper-wasm/Cargo.toml` -- `crates/temper-wasm/src/lib.rs` -- `crates/temper-wasm/src/types.rs` -- `crates/temper-wasm/src/host_trait.rs` -- `crates/temper-wasm/src/engine.rs` -- `crates/temper-server/src/wasm_registry.rs` - -## Files Modified -- `Cargo.toml` (workspace) — added temper-wasm, wasmtime, sha2, async-trait -- `crates/temper-spec/src/automaton/types.rs` — Effect::Trigger variant, Integration fields -- `crates/temper-spec/src/automaton/parser.rs` — trigger parsing, WASM validation -- `crates/temper-jit/src/table/builder.rs` — Trigger → Custom conversion -- `crates/temper-store-turso/src/schema.rs` — wasm_modules table -- `crates/temper-store-turso/src/store.rs` — CRUD + migration -- `crates/temper-store-turso/src/lib.rs` — re-exports -- `crates/temper-store-postgres/src/schema.rs` — wasm_modules table -- `crates/temper-server/src/lib.rs` — wasm_registry module -- `crates/temper-server/src/state.rs` — dispatch wiring, WASM module CRUD -- `crates/temper-runtime/src/scheduler/sim_handler.rs` — pending_callbacks() -- `crates/temper-runtime/src/scheduler/sim_actor_system.rs` — SimIntegrationResponses -- `crates/temper-runtime/src/scheduler/mod.rs` — re-exports -- `crates/temper-server/src/entity_actor/sim_handler.rs` — custom effects capture -- `crates/temper-platform/src/deploy/pipeline.rs` — WASM validation step -- `crates/temper-verify/src/model/builder.rs` — Trigger → None mapping diff --git a/.progress/026_20260223_fix_6_audit_issues.md b/.progress/026_20260223_fix_6_audit_issues.md deleted file mode 100644 index 845888576..000000000 --- a/.progress/026_20260223_fix_6_audit_issues.md +++ /dev/null @@ -1,15 +0,0 @@ -# Plan: Fix 6 Audit Issues - -## Status: In Progress - -## Phase 1: Quick Wins -- [ ] Issue 3: Shadow Testing — Extend TestCase to Full EvalContext -- [ ] Issue 6: Action Dispatch Timeout — Make Configurable -- [ ] Issue 5: Query $expand — Support Nested Expansion - -## Phase 2: Medium -- [ ] Issue 1: OData Bound Functions — Wire Up Read-Only Dispatch - -## Phase 3: Large -- [ ] Issue 2: Deploy Hook — Wire Up Spec Sourcing + Pipeline -- [ ] Issue 4: Cross-Invariant Eventual Enforcement — Background Convergence diff --git a/.progress/027_20260224_move_mutations_out_of_observe.md b/.progress/027_20260224_move_mutations_out_of_observe.md deleted file mode 100644 index 079d9dcd0..000000000 --- a/.progress/027_20260224_move_mutations_out_of_observe.md +++ /dev/null @@ -1,51 +0,0 @@ -# 027 — Move Mutation Endpoints Out of /observe - -## Goal -Move all 7 POST/DELETE mutation endpoints from `/observe` (read-only) into `/api/*`. - -## Endpoints to Move - -| # | Old Path | New Path | Method | -|---|----------|----------|--------| -| 1 | `/observe/specs/load-dir` | `/api/specs/load-dir` | POST | -| 2 | `/observe/specs/load-inline` | `/api/specs/load-inline` | POST | -| 3 | `/observe/wasm/modules/{name}` | `/api/wasm/modules/{name}` | POST | -| 4 | `/observe/wasm/modules/{name}` | `/api/wasm/modules/{name}` | DELETE | -| 5 | `/observe/evolution/records/{id}/decide` | `/api/evolution/records/{id}/decide` | POST | -| 6 | `/observe/trajectories/unmet` | `/api/evolution/trajectories/unmet` | POST | -| 7 | `/observe/sentinel/check` | `/api/evolution/sentinel/check` | POST | - -## Phases - -### Phase 1: Backend — Create `/api` router -- [x] Make observe sub-modules `pub(crate)` in `observe/mod.rs` -- [x] Create `crates/temper-server/src/api.rs` with `build_api_router()` -- [x] Remove mutation routes from `build_observe_router()` -- [x] Wire `/api` into `router.rs` - -### Phase 2: Update backend tests -- [x] Update observe/mod.rs tests referencing moved endpoints (9 test URLs updated) - -### Phase 3: Frontend updates -- [x] Update `observe/lib/api.ts` — `triggerSentinelCheck` path -- [x] Update `observe/__tests__/lib/api.test.ts` — sentinel check URL -- [x] Update `observe/app/integrations/page.tsx` — help text - -### Phase 4: Docs + Skills -- [x] Update `.claude/skills/temper.md` -- [x] Update `.claude/skills/temper-user.md` -- [x] Update `.claude/commands/temper-user.md` -- [x] Update `observe/public/skills/builder.md` -- [x] Update `observe/public/skills/user.md` -- [x] Update `docs/GAP_TRACKER.md` -- [x] Update `skills/temper/SKILL.md` - -### Phase 5: Verify -- [x] `cargo test -p temper-server` — 145 tests pass -- [x] `cd observe && npx vitest run` — 104 tests pass -- [x] Reviews (DST, code, alignment) — all PASS -- [x] Readability baseline updated (PROD_MAX_FILE_LINES 1229→1234 from upstream specs_helpers extraction) -- [x] Committed: 51f0e6d -- [x] Pushed to main - -## Status: COMPLETE diff --git a/.progress/028_20260224_agent_backend_capabilities.md b/.progress/028_20260224_agent_backend_capabilities.md deleted file mode 100644 index bb2ca9355..000000000 --- a/.progress/028_20260224_agent_backend_capabilities.md +++ /dev/null @@ -1,48 +0,0 @@ -# Agent Backend Capabilities - -**Created**: 2026-02-24 -**Status**: Complete - -## Phases - -- [x] Phase 0: ADR — `docs/adrs/0003-agent-backend-capabilities.md` -- [x] Phase 1: Agent Identity / Sessions — X-Agent-Id, X-Session-Id threaded through dispatch -- [x] Phase 2: Credentials Vault — AES-256-GCM per-tenant secrets, injected into WASM host -- [x] Phase 3: Idempotency — Per-actor LRU cache with TTL eviction -- [x] Phase 4: Blocking Integrations — ?await_integration=true for inline WASM execution -- [x] Verification — `cargo test --workspace` all passing - -## Files Changed - -### New Files -- `docs/adrs/0003-agent-backend-capabilities.md` — ADR -- `crates/temper-server/src/secrets_vault.rs` — Encrypted vault with AES-256-GCM -- `crates/temper-server/src/idempotency.rs` — Idempotency LRU cache -- `crates/temper-server/src/state/dispatch_blocking.rs` — Blocking WASM integration dispatch - -### Modified Files -- `Cargo.toml` + `Cargo.lock` — aes-gcm workspace dep -- `crates/temper-server/Cargo.toml` — aes-gcm dep -- `crates/temper-server/src/dispatch.rs` — AgentContext, idempotency, await_integration -- `crates/temper-server/src/state/dispatch.rs` — Agent threading, secrets injection, blocking branch -- `crates/temper-server/src/state/mod.rs` — New fields: idempotency_cache, secrets_vault -- `crates/temper-server/src/state/trajectory.rs` — agent_id, session_id fields -- `crates/temper-server/src/state/persistence.rs` — Trajectory + secrets persistence -- `crates/temper-server/src/events.rs` — agent_id, session_id on EntityStateChange -- `crates/temper-server/src/router.rs` — CORS headers for new headers -- `crates/temper-server/src/api.rs` — Secrets management routes -- `crates/temper-server/src/lib.rs` — Module registration -- `crates/temper-wasm/src/types.rs` — agent_id, session_id on WasmInvocationContext -- `crates/temper-store-postgres/src/schema.rs` — Trajectories + secrets table schemas -- Various test files updated for new struct fields - -## Instance Log - -| Phase | Status | Notes | -|-------|--------|-------| -| 0 | Complete | ADR created | -| 1 | Complete | Agent identity threaded through full dispatch chain | -| 2 | Complete | Vault + API + WASM injection | -| 3 | Complete | Idempotency cache with budget + TTL | -| 4 | Complete | Blocking dispatch with Box::pin for recursive async | -| Verify | Complete | cargo test --workspace all passing | diff --git a/.progress/028_20260224_wasm_persistence.md b/.progress/028_20260224_wasm_persistence.md deleted file mode 100644 index c2dd206b0..000000000 --- a/.progress/028_20260224_wasm_persistence.md +++ /dev/null @@ -1,30 +0,0 @@ -# WASM Persistence — Survive Server Restarts - -## Status: COMPLETE - -## Steps -- [x] Step 1: Add wasm_invocation_logs schema (Postgres + Turso) -- [x] Step 2: Add migrations for new table (+ fixed missing wasm_modules migration in Postgres) -- [x] Step 3: Add Turso store methods (persist_wasm_invocation, load_recent_wasm_invocations, load_wasm_modules_all_tenants) -- [x] Step 4: Add ServerState persistence methods (persist_wasm_invocation, load_wasm_modules, load_recent_wasm_invocations) -- [x] Step 5: Wire dispatch.rs to persist invocations (fire-and-forget after each in-memory log push) -- [x] Step 6: Wire startup recovery in serve/mod.rs (load_wasm_modules + load_recent_wasm_invocations) -- [x] Step 7: Update observe/wasm.rs (kept in-memory fast path, limit raised to 10k) -- [x] Step 8: Run tests — 697 workspace tests pass, 0 failures - -## Files Changed -| File | Action | -|------|--------| -| `crates/temper-store-postgres/src/schema.rs` | Added `CREATE_WASM_INVOCATION_LOGS_TABLE` + 3 indexes + tests | -| `crates/temper-store-turso/src/schema.rs` | Same for Turso (SQLite syntax) | -| `crates/temper-store-postgres/src/migration.rs` | Added wasm_modules + invocation_logs migrations | -| `crates/temper-store-turso/src/store.rs` | Added persist/load methods + TursoWasmInvocationRow | -| `crates/temper-store-turso/src/lib.rs` | Exported new types (TursoWasmInvocationInsert, TursoWasmInvocationRow) | -| `crates/temper-server/src/state/persistence.rs` | Added persist_wasm_invocation, load_wasm_modules, load_recent_wasm_invocations | -| `crates/temper-server/src/state/dispatch.rs` | Fire-and-forget persist after each invocation log push | -| `crates/temper-server/src/observe/wasm.rs` | Raised limit to 10k, kept in-memory fast path | -| `crates/temper-cli/src/serve/mod.rs` | Startup recovery: load_wasm_modules + load_recent_wasm_invocations | - -## Bug Fix -Fixed pre-existing bug: `wasm_modules` table was defined in Postgres schema.rs but missing from -migration.rs. Now included in the migration runner. diff --git a/.progress/029_20260224_cedar_authorization_for_agents.md b/.progress/029_20260224_cedar_authorization_for_agents.md deleted file mode 100644 index 46e1666df..000000000 --- a/.progress/029_20260224_cedar_authorization_for_agents.md +++ /dev/null @@ -1,65 +0,0 @@ -# Cedar Authorization for Agent Backend - -## Status: Complete -## Started: 2026-02-24 -## Completed: 2026-02-24 - -## Prerequisite: Fix Compilation Error -- [x] Check loader.rs:364 — already fixed -- [x] Check webhooks.rs:222 — already fixed -- [x] Workspace compiles clean - -## Phase 1: Wire Real Context Into Cedar (Level 1 — Entity Actions) -- [x] Add `SecurityContext::with_agent_context()` in temper-authz/context.rs -- [x] Add `authorize_with_context()` on ServerState in entity_ops.rs -- [x] Fix bindings.rs to extract X-Temper-* headers and use rich context -- [x] Move entity state fetch before authz check (resource attrs available) -- [x] Pass HeaderMap through from write.rs -- [x] Tests: 3 new tests in context.rs, all 16 temper-authz tests pass - -## Phase 2: WASM Host Function Authorization (Level 2) -- [x] Add `WasmAuthzContext` to temper-wasm/types.rs -- [x] Create `WasmAuthzGate` trait + `AuthorizedWasmHost` decorator in temper-wasm/authorized_host.rs -- [x] Create `CedarWasmAuthzGate` + `PermissiveWasmAuthzGate` in temper-server/wasm_authz_gate.rs -- [x] Wire `AuthorizedWasmHost` in dispatch.rs (wraps ProductionWasmHost) -- [x] Wire `AuthorizedWasmHost` in dispatch_blocking.rs -- [x] Thread `agent_ctx` through `dispatch_wasm_integrations` -- [x] Domain extraction: pure string parsing (no url crate) -- [x] Tests: 11 new tests in authorized_host.rs, 5 new tests in wasm_authz_gate.rs - -## Phase 3: Secret Pre-Filtering (Level 3) -- [x] Add `get_authorized_wasm_secrets()` method on ServerState (dispatch.rs) -- [x] Use filtered secrets in dispatch.rs (fire-and-forget path) -- [x] Use filtered secrets in dispatch_blocking.rs (inline path) - -## Phase 4: Policy Lifecycle -- [x] Create test-fixtures/specs/policies/platform-presets.cedar (Tier 1) -- [x] Add authz_denied/denied_resource/denied_module fields to TrajectoryEntry -- [x] Add `AuthzDenied` variant to `ObservationClass` in temper-evolution/records.rs -- [x] Add `suggest_cedar_policies()` in temper-platform deploy pipeline (Tier 2) -- [x] Create docs/adrs/0004-cedar-authorization-for-agents.md -- [x] All 738 workspace tests pass - -## Files Changed -### New Files -- `crates/temper-wasm/src/authorized_host.rs` — WasmAuthzGate trait + AuthorizedWasmHost decorator -- `crates/temper-server/src/wasm_authz_gate.rs` — CedarWasmAuthzGate + PermissiveWasmAuthzGate -- `test-fixtures/specs/policies/platform-presets.cedar` — Platform preset policies (Tier 1) -- `docs/adrs/0004-cedar-authorization-for-agents.md` — Architecture Decision Record - -### Modified Files -- `crates/temper-authz/src/context.rs` — Added `with_agent_context()` method -- `crates/temper-wasm/src/types.rs` — Added `WasmAuthzContext` struct -- `crates/temper-wasm/src/lib.rs` — Export new authorized_host module -- `crates/temper-server/src/lib.rs` — Register wasm_authz_gate module -- `crates/temper-server/src/state/entity_ops.rs` — Added `authorize_with_context()` method -- `crates/temper-server/src/odata/bindings.rs` — Rich Cedar context with headers + entity state -- `crates/temper-server/src/odata/write.rs` — Pass HeaderMap to dispatch_bound_action -- `crates/temper-server/src/state/dispatch.rs` — AuthorizedWasmHost + agent_ctx threading + secret pre-filtering -- `crates/temper-server/src/state/dispatch_blocking.rs` — Same as dispatch.rs -- `crates/temper-server/src/state/trajectory.rs` — Added authz_denied fields -- `crates/temper-server/src/webhooks.rs` — Updated TrajectoryEntry constructor -- `crates/temper-server/src/observe/evolution.rs` — Updated TrajectoryEntry constructor -- `crates/temper-cli/src/serve/loader.rs` — Updated TrajectoryEntry constructor -- `crates/temper-evolution/src/records.rs` — Added AuthzDenied observation class -- `crates/temper-platform/src/deploy/pipeline.rs` — Cedar policy suggestion generation diff --git a/.progress/030_20260225_140739_agent-policy-audit-layer.md b/.progress/030_20260225_140739_agent-policy-audit-layer.md deleted file mode 100644 index cd8cd5de8..000000000 --- a/.progress/030_20260225_140739_agent-policy-audit-layer.md +++ /dev/null @@ -1,34 +0,0 @@ -# Plan 030: Agent Policy & Audit Layer - -## Status: COMPLETE (pending commit) - -## Overview -Implement the human-facing policy & audit layer: denial→surface→approve→reload loop + agent dashboard UX. - -## Workstreams (Parallel) - -### WS1: Backend Core (Phases 1-4) — backend-core agent -- [x] Phase 1: PendingDecision data model + bounded log -- [x] Phase 2: Denial interception at OData (trajectory + pending decision on 403) -- [x] Phase 3: Policy CRUD API (GET/PUT policies, POST rules) -- [x] Phase 4: Decision approve/deny API + Cedar generation + SSE stream - -### WS2: Backend Observe (Phase 5) — backend-observe agent -- [x] Phase 5: Agent audit endpoints (list agents, agent history) - -### WS3: Frontend (Phases 6-8) — frontend agent -- [x] Phase 6: Decisions page (pending cards, approve/deny with scope, history table, SSE) -- [x] Phase 7: Agents pages (list with stats + detail timeline with denial highlights) -- [x] Phase 8: Sidebar + types + API client - -## Verification -- `cargo check --workspace` — PASS (0 errors) -- `cargo test -p temper-server` — PASS (all 22 tests) -- `npx next lint` — no new errors (pre-existing only) - -## Key Files -- New: pending_decisions.rs, observe/agents.rs, decisions/page.tsx, agents/page.tsx, agents/[id]/page.tsx -- Modified: state/mod.rs, api.rs, observe/mod.rs, odata/bindings.rs, dispatch.rs, Sidebar.tsx, types.ts, api.ts - -## DST Compliance -- BTreeMap everywhere, sim_uuid/sim_now, VecDeque bounded log, broadcast for SSE diff --git a/.progress/031_20260225_code-mode-agent-interface.md b/.progress/031_20260225_code-mode-agent-interface.md deleted file mode 100644 index edbe895b6..000000000 --- a/.progress/031_20260225_code-mode-agent-interface.md +++ /dev/null @@ -1,41 +0,0 @@ -# Code Mode for Temper: Spec-Aware Agent Interface - -## Status: COMPLETE - -## Phases - -### Phase 1: Search helpers (runtime.rs) -- [ ] Inject spec as Dataclass (type_id: 10) in run_search() -- [ ] Add dispatch_spec_method() for tenants/entities/describe/actions/actions_from/raw -- [ ] Change run_search() from single-shot to loop - -### Phase 2: Dynamic tool descriptions (spec_loader.rs + protocol.rs) -- [ ] Add generate_loaded_summary() to spec_loader.rs -- [ ] Make tool_definitions() take &RuntimeContext -- [ ] Update search/execute descriptions with spec-aware text - -### Phase 3: Developer methods (tools.rs) -- [ ] show_spec — read from self.spec -- [ ] submit_specs — POST /api/specs/load-inline -- [ ] set_policy — PUT /api/tenants/{t}/policies -- [ ] get_policies — GET /api/tenants/{t}/policies - -### Phase 4: Governance methods (tools.rs) -- [ ] get_decisions — GET with optional status filter -- [ ] approve_decision — POST with scope -- [ ] deny_decision — POST -- [ ] poll_decision — loop with 1s sleep, 30s timeout - -### Phase 5: Agent identity (tools.rs) -- [ ] Add X-Temper-Principal-Kind: agent header -- [ ] Add X-Temper-Principal-Id: mcp-agent header - -### Phase 6: Structured errors (sandbox.rs) -- [ ] Detect 403 + AuthorizationDenied in format_http_error() -- [ ] Parse body for decision ID -- [ ] Format rich denial message with guidance - -### Phase 7: Tests + verification -- [ ] Update existing search test for new spec API -- [ ] cargo build -p temper-mcp -- [ ] cargo test -p temper-mcp diff --git a/.progress/032_20260226_122604_wasm-developer-experience.md b/.progress/032_20260226_122604_wasm-developer-experience.md deleted file mode 100644 index 28f4d533a..000000000 --- a/.progress/032_20260226_122604_wasm-developer-experience.md +++ /dev/null @@ -1,31 +0,0 @@ -# WASM Developer Experience — ADR-0009 + Implementation - -## Status: COMPLETE - -## Phases - -### Phase 1: ADR-0009 [COMPLETE] -- [x] Create `docs/adrs/0009-wasm-developer-experience.md` - -### Phase 2: temper-wasm-sdk Crate [COMPLETE] -- [x] Create `crates/temper-wasm-sdk/Cargo.toml` -- [x] Create `crates/temper-wasm-sdk/src/lib.rs` -- [x] Create `crates/temper-wasm-sdk/src/host.rs` -- [x] Create `crates/temper-wasm-sdk/src/context.rs` -- [x] Add to workspace `Cargo.toml` members - -### Phase 3: compile_wasm MCP Tool [COMPLETE] -- [x] Add `compile_wasm` method to `tools.rs` -- [x] Update `protocol.rs` tool descriptions -- [x] Add `uuid` + `sha2` deps to MCP `Cargo.toml` - -### Phase 4: http-fetch Rewrite [COMPLETE] -- [x] Rewrite `examples/wasm-modules/http-fetch/` using SDK (375 lines -> 85 lines) - -### Phase 5: Verification [COMPLETE] -- [x] SDK compiles for wasm32-unknown-unknown -- [x] http-fetch rewrite compiles (217 KB WASM binary) -- [x] All 58 test suites pass (0 failures) -- [x] protocol.rs docs updated with compile_wasm -- [x] Readability baseline updated -- [x] cargo fmt applied diff --git a/.proof/001_gepa_temperagent_failure_reflection.md b/.proof/001_gepa_temperagent_failure_reflection.md deleted file mode 100644 index d88719a07..000000000 --- a/.proof/001_gepa_temperagent_failure_reflection.md +++ /dev/null @@ -1,55 +0,0 @@ -# 001_gepa_temperagent_failure_reflection - -Date: 2026-03-23 -Branch: docs/positioning-rewrite -Scope: TemperAgent sandbox provisioning failure reflection + root-cause hardening - -## Problem -`sandbox_provisioner` previously swallowed TemperFS bootstrap errors and still emitted `SandboxReady` with empty IDs. This made failures look like partial success. - -## Code Changes -- Hard fail on TemperFS bootstrap failure with explicit error context (`temper_api_url`, tenant, agent id). -- Added `temper_api_url` to TemperAgent state + `Configure` action. -- Updated CSDL to expose `TemperApiUrl` and `Configure.temper_api_url`. -- All TemperAgent WASM modules now resolve Temper API URL from entity `fields.temper_api_url` first, then integration config. -- Installing `temper-agent` now auto-installs `temper-fs` dependency. -- Added regression test for dependency install behavior. - -## Verification -### Build/Test -- `cargo fmt --all` -- `cargo check -p temper-platform` -- `cargo test -p temper-platform os_apps::tests::test_install_temper_agent_auto_installs_temper_fs -- --nocapture` (passed) -- Built all TemperAgent WASM modules for `wasm32-unknown-unknown`. - -### Live Proof (port 3015) -Tenant: `proof-fix-20260323` - -Prereq: Uploaded tenant-scoped WASM modules: -- `sandbox_provisioner` -- `llm_caller` -- `tool_runner` -- `workspace_restorer` - -Case A (bad API URL): -- Agent: `019d1c5a-4a43-71b0-adc9-199cb90eefab` -- Configure with `temper_api_url=http://127.0.0.1:39999` -- Provision result: - - `status=Failed` - - `error_message="TemperFS bootstrap failed at http://127.0.0.1:39999/tdata ... Ensure os-app 'temper-fs' is installed ... temper_api_url is correct."` - - `workspace_id`, `conversation_file_id`, `file_manifest_id` all empty - -Case B (correct API URL): -- Agent: `019d1c5a-4f84-7d23-8c18-dfc23885e3b9` -- Configure with `temper_api_url=http://127.0.0.1:3015` -- Provision + callback result: - - `status=Failed` (intentional due `max_turns=0`) - - `error_message="turn budget exhausted (0/0)"` - - `workspace_id`, `conversation_file_id`, `file_manifest_id` all populated - -Interpretation: -- Failure reflection is now explicit and truthful for TemperFS bootstrap issues. -- With correct URL, TemperFS bootstrap succeeds and IDs are present. - -## Operational Caveat -If tenant-scoped WASM modules are not uploaded, integration dispatch fails with `WASM module '' not found`. This is separate from TemperFS bootstrap handling. diff --git a/.proof/2026-04-24-action-triggers-merge-main-live-e2e.md b/.proof/2026-04-24-action-triggers-merge-main-live-e2e.md deleted file mode 100644 index 7750b5596..000000000 --- a/.proof/2026-04-24-action-triggers-merge-main-live-e2e.md +++ /dev/null @@ -1,62 +0,0 @@ -# Action Triggers Merge-Main Live E2E - -Date: 2026-04-24 -Repo: `/Users/seshendranalla/Development/temper-action-triggers` -Branch: `feat/action-triggers-unified` - -## Scope - -- Verify the merged branch after folding in `origin/main`. -- Prove standalone Temper still executes the new `temper-fs` trigger architecture live. -- Confirm explicit `FileVersion` lineage and immutable batch reads on a fresh local server. - -## Server - -Started a fresh standalone Temper server: - -```bash -PORT=4461 \ -TEMPER_API_KEY=temper-live-key \ -TURSO_URL=file:/tmp/temper-merge-e2e.db \ -RUST_LOG=info \ -cargo run -p temper-cli -- serve --no-observe --port 4461 -``` - -## Commands - -1. `curl -fsS http://127.0.0.1:4461/healthz` - Result: healthy -2. `curl -fsS -H 'Authorization: Bearer temper-live-key' -H 'content-type: application/json' -d '{"tenant":"default"}' http://127.0.0.1:4461/api/os-apps/temper-fs/install` - Result: `temper-fs` installed for tenant `default` -3. Live file lineage replay: - - `POST /tdata/Files` - - `PUT /tdata/Files('')/$value` with `first version from temper merge proof` - - `PUT /tdata/Files('')/$value` with `second version from temper merge proof` - - `GET /tdata/Files('')` - - `GET /tdata/FileVersions?$top=200` - - `POST /api/files/read-version-text-batch` - -## Observed Result - -```json -{ - "file_id": "fl-019dbff6-27d2-70c3-ab13-a8db28ff6be6", - "file_status": "Ready", - "version_count": 2, - "last_version_id": "019dbff6-27f8-70f0-95ec-1bfed8a1ca62", - "current_version_status": "Current", - "previous_version_status": "Superseded", - "latest_text": "second version from temper merge proof", - "batch_texts": [ - "first version from temper merge proof", - "second version from temper merge proof" - ] -} -``` - -## What This Proves - -- The merged standalone Temper server still installs `temper-fs` successfully after folding in `origin/main`. -- Inline entity triggers create and supersede `FileVersion` entities live. -- `File.fields.last_version_id` updates to the newest version. -- Immutable batch reads return historical content from the live server, not just the mutable file head. diff --git a/.proof/GEPA_E2E_PROOF.md b/.proof/GEPA_E2E_PROOF.md deleted file mode 100644 index 828fdbd73..000000000 --- a/.proof/GEPA_E2E_PROOF.md +++ /dev/null @@ -1,1625 +0,0 @@ -# GEPA End-to-End Proof (TemperAgent + OTS + Workflow Replay) - -**Date**: 2026-03-23 -**Workspace**: `/Users/seshendranalla/Development/temper-gepa-tarjan` -**Server**: `temper serve --port 4455 --storage turso --no-observe` -**Primary tenant**: `gepa-live-fresh-20260319` -**Primary run**: `EvolutionRun('evo-live-fresh-20260319-v4')` - -## Scope and Constraint -- This document is the canonical live-proof report. -- It includes the full trajectory taxonomy and trigger semantics discussed in chat. -- GEPA naming and data-model naming are intentionally unchanged in this update. -- This report focuses on what was *actually* proven in live runs, and explicitly lists what did not work. - -## GEPA Optimizer-Only Policy (2026-03-23 update) -- GEPA is now explicitly scoped to optimization of existing capability. -- Structural mutations are blocked in `gepa-proposer-agent`: - - no entity rename/introduction/removal - - no action add/remove - - no state add/remove -- When a proposal implies net-new capability, proposer performs unmet-intent handoff: - - emits `UnmetIntentHandoff` metadata in proposer output - - best-effort POSTs to `/api/evolution/trajectories/unmet` for separate unmet-intent processing -- GEPA returns a no-op mutation (`MutatedSpecSource = original`) when the structural gate blocks mutation. -- `patterns.missing_capabilities` remains available in reflective data, but is routed to unmet-intent handoff rather than direct structural edits by GEPA. - -## 2026-03-23 Full-Loop Re-Proof (Latest) -- **Tenant**: `gepa-live-20260323-121726` -- **Primary terminal run**: `EvolutionRun('evo-live-20260323-121726-v3')` -- **Artifacts dir**: `/tmp/gepa_run_20260323-121726` - -### What was proven in this latest run -1. **Automatic verify/deploy path now works end-to-end** (no manual steering): - - Terminal action chain: - `Created -> Start -> SelectCandidate -> RecordEvaluation -> RecordDataset -> RecordMutation -> RecordVerificationPass -> RecordScore -> RecordFrontierAutoApprove -> Deploy` - - Run status reached `Completed` with: - - `VerificationReport = "verification passed: 4 levels passed"` - - `DeploymentId = "gepa-deploy-evo-live-20260323-121726-v3-m1"` -2. **Unmet-intent handoff persists even when optimizer mutation path is allowed/continues**: - - `UnmetIntentReport` present in run fields with `reported = 3, failed = 0`. - - Reported intents included: - - `Add action 'Reassign'` - - `PromoteToCritical` - - `Reassign` - - These were also persisted into trajectory telemetry as `source=Platform` unmet records (visible in `/observe/trajectories`). -3. **Workflow-level GEPA path remained active**: - - OTS was seeded by real `temper mcp` sessions (success/partial/failure). - - `SelectCandidate` still omitted `TrajectoryActions`/`Trajectories`; replay consumed OTS auto-injected server-side. - -### What was fixed during this cycle -- `EvolutionRun` automation: - - Added `gepa-verify` module + `verify_candidate` trigger from `RecordMutation`. - - Added `gepa-deploy` module + `deploy_candidate` trigger from auto-approve and manual approve paths. - - `gepa-pareto` now emits dynamic callback action based on `AutonomyLevel` (`RecordFrontierAutoApprove` vs `RecordFrontier`). -- SDK-callback pitfall addressed: - - `gepa-verify`, `gepa-deploy`, and `gepa-pareto` were moved to explicit callback action emission (not macro-default `callback`) so action dispatch works. -- Proposer unmet-intent behavior: - - `gepa-proposer-agent` now reports unmet intents even when mutation proceeds. - -### Still open (not yet fully proven fixed) -1. **OTS ID consistency issue remains**: - - `flush_trajectory()` returned IDs still did not match IDs listed by `/api/ots/trajectories`. - - This means row ID alignment between MCP flush/finalize and listed OTS rows is still not proven fixed in live evidence. -2. This is tracked as an active blocker for the “single stable OTS ID per session” guarantee. - -## 2026-03-23 Bounded Re-Proof (Current) -- **Tenant**: `gepa-live-20260323-125726` -- **Primary run**: `EvolutionRun('evo-live-20260323-125726')` -- **Artifacts dir**: `/tmp/gepa_run_20260323-125726` - -### What was proven in this run -1. **WASM + secret setup path worked**: - - 12/12 module uploads succeeded (`wasm_upload_results.json`). - - Tenant secret `anthropic_api_key` stored successfully (`put_secret_anthropic_code.txt = 204`). -2. **Real OTS generation path worked**: - - Real `temper mcp` sessions produced success/partial/failed trajectories. - - `flush_trajectory()` returned concrete trajectory IDs for each session. -3. **OTS row-vs-payload ID mismatch is fixed in this isolated DB run**: - - `ots_id_consistency_summary.json`: - - `total_rows = 3` - - `matching_rows = 3` - - `mismatching_rows = 0` - - This shows persisted row `trajectory_id` now matches payload `$.trajectory_id`. - -### What failed in this run -1. The GEPA run reached `Proposing` and then failed: - - Event trail: - - `Created -> Start -> SelectCandidate -> RecordEvaluation -> RecordDataset -> Fail` - - Failure payload: - - `error = "authorization denied for http_call: no matching permit policy"` - - `integration = "propose_mutation"` - - `authz_denied = true` -2. Because proposer failed at `Proposing`, this run did not reach: - - `RecordMutation` - - `RecordVerificationPass` - - `RecordFrontierAutoApprove` - - `Deploy` -3. This run therefore cannot be used to re-prove auto verify/deploy or unmet-intent persistence; those remain proven by the prior successful run (`evo-live-20260323-121726-v3`). - -## 2026-03-23 Consolidated Full-Loop Re-Proof (All Three Aspects) -- **Tenant**: `gepa-live-20260323-134346` -- **Run**: `EvolutionRun('evo-live-20260323-134346')` -- **Artifacts dir**: `/tmp/gepa_run_20260323-134346` -- **Terminal status**: `Completed` - -### End-to-end path proven in this run -- `Created -> Start -> SelectCandidate -> RecordEvaluation -> RecordDataset -> RecordMutation -> RecordVerificationPass -> RecordScore -> RecordFrontierAutoApprove -> Deploy` -- Final run fields include: - - `VerificationReport = "verification passed: 4 levels passed"` - - `DeploymentId = "gepa-deploy-evo-live-20260323-134346-m1"` - - `UnmetIntentReport.attempted = 4`, `reported = 4`, `failed = 0` - -### The three requested aspects are now proven together -1. **Unmet-intent storage during optimizer flow** - - Missing-capability suggestions were surfaced by reflective/proposer and persisted via `/api/evolution/trajectories/unmet`. - - Evidence: - - `UnmetIntentReport` in final `EvolutionRun` fields with all reports successful. - - DB rows (`trajectories`) with `source = Platform`, `intent != null` for this tenant (`4` rows). - - This confirms unmet-intent handoff is not dropped when GEPA continues optimizer flow. - -2. **OTS trajectory ID consistency (flush vs stored rows)** - - `ots_id_consistency_summary.json` reports: - - `total_rows = 3` - - `matching_rows = 3` - - `mismatching_rows = 0` - - Flush IDs now match persisted OTS row payload IDs in this live run. - -3. **No manual verification/deploy steering** - - Verifier and deploy steps fired from integrations automatically and reached `Completed`. - - No manual `RecordVerificationPass`, `Approve`, or `Deploy` action calls were needed. - -### Root causes fixed to get this full loop green -1. **WASM Cedar authz tenant scope** - - `CedarWasmAuthzGate` now uses tenant-scoped authorization (`authorize_for_tenant_or_bypass`) for `http_call` and `access_secret`. -2. **WASM HTTP policy context mismatch** - - Evolution policy switched from `resource.domain` to `context.domain` for HTTP-call host checks. -3. **Internal API auth for proposer/verifier** - - `gepa-proposer-agent` and `gepa-verify` now attach `Authorization: Bearer ...` using: - - integration config (`temper_api_key = {secret:temper_api_key}`), plus - - fallback `get_secret("temper_api_key")`. -4. **Policy permissions for proposer/verifier ops** - - Added Cedar permits for: - - `http_call` from `gepa-proposer-agent` and `gepa-verify` to localhost - - `access_secret` for those modules - - `write_trajectories` for proposer (`Agent::"gepa-proposer-agent"`) so unmet intents persist. -5. **State-machine terminal handling on verifier faults** - - `EvolutionRun.Fail` now allows `from = "Verifying"` so verifier integration failures terminate cleanly instead of stalling. - -### Remaining caveat observed (non-blocking for this proof) -- `sandbox_provisioner` logs `TemperFS setup failed: Workspace creation failed (HTTP 404)` during TemperAgent provisioning in this environment. -- Despite that warning, the GEPA run still completed end-to-end (proposer response returned, verifier passed, deploy completed). - -## Executive Result -1. Real OTS trajectories were generated by real `temper mcp` sessions (no fabricated JSON). -2. `SelectCandidate` was executed without `TrajectoryActions` and without `Trajectories`; replay still consumed OTS from server-side auto-injection. -3. `gepa-replay` produced workflow-level results (`workflows[]`, `workflow_completion_rate`, `partial_adjusted_rate`) and action-level aggregates. -4. `gepa-reflective` produced workflow-level triplets and cross-trajectory patterns (missing capabilities, common failure points, successful patterns). -5. Latest consolidated run (`evo-live-20260323-134346`) completed end-to-end through verify, score, frontier update, and deploy. -6. Unmet-intent handoff now persists successfully during optimizer flow (`attempted=4`, `reported=4`, `failed=0`) while GEPA remains optimizer-only. -7. OTS row ID and payload trajectory ID matched for all seeded trajectories in the latest run (`3/3`). -8. Historical failures (proposer authz/401, verifier authz, `Fail` not valid from `Verifying`) are documented in prior sections and were resolved for the latest run. - -## What "the run" means in this report -A "run" here means one full `EvolutionRun` entity state-machine attempt from `Start` through terminal state (`Completed` or `Failed`). - -For the latest consolidated proof run `evo-live-20260323-134346`, the terminal path was: -- `Created -> Start -> SelectCandidate -> RecordEvaluation -> RecordDataset -> RecordMutation -> RecordVerificationPass -> RecordScore -> RecordFrontierAutoApprove -> Deploy -> Completed` - -No manual trajectory payload was provided to `SelectCandidate`; OTS data came from tenant OTS storage. - -## Trajectory Taxonomy (Current Project) - -### 1. OTS trajectories (`ots_trajectories`) -- Purpose: full agent/session traces (turns, messages, decisions, consequences). -- Producer: MCP runtime (`TrajectoryBuilder`) auto-records each `execute` call turn. -- Upload paths: - - End-of-session upload (`finalize_trajectory`) - - Mid-session snapshot upload (`flush_trajectory`) -- Consumer in GEPA pipeline today: - - `gepa-replay` gets OTS auto-injected when `SelectCandidate` does not provide trajectory params. - - `gepa-reflective` works from replay output. - -### 2. Entity/platform/authz trajectories (`trajectories`) -- Purpose: action/event telemetry per entity action (`source = Entity|Platform|Authz`, success/failure, authz denied, etc). -- Producer: entity dispatch and related platform/authz paths. -- Consumer in GEPA run today: - - Not directly consumed by `gepa-replay` in `evaluate_candidate` (that path currently uses OTS injection for GEPA). -- Consumer elsewhere: - - Observe/Evolution insight/sentinel pipelines. - -### 3. Unmet intents -- Representation: unmet-intent signals are derived from trajectory data / failures (and can be recorded through evolution unmet endpoint path). -- Consumer today: - - Observe/Evolution insight generation and sentinel monitoring. -- Consumer in GEPA run today: - - Not directly wired into `gepa-replay`/`gepa-reflective` input payload for this run. - -## Should OTS + entity/authz/unmet be merged right now? -Current behavior is intentionally separated: -- GEPA run path: OTS-centric (session workflow replay). -- Observe evolution path: trajectory/authz/unmet-intent analytics and sentinel records. - -This report does **not** rename or merge those pipelines. It documents current behavior and limitations only. - -## Triggering Model (Current State) - -### What triggers evolution runs now -- Primary proven path in this report: manual `EvolutionRun.Start` + `SelectCandidate` action invocation. -- Sentinel path exists (`temper.check_sentinel(tenant)` / server sentinel check endpoint), but in this run it is not the reliable automatic launcher for the GEPA loop. - -### What happened when sentinel was called live -- `temper.check_sentinel('gepa-live-fresh-20260319')` returned HTTP 500. -- Server logs show sentinel alerts were generated, but persistence hit `UNIQUE constraint failed: evolution_records.id` while writing multiple records in same check path. -- So sentinel currently has a real blocker in this environment. - -## Real OTS Generation in this proof - -### How the OTS rows were produced -All OTS rows below were produced by real MCP sessions (`temper mcp` with `execute` calls), not manual DB insertion. - -Session patterns used: -1. Success workflow: `Assign -> Reassign` -2. Partial workflow: `Assign -> PromoteToCritical` (`PromoteToCritical` unknown) -3. Failed workflow: `Reassign` from `Backlog` (invalid transition) -4. Flush workflow: action turn -> `flush_trajectory()` -> action turn (same session, 3 turns) - -### Important nuance found during live proof -- Tenant extraction for OTS upload is based on parsed calls. -- If calls use a variable (`tenant = ...`) instead of literal tenant string in `temper.action(...)`, uploader can fall back to `default` tenant. -- For this proof, final portfolio sessions were rerun with literal tenant strings to guarantee storage under `gepa-live-fresh-20260319`. - -## How decisions/actions/reasons are extracted -1. MCP runtime records each execute turn as OTS: - - user message = submitted code - - assistant message = runtime result / error - - decision.consequence.success = execution success/failure -2. Runtime extracts `trajectory_actions` from code and stores under `decision.choice.arguments.trajectory_actions`. -3. In replay: - - It iterates OTS turn -> decision -> `choice.arguments.trajectory_actions` first. - - If absent, it can fall back to parsing user code for action calls. -4. In reflective dataset: - - It consumes replay workflows and outcomes. - - Produces triplets + pattern summaries. - -## Fresh E2E Run (`evo-live-fresh-20260319-v4`) - -### Start/select invocation -- `Start` invoked with: - - `SkillName = project-management` - - `TargetEntityType = Issue` - - `AutonomyLevel = auto` -- `SelectCandidate` invoked with: - - `CandidateId` - - `SpecSource` -- Omitted intentionally: - - `TrajectoryActions` - - `Trajectories` - -### Observed status timeline -- `Evaluating` -- `Proposing` -- `Failed` - -### Final failure reason -`TemperAgent Failed on retry 1: Anthropic API returned 401: invalid x-api-key` - -## Workflow-level replay result from the fresh run -- `workflows_total = 8` -- `workflows_completed = 1` -- `workflows_partial = 3` -- `workflows_failed = 1` -- `workflows_empty = 3` -- `workflow_completion_rate = 0.2` -- `partial_adjusted_rate = 0.5` -- `actions_attempted = 8` -- `succeeded = 4` -- `success_rate = 0.5` -- `coverage = 0.875` - -## Reflective dataset result from the fresh run -- `success_count = 1` -- `failure_count = 4` -- `workflow_counts = {completed:1, partial:3, failed:1}` -- `patterns.missing_capabilities = ["PromoteToCritical"]` -- `patterns.common_failure_points` includes repeated `Reassign` from `Backlog` -- `patterns.successful_patterns` includes preserved success pattern with `Assign` - -## What worked -1. Real MCP-generated OTS capture and persistence. -2. Mid-session OTS flush API path (`flush_trajectory`) returns real trajectory IDs. -3. OTS auto-injection into `gepa-replay` when trajectory params are omitted. -4. Workflow-level replay and reflective outputs produced in-run. -5. TemperAgent proposer integration is invoked (reaches proposer stage). - -## What did not work / current blockers -1. Anthropic auth for proposer failed (`401 invalid x-api-key`), so no mutation was produced in this run. -2. Sentinel check endpoint produced `500` due duplicate `evolution_records.id` collisions. -3. OTS row trajectory id and payload trajectory id are different values in storage (documented below); this can confuse artifact tracing if not explicitly mapped. -4. Outcome at OTS metadata level is often `success` even when inner decision consequence is failure; replay still classifies workflow failure correctly from decision/action-level errors. - -## Architecture Diagram (Proven Path) -```text -MCP execute sessions - -> OTS TrajectoryBuilder (turns/decisions) - -> /api/ots/trajectories persisted - -> EvolutionRun.Start - -> SelectCandidate (without TrajectoryActions/Trajectories) - -> server auto-injects OTS into gepa-replay trigger params - -> gepa-replay (workflow outcomes + action stats) - -> gepa-reflective (triplets + patterns) - -> gepa-proposer-agent via TemperAgent - -> FAILED in this run (Anthropic 401 invalid key) -``` - -## Data-Pipeline Diagram (Taxonomy) -```text - +-------------------------------+ - | trajectories (Entity/Platform/Authz) -Actions/dispatch ------>| source-tagged action records |----+ - +-------------------------------+ | - | used by - v - +-------------------------------+ Observe evolution - | unmet intent / insight paths |--- sentinel / insights - +-------------------------------+ - -MCP execute sessions ---> OTS (turn/message/decision traces) ---> GEPA replay -> reflective -> proposer - ^ - | - flush_trajectory() snapshot -``` - -## Evidence: entity/authz/platform/unmet in this environment -- For `gepa-live-fresh-20260319`, `trajectories` table had only `source=Entity` rows in this proof run. -- Authz/platform trajectory rows exist in other tenants (captured separately below). -- `intent IS NOT NULL` rows count is `0` in this DB snapshot. - -## Artifact Index -- OTS list (API): `/tmp/ots_fresh2_list.json` -- OTS row metadata (sqlite): `/tmp/ots_fresh2_rows_sqlite.json` -- OTS row-vs-payload trajectory IDs: `/tmp/ots_fresh2_row_vs_payload_ids.json` -- Full OTS examples: - - `/tmp/ots_fresh2_success_full.json` - - `/tmp/ots_fresh2_partial_full.json` - - `/tmp/ots_fresh2_failed_full.json` - - `/tmp/ots_fresh2_flushseq_full.json` -- Evolution run artifacts: - - `/tmp/evo_live_fresh_v4_report.json` - - `/tmp/evo_live_fresh_v4_final.json` - - `/tmp/evo_live_fresh_v4_replay.json` - - `/tmp/evo_live_fresh_v4_dataset.json` -- Auxiliary telemetry snapshots: - - `/tmp/fresh_entity_traj_source_counts.json` - - `/tmp/fresh_entity_traj_totals.json` - - `/tmp/fresh_entity_traj_recent20.json` - - `/tmp/trajectory_authz_platform_counts.json` - - `/tmp/trajectory_unmet_intents_count.json` - ---- - -## Appendix A: OTS Row vs Payload Trajectory IDs - -```json -[{"row_trajectory_id":"019d087a-6c0d-7801-8f8e-e9955ebebe01","payload_trajectory_id":"019d087a-6c0d-7e40-a0b1-a5aefd7b87bb","created_at":"2026-03-19 23:42:14","turn_count":1}, -{"row_trajectory_id":"019d087a-6c17-7be0-8413-40ff7c95bbfd","payload_trajectory_id":"019d087a-6c16-74b2-9094-5768718f8d71","created_at":"2026-03-19 23:42:14","turn_count":3}, -{"row_trajectory_id":"019d087a-349e-7782-a1ba-1b7649495a7b","payload_trajectory_id":"019d087a-349d-7071-b3b0-301fc9464305","created_at":"2026-03-19 23:41:59","turn_count":1}, -{"row_trajectory_id":"019d087a-34a3-7092-8fe7-904862e7baff","payload_trajectory_id":"019d087a-34a2-7cf1-a894-b4e50c0b0fd9","created_at":"2026-03-19 23:41:59","turn_count":1}, -{"row_trajectory_id":"019d0879-90af-7f10-a572-6a6d7021dfb6","payload_trajectory_id":"019d0879-90af-7922-a5ea-b08864af0ca9","created_at":"2026-03-19 23:41:17","turn_count":1}, -{"row_trajectory_id":"019d0874-845a-7a71-a9fd-023f18d71474","payload_trajectory_id":"019d0874-8459-7352-b4d4-e1cfc83f456b","created_at":"2026-03-19 23:35:47","turn_count":1}, -{"row_trajectory_id":"019d0874-451c-7370-9d82-0a110cd8507b","payload_trajectory_id":"019d0874-451a-7e12-b13d-9fd40c41f1e2","created_at":"2026-03-19 23:35:30","turn_count":1}, -{"row_trajectory_id":"019d0872-e05e-7430-8e89-32f8e4c2e41d","payload_trajectory_id":"019d0872-e05d-7953-87c6-99fcf0b68da0","created_at":"2026-03-19 23:33:59","turn_count":1}] -``` - -## Appendix B: Full OTS Example (Success) - -```json -{ - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "version": "0.1.0", - "metadata": { - "task_description": "mcp-session", - "timestamp_start": "2026-03-19T23:41:17.849216Z", - "timestamp_end": "2026-03-19T23:41:17.871124Z", - "duration_ms": 21.0, - "agent_id": "unknown", - "outcome": "success", - "human_reviewed": false - }, - "context": {}, - "turns": [ - { - "turn_id": 1, - "span_id": "019d0879-90ae-7e22-8c55-bb311785afdb", - "timestamp": "2026-03-19T23:41:17.870853Z", - "duration_ms": 0.0, - "error": false, - "messages": [ - { - "message_id": "019d0879-90ae-7e22-8c55-bb4bf38d9ef8", - "role": "user", - "timestamp": "2026-03-19T23:41:17.870853Z", - "content": { - "type": "text", - "text": "created = await temper.create(\"gepa-live-fresh-20260319\", \"Issues\", {\"Id\": \"issue-fresh2-success-1\", \"Title\": \"fresh2 ots success\", \"CreatedAt\": \"2026-03-19T00:00:00Z\", \"UpdatedAt\": \"2026-03-19T00:00:00Z\"})\nissue_id = created[\"entity_id\"]\na1 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"Assign\", {\"AgentId\": \"agent-success2-1\", \"Reason\": \"fresh2-success\"})\na2 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"Reassign\", {\"NewAssigneeId\": \"agent-success2-2\", \"Reason\": \"fresh2-success\"})\nreturn {\"issue_id\": issue_id, \"assign\": a1, \"reassign\": a2}" - } - }, - { - "message_id": "019d0879-90ae-7e22-8c55-bb554dd55c01", - "role": "assistant", - "timestamp": "2026-03-19T23:41:17.870853Z", - "content": { - "type": "text", - "text": "{\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\",\"Status\":\"Backlog\",\"AgentId\":\"agent-success2-1\",\"Reason\":\"fresh2-success\",\"assignee_set\":true},\"events\":[{\"action\":\"Created\",\"from_status\":\"\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.852263Z\",\"params\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\"}},{\"action\":\"Assign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.857935Z\",\"params\":{\"AgentId\":\"agent-success2-1\",\"Reason\":\"fresh2-success\"}}],\"total_event_count\":2,\"sequence_nr\":2,\"@odata.context\":\"$metadata#Issues/$entity\"},\"reassign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\",\"Status\":\"Backlog\",\"AgentId\":\"agent-success2-1\",\"Reason\":\"fresh2-success\",\"assignee_set\":true,\"NewAssigneeId\":\"agent-success2-2\"},\"events\":[{\"action\":\"Created\",\"from_status\":\"\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.852263Z\",\"params\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\"}},{\"action\":\"Assign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.857935Z\",\"params\":{\"AgentId\":\"agent-success2-1\",\"Reason\":\"fresh2-success\"}},{\"action\":\"Reassign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.865255Z\",\"params\":{\"NewAssigneeId\":\"agent-success2-2\",\"Reason\":\"fresh2-success\"}}],\"total_event_count\":3,\"sequence_nr\":3,\"@odata.context\":\"$metadata#Issues/$entity\"}}" - } - } - ], - "decisions": [ - { - "decision_id": "019d0879-90ae-7e22-8c55-bb6f3c7b52a4", - "decision_type": "tool_selection", - "choice": { - "action": "execute: created = await temper.create(\"gepa-live-fresh-20260319\", \"Issues\", {\"Id\": \"issue-fresh2-success-1\",", - "arguments": { - "trajectory_actions": [ - { - "action": "Assign", - "params": { - "AgentId": "agent-success2-1", - "Reason": "fresh2-success" - } - }, - { - "action": "Reassign", - "params": { - "NewAssigneeId": "agent-success2-2", - "Reason": "fresh2-success" - } - } - ] - } - }, - "consequence": { - "success": true - } - } - ] - } - ] -} -``` - -## Appendix C: Full OTS Example (Partial) - -```json -{ - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "version": "0.1.0", - "metadata": { - "task_description": "mcp-session", - "timestamp_start": "2026-03-19T23:41:59.826047Z", - "timestamp_end": "2026-03-19T23:41:59.842733Z", - "duration_ms": 16.0, - "agent_id": "unknown", - "outcome": "success", - "human_reviewed": false - }, - "context": {}, - "turns": [ - { - "turn_id": 1, - "span_id": "019d087a-34a2-7cf1-a894-b4ab375b2689", - "timestamp": "2026-03-19T23:41:59.842551Z", - "duration_ms": 0.0, - "error": false, - "messages": [ - { - "message_id": "019d087a-34a2-7cf1-a894-b4b497f31915", - "role": "user", - "timestamp": "2026-03-19T23:41:59.842551Z", - "content": { - "type": "text", - "text": "created = await temper.create(\"gepa-live-fresh-20260319\", \"Issues\", {\"Id\": \"issue-fresh2-partial-1\", \"Title\": \"fresh2 ots partial\", \"CreatedAt\": \"2026-03-19T00:00:00Z\", \"UpdatedAt\": \"2026-03-19T00:00:00Z\"})\nissue_id = created[\"entity_id\"]\na1 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"Assign\", {\"AgentId\": \"agent-partial2-1\", \"Reason\": \"fresh2-partial\"})\na2 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"PromoteToCritical\", {\"Reason\": \"fresh2-partial\"})\nreturn {\"issue_id\": issue_id, \"assign\": a1, \"promote\": a2}" - } - }, - { - "message_id": "019d087a-34a2-7cf1-a894-b4ce46dae713", - "role": "assistant", - "timestamp": "2026-03-19T23:41:59.842551Z", - "content": { - "type": "text", - "text": "RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical" - } - } - ], - "decisions": [ - { - "decision_id": "019d087a-34a2-7cf1-a894-b4d28293ce24", - "decision_type": "tool_selection", - "choice": { - "action": "execute: created = await temper.create(\"gepa-live-fresh-20260319\", \"Issues\", {\"Id\": \"issue-fresh2-partial-1\",", - "arguments": { - "trajectory_actions": [ - { - "action": "Assign", - "params": { - "AgentId": "agent-partial2-1", - "Reason": "fresh2-partial" - } - }, - { - "action": "PromoteToCritical", - "params": { - "Reason": "fresh2-partial" - } - } - ] - } - }, - "consequence": { - "success": false, - "error_type": "RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical" - } - } - ] - } - ] -} -``` - -## Appendix D: Full OTS Example (Failed) - -```json -{ - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "version": "0.1.0", - "metadata": { - "task_description": "mcp-session", - "timestamp_start": "2026-03-19T23:41:59.825756Z", - "timestamp_end": "2026-03-19T23:41:59.837842Z", - "duration_ms": 12.0, - "agent_id": "unknown", - "outcome": "success", - "human_reviewed": false - }, - "context": {}, - "turns": [ - { - "turn_id": 1, - "span_id": "019d087a-349d-7071-b3b0-2fd8152835bc", - "timestamp": "2026-03-19T23:41:59.837691Z", - "duration_ms": 0.0, - "error": false, - "messages": [ - { - "message_id": "019d087a-349d-7071-b3b0-2fed8b3e5341", - "role": "user", - "timestamp": "2026-03-19T23:41:59.837691Z", - "content": { - "type": "text", - "text": "created = await temper.create(\"gepa-live-fresh-20260319\", \"Issues\", {\"Id\": \"issue-fresh2-failed-1\", \"Title\": \"fresh2 ots failed\", \"CreatedAt\": \"2026-03-19T00:00:00Z\", \"UpdatedAt\": \"2026-03-19T00:00:00Z\"})\nissue_id = created[\"entity_id\"]\na1 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"Reassign\", {\"NewAssigneeId\": \"agent-failed2-1\", \"Reason\": \"fresh2-failed\"})\nreturn {\"issue_id\": issue_id, \"reassign\": a1}" - } - }, - { - "message_id": "019d087a-349d-7071-b3b0-2ff9a7fb3691", - "role": "assistant", - "timestamp": "2026-03-19T23:41:59.837691Z", - "content": { - "type": "text", - "text": "RuntimeError: HTTP 409 Conflict: Action 'Reassign' not valid from state 'Backlog'" - } - } - ], - "decisions": [ - { - "decision_id": "019d087a-349d-7071-b3b0-300b1bfc5d0f", - "decision_type": "tool_selection", - "choice": { - "action": "execute: created = await temper.create(\"gepa-live-fresh-20260319\", \"Issues\", {\"Id\": \"issue-fresh2-failed-1\", ", - "arguments": { - "trajectory_actions": [ - { - "action": "Reassign", - "params": { - "NewAssigneeId": "agent-failed2-1", - "Reason": "fresh2-failed" - } - } - ] - } - }, - "consequence": { - "success": false, - "error_type": "RuntimeError: HTTP 409 Conflict: Action 'Reassign' not valid from state 'Backlog'" - } - } - ] - } - ] -} -``` - -## Appendix E: Full OTS Example (Flush Sequence) - -```json -{ - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "version": "0.1.0", - "metadata": { - "task_description": "mcp-session", - "timestamp_start": "2026-03-19T23:42:14.020954Z", - "timestamp_end": "2026-03-19T23:42:14.038870Z", - "duration_ms": 17.0, - "agent_id": "unknown", - "outcome": "success", - "human_reviewed": false - }, - "context": {}, - "turns": [ - { - "turn_id": 1, - "span_id": "019d087a-6c0d-7e40-a0b1-a56042316d07", - "timestamp": "2026-03-19T23:42:14.029227Z", - "duration_ms": 0.0, - "error": false, - "messages": [ - { - "message_id": "019d087a-6c0d-7e40-a0b1-a57fac37a429", - "role": "user", - "timestamp": "2026-03-19T23:42:14.029227Z", - "content": { - "type": "text", - "text": "issue_id = \"019d0879-909a-73b3-a811-9b0cbfb0b89b\"\na1 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"Assign\", {\"AgentId\": \"agent-flush2-1\", \"Reason\": \"fresh2-flush\"})\nreturn {\"issue_id\": issue_id, \"assign\": a1}" - } - }, - { - "message_id": "019d087a-6c0d-7e40-a0b1-a5890bb3544f", - "role": "assistant", - "timestamp": "2026-03-19T23:42:14.029227Z", - "content": { - "type": "text", - "text": "{\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\",\"Status\":\"Backlog\",\"AgentId\":\"agent-flush2-1\",\"Reason\":\"fresh2-flush\",\"assignee_set\":true,\"NewAssigneeId\":\"agent-success2-2\"},\"events\":[{\"action\":\"Created\",\"from_status\":\"\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.852263Z\",\"params\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\"}},{\"action\":\"Assign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.857935Z\",\"params\":{\"AgentId\":\"agent-success2-1\",\"Reason\":\"fresh2-success\"}},{\"action\":\"Reassign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.865255Z\",\"params\":{\"NewAssigneeId\":\"agent-success2-2\",\"Reason\":\"fresh2-success\"}},{\"action\":\"Assign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:42:14.025360Z\",\"params\":{\"AgentId\":\"agent-flush2-1\",\"Reason\":\"fresh2-flush\"}}],\"total_event_count\":4,\"sequence_nr\":4,\"@odata.context\":\"$metadata#Issues/$entity\"}}" - } - } - ], - "decisions": [ - { - "decision_id": "019d087a-6c0d-7e40-a0b1-a594fd403bee", - "decision_type": "tool_selection", - "choice": { - "action": "execute: issue_id = \"019d0879-909a-73b3-a811-9b0cbfb0b89b\"\na1 = await temper.action(\"gepa-live-fresh-20260319", - "arguments": { - "trajectory_actions": [ - { - "action": "Assign", - "params": { - "AgentId": "agent-flush2-1", - "Reason": "fresh2-flush" - } - } - ] - } - }, - "consequence": { - "success": true - } - } - ] - }, - { - "turn_id": 2, - "span_id": "019d087a-6c0f-71b1-a2e6-1649d65bf242", - "timestamp": "2026-03-19T23:42:14.031216Z", - "duration_ms": 0.0, - "error": false, - "messages": [ - { - "message_id": "019d087a-6c0f-71b1-a2e6-1652f1b67067", - "role": "user", - "timestamp": "2026-03-19T23:42:14.031216Z", - "content": { - "type": "text", - "text": "return await temper.flush_trajectory()" - } - }, - { - "message_id": "019d087a-6c0f-71b1-a2e6-1668edd708d1", - "role": "assistant", - "timestamp": "2026-03-19T23:42:14.031216Z", - "content": { - "type": "text", - "text": "{\"trajectory_id\":\"019d087a-6c0d-7e40-a0b1-a5aefd7b87bb\",\"status\":\"flushed\"}" - } - } - ], - "decisions": [ - { - "decision_id": "019d087a-6c0f-71b1-a2e6-167dc178be5c", - "decision_type": "tool_selection", - "choice": { - "action": "execute: return await temper.flush_trajectory()" - }, - "consequence": { - "success": true - } - } - ] - }, - { - "turn_id": 3, - "span_id": "019d087a-6c16-74b2-9094-572b526c89ed", - "timestamp": "2026-03-19T23:42:14.038658Z", - "duration_ms": 0.0, - "error": false, - "messages": [ - { - "message_id": "019d087a-6c16-74b2-9094-5731acf871f4", - "role": "user", - "timestamp": "2026-03-19T23:42:14.038658Z", - "content": { - "type": "text", - "text": "issue_id = \"019d0879-909a-73b3-a811-9b0cbfb0b89b\"\na2 = await temper.action(\"gepa-live-fresh-20260319\", \"Issues\", issue_id, \"Reassign\", {\"NewAssigneeId\": \"agent-flush2-2\", \"Reason\": \"fresh2-flush\"})\nreturn {\"issue_id\": issue_id, \"reassign\": a2}" - } - }, - { - "message_id": "019d087a-6c16-74b2-9094-574dad1e8c03", - "role": "assistant", - "timestamp": "2026-03-19T23:42:14.038658Z", - "content": { - "type": "text", - "text": "{\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"reassign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\",\"Status\":\"Backlog\",\"AgentId\":\"agent-flush2-1\",\"Reason\":\"fresh2-flush\",\"assignee_set\":true,\"NewAssigneeId\":\"agent-flush2-2\"},\"events\":[{\"action\":\"Created\",\"from_status\":\"\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.852263Z\",\"params\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T00:00:00Z\",\"UpdatedAt\":\"2026-03-19T00:00:00Z\"}},{\"action\":\"Assign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.857935Z\",\"params\":{\"AgentId\":\"agent-success2-1\",\"Reason\":\"fresh2-success\"}},{\"action\":\"Reassign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:41:17.865255Z\",\"params\":{\"NewAssigneeId\":\"agent-success2-2\",\"Reason\":\"fresh2-success\"}},{\"action\":\"Assign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:42:14.025360Z\",\"params\":{\"AgentId\":\"agent-flush2-1\",\"Reason\":\"fresh2-flush\"}},{\"action\":\"Reassign\",\"from_status\":\"Backlog\",\"to_status\":\"Backlog\",\"timestamp\":\"2026-03-19T23:42:14.035170Z\",\"params\":{\"NewAssigneeId\":\"agent-flush2-2\",\"Reason\":\"fresh2-flush\"}}],\"total_event_count\":5,\"sequence_nr\":5,\"@odata.context\":\"$metadata#Issues/$entity\"}}" - } - } - ], - "decisions": [ - { - "decision_id": "019d087a-6c16-74b2-9094-5752c170c6e6", - "decision_type": "tool_selection", - "choice": { - "action": "execute: issue_id = \"019d0879-909a-73b3-a811-9b0cbfb0b89b\"\na2 = await temper.action(\"gepa-live-fresh-20260319", - "arguments": { - "trajectory_actions": [ - { - "action": "Reassign", - "params": { - "NewAssigneeId": "agent-flush2-2", - "Reason": "fresh2-flush" - } - } - ] - } - }, - "consequence": { - "success": true - } - } - ] - } - ] -} -``` - -## Appendix F: Full Replay Output (`gepa-replay`) - -```json -{ - "action_results": [ - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-flush2-1", - "Reason": "fresh2-flush" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d087a-6c0d-7e40-a0b1-a5aefd7b87bb", - "turn_index": 0 - }, - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-flush2-1", - "Reason": "fresh2-flush" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 0 - }, - { - "action": "Reassign", - "error": null, - "error_kind": "invalid_transition", - "from_state": "Backlog", - "params": { - "NewAssigneeId": "agent-flush2-2", - "Reason": "fresh2-flush" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 2 - }, - { - "action": "Reassign", - "error": null, - "error_kind": "invalid_transition", - "from_state": "Backlog", - "params": { - "NewAssigneeId": "agent-failed2-1", - "Reason": "fresh2-failed" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_index": 0 - }, - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-partial2-1", - "Reason": "fresh2-partial" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - }, - { - "action": "PromoteToCritical", - "error": "unknown action 'PromoteToCritical' in state 'Backlog'", - "error_kind": "unknown_action", - "from_state": "Backlog", - "params": { - "Reason": "fresh2-partial" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - }, - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-success2-1", - "Reason": "fresh2-success" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - }, - { - "action": "Reassign", - "error": null, - "error_kind": "invalid_transition", - "from_state": "Backlog", - "params": { - "NewAssigneeId": "agent-success2-2", - "Reason": "fresh2-success" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - } - ], - "action_stats": { - "attempted": 8, - "coverage": 0.875, - "guard_pass_rate": 1.0, - "guard_rejections": 0, - "invalid_transitions": 3, - "succeeded": 4, - "success_rate": 0.5, - "transition_validity": 0.625, - "unknown_actions": 1 - }, - "actions_attempted": 8, - "coverage": 0.875, - "errors": [ - { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 2 - }, - { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_index": 0 - }, - { - "action": "PromoteToCritical", - "error_kind": "unknown_action", - "from_state": "Backlog", - "message": "unknown action 'PromoteToCritical' in state 'Backlog'", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - }, - { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - } - ], - "guard_pass_rate": 1.0, - "guard_rejections": 0, - "invalid_transitions": 3, - "partial_adjusted_rate": 0.5, - "per_action": { - "Assign": { - "attempted": 4, - "guard_rejections": 0, - "invalid_transitions": 0, - "succeeded": 4, - "unknown_actions": 0 - }, - "PromoteToCritical": { - "attempted": 1, - "guard_rejections": 0, - "invalid_transitions": 0, - "succeeded": 0, - "unknown_actions": 1 - }, - "Reassign": { - "attempted": 3, - "guard_rejections": 0, - "invalid_transitions": 3, - "succeeded": 0, - "unknown_actions": 0 - } - }, - "succeeded": 4, - "success_rate": 0.5, - "transition_validity": 0.625, - "unknown_actions": 1, - "workflow_completion_rate": 0.2, - "workflows": [ - { - "action_results": [ - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-flush2-1", - "Reason": "fresh2-flush" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d087a-6c0d-7e40-a0b1-a5aefd7b87bb", - "turn_index": 0 - } - ], - "action_sequence": [ - "Assign" - ], - "actions_attempted": 1, - "actions_succeeded": 1, - "actions_total": 1, - "agent_goal": "success", - "breakdown": null, - "breakdown_point": null, - "errors": [], - "final_state": "Backlog", - "outcome": "completed", - "reasoning_chain": "turn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0", - "trajectory_id": "019d087a-6c0d-7e40-a0b1-a5aefd7b87bb" - }, - { - "action_results": [ - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-flush2-1", - "Reason": "fresh2-flush" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 0 - }, - { - "action": "Reassign", - "error": null, - "error_kind": "invalid_transition", - "from_state": "Backlog", - "params": { - "NewAssigneeId": "agent-flush2-2", - "Reason": "fresh2-flush" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 2 - } - ], - "action_sequence": [ - "Assign", - "Reassign" - ], - "actions_attempted": 2, - "actions_succeeded": 1, - "actions_total": 2, - "agent_goal": "success", - "breakdown": { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 2 - }, - "breakdown_point": { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 2 - }, - "errors": [ - { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_index": 2 - } - ], - "final_state": "Backlog", - "outcome": "partial", - "reasoning_chain": "turn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0\nturn 2: {\"trajectory_id\":\"019d087a-6c0d-7e40-a0b1-a5aefd7b87bb\",\"status\":\"flushed\"}\nturn 3: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"reassign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19", - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71" - }, - { - "action_results": [ - { - "action": "Reassign", - "error": null, - "error_kind": "invalid_transition", - "from_state": "Backlog", - "params": { - "NewAssigneeId": "agent-failed2-1", - "Reason": "fresh2-failed" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_index": 0 - } - ], - "action_sequence": [ - "Reassign" - ], - "actions_attempted": 1, - "actions_succeeded": 0, - "actions_total": 1, - "agent_goal": "success", - "breakdown": { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_index": 0 - }, - "breakdown_point": { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_index": 0 - }, - "errors": [ - { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_index": 0 - } - ], - "final_state": "Backlog", - "outcome": "failed", - "reasoning_chain": "turn 1: RuntimeError: HTTP 409 Conflict: Action 'Reassign' not valid from state 'Backlog'", - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305" - }, - { - "action_results": [ - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-partial2-1", - "Reason": "fresh2-partial" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - }, - { - "action": "PromoteToCritical", - "error": "unknown action 'PromoteToCritical' in state 'Backlog'", - "error_kind": "unknown_action", - "from_state": "Backlog", - "params": { - "Reason": "fresh2-partial" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - } - ], - "action_sequence": [ - "Assign", - "PromoteToCritical" - ], - "actions_attempted": 2, - "actions_succeeded": 1, - "actions_total": 2, - "agent_goal": "success", - "breakdown": { - "action": "PromoteToCritical", - "error_kind": "unknown_action", - "from_state": "Backlog", - "message": "unknown action 'PromoteToCritical' in state 'Backlog'", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - }, - "breakdown_point": { - "action": "PromoteToCritical", - "error_kind": "unknown_action", - "from_state": "Backlog", - "message": "unknown action 'PromoteToCritical' in state 'Backlog'", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - }, - "errors": [ - { - "action": "PromoteToCritical", - "error_kind": "unknown_action", - "from_state": "Backlog", - "message": "unknown action 'PromoteToCritical' in state 'Backlog'", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_index": 0 - } - ], - "final_state": "Backlog", - "outcome": "partial", - "reasoning_chain": "turn 1: RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical", - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9" - }, - { - "action_results": [ - { - "action": "Assign", - "error": null, - "error_kind": null, - "from_state": "Backlog", - "params": { - "AgentId": "agent-success2-1", - "Reason": "fresh2-success" - }, - "success": true, - "to_state": "Backlog", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - }, - { - "action": "Reassign", - "error": null, - "error_kind": "invalid_transition", - "from_state": "Backlog", - "params": { - "NewAssigneeId": "agent-success2-2", - "Reason": "fresh2-success" - }, - "success": false, - "to_state": "Backlog", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - } - ], - "action_sequence": [ - "Assign", - "Reassign" - ], - "actions_attempted": 2, - "actions_succeeded": 1, - "actions_total": 2, - "agent_goal": "success", - "breakdown": { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - }, - "breakdown_point": { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - }, - "errors": [ - { - "action": "Reassign", - "error_kind": "invalid_transition", - "from_state": "Backlog", - "message": "spec evaluation failed", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_index": 0 - } - ], - "final_state": "Backlog", - "outcome": "partial", - "reasoning_chain": "turn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0", - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9" - }, - { - "action_results": [], - "action_sequence": [], - "actions_attempted": 0, - "actions_succeeded": 0, - "actions_total": 0, - "agent_goal": "success", - "breakdown": null, - "breakdown_point": null, - "errors": [], - "final_state": "Backlog", - "outcome": "empty", - "reasoning_chain": "turn 1: {\"module_name\":\"gepa-replay\",\"sha256_hash\":\"b9ee1c39570c57f5e652063595787082b0cc7a3a2ddefd74fda6977a05900467\",\"size_bytes\":275659}", - "trajectory_id": "019d0874-8459-7352-b4d4-e1cfc83f456b" - }, - { - "action_results": [], - "action_sequence": [], - "actions_attempted": 0, - "actions_succeeded": 0, - "actions_total": 0, - "agent_goal": "success", - "breakdown": null, - "breakdown_point": null, - "errors": [], - "final_state": "Backlog", - "outcome": "empty", - "reasoning_chain": "turn 1: RuntimeError: temper.upload_wasm missing required argument `wasm_path` at position 2", - "trajectory_id": "019d0874-451a-7e12-b13d-9fd40c41f1e2" - }, - { - "action_results": [], - "action_sequence": [], - "actions_attempted": 0, - "actions_succeeded": 0, - "actions_total": 0, - "agent_goal": "success", - "breakdown": null, - "breakdown_point": null, - "errors": [], - "final_state": "Backlog", - "outcome": "empty", - "reasoning_chain": "turn 1: {\"tenant\":\"gepa-live-fresh-20260319\",\"project-management\":{\"app\":\"project-management\",\"tenant\":\"gepa-live-fresh-20260319\",\"added\":[\"Comment\",\"Cycle\",\"Issue\",\"Label\",\"Project\"],\"updated\":[],\"skipped\":[],\"status\":\"installed\"},\"evolution\":{\"app\":\"evolution\",\"tenant\":\"gepa-live-fresh-20260319\",\"added\":[\"EvolutionRun\",\"Sent", - "trajectory_id": "019d0872-e05d-7953-87c6-99fcf0b68da0" - } - ], - "workflows_attempted": 5, - "workflows_completed": 1, - "workflows_empty": 3, - "workflows_failed": 1, - "workflows_partial": 3, - "workflows_total": 8 -}``` - -## Appendix G: Full Reflective Dataset (`gepa-reflective`) - -```json -{ - "entity_type": "Issue", - "failure_count": 4, - "patterns": { - "common_failure_points": [ - { - "action": "Reassign", - "from_state": "Backlog", - "occurrences": 3 - }, - { - "action": "PromoteToCritical", - "from_state": "Backlog", - "occurrences": 1 - } - ], - "guard_friction": [], - "missing_capabilities": [ - "PromoteToCritical" - ], - "successful_patterns": [ - { - "actions": [ - "Assign" - ], - "trajectory_id": "019d087a-6c0d-7e40-a0b1-a5aefd7b87bb" - } - ] - }, - "skill_name": "project-management", - "success_count": 1, - "triplets": [ - { - "actions_succeeded": 0, - "actions_total": 1, - "entity_type": "Issue", - "feedback": "FIX: Update action 'Reassign' to allow transition from 'Backlog' (add 'Backlog' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d087a-349d-7071-b3b0-301fc9464305' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: RuntimeError: HTTP 409 Conflict: Action 'Reassign' not valid from state 'Backlog'", - "outcome": "failed", - "output": "Outcome=failed, actions_succeeded=0/1, final_state=Backlog. First failure: action='Reassign' from_state='Backlog' error_kind='invalid_transition' message='spec evaluation failed'.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_id": 2 - }, - { - "actions_succeeded": 0, - "actions_total": 0, - "entity_type": "Issue", - "feedback": "FIX: Update action 'unknown' to allow transition from 'unknown' (add 'unknown' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0874-8459-7352-b4d4-e1cfc83f456b' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"module_name\":\"gepa-replay\",\"sha256_hash\":\"b9ee1c39570c57f5e652063595787082b0cc7a3a2ddefd74fda6977a05900467\",\"size_bytes\":275659}", - "outcome": "empty", - "output": "Outcome=empty, actions_succeeded=0/0, final_state=Backlog.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d0874-8459-7352-b4d4-e1cfc83f456b", - "turn_id": 5 - }, - { - "actions_succeeded": 0, - "actions_total": 0, - "entity_type": "Issue", - "feedback": "FIX: Update action 'unknown' to allow transition from 'unknown' (add 'unknown' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0874-451a-7e12-b13d-9fd40c41f1e2' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: RuntimeError: temper.upload_wasm missing required argument `wasm_path` at position 2", - "outcome": "empty", - "output": "Outcome=empty, actions_succeeded=0/0, final_state=Backlog.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d0874-451a-7e12-b13d-9fd40c41f1e2", - "turn_id": 6 - }, - { - "actions_succeeded": 0, - "actions_total": 0, - "entity_type": "Issue", - "feedback": "FIX: Update action 'unknown' to allow transition from 'unknown' (add 'unknown' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0872-e05d-7953-87c6-99fcf0b68da0' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"tenant\":\"gepa-live-fresh-20260319\",\"project-management\":{\"app\":\"project-management\",\"tenant\":\"gepa-live-fresh-20260319\",\"added\":[\"Comment\",\"Cycle\",\"Issue\",\"Label\",\"Project\"],\"updated\":[],\"skipped\":[],\"status\":\"installed\"},\"evolution\":{\"app\":\"evolution\",\"tenant\":\"gepa-live-fresh-20260319\",\"added\":[\"EvolutionRun\",\"Sent", - "outcome": "empty", - "output": "Outcome=empty, actions_succeeded=0/0, final_state=Backlog.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d0872-e05d-7953-87c6-99fcf0b68da0", - "turn_id": 7 - }, - { - "actions_succeeded": 1, - "actions_total": 2, - "entity_type": "Issue", - "feedback": "FIX: Update action 'Reassign' to allow transition from 'Backlog' (add 'Backlog' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d087a-6c16-74b2-9094-5768718f8d71' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0\nturn 2: {\"trajectory_id\":\"019d087a-6c0d-7e40-a0b1-a5aefd7b87bb\",\"status\":\"flushed\"}\nturn 3: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"reassign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19", - "outcome": "partial", - "output": "Outcome=partial, actions_succeeded=1/2, final_state=Backlog. First failure: action='Reassign' from_state='Backlog' error_kind='invalid_transition' message='spec evaluation failed'.", - "preserve": false, - "score": 0.5, - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_id": 1 - }, - { - "actions_succeeded": 1, - "actions_total": 2, - "entity_type": "Issue", - "feedback": "FIX: Add [[action]] section 'PromoteToCritical' to the Issue spec with 'from' including 'Backlog' and a valid 'to' state.", - "input": "Trajectory '019d087a-34a2-7cf1-a894-b4e50c0b0fd9' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical", - "outcome": "partial", - "output": "Outcome=partial, actions_succeeded=1/2, final_state=Backlog. First failure: action='PromoteToCritical' from_state='Backlog' error_kind='unknown_action' message='unknown action 'PromoteToCritical' in state 'Backlog''.", - "preserve": false, - "score": 0.5, - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_id": 3 - }, - { - "actions_succeeded": 1, - "actions_total": 2, - "entity_type": "Issue", - "feedback": "FIX: Update action 'Reassign' to allow transition from 'Backlog' (add 'Backlog' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0879-90af-7922-a5ea-b08864af0ca9' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0", - "outcome": "partial", - "output": "Outcome=partial, actions_succeeded=1/2, final_state=Backlog. First failure: action='Reassign' from_state='Backlog' error_kind='invalid_transition' message='spec evaluation failed'.", - "preserve": false, - "score": 0.5, - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_id": 4 - }, - { - "actions_succeeded": 1, - "actions_total": 1, - "entity_type": "Issue", - "feedback": "PRESERVE: This workflow completed successfully (1 actions). Preserve this behavior and do not regress it.", - "input": "Trajectory '019d087a-6c0d-7e40-a0b1-a5aefd7b87bb' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0", - "outcome": "completed", - "output": "Outcome=completed, actions_succeeded=1/1, final_state=Backlog.", - "preserve": true, - "score": 1.0, - "trajectory_id": "019d087a-6c0d-7e40-a0b1-a5aefd7b87bb", - "turn_id": 0 - } - ], - "verification_feedback": [], - "workflow_completion_rate": 0.2, - "workflow_counts": { - "completed": 1, - "failed": 1, - "partial": 3 - }, - "workflow_triplets": [ - { - "actions_succeeded": 0, - "actions_total": 1, - "entity_type": "Issue", - "feedback": "FIX: Update action 'Reassign' to allow transition from 'Backlog' (add 'Backlog' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d087a-349d-7071-b3b0-301fc9464305' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: RuntimeError: HTTP 409 Conflict: Action 'Reassign' not valid from state 'Backlog'", - "outcome": "failed", - "output": "Outcome=failed, actions_succeeded=0/1, final_state=Backlog. First failure: action='Reassign' from_state='Backlog' error_kind='invalid_transition' message='spec evaluation failed'.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d087a-349d-7071-b3b0-301fc9464305", - "turn_id": 2 - }, - { - "actions_succeeded": 0, - "actions_total": 0, - "entity_type": "Issue", - "feedback": "FIX: Update action 'unknown' to allow transition from 'unknown' (add 'unknown' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0874-8459-7352-b4d4-e1cfc83f456b' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"module_name\":\"gepa-replay\",\"sha256_hash\":\"b9ee1c39570c57f5e652063595787082b0cc7a3a2ddefd74fda6977a05900467\",\"size_bytes\":275659}", - "outcome": "empty", - "output": "Outcome=empty, actions_succeeded=0/0, final_state=Backlog.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d0874-8459-7352-b4d4-e1cfc83f456b", - "turn_id": 5 - }, - { - "actions_succeeded": 0, - "actions_total": 0, - "entity_type": "Issue", - "feedback": "FIX: Update action 'unknown' to allow transition from 'unknown' (add 'unknown' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0874-451a-7e12-b13d-9fd40c41f1e2' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: RuntimeError: temper.upload_wasm missing required argument `wasm_path` at position 2", - "outcome": "empty", - "output": "Outcome=empty, actions_succeeded=0/0, final_state=Backlog.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d0874-451a-7e12-b13d-9fd40c41f1e2", - "turn_id": 6 - }, - { - "actions_succeeded": 0, - "actions_total": 0, - "entity_type": "Issue", - "feedback": "FIX: Update action 'unknown' to allow transition from 'unknown' (add 'unknown' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0872-e05d-7953-87c6-99fcf0b68da0' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"tenant\":\"gepa-live-fresh-20260319\",\"project-management\":{\"app\":\"project-management\",\"tenant\":\"gepa-live-fresh-20260319\",\"added\":[\"Comment\",\"Cycle\",\"Issue\",\"Label\",\"Project\"],\"updated\":[],\"skipped\":[],\"status\":\"installed\"},\"evolution\":{\"app\":\"evolution\",\"tenant\":\"gepa-live-fresh-20260319\",\"added\":[\"EvolutionRun\",\"Sent", - "outcome": "empty", - "output": "Outcome=empty, actions_succeeded=0/0, final_state=Backlog.", - "preserve": false, - "score": 0.0, - "trajectory_id": "019d0872-e05d-7953-87c6-99fcf0b68da0", - "turn_id": 7 - }, - { - "actions_succeeded": 1, - "actions_total": 2, - "entity_type": "Issue", - "feedback": "FIX: Update action 'Reassign' to allow transition from 'Backlog' (add 'Backlog' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d087a-6c16-74b2-9094-5768718f8d71' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0\nturn 2: {\"trajectory_id\":\"019d087a-6c0d-7e40-a0b1-a5aefd7b87bb\",\"status\":\"flushed\"}\nturn 3: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"reassign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19", - "outcome": "partial", - "output": "Outcome=partial, actions_succeeded=1/2, final_state=Backlog. First failure: action='Reassign' from_state='Backlog' error_kind='invalid_transition' message='spec evaluation failed'.", - "preserve": false, - "score": 0.5, - "trajectory_id": "019d087a-6c16-74b2-9094-5768718f8d71", - "turn_id": 1 - }, - { - "actions_succeeded": 1, - "actions_total": 2, - "entity_type": "Issue", - "feedback": "FIX: Add [[action]] section 'PromoteToCritical' to the Issue spec with 'from' including 'Backlog' and a valid 'to' state.", - "input": "Trajectory '019d087a-34a2-7cf1-a894-b4e50c0b0fd9' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical", - "outcome": "partial", - "output": "Outcome=partial, actions_succeeded=1/2, final_state=Backlog. First failure: action='PromoteToCritical' from_state='Backlog' error_kind='unknown_action' message='unknown action 'PromoteToCritical' in state 'Backlog''.", - "preserve": false, - "score": 0.5, - "trajectory_id": "019d087a-34a2-7cf1-a894-b4e50c0b0fd9", - "turn_id": 3 - }, - { - "actions_succeeded": 1, - "actions_total": 2, - "entity_type": "Issue", - "feedback": "FIX: Update action 'Reassign' to allow transition from 'Backlog' (add 'Backlog' to the action's 'from' states or correct transition topology).", - "input": "Trajectory '019d0879-90af-7922-a5ea-b08864af0ca9' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0", - "outcome": "partial", - "output": "Outcome=partial, actions_succeeded=1/2, final_state=Backlog. First failure: action='Reassign' from_state='Backlog' error_kind='invalid_transition' message='spec evaluation failed'.", - "preserve": false, - "score": 0.5, - "trajectory_id": "019d0879-90af-7922-a5ea-b08864af0ca9", - "turn_id": 4 - }, - { - "actions_succeeded": 1, - "actions_total": 1, - "entity_type": "Issue", - "feedback": "PRESERVE: This workflow completed successfully (1 actions). Preserve this behavior and do not regress it.", - "input": "Trajectory '019d087a-6c0d-7e40-a0b1-a5aefd7b87bb' goal='success' for entity 'Issue'.\nReasoning chain:\nturn 1: {\"issue_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"assign\":{\"entity_type\":\"Issue\",\"entity_id\":\"019d0879-909a-73b3-a811-9b0cbfb0b89b\",\"status\":\"Backlog\",\"item_count\":0,\"counters\":{},\"booleans\":{\"assignee_set\":true},\"lists\":{},\"fields\":{\"Id\":\"issue-fresh2-success-1\",\"Title\":\"fresh2 ots success\",\"CreatedAt\":\"2026-03-19T0", - "outcome": "completed", - "output": "Outcome=completed, actions_succeeded=1/1, final_state=Backlog.", - "preserve": true, - "score": 1.0, - "trajectory_id": "019d087a-6c0d-7e40-a0b1-a5aefd7b87bb", - "turn_id": 0 - } - ] -}``` - -## Appendix H: Entity/Authz/Platform Trajectory Counts - -### `gepa-live-fresh-20260319` source counts -```json -[{"source":"Entity","n":29,"ok":25,"fail":4}] -``` - -### `gepa-live-fresh-20260319` totals -```json -[{"total":29,"authz_denied":0}] -``` - -### Cross-tenant authz/platform counts -```json -[{"tenant":"gepa-live-ots-temperagent-20260319","source":"Authz","n":34,"failures":34,"authz_denied":34}, -{"tenant":"gepa-live-ots-temperagent-20260319","source":"Platform","n":18,"failures":16,"authz_denied":0}, -{"tenant":"rita-agents","source":"Platform","n":6,"failures":2,"authz_denied":0}, -{"tenant":"gepa-codex-liveproof-20260319","source":"Platform","n":4,"failures":4,"authz_denied":0}, -{"tenant":"rita-agents","source":"Authz","n":4,"failures":4,"authz_denied":4}, -{"tenant":"gepa-e2e-proof","source":"Platform","n":3,"failures":1,"authz_denied":0}, -{"tenant":"gepa-e2e-proof","source":"Authz","n":2,"failures":2,"authz_denied":2}, -{"tenant":"gepa-live-portfolio-20260319","source":"Platform","n":2,"failures":2,"authz_denied":0}] -``` - -### Unmet-intent row count snapshot -```json -[{"intents_rows":0}] -``` - -## Appendix I: Run Outcome Snapshot - -```json -{ - "final_status": "Failed", - "status_timeline": [ - { - "at": "2026-03-19T23:44:35.280334+00:00", - "status": "Evaluating" - }, - { - "at": "2026-03-19T23:44:35.810859+00:00", - "status": "Proposing" - }, - { - "at": "2026-03-19T23:44:36.340279+00:00", - "status": "Failed" - } - ], - "has_replay": true, - "has_dataset": true, - "has_mutation": false, - "has_scores": false, - "has_frontier": false, - "errors": [] -} -``` - -## Appendix J: Relationship to previous proof docs -- This file supersedes ad-hoc notes and includes both: - - end-to-end GEPA run proof artifacts, and - - taxonomy/triggering clarifications requested in chat. -- Existing `docs/gepa-real-claude-live-proof-2026-03-19.md` is retained as a historical run log. diff --git a/.proof/gepa-real-claude-live-proof-2026-03-19.md b/.proof/gepa-real-claude-live-proof-2026-03-19.md deleted file mode 100644 index 294baff61..000000000 --- a/.proof/gepa-real-claude-live-proof-2026-03-19.md +++ /dev/null @@ -1,199 +0,0 @@ -# GEPA Live Proof (OTS Portfolio + Workflow Metrics) — 2026-03-19 - -> Superseded by [`docs/GEPA_E2E_PROOF.md`](./GEPA_E2E_PROOF.md), which contains: -> - the latest fresh-tenant end-to-end run (`evo-live-fresh-20260319-v4`) -> - full OTS/entity/authz taxonomy and trigger semantics -> - full raw artifacts (OTS/replay/reflective) and explicit blockers - -## Scope -- Worktree: `/Users/seshendranalla/Development/temper-gepa-tarjan` -- Server: `temper serve --port 4455 --storage turso --no-observe` -- Tenant: `gepa-live-portfolio-20260319` -- Proof date: March 19, 2026 -- Primary run: `EvolutionRun('evo-live-ots-portfolio-20260319-v3')` - -## What Was Proven -1. Real OTS trajectories were produced automatically by real `temper mcp` sessions (not fabricated JSON). -2. `SelectCandidate` omitted both `TrajectoryActions` and `Trajectories`; `gepa-replay` auto-loaded OTS trajectories from tenant storage. -3. `gepa-replay` produced workflow-level metrics (`workflows[]`, `workflow_completion_rate`, `partial_adjusted_rate`) plus action-level metrics. -4. `gepa-reflective` produced workflow-level triplets with: - - `score` (`1.0` completed, `0.5` partial, `0.0` failed) - - `preserve=true` on successful workflows - - `patterns.missing_capabilities`, `patterns.common_failure_points`, `patterns.successful_patterns` -5. `flush_trajectory()` works live through MCP (`{"status":"flushed","trajectory_id":"..."}`) and uploads mid-session OTS snapshots. - -## What Was Not Fully Proven End-to-End -- Full terminal success of the proposer/deploy leg in this run was blocked by invalid Anthropic credentials: - - `Anthropic API returned 401 ... invalid x-api-key` -- Result: run reached `Proposing` with correct replay/dataset artifacts, then failed before `RecordMutation/RecordScore/RecordFrontier/Deploy`. - -## Exact OTS Production Path - -### MCP sessions used to generate trajectory portfolio -- `success` workflow: `Assign -> Reassign` (real entity) -- `partial` workflow: `Assign -> PromoteToCritical` (`PromoteToCritical` unknown) -- `failed` workflow: `Reassign` from backlog (invalid transition) -- `flush` proof session: `Assign`, then `await temper.flush_trajectory()` mid-session, then another execute call - -These were real `temper mcp` `tools/call -> execute` invocations. Temper auto-uploaded OTS trajectories at session end, and uploaded a snapshot on flush. - -### Full OTS example (real row) -`row_trajectory_id = 019d082e-74dc-7d30-8122-1bd451a6a352` - -```json -{ - "ots_trajectory_id": "019d082e-74db-7d43-b5b4-6b7dcbb3eaa6", - "metadata": { - "task_description": "mcp-session", - "agent_id": "unknown", - "outcome": "success" - }, - "turns": [ - { - "messages": [ - {"role": "user", "content": {"type": "text", "text": "...temper.action(...Assign...) ... temper.action(...PromoteToCritical...)"}}, - {"role": "assistant", "content": {"type": "text", "text": "RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical"}} - ], - "decisions": [ - { - "choice": { - "action": "execute: ...", - "arguments": { - "trajectory_actions": [ - {"action": "Assign", "params": {"AgentId": "agent-partial-a", "Reason": "ots-partial-1"}}, - {"action": "PromoteToCritical", "params": {"Reason": "ots-partial-1"}} - ] - } - }, - "consequence": {"success": false, "error_type": "RuntimeError: HTTP 409 Conflict: Unknown action: PromoteToCritical"} - } - ] - } - ] -} -``` - -## How Decisions, Actions, and Reasons Are Extracted -1. `temper-mcp` captures each `execute` turn into OTS. -2. For replay, `gepa-replay` reads each trajectory turn and prefers `decision.choice.arguments.trajectory_actions`. -3. For reflective reasoning context, `gepa-reflective` reads decision reasoning + assistant messages (`reasoning_chain`). -4. If `trajectory_actions` are absent, replay falls back to parsing user code for `temper.action(...)` calls. - -## Workflow-Level Replay Output (v3) -From `ReplayResultJson` in `EvolutionRun('evo-live-ots-portfolio-20260319-v3')`: - -```json -{ - "workflows_total": 5, - "workflows_completed": 2, - "workflows_partial": 2, - "workflows_failed": 1, - "workflow_completion_rate": 0.4, - "partial_adjusted_rate": 0.6, - "actions_attempted": 7, - "succeeded": 4, - "success_rate": 0.5714285714285714, - "coverage": 0.8571428571428572 -} -``` - -Per-workflow outcomes included both preserved successes and failure/partial paths: -- completed: `Assign` -- partial: `Assign -> PromoteToCritical` -- failed: `Reassign` from `Backlog` - -## Workflow-Level Reflective Output (v3) -From `DatasetJson`: - -```json -{ - "workflow_triplet_count": 5, - "success_count": 2, - "failure_count": 3, - "workflow_completion_rate": 0.4, - "workflow_counts": {"completed": 2, "partial": 2, "failed": 1}, - "patterns": { - "common_failure_points": [ - {"action": "Reassign", "from_state": "Backlog", "occurrences": 2}, - {"action": "PromoteToCritical", "from_state": "Backlog", "occurrences": 1} - ], - "missing_capabilities": ["PromoteToCritical"], - "successful_patterns": [ - {"trajectory_id": "019d082f-b5df-7381-ad61-d59327351a0d", "actions": ["Assign"]} - ] - } -} -``` - -Triplets now include `preserve=true` for completed workflows and targeted mutation feedback for failed/partial workflows. - -## Before/After Evidence (Flat vs Workflow-Layered) - -### Before (older module output, flat/action-centric) -```json -{ - "actions_attempted": 7, - "succeeded": 0, - "success_rate": 0.0, - "has_workflows": false, - "has_workflow_completion_rate": false -} -``` - -### After (current implementation) -```json -{ - "workflows_total": 5, - "workflows_completed": 2, - "workflows_partial": 2, - "workflows_failed": 1, - "workflow_completion_rate": 0.4, - "partial_adjusted_rate": 0.6, - "actions_attempted": 7, - "succeeded": 4, - "success_rate": 0.5714285714285714 -} -``` - -## Live Blockers and Limits (Explicit) -1. Proposer failure root cause in this proof run: invalid Anthropic keys provided (`401 invalid x-api-key`). -2. Because proposer failed, this specific run did not reach scoring/frontier/deploy. -3. Replay/reflective/scoring modules are functioning and producing workflow-level outputs before proposer step. - -## Architecture Diagram (What Was Proven) -```text -Real MCP sessions (execute) -> OTS persisted in ots_trajectories - -> (optional) temper.flush_trajectory() snapshot upload - -EvolutionRun.Start - -> SelectCandidate (no TrajectoryActions/Trajectories) - -> gepa-replay auto-loads OTS portfolio from tenant - -> RecordEvaluation (workflow metrics + action metrics) - -> gepa-reflective builds workflow triplets + patterns - -> RecordDataset - -> gepa-proposer-agent (TemperAgent + Anthropic) - -> BLOCKED in this run by invalid x-api-key (401) -``` - -## Code Fixes Verified in This Proof Iteration -- `gepa-replay` now infers initial state from candidate IOA (`initial = "..."`) instead of hardcoded fallback. -- `gepa-replay` ignores `execute:` pseudo-actions when no `trajectory_actions` are present. -- `gepa-replay` emits `actions_attempted` and `breakdown_point` at workflow level (in addition to existing fields). -- Added replay unit tests for: - - initial-state inference - - execute pseudo-action filtering - - embedded trajectory action extraction - -## Artifacts -- `/tmp/mcp_traj_success_in.jsonl`, `/tmp/mcp_traj_success_out.jsonl` -- `/tmp/mcp_traj_partial_in.jsonl`, `/tmp/mcp_traj_partial_out.jsonl` -- `/tmp/mcp_traj_failed_in.jsonl`, `/tmp/mcp_traj_failed_out.jsonl` -- `/tmp/mcp_traj_flush_in.jsonl`, `/tmp/mcp_traj_flush_out.jsonl` -- `/tmp/ots_portfolio_list.json`, `/tmp/ots_portfolio_rows.json`, `/tmp/ots_partial_full.json` -- `/tmp/evo_portfolio_v3_final.json` -- `/tmp/evo_portfolio_v3_replay.json` -- `/tmp/evo_portfolio_v3_dataset.json` - -## Bottom Line -- Working now: OTS capture, OTS auto-injection, workflow-level replay, workflow-level reflective dataset, preserve/failure pattern extraction, flush snapshot upload. -- Not fully completed in this run: proposer mutation/deploy, blocked solely by invalid external Anthropic credentials. diff --git a/.proof/golden-soaring-cerf.md b/.proof/golden-soaring-cerf.md deleted file mode 100644 index 5a7600764..000000000 --- a/.proof/golden-soaring-cerf.md +++ /dev/null @@ -1,321 +0,0 @@ -# Golden Soaring Cerf Proof Report - -## Scope - -Implemented the plan from `~/.claude/plans/golden-soaring-cerf.md` in the dedicated worktree: - -- Worktree: `/Users/seshendranalla/Development/temper/.claude/worktrees/golden-soaring-cerf` -- Branch: `worktree-golden-soaring-cerf` -- Required base branch: `feat/ticklish-weaving-tarjan` -- Verified merge-base: `64fe5b54353092349e66ebf18b8413ac32e369f0` - -## Deliverables Implemented - -### ADR - -- `docs/adrs/0035-intent-discovery-evolution-loop.md` - -### New OS App - -- `os-apps/intent-discovery/specs/intent_discovery.ioa.toml` -- `os-apps/intent-discovery/csdl/intent_discovery.csdl.xml` -- `os-apps/intent-discovery/policies/intent_discovery.cedar` -- `os-apps/intent-discovery/skill.md` -- `os-apps/intent-discovery/wasm/gather_signals/src/lib.rs` -- `os-apps/intent-discovery/wasm/spawn_analyst/src/lib.rs` -- `os-apps/intent-discovery/wasm/create_proposals/src/lib.rs` -- `os-apps/intent-discovery/wasm/build.sh` - -### Agent / Observability Changes - -- `os-apps/temper-agent/prompts/evolution_analyst.md` -- `os-apps/temper-agent/specs/temper_agent.ioa.toml` -- `os-apps/temper-agent/wasm/llm_caller/src/lib.rs` -- `os-apps/temper-agent/wasm/tool_runner/src/lib.rs` -- `crates/temper-observe/src/otel.rs` - -### Platform / Server Changes - -- `crates/temper-server/src/api/mod.rs` -- `crates/temper-server/src/observe/evolution.rs` -- `crates/temper-server/src/observe/evolution/operations.rs` -- `crates/temper-server/src/observe/entities.rs` -- `crates/temper-server/src/observe/mod.rs` -- `crates/temper-server/src/observe/mod_test.rs` -- `crates/temper-server/src/state/policy_suggestions.rs` -- `crates/temper-store-turso/src/schema.rs` -- `crates/temper-store-turso/src/store/policy.rs` -- `crates/temper-platform/src/os_apps/mod.rs` -- `crates/temper-platform/src/os_apps/tests.rs` -- `os-apps/project-management/policies/issue.cedar` - -## Final Architecture - -### IntentDiscovery workflow - -`IntentDiscovery` is the spec-governed orchestrator: - -- `Trigger -> Gathering` via `gather_signals` -- `Gathering -> Analyzing` via `spawn_analyst` -- `Analyzing -> Proposing` via `create_proposals` -- `Proposing -> Complete` - -### Real analyst execution - -The analyst path now supports both: - -- deterministic local `mock` runs -- real Anthropic-backed runs - -For the real proof run, `IntentDiscovery` configured `TemperAgent` with: - -- `provider = anthropic` -- `model = claude-sonnet-4-20250514` -- `tools_enabled = logfire_query` - -### Logfire design - -Logfire was implemented as a WASM-backed agent tool, not a Rust-only orchestration adapter. - -The live flow was: - -1. local Temper server exported telemetry to Logfire via `LOGFIRE_TOKEN` -2. `TemperAgent` invoked `logfire_query` through `tool_runner` -3. the agent fed Logfire evidence back into the next LLM turn -4. final analysis was materialized into records and PM issues - -### Orchestration fix - -The intent-shaped real-agent run exposed two orchestration defects: - -Fix applied: - -- added `GET /observe/entities/{entity_type}/{entity_id}/wait` -- changed `spawn_analyst` to use that bounded server-side wait endpoint instead of hot polling from WASM -- added `timeout_secs = "420"` to the `spawn_analyst` integration so the orchestrator can wait for a real multi-turn agent run instead of failing at the default 30 second WASM budget - -### Intent-shaped changes completed - -The five changes requested after the first shallow run are now implemented: - -1. Redefined upstream evidence around `intent_evidence`, not just grouped errors. -2. Fed richer signals into `gather_signals`, including intent candidates, workaround patterns, abandonment patterns, plans, comments, and projects. -3. Split analyst output into `symptom_title`, `intent_title`, `recommended_issue_title`, and `problem_statement`. -4. Materialized PM issues from intent-shaped titles instead of raw operational symptoms. -5. Used Logfire as a real agent tool for evidence deepening, not just passive export/validation. - -## Commands Executed - -### WASM builds - -```bash -bash os-apps/intent-discovery/wasm/build.sh -bash os-apps/temper-agent/wasm/build.sh -``` - -### Rust verification - -```bash -cargo fmt --all -cargo check -p temper-server -p temper-cli -p temper-observe -p temper-platform -cargo test -p temper-store-turso -cargo test -p temper-platform -cargo test -p temper-server -``` - -### Real local proof server - -```bash -TURSO_URL='file:/.../.tmp/intent-discovery-proof-intent-shaped-20260323-r5/intent-proof.db' \ -TEMPER_VAULT_KEY='...' \ -LOGFIRE_TOKEN='...' \ -LOGFIRE_ENVIRONMENT='local' \ -cargo run -p temper-cli -- serve \ - --port 3463 \ - --storage turso \ - --no-observe \ - --app project-management \ - --app temper-agent \ - --app intent-discovery -``` - -### Real end-to-end proof harness - -```bash -ANTHROPIC_TOKEN='...' \ -LOGFIRE_READ_TOKEN='...' \ -BASE='http://127.0.0.1:3463' \ -LOGFIRE_QUERY_BASE='https://logfire-us.pydantic.dev' \ -bash .tmp/intent-discovery-proof-intent-shaped-20260323-r5/run_proof.sh -``` - -## End-to-End Proof Result - -Proof summary from `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/proof_summary.json`: - -```json -{ - "discovery_id": "intent-discovery-019d1cad-bbe7-7e01-9efe-b314ab29697d", - "analyze_response_status": "Analyzing", - "entity_status": "Complete", - "analyst_agent_id": "intent-analyst-intent-discovery-019d1cad-bbe7-7e01-9efe-b314ab29697d", - "issues_created": 2, - "records_created": 5, - "issues_before": 1, - "issues_after": 3, - "evolution_record_total": 5, - "finding_count": 2, - "intent_titles_present": 2, - "enable_titles": 1 -} -``` - -## Verified Real-Agent Evidence - -### Anthropic was actually called - -From the live server log for the `r5` proof run: - -- `llm_caller: calling Anthropic API, model=claude-sonnet-4-20250514, oauth=true, messages=1` -- `llm_caller: calling Anthropic API, model=claude-sonnet-4-20250514, oauth=true, messages=3` - -### The agent actually used Logfire - -From the live server log for the `r5` proof run: - -- `tool_runner: executing tool 'logfire_query'` -- `tool_runner: querying Logfire, query_kind=alternate_success_paths` -- `tool_runner: querying Logfire, query_kind=intent_failure_cluster` -- `HandleToolResults` -- follow-up Anthropic turn after the tool results -- `RecordResult -> Completed` - -### Local server actually posted to Logfire - -From `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/logfire_probe.json`: - -- recent `temper-platform` records were queryable from Logfire before analysis started - -That proves both sides of the observability loop: - -- local Temper wrote telemetry to Logfire -- the real analyst agent read Logfire back through `logfire_query` - -## Verified Analysis Output - -From `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/analysis.json`: - -- finding 1: - - `symptom_title`: `GenerateInvoice hits EntitySetNotFound on Invoice` - - `intent_title`: `Enable invoice generation workflow` - - `recommended_issue_title`: `Enable invoice generation workflow` -- finding 2: - - `symptom_title`: `MoveToTodo denied with no matching permit policy` - - `intent_title`: `Allow worker agents to transition issues to todo` - - `recommended_issue_title`: `Allow worker agents to transition issues to todo` - -The returned summary was: - -- the billing workflow had an unmet intent surfaced through workaround evidence, not just raw `EntitySetNotFound` -- the issue workflow had a governance gap surfaced as a blocked workflow outcome, not just a denial string - -## Verified Materialization Output - -From `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/materialization_report.json`: - -- `issues_created_count = 2` -- `records_created_count = 5` - -Created issues: - -- `Enable invoice generation workflow` -- `Allow worker agents to transition issues to todo` - -Created evolution records: - -- `5 total` in the successful `r5` run - -Issue state after materialization, from `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/issues_after.json`: - -- seed issue remained `Backlog` -- both new issues advanced to `Todo` - -This is the key regression fix relative to the earlier real run: the created PM issues are now intent-shaped rather than error-shaped. - -## Verified Intent Evidence - -From `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/intent_evidence_before.json`: - -- candidate 1: `Send An Invoice To The Customer` - - had `workaround_count = 2` - - had `abandonment_count = 2` - - showed failed `GenerateInvoice` followed by successful `CreateDraft` -- candidate 2: `Allow issue to reach todo` - - had `authz_denials = 3` - - had `abandonment_count = 1` - - showed repeated `MoveToTodo` denials - -That proves the run was no longer naming work directly from raw error strings. The upstream evidence already expressed unmet outcomes, workaround patterns, and abandonment patterns before the model produced findings. - -## Build / Test Results - -### WASM builds - -- `IntentDiscovery` WASM build: passed -- `TemperAgent` WASM build: passed - -### Cargo check - -- `cargo check -p temper-server -p temper-cli -p temper-observe -p temper-platform`: passed - -### Rust suites - -- `cargo test -p temper-store-turso`: 14 passed, 0 failed -- `cargo test -p temper-platform`: 213 passed, 0 failed -- `cargo test -p temper-server`: 303 passed, 0 failed - -Total verified tests after final fixes: `530 passed, 0 failed` - -## Remaining Limitations - -- The proof dataset is still synthetic. The run is real, but the seeded signals were intentionally constructed local examples rather than long-horizon production history. -- Intent inference upstream is still heuristic. It uses explicit `intent`, `session_id`, action sequences, workaround detection, abandonment detection, and authz/error clustering, but it is not yet learning latent intents from arbitrary free-form user behavior. -- Logfire is a tool the agent can query for deeper evidence; it is not yet the primary storage/query layer for all intent mining. The first pass still comes from local Temper evidence and then the agent drills into Logfire selectively. -- `sandbox_provisioner` still falls back around the missing `Workspaces` entity set. That noise is no longer dominating the findings, but the platform gap still exists. -- There is still no first-class Temper environment model beyond passing `LOGFIRE_ENVIRONMENT=local` and tagging traces with the local deployment environment. - -## Definition Of Done - -- [x] ADR written for the IntentDiscovery evolution loop -- [x] `IntentDiscovery` IOA spec, CSDL, policy, and skill added -- [x] `gather_signals`, `spawn_analyst`, and `create_proposals` WASM modules implemented -- [x] evolution analyst prompt added for `TemperAgent` -- [x] `POST /api/evolution/analyze` implemented -- [x] policy denial suggestions persisted to Turso and surfaced to analysis -- [x] project management Cedar policies widened for system-driven issue materialization -- [x] real Anthropic-backed analyst run executed locally -- [x] local server exported telemetry to Logfire -- [x] analyst agent queried Logfire through a WASM-backed tool -- [x] `IntentDiscovery` reached `Complete` in the real run -- [x] real run created PM issues and evolution records -- [x] orchestration bug fixed with bounded wait endpoint for terminal agent state -- [x] build, check, and Rust test verification completed after final fixes -- [ ] GIF / screencast recorded - -## Remaining Non-Code Follow-Up - -The plan requested a GIF / screencast for a tweet demo. That artifact was not produced in this terminal-only implementation run. - -## Evidence Files - -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/proof_summary.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/intent_discovery_entity.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/intent_discovery_history.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/analyst_agent.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/analysis.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/materialization_report.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/issues_after.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/evolution_records_after.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/intent_evidence_before.json` -- `.tmp/intent-discovery-proof-intent-shaped-20260323-r5/logfire_probe.json` -- `.tmp/intent-discovery-proof-real-20260323-r2/run_proof.sh` diff --git a/.proof/pawfs-context-read-plane-batch-read.md b/.proof/pawfs-context-read-plane-batch-read.md deleted file mode 100644 index 83a0d5a18..000000000 --- a/.proof/pawfs-context-read-plane-batch-read.md +++ /dev/null @@ -1,45 +0,0 @@ -# PawFS Context Read Plane Batch Read - -Date: 2026-04-23 - -## What changed - -- Added `load_query_projection_fields_many(...)` in `temper-store-turso` to fetch sparse projected fields for many entities in one query. -- Added `ServerState::read_file_texts_batch(...)` in `temper-server` to: - - resolve `File` metadata from the durable query plane - - fall back to actor state only for projection misses - - read blob bytes directly from the local blob store or external blob endpoint -- Added `POST /api/files/read-text-batch` in `temper-server`. - -## Verification - -### Projection loader - -Command: - -```bash -cargo test -p temper-store-turso load_query_projection_fields_many_returns_requested_fields_by_entity -- --nocapture -``` - -Result: - -- passed -- proved projected `content_hash`, `mime_type`, and `has_content` load in one round trip - -### Batch read API - -Command: - -```bash -cargo test -p temper-server --features observe batch_file_text_read_returns_projected_file_contents_in_request_order -- --nocapture -``` - -Result: - -- passed -- verified `POST /api/files/read-text-batch` returns: - - a ready file with text content - - a found-but-empty file with empty text - - a missing file with `found=false` -- preserved request order in the response - diff --git a/.proof/pawfs-context-read-plane-live-e2e.md b/.proof/pawfs-context-read-plane-live-e2e.md deleted file mode 100644 index 1e5c1e8dc..000000000 --- a/.proof/pawfs-context-read-plane-live-e2e.md +++ /dev/null @@ -1,76 +0,0 @@ -# PawFS Context Read Plane Live E2E - -Date: 2026-04-24 -Worktree: `/Users/seshendranalla/Development/temper-worktrees/pawfs-context-read-plane` -Validated through local `temperpaw-server` using this Temper worktree via a temporary local dependency patch in the paired OpenPaw worktree. - -## Goal - -Prove the Temper-side architecture works in a real server process: - -- OS-app reactions load and execute after install -- `FileVersion` lineage is explicit at runtime -- `POST /api/files/read-version-text-batch` serves immutable historical content -- the internal blob endpoint fast path works for local server verification - -## Live proof - -On the local server: - -1. Created a `File` -2. Wrote two distinct text payloads through `PUT /tdata/Files('')/$value` -3. Queried the resulting `File` and `FileVersion` entities -4. Called `POST /api/files/read-version-text-batch` with both the superseded and current version ids - -Observed runtime facts: - -- `File.fields.last_version_id` updated to the newest `FileVersion` -- newest `FileVersion.status=Current` -- previous `FileVersion.status=Superseded` -- batch immutable read returned: - - `first version from live e2e` - - `second version from live e2e` - -## Runtime bugs found and fixed during live verification - -### 1. Reactions were not actually live after app install - -Symptom: - -- specs defined the supersede reaction -- but the previous version stayed `Current` at runtime after later app installs - -Root cause: - -- OS-app bundles were not loading `reactions/reactions.toml` -- bootstrap registration dropped reaction rules -- later app installs overwrote existing tenant reactions - -Fix: - -- load reactions into `AppBundle` -- register them during bootstrap -- rebuild the live `ReactionDispatcher` after install -- merge tenant reactions instead of replacing them wholesale - -### 2. Immutable batch reads used the wrong blob path locally - -Symptom: - -- `POST /api/files/read-version-text-batch` returned `401 Unauthorized` on the local server even though the content existed - -Root cause: - -- batch version reads followed the configured blob endpoint even when it pointed at the server's own internal `/_internal/blobs` route - -Fix: - -- detect the internal local blob endpoint -- read directly from the local store instead of doing an external HTTP GET - -## What this proves - -- Temper now has a real native immutable read plane for TemperFS hot paths. -- `FileVersion` lineage is not just modeled in specs; it executes correctly in the running platform. -- app-installed reactions are active immediately after install. -- live immutable batch reads work on the same stack OpenPaw uses for Session context assembly. diff --git a/.proof/temper-agent-e2e-proof.md b/.proof/temper-agent-e2e-proof.md deleted file mode 100644 index e02d87528..000000000 --- a/.proof/temper-agent-e2e-proof.md +++ /dev/null @@ -1,929 +0,0 @@ -# Governed Agent Architecture E2E Proof - -## Date -2026-03-24T12:14:51.222495+00:00 - -## Branch -feat/temper-claw - -## Commit -f58f58926fdce2a35aa4487bffb3015900c5a8e4 - -## Server -`http://127.0.0.1:3463` against tenant `temper-agent-proof-20260324121451` - -## Specs Deployed -- `temper-fs`: {"app": "temper-fs", "tenant": "temper-agent-proof-20260324121451", "added": ["Directory", "File", "FileVersion", "Workspace"], "updated": [], "skipped": [], "status": "installed"} -- `temper-agent`: {"app": "temper-agent", "tenant": "temper-agent-proof-20260324121451", "added": ["AgentMemory", "AgentSkill", "AgentSoul", "CronJob", "CronScheduler", "HeartbeatMonitor", "TemperAgent", "ToolHook"], "updated": [], "skipped": [], "status": "installed"} -- `temper-channels`: {"app": "temper-channels", "tenant": "temper-agent-proof-20260324121451", "added": ["AgentRoute", "Channel", "ChannelSession"], "updated": [], "skipped": [], "status": "installed"} - -## Trigger Path A: Direct OData API -| Step | Expected | Actual | Status | -|---|---|---|---| -| A1 | Agent created with soul_id bound | soul_id=019d1fc4-f103-7500-9043-a09663bebb2e | PASS | -| A4 | SSE replay returns lifecycle events | captured direct-events.sse | PASS | -| A5 | Prompt includes soul, skills, and memory blocks | # Proof Soul

## Identity
You are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite.

## Instructions
- Prefer deterministic mock runs for verification.
- Surface memory and skills in the prompt.
- Use tools only when the proof plan requires them.

## Capabilities
- Run | PASS | -| A6 | Thinking/Executing loop is visible in events | ProcessToolCalls/HandleToolResults present | PASS | -| A7 | Session tree persisted JSONL entries and steering branch | {"id":"h-019d1fc4-f16f-7452-9119-79ae692dc5ae","parentId":null,"tokens":0,"type":"header","version":1}
{"content":"{\"mock_plan\":{\"steps\":[{\"text\":\"Starting direct path\",\"tool_calls\":[{\"name\":\"bash\",\"input\":{\"command\":\"sle | PASS | -| A8 | Steering injection stored and observable | steering marker present | PASS | -| A9 | Steering caused a continue transition | ContinueWithSteering seen | PASS | -| A10 | Agent completed successfully | Direct path finished with memory keys user-profile, project-context, proof-direct-memory. | PASS | -| A11 | save_memory created a new AgentMemory | count=1 | PASS | - -## Trigger Path B: Channel Webhook -| Step | Expected | Actual | Status | -|---|---|---|---| -| B1 | Channel.ReceiveMessage accepted webhook payload | ReceiveMessage executed | PASS | -| B2 | ChannelSession created for thread | session_id=019d1fc5-0717-7c22-a206-598ccf05f8b7 | PASS | -| B3 | Channel route spawned agent with route soul_id | soul_id=019d1fc4-f103-7500-9043-a09663bebb2e | PASS | -| B4 | Channel-triggered agent completed | Channel proof reply | PASS | -| B5 | send_reply delivered the agent result | {"path": "/", "body": "{\"agent_entity_id\":\"019d1fc5-06f9-7ad0-a252-bc6d34187024\",\"content\":\"Channel proof reply\",\"thread_id\":\"thread-1\"}", "agent_entity_id": "019d1fc5-06f9-7ad0-a252-bc6d34187024", "content": "Channel proof reply", "thread_id": "thread-1"} | PASS | - -## Trigger Path C: WASM Orchestration -| Step | Expected | Actual | Status | -|---|---|---|---| -| C1 | An orchestrator entity ran WASM that spawned a TemperAgent | parent_agent=019d1fc5-0a03-78d0-a9d4-c410a869dd27 | PASS | -| C2 | Child TemperAgent created with parent_agent_id | parent_agent_id=019d1fc5-0a03-78d0-a9d4-c410a869dd27 | PASS | -| C3 | Child agent completed and result was observable | Child completed after steering: STEERED-CHILD | PASS | - -## Trigger Path D: MCP Tool Call -| Step | Expected | Actual | Status | -|---|---|---|---| -| D1 | MCP created, configured, and provisioned an agent | agent_id=019d1fc5-19ab-7043-9830-8b10b06e0d44 | PASS | -| D2 | MCP-observed agent reached Completed | MCP path ok | PASS | -| D3 | MCP result matched expected output | MCP path ok | PASS | - -## Trigger Path E: Cron Job -| Step | Expected | Actual | Status | -|---|---|---|---| -| E1 | CronJob entity created | cron_id=019d1fc5-1b06-71b1-bac0-ca53d64e2d5f | PASS | -| E2 | Cron job activated | status=Active | PASS | -| E3 | Manual Trigger action executed | last_agent_id=019d1fc5-1b28-7461-b862-709e30b2b274 | PASS | -| E4 | Cron-triggered TemperAgent was created | agent_id=019d1fc5-1b28-7461-b862-709e30b2b274 | PASS | -| E5 | CronJob tracked last_agent_id | LastAgentId=019d1fc5-1b28-7461-b862-709e30b2b274 | PASS | -| E6 | Second trigger incremented run_count | RunCount=2 | PASS | - -## Subagent + Coding Agent Verification -| Step | Expected | Actual | Status | -|---|---|---|---| -| S1 | Parent agent created with spawn_agent in tools | tools_enabled includes spawn_agent | PASS | -| S2 | Parent invoked spawn_agent | child id present in parent session | PASS | -| S3 | Child links back to parent | ParentAgentId=019d1fc5-0a03-78d0-a9d4-c410a869dd27 | PASS | -| S4 | Parent steered child agent | Child completed after steering: STEERED-CHILD | PASS | -| S5 | list_agents exposed child status | child id visible in tool result | PASS | -| S6 | Parent/child flow produced child result | Child completed after steering: STEERED-CHILD | PASS | -| S7 | Parent invoked run_coding_agent | tool result captured | PASS | -| S8 | CLI command matched expected claude-code pattern | command string present | PASS | -| S9 | agent_depth guard prevented deep recursion | guard message present | PASS | - -## Heartbeat Monitoring Verification -| Step | Expected | Actual | Status | -|---|---|---|---| -| H1 | Heartbeat test agent created with short timeout | agent_id=019d1fc5-1c79-7f01-9671-5e3c358057bf | PASS | -| H2 | Mock hang plan provisioned | provider=mock, mode=hang | PASS | -| H3 | Heartbeat monitor started and scanned | monitor_id=019d1fc5-2086-7c90-8536-4c33a96a7e45 | PASS | -| H4 | Stale agent transitioned to Failed | heartbeat timeout: no heartbeat observed within 300 seconds | PASS | -| H5 | SSE replay captured TimeoutFail state change | TimeoutFail present | PASS | - -## Cross-Session Memory -| Step | Expected | Actual | Status | -|---|---|---|---| -| M1 | Second agent created with same soul_id | agent_id=019d1fc5-29ab-7193-b32d-539ce4388c08 | PASS | -| M2 | Cross-session memory loaded into prompt | memory keys=user-profile, project-context, proof-direct-memory count=3 | PASS | -| M3 | Memory-aware mock response surfaced recalled knowledge | memory keys=user-profile, project-context, proof-direct-memory count=3 | PASS | - -## Compaction -| Step | Expected | Actual | Status | -|---|---|---|---| -| X1 | Compaction entry was written into the session tree | compaction entry present | PASS | -| X2 | Agent resumed after compaction | [Previous conversation summary]
## Goal
Preserve the active task.

## Constraints & Preferences
Stay within the current workspace and existing agent context.

## Progress
- Done: Earlier conversation was compacted.
- In Progress: Continue the active task with the remaining context.
- Blocked: None.

## Key Decisions
Use the deterministic mock compaction path when no real model is configured.

## Next Steps
Resume the agent loop after compaction.

## Critical Context
## user
{"notes": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | PASS | - -## Artifacts - -### Session Tree Dump -```jsonl -{"id":"h-019d1fc4-f16f-7452-9119-79ae692dc5ae","parentId":null,"tokens":0,"type":"header","version":1} -{"content":"{\"mock_plan\":{\"steps\":[{\"text\":\"Starting direct path\",\"tool_calls\":[{\"name\":\"bash\",\"input\":{\"command\":\"sleep 2 && printf direct-path-bash\",\"workdir\":\"/Users/seshendranalla/Development/temper-pi-agent-rewrite/.tmp/temper-agent-proof/sandbox\"}}]},{\"final_text\":\"Waiting for steering check.\"},{\"text\":\"Steering applied: {{latest_user}}\",\"tool_calls\":[{\"name\":\"save_memory\",\"input\":{\"key\":\"proof-direct-memory\",\"content\":\"saved from direct path\",\"memory_type\":\"project\"}}]},{\"final_text\":\"Direct path finished with memory keys {{memory_keys}}.\"}]}}","id":"u-019d1fc4-f16f-7452-9119-79ae692dc5ae-0","parentId":"h-019d1fc4-f16f-7452-9119-79ae692dc5ae","role":"user","tokens":135,"type":"message"} -{"content":[{"text":"Starting direct path","type":"text"},{"id":"mock-tool-0-0","input":{"command":"sleep 2 && printf direct-path-bash","workdir":"/Users/seshendranalla/Development/temper-pi-agent-rewrite/.tmp/temper-agent-proof/sandbox"},"name":"bash","type":"tool_use"}],"id":"a-2","parentId":"u-019d1fc4-f16f-7452-9119-79ae692dc5ae-0","role":"assistant","tokens":257,"type":"message"} -{"content":[{"content":"direct-path-bash","is_error":false,"tool_use_id":"mock-tool-0-0","type":"tool_result"}],"id":"t-3","parentId":"a-2","role":"user","tokens":25,"type":"message"} -{"content":[{"text":"Waiting for steering check.","type":"text"}],"id":"a-4","parentId":"t-3","role":"assistant","tokens":27,"type":"message"} -{"content":"Follow the steering marker ST-123","id":"s-5","parentId":"a-4","role":"user","tokens":8,"type":"steering"} -{"content":[{"text":"Steering applied: Follow the steering marker ST-123","type":"text"},{"id":"mock-tool-2-0","input":{"content":"saved from direct path","key":"proof-direct-memory","memory_type":"project"},"name":"save_memory","type":"tool_use"}],"id":"a-6","parentId":"s-5","role":"assistant","tokens":237,"type":"message"} -{"content":[{"content":"Memory saved: key=proof-direct-memory, type=project","is_error":false,"tool_use_id":"mock-tool-2-0","type":"tool_result"}],"id":"t-7","parentId":"a-6","role":"user","tokens":33,"type":"message"} -{"content":[{"text":"Direct path finished with memory keys user-profile, project-context, proof-direct-memory.","type":"text"}],"id":"a-8","parentId":"t-7","role":"assistant","tokens":89,"type":"message"} -``` - -### SSE Events Captured -```text -event: state_change -data: {"seq":1,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Created","status":"Created","tenant":"temper-agent-proof-20260324121451"} - -event: state_change -data: {"seq":2,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Configure","status":"Created","tenant":"temper-agent-proof-20260324121451"} - -event: state_change -data: {"seq":3,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Provision","status":"Provisioning","tenant":"temper-agent-proof-20260324121451"} - -event: integration_start -data: {"seq":4,"integration":"provision_sandbox","module":"sandbox_provisioner","trigger_action":"Provision"} - -event: integration_complete -data: {"seq":5,"integration":"provision_sandbox","module":"sandbox_provisioner","trigger_action":"Provision","result":"success","callback_action":"SandboxReady","duration_ms":285} - -event: state_change -data: {"seq":6,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"SandboxReady","status":"Thinking","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":7,"integration":"call_llm","module":"llm_caller","trigger_action":"SandboxReady"} - -event: prompt_assembled -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":8,"kind":"prompt_assembled","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"system prompt assembled","timestamp":"2026-03-24T12:14:54.161741+00:00","data":{"kind":"prompt_assembled","message":"system prompt assembled","system_prompt":"# Proof Soul\n\n## Identity\nYou are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite.\n\n## Instructions\n- Prefer deterministic mock runs for verification.\n- Surface memory and skills in the prompt.\n- Use tools only when the proof plan requires them.\n\n## Capabilities\n- Run sandbox tools\n- Spawn governed child agents\n- Save and recall memories\n\n## Constraints\n- Do not use destructive commands.\n- Stay inside the provided workspace.\n\n\nOverride: include the DIRECT-OVERRIDE marker.\n\n\n \n \n\n\n\n \n The proof user prefers exact verification over discussion.\n \n \n Temper Pi rewrite proof must capture SSE, session trees, cron, heartbeat, channels, and MCP.\n \n"}} - -event: state_change -data: {"seq":9,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Thinking","tenant":"temper-agent-proof-20260324121451"} - -event: llm_request_started -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":10,"kind":"llm_request_started","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"calling provider=mock model=mock-proof","timestamp":"2026-03-24T12:14:54.174224+00:00","data":{"kind":"llm_request_started","message":"calling provider=mock model=mock-proof"}} - -event: llm_response -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":11,"kind":"llm_response","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"provider returned stop_reason=tool_use","timestamp":"2026-03-24T12:14:54.174579+00:00","data":{"kind":"llm_response","message":"provider returned stop_reason=tool_use","stop_reason":"tool_use"}} - -event: integration_complete -data: {"seq":12,"integration":"call_llm","module":"llm_caller","trigger_action":"SandboxReady","result":"success","callback_action":"ProcessToolCalls","duration_ms":70} - -event: state_change -data: {"seq":13,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"ProcessToolCalls","status":"Executing","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":14,"integration":"run_tools","module":"tool_runner","trigger_action":"ProcessToolCalls"} - -event: state_change -data: {"seq":15,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Executing","tenant":"temper-agent-proof-20260324121451"} - -event: tool_execution_start -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":16,"kind":"tool_execution_start","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":"mock-tool-0-0","tool_name":"bash","task_id":null,"message":"executing tool bash","timestamp":"2026-03-24T12:14:54.235977+00:00","data":{"kind":"tool_execution_start","message":"executing tool bash","tool_call_id":"mock-tool-0-0","tool_name":"bash"}} - -event: state_change -data: {"seq":17,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Steer","status":"Executing","tenant":"temper-agent-proof-20260324121451"} - -event: state_change -data: {"seq":18,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Executing","tenant":"temper-agent-proof-20260324121451"} - -event: tool_execution_complete -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":19,"kind":"tool_execution_complete","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":"mock-tool-0-0","tool_name":"bash","task_id":null,"message":"completed tool bash","timestamp":"2026-03-24T12:14:56.350986+00:00","data":{"is_error":false,"kind":"tool_execution_complete","message":"completed tool bash","tool_call_id":"mock-tool-0-0","tool_name":"bash"}} - -event: integration_complete -data: {"seq":20,"integration":"run_tools","module":"tool_runner","trigger_action":"ProcessToolCalls","result":"success","callback_action":"HandleToolResults","duration_ms":2203} - -event: state_change -data: {"seq":21,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"HandleToolResults","status":"Thinking","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":22,"integration":"call_llm","module":"llm_caller","trigger_action":"HandleToolResults"} - -event: prompt_assembled -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":23,"kind":"prompt_assembled","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"system prompt assembled","timestamp":"2026-03-24T12:14:56.454475+00:00","data":{"kind":"prompt_assembled","message":"system prompt assembled","system_prompt":"# Proof Soul\n\n## Identity\nYou are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite.\n\n## Instructions\n- Prefer deterministic mock runs for verification.\n- Surface memory and skills in the prompt.\n- Use tools only when the proof plan requires them.\n\n## Capabilities\n- Run sandbox tools\n- Spawn governed child agents\n- Save and recall memories\n\n## Constraints\n- Do not use destructive commands.\n- Stay inside the provided workspace.\n\n\nOverride: include the DIRECT-OVERRIDE marker.\n\n\n \n \n\n\n\n \n The proof user prefers exact verification over discussion.\n \n \n Temper Pi rewrite proof must capture SSE, session trees, cron, heartbeat, channels, and MCP.\n \n"}} - -event: state_change -data: {"seq":24,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Thinking","tenant":"temper-agent-proof-20260324121451"} - -event: llm_request_started -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":25,"kind":"llm_request_started","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"calling provider=mock model=mock-proof","timestamp":"2026-03-24T12:14:56.464181+00:00","data":{"kind":"llm_request_started","message":"calling provider=mock model=mock-proof"}} - -event: llm_response -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":26,"kind":"llm_response","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"provider returned stop_reason=end_turn","timestamp":"2026-03-24T12:14:56.464566+00:00","data":{"kind":"llm_response","message":"provider returned stop_reason=end_turn","stop_reason":"end_turn"}} - -event: integration_complete -data: {"seq":27,"integration":"call_llm","module":"llm_caller","trigger_action":"HandleToolResults","result":"success","callback_action":"CheckSteering","duration_ms":141} - -event: state_change -data: {"seq":28,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"CheckSteering","status":"Steering","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":29,"integration":"check_steering","module":"steering_checker","trigger_action":"CheckSteering"} - -event: integration_complete -data: {"seq":30,"integration":"check_steering","module":"steering_checker","trigger_action":"CheckSteering","result":"success","callback_action":"ContinueWithSteering","duration_ms":33} - -event: state_change -data: {"seq":31,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"ContinueWithSteering","status":"Thinking","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":32,"integration":"call_llm","module":"llm_caller","trigger_action":"ContinueWithSteering"} - -event: prompt_assembled -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":33,"kind":"prompt_assembled","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"system prompt assembled","timestamp":"2026-03-24T12:14:56.657477+00:00","data":{"kind":"prompt_assembled","message":"system prompt assembled","system_prompt":"# Proof Soul\n\n## Identity\nYou are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite.\n\n## Instructions\n- Prefer deterministic mock runs for verification.\n- Surface memory and skills in the prompt.\n- Use tools only when the proof plan requires them.\n\n## Capabilities\n- Run sandbox tools\n- Spawn governed child agents\n- Save and recall memories\n\n## Constraints\n- Do not use destructive commands.\n- Stay inside the provided workspace.\n\n\nOverride: include the DIRECT-OVERRIDE marker.\n\n\n \n \n\n\n\n \n The proof user prefers exact verification over discussion.\n \n \n Temper Pi rewrite proof must capture SSE, session trees, cron, heartbeat, channels, and MCP.\n \n"}} - -event: state_change -data: {"seq":34,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Thinking","tenant":"temper-agent-proof-20260324121451"} - -event: llm_request_started -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":35,"kind":"llm_request_started","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"calling provider=mock model=mock-proof","timestamp":"2026-03-24T12:14:56.668361+00:00","data":{"kind":"llm_request_started","message":"calling provider=mock model=mock-proof"}} - -event: llm_response -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":36,"kind":"llm_response","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"provider returned stop_reason=tool_use","timestamp":"2026-03-24T12:14:56.668770+00:00","data":{"kind":"llm_response","message":"provider returned stop_reason=tool_use","stop_reason":"tool_use"}} - -event: integration_complete -data: {"seq":37,"integration":"call_llm","module":"llm_caller","trigger_action":"ContinueWithSteering","result":"success","callback_action":"ProcessToolCalls","duration_ms":84} - -event: state_change -data: {"seq":38,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"ProcessToolCalls","status":"Executing","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":39,"integration":"run_tools","module":"tool_runner","trigger_action":"ProcessToolCalls"} - -event: state_change -data: {"seq":40,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Executing","tenant":"temper-agent-proof-20260324121451"} - -event: tool_execution_start -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":41,"kind":"tool_execution_start","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":"mock-tool-2-0","tool_name":"save_memory","task_id":null,"message":"executing tool save_memory","timestamp":"2026-03-24T12:14:56.745453+00:00","data":{"kind":"tool_execution_start","message":"executing tool save_memory","tool_call_id":"mock-tool-2-0","tool_name":"save_memory"}} - -event: state_change -data: {"seq":42,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Executing","tenant":"temper-agent-proof-20260324121451"} - -event: tool_execution_complete -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":43,"kind":"tool_execution_complete","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":"mock-tool-2-0","tool_name":"save_memory","task_id":null,"message":"completed tool save_memory","timestamp":"2026-03-24T12:14:56.777472+00:00","data":{"is_error":false,"kind":"tool_execution_complete","message":"completed tool save_memory","tool_call_id":"mock-tool-2-0","tool_name":"save_memory"}} - -event: integration_complete -data: {"seq":44,"integration":"run_tools","module":"tool_runner","trigger_action":"ProcessToolCalls","result":"success","callback_action":"HandleToolResults","duration_ms":106} - -event: state_change -data: {"seq":45,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"HandleToolResults","status":"Thinking","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":46,"integration":"call_llm","module":"llm_caller","trigger_action":"HandleToolResults"} - -event: prompt_assembled -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":47,"kind":"prompt_assembled","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"system prompt assembled","timestamp":"2026-03-24T12:14:56.876983+00:00","data":{"kind":"prompt_assembled","message":"system prompt assembled","system_prompt":"# Proof Soul\n\n## Identity\nYou are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite.\n\n## Instructions\n- Prefer deterministic mock runs for verification.\n- Surface memory and skills in the prompt.\n- Use tools only when the proof plan requires them.\n\n## Capabilities\n- Run sandbox tools\n- Spawn governed child agents\n- Save and recall memories\n\n## Constraints\n- Do not use destructive commands.\n- Stay inside the provided workspace.\n\n\nOverride: include the DIRECT-OVERRIDE marker.\n\n\n \n \n\n\n\n \n The proof user prefers exact verification over discussion.\n \n \n Temper Pi rewrite proof must capture SSE, session trees, cron, heartbeat, channels, and MCP.\n \n \n saved from direct path\n \n"}} - -event: state_change -data: {"seq":48,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"Heartbeat","status":"Thinking","tenant":"temper-agent-proof-20260324121451"} - -event: llm_request_started -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":49,"kind":"llm_request_started","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"calling provider=mock model=mock-proof","timestamp":"2026-03-24T12:14:56.886359+00:00","data":{"kind":"llm_request_started","message":"calling provider=mock model=mock-proof"}} - -event: llm_response -data: {"tenant":"temper-agent-proof-20260324121451","entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","seq":50,"kind":"llm_response","agent_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","tool_call_id":null,"tool_name":"llm_caller","task_id":null,"message":"provider returned stop_reason=end_turn","timestamp":"2026-03-24T12:14:56.886868+00:00","data":{"kind":"llm_response","message":"provider returned stop_reason=end_turn","stop_reason":"end_turn"}} - -event: integration_complete -data: {"seq":51,"integration":"call_llm","module":"llm_caller","trigger_action":"HandleToolResults","result":"success","callback_action":"CheckSteering","duration_ms":69} - -event: state_change -data: {"seq":52,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"CheckSteering","status":"Steering","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: integration_start -data: {"seq":53,"integration":"check_steering","module":"steering_checker","trigger_action":"CheckSteering"} - -event: integration_complete -data: {"seq":54,"integration":"check_steering","module":"steering_checker","trigger_action":"CheckSteering","result":"success","callback_action":"FinalizeResult","duration_ms":11} - -event: state_change -data: {"seq":55,"entity_type":"TemperAgent","entity_id":"019d1fc4-f16f-7452-9119-79ae692dc5ae","action":"FinalizeResult","status":"Completed","tenant":"temper-agent-proof-20260324121451","agent_id":"system"} - -event: agent_complete -data: {"seq":56,"status":"Completed","action":"FinalizeResult","result":"Direct path finished with memory keys user-profile, project-context, proof-direct-memory.","error_message":null,"agent_id":"system","session_id":null} - - -``` - -### OTS Trajectory Summary -```json -{ - "total": 837, - "success_count": 796, - "error_count": 41, - "success_rate": 0.951015531660693, - "by_action": { - "Activate": { - "total": 5, - "success": 5, - "error": 0 - }, - "CheckSteering": { - "total": 79, - "success": 79, - "error": 0 - }, - "CompactionComplete": { - "total": 4, - "success": 4, - "error": 0 - }, - "Configure": { - "total": 80, - "success": 75, - "error": 5 - }, - "Connect": { - "total": 14, - "success": 14, - "error": 0 - }, - "ContinueWithSteering": { - "total": 18, - "success": 18, - "error": 0 - }, - "Create": { - "total": 11, - "success": 11, - "error": 0 - }, - "CreateGovernanceDecision": { - "total": 38, - "success": 38, - "error": 0 - }, - "Fail": { - "total": 13, - "success": 10, - "error": 3 - }, - "FinalizeResult": { - "total": 61, - "success": 61, - "error": 0 - }, - "HandleToolResults": { - "total": 62, - "success": 62, - "error": 0 - }, - "Heartbeat": { - "total": 283, - "success": 259, - "error": 24 - }, - "NeedsCompaction": { - "total": 4, - "success": 4, - "error": 0 - }, - "ProcessToolCalls": { - "total": 62, - "success": 62, - "error": 0 - }, - "Provision": { - "total": 79, - "success": 75, - "error": 4 - }, - "Publish": { - "total": 14, - "success": 14, - "error": 0 - }, - "Ready": { - "total": 14, - "success": 14, - "error": 0 - }, - "ReceiveMessage": { - "total": 14, - "success": 14, - "error": 0 - }, - "ReplyDelivered": { - "total": 11, - "success": 11, - "error": 0 - }, - "RouteFailed": { - "total": 3, - "success": 3, - "error": 0 - }, - "SandboxReady": { - "total": 65, - "success": 65, - "error": 0 - }, - "Save": { - "total": 14, - "success": 11, - "error": 3 - }, - "ScanComplete": { - "total": 4, - "success": 4, - "error": 0 - }, - "ScheduleFailed": { - "total": 3, - "success": 3, - "error": 0 - }, - "SendReply": { - "total": 11, - "success": 11, - "error": 0 - }, - "Start": { - "total": 4, - "success": 4, - "error": 0 - }, - "Steer": { - "total": 23, - "success": 18, - "error": 5 - }, - "StreamUpdated": { - "total": 666, - "success": 666, - "error": 0 - }, - "TimeoutFail": { - "total": 4, - "success": 4, - "error": 0 - }, - "Trigger": { - "total": 9, - "success": 9, - "error": 0 - }, - "TriggerComplete": { - "total": 9, - "success": 9, - "error": 0 - }, - "manage_policies": { - "total": 2, - "success": 0, - "error": 2 - } - }, - "failed_intents": [ - { - "tenant": "temper-agent-proof-20260324053709", - "entity_type": "TemperAgent", - "entity_id": "019d1e58-f3e7-7932-82cd-88058cbfb00b", - "action": "Fail", - "success": false, - "from_status": "Thinking", - "to_status": "Failed", - "error": "Action 'Fail' not valid from state 'Failed'", - "agent_id": "system", - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:37:29.553394+00:00", - "request_body": "{\"error\":\"mock hang scenario finished without heartbeat\",\"error_message\":\"mock hang scenario finished without heartbeat\",\"integration\":\"call_llm\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324053613", - "entity_type": "TemperAgent", - "entity_id": "019d1e58-1c78-7fe2-a021-c9c6e1abc8bc", - "action": "Fail", - "success": false, - "from_status": "Thinking", - "to_status": "Failed", - "error": "Action 'Fail' not valid from state 'Failed'", - "agent_id": "system", - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:36:34.382826+00:00", - "request_body": "{\"error\":\"mock hang scenario finished without heartbeat\",\"error_message\":\"mock hang scenario finished without heartbeat\",\"integration\":\"call_llm\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324052805", - "entity_type": "TemperAgent", - "entity_id": "019d1e51-9a6a-7422-80f0-54e98068cf18", - "action": "Fail", - "success": false, - "from_status": "Thinking", - "to_status": "Failed", - "error": "Action 'Fail' not valid from state 'Failed'", - "agent_id": "system", - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:29:27.904381+00:00", - "request_body": "{\"error\":\"mock hang scenario finished without heartbeat\",\"error_message\":\"mock hang scenario finished without heartbeat\",\"integration\":\"call_llm\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324052805", - "entity_type": "TemperAgent", - "entity_id": "019d1e50-ae5f-79d3-ac83-cad10d32daff", - "action": "Provision", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774330097", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e50-ae5f-79d3-ac83-cad10d32daff", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:28:17.278573+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324052805", - "entity_type": "TemperAgent", - "entity_id": "019d1e50-ae5f-79d3-ac83-cad10d32daff", - "action": "Configure", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774330097", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e50-ae5f-79d3-ac83-cad10d32daff", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:28:17.264156+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324052055", - "entity_type": "TemperAgent", - "entity_id": "019d1e4a-1ae5-7cb1-b72a-14a0fa822293", - "action": "Provision", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774329666", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e4a-1ae5-7cb1-b72a-14a0fa822293", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:21:06.312690+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324052055", - "entity_type": "TemperAgent", - "entity_id": "019d1e4a-1ae5-7cb1-b72a-14a0fa822293", - "action": "Configure", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774329666", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e4a-1ae5-7cb1-b72a-14a0fa822293", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:21:06.296204+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324052055", - "entity_type": "TemperAgent", - "entity_id": "proof-sub-child", - "action": "Steer", - "success": false, - "from_status": "", - "to_status": "Created", - "error": "Action 'Steer' not valid from state 'Created'", - "agent_id": null, - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:21:03.050500+00:00", - "request_body": "{\"steering_messages\":\"[{\\\"content\\\":\\\"STEERED-CHILD\\\"}]\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051844", - "entity_type": "TemperAgent", - "entity_id": "019d1e49-05ad-7750-93b9-c1495350f029", - "action": "Provision", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774329595", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e49-05ad-7750-93b9-c1495350f029", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:19:55.350968+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051844", - "entity_type": "TemperAgent", - "entity_id": "019d1e49-05ad-7750-93b9-c1495350f029", - "action": "Configure", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774329595", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e49-05ad-7750-93b9-c1495350f029", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:19:55.329985+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051844", - "entity_type": "TemperAgent", - "entity_id": "019d1e48-1ac0-7d53-b740-578b822ded2d", - "action": "Provision", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774329535", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e48-1ac0-7d53-b740-578b822ded2d", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:18:55.204184+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051844", - "entity_type": "TemperAgent", - "entity_id": "019d1e48-1ac0-7d53-b740-578b822ded2d", - "action": "Configure", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": "proof-1774329535", - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e48-1ac0-7d53-b740-578b822ded2d", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:18:55.187204+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051844", - "entity_type": "TemperAgent", - "entity_id": "proof-sub-child", - "action": "Steer", - "success": false, - "from_status": "", - "to_status": "Created", - "error": "Action 'Steer' not valid from state 'Created'", - "agent_id": null, - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:18:52.071697+00:00", - "request_body": "{\"steering_messages\":\"[{\\\"content\\\":\\\"STEERED-CHILD\\\"}]\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051726", - "entity_type": "TemperAgent", - "entity_id": "proof-sub-child", - "action": "Steer", - "success": false, - "from_status": "", - "to_status": "Created", - "error": "Action 'Steer' not valid from state 'Created'", - "agent_id": null, - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:17:32.808465+00:00", - "request_body": "{\"steering_messages\":\"[{\\\"content\\\":\\\"STEERED-CHILD\\\"}]\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051552", - "entity_type": "TemperAgent", - "entity_id": "proof-sub-child", - "action": "Steer", - "success": false, - "from_status": "", - "to_status": "Created", - "error": "Action 'Steer' not valid from state 'Created'", - "agent_id": null, - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:15:58.536092+00:00", - "request_body": "{\"steering_messages\":\"[{\\\"content\\\":\\\"STEERED-CHILD\\\"}]\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324051424", - "entity_type": "TemperAgent", - "entity_id": "proof-sub-child", - "action": "Steer", - "success": false, - "from_status": "", - "to_status": "Created", - "error": "Action 'Steer' not valid from state 'Created'", - "agent_id": null, - "session_id": null, - "authz_denied": null, - "denied_resource": null, - "denied_module": null, - "source": "Entity", - "spec_governed": null, - "created_at": "2026-03-24T05:14:31.455917+00:00", - "request_body": "{\"steering_messages\":\"[{\\\"content\\\":\\\"STEERED-CHILD\\\"}]\"}", - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324050057", - "entity_type": "TemperAgent", - "entity_id": "019d1e37-c5fb-7c90-9e72-fd2614d747bc", - "action": "Configure", - "success": false, - "from_status": "Created", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": null, - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e37-c5fb-7c90-9e72-fd2614d747bc", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:01:04.909880+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324050057", - "entity_type": "TemperAgent", - "entity_id": "019d1e37-b196-74d0-aa86-2adabd01af1d", - "action": "Heartbeat", - "success": false, - "from_status": "Thinking", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": null, - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e37-b196-74d0-aa86-2adabd01af1d", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:01:02.456299+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324050057", - "entity_type": "TemperAgent", - "entity_id": "019d1e37-b196-74d0-aa86-2adabd01af1d", - "action": "Heartbeat", - "success": false, - "from_status": "Executing", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": null, - "authz_denied": true, - "denied_resource": "TemperAgent:019d1e37-b196-74d0-aa86-2adabd01af1d", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:01:02.305524+00:00", - "request_body": null, - "intent": null - }, - { - "tenant": "temper-agent-proof-20260324050057", - "entity_type": "AgentMemory", - "entity_id": "019d1e37-bbbe-7f51-b4a0-ea126e832d58", - "action": "Save", - "success": false, - "from_status": "Active", - "to_status": null, - "error": "no matching permit policy", - "agent_id": "anonymous", - "session_id": null, - "authz_denied": true, - "denied_resource": "AgentMemory:019d1e37-bbbe-7f51-b4a0-ea126e832d58", - "denied_module": null, - "source": "Authz", - "spec_governed": null, - "created_at": "2026-03-24T05:01:02.290204+00:00", - "request_body": null, - "intent": null - } - ] -} -``` - -### System Prompt Assembly -```text -# Proof Soul - -## Identity -You are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite. - -## Instructions -- Prefer deterministic mock runs for verification. -- Surface memory and skills in the prompt. -- Use tools only when the proof plan requires them. - -## Capabilities -- Run sandbox tools -- Spawn governed child agents -- Save and recall memories - -## Constraints -- Do not use destructive commands. -- Stay inside the provided workspace. - - -Override: include the DIRECT-OVERRIDE marker. - - - - - - - - - The proof user prefers exact verification over discussion. - - - Temper Pi rewrite proof must capture SSE, session trees, cron, heartbeat, channels, and MCP. - - -``` - -## Current Limitations -- None observed in the proof run. - -## Post-Proof Code Review Fixes - -The following issues were identified by code review and fixed after the initial proof run: - -### Fix 1: Extract duplicate TemperFS helpers into `wasm-helpers` crate -- **Issue**: `resolve_temper_api_url`, `read_session_from_temperfs`, `write_session_to_temperfs`, `entity_field_str` were duplicated across steering_checker, context_compactor, heartbeat_scan, cron_scheduler_check, and cron_trigger. -- **Fix**: Created `os-apps/temper-agent/wasm/wasm-helpers/` shared library crate with 6 unit tests. Updated all 5 modules to import from `wasm_helpers::*` instead of duplicating. - -### Fix 2: Server-side filtering in route_message -- **Issue**: `find_active_session` fetched ALL ChannelSessions then filtered in WASM memory — O(n) scan on every message. -- **Fix**: Added `$filter=Status eq 'Active' and ChannelId eq '{channel_id}' and ThreadId eq '{thread_id}'` to the OData query, letting the server filter. - -### Fix 3: Real timestamp comparison in heartbeat_scan -- **Issue**: Agents with a non-empty `last_heartbeat_at` were only logged, never compared against the timeout. Only agents with no heartbeat at all were timed out. -- **Fix**: Added `parse_iso8601_to_epoch_secs` to `wasm-helpers` and updated heartbeat_scan to compare `now - last_heartbeat > timeout_secs`. Reference time comes from `last_scan_at` on the HeartbeatMonitor entity. - -### Fix 4: Allow agents to manage their own memories -- **Issue**: `memory.cedar` restricted Save/Update/Recall to `["system", "supervisor", "human"]` agent types. Regular agents (the ones that actually need memory) were denied. -- **Fix**: Added a permit rule: `principal.agent_type == "agent" && resource.SoulId == principal.soul_id` — agents can manage memories scoped to their own soul. - -## Reproduction Commands -```bash -python3 scripts/temper_agent_e2e_proof.py -cargo test --workspace -``` diff --git a/.proofs/0129-directed-evolution-repair-aware-variant-lanes.md b/.proofs/0129-directed-evolution-repair-aware-variant-lanes.md deleted file mode 100644 index d4fa27938..000000000 --- a/.proofs/0129-directed-evolution-repair-aware-variant-lanes.md +++ /dev/null @@ -1,55 +0,0 @@ -# Directed Evolution Repair-Aware Variant Lanes Proof - -Date: 2026-05-27 - -## Scope - -The episode orchestrator now switches variant lane suggestions to repair-focused -lanes when the episode context is repair pressure. This addresses the live -repair cycle where growth-flavored lanes produced variants that were correctly -eliminated for missing the repair target. - -## Verification - -```text -cargo test --manifest-path os-apps/directed-evolution/wasm/episode_orchestrator/Cargo.toml --quiet -running 2 tests -2 passed - -./os-apps/directed-evolution/wasm/build.sh -signal_observer built -episode_orchestrator built -work_item_result_router built - -git diff --check -passed -``` - -## Live Evidence That Drove The Change - -Tenant: - -```text -de-live-repair-cycle-20260527081922 -``` - -The cycle reached a terminal failed state: - -```text -Episode en-019e6885-f57b-7e90-8a7e-6923e953b5f5: Failed -Generation en-019e6885-f88d-71f2-b37b-8f691f4d797a: Failed -Reason: All variants were eliminated before selection. -``` - -Evaluator evidence showed prompt drift: - -```text -Variant 2 was eliminated because it added intent-capture/product metadata while -the observed pressure was submitted-answer visibility and acceptance -actionability. -``` - -## Deployment Note - -This is hot-loadable Directed Evolution app/WASM behavior. It does not require a -Railway deployment. diff --git a/.proofs/2026-04-27-delta-os-app-reconcile.md b/.proofs/2026-04-27-delta-os-app-reconcile.md deleted file mode 100644 index 01a31d353..000000000 --- a/.proofs/2026-04-27-delta-os-app-reconcile.md +++ /dev/null @@ -1,107 +0,0 @@ -# Delta OS-App Reconcile Proof - -Date: 2026-04-27 - -Worktree: `/Users/seshendranalla/Development/temper-worktrees/os-app-delta-reconcile` - -## Red Tests - -Added failing tests before implementation: - -- `upsert_specs_and_commit_preserves_identical_spec_version` -- `upsert_wasm_module_preserves_version_for_identical_hash` -- `upsert_wasm_module_stores_artifact_outside_metadata_row` -- `test_reconcile_plan_for_wasm_only_digest_skips_unrelated_phases` -- `test_reconcile_os_app_repairs_spec_content_drift_despite_matching_digest` - -Initial failures on `main` behavior: - -- identical spec commit bumped version from `1` to `2` -- identical WASM upsert bumped version from `1` to `2` -- new WASM metadata row stored `7` inline bytes instead of `0` -- reconcile plan API did not exist yet -- matching installed bundle digest skipped hot reconcile even when live runtime - spec content had drifted - -## Verification Commands - -```bash -cargo test -p temper-store-turso -``` - -Result: passed. `34` unit tests, `5` blob TTL e2e tests, and doctests passed. - -```bash -cargo test -p temper-platform os_apps::mod_test -``` - -Result: passed. `52` OS-app platform tests passed. - -```bash -cargo run -p temper-cli -- --help -``` - -Result: passed. The Temper CLI built and printed help. - -## Local Server Smoke - -Started a local Temper server with a temp Turso DB and the project-management app: - -```bash -rm -f /tmp/temper-delta-reconcile-e2e.db /tmp/temper-delta-reconcile-e2e.db-wal /tmp/temper-delta-reconcile-e2e.db-shm -TURSO_URL=file:/tmp/temper-delta-reconcile-e2e.db \ - cargo run -p temper-cli -- serve --storage turso --no-observe --port 39876 --skill project-management -``` - -Observed: - -- server listened on `http://0.0.0.0:39876` -- `curl -fsS http://127.0.0.1:39876/healthz` exited successfully -- project-management installed for tenant `default` - -Queried the local DB: - -```bash -sqlite3 /tmp/temper-delta-reconcile-e2e.db \ - "select app_name, length(bundle_digest)>0 as has_bundle, length(spec_digest)>0 as has_spec from tenant_installed_apps where tenant_id='default'; - select entity_type, version, committed from specs where tenant='default' order by entity_type;" -``` - -Observed: - -- `tenant_installed_apps` had `project-management` with bundle and spec digests present -- committed specs for `default` included `Comment`, `Cycle`, `Issue`, `Label`, and `Project` -- project-management spec versions were `1` and committed - -Stopped the server with `SIGTERM` and confirmed no matching process remained. - -## Notes - -The local smoke run is a cold install because it starts from an empty DB. Delta reconcile behavior is covered by the OS-app tests: - -- `test_reconcile_os_app_delta_content_change_skips_specs` forces a changed bundle/content digest with matching spec/policy/WASM/seed digests and verifies reconcile does not classify or bootstrap specs. -- `test_install_plan_without_spec_phase_does_not_reclassify_specs` verifies the installer obeys a plan with the spec phase disabled. -- `test_reconcile_os_app_repairs_spec_content_drift_despite_matching_digest` verifies a matching installed bundle record does not hide drifted live spec content. - -## OpenPaw Live E2E - -Ran OpenPaw in a disposable worktree using patched local Temper crates via -Cargo `[patch]` entries, a disposable home, and a file-backed Turso DB. - -Cold boot from an empty DB: - -- `/readyz` returned HTTP `200` -- `phase_6b_os_app_reconcile`: `11,813ms` -- startup time to ready: `12,227ms` -- `wasm_modules`: `31` rows, `31` metadata-only rows, `31` blob artifacts, - `15,272,520` bytes in blobs, min/max metadata version `1/1` -- `specs`: `default` had `32` committed specs, min/max version `1/1`; - `temper-system` had `13` committed specs, min/max version `1/1` - -Warm boot using the same DB and `TEMPERPAW_WASM_STARTUP_POLICY=load-only`: - -- `/readyz` returned HTTP `200` -- all six startup apps logged `Skipped unchanged OS app` -- `phase_6b_os_app_reconcile`: `1,267ms` -- startup time to ready: `1,862ms` -- `wasm_modules` and `specs` row counts and versions stayed unchanged diff --git a/.proofs/full-trace-and-session-perf-2026-04-23.md b/.proofs/full-trace-and-session-perf-2026-04-23.md deleted file mode 100644 index 3fc474906..000000000 --- a/.proofs/full-trace-and-session-perf-2026-04-23.md +++ /dev/null @@ -1,57 +0,0 @@ -# Full Trace And Session Perf Verification - -Date: 2026-04-23 -Worktree: `/Users/seshendranalla/Development/temper-worktrees/full-trace-and-session-perf` -Branch: `codex/full-trace-and-session-perf` - -## Scope - -- Keep LLM spans on the active dispatch trace instead of detaching them. -- Add a local text fast path for internal `GET/PUT /tdata/Files('{id}')/$value` requests so WASM does not pay loopback HTTP for UTF-8 file traffic. -- Add a host-side batch HTTP primitive so WASM modules can execute independent text HTTP requests concurrently without inventing their own loopback orchestration. - -## Commands - -1. `cargo test -p temper-wasm current_traceparent_header_prefers_active_span_context -- --nocapture` - Result: passed -2. `cargo test -p temper-server parse_internal_file_value_request_matches_only_value_paths -- --nocapture` - Result: passed -3. `cargo test -p temper-server llm_root_span_stays_on_active_trace -- --nocapture` - Result: passed -4. `cargo test -p temper-wasm default_http_call_batch_runs_requests_concurrently -- --nocapture` - Result: passed -5. `cargo test -p temper-wasm --lib -- --nocapture` - Result: passed (`71` tests) -6. `cargo run -p temper-cli -- serve --no-observe --port 3313` - Result: built successfully and reached `Listening on http://0.0.0.0:3313` - -## Notes - -- Server boot emitted existing ADR-0050 liveness warnings from loaded specs; these predate this change. -- The runtime boot check confirms the tracing changes, local file fast path, and new batch host ABI did not prevent Temper from starting. - -## Addendum: 2026-04-24 Query-Plane Stability Verification - -### Additional Scope - -- Add `query_indexed = false` support to spec parsing and observation output. -- Filter query-plane projections using that spec metadata. -- Add `projection_hash` to `entity_catalog` so unchanged projections update catalog metadata without rebuilding field rows. - -### Additional Commands - -7. `cargo test -p temper-spec test_state_query_index_flag_parsed -- --nocapture` - Result: passed -8. `cargo test -p temper-store-turso unchanged_projection_updates_catalog_without_rebuilding_field_rows -- --nocapture` - Result: passed -9. `cargo test -p temper-server query_projection_excludes_fields_marked_not_query_indexed -- --nocapture` - Result: passed -10. `cargo test -p temper-store-turso --lib -- --nocapture` - Result: passed (`29` tests) -11. `cargo test -p temper-server --test query_projection_backfill -- --nocapture` - Result: passed (`3` tests) - -### Additional Notes - -- The no-op projection test proves `entity_catalog.sequence_nr` still advances even when the durable field-index rows do not need to be rewritten. -- OpenPaw’s Session E2E on the patched stack confirmed the practical outcome: heartbeat/progress hot fields were excluded from `entity_field_index` while the session still completed and the catalog row advanced. diff --git a/.proofs/knuth-postgres-migration-full.md b/.proofs/knuth-postgres-migration-full.md deleted file mode 100644 index bfb7dc9e5..000000000 --- a/.proofs/knuth-postgres-migration-full.md +++ /dev/null @@ -1,164 +0,0 @@ -# Knuth Postgres Migration Full Proof - -Date: 2026-04-28 -Updated: 2026-04-29 -Branch: `codex/knuth-postgres-migration-full` - -## Commits - -- `f6b1996` Add Postgres platform storage parity -- `f4a07ee` Bound trajectory persistence outbox -- `e222a3b` Select storage backend from environment -- `477ec47` Route Cedar policies through Postgres storage -- `0e6b52d` Allow Railway storage selection from env -- `d43a879` Fix local Postgres compose bootstrap -- `7749307` Route platform long-tail reads through Postgres -- `f66a2a2` Add Turso to Postgres migration command -- `727dab6` Route batched projections through Postgres -- `199ed52` Record Postgres migration proof -- `b80fa68` Record Datadog migration evidence -- `9bce8ad` Introduce StorageStack object-safe adapter -- `42bfdd2` Record Postgres cutover ADR gates -- `f43e0ab` Use versioned Postgres migrations -- `a7fc9fd` Remove stale bootstrap import -- `25d99c6` Record post-review migration proof - -## Verification Run - -- Red tests observed: - - `cargo test -p temper-store-postgres postgres_long_tail_methods_are_part_of_the_store_surface` failed before long-tail methods existed. - - `cargo test -p temper-cli test_cli_parse_migrate_turso_to_postgres` failed before the CLI command existed. - - `cargo test -p temper-store-postgres postgres_query_projection_batch_method_is_part_of_the_store_surface` failed before the Postgres batch projection reader existed. - - `cargo test -p temper-server --test storage_stack` failed before `temper_server::storage` existed. - - `cargo test -p temper-server --test storage_stack` failed again before `QueryPlaneStore`, `TrajectorySink`, and backend-neutral projection rows existed on `StorageStack`. - - `cargo test -p temper-store-postgres versioned_migration_is_the_schema_source` failed before `migrations/0001_initial.sql` existed. -- Green checks: - - `cargo fmt` - - `cargo check -p temper-server` - - `cargo check -p temper-cli` - - `cargo test -p temper-server --test storage_stack` - - `cargo test -p temper-server --test query_projection_backfill` - - `cargo test -p temper-store-postgres` - - `DATABASE_URL=postgres://temper:temper_dev@localhost:5432/temper cargo test -p temper-store-postgres` - - `DATABASE_URL=postgres://temper:temper_dev@localhost:5432/temper cargo test -p temper-store-postgres query_projection` - - `cargo test -p temper-store-turso` - - `DATABASE_URL=postgres://temper:temper_dev@localhost:5432/temper cargo test -p temper-cli` - - `DATABASE_URL=postgres://temper:temper_dev@localhost:5432/temper cargo test -p temper-cli smoke_migration_copies_events_snapshots_specs_projections_and_blobs_when_database_url_set -- --nocapture` - - `git diff --check` - -## Local E2E Evidence - -- Started local Postgres with `docker compose up -d postgres`. -- Ran a real migration smoke test from a local Turso DB into Docker Postgres. It copied event, snapshot, spec, query projection, and blob rows and asserted all landed in Postgres. -- Ran the CLI in dry-run/verify mode: - - `temper migrate-turso-to-postgres --tenant all --dry-run --verify --from-snapshot --turso-url file:/tmp/temper-cli-migration-empty.db` - - Manifest written to `/tmp/temper-cli-migration-manifest.json` with verified checksums for all empty source tables. -- Booted the actual server: - - `DATABASE_URL=postgres://temper:temper_dev@localhost:5432/temper TEMPER_EVENT_STORE=postgres cargo run -p temper-cli -- serve --port 43123 --no-observe` - - `/healthz` returned `HTTP/1.1 200 OK`. - - The boot path ran `sqlx::migrate!()` against `crates/temper-store-postgres/migrations/0001_initial.sql`; logs showed `_sqlx_migrations` present and Postgres migrations applied. - - `information_schema` check found all 7 sampled platform tables: `events`, `specs`, `trajectories`, `entity_catalog`, `blobs`, `tenant_secrets`, `policy_denial_patterns`. - -## Post-Review Remediation - -- Added `crates/temper-server/src/storage/mod.rs` with `DynEventStore`, `BoxedEventStore`, `BackendLabel`, and `StorageStack`. -- Wired `ServerState::set_event_store` so serve/bootstrap paths derive a first-class storage stack alongside the transitional `ServerEventStore` compatibility handle. -- Updated ADR-0066 to remove the object-safety deferral and document the query-plane / trajectory capability split. -- Added `QueryPlaneStore` and `TrajectorySink` capabilities to `StorageStack`, then moved production query-plane/trajectory writers and readers onto those traits: - - dispatch background projection updates - - entity create/update/delete projection writes - - projection backfill writes - - file metadata projection reads - - OData filter push-down - - runtime projection coverage metrics - - trajectory outbox and direct trajectory persistence -- Replaced the hand-rolled Postgres migration runner with `sqlx::migrate!()` and added `crates/temper-store-postgres/migrations/0001_initial.sql`. -- Added or corrected ADRs for the missing architectural records: - - ADR-0069: HttpEndpoint (renumbered from the duplicate ADR-0056) - - ADR-0070: Postgres multi-tenant isolation - - ADR-0071: storage retry classification - - ADR-0072: ProgressMade cadence - - ADR-0073: runtime index recovery - - ADR-0074: Turso to Postgres ETL methodology - - ADR-0075: tenant secrets key management -- Added `docs/runbooks/postgres-cutover.md` with Railway Postgres dry-run instructions, real-stack e2e gates, Datadog tripwires, and rollback steps. - -## Production Datadog Evidence - -Queried Datadog on 2026-04-29 through the Datadog MCP connector. The local shell environment still has no `DD_`/`DATADOG_` credentials, so dashboard, monitor, and log evidence below came from the connector rather than local env vars. - -- Dashboard found: `mn4-k3k-i66` (`TemperPaw -- Platform Overview`) - - URL: `https://app.datadoghq.com/dashboard/mn4-k3k-i66` - - The `State Liveness (ADR-0049 / ADR-0050)` group contains the production metric queries for: - - `sum:temper_state_timeout_fired_total{service:openpaw} by {entity_type,state}.as_count()` - - `sum:temper_state_timeout_reset_total{service:openpaw}.as_count() / (sum:temper_state_timeout_reset_total{service:openpaw}.as_count() + sum:temper_state_timeout_fired_total{service:openpaw}.as_count())` - - `avg:temper_scheduler_pending_timers{service:openpaw} by {entity_type}` - - `sum:temper_scheduler_overdue_on_replay_total{service:openpaw} by {entity_type}.as_count()` - - `sum:temper_spec_liveness_violations_total{service:openpaw} by {entity_type,state}.as_count()` - - `sum:temper_state_timeout_cancelled_total{service:openpaw} by {entity_type,state}.as_count()` - - `sum:temper_state_timeout_reset_total{service:openpaw} by {entity_type,state}.as_count()` -- Monitor search for `openpaw AND (session OR timeout OR orphan)` returned active production monitors: - - `275383770` `[Temper] Abnormal State Timeout Firing`: `OK` - - Query: `sum(last_15m):sum:temper_state_timeout_fired_total{service:openpaw} by {entity_type,state}.as_count() > 25` - - `275384441` `[Temper] State Timeout Reset Rate Drop`: `No Data` - - Query: `sum(last_1h):sum:temper_state_timeout_reset_total{service:openpaw,state:Executing}.as_count() < 1` - - `275383795` `[Temper] WASM Default Timeout Fallback Rate`: `OK` - - Query: `sum(last_1h):sum:temper_wasm_integration_default_timeout_used_total{service:openpaw}.as_count() > 500` - - `275383796` `[Temper] Session Memory Externalization Spike`: `No Data` - - `275384307` `[Temper] Session Memory Budget Exceeded`: `No Data` -- Log pattern query `service:openpaw (session OR timeout OR orphan)`, `from=now-7d`, `to=now`, grouped by `service,env,status`, returned 42 patterns. Relevant production counts: - - `Orphaned session recovery skipped; set TEMPERPAW_ORPHANED_SESSION_RECOVERY=true to enable bounded recovery`: 24 info logs from 2026-04-27 to 2026-04-28. - - `Failing orphaned session`: 22 info logs from 2026-04-22 to 2026-04-26. - - `Session recovery complete`: 8 info logs from 2026-04-22 to 2026-04-26. - - `Deferred session recovery complete`: 6 info logs from 2026-04-27 to 2026-04-28. - - `Deferred session recovery scheduled after readiness`: 6 info logs from 2026-04-27 to 2026-04-28. - - `Session recovery deferred until after readiness`: 6 info logs from 2026-04-27 to 2026-04-28. - - `ChannelSession ... points at unreadable Session ... HTTP 404; starting a fresh session`: 3 warn logs on 2026-04-28. - - `route_message: routed ... to fresh session ... after stale binding`: 3 info logs on 2026-04-28. - - `route_message: dispatched ResumeTools on stale session ...`: 2 info logs from 2026-04-26 to 2026-04-28. -- Aggregate Datadog log analytics over `now-7d`: - - `service:openpaw (orphan OR "orphaned session" OR "stale session" OR "unreadable Session")`: 51 logs total (`info=48`, `warn=3`). - - `service:openpaw timeout`: 114 logs total (`warn=105`, `error=9`). - - `service:openpaw (recovery OR recovered OR "Session recovery")`: 226 logs total (`info=226`). - - Daily liveness-related query `service:openpaw (orphan OR timeout OR recovery OR stale)`: - - 2026-04-28: `info=45` - - 2026-04-27: `info=97`, `error=2` - - 2026-04-26: `info=46`, `error=6`, `warn=5` - - 2026-04-25: `info=22`, `error=1` - - 2026-04-24: `info=5` - - 2026-04-23: `info=6` - - 2026-04-22: `warn=100`, `info=10` - -## Not Exercised - -- Real Discord DM flow was not exercised because this local environment has no `DISCORD_BOT_TOKEN`. -- Real Katagami `CurationJob` review loop remains the next OpenPaw-side gate; the Temper storage query/trajectory compatibility gap has been closed for production callers. -- Production-shaped ETL into a disposable Railway Postgres database was not run from this environment. -- Production cutover and 48-hour Postgres soak have not happened; Turso write-gate/priority/bypass removals remain correctly gated on that soak. -- `ServerEventStore` compatibility still exists for event-journal, platform long-tail, and observe-read paths that have not yet grown dedicated capability traits. - -## ServerEventStore Retirement Follow-Up - -Addressed the remaining abstraction concern on 2026-04-29: - -- `StorageStack::from_server_event_store` now fills event, platform, - query-plane, and trajectory slots from concrete backend handles - (`PostgresEventStore`, `TursoEventStore`, `TenantStoreRouter`) instead of - `Arc` trait adapters. -- Added direct `QueryPlaneStore` and `TrajectorySink` implementations for - Postgres, Turso, and tenant-routed Turso. -- Kept `ServerEventStore` only as the explicit compatibility handle for - unmigrated methods. -- Added `docs/adrs/0076-eliminate-server-event-store-enum.md` with the - complete method inventory, target traits, and definition of done for - deleting the enum. - -Verification: - -```bash -cargo test -p temper-server --test storage_stack storage_stack_from_event_store_uses_concrete_capability_handles -``` - -Result: passed. The regression test failed before the implementation because -the capability trait object data pointer matched the compatibility enum pointer; -it now proves the stack capabilities are concrete handles. diff --git a/.proofs/session-stall-remediation.md b/.proofs/session-stall-remediation.md deleted file mode 100644 index 4d603b223..000000000 --- a/.proofs/session-stall-remediation.md +++ /dev/null @@ -1,79 +0,0 @@ -# Proof Report: Session Stall Remediation - -## Date - -2026-04-24 - -## Branch / Commit - -- Branch: `codex/session-stall-remediation` -- Commit: PR head commit -- Companion OpenPaw branch: `codex/session-stall-remediation` - -## What Was Done - -- Moved live query projection maintenance off the dispatch success path. -- Added background projection metrics: - - `temper_query_projection_update_enqueued_total` - - `temper_query_projection_update_duration_ms` - - `temper_query_projection_update_error_total` -- Kept those metrics in a small dedicated module so `runtime_metrics.rs` stays under the readability ratchet threshold. -- Wrapped background projection work in a `dispatch.phase.query_projection` span. -- Updated query projection integration tests to poll for eventual projection updates. -- Fixed guest metric kind handling so `kind = "count"` and `kind = "counter"` both produce OTEL counters. - -## Verification Flow - -| Step | Expected | Actual | Status | -|------|----------|--------|--------| -| Red test: projection mode | Test fails before background mode helper exists | `cargo test -p temper-server query_projection_updates_are_not_on_the_dispatch_critical_path` failed with missing helper/type | Pass | -| Red test: guest metric count kind | Test fails before count alias helper exists | `cargo test -p temper-wasm guest_metric_count_kind_is_counter` failed with missing helper | Pass | -| Projection mode unit test | Query projection mode is background | `cargo test -p temper-server query_projection_updates_are_not_on_the_dispatch_critical_path` passed | Pass | -| Query projection integration tests | Live upsert/delete and backfill remain correct under eventual updates | `cargo test -p temper-server --test query_projection_backfill` passed, 3 tests | Pass | -| Guest metric kind unit test | `count` and `counter` both classify as counters | `cargo test -p temper-wasm guest_metric_count_kind_is_counter` passed | Pass | -| Instrumentation guard | All registered runtime metrics have emission sites | `cargo run -p temper-server --bin check_instrumentation` passed | Pass | -| Readability ratchet | New metrics do not add a >1000-line production file | `bash scripts/readability-ratchet.sh check .ci/readability-baseline.env` passed | Pass | - -## Verification Results - -- Dispatch now enqueues durable query projection maintenance and returns without awaiting projection storage writes. -- Projection integration tests pass by observing eventual query index and catalog state. -- Projection update latency and error metrics have both registration and emission sites. -- Guest-emitted budget counters from OpenPaw will be interpreted as counters when they use the existing `kind = "count"` convention. - -## What Worked - -- Existing query projection coverage adapted cleanly to eventual consistency with bounded polling. -- The instrumentation checker accepted the new runtime metrics. -- The guest metric fix improves old and new WASM counter emissions without changing guest module call sites. - -## What Didn't Work - -- No issue in the focused Temper verification. - -## Limitations - -- This proof does not include a live OpenPaw source-search replay. It verifies the platform dispatch/projection behavior and metric plumbing that the OpenPaw remediation depends on. - -## What Still Doesn't Work - -- Query projections are now eventually consistent after live dispatch. This is intentional, but consumers that require read-your-write query-plane visibility must poll or read the entity directly. - -## Artifacts - -- `crates/temper-server/src/state/dispatch/effects.rs` -- `crates/temper-server/src/query_projection_metrics.rs` -- `crates/temper-server/src/runtime_metrics.rs` -- `crates/temper-server/tests/query_projection_backfill.rs` -- `crates/temper-wasm/src/host_trait.rs` - -## Architecture Diagram - -```text -entity dispatch succeeds - -> state transition durable - -> timers / reactions / callbacks continue - -> query projection update enqueued - -> background task upserts/removes query projection - -> emits duration and error metrics -``` diff --git a/AGENTS.md b/AGENTS.md index a3c204741..72ff8c9c2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,17 @@ In simulation-visible crates (`temper-runtime`, `temper-jit`, `temper-server`): - No `chrono::Utc::now()`, `std::thread::sleep()`, `OsRng`, `getrandom` - simulated time, seeded PRNG - `SimActorHandler::spec_invariants()` auto-checks `[[invariant]]` sections - `// determinism-ok` suppresses guard false positives -- Full ruleset: `.agents/agents/dst-reviewer.md` +- Full ruleset: `.agents/agents/dst-reviewer.md`; method: `.agents/skills/deterministic-simulation/` + +DST-driven development replaces TDD for stateful kernel code: + +1. Harness first: extend the simulator with the scenario - workload, faults, invariants (the things that must never happen). The invariant must FAIL before you implement; a harness that cannot catch the missing behavior proves nothing. +2. Implement. Production code runs inside the simulation - not mocks, not a parallel reimplementation. +3. Run many seeds until invariants hold. One green seed is one execution, not correctness. +4. A failing seed found later is committed as a regression case and stays in the suite forever. The seed is the bug report: seed, invariant violated, minimal trace. +5. Fix by root cause. Never by weakening the invariant or narrowing the workload. + +Suites: `platform_e2e_dst`, `system_entity_dst` (crates/temper-platform/tests/). Cooperative fault points (BUGGIFY-style rare branches under simulation) belong in production code. ## Multi-tenancy and identity diff --git a/REVIEW.md b/REVIEW.md index c3f3229c8..8e4b09673 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -6,6 +6,11 @@ Repo-specific passes on top of the global review bar. Severity + `file:line` + c Any touched code in `temper-runtime`, `temper-jit`, `temper-server`: scan against the DST ruleset in `.agents/agents/dst-reviewer.md`. Wall clock, random UUIDs, HashMap iteration, thread spawns, direct I/O, and global state are all findings even when tests pass - they break seeded reproduction. Check that new `// determinism-ok` suppressions are justified, not convenient. +DST-driven development checks (`.agents/skills/deterministic-simulation/`): +- New stateful behavior with no simulator scenario + invariant covering it is a finding. +- A fix that weakens an invariant or narrows a workload to go green is a finding. +- A bug fixed without its failing seed committed as a regression case is a finding. + ## Pass 2: Invariants and the spec contract - A spec change and its TransitionTable behavior must say the same thing; look for code paths that bypass the table. diff --git a/docs/internal/GAP_TRACKER.md b/docs/internal/GAP_TRACKER.md deleted file mode 100644 index 0c04716be..000000000 --- a/docs/internal/GAP_TRACKER.md +++ /dev/null @@ -1,68 +0,0 @@ -# Temper Gap Tracker - -> Generated: 2026-02-11 -> Last updated: 2026-02-12 (WS1/WS2/WS3 execution) - -## Blocking (must fix for basic functionality) - -| # | Gap | Crate | Status | Resolution | -|---|-----|-------|--------|------------| -| 1 | `_query_options` unused in dispatch.rs | temper-server | **RESOLVED** | `query_eval.rs` implements full $filter, $select, $orderby, $top, $skip, $count evaluation; wired into dispatch. | -| 2 | `$expand` parsed but not evaluated | temper-server, temper-odata | **RESOLVED** | `expand_entity()` resolves navigation properties via CSDL, fetches related entities, supports nested query options. | -| 3 | Entity set listing returns empty array | temper-server | **RESOLVED** | `list_entity_ids()` + `entity_index` tracks all created entities per tenant/type. | -| 4 | No PATCH/PUT/DELETE handlers | temper-server | **RESOLVED** | `handle_odata_patch`, `handle_odata_put`, `handle_odata_delete` all registered in router. DELETE removes from actor registry + entity index. | -| 5 | `_body_json` unused in entity creation | temper-server | **RESOLVED** | Body fields passed as `initial_fields` to entity actor on creation. | -| 6 | Entity GET returns 200 on not-found | temper-server | **RESOLVED** | `dispatch.rs` checks `entity_exists()` and returns 404 with OData error body. | -| 7 | No codegen from IOA specs | temper-codegen | **RESOLVED** | `build_spec_model_mixed()` handles IOA→StateMachine conversion; `generate_entity_module()` works transparently with both IOA and TLA+ specs. | - -**All 7 P0 items resolved.** - -## Important (needed for real-world usage) - -| # | Gap | Crate | Status | Resolution | -|---|-----|-------|--------|------------| -| 8 | `Custom(String)` effect not dispatched | temper-jit, temper-server | **RESOLVED** | `custom_effects: Vec` field added to `EntityResponse`. Success path collects `Effect::Custom(name)` values during transition processing. Surfaced in HTTP response JSON for downstream hook dispatch. | -| 9 | No event subscription mechanism | temper-server | **RESOLVED** | SSE endpoint at `GET /tdata/$events` with `tokio::sync::broadcast` channel. `EntityStateChange` notifications published after successful transitions. Also `GET /observe/events/stream` with entity_type/entity_id filtering. | -| 10 | No cross-entity coordination | temper-runtime, temper-server | **RESOLVED** | Choreography via reaction rules in `temper-server::reaction`. `ReactionRegistry` indexes rules per-tenant, `SimReactionSystem` for DST, `ReactionDispatcher` for async production. Bounded cascade (MAX_DEPTH=8). | -| 11 | No append-only collections | temper-jit, temper-server | **RESOLVED** | Full stack: `Guard::ListContains`, `Guard::ListLengthMin`, `Effect::ListAppend`, `Effect::ListRemoveAt` across temper-spec, temper-jit, temper-server, temper-verify. `lists: BTreeMap>` on EntityState. | -| 12 | No persistent evolution record storage | temper-evolution | **RESOLVED** | `PostgresRecordStore` in `pg_store.rs` — `evolution_records` table with JSONB payload, indexes on `(record_type, status)` and `(derived_from)`. Full CRUD + `ranked_insights()` + `update_status()`. | -| 13 | No real Redis adapter | temper-store-redis | **RESOLVED** | `RedisMailbox` (RPUSH/LPOP/LLEN), `RedisPlacement` (GET/SET/DEL + scan), `RedisCache` (SET with EX + scan) — all via fred v10. | -| 14 | Integration webhook has no retry | temper-platform | **RESOLVED** | Exponential backoff retry in `WebhookDispatcher`. `DeadLetterQueue` trait + `InMemoryDeadLetterQueue` for permanently failed deliveries. Wired via `with_dlq()`. | -| 15 | Liveness properties not verified | temper-verify | **RESOLVED** | `LivenessViolation` type added. `check_liveness_post_simulation()` checks NoDeadlock + ReachesState. Wired into L2 cascade. `check_reaches_state()` cleaned up in stateright_impl. | -| 16 | No Cedar policy hot-reload | temper-authz | **RESOLVED** | `RwLock` with atomic swap via `reload_policies()`. Invalid policies preserve existing set. `policy_count()` helper added. | -| 17 | No entity state persistence wiring | temper-server, temper-store-postgres | **RESOLVED** | `EntityActorHandler::handle()` persists events after transitions via `event_store.append()`. Actor recovery replays events via `replay_events()`. Full Postgres EventStore implementation with schema/migrations. | -| 18 | Claude API client has no mock | temper-platform | **RESOLVED** | `ChatClient` trait extracted. `MockClaudeClient` supports fixed and sequential canned responses. `ObservationAgent` and `AnalysisAgent` now generic with `with_client()` constructors. | -| 19 | Query options not applied | temper-server, temper-odata | **RESOLVED** | Same as #1 — `query_eval.rs` applies all parsed options to entity data. | - -**All 12 P1 items resolved.** - -## Nice to Have (future improvements) - -| # | Gap | Crate | Status | Notes | -|---|-----|-------|--------|-------| -| 20 | MaxCount guard never parsed | temper-spec | OPEN | | -| 21 | Hand-rolled TOML parser is fragile | temper-spec | OPEN | | -| 22 | Shadow testing uses legacy API only | temper-jit | OPEN | | -| 23 | `rule_index` lost on deserialization | temper-jit | **RESOLVED** | Custom `Deserialize` impl calls `rebuild_index()` after deserialization (commit 659c195). | -| 24 | No batch request support | temper-odata, temper-server | OPEN | | -| 25 | No `$search` support | temper-odata | OPEN | | -| 26 | No `$apply` aggregation support | temper-odata | OPEN | | -| 27 | init template uses relative path | temper-cli | OPEN | | -| 28 | No graceful shutdown | temper-cli | OPEN | | -| 29 | Optimization recommendations not applied | temper-optimize | OPEN | | -| 30 | No observability provider adapters beyond ClickHouse | temper-observe | OPEN | Prometheus metrics now available at `GET /observe/metrics` (text format). | -| 31 | Legacy TLA+ extractor is brittle | temper-spec | OPEN | | -| 32 | Generated code not validated | temper-codegen | OPEN | | -| 33 | No developer approval UI | temper-platform | **PARTIALLY RESOLVED** | Evolution API endpoints added: `GET /observe/evolution/records`, `GET /observe/evolution/records/{id}`, `POST /api/evolution/records/{id}/decide`, `GET /observe/evolution/insights`. Dashboard page deferred. | -| 34 | Proc macros limited to marker traits | temper-macros | OPEN | | -| 35 | No Sentinel anomaly detection | temper-evolution, temper-observe | **RESOLVED** | `SentinelActor` with 3 default rules: error rate spike, guard rejection rate, no activity. Auto-generates O-Records via `RecordStore`. Uses `sim_now()`/`sim_uuid()`. | -| 36 | IncrementItems/DecrementItems legacy aliases | temper-jit | OPEN | | - -## Summary - -| Priority | Total | Resolved | Open | -|----------|-------|----------|------| -| P0 | 7 | **7** | 0 | -| P1 | 12 | **12** | 0 | -| P2 | 17 | **3** (#23, #33 partial, #35) | 14 | -| **Total** | **36** | **22** | **14** | diff --git a/docs/internal/crate-refactor-plan.md b/docs/internal/crate-refactor-plan.md deleted file mode 100644 index 1cfc70451..000000000 --- a/docs/internal/crate-refactor-plan.md +++ /dev/null @@ -1,209 +0,0 @@ -# Crate Refactor Plan (2026-02-24) - -## Why - -The workspace has grown to ~60k LOC, with the biggest concentration in `temper-server`. -Current priorities are: - -1. Reduce bloat and line count. -2. Make behavior easier to read and reason about. -3. Close semantic gaps where implementation and intent diverge. - -## Baseline Size (Rust LOC) - -| Crate | LOC | Files | -|---|---:|---:| -| temper-server | 15,591 | 43 | -| temper-platform | 6,365 | 27 | -| temper-spec | 4,023 | 16 | -| temper-verify | 3,287 | 10 | -| temper-runtime | 3,177 | 18 | -| temper-cli | 2,815 | 9 | - -Largest single files: - -- `crates/temper-server/src/observe/mod.rs` (1234) -- `crates/temper-server/src/dispatch.rs` (1191) -- `crates/temper-mcp/src/lib.rs` (1151) - -## Findings - -### High - -1. SMT guard semantics are currently over-approximate for state/list value checks. - - `ModelGuard::StateIn` in SMT encodes only `!states.is_empty()` instead of status membership. - - `ModelGuard::ListContains` in SMT encodes `len > 0` and ignores requested value. - - Files: - - `crates/temper-verify/src/smt.rs:353` - - `crates/temper-verify/src/smt.rs:379` - - Impact: symbolic verification can report false results versus runtime/Stateright semantics. - -### Medium - -2. OData handler logic is duplicated across HTTP verbs in one large module. - - Repeated: tenant/entity-set resolution, verification gate checks, entity existence checks, response shaping. - - File: - - `crates/temper-server/src/dispatch.rs:150` - - `crates/temper-server/src/dispatch.rs:532` - - `crates/temper-server/src/dispatch.rs:811` - - `crates/temper-server/src/dispatch.rs:949` - - `crates/temper-server/src/dispatch.rs:1067` - - Impact: behavior drift risk and hard-to-review changes. - -3. Metadata persistence backend fan-out is hand-coded repeatedly. - - Same Postgres/Turso/Redis branching in each operation. - - File: - - `crates/temper-server/src/state/persistence.rs:19` - - `crates/temper-server/src/state/persistence.rs:103` - - `crates/temper-server/src/state/persistence.rs:156` - - `crates/temper-server/src/state/persistence.rs:212` - - Impact: repetitive control flow and inconsistent error handling over time. - -4. Variable-initial parsing logic is duplicated across crates. - - File: - - `crates/temper-mcp/src/lib.rs:451` - - `crates/temper-verify/src/model/builder.rs:178` - - `crates/temper-verify/src/model/builder.rs:185` - - Impact: parse behavior can diverge by surface area. - -### Low - -5. Test-heavy modules keep prod and tests mixed in very large files. - - File: - - `crates/temper-server/src/observe/mod.rs:256` - - `crates/temper-server/src/router.rs:88` - - `crates/temper-mcp/src/lib.rs:881` - - Impact: increases navigation and review friction. - -## Refactor Program - -## Phase 1 (Quick Wins, low risk) - -1. Extract OData shared helpers from `dispatch.rs`. - - Add helpers: - - `resolve_entity_context(...)` - - `guard_write_access(...)` - - `load_entity_or_404(...)` - - `shape_entity_response(...)` - - Goal: reduce `dispatch.rs` by 200-300 LOC without changing behavior. - -2. Introduce backend dispatcher helper in persistence. - - Centralize backend selection into one internal function. - - Keep explicit Redis ephemeral errors, but remove repeated branch scaffolding. - - Goal: reduce `state/persistence.rs` by 80-120 LOC. - -3. Move large inline tests into dedicated integration/unit test files. - - Split `observe/mod.rs`, `router.rs`, and `temper-mcp/src/lib.rs` test sections. - - Goal: smaller production files and clearer module boundaries. - -## Phase 2 (Semantic Alignment) - -1. Make SMT status semantics explicit. - - Add symbolic status variable. - - Encode `StateIn` and transition constraints using status membership. - - Align guard encoding with `stateright_impl` semantics. - -2. Make list semantics explicit in SMT. - - Keep current bounded abstraction but encode when value-sensitive checks are approximated. - - Option A: model membership as symbolic finite set. - - Option B: declare approximation mode and downgrade assertion confidence. - -3. Unify guard/effect semantics in one shared adapter. - - Create shared evaluator/normalizer module used by both Stateright and SMT pipelines. - - Remove duplicate logic branches in `stateright_impl.rs` and `smt.rs`. - -## Phase 3 (Crate Boundary Cleanup) - -1. `temper-server`: split into focused OData modules. - - `odata/read.rs` - - `odata/write.rs` - - `odata/bindings.rs` - - `odata/response.rs` - -2. `temper-mcp`: split monolith `lib.rs`. - - `protocol.rs` (JSON-RPC framing) - - `tools.rs` (tool defs) - - `runtime.rs` (dispatch and HTTP bridge) - - `convert.rs` (Monty/JSON transforms) - -3. `temper-platform` tests: shared fixtures/harness. - - Consolidate repeated setup into `tests/common/`. - - Reduce duplication in large system tests. - -## Coding Rules to Lock In - -1. No silent fallback success for missing implementations. - - Return explicit capability/unsupported errors. - -2. One semantic source of truth for guards/effects. - - If symbolic execution approximates, it must be explicit in API/result fields. - -3. Max target file size for production modules. - - Soft limit: 400 LOC. - - Hard review warning above 600 LOC. - -## Execution Order - -1. Phase 1.1 OData helper extraction. -2. Phase 1.2 persistence backend dispatch extraction. -3. Phase 1.3 test splitting. -4. Phase 2 SMT semantic alignment. -5. Phase 3 crate/module boundary reorganization. - -## Expected Outcome - -- ~10-20% reduction in core server module LOC. -- Lower duplication in handler and persistence pathways. -- Clearer behavior guarantees between runtime, Stateright, and SMT. -- Faster onboarding for contributors via smaller, purpose-focused modules. - -## Completion Status (2026-02-24) - -### Phase 1 (Quick Wins) - -- [x] OData helper extraction and split from `dispatch.rs`. - - Completed via `crates/temper-server/src/odata/{read,write,common}.rs`. - - Additional follow-up split completed in Phase 3 (`bindings.rs`, `response.rs`). -- [x] Persistence backend dispatch extraction. - - Completed in `crates/temper-server/src/state/persistence.rs` via centralized metadata backend selection. -- [x] Large inline test extraction. - - Completed for: - - `crates/temper-server/src/router.rs` → `router_tests.rs` - - `crates/temper-server/src/observe/mod.rs` → `mod_tests.rs` - - `crates/temper-mcp/src/lib.rs` → `lib_tests.rs` - -### Phase 2 (Semantic Alignment) - -- [x] SMT status semantics explicit modeling. - - `ModelGuard::StateIn` now uses symbolic status membership (not non-empty shortcuts). -- [x] List semantics alignment with explicit approximation disclosure. - - `ModelGuard::ListContains` now uses exact bounded slot semantics in SMT. - - Conflicting `ListContains` guards are rejected under tight bounds (e.g. max list size = 1). -- [x] Shared guard/effect semantics adapter. - - `crates/temper-verify/src/model/semantics.rs` is now the shared concrete semantics source. - - Stateright and SMT paths consume shared guard traversal/evaluation utilities. - -### Phase 3 (Crate Boundary Cleanup) - -- [x] `temper-server` OData focused modules. - - Completed with: - - `odata/read.rs` - - `odata/write.rs` - - `odata/bindings.rs` - - `odata/response.rs` -- [x] `temper-mcp` monolith split. - - Completed with: - - `protocol.rs` (JSON-RPC framing) - - `runtime.rs` (stdio runtime loop + sandbox execution orchestration) - - `tools.rs` (Temper method dispatch + HTTP bridge) - - `convert.rs` (Monty/JSON conversion) - - `sandbox.rs` and `spec_loader.rs` retained as focused support modules -- [x] `temper-platform` test harness consolidation. - - Completed with shared fixtures in `crates/temper-platform/tests/common/`. - - High-duplication setup moved to shared helpers (`http`, `platform`, `specs`, `dst`). - -### Post-Refactor Snapshot (selected files) - -- `crates/temper-server/src/odata/write.rs`: **484 LOC** (was 617 after initial split) -- `crates/temper-mcp/src/lib.rs`: **37 LOC** (runtime/tool internals extracted) -- `crates/temper-platform/tests/system_entity_dst.rs`: **736 LOC** (was 1057) diff --git a/docs/proofs/2026-04-28-idempotent-spec-persistence-local-e2e.md b/docs/proofs/2026-04-28-idempotent-spec-persistence-local-e2e.md deleted file mode 100644 index 2c3053293..000000000 --- a/docs/proofs/2026-04-28-idempotent-spec-persistence-local-e2e.md +++ /dev/null @@ -1,63 +0,0 @@ -# 2026-04-28 Idempotent Spec Persistence Local E2E - -## Build and Tests - -- `cargo test -p temper-store-turso --lib -- --nocapture` - - Result: 41 passed. -- `cargo test -p temper-platform test_hashes_requiring_persistence_skip_cached_verified_specs -- --nocapture` - - Result: passed. -- `cargo test -p temper-cli -- --nocapture` - - Result: 44 passed. -- `cargo check -p temper-cli -p temper-platform -p temper-store-turso` - - Result: passed. -- `cargo build -p temper-cli` - - Result: passed. - -## Live E2E - -Started `temper serve` twice against the same file-backed Turso/libSQL database -with an isolated home directory: - -```bash -HOME=/tmp/temper-spec-e2e-home \ -XDG_DATA_HOME=/tmp/temper-spec-e2e-xdg \ -TURSO_URL=file:/tmp/temper-spec-e2e-clean.db \ -RUST_LOG=error \ -./target/debug/temper serve \ - --storage turso \ - --no-observe \ - --app pipeline=docs/examples/pipeline-specs \ - --port 3123 -``` - -First boot: - -- `/healthz`: 200 -- Spec counts: - - `default|8|8|8` - - `pipeline|12|12|12` - - `temper-system|13|13|13` -- Snapshot saved from: - - `tenant` - - `entity_type` - - `updated_at` - - `version` - - `verified` - - `committed` - - `content_hash` - -Second boot: - -- `/healthz`: 200 -- Log evidence: `[verify] Skipped 4 unchanged verified specs for tenant pipeline` -- Spec counts: - - `default|8|8|8` - - `pipeline|12|12|12` - - `temper-system|13|13|13` -- `diff -u /tmp/temper-spec-e2e-before.tsv /tmp/temper-spec-e2e-after.tsv` - returned no diff. - -## Result - -Warm boot with unchanged verified specs did not rewrite any persisted spec rows, -including `updated_at`, `version`, `verified`, `committed`, and `content_hash`. diff --git a/docs/runbooks/latency-observability-release.md b/docs/runbooks/latency-observability-release.md deleted file mode 100644 index 255015021..000000000 --- a/docs/runbooks/latency-observability-release.md +++ /dev/null @@ -1,377 +0,0 @@ -# Temper Latency Observability Release Runbook - -Status: OBS-001 through OBS-005 complete locally; PERF-001 measurement prep in progress -Owner: Temper/OpenPaw operator -Living dashboard: `docs/temper-latency-observability-report.html` - -## Purpose - -This runbook turns the local latency/observability repair slices into a -repeatable release. The goal is not "merge code and hope Datadog lights up"; -the goal is to prove that every new measurement surface is visible, current, -and useful before using it to drive performance work. - -## Repositories And Order - -Use the main-based worktrees created for this program: - -- Temper: - `/Users/seshendranalla/Development/temper-worktrees/latency-observability-program` -- TemperPaw: - `/Users/seshendranalla/Development/temperpaw-worktrees/latency-observability-program` - -Release order: - -1. Merge and deploy the Temper runtime/store instrumentation. -2. Run the DBM setup SQL against staging/live Postgres. -3. Deploy the TemperPaw Datadog dashboard and monitor config. -4. Enable guarded runtime flags and run live verification. -5. Record links/screenshots and status in the living dashboard. - -## Local Verification Before PR - -Fast package consistency check: - -```sh -scripts/verify-latency-observability-package.sh quick -``` - -Run these in the Temper worktree: - -```sh -cargo fmt --check -cargo check -p temper-cli -cargo test -p temper-server profiling::tests --lib -- --nocapture -cargo test -p temper-store-postgres metrics --lib -- --nocapture -cargo test -p temper-observe otel --lib -- --nocapture -cargo test -p temper-authz -- --nocapture -cargo check -p temper-server -cargo test -p temper-server query_projection_metrics --lib -- --nocapture -cargo test -p temper-server odata::read_support --lib -- --nocapture -cargo test -p temper-server --test query_projection_backfill -- --nocapture -cargo test -p temper-store-turso load_entity_catalog_rows_preserves_projected_fields_json --lib -- --nocapture -cargo test -p temper-store-turso query_projection_catalog_preserves_projected_fields --lib -- --nocapture -cargo test -p temper-store-turso export_query_projections_returns_all_fields_for_migration --lib -- --nocapture -git diff --check -``` - -Run these in the TemperPaw worktree: - -```sh -python3 -m json.tool dd-dashboards/temperpaw-overview.json >/tmp/temperpaw-overview.json.check -python3 -m json.tool dd-monitors/temperpaw-monitors.json >/tmp/temperpaw-monitors.json.check -git diff --check -``` - -Or run the focused full preflight, which executes the static package checks, -the focused Temper checks above, and the TemperPaw JSON/diff checks: - -```sh -scripts/verify-latency-observability-package.sh full -``` - -Before committing Temper code, run the required project reviews described in -`docs/HARNESS.md` and `AGENTS.md`: DST compliance review for sim-visible code -and code-quality review for the full change set. - -## PR Split - -Use two PRs so runtime behavior and Datadog configuration can be reviewed and -rolled back independently: - -1. **Temper runtime/store observability PR** - - ADR-0081, ADR-0082, ADR-0083, and ADR-0084. - - Profiler freshness/capture/upload metrics and continuous profiler gate. - - Postgres app-side pool/transaction/projection metrics. - - Projection queue/backfill/shadow/replay-parity metrics. - - Turso `entity_catalog.fields` preservation and migration. - - Trace sampler decision/config/rate metrics and reaction fanout summary - span fields. - - Cedar/AuthZ canonical counter cleanup, millisecond duration metric, phase - duration metrics, and request-shape histograms. - - DBM setup SQL and runbooks. - -2. **TemperPaw Datadog config PR** - - Dashboard widgets for projection correctness, replay parity, Postgres, - dispatch, WASM, blob, Monty, session tail latency, and trace budgets. - - Monitors for drift/errors and p95/p99 regressions. - - Monitors for missing sampler metrics, unusual delegated span volume, and - disabled background trace budgeting. - - AuthZ phase dashboard widgets and monitors for Cedar max-duration fallback - and error-path phase instrumentation. - - The request-latency monitor must use `p95:temper_dispatch_ask_latency_ms`, - not an average. - -Merge Temper first. Merge/deploy the Datadog config after the runtime emits the -new metrics in at least staging, otherwise the config can validate but not -prove usefulness. - -## Runtime Flags - -Profiler gates: - -```sh -TEMPER_PROFILING_ENABLED=true -TEMPER_PROFILING_CONTINUOUS=true -TEMPER_PROFILING_AUTO_UPLOAD=true -``` - -Use conservative initial profiler windows/intervals unless the deployment -already has an accepted profile cost budget. Compare the in-process profiler -signal with Datadog `ddprof` before deciding which path is the long-term Rust -profiling standard. - -Projection shadow checks: - -```sh -TEMPER_ODATA_CATALOG_SHADOW_READ_EVERY=1000 -``` - -Start with a low sample rate. Increase only after Datadog shows no routine -shadow drift/errors and the additional actor reads do not affect request -latency. - -Catalog-fast reads: - -```sh -TEMPER_ODATA_CATALOG_FAST_READ=false -``` - -Keep catalog-fast reads off for broad rollout until replay parity is clean on -deployed data. Enable per environment only after a clean parity window is -recorded in the living dashboard. - -Trace budget: - -```sh -TEMPER_TRACE_WASM_AUX_SAMPLE_PCT=5 -TEMPER_TRACE_DISPATCH_BACKGROUND_SAMPLE_PCT=25 -``` - -Raise either value temporarily during an incident when full child-span detail -is more important than trace volume. Restore conservative values before normal -traffic if high-fanout traces approach the 100k-span failure mode observed in -production. - -## DBM Repair - -Run the DBM setup SQL in each monitored Postgres database: - -```sh -psql "$DATABASE_URL" -f scripts/datadog-postgres-dbm-setup.sql -``` - -If the DBM Agent uses a role other than the default `datadog`, pass it -explicitly. Railway production on May 15, 2026 used the -`datadog-postgres-agent` service with `PGUSER=postgres`: - -```sh -psql "$DATABASE_URL" -v dbm_agent_role=postgres -f scripts/datadog-postgres-dbm-setup.sql -``` - -Then follow `docs/runbooks/datadog-postgres-dbm.md` to validate: - -- `datadog.pg_stat_activity()`, -- `datadog.pg_stat_statements()`, -- `datadog.explain_statement(TEXT)`, -- no routine `plan.collection_errors:invalid_schema` on hot Temper queries. - -## Deploy Datadog Config - -Railway production discovery on May 14, 2026: - -- Railway project: `openpaw-seshendranalla` -- Railway environment: `production` -- Railway service: `openpaw` -- Public domains: `https://openpaw-production.up.railway.app` and `https://temperpaw.katagami.ai` -- Runtime Datadog tag: `DD_SERVICE=temperpaw` -- Current production build variables observed: `BUILD_VERSION=sha-6c352c5` - -The Railway service is named `openpaw`, but Datadog dashboard and monitor -queries must target `service:temperpaw` unless production intentionally changes -`DD_SERVICE`. The package preflight fails if TemperPaw Datadog config regresses -to `service:openpaw`. - -Requires `DD_API_KEY`, `DD_APP_KEY`, and optionally `DD_SITE`. These are present -in the Railway production environment for the `openpaw` service as of the -May 14, 2026 check. Prefer `railway run` for dry-runs so the local shell does -not need secret exports. - -Dry-run monitors first: - -```sh -railway run --service openpaw --environment production -- python3 scripts/deploy_monitors.py --dry-run -``` - -May 14, 2026 dry-run result: the Railway-injected Datadog credentials worked -and the script would create all 62 source-of-truth monitors under the -`team:openpaw` monitor tag scope. On May 15, 2026, the live source set was -expanded to 66 monitors with four standard APM coverage monitors, and the -Datadog deploy created or updated the full set. Before running future -`--reconcile` deployments, confirm that older dashboards or monitors are not -still intentionally managed under different tags or names. - -Deploy dashboard: - -```sh -railway run --service openpaw --environment production -- python3 scripts/deploy_dashboard.py dd-dashboards/temperpaw-overview.json -``` - -Deploy monitors: - -```sh -railway run --service openpaw --environment production -- python3 scripts/deploy_monitors.py -``` - -Record the dashboard URL and newly created/updated monitor IDs in the living -dashboard. - -## Railway Health Probe - -Production health probe on May 14, 2026: - -```sh -curl -fsS https://openpaw-production.up.railway.app/readyz -curl -i -fsS https://openpaw-production.up.railway.app/healthz -``` - -Observed result: `/readyz` returned JSON with `status:"ready"` and Discord -connected/configured, while `/healthz` returned HTTP 200 with an empty body. -The custom domain `temperpaw.katagami.ai` did not resolve from this local -environment during the check, so use the Railway domain for immediate smoke -tests unless DNS is repaired or verified from another network. - -## Read-Only Datadog Snapshot - -Use Railway-injected Datadog credentials for read-only metric snapshots: - -```sh -railway run --service openpaw --environment production -- python3 scripts/read_datadog_snapshot.py -``` - -May 14, 2026 result for `service:temperpaw`: - -- `temper_up` returned one live series over 1h/24h with latest value `1`. -- `temper_cedar_evaluations_total` returned counted traffic: 8,876 over 1h and - 17,005 over 24h in the query output. -- `temper_cedar_evaluation_duration` returned live averages; the 24h max - sampled value was about 122 ms. -- `p95:temper_dispatch_ask_latency_ms`, new Postgres p95 metrics, projection - update errors, and profiler upload/error metrics returned no series. - -Interpretation: the deployed service and Cedar metrics are live under -`service:temperpaw`, but profiler and several tail/correctness surfaces remain -measurement gaps until this package is deployed and Datadog percentile -aggregations are configured. - -## Datadog Metric Configuration - -Before p95/p99 gates are treated as live, enable percentile aggregations for -the distribution metrics that drive the latency program: - -- `temper_cedar_evaluation_duration` -- `temper_cedar_evaluation_duration_ms` -- `temper_cedar_evaluation_phase_duration_ms` -- `temper_dispatch_ask_latency_ms` -- `temper_query_projection_update_duration_ms` -- `temper_query_projection_update_end_to_end_duration_ms` -- `temper_postgres_pool_acquire_duration_ms` -- `temper_postgres_transaction_duration_ms` - -This is a required Datadog configuration step. The local code can emit -histograms, but Datadog must be configured to expose p95/p99 aggregations. If a -percentile widget or monitor is No Data while avg/max is live, treat it as a -measurement gap, not a product latency conclusion. - -The TemperPaw Datadog config repository keeps this step repeatable: - -```sh -railway run --service openpaw --environment production -- python3 scripts/configure_metric_percentiles.py -railway run --service openpaw --environment production -- python3 scripts/configure_metric_percentiles.py --apply -``` - -The script manages a bounded list of latency distribution metrics and excludes -known high-cardinality tags such as `session_id` from the queryable tag list. -Datadog cannot configure metrics that have never emitted, so missing future -runtime metrics are skipped and must be re-run after the Temper runtime PR is -deployed. - -## Live Verification - -Verify every signal in Datadog after deployment. Passing local tests is not a -substitute for this table. - -| Area | Required live proof | -| --- | --- | -| Profiler | Fresh `datadog.profiling.rust.profiles_uploaded` for `service:temperpaw`, upload errors near zero, flamegraph visible for the deployed version. | -| AuthZ | `temper_cedar_evaluation_duration_ms`, `temper_cedar_evaluation_phase_duration_ms`, and `temper_cedar_request_attribute_count` visible; p95/p99 aggregations enabled; duplicate `temper-authz` counter scope absent from new versions. | -| DBM | Hot `entity_catalog`, `entity_field_index`, and event append/query signatures have explain plans without `invalid_schema`. | -| Postgres app metrics | p50/p95/p99 visible for pool acquire and transaction duration by `operation` and `outcome`. | -| Projection updates | Queue wait, update duration, end-to-end duration, applied sequence, and update errors visible by `source`, `entity_type`, and operation. | -| Shadow checks | Shadow check match/drift/error rates visible; no unexplained drift before increasing sampling or enabling fast reads. | -| Replay parity | Replay parity match/drift/error rates visible; parity clean before projection write-amplification changes. | -| Turso catalog | Fast-read catalog rows preserve JSON types and parity stays clean after migration/backfill. | -| Trace budget | `temper_trace_sampler_*` metrics visible; sampler configured-rule gauges are nonzero; high-fanout smoke traces retain roots and reaction summaries without routine 100k-span child traces. | -| Dispatch | `temper_dispatch_ask_latency_ms` p95/p99 visible and request-latency monitor evaluates percentile data. | -| WASM | Invocation and host HTTP duration p95/p99 visible by trigger/call kind. | -| Blob/Monty | Blob I/O, blob transport, and Monty wait p95/p99 visible. | -| Sessions | Context prepare and session phase p95/p99 visible by phase/result where applicable. | - -## Live End-To-End Runs - -Run at least one staging and one live smoke path that exercises: - -1. entity creation, -2. entity action dispatch, -3. OData collection read, -4. projection update and backfill/parity signal, -5. a session/tool/WASM path if available in the environment, -6. a blob/content path if available in the environment. - -For the existing Temper agent proof harness, run: - -```sh -python3 scripts/temper_agent_e2e_proof.py -``` - -The script writes its markdown proof to `.proof/temper-agent-e2e-proof.md` and -JSON artifacts under `.tmp/temper-agent-proof/artifacts`. Attach or link the -relevant proof summary from the living dashboard after staging/live execution. - -For each run, record: - -- exact environment, -- deployed git SHA/version, -- start/end time, -- request/trace links, -- Datadog dashboard screenshot or URL, -- any monitor state changes, -- pass/fail result. - -## Rollback - -Runtime rollback: - -- Disable profiler continuous upload flags. -- Set `TEMPER_ODATA_CATALOG_SHADOW_READ_EVERY=0`. -- Keep `TEMPER_ODATA_CATALOG_FAST_READ=false`. -- Roll back runtime deployment if metrics cause unexpected overhead. - -Datadog config rollback: - -- Revert the TemperPaw Datadog PR and run the dashboard/monitor deploy scripts. -- Prefer disabling noisy monitors over deleting evidence while investigating - threshold tuning. - -## Done Criteria - -This release package is done only when: - -- both PRs are merged, -- runtime is deployed, -- DBM repair is applied and verified, -- Datadog dashboard/monitors are deployed, -- live e2e runs pass, -- fresh Datadog evidence is recorded in - `docs/temper-latency-observability-report.html`, -- the living dashboard names any remaining optimization work with evidence. diff --git a/docs/temper-latency-observability-report.html b/docs/temper-latency-observability-report.html deleted file mode 100644 index 03203956e..000000000 --- a/docs/temper-latency-observability-report.html +++ /dev/null @@ -1,3283 +0,0 @@ - - - - - - Temper Latency and Observability Acceleration Report - - - - - - -
- - -
-
-
Vapor Terminal OVA Console / May 15, 2026
-

Temper Latency and Observability Acceleration Report

-

- This report combines a whole-repository architecture pass with live Datadog APM, DBM, metrics, - monitors, dashboards, and profiling signals. It is built as an interactive console: click any card, - table row, chip, task, or diagram for a detail drawer. -

-
- -
- -
- Click almost anywhere for more detail. Diagram toolbars can focus large diagrams for easier reading. - -
- -
-
-
live / program control
-

Program Progress

-
-
-
-

Current status

-

- This HTML file is now the living dashboard for the Temper latency and observability acceleration goal. - It must be updated as milestones move, evidence arrives, tasks split, ADRs land, PRs open, PRs merge, - live runs pass, and deployment completes. -

-
-
98%
-
-
-
PhaseThe measurement and observability foundation is live, and eight shipped slices have moved through the full lane: ADR-0088/File $value fast path, ADR-0089/AuthZ candidate filtering, PERF-001B/ADR-0090 AuthZ candidate indexing, PERF-002/ADR-0091 projection diff-index upserts, PERF-005B/ADR-0092 bounded background File reactions, PERF-005C/ADR-0093 native blob/object-store observability, PERF-005D/ADR-0094 native-first File blob reads, and PERF-003/ADR-0095 projection transaction fast path. PERF-003 is deployed and measured; PERF-003B/ADR-0096 is now active to attack the remaining changed-projection SQL shape before moving to broader workflow executor work.
-
BranchesMerged: Temper codex/latency-observability-program; TemperPaw codex/latency-observability-program; final dashboard evidence codex/latency-observability-final-report; Mac mini/auth checkpoint branches; corrected production e2e report codex/latency-observability-auth-timeout-20260515-1324; File fast-path branches; AuthZ candidate-filter branches; PERF-001B branches; PERF-002 branches; PERF-005B branches; PERF-005C branches; PERF-005D Temper/TemperPaw branches; PERF-003 Temper branch codex/latency-db-transaction-shape-20260517; and PERF-003 TemperPaw rollout branch codex/bump-temper-db-transaction-shape-20260517.
-
WorktreesCurrent report worktree /Users/seshendranalla/Development/temper-worktrees/latency-observability-auth-timeout-20260515-1324. PERF-003 implementation worktree /Users/seshendranalla/Development/temper-worktrees/latency-db-transaction-shape-20260517 produced merged Temper commit 6439a8a0be134ffa37933701fb8fca140121d44d. PERF-003 rollout worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-db-transaction-shape-20260517 produced merged TemperPaw commit c16e0201c1490e0496f4964f5c72b704bb8cd216. Active PERF-003B worktree /Users/seshendranalla/Development/temper-worktrees/latency-session-projection-shape-20260517 is branched from latest main at report merge 37e269ed.
-
PRsAll PERF-003 PR gates are complete. Temper PR nerdsane/temper#245 passed CI run 25979739482 and merged as 6439a8a0. TemperPaw rollout PR nerdsane/temperpaw#276 passed PR CI run 25980160694 and merged as c16e0201. Dashboard evidence PR nerdsane/temper#247 passed CI run 25981859177 and merged as 37e269ed. PERF-003B has ADR/code/tests locally and is not opened as a PR yet.
-
RailwayCLI authentication is confirmed for the correct account. Project openpaw-seshendranalla, environment production, service openpaw. Current deployment 6e42424f-7ad4-4577-b127-9e3a0a36abeb succeeded from Dockerfile.deploy with image tag ghcr.io/nerdsane/temperpaw:sha-c16e020 and image digest sha256:9c8a844fdfe52a201adea5755a3d5d61c498d8ba6cdc84c2f64a151d3bf27c89. Authenticated /paw/version reports sha-c16e020 / c16e0201c1490e0496f4964f5c72b704bb8cd216; /readyz is ready and Discord is connected. A transient version-variable mismatch and accidental config/source deploy were corrected before final proof traffic.
-
Datadog tagProduction exports DD_SERVICE=temperpaw and current version tag DD_VERSION=c16e0201c1490e0496f4964f5c72b704bb8cd216. Latest 12-hour c16 query shows Session/background_dispatch projection p95 bins still peaking at 386.2 ms, while pool acquire remains about 1.4-10.1 ms. Reconciliation counters now show the true next target: diff=408, insert=64, skipped_unchanged=26, with hot bins averaging about 25-28 indexed fields per projection update.
-
Production checksLatest live proof perf-003-projection-fast-path-proof-20260517041730.md ran against https://openpaw-production.up.railway.app on deployment 6e42424f. It created File fl-019e3427-4b90-7801-be94-d4af896aa315, wrote three versions through $value with client wall clocks 349.2 ms, 366.5 ms, and 344.8 ms, verified exact readback after each write, verified VersionCount=3, verified FileVersions transitioned Superseded, Superseded, Current, and confirmed indexed DB rows directly in production Postgres.
-
Current slicePERF-003B/ADR-0096 is in local implementation. It keeps the ADR-0095 catalog serialization and no-op fast path, but replaces changed projection row-by-row entity_field_index delete/upsert loops with one anti-join delete plus one batched INSERT ... SELECT FROM unnest(...) ON CONFLICT. The goal is to reduce lock-held SQL round trips for the measured diff-heavy Session residual without changing OData semantics or projection correctness.
-
Current rolloutRollout complete, proof complete, but measurement continues. Post-deploy projection update metrics show File and FileVersion background_dispatch p95 buckets around 50-73 ms, while Session/background_dispatch remains the notable residual at about 245-386 ms in the proof window. Profiling is packaged but not continuously useful yet: TEMPER_DDPROF_ENABLED=false, ddprof is present, Railway exposes perf_event_paranoid=3, and CAP_PERFMON=false.
-
Next task orderFinish PERF-003B local/full gates, open and merge the Temper PR, bump TemperPaw, deploy, then rerun live File/OData plus Session-oriented proof traffic and compare current-version Session/background_dispatch, query_projection_upsert, reconciliation-path, DBM, and projection correctness signals. After that, choose between continuous replay/shadow activation, profiler canary hardening, startup/cutover cost, workflow executor, or event-append sequence shape.
-
Review gatesPERF-003 code-quality review marker passed with no findings. PERF-003B code-quality review marker also passed locally with no findings; DST review is not required because the changed code remains confined to temper-store-postgres, not temper-runtime, temper-jit, or temper-server. The full pre-push gate is still pending for PERF-003B.
-
Dashboarddocs/temper-latency-observability-report.html
-
ADRsdocs/adrs/0081-latency-observability-acceleration-program.md; docs/adrs/0082-projection-correctness-observability.md; docs/adrs/0083-trace-budget-and-fanout-summarization.md; docs/adrs/0084-authz-latency-phase-instrumentation.md; docs/adrs/0088-native-file-value-write-fast-path.md; docs/adrs/0089-authz-policy-candidate-index.md; PERF-001B ADR /Users/seshendranalla/Development/temper-worktrees/latency-authz-candidate-index-20260516/docs/adrs/0090-authz-candidate-selection-index.md; PERF-002 ADR /Users/seshendranalla/Development/temper-worktrees/latency-projection-diff-index-20260516/docs/adrs/0091-query-projection-diff-index-upserts.md; PERF-005B ADR /Users/seshendranalla/Development/temper-worktrees/latency-file-value-residual-20260516/docs/adrs/0092-bounded-background-file-reactions.md; PERF-005C ADR /Users/seshendranalla/Development/temper-worktrees/latency-blob-transport-observability-20260516/docs/adrs/0093-native-blob-transport-observability.md; PERF-005D ADR /Users/seshendranalla/Development/temper-worktrees/latency-file-blob-read-key-order-20260517/docs/adrs/0094-native-file-blob-read-key-order.md; PERF-003 ADR /Users/seshendranalla/Development/temper-worktrees/latency-db-transaction-shape-20260517/docs/adrs/0095-projection-transaction-fast-path.md; active PERF-003B ADR /Users/seshendranalla/Development/temper-worktrees/latency-session-projection-shape-20260517/docs/adrs/0096-set-based-projection-index-reconciliation.md
-
Runbooksdocs/runbooks/datadog-postgres-dbm.md; docs/runbooks/latency-observability-release.md; scripts/datadog-postgres-dbm-setup.sql
-
Local checksscripts/verify-latency-observability-package.sh full passed after Railway access and again after the readability-safe module split. File fast-path, AuthZ, PERF-001B, PERF-002, PERF-005B, PERF-005C, PERF-005D, and PERF-003 have all passed the relevant local checks, GitHub CI, TemperPaw rollout checks, main CI, Docker, Railway deployment, live proof, and Datadog proof for their targeted behavior. PERF-003B currently has green cargo check -p temper-store-postgres, focused upsert_query_projection tests, full cargo test -p temper-store-postgres, cargo clippy -p temper-store-postgres --all-targets -- -D warnings, rustfmt check, and git diff --check; full workspace/pre-push is pending.
-
TrackingThread goal was resumed by the user's go on instruction; Temper MCP issue tracking is unavailable in this session.
-
Program splitDone or mostly done: architecture study, ADRs, living dashboard, Datadog dashboards/monitors, targeted profiling path, DBM repair, runtime percentile metrics, AuthZ phase measurement, projection correctness metrics, trace budget instrumentation, deploy proof, representative production e2e proof, File $value fast path, first AuthZ candidate-filter deployment, PERF-001B candidate-index deployment/proof, PERF-002 projection diff-index deployment/proof, PERF-005B/005C/005D full rollout proof, and PERF-003 full rollout proof. In progress: PERF-003B set-based changed projection reconciliation. Remaining work is deliberately narrower and evidence-driven, not a mystery bucket.
-
Finish lineThe goal remains open because the whole-system latency program is broader than one shipped projection transaction slice. The next phase should not start with another blind optimization; it should gather the longer c16 window, turn replay/shadow correctness checks into continuous production evidence, make profiler activation useful on Railway or document the platform limit, and then implement the highest residual latency item with the same ADR, worktree, tests, PR, merge, deploy, live e2e, Datadog proof, and correctness proof discipline.
-
-
-
-

Status log

-
-
- -

Created main-based worktree codex/latency-observability-program for all tracked program setup edits.

-
-
- -

Promoted this report from a static diagnosis into the living program dashboard with a progress bar, milestone tracker, task board, and evidence target.

-
-
- -

Created ADR-0081 to define the measurement-first program, worktree discipline, correctness gates, PR/merge/deploy finish line, and dashboard-as-source-of-status requirement.

-
-
- -

Ran live Datadog profiling metric check. Metrics exist for datadog.profiling.rust.profiles_uploaded and datadog.profiling.rust.upload_errors, but the last 24 hours for service:temperpaw only returned stale-looking CPU upload points with value 0 and no useful nonzero production profile series.

-
-
- -

Ran live Datadog DBM check. temperpaw-postgres is healthy and low-latency overall, but plan search returned 2,408 plan samples with plan.collection_errors / invalid_schema, so query plan evidence is present but not usable yet.

-
-
- -

Implemented OBS-001 local repair in crates/temper-server/src/profiling.rs and crates/temper-cli/src/serve/mod.rs: explicit profiler config/freshness/capture metrics plus opt-in continuous CPU profile capture gated by TEMPER_PROFILING_ENABLED, TEMPER_PROFILING_CONTINUOUS, and TEMPER_PROFILING_AUTO_UPLOAD.

-
-
- -

Verified the local profiler repair: cargo check -p temper-cli passed; cargo test -p temper-server profiling::tests --lib -- --nocapture passed 4 profiler tests; cargo fmt --check and git diff --check passed. Remaining warnings are pre-existing dead-code warnings in crates/temper-server/src/authz/helpers.rs.

-
-
- -

Rendered this report with Playwright at #progress, confirmed the progress surface loads, and fixed section anchor spacing so sticky navigation no longer covers headings on direct links.

-
-
- -

Drafted OBS-002 DBM repair package: scripts/datadog-postgres-dbm-setup.sql creates the required Datadog schema/functions/grants/search path, and docs/runbooks/datadog-postgres-dbm.md gives operator validation steps for the current invalid_schema failure.

-
-
- -

Added OBS-002 app-side PostgreSQL timing metrics in crates/temper-store-postgres: pool acquire, transaction begin, transaction commit, end-to-end transaction duration, operation outcomes, projection indexed field count, and oversized-field skips for event append and projection write transactions.

-
-
- -

Verified the Postgres metrics slice: cargo test -p temper-store-postgres metrics --lib -- --nocapture, cargo check -p temper-cli, cargo fmt --check, and git diff --check passed.

-
-
- -

Created ADR-0082 for projection correctness observability: projection source tagging, applied sequence visibility, background queue delay, backfill coverage, and a follow-up drift/parity gate before projection write optimization.

-
-
- -

Implemented OBS-003 first metrics slice in crates/temper-server/src/query_projection_metrics.rs, state/dispatch/effects.rs, state/entity_ops.rs, and state/projection_backfill.rs. It adds source-tagged projection update metrics, queue wait, end-to-end duration, applied sequence gauges, and backfill coverage/replay-event metrics.

-
-
- -

Verified OBS-003 locally: cargo test -p temper-server query_projection_metrics --lib -- --nocapture, cargo check -p temper-cli, cargo fmt --check, and git diff --check passed. Remaining warnings are the same pre-existing authz helper dead-code warnings.

-
-
- -

Implemented the OBS-003 sampled projection drift hook in crates/temper-server/src/odata/read_support.rs. When TEMPER_ODATA_CATALOG_SHADOW_READ_EVERY is set above 0, catalog-fast OData hits deterministically sample background actor shadow reads and compare projected status, fields, and sequence against authoritative actor state.

-
-
- -

Verified the sampled drift hook locally: cargo test -p temper-server odata::read_support --lib -- --nocapture passed 9 tests, including stable sampling, drift classification, and sequence-gap direction tests. cargo check -p temper-cli, cargo fmt --check, and git diff --check passed.

-
-
- -

Created the TemperPaw main-based worktree at /Users/seshendranalla/Development/temperpaw-worktrees/latency-observability-program on branch codex/latency-observability-program for Datadog dashboard and monitor configuration.

-
-
- -

Added OBS-004 Datadog config coverage in TemperPaw: projection queue wait p95/p99, projection end-to-end p95/p99, applied sequence, backfill coverage/replay events, shadow check totals/sequence gap, Postgres pool acquire p95/p99, Postgres transaction p95/p99, and monitors for projection drift, shadow errors, projection queue/e2e regressions, backfill failures, and Postgres pool/transaction regressions.

-
-
- -

Verified TemperPaw Datadog config locally: python3 -m json.tool dd-dashboards/temperpaw-overview.json, python3 -m json.tool dd-monitors/temperpaw-monitors.json, and git diff --check passed in the TemperPaw worktree.

-
-
- -

Implemented the OBS-003 replay parity verifier in state/projection_backfill.rs and exposed it through ServerState::verify_query_projection_replay_parity. It rebuilds active entity state from the event journal, compares durable catalog rows, emits temper_query_projection_replay_parity_* metrics, and returns bounded drift examples without using entity IDs as metric tags.

-
-
- -

The parity test exposed a real Turso correctness issue: entity_catalog did not preserve the full projected fields JSON. Fixed Turso schema/migration/upsert/load/export paths so the catalog stores fields as a JSON text blob while entity_field_index remains the scalar filter index.

-
-
- -

Verified replay parity and Turso field preservation locally: cargo test -p temper-server --test query_projection_backfill -- --nocapture passed 4 tests, and focused temper-store-turso tests for catalog fields, schema migration, and projection export passed.

-
-
- -

Extended TemperPaw Datadog config with replay parity widgets and monitors for parity drift/errors. JSON validation and git diff --check pass locally; deployment was blocked on Datadog API keys at this point in the run.

-
-
- -

Expanded OBS-004 tail-latency coverage in TemperPaw Datadog config: dispatch p95/p99, WASM invocation p95/p99, WASM host HTTP p95/p99, blob I/O and remote transport p95/p99, Monty REPL wait p95/p99, context-prepare p95/p99, and session-phase p95/p99 widgets.

-
-
- -

Added or repaired percentile monitors for request latency, WASM invocation, WASM host HTTP, blob I/O, blob transport, Monty REPL wait, session context prepare, and session phase duration. TemperPaw dashboard/monitor JSON validation and git diff --check passed locally.

-
-
- -

Added docs/runbooks/latency-observability-release.md to define the two-PR release split, local verification commands, runtime flags, DBM repair, Datadog deployment commands, live proof matrix, e2e run requirements, rollback, and done criteria.

-
-
- -

Created ADR-0083 for trace budget and fanout summarization. The decision keeps canonical request roots, makes sampler decisions measurable, adds startup-tunable reduced-prefix sampling, and records bounded reaction fanout summaries on parent spans.

-
-
- -

Implemented OBS-005 local code in crates/temper-observe/src/otel.rs and crates/temper-server/src/trigger/dispatcher.rs: temper_trace_sampler_decisions_total, temper_trace_sampler_configured_rules, temper_trace_sampler_reduced_sample_rate_pct, TEMPER_TRACE_WASM_AUX_SAMPLE_PCT, TEMPER_TRACE_DISPATCH_BACKGROUND_SAMPLE_PCT, and reaction fanout fields.

-
-
- -

Verified OBS-005 locally: cargo test -p temper-observe otel --lib -- --nocapture passed 9 tests, and cargo check -p temper-server passed with only the known pre-existing authz helper dead-code warnings.

-
-
- -

Added OBS-005 Datadog config in TemperPaw: a Trace Budget dashboard group and monitors for missing sampler metrics, delegated span-volume spikes, and disabled dispatch-background trace budgeting. Dashboard JSON, monitor JSON, and git diff --check passed locally.

-
-
- -

Added scripts/verify-latency-observability-package.sh as the package preflight for OBS-001 through OBS-005. The quick mode verifies the living dashboard, ADRs, runbooks, DBM SQL, required metric/source strings, TemperPaw JSON, dashboard widgets, and monitors. Its first run exposed a missing e2e proof command in the release runbook; the runbook now names python3 scripts/temper_agent_e2e_proof.py.

-
-
- -

Verified the new preflight quick mode: scripts/verify-latency-observability-package.sh quick passed.

-
-
- -

Verified the full focused preflight: scripts/verify-latency-observability-package.sh full passed. It ran the static cross-repo package checks, cargo fmt --check, cargo check -p temper-cli, cargo check -p temper-server, profiler tests, OTEL sampler tests, Postgres metrics tests, projection metrics tests, OData shadow tests, replay parity integration tests, focused Turso projection tests, and TemperPaw git diff --check. The only warnings were the known pre-existing authz helper dead-code warnings.

-
-
- -

Used current Datadog context to check Cedar/AuthZ signals. temper_cedar_evaluations_total and temper_cedar_evaluation_duration are live, but temper_cedar_evaluation_duration has percentile aggregations disabled and temper_cedar_evaluations_total is emitted from both temper-authz and temper.authz instrumentation scopes.

-
-
- -

Created ADR-0084 for AuthZ latency phase instrumentation. The decision records one canonical Cedar counter scope, keeps the existing seconds metric, adds temper_cedar_evaluation_duration_ms, adds temper_cedar_evaluation_phase_duration_ms, and records bounded request-shape histograms with temper_cedar_request_attribute_count.

-
-
- -

Implemented the ADR-0084 local AuthZ metrics slice in crates/temper-authz: canonical allow/deny/error counter path, millisecond duration metric, bounded phase timers, request attribute-count histograms, and a focused invalid-action error-path test. cargo check -p temper-authz and cargo test -p temper-authz invalid_action_denies_through_instrumented_error_path -- --nocapture passed.

-
-
- -

Extended TemperPaw Datadog config with an AuthZ Phase Breakdown (ADR-0084) dashboard group, ms duration p95/p99 and avg/max fallback panels, phase duration panels, request-shape panels, a Cedar max-duration fallback monitor, and a phase-error monitor. Dashboard and monitor JSON parse locally.

-
-
- -

Verified the updated ADR-0084 package with scripts/verify-latency-observability-package.sh full. It passed package consistency checks, cargo fmt --check, cargo check -p temper-cli, cargo check -p temper-server, all 56 temper-authz tests, profiler tests, OTEL sampler tests, Postgres metrics tests, projection metrics tests, OData shadow tests, replay parity integration tests, focused Turso projection tests, and both worktree diff checks. The only warnings remain the known pre-existing authz helper dead-code warnings.

-
-
- -

Checked this report after the progress update: inline module JavaScript parses, the document contains the 49% progress marker, ADR-0084 link, and AuthZ phase metric strings, and both Temper/TemperPaw git diff --check commands pass. The in-app browser connection detected the open file tab, but its URL policy blocked a reload of the local file:// report, so latest visual refresh remained pending instead of bypassed.

-
-
- -

Next action at that point: prepare the PR split and deployment checklist. Live Datadog deployment was still blocked until API credentials and environment deployment access became available.

-
-
- -

Railway access is now available for the correct account. Production inspection found project openpaw-seshendranalla, environment production, Railway service openpaw, domains openpaw-production.up.railway.app and temperpaw.katagami.ai, Datadog API/app key variables present, and runtime tag DD_SERVICE=temperpaw.

-
-
- -

Retargeted TemperPaw dashboard and monitor queries from service:openpaw to service:temperpaw and added a package preflight guard so config cannot regress to the wrong service tag. Runtime deployment is still gated by PR/merge and by updating TemperPaw's pinned Temper revision after the Temper PR lands.

-
-
- -

Ran railway run --service openpaw --environment production -- python3 scripts/deploy_monitors.py --dry-run. The Datadog API call succeeded without exposing secrets and reported that all 62 source-of-truth monitors would be created under the team:openpaw tag scope.

-
-
- -

Probed production health through the Railway domain. https://openpaw-production.up.railway.app/readyz returned status:"ready" with Discord connected/configured, and /healthz returned HTTP 200. The custom domain temperpaw.katagami.ai did not resolve from this local environment during the check.

-
-
- -

Verified the in-app browser is on this report file and the title resolves to Temper Latency and Observability Acceleration Report. A reload of the local file:// URL was blocked by the browser security policy again, so latest visual refresh remains pending; static HTML JavaScript parsing, package preflight, JSON parsing, and diff checks passed instead.

-
-
- -

Added and verified scripts/read_datadog_snapshot.py in TemperPaw. Running it through Railway-injected credentials for service:temperpaw showed temper_up stayed at 1 over 24h, temper_cedar_evaluations_total returned 17,005 counted evaluations, and temper_cedar_evaluation_duration returned live averages with max around 122 ms. Dispatch p95, projection-update error, new Postgres p95, and profiler upload/error queries returned no series.

-
-
- -

Re-ran scripts/verify-latency-observability-package.sh full after the Railway/service-tag/snapshot-helper updates. It passed package consistency, Rust formatting/check/tests, focused projection/AuthZ/profiler/OTEL/Postgres/Turso tests, TemperPaw JSON validation, snapshot helper compilation, and both worktree git diff --check passes. The only warnings are the known pre-existing unused authz helper warnings.

-
-
- -

Ran the changed-file determinism hook loop after cleanup; sim-visible Rust files now pass. Independent DST review returned PASS for read_support.rs, profiling.rs, query_projection_metrics.rs, dispatch effects, entity ops, state, projection backfill, reaction dispatcher, and the query projection backfill test. Independent code-quality review returned PASS with no blocker findings.

-
-
- -

Temper pre-push exposed a maintainability regression rather than a runtime bug: four production files crossed the 500-line readability threshold and two temporary dead-code allowances increased the baseline. Fixed this by splitting catalog shadow checks, profiler metrics/tests, and projection replay parity into child modules; changed the staged governed mutation helper to a precise lint expectation; and removed redundant Turso schema test coverage.

-
-
- -

Re-ran local quality gates after the module split. scripts/verify-latency-observability-package.sh full, cargo check -p temper-server, cargo clippy -p temper-server --all-targets -- -D warnings, bash scripts/readability-ratchet.sh check .ci/readability-baseline.env, the changed/new sim-visible determinism hook loop, and scripts/verify-latency-observability-package.sh quick all pass. The verifier was updated so profiler upload and projection shadow-read checks follow the new module paths.

-
-
- -

Pushed the Temper branch after the repository pre-push hook passed all four gates: rustfmt, clippy, readability ratchet, and full cargo test --workspace. Created draft PRs: nerdsane/temper#229 for the runtime/report/ADR package and nerdsane/temperpaw#266 for Datadog dashboards, monitors, and snapshot tooling.

-
-
- -

Confirmed both draft PRs point at the pushed worktree heads: Temper 1ae2c3fe and TemperPaw ea86a6d5. Both worktrees are clean and tracking their pushed branches.

-
-
- -

Used live Datadog MCP with the correct Railway/Datadog account. Datadog services now visible for this program are openpaw, temperpaw, and temperpaw-postgres. Dashboard TemperPaw - Platform Overview exists and already contains profiler upload/error widgets.

-
-
- -

Datadog monitor coverage still reports missing service-level APM error, duration, and rate coverage for service:temperpaw, even though many domain-specific Temper/TemperPaw monitors exist. This confirms OBS-004 still needs Datadog config deployment plus coverage reconciliation, not just local JSON changes.

-
-
- -

Live APM over the latest two-hour window showed sustained HTTP traffic under service:temperpaw env:prod. Longest raw traces were mostly long-lived event streams with almost all time in idle_ns, so they are not server slowness. Excluding GET /observe/events/stream and GET /tdata/$events, the slowest observed paths were file $value writes at about 2.1-2.4 s, a WASM observability probe at about 1.3 s, policy creation around 0.4 s, and regular OData/list reads mostly in the single-digit to low-tens of milliseconds.

-
-
- -

Live custom metric check showed temper_dispatch_ask_latency_ms averaging about 15-21 ms with max buckets around 25-28 ms, and temper_cedar_evaluation_duration averaging about 35-59 ms with max buckets around 64 ms in the sampled window. Both are Datadog distribution metrics with is_percentiles_enabled:false; no temper* metric under service:temperpaw currently has percentile aggregation enabled.

-
-
- -

Live DBM is present but incomplete: activity rows and APM correlation are visible for temperpaw-postgres, and DBM found the instance suspicious only for transient latency spikes rather than sustained contention. Plan search returned 191 recent plan records, but plan payloads still contain invalid_schema collection errors, so explain-plan usefulness remains blocked until the DBM SQL/runbook repair is applied.

-
-
- -

DBM query-performance refresh separated database costs from application costs: entity_catalog batch lookup is high volume but usually sub-millisecond database latency, tenant-count scans average around 8.9 ms and touch about 3,846 shared blocks, field-index filtered reads are around tens of microseconds, and sampled entity_field_index upserts are also tens of microseconds with a single observed transaction-duration sample around 15.8 ms. The live slow file writes are therefore unlikely to be explained by database saturation alone.

-
-
- -

Deployed the TemperPaw Datadog dashboard through Railway. railway run --service openpaw --environment production -- python3 scripts/deploy_dashboard.py dd-dashboards/temperpaw-overview.json updated dashboard mn4-k3k-i66.

-
-
- -

Datadog rejected one monitor before deployment because default_zero(...) is incompatible with on_missing_data: resolve. Fixed [OpenPaw] Error Rate Spike to use on_missing_data: default and added validation plus 429 retry/backoff to scripts/deploy_monitors.py before retrying.

-
-
- -

Deployed the source monitor set through Railway. The first live deploy created 62 monitors; follow-up coverage passes added four standard APM monitors (APM Request Rate Missing id 283342070, APM HTTP 5xx Spike id 283342074, APM HTTP Duration p95 Regression id 283342076, and APM Error Rate Spike id 283345345) and updated the set to 66 monitors.

-
-
- -

Added repeatable Datadog percentile configuration in TemperPaw with scripts/configure_metric_percentiles.py. Applying it through Railway enabled percentiles for 18 currently live latency distributions including temper_dispatch_ask_latency_ms, temper_cedar_evaluation_duration, temper_session_phase_duration_ms, temper_wasm_invocation_duration_ms, temper_wasm_host_http_duration_ms, blob, actor, admission, event-store, projection update, session-context, and trajectory outbox metrics.

-
-
- -

Datadog could not preconfigure metrics that have never emitted. The percentile script skipped temper_cedar_evaluation_duration_ms, temper_cedar_evaluation_phase_duration_ms, temper_postgres_pool_acquire_duration_ms, temper_postgres_transaction_duration_ms, temper_query_projection_update_end_to_end_duration_ms, and temper_query_projection_update_queue_wait_ms; these must be rerun after the Temper runtime PR is deployed and emits them.

-
-
- -

Verified percentile configuration live: search_datadog_metrics(percentiles_enabled=true) now returns 18 temper* metrics for service:temperpaw, and metric context confirms temper_dispatch_ask_latency_ms, temper_cedar_evaluation_duration, and temper_session_phase_duration_ms have is_percentiles_enabled:true. After Datadog indexing caught up, the service coverage analyzer recognized all three standard APM signals for service:temperpaw: rate via monitors 283342070/283342074, duration via 283342076, and error via 283345345.

-
-
- -

Applied the DBM repair to production Postgres using Railway's Postgres public URL without printing secrets. Production's datadog-postgres-agent uses PGUSER=postgres, so scripts/datadog-postgres-dbm-setup.sql now accepts -v dbm_agent_role=postgres. SQL probes returned JSON explain plans, the Datadog Postgres agent was restarted, and fresh DBM plans after a live read smoke showed real index plans with zero fresh invalid_schema matches. DBM schema collection is still absent and remains a separate gap.

-
-
- -

Verified production on-demand profiling through Railway. Current production has TEMPER_PROFILING_ENABLED=true, TEMPER_PROFILING_AUTO_UPLOAD=true, TEMPER_PROFILING_CONTINUOUS unset, and DD_PROFILING_ENABLED=false. Idle CPU and wall captures returned HTTP 200 with 83-byte payloads; an 8-second CPU capture at 199 Hz while issuing 140 authenticated /readyz, /tdata/Channels, and /tdata/WorkerRuns reads returned HTTP 200 for all probes and produced a 26.4 KB pprof payload. Railway logs show uploads to the Datadog Agent intake, Datadog now has fresh profile_type:cpu upload buckets and no upload-error series, and go tool pprof parsed 70.35 ms of samples showing Axum middleware, OData read, Postgres catalog loading, SQLx receive, and tracing/OTel span work.

-
-
- -

Started refreshing Temper PR #229 onto latest origin/main. Resolved conflicts across serve startup, OTEL trace sampling, OData catalog reads, profiling upload/tests, Postgres DBM-attributed projection writes, and Turso catalog/projection tests while preserving both mainline features and the latency-observability instrumentation. Current local checks pass: cargo fmt --check, git diff --check, cargo check -p temper-cli, cargo test -p temper-server profiling::tests --lib -- --nocapture, cargo test -p temper-store-postgres metrics --lib -- --nocapture, and focused temper-store-turso catalog/export/published-artifact tests.

-
-
- -

Reran the full latency-observability package preflight after the mainline refresh. It initially exposed a verifier test filter that matched zero Turso tests after the merged mainline rename; fixed scripts/verify-latency-observability-package.sh to call load_entity_catalog_rows_returns_full_projected_fields. The corrected scripts/verify-latency-observability-package.sh full pass now covers package consistency, cargo formatting/checks, all temper-authz tests, profiling tests, OTEL sampler tests, Postgres metrics tests, projection metrics, OData shadow tests, replay parity integration tests, and the focused Turso full-fields/export tests.

-
-
- -

The first post-merge push attempt was blocked by the readability ratchet because merged mainline files raised PROD_FILES_GT500 and PROD_FILES_GT1000. Fixed the regression without updating the baseline: moved OTEL environment resolution, trace sampler rules, and OTEL tests into crates/temper-observe/src/otel/config.rs, otel/sampler.rs, and otel/tests.rs; trimmed crates/temper-cli/src/serve/mod.rs to the 1000-line threshold; and updated the package verifier to follow the new sampler module. Verification now passes: cargo fmt --check, cargo test -p temper-observe otel --lib -- --nocapture, scripts/verify-latency-observability-package.sh full, bash scripts/readability-ratchet.sh check .ci/readability-baseline.env, and git diff --check.

-
-
- -

Pushed the refreshed Temper branch at 50fe9664. The repository pre-push hook passed rustfmt, clippy, readability ratchet, and the full workspace test suite; the slowest dst_platform_random workload finished in 457.87 seconds. GitHub now reports Temper PR #229 as MERGEABLE with CI running. TemperPaw PR #266 remains CONFLICTING and is the next refresh target.

-
-
- -

Railway CLI authentication is confirmed for the correct account, so the live deployment and Datadog redeploy lane is open again.

-

Resolved the TemperPaw observability restructure merge in the worktree /Users/seshendranalla/Development/temperpaw-worktrees/latency-observability-program. The merged Datadog source keeps the new TemperPaw identity cleanup, LLMObs/session/DBM/log surfaces, legacy monitor reconciliation, Datadog retry/backoff, validate-before-deploy, 76 source monitors, and the latency-program percentile coverage backed by scripts/configure_metric_percentiles.py.

-

Verified TemperPaw locally with python3 -m json.tool for dashboard and monitors, python3 -m py_compile for Datadog scripts, git diff --check, focused Datadog contracts, identity/session contracts, and full cargo test -p temperpaw --tests -- --nocapture with 187 passing tests. Pushed merge commit aa38b944; GitHub now reports TemperPaw PR #266 as MERGEABLE with CI running.

-
-
- -

Redeployed the refreshed TemperPaw Datadog source through Railway. The dashboard update completed for mn4-k3k-i66.

-

The first monitor deploy exposed a real live-state problem: Datadog had same-name legacy OpenPaw monitor copies, and updating one into the current query would duplicate an existing TemperPaw monitor. Hardened scripts/deploy_monitors.py to prefer current team:temperpaw monitors over legacy duplicates, validated it with the Datadog observability contract, deployed all 76 source monitors, then ran --reconcile to delete 41 legacy/orphan monitors.

-

Re-applied Datadog percentile configuration. Datadog reported 30 configured source metrics: 19 already enabled live and 11 skipped as missing until the Temper runtime PR emits them. Pushed the deployer fix to TemperPaw at 0a18dbd8; PR #266 remains MERGEABLE with CI queued.

-
-
- -

Used Datadog MCP against the live deployment after the Railway deploy. Datadog confirms dashboard mn4-k3k-i66, APM service coverage for rate/error/duration on service:temperpaw, live temper_dispatch_ask_latency_ms percentiles, and current DBM explain plans. It also showed that datadog.dbm.activity_rows is too sparse to be the primary DBM availability gate even while postgresql.queries.count and plans are live.

-

Replaced the live [TemperPaw] Postgres DBM Activity Missing monitor with [TemperPaw] Postgres DBM Query Metrics Missing, backed by postgresql.queries.count. The deploy created monitor 283379354, deleted the old activity-row monitor 282522099, and left the new monitor OK. The only remaining active Datadog alert is [Temper] Trace Sampler Metrics Missing, which is expected until the Temper runtime PR is deployed through TemperPaw.

-

Committed and pushed the source-of-truth correction to TemperPaw at 05f73f00 after JSON validation plus datadog_monitor_config and datadog_observability_contract passed.

-
-
- -

Completed the merge/deploy lane. Temper PR #229 merged into main as eafba6ab after green CI. TemperPaw PR #266 bumped all Temper git rev pins to eafba6ab, passed cargo check -p temperpaw, focused Datadog tests, full cargo test -p temperpaw --tests -- --nocapture with 187 passing tests, GitHub CI, and Docker, then merged as 2b0ec6e9.

-

Deployed production Railway service openpaw from image digest sha256:4a84ec3a62b45d9ee67f897c5ca4b17ab9acdd3277fa0b37a64c49c69e7d4b78. The first redeploy showed the new image but stale service-level version variables; corrected BUILD_VERSION, BUILD_SHA, and DD_VERSION, then redeployed as a3564c57-a16f-49e9-ac5a-355dd0013f88. Authenticated /paw/version now returns sha-2b0ec6e9 and 2b0ec6e9849d5eabacb480678a9bfc92fff6f1c4; public /readyz and /healthz are green.

-

Re-ran Datadog percentile configuration after runtime deployment. The new runtime metrics now emit and have p95/p99 enabled: temper_cedar_evaluation_duration_ms, temper_cedar_evaluation_phase_duration_ms, temper_postgres_pool_acquire_duration_ms, temper_postgres_transaction_duration_ms, temper_query_projection_update_queue_wait_ms, and temper_query_projection_update_end_to_end_duration_ms. Latest p95 samples show dispatch around 20.5 ms, Cedar total around 50 ms, Cedar phase:authorizer around 50 ms with request/context/resource phases in microseconds, Postgres pool acquire around 2.4 ms, Postgres transaction around 75 ms for projection upsert and around 19-25 ms for event append, projection queue wait under 0.04 ms, and projection update end-to-end around 75-102 ms.

-

Datadog MCP post-deploy verification found 14,133 APM spans in the latest hour for service:temperpaw env:prod, no matching active alert groups, no matching error spans for the deployed version, APM rate/error/duration monitor coverage, live trace sampler metrics, and 111 fresh DBM plan records in the latest hour. Sample DBM plans are tagged with version 2b0ec6e9849d5eabacb480678a9bfc92fff6f1c4 and include entity_catalog_pkey, idx_entity_catalog_type, and idx_efi_lookup.

-

Attempted a production observe-only Paw Patrol worker proof. The control plane accepted the run and created RepoGraphSnapshot en-019e2af1-84dd-7b62-abb1-f2d28216efac, WorkCycle wc-019e2af1-8e84-77f3-9dab-956bcc17d59b, and WorkerRun en-019e2af1-8f3a-70b0-928c-6963fbc38fa7. The worker was correctly claimed by mac-mini-codex-prod but failed before scan execution because the worker host's local Codex auth refresh token is invalid/reused and its local Temper skill file is missing current YAML frontmatter. Later review reclassified this as a blocked worker-specific proof route, not the final latency/observability blocker.

-
-
- -

Merged the final deployment evidence report PR #230 after green CI as 0be1b768. CI included verification contract, integrity/DST patterns, compile/lint, spec verification, instrumentation hygiene, full workspace tests, and DST/platform shards.

-

Confirmed SSH access to the Mac mini worker host as openclaw. The worker doctor reproduced the failed e2e cause: control-plane OData and event-stream checks passed, but codex_exec_smoke failed because the local Temper skill lacked YAML frontmatter and the Codex ChatGPT refresh token was invalid/reused.

-

Repaired /Users/openclaw/.agents/skills/temper/SKILL.md by adding current Codex skill YAML frontmatter and preserved a timestamped backup at /Users/openclaw/.agents/skills/temper/SKILL.md.bak-20260515T101729Z. A second doctor run no longer reported the skill-loader error; the remaining failure is only Codex auth refresh.

-

Started codex login --device-auth on the Mac mini after clearing the unusable login, but the device-code flow timed out without browser approval. This would be required only to revive that specific worker route; it is not required for the corrected production e2e proof.

-
-
- -

Merged Mac mini worker auth checkpoint PR #231 after green CI as 6650ac85. CI covered verification contract, compile/lint, integrity/DST patterns, spec verification, instrumentation hygiene, full workspace tests, and all DST/platform shards.

-

Re-checked the deployment lane after the account fix: Railway CLI authentication is valid for the correct account and the worktree is linked to project openpaw-seshendranalla in environment production. Because the local Railway link has no default service selected, production commands must continue passing --service openpaw explicitly.

-

Re-checked Mac mini Codex auth and it still reported Not logged in. Started a fresh codex login --device-auth session on the Mac mini, held it open for the full 15-minute approval window, and it timed out without browser approval. No production observe-only e2e rerun was attempted because the worker doctor cannot pass codex_exec_smoke until that login is approved.

-
-
- -

Merged auth retry report PR #232 after green CI as 36917639. CI included verification contract, compile/lint, integrity/DST patterns, spec verification, instrumentation hygiene, full workspace tests, and all DST/platform shards.

-

Inspected the Codex CLI auth surface on the Mac mini. codex login --with-api-key and codex login --with-agent-identity are available, but no OPENAI_API_KEY or CODEX_AGENT_IDENTITY is present locally, remotely, or in the worker environment file. The remote Mac mini has Codex config but no auth file; this local machine has a working ChatGPT Codex session and an auth file.

-

The only non-browser path discovered for that worker route was transferring existing local Codex auth material to the Mac mini over SSH and then setting restrictive file permissions. That would move credential material, so it was not executed. After the 15:47 correction, this path is no longer part of the latency/observability finish line.

-
-
- -

Merged auth-path report PR #233 after green CI as b47a7c7a. CI covered verification contract, compile/lint, integrity/DST patterns, spec verification, instrumentation hygiene, full workspace tests, and all DST/platform shards.

-

Confirmed Railway is logged in to the correct account and linked to project openpaw-seshendranalla, environment production. Some worktrees still report Service: None, so commands continue to pass --service openpaw explicitly.

-

Created a fresh main-based TemperPaw proof worktree at /Users/seshendranalla/Development/temperpaw-worktrees/production-observe-main-proof on origin/main commit 2b0ec6e9, built paw-codex-worker, and ran the production worker doctor through Railway-injected credentials. The doctor passed repo path, workspace path, worker token, capabilities, execution-off safety, Codex binary presence, OData, and event-stream checks; PAW_CODEX_DOCTOR_EXEC_SMOKE=0 intentionally skipped Codex auth execution.

-

Started a temporary authenticated local worker with WORKER_ID=mac-mini-codex-prod and PAW_CODEX_ENABLE_EXECUTION=0, then reran the guarded production observe-only proof. The live control plane created RepoGraphSnapshot en-019e2bcb-8cea-7463-b044-7475c5733934, WorkCycle wc-019e2bcb-970c-7200-88e3-6f1cf996e862, and WorkerRun en-019e2bcb-97df-75e0-bb2d-8109fea0ca05; the WorkerRun was claimable and claimed by mac-mini-codex-prod, then failed with create /Users/openclaw/Development/temperpaw-worktrees.

-

This proves the deployed control plane can still dispatch the worker proof and the worker protocol can claim it. Later review corrected the scope: a passing final proof can use any representative production TemperPaw/Temper request or agent path, so the Mac-mini-specific worktree/auth issue is not the goal blocker.

-
-
- -

Merged local-proof blocker PR #234 after green CI as d3ca7b98. CI included verification contract, compile/lint, integrity/DST patterns, spec verification, instrumentation hygiene, full workspace tests, and all DST/platform shards.

-

Re-verified Railway after the account correction: railway whoami reports the intended account, and railway status now resolves project openpaw-seshendranalla, environment production, service openpaw from the production proof worktree.

-

Re-checked the Mac mini host. The production worktree root exists and launchd reports com.temperpaw.paw-codex-worker running with PAW_CODEX_ENABLE_EXECUTION=1, WORKER_ID=mac-mini-codex-prod, and WORKSPACE_ROOT=/Users/openclaw/Development/temperpaw-worktrees. codex login status still reports Not logged in.

-

Started a fresh Mac mini codex login --device-auth session and held it open for the full approval window, but it timed out without completion. This remained a blocker only for the worker-specific proof route.

-
-
- -

Merged current blocker report PR #235 after green CI as b466a584. The rerun passed verification contract, compile/lint, integrity/DST patterns, spec verification, instrumentation hygiene, full workspace tests, and all DST/platform shards.

-

Reconfirmed the post-Railway-fix deployment lane from the production proof worktree: Railway resolves project openpaw-seshendranalla, environment production, and service openpaw. The Mac mini worker launch agent is still running and the expected worktree root exists.

-

Started another fresh Mac mini codex login --device-auth session, kept it open for the full approval window, and it timed out without completion. This was later reclassified as a worker-route issue rather than the required production e2e proof.

-
-
- -

Merged second auth-timeout report PR #236 after green CI as 5e4652d4. CI passed verification contract, compile/lint, integrity/DST patterns, spec verification, instrumentation hygiene, full workspace tests, and all DST/platform shards.

-

Rechecked the active goal and the Mac mini worker host. The goal remains active, codex login status still reports Not logged in, launchd still reports com.temperpaw.paw-codex-worker running, and /Users/openclaw/Development/temperpaw-worktrees still exists.

-

Opened another fresh Mac mini codex login --device-auth session and held it open for the full 15-minute approval window. It timed out without completion. The 15:47 correction removes this as a goal-level blocker and keeps it only as historical worker-route evidence.

-
-
- -

Corrected the verification strategy after human review. The final live e2e requirement is a production TemperPaw/Temper proof with observable latency and correctness evidence; it does not require the mac-mini-codex-prod Codex worker. Mac mini auth is now recorded as a historical attempted proof route rather than a blocker for the latency/observability goal.

-

Stopped the open Mac mini device-auth wait. PR #237 already had green GitHub CI for the previous report update; this amendment changes the finish line to a representative production request or agent flow with Datadog APM/metric/DBM evidence plus a correctness check.

-
-
- -

Recalibrated program progress after human review. The previous 97% progress marker over-counted observability/deployment preparation as if it were the full latency program. The dashboard now shows 58%: measurement, dashboards, monitors, DBM, profiling path, trace budgets, AuthZ phase metrics, and projection observability are mostly complete; actual latency improvement PRs remain the next phase.

-

Confirmed actual speed improvement is in scope. The prioritized implementation path is Cedar/AuthZ authorizer cost first, then projection write amplification, DB transaction shape, workflow/integration executor behavior, and blob/data-plane separation where evidence supports it. Each improvement must ship with before/after latency evidence and correctness gates.

-
-
- -

Ran the corrected production end-to-end proof through the deployed TemperPaw OData/file path, without using a Codex worker. Production created File latency-proof-20260515204138, uploaded 36 bytes through $value, projected the entity to Ready, and read back content whose SHA-256 matched e49fc50a33d635d264263ba3f6a5e4dddb179b2710cfbc5b346e5cba14996a28.

-

Queried Datadog for service:temperpaw env:prod over 2026-05-15T20:40:00Z to 2026-05-15T20:46:00Z. The proof trace shows POST /tdata/Files at about 189 ms, PUT /tdata/Files('latency-proof-{num}')/$value at about 3.16 s, File.StreamUpdated at about 355 ms, and read-back GET /tdata/Files('latency-proof-{num}')/$value at about 175 ms. That moves the dashboard to 63% and makes file/blob write latency an immediate optimization candidate alongside Cedar/AuthZ and projection write amplification.

-
-
- -

Merged corrected report/e2e PR #237 into main at f2ec2a59, then rebased the first actual latency branch onto the updated mainline.

-

Created ADR-0088 and implemented the first File $value latency slice in codex/latency-file-value-fast-path-20260515: built-in File uploads now use native content-addressed blob storage and dispatch the verified StreamUpdated action, avoiding the WASM blob adapter on the hot path and avoiding a remote HEAD before PUT for content-addressed writes. Local checks passed: cargo check -p temper-server, cargo test -p temper-server --test file_value_fast_path, cargo fmt --check, HTML module syntax check, and git diff --check. Browser reload of the local file:// report remains blocked by the in-app browser URL policy. Remaining for this slice: DST/code review gates, PR, CI, merge, deploy, and production before/after Datadog proof.

-
-
- -

Unblocked the first latency-improvement PR after the repository readability ratchet caught an oversized-file regression. The File write helper moved to crates/temper-server/src/state/file_writes.rs, and OData $value PUT moved to crates/temper-server/src/odata/stream_put.rs; crates/temper-server/src/odata/write.rs is back below the 1000-line threshold without changing the readability baseline.

-

Re-ran validation after the split: cargo check -p temper-server, cargo test -p temper-server --test file_value_fast_path (7 tests), cargo fmt --check, bash scripts/readability-ratchet.sh check .ci/readability-baseline.env, git diff --check, and HTML module syntax check all pass. The broad determinism script still reports 26 pre-existing repository patterns and exits 0. DST/code review markers were refreshed for the updated file set. Remaining for this slice: amend, push, PR, CI, merge, deploy, and production before/after Datadog proof.

-
-
- -

Amended and pushed codex/latency-file-value-fast-path-20260515 at 241c1829. The repository pre-push hook passed all four gates: rustfmt, clippy, readability ratchet, and full cargo test --workspace. The slowest leg was dst_platform_random at 464.28 seconds, and the new file_value_fast_path suite passed again inside the full run.

-

Opened draft PR nerdsane/temper#238. Remaining for this slice: GitHub CI, merge, deploy, production before/after Datadog proof, then either mark the PR ready/merge or address review feedback.

-
-
- -

Completed the first actual latency-improvement code slice in Temper. PR nerdsane/temper#238 passed GitHub CI, was marked ready, and merged into main as 98b497b2. Its local and CI evidence covered the native built-in File $value fast path, readability split, full workspace tests, DST/platform suites, instrumentation hygiene, verification contract, and spec verification.

-

Created the TemperPaw rollout worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-file-fast-path-20260515 from origin/main, bumped all Temper runtime and temper-wasm-sdk pins from eafba6ab to 98b497b2, and opened nerdsane/temperpaw#269. Local validation passed: cargo check --locked -p temperpaw, cargo test --locked -p temperpaw --quiet, cargo test --locked -p paw-codex-worker --quiet, cargo fmt --all -- --check, git diff --check, cargo test --manifest-path os-apps/paw-patrol/wasm/review_gate_lifecycle/Cargo.toml --quiet, and CI clippy. Remaining for this slice: TemperPaw CI, merge, Docker image build, Railway redeploy, production File $value e2e proof, and Datadog before/after trace evidence.

-
-
- -

Completed the TemperPaw rollout PR. nerdsane/temperpaw#269 passed GitHub CI and merged into main as fd83c31b00bff57ed39419824efbb99c56ee2854. This makes production eligible for the Temper fast-path commit 98b497b2.

-
-
- -

Verified the post-merge delivery lane. TemperPaw main CI run 25946125710 succeeded, Docker run 25946125708 succeeded, and GHCR edge image digest sha256:b45f94c648fa634a0bd8448ddef51db3cc164e5fff8790f08b2ed3e53f0daaa4 was deployed to Railway as deployment d950fb9e-feb7-498f-b794-665fce9913bc.

-

Railway production health is green on https://openpaw-production.up.railway.app: /readyz returns ready JSON, /healthz returns 200, and authenticated /paw/version returns sha-fd83c31 / fd83c31b00bff57ed39419824efbb99c56ee2854. The custom domain temperpaw.katagami.ai still failed DNS resolution from this local environment, so the Railway domain remains the proof target.

-
-
- -

Ran the live production File $value proof against the Railway domain. File latency-fastpath-20260515235557 was created, uploaded through $value, reached Status: Ready, and read back the same SHA-256 a1aa969a1dde6e33a628a3da2db19588b90785c53c006999b17434f741f41705. Measured wall times: create 159.8 ms, upload 412.4 ms, entity read 64.2 ms, byte read 295.3 ms.

-
-
- -

Queried Datadog MCP for the deployed proof. The new upload trace is f3723cc99d48e9a65d5cc7ac4b61fc23: HTTP PUT /tdata/Files('latency-fastpath-{num}')/$value lasted 353.3 ms, includes native state.put_file_stream_content.native, has no matching wasm.invoke or blob_adapter spans, and carries version fd83c31b00bff57ed39419824efbb99c56ee2854 plus code paths under Temper 98b497b. Baseline trace 20be98473ad99f787fa0d007f36b9b7b lasted 3,159.5 ms and included wasm.invoke plus R2 host HTTP work. This is a roughly 8.9x server-span improvement and a roughly 7.7x direct curl upload improvement for the representative proof.

-
-
- -

Started the second actual latency-improvement slice in fresh Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-authz-fast-path-20260515 on branch codex/latency-authz-fast-path-20260515. ADR-0089 now records the Cedar/AuthZ candidate-policy decision: reduce the per-request Cedar policy set using only public scope constraints that cannot match the current principal/action/resource; keep forbids, diagnostics, default-deny, and Cedar expression evaluation intact; do not cache allow/deny decisions.

-

Implemented the local AuthZ candidate filter in crates/temper-authz/src/engine/candidates.rs, wired it into evaluate_request, and added temper_cedar_policy_candidate_count with low-cardinality source and outcome labels. Initial local verification is green: cargo test -p temper-authz passes 62 tests and cargo check -p temper-server passes. Remaining for this slice: wider checks, PR, TemperPaw pin, Railway deploy, live e2e, and Datadog proof that candidate count falls and phase:authorizer improves.

-
-
- -

Widened local verification for the AuthZ candidate slice. cargo fmt --all -- --check passed, cargo clippy -p temper-authz -p temper-server --all-targets -- -D warnings passed, the report inline module parses under node --check --input-type=module, and git diff --check passes for the touched Rust, ADR, and HTML files.

-
-
- -

Ran the full local pre-push verification pipeline for the AuthZ candidate slice. All four gates passed: rustfmt, workspace clippy, readability ratchet, and full cargo test --workspace. The slowest DST workload was dst_platform_random at 526.04 seconds and passed. The branch is ready for commit/PR packaging from a local verification standpoint.

-
-
- -

Committed the AuthZ candidate-filter slice as eb02561f, pushed branch codex/latency-authz-fast-path-20260515, and opened draft PR nerdsane/temper#239. Next gates are GitHub CI, ready/merge, TemperPaw pin, Railway deployment, live e2e, and Datadog proof for temper_cedar_policy_candidate_count plus reduced phase:authorizer duration.

-
-
- -

GitHub CI passed for AuthZ candidate-filter PR nerdsane/temper#239: spec verification, instrumentation hygiene, compile/lint, verification contract, integrity/DST patterns, workspace tests, and all DST platform shards were green in run 25949367580. The PR is merged into Temper main as b0b898a2d71192e9534ec3b3d0529c6e54c3d3c5.

-

Current gate for the second latency slice is now rollout, not implementation: create a TemperPaw worktree from main, pin Temper to b0b898a2, pass local/CI checks, merge/deploy, then prove live candidate counts and phase:authorizer latency in Datadog.

-
-
- -

Created TemperPaw rollout worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-authz-candidates-20260515 from origin/main on branch codex/bump-temper-authz-candidates-20260515. Bumped all Temper crate, WASM SDK, Docker, and observability-contract pins from 98b497b2 to b0b898a2.

-

Local rollout verification is green: cargo check -p temperpaw --locked, all 31 datadog_observability_contract tests, cargo fmt --all -- --check, and git diff --check passed. Next gates are commit, TemperPaw PR, CI, Railway deployment, and live Datadog proof.

-
-
- -

Committed TemperPaw rollout as f8d4b477, pushed branch codex/bump-temper-authz-candidates-20260515, and opened PR nerdsane/temperpaw#270. GitHub CI run 25949959572 is running.

-
-
- -

GitHub PR CI passed for nerdsane/temperpaw#270: format, clippy, check, tests, WASM build, and dashboard build were green in run 25949959572. The PR merged into TemperPaw main as 7726e4dfba5453c337dafb7411f88c239f3d5bd5.

-

Current gate is production rollout: main CI run 25950252249 and Docker run 25950252254 are running, then Railway must deploy the new image before live AuthZ evidence can be collected.

-
-
- -

Completed the AuthZ rollout delivery lane. TemperPaw main CI run 25950252249 passed, Docker run 25950252254 passed, and GHCR published ghcr.io/nerdsane/temperpaw:sha-7726e4d with manifest digest sha256:960781072bd76d96af79148eb02033a478085aa4ee5ca27aee9ee449d2c61e46.

-

Railway initially deployed the new digest while service-level version variables still pointed at fd83c31b. Corrected BUILD_VERSION, BUILD_SHA, IMAGE_TAG, and then DD_VERSION; deployment 8ad1c472-1c63-4473-a9ed-10f7458df253 is now successful on sha-7726e4d. Live /readyz is ready, Discord is connected/configured, and authenticated /paw/version, BUILD_SHA, and DD_VERSION all agree on 7726e4dfba5453c337dafb7411f88c239f3d5bd5.

-
-
- -

Ran the corrected production File/OData proof authz-candidates-ddversion-20260516141213. Production created File fl-019e3121-6a28-7a90-b5ea-6d1931e03e47, uploaded 72 bytes, reached Ready in one poll, and read back matching SHA-256 07b8ecbc0eb538ec7ff67b492f42644152ad0c74c42163f2c5b74b14f3b6641f. Direct wall timings were metadata 89.8-120.7 ms, list 60.5 ms, create 274.5 ms, upload 677.7 ms, entity read 57.2 ms, byte read 554.3 ms, and version check 43.4 ms.

-

Datadog retained exact-file sampled spans for the same proof under version:7726e4dfba5453c337dafb7411f88c239f3d5bd5: POST trace 9a3f6fc470e35fa071fc78054333f6e6 at 222 ms, PUT $value trace 0878a09bf0de8803b19bac5b6764744f at 625 ms, entity GET trace 0748a07428770bc177b21ee73207cc01 at 9.8 ms, and byte GET trace 214be076717c233a8f127977ba161300 at 501 ms.

-
-
- -

Queried Datadog metrics for the corrected version. temper_cedar_policy_candidate_count now emits with source:full and source:candidate plus the correct version tag, but percentile aggregation is still disabled for this new distribution. In the latest proof bucket, full policy count averaged 16,497 per evaluation and candidate count averaged 48, a roughly 344x reduction in policies sent to Cedar.

-

The latency lesson is more measured than "done": temper_cedar_evaluation_phase_duration_ms has percentiles enabled and showed phase:authorizer p95 around 0.23 ms, but phase:policy_candidates p95 around 13.4 ms. The full PUT trace still shows authz.Create at 19.6 ms and authz.RecordVersion at 26.1 ms. So ADR-0089 successfully reduced Cedar evaluator work, while the next AuthZ slice should remove per-request candidate scanning/rebuilding with a correctness-preserving index or reusable candidate policy-set cache.

-
-
- -

Started PERF-001B in a fresh main-based Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-authz-candidate-index-20260516 on branch codex/latency-authz-candidate-index-20260516. Added ADR-0090 for a precomputed static Cedar candidate-selection index: no decision cache, no Cedar bypass, fallback to full policy set on unsafe/non-static policies, and atomic index rebuild on policy reload.

-

Implemented the local slice in crates/temper-authz/src/engine/candidates.rs, crates/temper-authz/src/engine/candidates_tests.rs, and crates/temper-authz/src/engine/mod.rs. The engine now stores a compiled bundle of the Cedar PolicySet plus a candidate index keyed by principal, action, and resource constraints. Tests now include a 1,002-policy equivalence case that proves indexed selection matches the ADR-0089 scan semantics for a production-shaped large policy set.

-

Local checks passed: focused large-policy test, full cargo test -p temper-authz with 64 tests, cargo check -p temper-server, cargo clippy -p temper-authz -p temper-server --all-targets -- -D warnings, cargo fmt --all -- --check, and git diff --check. Remaining gates are commit, PR, GitHub CI, TemperPaw dependency bump, Docker/Railway deploy, live e2e proof, and Datadog confirmation that phase:policy_candidates drops below 2 ms.

-
-
- -

Committed PERF-001B as 247c7d6b5f7765894284ba3ab688cc2cfba99da8, pushed branch codex/latency-authz-candidate-index-20260516, and opened ready PR nerdsane/temper#240. GitHub CI run 25965850115 is now the active gate.

-

Local pre-push evidence is strong. The first full run was blocked by one unrelated temper-store-turso concurrency assertion; the exact test passed immediately on rerun. The second normal git push kept hooks enabled and passed rustfmt, workspace clippy, readability ratchet, full workspace tests, and doctests before pushing.

-
-
- -

GitHub CI for nerdsane/temper#240 passed: compile/lint, tests, spec verification, instrumentation hygiene, verification contract, integrity/DST patterns, and all DST platform shards were green in run 25965850115. Merged into Temper main as 84e95c6677f4a03364dea796cc67237bbc8275ce.

-

Current gate is now rollout, not Temper implementation: create a fresh TemperPaw worktree from main, bump every Temper pin to 84e95c66, pass local/CI checks, merge, build Docker, deploy Railway, run the live File/OData proof, and confirm phase:policy_candidates latency in Datadog.

-
-
- -

Created the fresh TemperPaw rollout worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-authz-index-20260516 on branch codex/bump-temper-authz-index-20260516 from origin/main. Bumped the Temper server, Docker observability clone, Datadog contract expectations, WASM SDK manifests, and the WASM lockfiles to 84e95c6677f4a03364dea796cc67237bbc8275ce.

-

Local rollout validation is green: cargo check -p temperpaw --locked, all 31 datadog_observability_contract tests, cargo fmt --all -- --check, and git diff --check. The first contract run correctly caught stale WASM lockfile sources; that is now fixed and included in the rollout diff.

-
-
- -

Committed the TemperPaw rollout as a5fb5e04f4c6a017d0bc3175bd3d07b0a85e9faa, pushed codex/bump-temper-authz-index-20260516, and opened draft PR nerdsane/temperpaw#271. GitHub CI run 25966397547 is in progress.

-
-
- -

TemperPaw rollout PR #271 passed CI run 25966397547: fmt, clippy, check, worker smoke syntax, os-apps WASM build, Cargo tests, and dashboard build. Marked it ready and merged into main as 1118183f47239401f216a771d6cd6fa6c1376a07.

-

Post-merge gates are now active: main CI run 25966733023 and Docker run 25966733035.

-
-
- -

Post-merge gates passed. TemperPaw main CI 25966733023 was green, and Docker 25966733035 pushed ghcr.io/nerdsane/temperpaw:sha-1118183 with digest sha256:7843528813075858ba12403af1fc326e314425d00b44643da987e982687e78b9.

-
-
- -

Railway deployment ee997c77-7abb-4682-aa76-4a5155f05c1f succeeded for the PERF-001B TemperPaw rollout. The production service is reachable on https://openpaw-production.up.railway.app; /readyz returns ready JSON in about 77 ms, /healthz returns HTTP 200 in about 59 ms, and authenticated /paw/version returns sha-1118183f plus SHA 1118183f47239401f216a771d6cd6fa6c1376a07 in about 52 ms.

-

The first deploy command attempted a plain Railway redeploy and reused the old sha-7726e4d image; the successful deploy used the checked-in Dockerfile.deploy wrapper with IMAGE_TAG=sha-1118183 and railway up.

-
-
- -

Live production proof authz-index-live-proof-20260516164954 passed on deployed version 1118183f47239401f216a771d6cd6fa6c1376a07. It created six Files, uploaded bytes through $value, observed Ready in one poll for every File, and read back matching SHA-256 values. The proof trace is ba53ebea1c7c4952ac11400d6b8e90a8.

-

Datadog confirms the candidate-index win and the remaining nuance: phase:policy_candidates p95 was 13.77 ms in the first post-deploy/cold bucket, then 0.335 ms, 0.262 ms, 0.262 ms, and 2.685 ms in the live File proof bucket. Candidate volume remains 16,497 full policies versus 48 candidates on the heavy File path. Exact proof trace aggregation shows File create route p50/p95 at about 62.5/65.0 ms, dispatch.phase.query_projection p50/p95 at about 20.8/25.4 ms, and entity.get_or_spawn_tenant_actor_with_fields p95 around 41.9 ms.

-
-
- -

Dashboard validation passed after the progress update: inline module JavaScript parses, git diff --check is clean for the report, desktop and mobile Playwright renders show the 92% progress marker, and mobile no longer has page-level horizontal overflow while wide evidence tables remain scrollable inside their own containers.

-
-
- -

Started PERF-002 in a fresh main-based Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-projection-diff-index-20260516 on branch codex/latency-projection-diff-index-20260516. Added ADR-0091 for Postgres query-projection diff-index upserts: keep entity_catalog authoritative, serialize concurrent projection writers through the catalog upsert, lock actual field-index rows in the same transaction, delete stale or changed index rows, upsert only missing/changed/status-changed rows, and avoid projection caches or Turso rewrites in the first PR.

-

Implemented the local PERF-002 slice in crates/temper-store-postgres/src/platform.rs. Projection upserts now compute the scalar index once, compare it with locked entity_field_index rows, preserve unchanged rows, remove fields that disappeared or became too large for the btree lookup index, and keep the full JSON fields in entity_catalog.

-

Added focused Postgres storage tests in crates/temper-store-postgres/src/store_projection_test.rs. They prove unchanged field-index rows are not rewritten, changed rows are rewritten, removed fields disappear, oversized values are no longer indexed, and the catalog row still carries the full projected fields and sequence. Local checks are green: cargo fmt --all -- --check, cargo check -p temper-store-postgres, cargo clippy -p temper-store-postgres --all-targets -- -D warnings, full cargo test -p temper-store-postgres -- --nocapture, and git diff --check. The integration tests compiled and passed in skip mode because local DATABASE_URL is not exported; live DB proof remains a deployment-gate item.

-
-
- -

Dashboard validation passed after the PERF-002 update: inline module JavaScript parses, git diff --check is clean for the report, and Playwright renders at desktop and mobile show the 93% progress marker with no page-level horizontal overflow. Screenshots were saved to /tmp/temper-report-progress-desktop-perf002.png and /tmp/temper-report-progress-mobile-perf002.png.

-
-
- -

Packaging review found and fixed a concurrency edge in the first PERF-002 implementation. A missing catalog row cannot be locked with FOR UPDATE, so concurrent first-writer projection transactions could have diffed against an empty field index and left stale rows. The patch now upserts entity_catalog first so the catalog primary key serializes both new and existing entity updates before the transaction locks and diffs entity_field_index.

-

Re-verified after the transaction-order fix: cargo fmt --all -- --check, cargo check -p temper-store-postgres, cargo clippy -p temper-store-postgres --all-targets -- -D warnings, full cargo test -p temper-store-postgres -- --nocapture, projection worktree git diff --check, report inline JavaScript syntax check, and report git diff --check all pass.

-
-
- -

Pushed PERF-002 branch codex/latency-projection-diff-index-20260516 after the full pre-push gate passed: rustfmt, clippy, readability ratchet, and full cargo test --workspace. The long randomized DST/platform test completed green instead of blocking the work.

-

Opened draft Temper PR nerdsane/temper#241 for commit bd2cab1a. GitHub CI run 25968899080 is in progress; initial status shows the verification contract passed, bench build skipped, and compile/lint, integrity/DST, tests, platform DST shards, spec verification, and instrumentation hygiene still running.

-
-
- -

Validated the updated living dashboard: inline module JavaScript parses, report git diff --check is clean, desktop and mobile Playwright renders show the 94% marker, PR/CI links are present, and mobile has no page-level horizontal overflow. Screenshots were saved to /tmp/temper-report-progress-desktop-pr241.png and /tmp/temper-report-progress-mobile-pr241.png.

-
-
- -

Temper PR #241 passed GitHub CI run 25968899080, was marked ready, and merged as ed68e78539f5511b453e34e554bc95b0afb91a0d. The local gh cleanup reported a harmless worktree checkout error because another worktree already owns main; the PR state is merged.

-

Created fresh main-based TemperPaw rollout worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-projection-diff-index-20260516 on branch codex/bump-temper-projection-diff-index-20260516. It pins root Temper crates, packaged temper-wasm-sdk manifests/locks, Docker TEMPER_OBSERVABILITY_REV, and the Datadog observability contract test to ed68e785. Local rollout verification is green: locked check, fmt, diff check, CI shell script checks, clippy, TemperPaw/worker/review-gate tests, all packaged WASM build scripts, and dashboard build.

-
-
- -

Opened draft TemperPaw rollout PR nerdsane/temperpaw#272 for commit 65da324e. The PR carries only the dependency and deployment pin handoff for PERF-002: root Temper crates, packaged temper-wasm-sdk manifests and locks, Docker TEMPER_OBSERVABILITY_REV, and Datadog observability contract expectations now point at merged Temper commit ed68e78539f5511b453e34e554bc95b0afb91a0d.

-

PR CI run 25969575912 is in progress. If it passes, the next gate is to mark the PR ready, merge it, verify main CI and Docker, deploy with the checked-in Dockerfile.deploy wrapper through railway up, then run the live projection proof and Datadog/replay/shadow drift checks.

-
-
- -

Validated this dashboard update: inline module JavaScript parses, report git diff --check is clean, and desktop/mobile Playwright renders show the 96% progress marker, the PR #272 link, and CI run 25969575912 with no page-level horizontal overflow. Screenshots were saved to /tmp/temper-report-progress-desktop-pr272.png and /tmp/temper-report-progress-mobile-pr272.png.

-
-
- -

TemperPaw rollout PR #272 passed PR CI run 25969575912, was marked ready, and merged as 73e756a0bdb5f9a3084c5dd33c424154871fb873. The local merge cleanup again reported a harmless worktree checkout error because another worktree owns main; GitHub confirms the PR state is merged.

-

Next gate is post-merge verification on main: wait for main CI and Docker, then deploy the merged commit to Railway and prove the projection diff-index path live with request evidence, Datadog before/after metrics, and replay/shadow drift checks.

-
-
- -

Post-merge TemperPaw main gates are green for commit 73e756a0bdb5f9a3084c5dd33c424154871fb873: CI run 25969887628 passed in 14m10s and Docker run 25969887617 passed in 23m19s. Docker emitted a pre-existing GitHub Actions Node.js 20 deprecation annotation for docker/build-push-action@v6, docker/login-action@v3, and docker/metadata-action@v5; it did not fail the build.

-

Deployment will use a fresh main-based TemperPaw worktree at the merged commit, not the PR branch or the dirty base checkout.

-
-
- -

Deployed PERF-002 to Railway production from detached main worktree /Users/seshendranalla/Development/temperpaw-worktrees/deploy-projection-diff-index-20260516. Deployment 72635a45-5744-4c72-82e0-7b7143e9a548 succeeded for service openpaw using Dockerfile.deploy, IMAGE_TAG=sha-73e756a, and wrapper digest sha256:e89ad62667469b67eda4a67fdbeb618ac847757ede66ca5fe19a375ed50b9fed.

-

Production probes passed: /readyz returned ready JSON, /healthz returned HTTP 200, and authenticated /paw/version returned sha-73e756a with SHA 73e756a0bdb5f9a3084c5dd33c424154871fb873. The non-secret Railway release metadata and Datadog version tags were updated to the same SHA before deployment.

-
-
- -

Live projection proof projection-diff-live-proof-20260516191623 passed against https://openpaw-production.up.railway.app. It created four Files, double-uploaded bytes through $value, reached Ready in one poll after each upload, read matching bytes back, and listed each File by Name through the projection path.

-

Direct wall timings from the successful run: creates 171.9-189.2 ms, uploads 412.2-540.6 ms, byte reads 328.4-385.3 ms, and projection list-by-name reads 67.5-76.7 ms. The proof exercised production auth, OData dispatch, event append, projection update, blob/object storage, read-after-write, and Datadog APM under version 73e756a0bdb5f9a3084c5dd33c424154871fb873.

-
-
- -

Datadog confirms current-version projection metrics are active. temper_query_projection_update_duration_ms, temper_query_projection_update_end_to_end_duration_ms, temper_query_projection_update_started_total, temper_query_projection_update_enqueued_total, temper_query_projection_update_queue_wait_ms, and temper_query_projection_applied_sequence_nr are visible under service:temperpaw and version 73e756a0bdb5f9a3084c5dd33c424154871fb873.

-

In the proof bucket, File projection upsert p95 was 42.6 ms for source:create and 91.1 ms for source:background_dispatch. APM sampled the successful proof window: PUT $value averaged about 420.1 ms, byte GETs about 306.3 ms, File creates about 132.7 ms, list reads about 17.5 ms, and entity reads about 8.3 ms. Shadow and replay parity metric searches returned no recent data, so that remains an instrumentation/scheduling gap rather than a closed proof.

-
-
- -

Direct database correctness proof passed for the four successful proof Files. entity_catalog has all four rows at tenant default, entity type File, Status=Ready, and sequence_nr=5; entity_field_index has 14 indexed fields per File.

-

The indexed Name and MimeType values match the catalog fields, and each indexed content_hash matches the SHA-256 bytes read back by the live proof. This closes the production correctness slice for PERF-002 while keeping continuous replay/shadow execution as the next observability hardening item.

-
-
- -

Selected the next measured latency slice from current Datadog evidence: residual File $value latency still includes post-commit reaction work. Current-version APM/metrics show PUT $value around 490-611 ms p95 depending on span scope, File.StreamUpdated around 248 ms p95, and reaction.dispatch around 220 ms p95. Long-lived event streams and idle actor passivation are excluded from the user-visible target.

-

Created fresh main-based Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-file-value-residual-20260516 on branch codex/latency-file-value-residual-20260516, then wrote ADR-0092 docs/adrs/0092-bounded-background-file-reactions.md. The decision keeps File byte/blob/verified StreamUpdated commit synchronous, moves the FileVersion/RecordVersion reaction chain into a bounded background lane only for native File $value writes, and falls back to inline reactions when the background budget is exhausted.

-
-
- -

Implemented PERF-005B locally. The dispatch API now has an explicit await_reactions option; all default/OData/platform call sites preserve existing inline behavior, while put_file_stream_content_native opts into bounded background File reactions after the blob write and File transition commit. The background path emits reaction.dispatch.background and keeps overload behavior conservative by awaiting inline when no semaphore permit is available.

-

Local validation is green: cargo test -p temper-server --test trigger_e2e_prod inline_action_triggers_can_run_in_background_when_requested -- --nocapture, cargo test -p temper-server --test file_value_fast_path -- --nocapture, cargo check -p temper-server -p temper-platform, and cargo clippy -p temper-server -p temper-platform --all-targets -- -D warnings. Remaining gates are DST/code-review markers, pre-push/full CI, PR, TemperPaw rollout, Railway deployment, live File proof, Datadog before/after, and direct correctness proof.

-
-
- -

Expanded local validation for PERF-005B. The focused suite cargo test -p temper-server --test trigger_e2e_prod --test reaction_e2e_prod --test adapter_dispatch --test wasm_dispatch -- --nocapture passed, confirming default inline reaction behavior remains intact for existing reactions, adapter callbacks, triggers, and WASM callbacks. cargo fmt --all -- --check and git diff --check also pass.

-

Manual DST and code-quality review markers were written for this worktree because separate reviewer agents are not available under the current session tool rules. The determinism audit exits 0 with the known 26 pre-existing repository patterns and did not identify a new unsuppressed pattern from PERF-005B. Static dashboard validation passed; the in-app browser refused a local file:// reload under its URL policy, so responsive visual refresh remains unverified in-browser for this edit rather than silently bypassed.

-
-
- -

Packaged PERF-005B for GitHub. Commit 77e97f92d855d612d9aa5d4f925ed3eb6f0f3109 is pushed on branch codex/latency-file-value-residual-20260516, and draft Temper PR nerdsane/temper#242 is open.

-

The full local pre-push gate passed before push: rustfmt, workspace clippy, readability ratchet, full workspace tests, and doctests. GitHub CI run 25971893778 is running; early completed jobs include verification contract, Integrity & DST Patterns, platform random DST shard, and instrumentation hygiene.

-
-
- -

Temper PR #242 passed CI run 25971893778, including verification contract, compile/lint, integrity/DST patterns, tests, DST/platform shards, spec verification, and instrumentation hygiene. Only pre-existing GitHub Actions Node.js 20 deprecation annotations remained.

-

Marked the PR ready and merged it as 1796c4f0f5cc9b81557107bae19795db5b578d0c. This closes the Temper half of PERF-005B; the remaining work is the TemperPaw rollout, Railway deployment, live File proof, Datadog proof, and correctness proof.

-
-
- -

Created a fresh main-based TemperPaw worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-file-value-residual-20260516 on branch codex/bump-temper-file-value-residual-20260516 from origin/main commit 73e756a0. Updated the pinned Temper revision across root manifests, Docker observability source, Datadog observability contracts, packaged WASM manifests, and packaged WASM lockfiles to 1796c4f0f5cc9b81557107bae19795db5b578d0c.

-

The rollout needed one compatibility patch: crates/temperpaw/src/setup_api.rs now sets await_reactions: true when constructing DispatchExtOptions, preserving the old synchronous setup behavior for TemperPaw while the native File $value path uses the new background-reaction option inside Temper.

-

Local rollout validation is green: locked cargo check -p temperpaw, 31 Datadog observability contract tests, stale-revision scan, rustfmt, diff check, clippy/check for temperpaw and paw-codex-worker, CI script syntax/smokes, full temperpaw tests, paw-codex-worker tests, review-gate lifecycle tests, every packaged WASM build script, and the dashboard production build after npm ci.

-
-
- -

Committed the TemperPaw rollout as af3546778cb9c8d5c6150f11a5e27f5849a13639, pushed branch codex/bump-temper-file-value-residual-20260516, and opened draft PR nerdsane/temperpaw#273.

-

GitHub queued PR CI run 25972634147. The next decision point is remote CI: if green, mark the PR ready and merge; if red, fix the rollout before deployment.

-
-
- -

TemperPaw PR #273 passed PR CI run 25972634147 in 14m44s: fmt, clippy, check, worker smoke, packaged WASM build, tests, and dashboard build all succeeded.

-

Marked the PR ready and squash-merged it as e8457ca4a4891b05710980c7aebe1ed822d55f87. TemperPaw main CI run 25972955841 and Docker run 25972955833 are now running against the merge commit.

-
-
- -

TemperPaw main CI 25972955841 passed in 14m55s, repeating the full mainline gate: fmt, clippy, check, worker smoke, packaged WASM build, tests, and dashboard build.

-

TemperPaw Docker 25972955833 passed in 24m49s. It pushed ghcr.io/nerdsane/temperpaw:sha-e8457ca with image digest sha256:4956e8d7ed26af68168b64643b6aefc0e03e5cc114cbb8377382649c3a9eadfa. The only annotation is the pre-existing Node.js 20 deprecation warning from Docker GitHub Actions.

-
-
- -

Deployed PERF-005B to Railway production from the merged TemperPaw commit e8457ca4a4891b05710980c7aebe1ed822d55f87. Railway deployment 5778571f-b325-4237-a792-cb13342963a4 succeeded with wrapper image digest sha256:3280c36ae065391ecc7741abd9051faae0a983150db312c4e920a5eb977f1125 and runtime image tag sha-e8457ca.

-

Post-deploy probes are version-correct: /readyz returned ready JSON in about 75.7 ms, /healthz returned HTTP 200 in about 40.3 ms, production variables now expose DD_VERSION=e8457ca4a4891b05710980c7aebe1ed822d55f87, and authenticated /paw/version returns sha-e8457ca4 with SHA e8457ca4a4891b05710980c7aebe1ed822d55f87.

-
-
- -

Ran live proof file-reaction-background-live-proof-20260516215205 against Railway production on e8457ca4. It created six fresh Files and performed twelve native File $value writes. Every write reached Status=Ready in one poll, every byte readback matched the final SHA-256, and every File was findable by Name through the projection path.

-

Measured request wall times from the proof client: first-write average 288.5 ms and max 317.0 ms; final-write average 281.6 ms and max 294.2 ms; final byte reads averaged in the low-to-mid 300 ms range. Each File produced the expected Superseded + Current FileVersion chain.

-
-
- -

Ran a direct production Postgres proof for the same six File IDs using Railway's Postgres public URL without printing secrets. It found all six File catalog rows at Status=Ready, sequence_nr=5, version_count=2, matching content_hash, and last_version_id equal to the Current FileVersion ID.

-

The DB proof also found ok_index_files=6: each File had 13 projection index rows, each File had 5 File events, and each FileVersion pair had 5 combined FileVersion events. Current FileVersion rows held the final content hash and size; Superseded rows remained linked as the predecessor.

-
-
- -

Queried Datadog APM for the proof window. Sampled PUT /tdata/Files('fl-{guid}')/$value spans under version:e8457ca4a4891b05710980c7aebe1ed822d55f87 had p95 238.7 ms, average 238.6 ms, and max 271.6 ms. The previous projection-diff proof window had sampled PUT $value p95 490.3 ms, average 594.8 ms, and max 1440.5 ms.

-

Datadog also shows File.StreamUpdated p95 improving from 223.1 ms in the previous proof window to 13.7 ms now. New reaction.dispatch.background spans are present with p95 89.6 ms; trace fdd583aecbc17ec4b089b0e0dabb3580 shows the HTTP response ending at 21:52:12.908Z and the background reaction span starting at 21:52:12.909Z.

-
-
- -

Selected PERF-005C from the post-PERF-005B evidence. The residual upload p95 is now inside state.put_file_stream_content.native, while temper_blob_io_wait_duration_ms only measures semaphore queue wait and returned no useful production samples. The missing surface is actual native blob transport duration for local filesystem and S3/R2 operations.

-

Created fresh main-based Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-blob-transport-observability-20260516 on branch codex/latency-blob-transport-observability-20260516, then wrote ADR-0093 docs/adrs/0093-native-blob-transport-observability.md.

-
-
- -

Implemented PERF-005C locally in Temper. Native BlobStore operations now emit bounded blob.transport.put, blob.transport.put_content, blob.transport.get, and blob.transport.head spans plus temper_blob_native_transport_duration_ms, temper_blob_native_transport_requests_total, temper_blob_native_transport_request_bytes, and temper_blob_native_transport_response_bytes. Labels are bounded to operation, backend, outcome, and status class, with no tenant, key, hash, URL, or auth material.

-

Validation passed: cargo check -p temper-server, cargo clippy -p temper-server --all-targets -- -D warnings, cargo test -p temper-server blob, cargo test -p temper-server --test file_value_fast_path, cargo fmt --all -- --check, and git diff --check. One malformed local Cargo command was corrected and rerun with proper filters.

-
-
- -

Committed PERF-005C as 8fc79f3c50eb05e3fdf419749b444ef6b45bbfae, pushed branch codex/latency-blob-transport-observability-20260516, and opened draft Temper PR nerdsane/temper#243.

-

The repository pre-push gate passed rustfmt, workspace clippy, readability ratchet, and full cargo test --workspace. GitHub CI run 25974966315 is now running and the next decision point is green CI, then ready/merge, then the TemperPaw rollout.

-
-
- -

Temper PR nerdsane/temper#243 passed CI run 25974966315: verification contract, compile/lint, integrity/DST patterns, full non-DST workspace tests, all DST/platform shards, spec verification, and instrumentation hygiene are green.

-

Marked the PR ready and squash-merged it into Temper main as 88c9d797b398df04d845ff202738b09542817415. The active PERF-005C gate is now a fresh main-based TemperPaw rollout worktree, dependency pin bump, dashboard/monitor contract update for temper_blob_native_transport_*, PR, Docker, Railway deploy, live File proof, and Datadog proof.

-
-
- -

Created and validated the PERF-005C TemperPaw rollout in /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-blob-transport-observability-20260516 on branch codex/bump-temper-blob-transport-observability-20260516. The rollout pins Temper to merged commit 88c9d797b398df04d845ff202738b09542817415 and extends Datadog dashboard, monitor, percentile configuration, docs, and contract tests for temper_blob_native_transport_*.

-

Local rollout validation passed: dashboard and monitor JSON parse, percentile Python script compiles, git diff --check is clean, cargo fmt --all -- --check passes, cargo check --locked -p temperpaw passes, Datadog observability contract tests pass, Datadog monitor config tests pass, full cargo test --locked -p temperpaw --tests -- --nocapture passes with 187 tests, and the focused workspace_fs WASM SDK test passes.

-
-
- -

Committed the TemperPaw PERF-005C rollout as 04c6d3bf7c0b0e237675e4dba7f1beb7471ce220, pushed branch codex/bump-temper-blob-transport-observability-20260516, and opened draft PR nerdsane/temperpaw#274.

-

GitHub PR CI run 25975556316 is in progress. The next gate is green CI, then mark ready, merge, wait for main CI and Docker, deploy the new image to Railway, and run live File/Datadog/DB proof for native blob transport.

-
-
- -

TemperPaw PR #274 passed PR CI run 25975556316: fmt, clippy, check, worker smoke syntax, os-app WASM build, tests, and dashboard build are green.

-

Marked the PR ready and squash-merged it into TemperPaw main as 9fcd4b2b1f6c651d8c12c954f836d250efb443be. Mainline CI run 25975834289 and Docker run 25975834282 are now running for that merge commit.

-
-
- -

TemperPaw main CI run 25975834289 and Docker run 25975834282 passed for merge commit 9fcd4b2b1f6c651d8c12c954f836d250efb443be.

-

Docker published ghcr.io/nerdsane/temperpaw:sha-9fcd4b2 with digest sha256:74d398d56362de4b31cf09334e975952ec8f455a29e92a80fee99a6d5edf65bb. Next gate is a fresh main-based deployment worktree, Railway variable update/deploy, health/version probes, live File proof, Datadog native blob proof, and direct DB correctness proof.

-
-
- -

Deployed PERF-005C to Railway production from merged TemperPaw commit 9fcd4b2b1f6c651d8c12c954f836d250efb443be. Railway deployment 37edf6ad-5f71-4865-85e3-46a05dd7cc78 succeeded; wrapper digest is sha256:4f3da001ee59f15b87833713d6fa65a84c0cbffe5a5711a0fdf48cbc2b9c6f20.

-

Post-deploy probes are version-correct: /readyz HTTP 200 in 96.5 ms, /healthz HTTP 200 in 63.6 ms, authenticated /paw/version HTTP 200 in 55.9 ms, and Railway env reports DD_VERSION=9fcd4b2b1f6c651d8c12c954f836d250efb443be.

-
-
- -

Applied the native blob transport observability objects to live Datadog. Percentile configs were created for temper_blob_native_transport_duration_ms, temper_blob_native_transport_request_bytes, and temper_blob_native_transport_response_bytes. Dashboard mn4-k3k-i66 was updated, and monitors [Temper] Native Blob Transport Duration Spike id 283877764 plus [Temper] Native Blob Transport p95 Regression id 283877770 were created.

-

Ran fresh live proof blob-transport-live-proof-20260517001121 after percentile config so Datadog p95/p99 has post-configuration samples. It created six Files, ran twelve $value writes, read final bytes back, listed by Name, and verified Superseded + Current FileVersion chains. Final-write average was 347.25 ms, max 393.24 ms.

-
-
- -

Ran direct production Postgres proof for the same six File IDs using Railway's Postgres public URL without printing secrets. It found ok_file_catalog_rows=6, ok_current_file_versions=6, ok_superseded_file_versions=6, ok_file_event_chains=6, and ok_fileversion_event_chains=12.

-

The DB proof also found file_event_rows_total=30, fileversion_event_rows_total=30, ok_file_index_rows_14=6, and ok_fileversion_index_rows_9=12; no bad File catalog rows or bad FileVersion event rows were returned.

-
-
- -

Queried Datadog for the post-configuration proof window. Metrics show put_content count 12, average 195.2 ms, p95/p99 223.8 ms; successful get count 12, average 106.8 ms, p95/p99 177.4 ms; 404 read-probe get count 12, average 73.3 ms, p95/p99 88.3 ms. Request and response byte metrics both averaged 98,304 bytes for the expected 64 KiB / 128 KiB mix.

-

Expanded trace a29ccff5afb451042e8bcd8ec1247b88 proves the new span placement: PUT $value 264.4 ms, state.put_file_stream_content.native 235.7 ms, child blob.transport.put_content 186.5 ms, followed by File.StreamUpdated and bounded background reactions.

-
-
- -

Started PERF-005D from the PERF-005C Datadog evidence. Fresh native File reads were paying one failed legacy external-key blob.transport.get before the successful native temper-fs/{content_hash} get. Created fresh main-based Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-file-blob-read-key-order-20260517 on branch codex/latency-file-blob-read-key-order-20260517 and added ADR-0094 for native-first File blob read-key ordering with legacy fallback.

-

Implemented the local slice in crates/temper-server/src/state/file_read_blobs.rs. The read helper now tries temper-fs/{content_hash} first for every File read, then falls back to the legacy external {content_hash} key only when the tenant blob endpoint is external. File state, IOA specs, projections, OData, and acknowledgement semantics are unchanged.

-
-
- -

PERF-005D focused local checks are green: cargo test -p temper-server file_blob_read_keys passed the native-first and internal-only helper tests; cargo test -p temper-server --test file_value_fast_path passed all seven File value fast-path tests; cargo check -p temper-server, cargo fmt --all -- --check, and git diff --check passed.

-

Next gates are the required DST/code-quality review markers for the sim-visible temper-server change, commit, Temper PR, GitHub CI, TemperPaw pin bump, Railway deploy, live File proof, and Datadog confirmation that fresh native reads no longer emit the legacy-key 404 while legacy fallback remains available.

-
-
- -

Packaged PERF-005D for GitHub. DST and code-review markers were written for the touched sim-visible server file and ADR; the broad determinism audit still reports the known 26 pre-existing repository patterns, and the PERF-005D diff adds none.

-

Committed a7e942cf with message Prefer native File blob read key, pushed branch codex/latency-file-blob-read-key-order-20260517, and opened draft Temper PR nerdsane/temper#244. The local pre-push gate passed all four gates: rustfmt, workspace clippy, readability ratchet, full cargo test --workspace, and doctests. GitHub CI run 25977404056 is in progress.

-
-
- -

Checked the active Temper PR gate while resuming the thread. PR #244 is still draft and mergeable. All checks except the broad cargo test --workspace -- --skip dst_ job are green; that final job has been running since 2026-05-17T00:56:44Z.

-

While waiting, inspected the TemperPaw rollout surface. TemperPaw pins all Temper crates and all packaged WASM SDK manifests to one Temper revision, with crates/temperpaw/tests/datadog_observability_contract.rs enforcing the pin so the rollout remains a coordinated runtime/SDK bump.

-

Created fresh TemperPaw worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-file-blob-read-key-order-20260517 on branch codex/bump-temper-file-blob-read-key-order-20260517 from origin/main commit 9fcd4b2b. No pin edit has been made yet; that waits for the Temper merge SHA.

-
-
- -

PERF-005D Temper PR #244 passed CI run 25977404056, was marked ready, and merged as ff79974d88a5b1a67e1fa9c4a746b422e12b29c3. The merge command reported the known local checkout issue because another worktree owns main, but GitHub confirms the PR state is merged.

-

Applied the TemperPaw coordinated pin bump to commit 709876f3, pushed codex/bump-temper-file-blob-read-key-order-20260517, and opened draft rollout PR nerdsane/temperpaw#275. Local rollout checks are green: cargo check --locked -p temperpaw -p paw-codex-worker, cargo test -p temperpaw --test datadog_observability_contract with 31 passing tests, cargo fmt --all -- --check, and git diff --check. GitHub CI run 25977831129 is in progress.

-
-
- -

TemperPaw rollout PR #275 passed PR CI run 25977831129, was marked ready, and merged as 821d6c6ee28845e6b1365d78f3b85c3b5cc1ad15. GitHub queued main CI run 25978275903 and Docker run 25978275890.

-
-
- -

TemperPaw main CI run 25978275903 and Docker run 25978275890 passed for 821d6c6ee28845e6b1365d78f3b85c3b5cc1ad15. Docker published ghcr.io/nerdsane/temperpaw:sha-821d6c6 with base image digest sha256:c8e6ac6ff9df7a47ceec29fa80ebc4228d3cb99fdf3f285ac27189714e341aff.

-

Railway deployment b628e727-b013-4c89-9eaa-f65f4f7e7433 succeeded from the deployment worktree. The first deploy correctly used the image but still reported old runtime version variables; the second deploy fixed BUILD_VERSION, BUILD_SHA, DD_VERSION, DD_GIT_COMMIT_SHA, and OTEL service.version. Authenticated /paw/version now reports sha-821d6c6 and full SHA 821d6c6ee28845e6b1365d78f3b85c3b5cc1ad15.

-
-
- -

Ran production proof blob-read-key-order-live-proof-20260517022158: 12 fresh Files, 1,802 bytes written through $value, exact byte readback, matching SHA-256 ContentHash, matching SizeBytes, Ready status, VersionCount=1, and non-empty LastVersionId for every File. Client read wall clock: min 221.9 ms, p50 247.7 ms, max 287.7 ms.

-

Datadog before/after confirms the fix: previous version 9fcd4b2b proof window had 12 successful fresh-read gets and 12 legacy-key not_found gets; current version 821d6c6e proof window has successful get and put_content activity with no get/outcome:not_found series. DBM also captured proof-window PostgreSQL samples with propagated service/version tags.

-
-
- -

Selected PERF-003 from a fresh Datadog residual pass. Current deployed version 821d6c6e shows query_projection_upsert transaction p95 around 429-443 ms while pool acquire stays around 4.8-5.0 ms; projection end-to-end p95 reaches 471-478 ms for Session/background_dispatch. That points at transaction work/shape rather than connection starvation.

-

Created fresh main-based Temper worktree /Users/seshendranalla/Development/temper-worktrees/latency-db-transaction-shape-20260517 on branch codex/latency-db-transaction-shape-20260517, added ADR-0095, and implemented the first local fast path: scalar index extraction moves before BEGIN, unchanged projection hash/status updates only catalog sequence/metadata, and changed projections still use the ADR-0091 diff reconciliation. Added temper_postgres_projection_index_reconciliations_total to distinguish insert, diff, and skipped_unchanged paths in Datadog.

-

Local verification is green: cargo test -p temper-store-postgres upsert_query_projection -- --nocapture, full cargo test -p temper-store-postgres -- --nocapture, cargo clippy -p temper-store-postgres --all-targets -- -D warnings, cargo check -p temper-store-postgres, cargo check -p temper-server, cargo fmt --all -- --check, and git diff --check. The Postgres integration portions still skip without DATABASE_URL; production DB proof remains a rollout gate.

-
-
- -

Completed the PERF-003 local-to-PR gate. Code-quality review marker passed with no findings, commit f0e3a18c was created with message Shorten projection transaction fast path, and branch codex/latency-db-transaction-shape-20260517 pushed after the full Temper pre-push gate passed rustfmt, workspace clippy, readability ratchet, full cargo test --workspace, and doctests.

-

Opened draft Temper PR nerdsane/temper#245. Remaining gates for this slice are GitHub CI, mark-ready/merge, TemperPaw pin bump and rollout PR, Railway deployment, live OData/File proof, Datadog before/after query_projection_upsert transaction p95, temper_postgres_projection_index_reconciliations_total path mix, and direct projection correctness proof against production data.

-
-
- -

Finished the Temper half of PERF-003: PR nerdsane/temper#245 passed GitHub CI run 25979739482, was marked ready, and merged as 6439a8a0be134ffa37933701fb8fca140121d44d. CI passed verification contract, compile/lint, integrity/DST, non-DST workspace tests, all DST/platform shards, spec verification, and instrumentation hygiene.

-

Created fresh main-based TemperPaw worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-db-transaction-shape-20260517, bumped all Temper runtime/server/store/SDK pins and checked-in WASM lockfiles to 6439a8a0, and committed rollout commit beaaf466. Local rollout checks passed: cargo check --locked -p temperpaw -p paw-codex-worker, cargo test -p temperpaw --test datadog_observability_contract -- --nocapture with 31 passing tests, cargo fmt --all -- --check, git diff --check, and rg --no-ignore proof that the old Temper rev is gone.

-

Opened draft TemperPaw rollout PR nerdsane/temperpaw#276. PR CI run 25980160694 is in progress; after that the remaining gates are mark-ready/merge, main CI, Docker, Railway deploy, live OData/File proof, Datadog transaction p95 and reconciliation-path proof, and direct projection correctness proof.

-
-
- -

Completed the TemperPaw rollout gate for PERF-003. PR nerdsane/temperpaw#276 passed PR CI run 25980160694, was marked ready, and merged as c16e0201c1490e0496f4964f5c72b704bb8cd216. TemperPaw main CI run 25980450147 and Docker run 25980450153 passed.

-
-
- -

Deployed PERF-003 to Railway production from detached deployment worktree /Users/seshendranalla/Development/temperpaw-worktrees/deploy-db-transaction-shape-20260517. Final successful deployment is 6e42424f-7ad4-4577-b127-9e3a0a36abeb, using Dockerfile.deploy, pinned image tag sha-c16e020, and image digest sha256:9c8a844fdfe52a201adea5755a3d5d61c498d8ba6cdc84c2f64a151d3bf27c89.

-

Corrected the Railway version environment so BUILD_VERSION, BUILD_SHA, DD_VERSION, and Datadog version tagging match c16e0201c1490e0496f4964f5c72b704bb8cd216. Authenticated /paw/version reports sha-c16e020 and /readyz reports ready with Discord connected.

-
-
- -

Ran live PERF-003 OData proof perf-003-projection-fast-path-proof-20260517041730.md against production. It created File fl-019e3427-4b90-7801-be94-d4af896aa315, performed three PUT $value writes at 349.2 ms, 366.5 ms, and 344.8 ms, verified exact GET $value readback after each write, and verified direct File reads with min 54.9 ms, p50 65.3 ms, and max 71.9 ms.

-

Correctness proof passed: the File reached Ready with VersionCount=3 and last version 019e3427-58a6-7331-9614-b6982a5347dc; FileVersions were Superseded, Superseded, and Current; filtered OData reads by Name and Path found the proof row. Lowercase name returned zero rows, which is an API ergonomics caveat rather than a projection failure.

-
-
- -

Completed direct production database proof through Railway Postgres. entity_catalog contains the proof File at sequence 7, status Ready, projection version 2, version_count=3, and the expected last version id. entity_field_index contains 14 indexed File rows for the proof entity, and the three FileVersion catalog rows preserve the expected previous_version_id chain. FileVersion file_id index count is 3.

-

Datadog proof is useful but still a small sample. Current-version c16 metrics show query_projection_upsert transaction p95 buckets between 12.9 ms and 385.6 ms, event_append p95 between 12.7 ms and 25.2 ms, and reconciliation-path counts diff=233, insert=25, skipped_unchanged=17. Projection end-to-end p95 is healthy for File/FileVersion background dispatch at about 50-73 ms, while Session/background_dispatch remains the next suspicious residual at about 245-386 ms.

-
-
- -

Queried Datadog profiling and DBM. No fresh continuous profiler upload series appeared in the latest window; the Railway capability endpoint reports TEMPER_DDPROF_ENABLED=false, ddprof_present=true, perf_event_paranoid=3, and CAP_PERFMON=false, so profiling should be treated as targeted/canary evidence on Railway until the platform permission gap is solved or documented.

-

DBM is now giving actionable current-version samples with service:temperpaw, database_instance:temperpaw-postgres, and version:c16e0201.... Samples include low-duration entity_field_index upserts and one SELECT status, projection_hash FROM entity_catalog ... FOR UPDATE sample with about 56 ms transaction time and a transaction-id lock wait during concurrent proof traffic. That keeps the next work measured: longer c16 observation first, then the Session/background_dispatch and catalog-lock residuals.

-
-
- -

Merged the dashboard evidence PR nerdsane/temper#247 after CI run 25981859177 passed. The merge commit is 37e269ed217a6c3f066717f6b7ec4adc98ae43a1, so main now contains the PERF-003 live proof, Railway deployment evidence, Datadog/DBM proof, direct production DB proof, and updated task matrix.

-
-
- -

Started PERF-003B from fresh main-based worktree /Users/seshendranalla/Development/temper-worktrees/latency-session-projection-shape-20260517 on branch codex/latency-session-projection-shape-20260517. Latest Datadog query confirmed the next measured residual: current c16 Session/background_dispatch projection p95 bins still peak around 386.2 ms, pool acquire is only 1.4-10.1 ms, and reconciliation work is dominated by changed diff paths.

-

Created ADR-0096 for set-based projection index reconciliation. The local implementation keeps ADR-0095 catalog serialization and no-op skip behavior, but changes changed projections from per-field SQL loops to one stale-row anti-join delete plus one batched unnest upsert with a conflict WHERE clause that avoids rewriting unchanged rows.

-
-
- -

PERF-003B local gates are green so far: cargo check -p temper-store-postgres, focused upsert_query_projection tests, full cargo test -p temper-store-postgres, cargo clippy -p temper-store-postgres --all-targets -- -D warnings, cargo fmt --all -- --check, and git diff --check. Added focused tests for unchanged-row preservation, status-only denormalized index updates, long field removal, and clearing stale index rows when scalar fields disappear.

-
-
-
-
- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Evidence checkLatest live resultMeaningNext action
Datadog profiling metricsProduction on-demand profiler routes are live. Idle CPU/wall captures returned HTTP 200 but only 83-byte profiles; an 8-second CPU capture at 199 Hz during 140 authenticated read probes produced a 26.4 KB pprof file, uploaded to the Datadog Agent intake, and yielded fresh profile_type:cpu upload buckets with no upload-error series.Profiling is now useful for targeted CPU investigations when captured under live load. It is still not a continuous always-on surface: TEMPER_PROFILING_CONTINUOUS is unset, native ddprof is disabled, wall counters are not yet trustworthy, and allocation/lock profiles are not proven.Enable continuous profiling or ddprof deliberately, add/verify freshness monitors, capture flamegraph links/screenshots, and use loaded CPU captures for Cedar/AuthZ and projection hot-path work.
Datadog DBM instance healthtemperpaw-postgres is visible in DBM. Datadog ranked it suspicious for transient latency spikes around 2026-05-15 02:23-02:34 UTC, but baseline behavior stabilized afterward; wait events were minimal.The database is not broadly saturated in the sampled window. The transient spikes are real enough to keep watching, but they do not justify speculative query-shape changes without plan and app-side timing evidence.Keep DBM health in the dashboard and separate global DB health from hot query design.
Datadog DBM query plansDBM explain-plan repair was applied on 2026-05-15. Post-deploy Datadog returned 111 fresh plan records in the latest hour for temperpaw-postgres, including app plans tagged with version:2b0ec6e9849d5eabacb480678a9bfc92fff6f1c4. Fresh plans include real payloads for entity_catalog, entity_field_index, and events; examples include idx_entity_catalog_type, idx_efi_lookup, entity_catalog_pkey, idx_events_tenant_entity, and events_tenant_entity_type_entity_id_sequence_nr_key.Explain plans are now usable for hot-query design. Remaining DBM gaps are schema collection, occasional prepared-statement notices, and one long Datadog-agent query truncated by track_activity_query_size=1024.Keep watching fresh plan samples for invalid_schema, enable schema collection if supported by the Railway agent config, and decide whether track_activity_query_size should be raised.
Datadog DBM availability monitorLive Datadog showed postgresql.queries.count and plan samples current while datadog.dbm.activity_rows had no samples in the last hour. Replaced the activity-row availability monitor with [TemperPaw] Postgres DBM Query Metrics Missing using postgresql.queries.count; the new monitor is OK.The old monitor was an observability false positive: it treated sparse DBM activity sampling as an integration outage. DBM activity rows remain useful for sampled correlation, but query-count and plan telemetry are better liveness signals.Keep the new monitor live, use samples/plans for diagnosis, and keep schema collection as the remaining DBM instrumentation gap.
APM endpoint latency refreshOver the latest live two-hour window, long-lived event stream spans were 45-60 s but almost entirely idle_ns. Excluding those streams, file $value writes were about 2.1-2.4 s, one WASM probe was about 1.3 s, policy creation was about 0.4 s, and regular OData/list reads were mostly 6-20 ms.The prior diagnosis still applies: ordinary reads are already fast on the projection plane, while blob/file writes, integration/WASM work, and selected control-plane work need separate optimization lanes. Raw APM averages must exclude long-lived streams or they overstate server work.Add dashboard filters/notes that separate streaming spans from request/response spans; keep slow-file and WASM paths in the next optimization queue after measurement deploy.
Datadog percentile configurationApplied scripts/configure_metric_percentiles.py --apply through Railway before and after runtime deployment. Datadog now returns percentile-enabled post-deploy series for temper_cedar_evaluation_duration_ms, temper_cedar_evaluation_phase_duration_ms, temper_postgres_pool_acquire_duration_ms, temper_postgres_transaction_duration_ms, temper_query_projection_update_queue_wait_ms, and temper_query_projection_update_end_to_end_duration_ms.The p95/p99 path is now usable for the newly deployed runtime metrics, not just the pre-existing averages. Remaining skipped metrics are workflow-dependent surfaces that need traffic or scheduled parity/backfill execution.Keep the percentile script in the release checklist and rerun it after projection parity/backfill workflows emit their first samples.
DBM query cost refreshLive DBM query-performance analysis showed entity_catalog batch lookups are high volume but usually sub-millisecond database latency; tenant-count scans average around 8.9 ms and touch about 3,846 shared blocks; field-index filtered reads are tens of microseconds; sampled entity_field_index upserts are tens of microseconds with one transaction-duration sample around 15.8 ms.Database-side cost is measurable and currently not the whole explanation for the slowest file writes. The likely next wins are app-side transaction shape, write amplification/coalescing, blob path timing, and stream-aware latency accounting.Deploy app-side Postgres timing metrics, then correlate slow APM traces against DBM signatures before rewriting storage paths.
OBS-002 repair packageAdded scripts/datadog-postgres-dbm-setup.sql and docs/runbooks/datadog-postgres-dbm.md based on Datadog's Postgres DBM requirements for datadog.explain_statement, role grants, pg_stat_statements, and search path.The most likely invalid_schema cause now has an explicit operator fix path and validation query set.Run against staging/live Postgres and verify Datadog explains hot query signatures without plan.collection_errors.
OBS-002 app-side Postgres metricsAdded and deployed temper_postgres_pool_acquire_duration_ms, temper_postgres_transaction_begin_duration_ms, temper_postgres_transaction_commit_duration_ms, temper_postgres_transaction_duration_ms, temper_postgres_operation_outcomes_total, temper_postgres_projection_index_fields, and temper_postgres_projection_skipped_index_fields_total. Latest p95 pool acquire is about 2.4 ms; p95 transaction duration is about 75 ms for query_projection_upsert and about 19-25 ms for event_append.DBM can show database-side behavior, while Temper can now separate pool waiting, transaction overhead, write amplification, and failed/concurrent operation outcomes. Current evidence points away from pool starvation and toward projection transaction/update work.Correlate the next slow projection/write traces against these app-side operation metrics before changing storage shape.
OBS-003 projection correctness metricsAdded and deployed source-tagged projection metrics. The previous projection proof returned File projection p95 of 42.6 ms for source:create and 91.1 ms for source:background_dispatch. The PERF-005B proof window emitted File projection throughput counters: 30 update-started and 24 update-enqueued samples. The broader set includes applied sequence, backfill, shadow-check, replay parity, and sequence-gap metrics.Projection writes are now observable by source, catalog-fast reads can opt into deterministic sampled shadow checks via TEMPER_ODATA_CATALOG_SHADOW_READ_EVERY, and active projection rows can be compared against event-replayed authoritative state. PERF-002 improved write amplification and live DB proof confirms the indexed catalog state for the proof rows, but replay/shadow metrics did not emit recent samples.Turn replay parity and sampled shadow reads into scheduled production checks so correctness remains continuously visible instead of only probe-driven.
Railway production accessRailway CLI is logged in and linked to production. The service is openpaw, but production exports DD_SERVICE=temperpaw. Datadog key variables are present in the service environment without exposing secret values.Datadog deploys now use railway run, but query scopes must follow the runtime service tag, not the Railway service name.Use Railway for the remaining DBM SQL/runbook validation, runtime deployment checks, and live e2e proof on the Railway domain.
Datadog monitor deploymentDashboard mn4-k3k-i66 was updated again for PERF-005C. Native blob transport percentile configs were created for duration/request bytes/response bytes. The monitor catalog now includes native blob duration spike id 283877764 and native blob p95 regression id 283877770, alongside the standard APM request-rate id 283342070, 5xx id 283342074, duration id 283342076, and error-rate id 283345345.The Datadog dashboard/monitor source now exists in production Datadog and covers the formerly blind native object-store boundary. Service coverage analyzer recognizes rate, duration, and error coverage for service:temperpaw.Keep source-of-truth monitor deployment in the release checklist, add SLO objects after latency targets are finalized, and keep native blob percentiles in every future File/data-plane rollout proof.
Production health probehttps://openpaw-production.up.railway.app/readyz returned ready JSON after PERF-003 deployment 6e42424f-7ad4-4577-b127-9e3a0a36abeb, including Discord connected. Authenticated /paw/version returns sha-c16e020 with SHA c16e0201c1490e0496f4964f5c72b704bb8cd216. The deployment uses ghcr.io/nerdsane/temperpaw:sha-c16e020; the image digest is sha256:9c8a844fdfe52a201adea5755a3d5d61c498d8ba6cdc84c2f64a151d3bf27c89.The Railway app is alive on the Railway domain and version-correct for the merged PERF-003 rollout. A version-variable mismatch and accidental config/source deploy were found and corrected before final proof traffic, so Datadog tags now match the deployed code.Keep immutable image tags for proof deployments and automate BUILD_SHA/DD_VERSION/DD_GIT_COMMIT_SHA/OTEL version updates so Datadog tags cannot drift on future rollouts.
Datadog metric snapshotPERF-003 metrics are live under version:c16e0201c1490e0496f4964f5c72b704bb8cd216. The post-deploy window shows temper_postgres_projection_index_reconciliations_total path counts diff=233, insert=25, and skipped_unchanged=17. query_projection_upsert transaction p95 is currently a tiny sample, with buckets from 12.9 ms to 385.6 ms; event_append p95 is 12.7-25.2 ms; File/FileVersion projection e2e p95 is about 50-73 ms; Session/background dispatch remains about 245-386 ms.The core service, current-tail distributions, Postgres app timing, projection timing, trace sampler metrics, DBM plans, targeted CPU profiler path, AuthZ candidate metrics, projection diff-index proof metrics, background reaction spans, native blob transport metrics, and new reconciliation-path counters are live under service:temperpaw. The next observability gap remains continuous shadow/replay parity emission and truly useful continuous profiler uploads.Run a longer c16 observation window, activate scheduled projection replay parity and sampled shadow reads, then choose the next latency slice from Session/background_dispatch or catalog-lock residuals.
Production observe-only e2eBaseline proof passed at 2026-05-15 16:41 EDT with trace 20be98473ad99f787fa0d007f36b9b7b. Post-fast-path proof passed at 2026-05-15 19:56 EDT with trace f3723cc99d48e9a65d5cc7ac4b61fc23. Corrected AuthZ rollout proof passed at 2026-05-16 10:12 EDT. Projection diff proof projection-diff-live-proof-20260516191623 passed at 2026-05-16 15:16 EDT. Background reaction proof file-reaction-background-live-proof-20260516215205 passed at 2026-05-16 17:52 EDT. Native blob transport proof blob-transport-live-proof-20260517001121 passed at 2026-05-16 20:11 EDT. Native-first read proof blob-read-key-order-live-proof-20260517022158 passed at 2026-05-16 22:21 EDT. PERF-003 proof perf-003-projection-fast-path-proof-20260517041730.md passed at 2026-05-17 00:17 EDT with three File $value versions, exact readbacks, FileVersion chain validation, Datadog reconciliation metrics, DBM samples, and direct production DB correctness proof.The proof exercises production auth, OData dispatch, event append, projection update, blob/object storage, read-after-write, FileVersion creation, Datadog APM/logs/native blob metrics, reconciliation-path metrics, DBM, and the projection lookup path. It now proves correctness and version-tagged observability for every shipped latency slice through PERF-003.Promote this into a repeatable smoke/load script and add a small load loop so future before/after p95/p99 values have enough samples, not just one proof bucket.
AuthZ candidate filterTemper PR #239 and TemperPaw PR #270 are merged, CI/Docker are green, Railway deployment 8ad1c472-1c63-4473-a9ed-10f7458df253 is live, and Datadog shows candidate metrics under version:7726e4dfba5453c337dafb7411f88c239f3d5bd5. The live proof reduced policy volume from 16,497 to 48 candidates per evaluation. PUT trace 0878a09bf0de8803b19bac5b6764744f still shows authz.Create at 19.6 ms and authz.RecordVersion at 26.1 ms.This is a correctness-preserving optimization that moved the bottleneck: Cedar evaluator time is now small, but per-request candidate selection/rebuilding remains visible. The design direction is right; the implementation needs an index/reuse layer to become "really fast."Create the next main-based Temper worktree, write an ADR for static policy candidate indexes or reusable candidate policy sets, prove conservative invalidation, and target policy_candidates p95 under 2 ms without any decision cache.
File $value fast pathBefore: trace 20be98473ad99f787fa0d007f36b9b7b, HTTP server span 3,159.5 ms, PUT /odata/{path} 3,159.4 ms, PUT $value 3,159.4 ms, wasm.invoke 516.1 ms, and R2 host PUT 220.8 ms. After: trace f3723cc99d48e9a65d5cc7ac4b61fc23, HTTP server span 353.3 ms, PUT /odata/{path} 353.2 ms, PUT $value 353.2 ms, native state.put_file_stream_content.native 353.1 ms, no matching wasm.invoke/blob_adapter span, and direct curl upload wall time 412.4 ms.This is the first completed actual latency win: about 8.9x faster by Datadog server-span timing and about 7.7x faster by direct proof wall timing, while preserving byte hash equality, StreamUpdated, FileVersion behavior, projection status, and audit/event flow.Keep this fast path deployed, add a reusable proof script, then move to the next measured latency slice rather than widening the data-plane change prematurely.
Review gatesDST review PASS, code-quality review PASS, changed sim-visible files passed the determinism guard loop, Temper pre-push passed rustfmt/clippy/readability/full workspace tests, TemperPaw pin-bump local validation passed, GitHub CI passed for all shipped rollout PRs, TemperPaw main CI passed, Docker main build pushed the deployed image digest, Railway deployed it, live e2e proofs passed, Datadog current-version evidence exists, native blob monitors/percentiles are deployed, and direct DB correctness checks passed for the projection, FileVersion, and native blob proof slices.Five latency-improvement packages plus one key observability package are merged, CI-proven, deployed, and backed by production evidence. Residual risk has moved to choosing and safely executing the next latency slice while making projection parity checks continuous.Start the next latency-improvement PR from main in a new worktree, write an ADR first if the change alters architecture, and keep this dashboard updated with before/after evidence.
OBS-004 Datadog dashboard and monitorsThe refreshed TemperPaw source has a deployed dashboard combining main's session/LLMObs/DBM/log restructuring with latency-program projection, Postgres, dispatch, WASM, blob, Monty, AuthZ, native blob transport, and trace-budget coverage. Queries are scoped to production's service:temperpaw tag, and native blob percentile panels now have live samples.The live Datadog account has the merged target config plus post-deploy runtime data. Some workflow-specific panels can remain sparse until parity/backfill/profiler schedules run.Capture screenshots/links for the next review packet and add formal SLO objects after targets are approved.
OBS-005 trace budgetAdded and deployed ADR-0083, sampler decision/config/rate metrics, startup-tunable reduced sampling for WASM auxiliary and background dispatch helper spans, reaction fanout summary span fields, and a TemperPaw Trace Budget dashboard/monitor group. Datadog now finds temper_trace_sampler_decisions_total, temper_trace_sampler_configured_rules, and temper_trace_sampler_reduced_sample_rate_pct.Trace volume now has an explicit, observable budget instead of relying on ad hoc trace-agent behavior; high-fanout traces should retain root and summary context without routine 100k-child-span payloads.Run a high-fanout smoke path and record representative trace span counts plus slow/error trace links here.
PERF-001 measurement prepDatadog now shows live p95 for temper_cedar_evaluation_duration_ms and temper_cedar_evaluation_phase_duration_ms. Latest p95 is about 50 ms overall and about 50 ms in phase:authorizer; action_uid, entities, request, resource_attrs, context_attrs, resource_uid, and principal_attrs are all tiny by comparison.We can now separate Cedar authorizer cost from Temper request-construction cost before optimizing or proposing a deny-safe cache. The first evidence says policy/entity evaluation dominates.Use targeted profiling around Cedar authorizer execution, then decide between policy/entity reuse, compiled-entity strategy, or a separate deny-safe cache ADR.
Release verification packageAdded docs/runbooks/latency-observability-release.md with the two-PR split, exact local checks, runtime flags, DBM setup, Datadog deploy commands, live proof matrix, e2e smoke requirements, rollback plan, and done criteria. Quick/full preflight passed; the two PRs merged; production deploy and Datadog proof are recorded here.The work has a concrete path from local patches to deployed evidence. The representative production e2e proof is now complete and independent of any specific worker implementation.Turn the proof into a reusable verification script and run it as the before/after check for upcoming latency PRs.
OBS-003 Turso projection catalog correctnessFixed Turso entity_catalog to preserve full projected fields JSON, added an idempotent migration for existing catalogs, and updated catalog load/export paths. Focused Turso tests and the server replay parity integration test pass.The catalog-fast read contract now matches the Postgres path: the catalog owns the full projected row, while the field index remains a scalar filter index.Deploy, backfill/rewrite existing Turso projection rows, then verify replay parity is clean before increasing fast-read traffic.
OBS-001 local implementationAdded and deployed profiler config/freshness/capture metrics, opt-in continuous CPU profile capture, and startup wiring from the CLI serve path. Production still has TEMPER_PROFILING_CONTINUOUS unset and DD_PROFILING_ENABLED=false.On-demand in-process pprof is proven live and useful under load, and the deployed runtime can now support a controlled continuous-capture experiment. It is not yet an always-on profiling posture.Enable continuous capture intentionally or compare against native ddprof, then record flamegraph links/screenshots.
Datadog native profiler pathProduction currently has DD_PROFILING_ENABLED=false, so scripts/temperpaw-entrypoint.sh does not wrap the process in ddprof. Current Datadog docs recommend ddprof for compiled languages including Rust, C, and C++ when runtime, symbols, OS permissions, service/env/version, and Agent/export settings are correct.The in-process profiler is a proven fallback/control for targeted captures, but ddprof remains the likely first-class continuous profiler for Temper's Rust process if Railway permissions and image contents support it.Add a deployment task to install/wrap with ddprof, verify perf_event_paranoid/capabilities and symbols, then compare native continuous profiles with the in-process fallback.
OBS-001 local verificationcargo check -p temper-cli, cargo test -p temper-server profiling::tests --lib, cargo test -p temper-server query_projection_metrics --lib, cargo test -p temper-server odata::read_support --lib, cargo test -p temper-store-postgres metrics --lib, cargo fmt --check, git diff --check, JS syntax check, and Playwright screenshot smoke check passed.The local code path compiles and profiler tuning behavior is covered by unit tests; production on-demand CPU profile capture and Datadog Agent upload are now live-proven.OBS-001 is not done until continuous or native profiling is deployed, flamegraph evidence is recorded, and stale-profiler monitoring stays clean.
-
- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
MilestoneStatusEvidence required in this dashboardExit criteria
Program setupBaseline completeThread goal, worktree, branch, ADR link, progress dashboard.ADR-0081 exists and this page records the plan/status baseline.
Observability repairRuntime metrics liveProfiler freshness, DBM plan health, percentile metrics, trace budget, projection drift/lag/replay metrics.Dashboards/monitors show trustworthy current data, not stale or average-only proxies. Remaining nuance: continuous/native profiling and scheduled projection parity still need a follow-up pass.
Performance slice 1First AuthZ slice liveCedar/AuthZ phase metrics, request-shape metrics, candidate-count metrics, percentile config proof, tests, deployment, and live latency evidence.AuthZ candidate filtering is merged and deployed. It reduced policy volume from 16,497 to 48 candidates per sampled evaluation, but live evidence now points at policy_candidates selection/rebuild overhead as the follow-up target.
Performance slice 2Live proof completeADR-0091, Postgres projection diff-index implementation, storage tests, before/after Datadog projection latency/write-count proof, direct DB correctness proof, and replay/shadow gap classification.PERF-002 is merged, rolled out, deployed, live-proven, Datadog-proven, and direct-DB-proven. Remaining follow-up is to run replay/shadow parity continuously rather than probe-only.
Performance slice 3Live proof completeADR-0092, bounded background File reactions, focused tests, clippy/check, PR/CI, TemperPaw rollout, Railway deployment, live File proof, Datadog before/after, and direct File/FileVersion correctness.PERF-005B is merged, deployed, live-proven, Datadog-proven, and direct-DB-proven. The proof shows File $value response latency improved while bytes remain immediately readable and the FileVersion/RecordVersion chain converges correctly.
Structural slicesFirst data-plane slice liveWorkflow executor/blob data-plane ADRs, implementation PRs, live run evidence.ADR-0088, local tests, GitHub CI, Temper/TemperPaw PR merges, Railway deployment, and Datadog before/after proof show built-in File uploads no longer force a WASM blob-adapter invocation. ADR-0093 now shows the remaining native object-store transport cost directly. Remaining structural work is broader data-plane design and workflow executor latency.
Final verificationFive latency e2es plus blob-observability proof passedLocal tests, live runs, live e2e tests, PR links, merge proof, deployment proof.Observability code, five actual latency-improvement PR sets, and the native blob transport observability rollout are merged and deployed; this dashboard contains the evidence trail and production File/OData read-after-write proofs. Final goal completion still requires the next evidence-driven latency PRs, plus continuous profiling/parity, to merge, deploy, and prove before/after wins.
-
- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Task IDTaskStatusWorktree / PREvidence required before done
PLAN-001Create living dashboard and program tracking baseline.Donecodex/latency-observability-programThis progress section, progress bar, status log, and milestone table exist.
ADR-0081Define latency/observability acceleration program, gates, order, and finish line.Donecodex/latency-observability-programdocs/adrs/0081-latency-observability-acceleration-program.md exists.
OBS-001Restore useful Datadog profiling: freshness, upload rate, upload errors, CPU/allocation/lock profiles.Runtime deployed; continuous pendingTemper PR #229 merged; deployed via TemperPaw #266Local check/test passed, production on-demand CPU capture under read load produced a valid 26.4 KB pprof profile plus fresh Datadog upload counters, and runtime deploy is live. Remaining profiler work is continuous or native ddprof evaluation, allocation/lock or documented Rust limitation, and durable flamegraph links/screenshots.
OBS-002Repair DBM usefulness: query plans, DB CPU/vacuum coverage, lock/wait/pool/transaction visibility.DBM and app metrics liveTemper PR #229 merged; deployed via TemperPaw #266Runbook/setup SQL were applied to production Postgres, DBM plans are live, and app-side Postgres metrics now emit. Latest p95: pool acquire about 2.4 ms, projection transaction about 75 ms, event append about 19-25 ms. Remaining work: DBM schema collection decision, DB CPU/vacuum/wait coverage classification, and continued clean plan samples.
OBS-003Add projection correctness observability: lag, sequence, version/hash, drift samples, shadow reads, replay parity.Runtime metrics live; parity run pendingTemper PR #229 merged; deployed via TemperPaw #266Source-tagged queue/update metrics emit in production. Latest p95: projection queue wait under 0.04 ms and projection end-to-end update about 75-102 ms. Opt-in shadow checks, replay parity metrics/tests, and Turso catalog field preservation exist; scheduled/live replay parity execution against deployed data remains pending.
OBS-004Convert key metrics to distributions/percentiles and fix misleading or No Data monitors.Runtime percentiles liveTemperPaw PR #266 mergedDashboard mn4-k3k-i66 and the refreshed 76-monitor catalog are deployed live; 41 legacy/orphan monitors were reconciled away. Datadog service coverage recognizes APM rate, duration, and error. The post-deploy percentile pass enabled the new runtime metrics that were previously skipped: Cedar ms/phase, Postgres pool/transaction, and projection queue/e2e timings.
OBS-005Add trace budgets and high-fanout summary spans with exemplars.Sampler metrics liveTemper PR #229 merged; TemperPaw PR #266 mergedADR-0083, sampler metrics, configurable reduced-prefix rules, reaction fanout span summaries, dashboard widgets, and monitors are deployed. Datadog now finds temper_trace_sampler_decisions_total, temper_trace_sampler_configured_rules, and temper_trace_sampler_reduced_sample_rate_pct. Done requires high-fanout smoke evidence that routine traces no longer create 100k-span payloads while slow/error traces retain root and summary detail.
PERF-001Measure then optimize Cedar/AuthZ CPU path without weakening governance.Live; follow-up target foundADR-0089; Temper PR #239 merged as b0b898a2; TemperPaw PR #270 merged as 7726e4df; CI runs Temper / TemperPaw PR / TemperPaw main / DockerADR-0084 proved the old authorizer phase dominated; ADR-0089 implements the first safe optimization: per-request Cedar candidate policy sets using principal/action/resource scope constraints only. Local semantic tests, focused server check, focused clippy, full pre-push pipeline, GitHub CI, TemperPaw pin checks, main CI, Docker, Railway deploy, and live proof all pass. Datadog now shows the candidate-count win and the next bottleneck: phase:policy_candidates selection/rebuild time.
PERF-001BMake AuthZ candidate selection itself fast through indexed/reusable policy subsets.Live; narrow follow-upADR-0090 and branch codex/latency-authz-candidate-index-20260516 merged through PR nerdsane/temper#240. CI run 25965850115 passed and Temper main now contains 84e95c6677f4a03364dea796cc67237bbc8275ce. TemperPaw rollout PR #271 passed PR CI run 25966397547 and merged as 1118183f; main CI and Docker are green with image sha-1118183.Precomputed per-policy-set index preserves Cedar semantics: no decision cache, no skipped forbids, default deny intact, conservative fallback on unsafe policy sets, explicit rebuild on policy reload. Railway deploy ee997c77-7abb-4682-aa76-4a5155f05c1f, live e2e proof authz-index-live-proof-20260516164954, and Datadog proof are complete. Outcome: warm policy_candidates p95 dropped from about 13.4 ms to 0.26-0.34 ms, but the heavy File proof bucket reached 2.685 ms; keep a small follow-up if the hard target remains sub-2 ms for every heavy path.
PERF-002Reduce projection write amplification with diffing, batching, coalescing, and selective indexing.Live proof completeADR-0091; Temper PR nerdsane/temper#241 merged as ed68e785; TemperPaw rollout PR nerdsane/temperpaw#272 passed PR CI run 25969575912, merged as 73e756a0bdb5f9a3084c5dd33c424154871fb873, passed main CI 25969887628 and Docker 25969887617, then deployed to Railway as 72635a45-5744-4c72-82e0-7b7143e9a548.Temper local, full pre-push, and GitHub CI passed. TemperPaw rollout local checks, tests, WASM build, dashboard build, PR CI, merge, main CI, Docker, Railway deployment, live File/OData proof, Datadog current-version projection metrics, and direct production DB correctness proof passed. Residual gap: replay/shadow parity metrics are implemented but did not emit recent samples, so they need scheduled/live activation.
PERF-003Improve DB transaction shape and event append path where evidence justifies it.Live proof completeADR-0095 and Temper PR #245 merged as 6439a8a0. TemperPaw rollout PR #276 passed PR CI 25980160694, merged as c16e0201, then passed main CI 25980450147 and Docker 25980450153.Railway deployment 6e42424f-7ad4-4577-b127-9e3a0a36abeb is live and version-correct. Live File/OData proof, direct production DB proof, Datadog transaction/reconciliation metrics, and DBM/APM correlation are complete. Remaining work is not rollout: collect a longer c16 post-deploy window before claiming tail improvement, then decide whether the next DB slice is Session/background_dispatch, catalog locking under concurrency, or event append sequence shape.
PERF-003BMake changed projection reconciliation set-based so Session/background_dispatch stops paying per-field SQL round trips.Local implementationADR-0096; active branch codex/latency-session-projection-shape-20260517 in worktree /Users/seshendranalla/Development/temper-worktrees/latency-session-projection-shape-20260517.Local store checks are green. Remaining gates: full Temper pre-push, Temper PR/CI/merge, TemperPaw pin bump, PR/main/Docker, Railway deploy, live OData/File and Session-oriented proof, Datadog before/after proof for Session/background_dispatch and query_projection_upsert, plus direct production projection correctness proof.
PERF-004Move long integration workflows to durable, observable execution with idempotent callbacks.PlannedPending ADR/PRControl-plane response latency improves; live workflow e2e passes; duplicate side effects prevented.
PERF-005Split blob/data plane with streaming or direct upload while preserving Temper metadata and verification.First fast path deployedTemper PR #238 merged; TemperPaw PR #269 merged; Railway deployment d950fb9e-feb7-498f-b794-665fce9913bcADR-0088 exists. Temper local checks, pre-push, GitHub CI, and merge are complete. TemperPaw pin-bump validation, CI, merge, main CI, Docker, Railway deployment, health/version probes, live read-after-write proof, and Datadog before/after trace evidence are complete. Remaining data-plane work is to turn the proof into a reusable script and decide whether further direct-upload/streaming architecture is justified by evidence.
PERF-005BRemove post-commit File reaction fanout from the synchronous native File $value response path.Live proof completeADR-0092 and branch codex/latency-file-value-residual-20260516 merged through Temper PR nerdsane/temper#242 as 1796c4f0. Rollout branch codex/bump-temper-file-value-residual-20260516 merged through TemperPaw PR #273 as e8457ca4 after PR CI run 25972634147. Main CI 25972955841 and Docker 25972955833 are green; Docker image digest sha256:4956e8d7ed26af68168b64643b6aefc0e03e5cc114cbb8377382649c3a9eadfa. Railway deployment 5778571f-b325-4237-a792-cb13342963a4 is live and version-correct.Temper local/pre-push/CI are green and merged. TemperPaw rollout local checks, PR CI, main CI, Docker, Railway deploy, health probe, version probe, live File proof, Datadog evidence, and direct DB correctness proof are green. The sampled proof window shows PUT $value p95 238.7 ms versus previous 490.3 ms, with reaction.dispatch.background running after the HTTP span.
PERF-005CInstrument native blob transport duration, status, and bytes so the remaining File $value residual can be attributed before the next data-plane architecture change.Live proof completeADR-0093 and branch codex/latency-blob-transport-observability-20260516 merged through Temper PR nerdsane/temper#243 as 88c9d797 after CI run 25974966315 passed. TemperPaw rollout PR nerdsane/temperpaw#274 passed PR CI run 25975556316 and merged as 9fcd4b2b. Main CI 25975834289 and Docker 25975834282 passed; Docker image digest is sha256:74d398d56362de4b31cf09334e975952ec8f455a29e92a80fee99a6d5edf65bb.Temper local validation, pre-push gates, GitHub CI, and merge are green. TemperPaw rollout local validation, PR CI, main CI, Docker, Railway deploy 37edf6ad-5f71-4865-85e3-46a05dd7cc78, live proof blob-transport-live-proof-20260517001121, Datadog native transport proof, dashboard/monitor/percentile deploy, and direct DB correctness check are green. Outcome: the residual File $value path is now attributable down to native object-store transport.
PERF-005DRemove the extra legacy external-key 404 from fresh native File reads by preferring the native temper-fs/{content_hash} blob key while preserving legacy WASM fallback.Deployed/provenADR-0094 and branch codex/latency-file-blob-read-key-order-20260517 merged through Temper PR #244 as ff79974d after CI run 25977404056 passed. TemperPaw rollout PR #275 passed PR CI run 25977831129 and merged as 821d6c6e. Main CI 25978275903 and Docker 25978275890 passed. Railway deployment b628e727-b013-4c89-9eaa-f65f4f7e7433 is live and version-correct.Temper local checks, full pre-push gate, GitHub CI, TemperPaw rollout local checks, PR CI, main CI, Docker, Railway deploy, health/version probes, live proof blob-read-key-order-live-proof-20260517022158, Datadog no-extra-404 proof, and DBM sample proof are green. Legacy fallback is still covered locally; a live legacy-only fixture remains optional if we want production proof for historical WASM-only blobs.
VERIFY-001Build repeatable load/replay/e2e verification and deployment evidence trail.Representative e2e passed; repeatable script pendingTemper PR #229 merged; TemperPaw PR #266 merged; final evidence report Temper PR #230 merged; auth checkpoint Temper PR #231 merged; auth retry Temper PR #232 merged; auth-path Temper PR #233 merged; local-proof blocker Temper PR #234 merged; current blocker report Temper PR #235 merged; second auth-timeout report Temper PR #236 merged; correction report Temper PR #237 merged; File fast-path Temper PR #238 merged; rollout TemperPaw PR #269 mergedLocal tests, GitHub CI, Docker image push, Railway deployment, health probes, version proof, Datadog monitor coverage, APM spans, DBM plans, runtime metrics, baseline production OData/File proof, and fast-path production OData/File proof are recorded. Remaining verification work is to make the proof repeatable and run it before/after each latency PR.
-
-
- -
-
-
00 / direct answer
-

Does The Previous Diagnosis Still Apply?

-
-
-
-

Yes, with a production update

-

- The central conclusion still holds: Temper's verified transition table and actor admission path are not - the main latency limit in the observed production system. The slow areas are around the mission-preserving - core: authorization, projection/index maintenance, event persistence shape, WASM integrations, blob - transfer, and observability cardinality. -

-

- The important update is that production Datadog traces show a newer projection read plane: - entity_catalog, entity_field_index, and - dispatch.phase.query_projection. Those symbols are not present in this local checkout, - so Datadog is the fresher source for production behavior. -

-
-
-

Measured approach

-

- We should treat speed as a correctness program. First repair measurement: profiling, DBM, percentile - metrics, trace budgets, and projection drift observability. Then optimize the CPU and write-amplified - paths with repeatable load tests and replay parity. -

-
- keep verified specs - keep Cedar default-deny - keep event audit - bound projection lag - control trace cardinality - make profiling continuous -
-
-
-
- -
-
-
01 / current system
-

Current Architecture

-
-
-
Current platform architecturerepo + prod delta
- 
-flowchart LR
-  Dev["Developer Chat"] --> SpecGen["Interview -> IOA / CSDL / Cedar / WASM"]
-  SpecGen --> Verify["Verification Cascade
L0 symbolic
L1 model check
L2 deterministic simulation
L3 property"]
-  Verify --> Registry["SpecRegistry
(TenantId, EntityType) -> spec + TransitionTable"]
-  User["Production Chat / Agents / SDKs"] --> Server["Temper Server
HTTP + OData + MCP"]
-  Registry --> Server
-  Server --> Authz["Cedar AuthZ
principal + action + resource + context"]
-  Server --> Actors["Entity Actors
verified transitions
bounded mailboxes"]
-  Actors --> Events["Event Store
tenant-scoped Postgres"]
-  Actors --> Wasm["WASM Integrations
sandbox + host calls"]
-  Wasm --> External["External APIs
blob store
LLM/tool providers"]
-  Events --> Projection["Production projection plane
entity_catalog + entity_field_index"]
-  Projection --> Reads["OData read/query path"]
-  Server --> Observe["temper-observe
WideEvents + spans + metrics + trajectories"]
-  Observe --> Evolution["Evolution Engine
O / P / A / D / I records"]
-  Observe --> Datadog["Datadog
APM + DBM + metrics + monitors + profiler"]
-
-
- -
-
-
Read pathprojection first
- 
-sequenceDiagram
-  participant C as Client
-  participant O as OData route
-  participant A as Cedar/Auth
-  participant P as Projection DB
-  participant E as Entity actor
-  C->>O: GET /odata/{path}
-  O->>A: authorize request
-  alt projection hit
-    O->>P: SELECT entity_catalog / entity_field_index
-    P-->>O: projected entity or query result
-  else fallback / hydration
-    O->>E: get_or_spawn + GetState
-    E-->>O: entity state
-  end
-  O-->>C: OData response
-
-
-
-
Action/write pathgoverned transition
- 
-sequenceDiagram
-  participant C as Client
-  participant D as Dispatch
-  participant Z as Cedar
-  participant E as Entity actor
-  participant J as Event journal
-  participant Q as Projection updater
-  participant W as WASM integration
-  C->>D: POST action / PUT stream
-  D->>Z: authorize_with_context
-  Z-->>D: allow or deny
-  D->>E: ask/action
-  E->>J: append events
-  E-->>D: new state + effects
-  D->>Q: update projection
-  D->>W: inline or background integration
-  W-->>D: callback action
-  D-->>C: response
-
-
-
-
- -
-
-
02 / production signals
-

Datadog Evidence Snapshot

-
-
-
- Scope. - Datadog was queried for temperpaw on May 14-17, 2026. The first pass used 24-hour - APM/DBM evidence; the latest refresh used live deployment-corrected APM, metric, monitor-coverage, - DBM query-performance, Datadog monitor deployment, percentile-configuration, and post-PERF-003 deployment windows. - Production observability includes projection symbols missing from this local checkout, - and production currently emits DD_SERVICE=temperpaw with version - c16e0201c1490e0496f4964f5c72b704bb8cd216. -
-
- -
-
OData reads
7 / 17 / 30 ms

GET /odata/{path} p50 / p95 / p99.

-
Projection e2e
386 ms

Latest c16 Session/background_dispatch p95 peak; File/FileVersion sit closer to 50-73 ms.

-
AuthZ CPU
81 / 203 / 253 ms

authorize_with_context p50 / p95 / p99.

-
WASM integrations
4.4 / 19 / 50 s

background_wasm_integrations p50 / p95 / p99.

-
Fan-out trace
166k spans

One trace had 104k field-index inserts.

-
Reconcile paths
408 / 64 / 26

Latest c16 counts: diff / insert / skipped_unchanged; this selects PERF-003B.

-
Indexed fields
25-28

Average scalar fields in the hot projection bins, which makes per-field SQL round trips visible.

-
Catalog lock
~56 ms

DBM sampled one bounded FOR UPDATE transaction-id wait under concurrent proof traffic.

-
Profiler
canary only

Targeted CPU captures work; continuous ddprof is packaged but not active/useful on Railway yet.

-
Runtime deploy
c16e020

Railway /paw/version, DD_VERSION, APM spans, DBM samples, and production code paths show the merged PERF-003 rollout.

-
Monitor coverage
3 / 3

Datadog recognizes request rate, error, and duration monitors for service:temperpaw.

-
APM traffic
14,133 spans

Latest one-hour service:temperpaw env:prod query; no matching error spans for deployed version.

-
Dispatch p95
20.5 ms

Latest post-deploy p95:temper_dispatch_ask_latency_ms bucket.

-
Cedar p95
50 ms

temper_cedar_evaluation_duration_ms is now live and percentile-enabled.

-
AuthZ policy count
344x less

Latest corrected proof: 16,497 full policies vs 48 candidates per evaluation.

-
AuthZ phase p95
0.26-2.69 ms

phase:policy_candidates is mostly sub-ms after warmup; the heavy File proof bucket is just above the 2 ms target.

-
Proof trace
62.5 / 65.0 ms

Exact live File create trace p50 / p95 for POST /odata/{path}; projection phase p95 is 25.4 ms.

-
Postgres pool p95
4.1 ms

Small c16 proof sample; pool acquire is still not the primary residual.

-
Session p95
245-386 ms

Session/background_dispatch is the next visible projection-update residual.

-
File write proof
353 ms

Fast-path trace f3723cc99d48e9a65d5cc7ac4b61fc23; down from 3.16 s baseline trace 20be98473ad99f787fa0d007f36b9b7b.

-
DBM plan health
usable

Fresh plans now include real index payloads; schema collection remains a separate gap.

-
E2E proof
passed x8

All shipped slices through PERF-003 have production proof traffic and version-tagged evidence.

-
DB proof
1 / 14 / 3

Latest proof: one File, fourteen File index rows, three FileVersion rows and file_id indexes.

-
Parity gap
no recent data

Replay and sampled shadow metrics exist but returned no recent samples; schedule or trigger them continuously.

-
- -
-
Latency anatomywhere the heat is
- 
-flowchart TB
-  Request["User-visible request"] --> Core["Verified actor core
fast: microseconds to sub-ms phases"]
-  Request --> Auth["Cedar authorization
improved: scoped candidates + index"]
-  Request --> Read["Projected reads
healthy: proof GET p50 65 ms"]
-  Request --> Projection["Projection writes
File healthy; Session residual"]
-  Request --> Event["Event journal
currently modest p95 in c16 sample"]
-  Request --> Wasm["WASM/external workflows
seconds to tens of seconds"]
-  Request --> Blob["Blob path
native transport attributed"]
-  Projection --> Lock["Catalog lock samples
bounded but visible"]
-  Projection --> Drift["Need continuous replay/shadow proof"]
-  Auth --> Profile["Profiling
targeted/canary only on Railway"]
-
-
- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
LayerWhat Datadog showedInterpretationConfidence
Transition table / actor admissionActor spawn and admission p95 were sub-millisecond. Local JIT transition evaluation is microseconds.The mission core is not the current latency limit.High
Read/query planeOData GET p95 was around 17 ms using projection tables.The new projection plane largely fixed regular read latency.High
Projection writesQuery projection tails into seconds during fan-out; one trace emitted 104k field-index insert spans.Write amplification and trace fan-out are primary architecture issues.High
Cedar authorizationAuthZ p95 around 200 ms; sampled trace was almost entirely busy CPU time.Needs profiling, context dimensions, cache/precompile strategy.High
WASM/external integrationsSession integration actions had p50s of seconds and p95s of tens of seconds.External workflow work needs queueing, budgets, and async progress semantics.High
Postgres DBMDBM sees temperpaw-postgres as healthy and fresh post-repair plans include real payloads for entity_catalog, entity_field_index, and events.Instance health and plan-level diagnosis are now usable; schema collection and track_activity_query_size remain follow-up gaps.Medium
ProfilingProduction on-demand CPU profile under read load produced a valid 26.4 KB pprof payload, uploaded to the Datadog Agent, and exposed fresh profile_type:cpu upload buckets.Targeted CPU profiling is useful now; continuous profiling, native ddprof, wall/allocation/lock coverage, and freshness monitors still need completion.Medium
-
-
- -
-
-
03 / diagnosis
-

Architecture Versus Overlooked Costs

-
-
-
-

Architecture constraints to preserve

-
    -
  • Specs are generated from conversation and verified before deployment.
    • -
  • Cedar remains default-deny and auditable.
    • -
  • Events remain durable enough for replay, audit, evolution, and proof.
    • -
  • Tenant isolation remains a first-class key in stores, caches, and metrics.
    • -
  • The deterministic simulation boundary remains clean.
    • -
-
-
-

Likely overlooked costs

-
    -
  • Per-field projection writes and row-by-row index maintenance.
    • -
  • Per-call AuthZ context cloning and Cedar evaluation CPU.
    • -
  • Event append shape with SELECT MAX(sequence_nr).
    • -
  • Blob upload path copying bytes before blob_adapter.
    • -
  • Observability cardinality: huge traces and average-only metrics.
    • -
-
-
- -
-
Cause and remedy mapmission-safe speed
- 
-flowchart LR
-  Slow["Latency pressure"] --> Auth["AuthZ CPU"]
-  Slow --> Proj["Projection write amp"]
-  Slow --> Integrations["External/WASM workflows"]
-  Slow --> Blob["Blob data plane"]
-  Slow --> DB["DB transaction shape"]
-  Slow --> Obs["Observability overhead/gaps"]
-  Auth --> AuthFix["profile -> compile/cache -> avoid clones"]
-  Proj --> ProjFix["diff -> batch -> selective index -> drift monitor"]
-  Integrations --> IntFix["durable executor -> idempotent callbacks"]
-  Blob --> BlobFix["direct upload or true streaming"]
-  DB --> DBFix["pool wait + tx duration + CAS append"]
-  Obs --> ObsFix["distributions + exemplars + trace budgets"]
-  AuthFix --> Mission["Verified + governed + fast"]
-  ProjFix --> Mission
-  IntFix --> Mission
-  BlobFix --> Mission
-  DBFix --> Mission
-  ObsFix --> Mission
-
-
-
- -
-
-
04 / observability
-

Coverage And Gaps

-
-
-
-

What is good today

-
    -
  • APM has meaningful phase spans for dispatch, projection, Cedar, WASM, and SQL.
    • -
  • SQL spans include busy/idle clues that distinguish actual query time from waiting.
    • -
  • DBM can see projection/event/trajectory query samples and post-repair explain plans.
    • -
  • Custom metrics exist for Cedar phase timing, Postgres pool/transaction timing, projection queue/e2e timing, dispatch, WASM, trace sampling, and profiling uploads.
    • -
  • Dashboard and monitor coverage exists, including DBM, Cedar, WASM, actor, mailbox, trace-budget, and standard APM rate/error/duration signals.
    • -
-
-
-

What is not good enough

-
    -
  • Targeted CPU profiling works, but continuous/native profiling is still not proven as an always-on surface.
    • -
  • DBM plan collection is repaired, but schema collection, CPU/vacuum coverage, and track_activity_query_size tuning remain open.
    • -
  • Formal SLO objects are not yet defined, even though p95/p99 metrics and monitors now exist.
    • -
  • WASM metrics have module:N/A, hiding the key latency dimension.
    • -
  • Projection correctness instrumentation exists, but scheduled/live drift, replay parity, and version/hash checks need production execution.
    • -
  • Service dependency graph did not cleanly show temperpaw -> temperpaw-postgres.
    • -
-
-
- -
-
Current observability flowwhat exists
- 
-flowchart TB
-  App["temperpaw Rust service"] --> Tracing["tracing + OpenTelemetry spans"]
-  App --> Metrics["custom OTEL metrics
dispatch + projection + Cedar + WASM"]
-  App --> Logs["structured logs"]
-  PG["Postgres / Railway"] --> DBM["Datadog DBM samples"]
-  App --> Prof["Datadog Rust profiler uploads"]
-  Tracing --> APM["Datadog APM"]
-  Metrics --> DDM["Datadog Metrics + Monitors"]
-  Logs --> DDL["Datadog Logs"]
-  DBM --> DDDB["DBM query analysis"]
-  Prof --> DDProf["Profiler UI"]
-  APM --> Dash["TemperPaw Platform Overview"]
-  DDM --> Dash
-  DDDB --> Dash
-  DDProf --> Dash
-
-
- -
-
Observability repair loopmake speed measurable
- 
-flowchart LR
-  Baseline["Baseline SLOs
workflow p50/p95/p99"] --> Profiles["Fresh profiles
CPU + alloc + lock"]
-  Profiles --> DBM2["Reliable DBM
plans + waits + CPU + vacuum"]
-  DBM2 --> Distributions["Distribution metrics
not averages"]
-  Distributions --> Correct["Correctness metrics
lag + drift + replay parity"]
-  Correct --> TraceBudget["Trace budgets
summaries + exemplars"]
-  TraceBudget --> Decide["Optimization decisions"]
-  Decide --> LoadLab["Replay + synthetic load lab"]
-  LoadLab --> Baseline
-
-
- -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
QuestionCurrent answerNeeded instrumentation
Is Datadog profiling instrumented?Yes for targeted in-process CPU profiling. Production routes, Agent upload, upload counters, and dashboard widgets exist; loaded captures produce parseable pprof payloads.Continuous capture, native ddprof, freshness monitor, trustworthy wall counters, allocation/lock profile proof or documented limits, and flamegraph runbook evidence.
Is profiling useful right now?Useful for targeted CPU captures under real load. Not enough yet for always-on diagnosis because continuous profiling is unset and native ddprof is disabled.Use loaded CPU captures for Cedar, JSON, projection diffing, and WASM host overhead while completing continuous/native profiling.
Is DB monitoring useful?Much more useful after the live DBM helper repair. DBM now shows activity rows, APM trace correlation, query performance, transient-latency health signals, and fresh explain plans for hot entity_catalog, entity_field_index, and events signatures. Schema definitions still do not show up through DBM schema collection.Use the new plan data and live app-side pool/transaction metrics for measured query-shape work, but still fix/decide schema collection, CPU/vacuum metrics, and track_activity_query_size.
Can we prove projections are correct?Partially, and now with a stronger local proof. We can emit source-tagged projection queue/update/backfill/applied-sequence metrics, opt into sampled catalog shadow reads, and run a replay parity verifier that compares active catalog rows with state rebuilt from the event journal.Scheduled/live parity execution, row-count parity, hash/version parity, deployed dashboard proof, and clean drift/error monitors.
-
-
- -
-
-
05 / target state
-

Proposed Fast Architecture

-
-
-
Target architecturefast without losing Temper
- 
-flowchart LR
-  Client["Clients / agents"] --> Edge["OData + MCP edge
workflow SLO tags"]
-  Edge --> AuthCache["AuthZ decision layer
compiled policy sets + deny-safe cache"]
-  AuthCache --> Actor["Verified entity actor
TransitionTable + event append"]
-  Actor --> Journal["Event journal
single-statement CAS + snapshots"]
-  Actor --> Outbox["Projection outbox
tenant/entity/version sequenced"]
-  Outbox --> Coalescer["Projection coalescer
diff fields + batch SQL + selective index"]
-  Coalescer --> Catalog["entity_catalog"]
-  Coalescer --> FieldIndex["entity_field_index"]
-  Catalog --> Reads["low-latency read plane"]
-  FieldIndex --> Reads
-  Actor --> Workflow["Durable integration executor
queue + idempotency + budgets"]
-  Workflow --> External["External APIs / providers"]
-  Client --> BlobPlane["Direct blob data plane
presigned or streamed upload"]
-  BlobPlane --> BlobStore["Object storage"]
-  BlobPlane --> Actor
-  Actor --> Correctness["Correctness plane
replay + shadow reads + drift gauges"]
-  Edge --> Obs["Observability plane
spans + distributions + profiles + DBM + exemplars"]
-  Coalescer --> Obs
-  Workflow --> Obs
-  Correctness --> Obs
-
-
- -
-
-

Keep the verified core

-

Do not replace the actor/state-machine model. Optimize persistence, authorization, projections, and external work around it.

-
-
-

Move derived work out

-

Projection and integration work should be bounded, coalesced, and observable. Synchronous semantics should be explicit.

-
-
-

Instrument correctness

-

If reads depend on projections, drift detection, replay parity, and read-your-write guarantees belong in dashboards.

-
-
- -
-
Projection correctness loopspeed plus proof
- 
-sequenceDiagram
-  participant A as Actor/Event source
-  participant O as Projection outbox
-  participant C as Coalescer
-  participant P as Projection tables
-  participant S as Shadow reader
-  participant M as Monitors
-  A->>O: committed event sequence
-  O->>C: ordered projection work
-  C->>P: diff + batch update
-  P-->>C: version + row counts
-  S->>A: authoritative sample
-  S->>P: projected sample
-  S->>M: drift hash / lag / parity
-  M-->>C: alert or allow rollout
-
-
-
- -
-
-
06 / work plan
-

Task Strategy

-
-
-
Task dependency mapmeasured order
- 
-flowchart LR
-  OBS1["OBS-001
Profiler restoration"] --> PERF1["PERF-001
Cedar CPU path"]
-  OBS2["OBS-002
DBM repair"] --> PERF3["PERF-003
DB transaction shape"]
-  OBS3["OBS-003
Projection correctness"] --> PERF2["PERF-002
Projection write amp"]
-  OBS4["OBS-004
Metric percentiles"] --> SLO["SLO gates"]
-  OBS5["OBS-005
Trace budgets"] --> PERF2
-  PERF2 --> VERIFY["VERIFY-001
Replay load lab"]
-  PERF1 --> VERIFY
-  PERF3 --> VERIFY
-  PERF4["PERF-004
Workflow executor"] --> VERIFY
-  PERF5["PERF-005
Blob data plane"] --> VERIFY
-  VERIFY --> Fast["Fast + governed + correct"]
-
-
- -
-
-
OBS-001 / profiler restoration
-
-

Make Datadog profiling continuously useful. Targeted production CPU profiling is live-proven under load; continuous/native profiling is packaged but currently blocked or canary-only on Railway.

-
    -
  • Local: added profiler config, capture, freshness, upload, and continuous-mode metrics.
    • -
  • Local: added opt-in continuous CPU capture with Datadog Agent auto-upload requirement.
    • -
  • Live: verified /_admin/profile/cpu under authenticated read load; Datadog received fresh CPU upload counters and the pprof file parses locally.
    • -
  • Live: capability endpoint reports TEMPER_DDPROF_ENABLED=false, ddprof_present=true, perf_event_paranoid=3, and CAP_PERFMON=false; Datadog continuous profiler uploads did not appear in the latest window.
    • -
  • Next: run profiler as a controlled canary or targeted capture for suspected CPU paths, and document the Railway permission limit if native continuous profiling cannot be enabled safely.
    • -
  • Next: confirm allocation/lock profile path or document Rust profiler limitations.
    • -
-

Acceptance: either fresh profiles within 10 minutes for the canary path, or an explicit documented platform limit plus a repeatable targeted-profile fallback.

-
-
-
-
OBS-002 / DBM repair
-
-

Make database monitoring reliable enough for query design. The setup/runbook repair package was applied live to production Postgres, and app-side Postgres metrics are implemented locally but still need runtime deployment.

-
    -
  • Local: added Datadog Postgres DBM setup SQL for schema, helper functions, grants, search path, RLS planning context, and configurable DBM Agent role.
    • -
  • Local: added operator runbook with psql validation and Datadog plan-search criteria.
    • -
  • Local: added Postgres pool acquire, transaction begin, transaction commit, transaction duration, operation outcome, and projection field-count metrics.
    • -
  • Live: production DBM Agent role was postgres, not datadog; applied setup with -v dbm_agent_role=postgres.
    • -
  • Live: SQL probes returned JSON explain plans for entity_catalog queries, the Datadog Postgres agent was restarted, and a Railway-domain read smoke returned 200 for /readyz, /tdata/Channels?$top=1, and /tdata/WorkerRuns?$top=1.
    • -
  • Live: fresh DBM plan search after the smoke returned plans for events using idx_events_tenant_entity and events_tenant_entity_type_entity_id_sequence_nr_key, and fresh invalid_schema search returned zero matches.
    • -
  • Next: verify DB CPU/vacuum/wait coverage, decide schema collection support, decide whether to raise track_activity_query_size, and verify app-side pool/transaction metrics after runtime deploy.
    • -
-

Acceptance: plans visible for top projection/event queries, app-side DB p50/p95/p99 visible by operation, schema/CPU/vacuum/wait gaps classified, and DB health warnings cleared.

-
-
-
-
OBS-003 / projection correctness
-
-

Treat projection correctness as first-class observability. Metrics, sampled shadow checks, replay parity, and the Turso catalog field-preservation repair are implemented and verified locally; live scheduled execution is next.

-
    -
  • Local: added ADR-0082 for projection correctness observability.
    • -
  • Local: added source-tagged update started/error/duration, queue wait, end-to-end duration, and applied sequence metrics.
    • -
  • Local: added backfill coverage, backfill duration, and replay event-count metrics.
    • -
  • Local: added opt-in deterministic sampled shadow checks for catalog-fast reads via TEMPER_ODATA_CATALOG_SHADOW_READ_EVERY.
    • -
  • Local: added replay parity verifier metrics/tests against authoritative event replay.
    • -
  • Local: fixed Turso entity_catalog so full projected fields JSON is preserved separately from the scalar field index.
    • -
  • Live: deployed projection queue and end-to-end update metrics; Datadog now shows p95 queue wait under 0.04 ms and update end-to-end around 75-102 ms for source:background_dispatch.
    • -
  • Next: schedule/live-run parity verification and shadow-read sampling against deployed data.
    • -
-

Acceptance: dashboard shows latency, lag, backlog, and drift together.

-
-
-
-
OBS-004 / metric percentiles
-
-

Convert key averages into useful distributions and make correctness and tail-latency signals visible in Datadog config. The TemperPaw dashboard/monitor slices are implemented, retargeted to production's service:temperpaw tag, deployed live, and now receiving post-runtime-deploy metric data.

-
    -
  • Local: added projection queue wait and end-to-end p95/p99 dashboard widgets.
    • -
  • Local: added projection applied-sequence, backfill, shadow-check, and shadow sequence-gap widgets.
    • -
  • Local: added replay parity check, duration, and sequence-gap widgets.
    • -
  • Local: added Postgres pool acquire and transaction p95/p99 widgets.
    • -
  • Local: added dispatch, WASM invocation, WASM host HTTP, blob I/O, blob transport, Monty wait, context-prepare, and session-phase p95/p99 widgets.
    • -
  • Local: added monitors for projection drift, replay parity drift/errors, shadow check errors, projection queue/e2e regressions, backfill failures, and Postgres pool/transaction regressions.
    • -
  • Local: repaired the request latency monitor to use p95:temper_dispatch_ask_latency_ms instead of an average, and added p95 monitors for WASM, blob, Monty, and session phases.
    • -
  • Local: corrected Datadog query scopes from the Railway service name to the runtime tag service:temperpaw and added a preflight regression guard.
    • -
  • Live: deployed dashboard mn4-k3k-i66, created or updated 66 source monitors, and added four standard APM coverage monitors before the main refresh.
    • -
  • Live: redeployed the refreshed 76-monitor source catalog after the TemperPaw main refresh.
    • -
  • Live: reconciled 41 legacy/orphan monitors, including stale OpenPaw monitor copies and the obsolete continuous profiler upload-stalled monitor.
    • -
  • Live: enabled percentiles for existing Temper latency distributions, then reran after runtime deployment so Cedar ms/phase, Postgres pool/transaction, and projection queue/e2e metrics now have p95/p99.
    • -
  • Live: Datadog service coverage now recognizes APM rate, duration, and error for service:temperpaw.
    • -
  • Local: merged the latest TemperPaw observability restructure into the latency branch; source now has 76 monitors and a 24-widget dashboard with no unsupported percentile queries.
    • -
  • Next: rerun percentile config after parity/backfill workflows emit and capture dashboard screenshots/links for review.
    • -
-

Acceptance: SLO dashboards use percentiles, not misleading averages.

-
-
-
-
PERF-001 / Cedar CPU path
-
-

Reduce authorization latency without weakening governance. The measurement layer is deployed, and the first local optimization now reduces the Cedar candidate set before authorization while preserving default-deny and forbid semantics.

-
    -
  • Local: added ADR-0084 for AuthZ phase instrumentation and Datadog percentile gates.
    • -
  • Local: canonicalized temper_cedar_evaluations_total to the decision-tagged temper.authz scope and added an error decision label.
    • -
  • Local: added temper_cedar_evaluation_duration_ms, temper_cedar_evaluation_phase_duration_ms, and temper_cedar_request_attribute_count.
    • -
  • Local: added TemperPaw AuthZ phase dashboard widgets plus max-duration and phase-error monitors.
    • -
  • Live: enabled percentiles for the new Cedar ms/phase metrics after runtime deployment. Latest p95 is about 50 ms overall and about 50 ms in phase:authorizer; request-shape phases are microseconds.
    • -
  • Local: added ADR-0089 and crates/temper-authz/src/engine/candidates.rs for scope-safe Cedar policy candidate selection. This drops only impossible principal/action/resource scope mismatches, includes all matching forbids, preserves named policy diagnostics, and does not cache decisions.
    • -
  • Local: added temper_cedar_policy_candidate_count with source=full|candidate and outcome=full|filtered|fallback labels so Datadog can correlate policy volume with AuthZ latency.
    • -
  • Local: cargo test -p temper-authz passes 62 tests and cargo check -p temper-server passes.
    • -
  • Local: cargo fmt --all -- --check, focused clippy for temper-authz and temper-server, HTML module syntax, and git diff --check pass.
    • -
  • Local: full pre-push gate passed rustfmt, workspace clippy, readability ratchet, and full cargo test --workspace; slowest DST random workload was 526.04 seconds.
    • -
  • Remote: GitHub CI run 25949367580 passed, and Temper PR #239 merged as b0b898a2.
    • -
  • TemperPaw local: pinned crate, WASM SDK, Docker, and observability contract references to b0b898a2; cargo check -p temperpaw --locked, all 31 datadog_observability_contract tests, formatting, and diff checks pass.
    • -
  • Remote: TemperPaw rollout PR #270 passed PR CI and merged as 7726e4df.
    • -
  • Remote: TemperPaw main CI and Docker passed; Railway deployment 8ad1c472-1c63-4473-a9ed-10f7458df253 is live on sha-7726e4d with /paw/version and DD_VERSION both set to 7726e4dfba5453c337dafb7411f88c239f3d5bd5.
    • -
  • Live: File proof authz-candidates-ddversion-20260516141213 passed read-after-write correctness and produced exact-file APM traces for POST, PUT $value, entity GET, and byte GET.
    • -
  • Live: Datadog candidate metrics show 16,497 full policies versus 48 candidates per sampled evaluation, about 344x fewer policies reaching Cedar.
    • -
  • Measured next target: phase:authorizer p95 is down around 0.23 ms, but phase:policy_candidates is around 13.4 ms and full AuthZ spans in the PUT trace remain 19.6-26.1 ms.
    • -
  • Local PERF-001B: created branch codex/latency-authz-candidate-index-20260516, added ADR-0090, implemented a precomputed candidate index, and split selector tests into crates/temper-authz/src/engine/candidates_tests.rs.
    • -
  • Local PERF-001B: the new 1,002-policy equivalence test, full cargo test -p temper-authz with 64 tests, cargo check -p temper-server, focused clippy, rustfmt, and diff check are green.
    • -
  • Remote PERF-001B: committed 247c7d6b, pushed codex/latency-authz-candidate-index-20260516, and opened PR #240. The second full pre-push run passed rustfmt, workspace clippy, readability ratchet, full workspace tests, and doctests.
    • -
  • Remote PERF-001B: GitHub CI run 25965850115 passed and PR #240 merged into Temper main as 84e95c66.
    • -
  • TemperPaw rollout PERF-001B: created branch codex/bump-temper-authz-index-20260516 in worktree /Users/seshendranalla/Development/temperpaw-worktrees/bump-temper-authz-index-20260516, pinned crate dependencies, Docker observability clone, Datadog contract expectations, WASM SDK manifests, and stale WASM lockfiles to 84e95c66.
    • -
  • TemperPaw local PERF-001B: cargo check -p temperpaw --locked, all 31 datadog_observability_contract tests, formatting, and diff checks pass.
    • -
  • Remote TemperPaw PERF-001B: committed a5fb5e04, pushed the rollout branch, opened PR #271, passed PR CI run 25966397547, and merged as 1118183f.
    • -
  • Remote TemperPaw main PERF-001B: main CI run 25966733023 and Docker run 25966733035 passed; Docker pushed ghcr.io/nerdsane/temperpaw:sha-1118183 with digest sha256:7843528813075858ba12403af1fc326e314425d00b44643da987e982687e78b9.
    • -
  • Railway PERF-001B: production deployment ee997c77-7abb-4682-aa76-4a5155f05c1f succeeded through Dockerfile.deploy; /readyz, /healthz, and authenticated /paw/version are healthy and version-correct for 1118183f47239401f216a771d6cd6fa6c1376a07.
    • -
  • Live PERF-001B: production proof authz-index-live-proof-20260516164954 created six Files, uploaded bytes through $value, reached Ready, read matching bytes back, and retained Datadog trace ba53ebea1c7c4952ac11400d6b8e90a8.
    • -
  • Datadog PERF-001B: current-version phase:policy_candidates p95 moved from the old 13.4 ms class to post-warm buckets of 0.335 ms and 0.262 ms; the live File proof bucket is 2.685 ms. Candidate volume remains 16,497 full policies versus 48 candidates for heavy File evaluations.
    • -
  • Next: decide whether to spend one more small AuthZ pass to make the heavy-path p95 strictly sub-2 ms, or move directly to the larger measured latency pools: projection write amplification, actor-spawn/query-projection fanout, and File $value transfer time.
    • -
-

Target: p95 under 20 ms initially, p99 under 50 ms, with no Cedar bypass and no cache until invalidation is proven.

-
-
-
-
PERF-002 / projection write amp
-
-

Make projection updates fast and bounded. This is an actual latency-improvement slice, not additional observability. The first Postgres-only implementation is merged into Temper and TemperPaw, deployed to Railway, and proven with live requests, Datadog, and direct database correctness checks.

-
    -
  • Local: created ADR-0091 and branch codex/latency-projection-diff-index-20260516 in fresh worktree /Users/seshendranalla/Development/temper-worktrees/latency-projection-diff-index-20260516.
    • -
  • Local: implemented Postgres field-index diff upserts. Unchanged rows are preserved, changed rows are rewritten, removed fields are deleted, and oversized fields stay in entity_catalog while leaving entity_field_index.
    • -
  • Local: added storage tests for unchanged row preservation, changed row rewrite, stale row removal, oversized field removal, catalog field preservation, and sequence preservation.
    • -
  • Local: cargo fmt --all -- --check, cargo check -p temper-store-postgres, cargo clippy -p temper-store-postgres --all-targets -- -D warnings, full cargo test -p temper-store-postgres -- --nocapture, git diff --check, readability ratchet, and full pre-push cargo test --workspace pass.
    • -
  • GitHub: Temper PR nerdsane/temper#241 passed CI run 25968899080 and merged as ed68e78539f5511b453e34e554bc95b0afb91a0d.
    • -
  • TemperPaw rollout: branch codex/bump-temper-projection-diff-index-20260516 pins root Temper crates, packaged temper-wasm-sdk manifests/locks, Docker TEMPER_OBSERVABILITY_REV, and Datadog observability contract expectations to ed68e785.
    • -
  • TemperPaw local: locked check, fmt, diff check, CI script syntax/smokes, clippy, TemperPaw/worker/review-gate tests, all packaged WASM build scripts, and dashboard build pass.
    • -
  • GitHub: TemperPaw rollout PR nerdsane/temperpaw#272 passed PR CI run 25969575912, was marked ready, and merged as 73e756a0bdb5f9a3084c5dd33c424154871fb873.
    • -
  • Main: CI run 25969887628 and Docker run 25969887617 are green for 73e756a0.
    • -
  • Railway: deployment 72635a45-5744-4c72-82e0-7b7143e9a548 succeeded on sha-73e756a; /readyz, /healthz, and authenticated /paw/version pass.
    • -
  • Live: proof projection-diff-live-proof-20260516191623 created four Files, double-uploaded bytes, reached Ready in one poll, read matching bytes, and listed by Name through projection.
    • -
  • Datadog: current-version File projection upsert p95 is 42.6 ms for source:create and 91.1 ms for source:background_dispatch; sampled APM shows list/entity reads mostly below 20 ms and PUT $value around 420 ms average in the proof window.
    • -
  • Correctness: direct DB proof shows the four proof Files in entity_catalog at Status=Ready, sequence_nr=5, with 14 matching entity_field_index rows per File, including Name, MimeType, and content_hash.
    • -
  • Next: turn replay parity and sampled shadow reads into scheduled/current production checks, then pick the next measured latency pool.
    • -
-

Target: normal query_projection p95 under 50 ms and zero unexplained drift.

-
-
-
-
PERF-003 / DB transaction shape
-
-

Remove avoidable database tail latency. The first shipped slice targeted measured projection transaction tail, not speculative pool tuning.

-
    -
  • Datadog selection: current version 821d6c6e shows query_projection_upsert transaction p95 around 429-443 ms, pool-acquire p95 around 4.8-5.0 ms, and projection end-to-end p95 around 471-478 ms for Session/background_dispatch.
    • -
  • Local: created fresh main-based worktree /Users/seshendranalla/Development/temper-worktrees/latency-db-transaction-shape-20260517 on branch codex/latency-db-transaction-shape-20260517.
    • -
  • ADR: added docs/adrs/0095-projection-transaction-fast-path.md for a Postgres-only projection transaction fast path.
    • -
  • Implementation: upsert_query_projection now computes scalar index fields before opening the transaction, locks the existing catalog fingerprint first, updates only catalog sequence/metadata when projection hash and status are unchanged, and preserves full diff reconciliation when fields or status change.
    • -
  • Observability: added temper_postgres_projection_index_reconciliations_total with low-cardinality path tags so rollout proof can separate insert, diff, and skipped_unchanged traffic.
    • -
  • Correctness: added a focused no-op projection test proving sequence advancement does not rewrite unchanged entity_field_index rows; existing changed-field and long-field reconciliation tests still pass.
    • -
  • Local checks: focused projection tests, full temper-store-postgres tests, clippy, store check, server check, rustfmt, diff check, code-quality review, and the full Temper pre-push gate are green. Local Postgres integration paths still skip without DATABASE_URL.
    • -
  • Temper GitHub: commit f0e3a18c went through draft PR #245, passed CI run 25979739482, and merged as 6439a8a0.
    • -
  • TemperPaw rollout: commit beaaf466 bumped all Temper runtime/server/store/SDK pins and checked-in WASM lockfiles to 6439a8a0; PR #276 passed PR CI run 25980160694 and merged as c16e0201.
    • -
  • Main/Docker: TemperPaw main CI run 25980450147 and Docker run 25980450153 passed; Docker image sha-c16e020 is the production image.
    • -
  • Railway: deployment 6e42424f-7ad4-4577-b127-9e3a0a36abeb is live, /paw/version reports sha-c16e020 / c16e0201c1490e0496f4964f5c72b704bb8cd216, and /readyz is ready.
    • -
  • Live proof: proof File fl-019e3427-4b90-7801-be94-d4af896aa315 received three PUT $value writes, exact readback after each write, VersionCount=3, correct FileVersion chain, and direct File GET p50 65.3 ms.
    • -
  • Production DB proof: entity_catalog and entity_field_index contain the expected File and FileVersion rows, including 14 indexed File fields and 3 FileVersion file_id index rows.
    • -
  • Datadog proof: current-version c16 tags are present, reconciliation paths emit diff=233, insert=25, and skipped_unchanged=17, and DBM samples correlate to APM with ddpv='c16e0201...'.
    • -
  • Residual: the c16 sample is still small. File/FileVersion background projection p95 is roughly 50-73 ms, but Session/background_dispatch remains around 245-386 ms; catalog-lock waits are visible but bounded in DBM. Next gate is a longer post-deploy observation window, then the next measured slice.
    • -
  • Follow-up after this slice: replace event append SELECT MAX shape with optimistic insert or sequence metadata only if fresh evidence still points there after the Session/background_dispatch residual is explained.
    • -
-
-
PERF-003 shipped pathtransaction shape
- 
-flowchart LR
-  Event["Entity event applied"] --> Queue["Projection update"]
-  Queue --> Precompute["Precompute scalar index fields
outside pool + transaction"]
-  Precompute --> Begin["Acquire connection + BEGIN"]
-  Begin --> Lock["Lock catalog fingerprint
status + projection_hash"]
-  Lock --> Same{"Hash/status unchanged?"}
-  Same -->|yes| CatalogOnly["Update catalog sequence + metadata only"]
-  Same -->|no| Diff["Diff/reconcile entity_field_index"]
-  CatalogOnly --> Metrics["path:skipped_unchanged"]
-  Diff --> Metrics
-  Metrics --> DBM["Datadog metrics + DBM/APM correlation"]
-
-
-

Target: no recurring idle-in-transaction samples for hot writes.

-
-
-
-
PERF-004 / workflow executor
-
-

Keep slow integrations from defining the whole platform latency.

-
    -
  • Move long-running WASM/external calls to a durable executor.
    • -
  • Use idempotency keys and verified callback transitions.
    • -
  • Emit progress state and queue wait metrics.
    • -
-

Target: control-plane responses are fast while integration progress remains visible.

-
-
-
-
PERF-005 / blob data plane
-
-

Stop routing large bytes through the hottest control path.

-
    -
  • Local: ADR-0088 defines a native built-in File $value write fast path.
    • -
  • Local: built-in File uploads now store content through native content-addressed blob storage, then dispatch the verified StreamUpdated action.
    • -
  • Local: remote content-addressed writes avoid the existing native HEAD-then-PUT shape and use a direct PUT.
    • -
  • Local: cargo check -p temper-server and cargo test -p temper-server --test file_value_fast_path passed; OData File $value succeeds without any blob_adapter registered.
    • -
  • Live: Temper PR #238 and TemperPaw PR #269 merged, Railway deployed fd83c31b, and production read-back proof passed.
    • -
  • Measured: Datadog upload trace moved from 3,159.5 ms with wasm.invoke to 353.3 ms with native state.put_file_stream_content.native and no matching WASM/blob-adapter span.
    • -
  • PERF-005B local: ADR-0092 and branch codex/latency-file-value-residual-20260516 target the remaining measured tail where File upload still waits on post-commit reaction fanout.
    • -
  • PERF-005B local: dispatch now has explicit await_reactions; all existing generic/OData/platform callers preserve inline reaction semantics, while native File $value writes commit bytes and StreamUpdated synchronously, then run the FileVersion/RecordVersion trigger chain in a bounded background lane.
    • -
  • PERF-005B local checks: background trigger test, File $value fast-path test suite, focused adapter/reaction/trigger/WASM suite, cargo check -p temper-server -p temper-platform, clippy for both crates, and the full pre-push gate all pass.
    • -
  • PERF-005B Temper PR: #242 passed CI run 25971893778 and merged as 1796c4f0.
    • -
  • PERF-005B TemperPaw rollout local: branch codex/bump-temper-file-value-residual-20260516 pins Temper to 1796c4f0, preserves setup API inline reaction behavior with await_reactions: true, and passes locked check, Datadog contracts, rustfmt, clippy/check, worker/review-gate tests, full TemperPaw tests, packaged WASM builds, and dashboard build.
    • -
  • PERF-005B TemperPaw rollout PR: #273 passed PR CI run 25972634147 and merged as e8457ca4.
    • -
  • PERF-005B mainline: TemperPaw main CI 25972955841 and Docker 25972955833 passed; Docker pushed ghcr.io/nerdsane/temperpaw:sha-e8457ca with digest sha256:4956e8d7ed26af68168b64643b6aefc0e03e5cc114cbb8377382649c3a9eadfa.
    • -
  • PERF-005B deploy: Railway deployment 5778571f-b325-4237-a792-cb13342963a4 succeeded on sha-e8457ca4; /readyz, /healthz, and authenticated /paw/version pass with DD_VERSION=e8457ca4a4891b05710980c7aebe1ed822d55f87.
    • -
  • PERF-005B live: proof file-reaction-background-live-proof-20260516215205 created six Files, ran twelve $value writes, read matching bytes back, verified projection reads, confirmed FileVersion chains, and passed direct production DB checks.
    • -
  • Measured: Datadog sampled PUT $value p95 moved from 490.3 ms to 238.7 ms; File.StreamUpdated p95 moved from 223.1 ms to 13.7 ms; reaction.dispatch.background p95 is 89.6 ms and starts after the HTTP response span.
    • -
  • PERF-005C selected: current residual latency sits inside native File byte work, but native blob observability only measured semaphore queue wait, not object-store transport. ADR-0093 defines bounded transport spans and percentile metrics.
    • -
  • PERF-005C local: branch codex/latency-blob-transport-observability-20260516 adds native blob.transport.* spans and transport duration/request/byte metrics for local filesystem and S3/R2 put, put_content, get, and head operations.
    • -
  • PERF-005C local checks: cargo check -p temper-server, cargo clippy -p temper-server --all-targets -- -D warnings, cargo test -p temper-server blob, cargo test -p temper-server --test file_value_fast_path, cargo fmt --all -- --check, and git diff --check all pass.
    • -
  • PERF-005C GitHub: commit 8fc79f3c was pushed, the full pre-push gate passed, Temper PR #243 passed CI run 25974966315, and merged into main as 88c9d797.
    • -
  • PERF-005C TemperPaw rollout local: branch codex/bump-temper-blob-transport-observability-20260516 pins Temper to 88c9d797, extends Datadog dashboard/monitor/percentile contract coverage for native blob transport metrics, and passes JSON validation, script compile, diff check, rustfmt, locked package check, Datadog contract tests, monitor config tests, full 187-test TemperPaw suite, and focused workspace_fs WASM SDK test.
    • -
  • PERF-005C TemperPaw rollout PR: commit 04c6d3bf was pushed, PR #274 passed CI run 25975556316, and the PR merged as 9fcd4b2b.
    • -
  • PERF-005C TemperPaw rollout merge: main CI run 25975834289 and Docker run 25975834282 passed for 9fcd4b2b.
    • -
  • PERF-005C TemperPaw mainline: CI run 25975834289 and Docker run 25975834282 passed; GHCR image sha-9fcd4b2 has digest sha256:74d398d56362de4b31cf09334e975952ec8f455a29e92a80fee99a6d5edf65bb.
    • -
  • PERF-005C deploy: Railway deployment 37edf6ad-5f71-4865-85e3-46a05dd7cc78 succeeded on sha-9fcd4b2b; /readyz, /healthz, authenticated /paw/version, and Railway DD_VERSION checks pass.
    • -
  • PERF-005C Datadog config: percentile configs for native transport duration/request bytes/response bytes are created; dashboard mn4-k3k-i66 is updated; native blob duration spike monitor 283877764 and native blob p95 regression monitor 283877770 are live.
    • -
  • PERF-005C live: proof blob-transport-live-proof-20260517001121 created six Files, ran twelve $value writes, read matching bytes back, verified projection reads, confirmed FileVersion chains, and passed direct production DB checks.
    • -
  • Measured: Datadog native blob metrics show put_content count 12, average 195.2 ms, p95/p99 223.8 ms; expanded trace a29ccff5afb451042e8bcd8ec1247b88 shows PUT $value 264.4 ms, state.put_file_stream_content.native 235.7 ms, and child blob.transport.put_content 186.5 ms.
    • -
  • PERF-005D selected: the same Datadog proof showed fresh native reads paying one legacy external-key blob.transport.get 404 before the successful native temper-fs/{content_hash} get. ADR-0094 defines native-first read-key order plus legacy fallback.
    • -
  • PERF-005D local: branch codex/latency-file-blob-read-key-order-20260517 changes file_read_blobs.rs so File reads try temper-fs/{content_hash} first and only probe the legacy external {content_hash} key for external endpoints when the native key is missing.
    • -
  • PERF-005D local checks: cargo test -p temper-server file_blob_read_keys, cargo test -p temper-server --test file_value_fast_path, cargo check -p temper-server, cargo fmt --all -- --check, and git diff --check all pass.
    • -
  • PERF-005D GitHub: Temper PR #244 passed CI run 25977404056 and merged as ff79974d.
    • -
  • PERF-005D rollout: TemperPaw commit 709876f3 bumped all runtime/server/SDK pins to ff79974d88a5b1a67e1fa9c4a746b422e12b29c3. PR #275 passed CI run 25977831129 and merged as 821d6c6e; main CI 25978275903 and Docker 25978275890 passed.
    • -
  • PERF-005D deploy: Railway deployment b628e727-b013-4c89-9eaa-f65f4f7e7433 succeeded from ghcr.io/nerdsane/temperpaw:sha-821d6c6. Runtime version tags now match 821d6c6ee28845e6b1365d78f3b85c3b5cc1ad15 across /paw/version, DD_VERSION, DD_GIT_COMMIT_SHA, and OTEL service.version.
    • -
  • PERF-005D live proof: blob-read-key-order-live-proof-20260517022158 created 12 fresh Files, wrote and read back 1,802 bytes, verified SHA-256 ContentHash, SizeBytes, Ready status, VersionCount=1, and LastVersionId for every File.
    • -
  • PERF-005D Datadog proof: old version window showed 12 get/outcome:ok plus 12 get/outcome:not_found; new version window shows get/outcome:ok count 19, put_content/outcome:ok count 13, and no get/outcome:not_found series. New get duration p50/p95 is 98.4 ms/223.8 ms.
    • -
-

Target: large upload latency scales with storage path, not actor/control-plane CPU.

-
-
-
-
OBS-005 / trace budgets
-
-

Keep traces useful without making them the bottleneck.

-
    -
  • ADR-0083 defines sampler policy and rollback.
    • -
  • Sampler decision/config/rate metrics are implemented.
    • -
  • Reaction fanout summary span fields are implemented.
    • -
  • Datadog widgets and monitors are ready to deploy.
    • -
-

Target: no routine 100k-span traces while slow/error trace roots and fanout summaries remain available.

-
-
-
-
VERIFY-001 / replay load lab
-
-

Make performance regressions reproducible.

-
    -
  • scripts/verify-latency-observability-package.sh full passed locally.
    • -
  • Run python3 scripts/temper_agent_e2e_proof.py during staging/live proof.
    • -
  • Generate synthetic tenants with large entities and many fields.
    • -
  • Compare projected reads to event/actor state during load.
    • -
-

Target: every speed claim has a repeatable benchmark and correctness check.

-
-
-
-
DECIDE / explicit architecture calls
-
-

These should become ADR-backed decisions.

-
    -
  • What read-your-write guarantee does projection promise?
    • -
  • Which fields are query-indexed by default?
    • -
  • Which integrations must remain inline?
    • -
  • What data sizes use direct blob upload?
    • -
-

Target: performance choices are contracts, not accidental behavior.

-
-
-
-
- -
-
-
07 / targets
-

Candidate SLOs

-
-
- - - - - - - - - - - - - - - - - -
WorkflowInitial targetStretch targetCorrectness guardrail
OData projected readp95 under 20 msp95 under 10 msProjection drift zero for verified sample set.
Entity action, no integrationp95 under 75 msp95 under 35 msDurable event append before success unless spec says otherwise.
Cedar authorizationp95 under 20 msp95 under 5 ms for cache hitsCache invalidates on policy hash and principal/context changes.
Projection updatep95 under 50 ms normal pathp99 under 100 ms normal pathLag and drift alerts plus shadow reads during rollout.
Integration actioncontrol response under 250 ms when asyncprogress event under 100 msVerified callback transition and idempotency key.
Blob metadata transitionp95 under 150 ms excluding object-store uploadp95 under 75 msHash/content verification after durable object commit.
-
-
- -
-
-
08 / sources
-

Sources And Anchors

-
-
-
-

Repository anchors

-
    -
  • README.md: verified, policy-driven runtime and high-level architecture.
    • -
  • crates/temper-server/src/state/entity_ops.rs: authorize_with_context.
    • -
  • crates/temper-server/src/state/dispatch/mod.rs and dispatch/wasm.rs: WASM dispatch.
    • -
  • crates/temper-server/src/odata/write.rs: stream PUT $value and blob_adapter.
    • -
  • crates/temper-store-postgres/src/store.rs: event append transaction shape.
    • -
  • crates/temper-observe/src/wide_event.rs: WideEvent span/metric emission.
    • -
  • crates/temper-authz/src/metrics.rs and crates/temper-authz/src/engine/mod.rs: Cedar evaluation and ADR-0084 phase metrics.
    • -
-
-
-

Live observability sources

-
    -
  • Datadog APM spans for temperpaw: OData, dispatch phases, Cedar, projection, WASM, and SQL.
    • -
  • Datadog DBM for temperpaw-postgres: query samples, health signals, lock/wait evidence, plan gaps.
    • -
  • Datadog metrics: Cedar, AuthZ phase, projection, dispatch, WASM, and profiler upload/error metrics.
    • -
  • Datadog monitors and dashboard: TemperPaw - Platform Overview.
    • -
  • Vapor Terminal OVA Console design language.
    • -
-

- Caveat: production traces include projection symbols absent from this local checkout, so Datadog is treated - as the fresher source for production performance. -

-
-
-
-
- -
- - - - - diff --git a/scripts/bench.sh b/scripts/bench.sh deleted file mode 100755 index 99efc245f..000000000 --- a/scripts/bench.sh +++ /dev/null @@ -1,50 +0,0 @@ -#!/usr/bin/env bash -# Run all Temper benchmarks. -# -# Usage: -# ./scripts/bench.sh # Run all benchmarks -# ./scripts/bench.sh jit # TransitionTable micro-benchmarks -# ./scripts/bench.sh actor # Server actor dispatch overhead -# ./scripts/bench.sh ecommerce # E-commerce agent checkout (realistic) -# -# Set DATABASE_URL to include Postgres benchmarks: -# DATABASE_URL=postgres://user:pass@localhost/db ./scripts/bench.sh - -set -euo pipefail - -cd "$(git rev-parse --show-toplevel)" - -case "${1:-all}" in - jit) - echo "=== TransitionTable micro-benchmarks ===" - cargo bench -p temper-jit --bench table_eval - ;; - actor) - echo "=== Actor dispatch overhead ===" - cargo bench -p temper-server --bench actor_throughput - ;; - ecommerce) - echo "=== E-commerce agent checkout ===" - cargo bench -p ecommerce-reference --bench agent_checkout - ;; - all) - echo "=== TransitionTable micro-benchmarks ===" - cargo bench -p temper-jit --bench table_eval - echo "" - echo "=== Actor dispatch overhead ===" - cargo bench -p temper-server --bench actor_throughput - echo "" - echo "=== E-commerce agent checkout ===" - cargo bench -p ecommerce-reference --bench agent_checkout - ;; - *) - echo "Usage: $0 [jit|actor|ecommerce|all]" - exit 1 - ;; -esac - -echo "" -if [ -z "${DATABASE_URL:-}" ]; then - echo "Note: Postgres benchmarks skipped (set DATABASE_URL to enable)" -fi -echo "Done. HTML reports in target/criterion/" diff --git a/scripts/demo.sh b/scripts/demo.sh deleted file mode 100755 index 13b0a11a9..000000000 --- a/scripts/demo.sh +++ /dev/null @@ -1,184 +0,0 @@ -#!/usr/bin/env bash -# Temper Agent OS — Governance Demo -# -# Prerequisites: -# 1. Start the server: -# DATABASE_URL=postgres://localhost:5432/temper \ -# cargo run -p temper-cli -- serve --port 3333 \ -# --app ecommerce=reference-apps/ecommerce/specs -# -# 2. Start the observe dashboard: -# cd observe && npm run dev -# -# 3. Open http://localhost:3000 in your browser -# -# Then run this script to see the full agent governance loop. - -set -euo pipefail - -API="http://localhost:3333" -TENANT="ecommerce" - -echo "=========================================" -echo " TEMPER AGENT OS — GOVERNANCE DEMO" -echo "=========================================" -echo "" - -# Verify server is running -if ! curl -s "$API/tdata" > /dev/null 2>&1; then - echo "ERROR: Server not running on $API" - echo "Start it with: DATABASE_URL=postgres://localhost:5432/temper cargo run -p temper-cli -- serve --port 3333 --app ecommerce=reference-apps/ecommerce/specs" - exit 1 -fi - -# Step 1: Enable Cedar default-deny for the tenant -echo "Step 1: Enable Cedar default-deny" -curl -s -X PUT "$API/api/tenants/$TENANT/policies" \ - -H "Content-Type: application/json" \ - -d "{\"policy_text\": \"// Default deny — no permits\\n\"}" > /dev/null -echo " Cedar policies loaded (empty = default-deny for agents)" -echo "" - -# Step 2: Create an Order as an agent -echo "Step 2: Create Order as agent 'checkout-bot'" -RESULT=$(curl -s -X POST "$API/tdata/Orders" \ - -H "Content-Type: application/json" \ - -H "X-Tenant-Id: $TENANT" \ - -H "X-Temper-Principal-Kind: agent" \ - -H "X-Temper-Principal-Id: checkout-bot" \ - -d '{}') -ORDER_ID=$(echo "$RESULT" | python3 -c "import sys,json; print(json.load(sys.stdin)['entity_id'])") -echo " Created: Orders('$ORDER_ID') in Draft" -echo "" - -# Step 3: Agent tries a bound action — gets DENIED -echo "Step 3: Agent tries AddItem → DENIED (no matching permit policy)" -HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \ - -X POST "$API/tdata/Orders('${ORDER_ID}')/Temper.AddItem" \ - -H "Content-Type: application/json" \ - -H "X-Tenant-Id: $TENANT" \ - -H "X-Temper-Principal-Kind: agent" \ - -H "X-Temper-Principal-Id: checkout-bot" \ - -d '{}') -echo " HTTP $HTTP_CODE — Authorization denied" -echo " → Check the Decisions page in the dashboard!" -echo "" - -# Step 4: Show the pending decision -echo "Step 4: Pending Decision created" -DECISIONS=$(curl -s "$API/api/tenants/$TENANT/decisions") -PD_ID=$(echo "$DECISIONS" | python3 -c " -import sys,json -d = json.load(sys.stdin)['decisions'] -pending = [x for x in d if x['status'] == 'pending'] -if pending: - p = pending[-1] - print(p['id']) -" 2>/dev/null) -echo "$DECISIONS" | python3 -c " -import sys,json -d = json.load(sys.stdin)['decisions'] -pending = [x for x in d if x['status'] == 'pending'] -if pending: - p = pending[-1] - print(f\" ID: {p['id']}\") - print(f\" Agent: {p['agent_id']}\") - print(f\" Action: {p['action']} on {p['resource_type']}\") - print(f\" Reason: {p['denial_reason']}\") -" 2>/dev/null -echo "" - -# Step 5: Human approves with broad scope -echo "Step 5: Human approves with BROAD scope" -APPROVE=$(curl -s -X POST "$API/api/tenants/$TENANT/decisions/${PD_ID}/approve" \ - -H "Content-Type: application/json" \ - -d '{"scope": "broad", "decided_by": "admin"}') -echo "$APPROVE" | python3 -c " -import sys,json -r = json.load(sys.stdin) -print(f\" Status: {r['status']}\") -print(f\" Generated Cedar policy:\") -for line in r['generated_policy'].split('\\n'): - print(f\" {line}\") -" 2>/dev/null -echo "" - -# Step 6: Agent retries — now succeeds -echo "Step 6: Agent retries AddItem → SUCCESS" -curl -s -X POST "$API/tdata/Orders('${ORDER_ID}')/Temper.AddItem" \ - -H "Content-Type: application/json" \ - -H "X-Tenant-Id: $TENANT" \ - -H "X-Temper-Principal-Kind: agent" \ - -H "X-Temper-Principal-Id: checkout-bot" \ - -d '{}' | python3 -c " -import sys,json -r = json.load(sys.stdin) -print(f\" Status: {r['status']}, Items: {r['item_count']}\") -" 2>/dev/null -echo "" - -# Step 7: Continue the lifecycle -echo "Step 7: SubmitOrder → SUCCESS (broad scope covers all Order actions)" -curl -s -X POST "$API/tdata/Orders('${ORDER_ID}')/Temper.SubmitOrder" \ - -H "Content-Type: application/json" \ - -H "X-Tenant-Id: $TENANT" \ - -H "X-Temper-Principal-Kind: agent" \ - -H "X-Temper-Principal-Id: checkout-bot" \ - -d '{}' | python3 -c " -import sys,json -r = json.load(sys.stdin) -print(f\" Status: {r['status']}\") -" 2>/dev/null -echo "" - -echo "Step 8: ConfirmOrder → SUCCESS" -curl -s -X POST "$API/tdata/Orders('${ORDER_ID}')/Temper.ConfirmOrder" \ - -H "Content-Type: application/json" \ - -H "X-Tenant-Id: $TENANT" \ - -H "X-Temper-Principal-Kind: agent" \ - -H "X-Temper-Principal-Id: checkout-bot" \ - -d '{}' | python3 -c " -import sys,json -r = json.load(sys.stdin) -print(f\" Status: {r['status']}\") -" 2>/dev/null -echo "" - -# Step 9: Agent audit trail -echo "=========================================" -echo " AGENT AUDIT TRAIL" -echo "=========================================" -curl -s "$API/observe/agents" | python3 -c " -import sys,json -data = json.load(sys.stdin) -for a in data['agents']: - print(f\"Agent: {a['agent_id']}\") - print(f\" Actions: {a['total_actions']} total\") - print(f\" Success: {a['success_count']}\") - print(f\" Denied: {a['denial_count']}\") - print(f\" Rate: {a['success_rate']*100:.0f}%\") - print(f\" Entities: {', '.join(a['entity_types'])}\") -" 2>/dev/null -echo "" - -echo "=========================================" -echo " ACTION HISTORY" -echo "=========================================" -curl -s "$API/observe/agents/checkout-bot/history" | python3 -c " -import sys,json -data = json.load(sys.stdin) -print(f\"{'Action':<20} {'Result':<10} {'From':<15} {'To':<15}\") -print('-' * 60) -for h in data['history']: - denied = 'DENIED' if h.get('authz_denied') else ('OK' if h['success'] else 'FAIL') - frm = h.get('from_status') or '-' - to = h.get('to_status') or '-' - print(f\"{h['action']:<20} {denied:<10} {frm:<15} {to:<15}\") -" 2>/dev/null -echo "" - -echo "=========================================" -echo " Open the dashboard: http://localhost:3000" -echo " - Decisions page: see approval history" -echo " - Agents page: see checkout-bot stats" -echo "=========================================" diff --git a/scripts/e2e-trusted-issuer.sh b/scripts/e2e-trusted-issuer.sh deleted file mode 100755 index 0b945dd81..000000000 --- a/scripts/e2e-trusted-issuer.sh +++ /dev/null @@ -1,129 +0,0 @@ -#!/bin/bash -# Live end-to-end check for platform-issued token verification (ARN-255). -# -# Boots a real temper server, activates a trusted issuer through the same -# environment configuration a deployment uses, mints real ES256 tokens with the -# matching private key, and drives the real HTTP surface to prove: -# 1. a valid token authenticates (not 401) -# 2. a token signed by an unknown key is rejected (401) -# 3. an expired token is rejected (401) -# 4. a token from an unregistered issuer is rejected (401) -# 5. a garbage/tampered token is rejected (401) -# 6. the operator key still works — the change is additive (200) -# 7. a verified agent token CANNOT register an issuer (403) -# (the takeover path: register your own key, mint owner tokens) -# 8. a verified agent token CANNOT bump a generation (403) -# (per-user sign-out denial of service) -# -# Requires: cargo, python3 with 'cryptography', curl. Usage: -# scripts/e2e-trusted-issuer.sh [port] -set -uo pipefail - -PORT="${1:-3477}" -BASE="http://localhost:${PORT}" -TENANT="default" -API_KEY="local-e2e-operator-key" -ISSUER="https://e2e.issuer.local" -AUD="temper-e2e" -WORK="$(mktemp -d)" -SERVER_PID="" -cleanup() { [ -n "$SERVER_PID" ] && kill "$SERVER_PID" 2>/dev/null; rm -rf "$WORK"; } -trap cleanup EXIT - -say() { printf '\n\033[1m== %s\033[0m\n' "$*"; } - -say "Minting a P-256 key, its JWKS, and four test tokens" -python3 - "$WORK" "$ISSUER" "$AUD" <<'PY' -import base64, json, sys, time -from cryptography.hazmat.primitives.asymmetric import ec -from cryptography.hazmat.primitives.asymmetric.utils import decode_dss_signature -from cryptography.hazmat.primitives import hashes - -work, issuer, aud = sys.argv[1], sys.argv[2], sys.argv[3] -b64 = lambda b: base64.urlsafe_b64encode(b).rstrip(b"=").decode() - -def mint(key, claims, kid="e2e-k1"): - head = {"alg": "ES256", "kid": kid, "typ": "JWT"} - si = f'{b64(json.dumps(head).encode())}.{b64(json.dumps(claims).encode())}' - r, s = decode_dss_signature(key.sign(si.encode(), ec.ECDSA(hashes.SHA256()))) - return f'{si}.{b64(r.to_bytes(32,"big") + s.to_bytes(32,"big"))}' - -key = ec.generate_private_key(ec.SECP256R1()) -pn = key.public_key().public_numbers() -open(f"{work}/jwks.json","w").write(json.dumps({"keys":[{ - "kty":"EC","crv":"P-256","kid":"e2e-k1", - "x": b64(pn.x.to_bytes(32,"big")), "y": b64(pn.y.to_bytes(32,"big"))}]})) - -now = int(time.time()) -base = {"iss": issuer, "aud": aud, "sub": "human-e2e", "client_id": "kc_agent_e2e", - "agent_type": "contributor", "grant_id": "grant-e2e", "nbf": now - 300} -open(f"{work}/valid.txt","w").write(mint(key, {**base, "exp": now + 900})) -open(f"{work}/expired.txt","w").write(mint(key, {**base, "exp": now - 600})) -open(f"{work}/bad_iss.txt","w").write(mint(key, {**base, "iss": "https://unregistered.example", "exp": now + 900})) -open(f"{work}/rogue.txt","w").write(mint(ec.generate_private_key(ec.SECP256R1()), {**base, "exp": now + 900})) -print(" 4 tokens + JWKS ready") -PY -[ -f "$WORK/valid.txt" ] || { echo "token minting failed"; exit 1; } - -say "Starting a real temper server on :$PORT with the issuer activated by env" -TEMPER_API_KEY="$API_KEY" \ -TEMPER_TRUSTED_ISSUER_URL="$ISSUER" \ -TEMPER_TRUSTED_ISSUER_JWKS="$(cat "$WORK/jwks.json")" \ -TEMPER_TRUSTED_ISSUER_AUD="$AUD" \ - cargo run -q -p temper-cli --bin temper -- serve --port "$PORT" --no-observe \ - >"$WORK/server.log" 2>&1 & -SERVER_PID=$! -for _ in $(seq 1 150); do - curl -sf "$BASE/healthz" >/dev/null 2>&1 && break - sleep 2 - kill -0 "$SERVER_PID" 2>/dev/null || { echo "server died:"; tail -30 "$WORK/server.log"; exit 1; } -done -curl -sf "$BASE/healthz" >/dev/null || { echo "never healthy:"; tail -30 "$WORK/server.log"; exit 1; } -echo " healthy" -grep -q "Trusted issuer '$ISSUER' registered" "$WORK/server.log" \ - && echo " issuer registered from environment at boot" \ - || { echo " ISSUER NOT REGISTERED — see log"; tail -20 "$WORK/server.log"; } - -code() { # code [method] [path] [body] - local tok="$1" method="${2:-GET}" path="${3:-/tdata/TrustedIssuers}" body="${4:-}" - if [ -n "$body" ]; then - curl -s -o /dev/null -w '%{http_code}' -X "$method" "$BASE$path" \ - -H "Authorization: Bearer $tok" -H "X-Tenant-Id: $TENANT" \ - -H "Content-Type: application/json" -d "$body" - else - curl -s -o /dev/null -w '%{http_code}' -X "$method" "$BASE$path" \ - -H "Authorization: Bearer $tok" -H "X-Tenant-Id: $TENANT" - fi -} - -PASS=0; FAIL=0 -check() { # check - local name="$1" got="$2"; shift 2 - for want in "$@"; do - if [ "$got" = "$want" ]; then printf ' \033[32mPASS\033[0m %s (HTTP %s)\n' "$name" "$got"; PASS=$((PASS+1)); return; fi - done - printf ' \033[31mFAIL\033[0m %s (got HTTP %s, wanted %s)\n' "$name" "$got" "$*"; FAIL=$((FAIL+1)) -} - -VALID=$(cat "$WORK/valid.txt") -ISS_ENC="https%3A%2F%2Fe2e.issuer.local" - -say "Token verification" -check "valid token authenticates" "$(code "$VALID")" 200 403 404 -check "rogue-key token rejected" "$(code "$(cat "$WORK/rogue.txt")")" 401 -check "expired token rejected" "$(code "$(cat "$WORK/expired.txt")")" 401 -check "unregistered issuer rejected" "$(code "$(cat "$WORK/bad_iss.txt")")" 401 -check "garbage token rejected" "$(code 'not.a.jwt')" 401 -check "operator key still works (additive)" "$(code "$API_KEY")" 200 - -say "Privilege boundary on the identity entities" -REG_BODY='{"issuer":"https://attacker.example","jwks_json":"{\"keys\":[]}","audience":"x","algorithms":"ES256","description":"takeover attempt","created_by":"attacker"}' -check "agent token CANNOT register an issuer" \ - "$(code "$VALID" POST "/tdata/TrustedIssuers('https%3A%2F%2Fattacker.example')/Temper.RegisterIssuer" "$REG_BODY")" 403 -check "agent token CANNOT rotate issuer keys" \ - "$(code "$VALID" POST "/tdata/TrustedIssuers('$ISS_ENC')/Temper.RotateIssuerKeys" '{"jwks_json":"{\"keys\":[]}"}')" 403 -check "agent token CANNOT bump a generation" \ - "$(code "$VALID" POST "/tdata/PrincipalGenerations('human-e2e')/Temper.BumpGeneration" '{}')" 403 - -say "Result: $PASS passed, $FAIL failed" -[ "$FAIL" -eq 0 ] diff --git a/scripts/set-branch-protection.sh b/scripts/set-branch-protection.sh deleted file mode 100755 index 8bd5b3e42..000000000 --- a/scripts/set-branch-protection.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -# Configure branch protection for mainline CI enforcement. -# Usage: -# scripts/set-branch-protection.sh [owner/repo] [branch] -# Example: -# scripts/set-branch-protection.sh nerdsane/temper main - -REPO="${1:-nerdsane/temper}" -BRANCH="${2:-main}" - -if ! command -v gh >/dev/null 2>&1; then - echo "ERROR: gh CLI is required." >&2 - exit 1 -fi - -if ! gh auth status >/dev/null 2>&1; then - echo "ERROR: gh auth is not valid. Run: gh auth login -h github.com" >&2 - exit 1 -fi - -echo "Applying branch protection to ${REPO}:${BRANCH} ..." - -gh api \ - --method PUT \ - -H "Accept: application/vnd.github+json" \ - "repos/${REPO}/branches/${BRANCH}/protection" \ - --input - <<'JSON' -{ - "required_status_checks": { - "strict": true, - "contexts": [ - "Verification Contract (verification.v1)", - "Compile & Lint", - "Integrity & DST Patterns", - "Tests", - "Spec Verification (L0-L3)" - ] - }, - "enforce_admins": true, - "required_pull_request_reviews": { - "dismiss_stale_reviews": true, - "require_code_owner_reviews": false, - "required_approving_review_count": 1 - }, - "required_conversation_resolution": true, - "restrictions": null -} -JSON - -echo "Branch protection applied successfully." diff --git a/scripts/temper_agent_e2e_proof.py b/scripts/temper_agent_e2e_proof.py deleted file mode 100644 index 7d7045e79..000000000 --- a/scripts/temper_agent_e2e_proof.py +++ /dev/null @@ -1,1424 +0,0 @@ -#!/usr/bin/env python3 - -import json -import os -import subprocess -import sys -import time -import urllib.error -import urllib.parse -import urllib.request -from datetime import datetime, timezone -from pathlib import Path - - -REPO_ROOT = Path(__file__).resolve().parents[1] -ARTIFACT_ROOT = REPO_ROOT / ".tmp" / "temper-agent-proof" / "artifacts" -REPORT_PATH = REPO_ROOT / ".proof" / "temper-agent-e2e-proof.md" - -SERVER = os.environ.get("TEMPER_PROOF_SERVER", "http://127.0.0.1:3463") -BLOB_ENDPOINT = os.environ.get("TEMPER_PROOF_BLOB", "http://127.0.0.1:9987") -SANDBOX_URL = os.environ.get("TEMPER_PROOF_SANDBOX", "http://127.0.0.1:9989") -REPLY_LOG = Path( - os.environ.get( - "TEMPER_PROOF_REPLY_LOG", - str(REPO_ROOT / ".tmp" / "temper-agent-proof" / "reply" / "replies.jsonl"), - ) -) -SANDBOX_WORKDIR = os.environ.get( - "TEMPER_PROOF_WORKDIR", - str(REPO_ROOT / ".tmp" / "temper-agent-proof" / "sandbox"), -) -TENANT = os.environ.get( - "TEMPER_PROOF_TENANT", - f"temper-agent-proof-{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}", -) -MCP_BIN = os.environ.get("TEMPER_PROOF_MCP_BIN", str(REPO_ROOT / "target" / "debug" / "temper-mcp")) - -ADMIN_HEADERS = {"x-temper-principal-kind": "admin"} -SYSTEM_HEADERS = {"x-temper-principal-kind": "system"} - - -def ensure_dirs() -> None: - ARTIFACT_ROOT.mkdir(parents=True, exist_ok=True) - REPORT_PATH.parent.mkdir(parents=True, exist_ok=True) - REPLY_LOG.parent.mkdir(parents=True, exist_ok=True) - - -def now_utc() -> str: - return datetime.now(timezone.utc).isoformat() - - -def write_text(path: Path, text: str) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(text, encoding="utf-8") - - -def append_jsonl(path: Path, value) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - with path.open("a", encoding="utf-8") as handle: - handle.write(json.dumps(value, sort_keys=True) + "\n") - - -def lookup(mapping, *keys): - if not isinstance(mapping, dict): - return None - - def normalize_key(value) -> str: - return "".join(ch for ch in str(value) if ch.isalnum()).lower() - - lowered = {normalize_key(k): v for k, v in mapping.items()} - for key in keys: - if key in mapping: - return mapping[key] - lower = normalize_key(key) - if lower in lowered: - return lowered[lower] - return None - - -def entity_fields(entity): - return lookup(entity, "fields") or {} - - -def entity_id(entity): - return lookup(entity, "entity_id", "Id", "id") - - -def entity_status(entity): - return lookup(entity, "status", "Status") - - -def entity_field(entity, *keys): - fields = entity_fields(entity) - value = lookup(fields, *keys) - if value is not None: - return value - return lookup(entity, *keys) - - -def json_body_bytes(body) -> bytes: - return json.dumps(body).encode("utf-8") - - -def request( - method: str, - path: str, - *, - tenant: str | None = None, - headers: dict | None = None, - json_body=None, - body: bytes | None = None, - content_type: str | None = None, - accept: str | None = "application/json", - expect: tuple[int, ...] | None = None, -): - if path.startswith("http://") or path.startswith("https://"): - url = path - else: - url = SERVER.rstrip("/") + path - all_headers = {} - if tenant: - all_headers["x-tenant-id"] = tenant - if accept: - all_headers["accept"] = accept - if headers: - all_headers.update(headers) - if json_body is not None: - payload = json_body_bytes(json_body) - all_headers.setdefault("content-type", "application/json") - else: - payload = body - if content_type: - all_headers.setdefault("content-type", content_type) - req = urllib.request.Request(url, data=payload, method=method.upper(), headers=all_headers) - try: - with urllib.request.urlopen(req, timeout=120) as resp: - raw = resp.read() - status = resp.getcode() - resp_headers = dict(resp.headers.items()) - except urllib.error.HTTPError as err: - raw = err.read() - status = err.code - resp_headers = dict(err.headers.items()) - text = raw.decode("utf-8", errors="replace") - parsed = None - ctype = resp_headers.get("Content-Type", "") - if "json" in ctype or text.startswith("{") or text.startswith("["): - try: - parsed = json.loads(text) - except json.JSONDecodeError: - parsed = None - if expect and status not in expect: - raise RuntimeError(f"{method} {url} failed with HTTP {status}: {text[:600]}") - return { - "status": status, - "text": text, - "json": parsed, - "headers": resp_headers, - "url": url, - } - - -def post_json(path: str, body, *, tenant: str | None = None, headers: dict | None = None, expect=(200, 201, 204)): - return request("POST", path, tenant=tenant, headers=headers, json_body=body, expect=expect) - - -def put_json(path: str, body, *, tenant: str | None = None, headers: dict | None = None, expect=(200, 201, 204)): - return request("PUT", path, tenant=tenant, headers=headers, json_body=body, expect=expect) - - -def put_text(path: str, text: str, *, tenant: str | None = None, headers: dict | None = None, expect=(200, 201, 204)): - return request( - "PUT", - path, - tenant=tenant, - headers=headers, - body=text.encode("utf-8"), - content_type="text/plain", - accept=None, - expect=expect, - ) - - -def get_json(path: str, *, tenant: str | None = None, headers: dict | None = None, expect=(200,)): - return request("GET", path, tenant=tenant, headers=headers, expect=expect) - - -def install_app(tenant: str, app_name: str): - return post_json( - f"/api/os-apps/{app_name}/install", - {"tenant": tenant}, - headers=ADMIN_HEADERS, - )["json"] - - -def put_secret(tenant: str, key: str, value: str) -> None: - put_json( - f"/api/tenants/{tenant}/secrets/{key}", - {"value": value}, - headers=ADMIN_HEADERS, - expect=(204,), - ) - - -def upload_wasm(tenant: str, name: str, wasm_path: Path): - return request( - "POST", - f"/api/wasm/modules/{name}", - tenant=tenant, - headers=ADMIN_HEADERS, - body=wasm_path.read_bytes(), - content_type="application/wasm", - expect=(200,), - )["json"] - - -def create_entity(tenant: str, entity_set: str, fields: dict): - return post_json( - f"/tdata/{entity_set}", - fields, - tenant=tenant, - headers=ADMIN_HEADERS, - )["json"] - - -def get_entity(tenant: str, entity_set: str, entity_id_value: str): - key = urllib.parse.quote(entity_id_value, safe="") - return get_json( - f"/tdata/{entity_set}('{key}')", - tenant=tenant, - headers=ADMIN_HEADERS, - )["json"] - - -def list_entities(tenant: str, entity_set: str): - return get_json( - f"/tdata/{entity_set}", - tenant=tenant, - headers=ADMIN_HEADERS, - )["json"]["value"] - - -def action_with_fallback(tenant: str, entity_set: str, entity_id_value: str, action_paths: list[str], body: dict): - key = urllib.parse.quote(entity_id_value, safe="") - last_error = None - for action_path in action_paths: - resp = request( - "POST", - f"/tdata/{entity_set}('{key}')/{action_path}", - tenant=tenant, - headers=ADMIN_HEADERS, - json_body=body, - ) - if 200 <= resp["status"] < 300: - return resp["json"] or resp["text"] - last_error = resp - if resp["status"] not in (400, 404): - break - if last_error is None: - raise RuntimeError(f"no action path tried for {entity_set} {entity_id_value}") - raise RuntimeError( - f"action failed for {entity_set} {entity_id_value} via {action_paths}: " - f"HTTP {last_error['status']} {last_error['text'][:400]}" - ) - - -def wait_entity(tenant: str, entity_type: str, entity_id_value: str, statuses: list[str], timeout_ms: int = 120000): - query = urllib.parse.urlencode( - { - "statuses": ",".join(statuses), - "timeout_ms": str(timeout_ms), - "poll_ms": "250", - } - ) - return get_json( - f"/observe/entities/{entity_type}/{urllib.parse.quote(entity_id_value, safe='')}/wait?{query}", - tenant=tenant, - headers=ADMIN_HEADERS, - expect=(200, 408), - )["json"] - - -def wait_for_entities(tenant: str, entity_set: str, predicate, timeout_s: float = 10.0, poll_s: float = 0.25): - deadline = time.time() + timeout_s - while True: - matches = [entry for entry in list_entities(tenant, entity_set) if predicate(entry)] - if matches or time.time() >= deadline: - return matches - time.sleep(poll_s) - - -def read_reply_lines() -> list[dict]: - if not REPLY_LOG.exists(): - return [] - raw_reply_lines = [ - json.loads(line) - for line in REPLY_LOG.read_text(encoding="utf-8").splitlines() - if line.strip() - ] - reply_lines = [] - for line in raw_reply_lines: - body = line.get("body") - if isinstance(body, str): - try: - parsed_body = json.loads(body) - except json.JSONDecodeError: - parsed_body = body - if isinstance(parsed_body, dict): - merged = dict(line) - merged.update(parsed_body) - line = merged - reply_lines.append(line) - return reply_lines - - -def wait_for_reply(predicate, timeout_s: float = 10.0, poll_s: float = 0.25) -> list[dict]: - deadline = time.time() + timeout_s - while True: - reply_lines = read_reply_lines() - if any(predicate(line) for line in reply_lines) or time.time() >= deadline: - return reply_lines - time.sleep(poll_s) - - -def capture_sse(tenant: str, entity_type: str, entity_id_value: str, output_path: Path, since: int = 0, max_time: int = 2): - cmd = [ - "curl", - "-sN", - "--max-time", - str(max_time), - "-H", - f"x-tenant-id: {tenant}", - "-H", - "x-temper-principal-kind: admin", - f"{SERVER}/observe/entities/{entity_type}/{entity_id_value}/events?since={since}", - ] - result = subprocess.run(cmd, cwd=REPO_ROOT, capture_output=True, text=True) - write_text(output_path, result.stdout) - return result.stdout - - -def create_file_asset(tenant: str, workspace_id: str, directory_id: str, path: str, content: str): - file_entity = create_entity( - tenant, - "Files", - { - "Name": Path(path).name, - "Path": path, - "DirectoryId": directory_id, - "WorkspaceId": workspace_id, - "MimeType": "text/markdown" if path.endswith(".md") else "text/plain", - }, - ) - file_id = entity_id(file_entity) - put_text( - f"/tdata/Files('{file_id}')/$value", - content, - tenant=tenant, - headers=ADMIN_HEADERS, - ) - return file_entity - - -def get_file_text(tenant: str, file_id: str) -> str: - return request( - "GET", - f"/tdata/Files('{urllib.parse.quote(file_id, safe='')}')/$value", - tenant=tenant, - headers=ADMIN_HEADERS, - accept=None, - expect=(200,), - )["text"] - - -def clean_sandbox() -> None: - request( - "POST", - f"{SANDBOX_URL}/v1/processes/run", - headers={}, - json_body={"command": f"rm -rf '{SANDBOX_WORKDIR}'/* 2>/dev/null || true", "workdir": SANDBOX_WORKDIR}, - expect=(200,), - ) - - -def extract_prompt_from_sse(raw_sse: str) -> str: - event_name = None - for line in raw_sse.splitlines(): - if line.startswith("event:"): - event_name = line.split(":", 1)[1].strip() - elif line.startswith("data:"): - payload = line.split(":", 1)[1].strip() - try: - data = json.loads(payload) - except json.JSONDecodeError: - continue - if event_name == "prompt_assembled": - nested = lookup(data, "data") or {} - return lookup(nested, "system_prompt") or lookup(data, "system_prompt") or "" - if event_name == "integration_progress" and lookup(data, "kind") == "prompt_assembled": - nested = lookup(data, "data") or {} - return lookup(nested, "system_prompt") or lookup(data, "system_prompt") or "" - return "" - - -def parse_sse_events(raw_sse: str): - events = [] - current = None - for line in raw_sse.splitlines(): - if line.startswith("event:"): - current = {"event": line.split(":", 1)[1].strip()} - elif line.startswith("data:") and current is not None: - payload = line.split(":", 1)[1].strip() - try: - current["data"] = json.loads(payload) - except json.JSONDecodeError: - current["data"] = payload - events.append(current) - current = None - return events - - -def latest_text_result_from_session(session_jsonl: str) -> str: - last = "" - for line in session_jsonl.splitlines(): - if not line.strip(): - continue - entry = json.loads(line) - if lookup(entry, "type") != "message": - continue - if lookup(entry, "role") != "assistant": - continue - content = lookup(entry, "content") - if isinstance(content, list): - texts = [block.get("text", "") for block in content if block.get("type") == "text"] - if texts: - last = "\n".join(texts) - elif isinstance(content, str): - last = content - return last - - -def entity_result(entity, session_jsonl: str | None = None) -> str: - for key in ("result", "Result"): - value = entity_field(entity, key) - if isinstance(value, str) and value: - return value - if session_jsonl: - return latest_text_result_from_session(session_jsonl) - return "" - - -def step(status: bool, expected: str, actual: str): - return {"status": "PASS" if status else "FAIL", "expected": expected, "actual": actual} - - -class McpClient: - def __init__(self, binary_path: str, port: int, stderr_path: Path): - self.stderr_handle = stderr_path.open("w", encoding="utf-8") - self.process = subprocess.Popen( - [ - binary_path, - "--port", - str(port), - "--agent-id", - "proof-harness", - "--agent-type", - "human", - "--session-id", - f"proof-{int(time.time())}", - ], - cwd=REPO_ROOT, - stdin=subprocess.PIPE, - stdout=subprocess.PIPE, - stderr=self.stderr_handle, - text=True, - bufsize=1, - ) - self.next_id = 1 - - def send(self, payload): - assert self.process.stdin is not None - self.process.stdin.write(json.dumps(payload) + "\n") - self.process.stdin.flush() - - def recv(self, expected_id: int): - assert self.process.stdout is not None - while True: - line = self.process.stdout.readline() - if not line: - raise RuntimeError("temper-mcp closed stdout unexpectedly") - message = json.loads(line) - if message.get("id") != expected_id: - continue - return message - - def initialize(self): - req_id = self.next_id - self.next_id += 1 - self.send( - { - "jsonrpc": "2.0", - "id": req_id, - "method": "initialize", - "params": { - "protocolVersion": "2024-11-05", - "capabilities": {}, - "clientInfo": {"name": "pi-proof", "version": "1.0.0"}, - }, - } - ) - self.recv(req_id) - self.send({"jsonrpc": "2.0", "method": "notifications/initialized"}) - - def execute(self, code: str): - req_id = self.next_id - self.next_id += 1 - self.send( - { - "jsonrpc": "2.0", - "id": req_id, - "method": "tools/call", - "params": { - "name": "execute", - "arguments": {"code": code}, - }, - } - ) - response = self.recv(req_id) - if "error" in response: - raise RuntimeError(response["error"]["message"]) - result = response["result"] - text = "" - content = result.get("content") or [] - if content: - text = content[0].get("text", "") - if result.get("isError"): - raise RuntimeError(text) - try: - return json.loads(text) - except json.JSONDecodeError: - return text - - def close(self): - if self.process.poll() is None: - self.process.terminate() - try: - self.process.wait(timeout=5) - except subprocess.TimeoutExpired: - self.process.kill() - self.stderr_handle.close() - - -def build_mock_plan(steps: list[dict]) -> str: - return json.dumps({"mock_plan": {"steps": steps}}, separators=(",", ":")) - - -def main() -> int: - ensure_dirs() - clean_sandbox() - REPLY_LOG.write_text("", encoding="utf-8") - - artifact_log = ARTIFACT_ROOT / "proof-log.jsonl" - artifact_log.unlink(missing_ok=True) - - report = { - "date": now_utc(), - "tenant": TENANT, - "branch": subprocess.check_output(["git", "branch", "--show-current"], cwd=REPO_ROOT, text=True).strip(), - "commit": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=REPO_ROOT, text=True).strip(), - "steps": {}, - } - - health = get_json("/observe/health", headers=ADMIN_HEADERS)["json"] - write_text(ARTIFACT_ROOT / "server-health.json", json.dumps(health, indent=2)) - - apps = { - "temper-fs": install_app(TENANT, "temper-fs"), - "temper-agent": install_app(TENANT, "temper-agent"), - "temper-channels": install_app(TENANT, "temper-channels"), - } - write_text(ARTIFACT_ROOT / "installed-apps.json", json.dumps(apps, indent=2)) - - put_secret(TENANT, "temper_api_url", SERVER) - put_secret(TENANT, "blob_endpoint", BLOB_ENDPOINT) - - modules = { - "blob_adapter": REPO_ROOT / "os-apps" / "temper-fs" / "wasm" / "blob_adapter.wasm", - "llm_caller": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "llm_caller" / "target" / "wasm32-unknown-unknown" / "release" / "llm_caller.wasm", - "tool_runner": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "tool_runner" / "target" / "wasm32-unknown-unknown" / "release" / "tool_runner.wasm", - "sandbox_provisioner": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "sandbox_provisioner" / "target" / "wasm32-unknown-unknown" / "release" / "sandbox_provisioner.wasm", - "context_compactor": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "context_compactor" / "target" / "wasm32-unknown-unknown" / "release" / "context_compactor.wasm", - "steering_checker": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "steering_checker" / "target" / "wasm32-unknown-unknown" / "release" / "steering_checker.wasm", - "coding_agent_runner": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "coding_agent_runner" / "target" / "wasm32-unknown-unknown" / "release" / "coding_agent_runner.wasm", - "heartbeat_scan": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "heartbeat_scan" / "target" / "wasm32-unknown-unknown" / "release" / "heartbeat_scan.wasm", - "heartbeat_scheduler": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "heartbeat_scheduler" / "target" / "wasm32-unknown-unknown" / "release" / "heartbeat_scheduler.wasm", - "cron_trigger": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "cron_trigger" / "target" / "wasm32-unknown-unknown" / "release" / "cron_trigger.wasm", - "cron_scheduler_check": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "cron_scheduler_check" / "target" / "wasm32-unknown-unknown" / "release" / "cron_scheduler_check.wasm", - "cron_scheduler_heartbeat": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "cron_scheduler_heartbeat" / "target" / "wasm32-unknown-unknown" / "release" / "cron_scheduler_heartbeat.wasm", - "workspace_restorer": REPO_ROOT / "os-apps" / "temper-agent" / "wasm" / "workspace_restorer" / "target" / "wasm32-unknown-unknown" / "release" / "workspace_restorer.wasm", - "channel_connect": REPO_ROOT / "os-apps" / "temper-channels" / "wasm" / "channel_connect" / "target" / "wasm32-unknown-unknown" / "release" / "channel_connect.wasm", - "route_message": REPO_ROOT / "os-apps" / "temper-channels" / "wasm" / "route_message" / "target" / "wasm32-unknown-unknown" / "release" / "route_message.wasm", - "send_reply": REPO_ROOT / "os-apps" / "temper-channels" / "wasm" / "send_reply" / "target" / "wasm32-unknown-unknown" / "release" / "send_reply.wasm", - } - upload_results = {} - for name, wasm_path in modules.items(): - upload_results[name] = upload_wasm(TENANT, name, wasm_path) - append_jsonl(artifact_log, {"type": "wasm_upload", "name": name, "path": str(wasm_path), "result": upload_results[name]}) - write_text(ARTIFACT_ROOT / "uploaded-modules.json", json.dumps(upload_results, indent=2)) - - workspace = create_entity(TENANT, "Workspaces", {"Name": "Pi Proof Workspace", "QuotaLimit": 100000000}) - directory = create_entity( - TENANT, - "Directories", - {"Name": "root", "Path": "/", "WorkspaceId": entity_id(workspace)}, - ) - write_text(ARTIFACT_ROOT / "fs-root.json", json.dumps({"workspace": workspace, "directory": directory}, indent=2)) - - soul_md = """# Proof Soul - -## Identity -You are Proof Soul, a governed Temper agent used to verify the Pi architecture rewrite. - -## Instructions -- Prefer deterministic mock runs for verification. -- Surface memory and skills in the prompt. -- Use tools only when the proof plan requires them. - -## Capabilities -- Run sandbox tools -- Spawn governed child agents -- Save and recall memories - -## Constraints -- Do not use destructive commands. -- Stay inside the provided workspace. -""" - skill_one_md = """# code-reviewer - -Inspect code changes for regressions, missing tests, and risky assumptions. -""" - skill_two_md = """# file-search - -Locate relevant files quickly and summarize the signal, not the noise. -""" - - soul_file = create_file_asset(TENANT, entity_id(workspace), entity_id(directory), "/soul.md", soul_md) - skill_one_file = create_file_asset(TENANT, entity_id(workspace), entity_id(directory), "/skills/code-reviewer.md", skill_one_md) - skill_two_file = create_file_asset(TENANT, entity_id(workspace), entity_id(directory), "/skills/file-search.md", skill_two_md) - - soul = create_entity( - TENANT, - "AgentSouls", - { - "Name": "Proof Soul", - "Description": "Pi agent rewrite proof identity", - "ContentFileId": entity_id(soul_file), - "AuthorId": "proof-harness", - }, - ) - soul_id = entity_id(soul) - action_with_fallback( - TENANT, - "AgentSouls", - soul_id, - ["Temper.Agent.AgentSoul.Publish", "Temper.Agent.Publish"], - {}, - ) - - skill_one = create_entity( - TENANT, - "AgentSkills", - { - "Name": "code-reviewer", - "Description": "Review changes for bugs and missing tests.", - "ContentFileId": entity_id(skill_one_file), - "Scope": "global", - }, - ) - skill_two = create_entity( - TENANT, - "AgentSkills", - { - "Name": "file-search", - "Description": "Find relevant files and summarize their purpose.", - "ContentFileId": entity_id(skill_two_file), - "Scope": "global", - }, - ) - seeded_memory = [ - create_entity( - TENANT, - "AgentMemorys", - { - "Key": "user-profile", - "Content": "The proof user prefers exact verification over discussion.", - "MemoryType": "user", - "SoulId": soul_id, - "AuthorAgentId": "proof-harness", - }, - ), - create_entity( - TENANT, - "AgentMemorys", - { - "Key": "project-context", - "Content": "Temper Pi rewrite proof must capture SSE, session trees, cron, heartbeat, channels, and MCP.", - "MemoryType": "project", - "SoulId": soul_id, - "AuthorAgentId": "proof-harness", - }, - ), - ] - setup_snapshot = { - "soul": get_entity(TENANT, "AgentSouls", soul_id), - "skills": [get_entity(TENANT, "AgentSkills", entity_id(skill_one)), get_entity(TENANT, "AgentSkills", entity_id(skill_two))], - "memory": [get_entity(TENANT, "AgentMemorys", entity_id(entry)) for entry in seeded_memory], - } - write_text(ARTIFACT_ROOT / "setup-assets.json", json.dumps(setup_snapshot, indent=2)) - - channel = create_entity( - TENANT, - "Channels", - { - "ChannelType": "webhook", - "ChannelId": "proof-webhook", - "DefaultAgentConfig": json.dumps( - { - "provider": "mock", - "model": "mock-proof", - "tools_enabled": "", - "max_turns": "4", - "sandbox_url": SANDBOX_URL, - "workdir": SANDBOX_WORKDIR, - "soul_id": soul_id, - }, - separators=(",", ":"), - ), - "WebhookUrl": "http://127.0.0.1:9988", - }, - ) - channel_id = entity_id(channel) - action_with_fallback( - TENANT, - "Channels", - channel_id, - ["Temper.OpenClaw.Channel.Connect", "Temper.OpenClaw.Connect"], - {}, - ) - route = create_entity( - TENANT, - "AgentRoutes", - { - "BindingTier": "channel", - "ChannelId": "proof-webhook", - "MatchPattern": ".*", - "AgentConfig": json.dumps( - { - "provider": "mock", - "model": "mock-proof", - "tools_enabled": "", - "max_turns": "4", - "sandbox_url": SANDBOX_URL, - "workdir": SANDBOX_WORKDIR, - }, - separators=(",", ":"), - ), - "SoulId": soul_id, - }, - ) - write_text( - ARTIFACT_ROOT / "channel-setup.json", - json.dumps( - { - "channel": get_entity(TENANT, "Channels", channel_id), - "route": get_entity(TENANT, "AgentRoutes", entity_id(route)), - }, - indent=2, - ), - ) - - direct_plan = build_mock_plan( - [ - { - "text": "Starting direct path", - "tool_calls": [ - { - "name": "bash", - "input": { - "command": "sleep 2 && printf direct-path-bash", - "workdir": SANDBOX_WORKDIR, - }, - } - ], - }, - {"final_text": "Waiting for steering check."}, - { - "text": "Steering applied: {{latest_user}}", - "tool_calls": [ - { - "name": "save_memory", - "input": { - "key": "proof-direct-memory", - "content": "saved from direct path", - "memory_type": "project", - }, - } - ], - }, - {"final_text": "Direct path finished with memory keys {{memory_keys}}."}, - ] - ) - - direct_agent = create_entity(TENANT, "TemperAgents", {"TemperAgentId": "proof-direct"}) - direct_id = entity_id(direct_agent) - action_with_fallback( - TENANT, - "TemperAgents", - direct_id, - ["Temper.Agent.TemperAgent.Configure", "Temper.Agent.Configure"], - { - "system_prompt": "Override: include the DIRECT-OVERRIDE marker.", - "user_message": direct_plan, - "model": "mock-proof", - "provider": "mock", - "max_turns": "8", - "tools_enabled": "bash,save_memory", - "workdir": SANDBOX_WORKDIR, - "sandbox_url": SANDBOX_URL, - "soul_id": soul_id, - "max_follow_ups": "5", - }, - ) - action_with_fallback( - TENANT, - "TemperAgents", - direct_id, - ["Temper.Agent.TemperAgent.Provision", "Temper.Agent.Provision"], - {}, - ) - time.sleep(0.5) - action_with_fallback( - TENANT, - "TemperAgents", - direct_id, - ["Temper.Agent.TemperAgent.Steer", "Temper.Agent.Steer"], - {"steering_messages": json.dumps([{"content": "Follow the steering marker ST-123"}])}, - ) - direct_wait = wait_entity(TENANT, "TemperAgent", direct_id, ["Completed", "Failed", "Cancelled"], 120000) - direct_entity = get_entity(TENANT, "TemperAgents", direct_id) - direct_session = get_file_text(TENANT, entity_field(direct_entity, "session_file_id", "SessionFileId")) - direct_sse = capture_sse(TENANT, "TemperAgent", direct_id, ARTIFACT_ROOT / "direct-events.sse") - direct_result = entity_result(direct_wait, direct_session) - direct_prompt = extract_prompt_from_sse(direct_sse) - write_text(ARTIFACT_ROOT / "direct-agent.json", json.dumps(direct_entity, indent=2)) - write_text(ARTIFACT_ROOT / "direct-session.jsonl", direct_session) - write_text(ARTIFACT_ROOT / "direct-prompt.txt", direct_prompt) - - direct_memories = list_entities(TENANT, "AgentMemorys") - direct_saved = [entry for entry in direct_memories if entity_field(entry, "Key") == "proof-direct-memory"] - - report["steps"]["A"] = { - "A1": step(entity_field(direct_entity, "SoulId") == soul_id, "Agent created with soul_id bound", f"soul_id={entity_field(direct_entity, 'SoulId')}"), - "A4": step("event: state_change" in direct_sse, "SSE replay returns lifecycle events", "captured direct-events.sse"), - "A5": step( - all(marker in direct_prompt for marker in ["Proof Soul", "", ""]), - "Prompt includes soul, skills, and memory blocks", - direct_prompt[:300], - ), - "A6": step( - "ProcessToolCalls" in direct_sse and "HandleToolResults" in direct_sse, - "Thinking/Executing loop is visible in events", - "ProcessToolCalls/HandleToolResults present" if "ProcessToolCalls" in direct_sse else "missing loop markers", - ), - "A7": step('"type":"message"' in direct_session and "s-" in direct_session, "Session tree persisted JSONL entries and steering branch", direct_session[:240]), - "A8": step("ST-123" in direct_sse or "ST-123" in direct_session, "Steering injection stored and observable", "steering marker present"), - "A9": step("ContinueWithSteering" in direct_sse, "Steering caused a continue transition", "ContinueWithSteering seen" if "ContinueWithSteering" in direct_sse else "missing"), - "A10": step(entity_status(direct_wait) == "Completed", "Agent completed successfully", direct_result), - "A11": step(bool(direct_saved), "save_memory created a new AgentMemory", f"count={len(direct_saved)}"), - } - - channel_plan = build_mock_plan([{"final_text": "Channel proof reply"}]) - receive_result = action_with_fallback( - TENANT, - "Channels", - channel_id, - ["Temper.OpenClaw.Channel.ReceiveMessage", "Temper.OpenClaw.ReceiveMessage"], - { - "message_id": "msg-1", - "author_id": "user-1", - "thread_id": "thread-1", - "content": channel_plan, - }, - ) - channel_sessions = wait_for_entities( - TENANT, - "ChannelSessions", - lambda entry: entity_field(entry, "ThreadId") == "thread-1", - ) - channel_session = channel_sessions[0] - channel_agent_id = entity_field(channel_session, "AgentEntityId") - channel_agent = get_entity(TENANT, "TemperAgents", channel_agent_id) - channel_wait = wait_entity(TENANT, "TemperAgent", channel_agent_id, ["Completed", "Failed", "Cancelled"], 60000) - reply_lines = wait_for_reply( - lambda line: line.get("content") == "Channel proof reply" - and line.get("thread_id") == "thread-1", - timeout_s=10.0, - poll_s=0.25, - ) - write_text( - ARTIFACT_ROOT / "channel-result.json", - json.dumps( - { - "receive_result": receive_result, - "session": channel_session, - "agent": channel_agent, - "wait": channel_wait, - "reply_lines": reply_lines, - }, - indent=2, - ), - ) - report["steps"]["B"] = { - "B1": step(True, "Channel.ReceiveMessage accepted webhook payload", "ReceiveMessage executed"), - "B2": step(bool(channel_sessions), "ChannelSession created for thread", f"session_id={entity_id(channel_session)}"), - "B3": step(entity_field(channel_agent, "SoulId") == soul_id, "Channel route spawned agent with route soul_id", f"soul_id={entity_field(channel_agent, 'SoulId')}"), - "B4": step(entity_status(channel_wait) == "Completed", "Channel-triggered agent completed", entity_result(channel_wait)), - "B5": step(any(line.get("content") == "Channel proof reply" for line in reply_lines), "send_reply delivered the agent result", json.dumps(reply_lines[-1]) if reply_lines else "no reply"), - } - - child_plan = build_mock_plan( - [ - { - "text": "child start", - "tool_calls": [ - { - "name": "bash", - "input": { - "command": "sleep 2 && printf child-ready", - "workdir": SANDBOX_WORKDIR, - }, - } - ], - }, - {"final_text": "Child waiting for steering."}, - {"final_text": "Child completed after steering: {{latest_user}}"}, - ] - ) - subagent_plan = build_mock_plan( - [ - { - "text": "spawning child", - "tool_calls": [ - { - "name": "spawn_agent", - "input": { - "task": child_plan, - "agent_id": "proof-sub-child", - "provider": "mock", - "model": "mock-proof", - "max_turns": 6, - "tools": "bash", - "soul_id": soul_id, - "background": True, - }, - } - ], - }, - { - "text": "managing child", - "tool_calls": [ - {"name": "list_agents", "input": {}}, - {"name": "steer_agent", "input": {"agent_id": "proof-sub-child", "message": "STEERED-CHILD"}}, - {"name": "run_coding_agent", "input": {"agent_type": "claude-code", "task": "subagent proof task", "workdir": SANDBOX_WORKDIR}}, - ], - }, - {"final_text": "Subagent parent done"}, - ] - ) - - sub_parent = create_entity(TENANT, "TemperAgents", {"TemperAgentId": "proof-sub-parent"}) - sub_parent_id = entity_id(sub_parent) - action_with_fallback( - TENANT, - "TemperAgents", - sub_parent_id, - ["Temper.Agent.TemperAgent.Configure", "Temper.Agent.Configure"], - { - "system_prompt": "Subagent proof parent.", - "user_message": subagent_plan, - "model": "mock-proof", - "provider": "mock", - "max_turns": "8", - "tools_enabled": "spawn_agent,list_agents,steer_agent,run_coding_agent", - "workdir": SANDBOX_WORKDIR, - "sandbox_url": SANDBOX_URL, - "soul_id": soul_id, - }, - ) - action_with_fallback( - TENANT, - "TemperAgents", - sub_parent_id, - ["Temper.Agent.TemperAgent.Provision", "Temper.Agent.Provision"], - {}, - ) - sub_parent_wait = wait_entity(TENANT, "TemperAgent", sub_parent_id, ["Completed", "Failed", "Cancelled"], 120000) - sub_parent_entity = get_entity(TENANT, "TemperAgents", sub_parent_id) - sub_parent_session = get_file_text(TENANT, entity_field(sub_parent_entity, "session_file_id", "SessionFileId")) - sub_child_entities = wait_for_entities( - TENANT, - "TemperAgents", - lambda entry: entity_field(entry, "TemperAgentId") == "proof-sub-child" - and entity_field(entry, "ParentAgentId") == sub_parent_id, - ) - sub_child_entity = sub_child_entities[0] - sub_child_id = entity_id(sub_child_entity) - sub_child_wait = wait_entity(TENANT, "TemperAgent", sub_child_id, ["Completed", "Failed", "Cancelled"], 120000) - sub_child_session = get_file_text(TENANT, entity_field(sub_child_entity, "session_file_id", "SessionFileId")) - write_text(ARTIFACT_ROOT / "subagent-parent-session.jsonl", sub_parent_session) - write_text(ARTIFACT_ROOT / "subagent-child-session.jsonl", sub_child_session) - - report["steps"]["C"] = { - "C1": step(True, "An orchestrator entity ran WASM that spawned a TemperAgent", f"parent_agent={sub_parent_id}"), - "C2": step(entity_field(sub_child_entity, "ParentAgentId") == sub_parent_id, "Child TemperAgent created with parent_agent_id", f"parent_agent_id={entity_field(sub_child_entity, 'ParentAgentId')}"), - "C3": step(entity_status(sub_child_wait) == "Completed", "Child agent completed and result was observable", entity_result(sub_child_wait, sub_child_session)), - } - report["steps"]["S"] = { - "S1": step(True, "Parent agent created with spawn_agent in tools", "tools_enabled includes spawn_agent"), - "S2": step("proof-sub-child" in sub_parent_session, "Parent invoked spawn_agent", "child id present in parent session"), - "S3": step(entity_field(sub_child_entity, "ParentAgentId") == sub_parent_id, "Child links back to parent", f"ParentAgentId={entity_field(sub_child_entity, 'ParentAgentId')}"), - "S4": step("STEERED-CHILD" in sub_child_session or "STEERED-CHILD" in entity_result(sub_child_wait, sub_child_session), "Parent steered child agent", entity_result(sub_child_wait, sub_child_session)), - "S5": step("proof-sub-child" in sub_parent_session and "- proof-sub-child:" in sub_parent_session, "list_agents exposed child status", "child id visible in tool result"), - "S6": step("Child completed after steering" in entity_result(sub_child_wait, sub_child_session), "Parent/child flow produced child result", entity_result(sub_child_wait, sub_child_session)), - "S7": step("run_coding_agent" in sub_parent_session, "Parent invoked run_coding_agent", "tool result captured"), - "S8": step("claude --permission-mode bypassPermissions --print 'subagent proof task'" in sub_parent_session, "CLI command matched expected claude-code pattern", "command string present"), - } - - depth_plan = build_mock_plan( - [ - {"tool_calls": [{"name": "spawn_agent", "input": {"task": build_mock_plan([{"final_text": "never"}])}}]}, - {"final_text": "depth-guard-done"}, - ] - ) - depth_agent = create_entity(TENANT, "TemperAgents", {"TemperAgentId": "proof-depth-guard"}) - depth_id = entity_id(depth_agent) - action_with_fallback( - TENANT, - "TemperAgents", - depth_id, - ["Temper.Agent.TemperAgent.Configure", "Temper.Agent.Configure"], - { - "user_message": depth_plan, - "model": "mock-proof", - "provider": "mock", - "max_turns": "4", - "tools_enabled": "spawn_agent", - "agent_depth": 5, - "soul_id": soul_id, - "sandbox_url": SANDBOX_URL, - "workdir": SANDBOX_WORKDIR, - }, - ) - action_with_fallback( - TENANT, - "TemperAgents", - depth_id, - ["Temper.Agent.TemperAgent.Provision", "Temper.Agent.Provision"], - {}, - ) - depth_wait = wait_entity(TENANT, "TemperAgent", depth_id, ["Completed", "Failed", "Cancelled"], 60000) - depth_entity = get_entity(TENANT, "TemperAgents", depth_id) - depth_session_file_id = entity_field(depth_entity, "session_file_id", "SessionFileId") - depth_session = get_file_text(TENANT, depth_session_file_id) if depth_session_file_id else "" - report["steps"]["S"]["S9"] = step( - "agent_depth guard hit" in depth_session, - "agent_depth guard prevented deep recursion", - "guard message present" if "agent_depth guard hit" in depth_session else "guard missing", - ) - - mcp = McpClient(MCP_BIN, 3463, ARTIFACT_ROOT / "temper-mcp.stderr.log") - try: - mcp.initialize() - mcp_plan = json.dumps(build_mock_plan([{"final_text": "MCP path ok"}])) - mcp_create = mcp.execute( - f""" -agent = await temper.create('{TENANT}', 'TemperAgents', {{}}) -aid = agent['entity_id'] -await temper.action('{TENANT}', 'TemperAgents', aid, 'Agent.TemperAgent.Configure', {{ - 'user_message': {mcp_plan}, - 'model': 'mock-proof', - 'provider': 'mock', - 'max_turns': '4', - 'tools_enabled': '', - 'soul_id': '{soul_id}', - 'sandbox_url': '{SANDBOX_URL}', - 'workdir': {json.dumps(SANDBOX_WORKDIR)} -}}) -await temper.action('{TENANT}', 'TemperAgents', aid, 'Agent.TemperAgent.Provision', {{}}) -return {{'agent_id': aid}} -""" - ) - mcp_agent_id = mcp_create["agent_id"] - mcp_wait = wait_entity(TENANT, "TemperAgent", mcp_agent_id, ["Completed", "Failed", "Cancelled"], 60000) - mcp_entity = mcp.execute(f"return await temper.get('{TENANT}', 'TemperAgents', '{mcp_agent_id}')") - write_text( - ARTIFACT_ROOT / "mcp-results.json", - json.dumps({"create": mcp_create, "entity": mcp_entity, "wait": mcp_wait}, indent=2), - ) - finally: - mcp.close() - report["steps"]["D"] = { - "D1": step(True, "MCP created, configured, and provisioned an agent", f"agent_id={mcp_agent_id}"), - "D2": step(entity_status(mcp_wait) == "Completed", "MCP-observed agent reached Completed", entity_result(mcp_wait)), - "D3": step(entity_result(mcp_wait) == "MCP path ok", "MCP result matched expected output", entity_result(mcp_wait)), - } - - cron_template = build_mock_plan([{"final_text": "cron run {{run_count}}"}]) - cron_job = create_entity( - TENANT, - "CronJobs", - { - "Name": "proof-cron", - "Schedule": "* * * * *", - "SoulId": soul_id, - "UserMessageTemplate": cron_template, - "Model": "mock-proof", - "Provider": "mock", - "ToolsEnabled": "", - "SandboxUrl": SANDBOX_URL, - "MaxTurns": "4", - "MaxRuns": "2", - }, - ) - cron_id = entity_id(cron_job) - action_with_fallback( - TENANT, - "CronJobs", - cron_id, - ["Temper.Agent.CronJob.Activate", "Temper.Agent.Activate"], - {}, - ) - action_with_fallback( - TENANT, - "CronJobs", - cron_id, - ["Temper.Agent.CronJob.Trigger", "Temper.Agent.Trigger"], - {"last_run_at": now_utc()}, - ) - cron_after_first_matches = wait_for_entities( - TENANT, - "CronJobs", - lambda entry: entity_id(entry) == cron_id and bool(entity_field(entry, "LastAgentId")), - timeout_s=20.0, - poll_s=0.25, - ) - if not cron_after_first_matches: - raise RuntimeError(f"cron proof: no last_agent_id observed for CronJob {cron_id}") - cron_after_first = cron_after_first_matches[0] - cron_agent_id = entity_field(cron_after_first, "LastAgentId") - cron_agent_wait = wait_entity(TENANT, "TemperAgent", cron_agent_id, ["Completed", "Failed", "Cancelled"], 60000) - action_with_fallback( - TENANT, - "CronJobs", - cron_id, - ["Temper.Agent.CronJob.Trigger", "Temper.Agent.Trigger"], - {"last_run_at": now_utc()}, - ) - cron_after_second_matches = wait_for_entities( - TENANT, - "CronJobs", - lambda entry: entity_id(entry) == cron_id and int(entity_field(entry, "RunCount") or 0) >= 2, - timeout_s=20.0, - poll_s=0.25, - ) - if not cron_after_second_matches: - raise RuntimeError(f"cron proof: run_count did not reach 2 for CronJob {cron_id}") - cron_after_second = cron_after_second_matches[0] - write_text( - ARTIFACT_ROOT / "cron-results.json", - json.dumps({"job_after_first": cron_after_first, "job_after_second": cron_after_second, "agent_wait": cron_agent_wait}, indent=2), - ) - report["steps"]["E"] = { - "E1": step(True, "CronJob entity created", f"cron_id={cron_id}"), - "E2": step(entity_status(cron_after_first) == "Active", "Cron job activated", f"status={entity_status(cron_after_first)}"), - "E3": step(True, "Manual Trigger action executed", f"last_agent_id={cron_agent_id}"), - "E4": step(bool(cron_agent_id), "Cron-triggered TemperAgent was created", f"agent_id={cron_agent_id}"), - "E5": step(entity_field(cron_after_first, "LastAgentId") == cron_agent_id, "CronJob tracked last_agent_id", f"LastAgentId={entity_field(cron_after_first, 'LastAgentId')}"), - "E6": step(int(entity_field(cron_after_second, "RunCount") or 0) >= 2, "Second trigger incremented run_count", f"RunCount={entity_field(cron_after_second, 'RunCount')}"), - } - - heartbeat_agent = create_entity(TENANT, "TemperAgents", {"TemperAgentId": "proof-heartbeat"}) - heartbeat_agent_id = entity_id(heartbeat_agent) - action_with_fallback( - TENANT, - "TemperAgents", - heartbeat_agent_id, - ["Temper.Agent.TemperAgent.Configure", "Temper.Agent.Configure"], - { - "user_message": build_mock_plan([{"mode": "hang"}]), - "model": "mock-proof", - "provider": "mock", - "max_turns": "4", - "tools_enabled": "", - "soul_id": soul_id, - "heartbeat_timeout_seconds": "5", - "sandbox_url": SANDBOX_URL, - "workdir": SANDBOX_WORKDIR, - }, - ) - action_with_fallback( - TENANT, - "TemperAgents", - heartbeat_agent_id, - ["Temper.Agent.TemperAgent.Provision", "Temper.Agent.Provision"], - {}, - ) - time.sleep(1) - heartbeat_monitor = create_entity(TENANT, "HeartbeatMonitors", {"ScanIntervalSeconds": "1"}) - heartbeat_monitor_id = entity_id(heartbeat_monitor) - action_with_fallback( - TENANT, - "HeartbeatMonitors", - heartbeat_monitor_id, - ["Temper.Agent.HeartbeatMonitor.Start", "Temper.Agent.Start"], - {}, - ) - heartbeat_wait = wait_entity(TENANT, "TemperAgent", heartbeat_agent_id, ["Failed", "Completed"], 30000) - heartbeat_sse = capture_sse(TENANT, "TemperAgent", heartbeat_agent_id, ARTIFACT_ROOT / "heartbeat-events.sse") - report["steps"]["H"] = { - "H1": step(True, "Heartbeat test agent created with short timeout", f"agent_id={heartbeat_agent_id}"), - "H2": step(True, "Mock hang plan provisioned", "provider=mock, mode=hang"), - "H3": step(True, "Heartbeat monitor started and scanned", f"monitor_id={heartbeat_monitor_id}"), - "H4": step(entity_status(heartbeat_wait) == "Failed", "Stale agent transitioned to Failed", entity_field(heartbeat_wait, "ErrorMessage", "error_message") or entity_result(heartbeat_wait)), - "H5": step("TimeoutFail" in heartbeat_sse, "SSE replay captured TimeoutFail state change", "TimeoutFail present" if "TimeoutFail" in heartbeat_sse else "missing"), - } - - memory_agent = create_entity(TENANT, "TemperAgents", {"TemperAgentId": "proof-memory"}) - memory_agent_id = entity_id(memory_agent) - action_with_fallback( - TENANT, - "TemperAgents", - memory_agent_id, - ["Temper.Agent.TemperAgent.Configure", "Temper.Agent.Configure"], - { - "user_message": build_mock_plan([{"final_text": "memory keys={{memory_keys}} count={{memory_count}}"}]), - "model": "mock-proof", - "provider": "mock", - "max_turns": "4", - "tools_enabled": "", - "soul_id": soul_id, - "sandbox_url": SANDBOX_URL, - "workdir": SANDBOX_WORKDIR, - }, - ) - action_with_fallback( - TENANT, - "TemperAgents", - memory_agent_id, - ["Temper.Agent.TemperAgent.Provision", "Temper.Agent.Provision"], - {}, - ) - memory_wait = wait_entity(TENANT, "TemperAgent", memory_agent_id, ["Completed", "Failed", "Cancelled"], 60000) - memory_entity = get_entity(TENANT, "TemperAgents", memory_agent_id) - memory_session = get_file_text(TENANT, entity_field(memory_entity, "session_file_id", "SessionFileId")) - memory_result = entity_result(memory_wait, memory_session) - report["steps"]["M"] = { - "M1": step(True, "Second agent created with same soul_id", f"agent_id={memory_agent_id}"), - "M2": step("proof-direct-memory" in memory_result and "project-context" in memory_result, "Cross-session memory loaded into prompt", memory_result), - "M3": step("count=" in memory_result, "Memory-aware mock response surfaced recalled knowledge", memory_result), - } - - compaction_notes = "X" * 6000 - compaction_agent = create_entity(TENANT, "TemperAgents", {"TemperAgentId": "proof-compaction"}) - compaction_agent_id = entity_id(compaction_agent) - action_with_fallback( - TENANT, - "TemperAgents", - compaction_agent_id, - ["Temper.Agent.TemperAgent.Configure", "Temper.Agent.Configure"], - { - "user_message": json.dumps({"notes": compaction_notes, "mock_plan": {"steps": [{"final_text": "compaction proof ok"}]}}), - "model": "mock-proof", - "provider": "mock", - "max_turns": "6", - "tools_enabled": "", - "soul_id": soul_id, - "reserve_tokens": "199500", - "keep_recent_tokens": "100", - "sandbox_url": SANDBOX_URL, - "workdir": SANDBOX_WORKDIR, - }, - ) - action_with_fallback( - TENANT, - "TemperAgents", - compaction_agent_id, - ["Temper.Agent.TemperAgent.Provision", "Temper.Agent.Provision"], - {}, - ) - compaction_wait = wait_entity(TENANT, "TemperAgent", compaction_agent_id, ["Completed", "Failed", "Cancelled"], 60000) - compaction_entity = get_entity(TENANT, "TemperAgents", compaction_agent_id) - compaction_session = get_file_text(TENANT, entity_field(compaction_entity, "session_file_id", "SessionFileId")) - write_text(ARTIFACT_ROOT / "compaction-session.jsonl", compaction_session) - report["steps"]["X"] = { - "X1": step("compaction" in compaction_session, "Compaction entry was written into the session tree", "compaction entry present" if "compaction" in compaction_session else "missing"), - "X2": step(entity_status(compaction_wait) == "Completed", "Agent resumed after compaction", entity_result(compaction_wait, compaction_session)), - } - - trajectories_summary = get_json("/observe/trajectories?entity_type=TemperAgent&failed_limit=20", tenant=TENANT, headers=ADMIN_HEADERS)["json"] - write_text(ARTIFACT_ROOT / "trajectories.json", json.dumps(trajectories_summary, indent=2)) - - specs_summary = { - "temper-agent": apps["temper-agent"], - "temper-channels": apps["temper-channels"], - "temper-fs": apps["temper-fs"], - } - - def table(section): - rows = [ - "| Step | Expected | Actual | Status |", - "|---|---|---|---|", - ] - for key, value in report["steps"][section].items(): - actual = value["actual"].replace("|", "\\|").replace("\n", "
") - rows.append(f"| {key} | {value['expected']} | {actual} | {value['status']} |") - return "\n".join(rows) - - limitations = [] - if report["steps"]["H"]["H4"]["status"] != "PASS": - limitations.append("Heartbeat timeout did not fail the hanging agent.") - if report["steps"]["X"]["X1"]["status"] != "PASS": - limitations.append("Compaction scenario did not emit a compaction entry.") - if not limitations: - limitations.append("None observed in the proof run.") - - report_text = f"""# Governed Agent Architecture E2E Proof - -## Date -{report['date']} - -## Branch -{report['branch']} - -## Commit -{report['commit']} - -## Server -`{SERVER}` against tenant `{TENANT}` - -## Specs Deployed -- `temper-fs`: {json.dumps(specs_summary['temper-fs'])} -- `temper-agent`: {json.dumps(specs_summary['temper-agent'])} -- `temper-channels`: {json.dumps(specs_summary['temper-channels'])} - -## Trigger Path A: Direct OData API -{table('A')} - -## Trigger Path B: Channel Webhook -{table('B')} - -## Trigger Path C: WASM Orchestration -{table('C')} - -## Trigger Path D: MCP Tool Call -{table('D')} - -## Trigger Path E: Cron Job -{table('E')} - -## Subagent + Coding Agent Verification -{table('S')} - -## Heartbeat Monitoring Verification -{table('H')} - -## Cross-Session Memory -{table('M')} - -## Compaction -{table('X')} - -## Artifacts - -### Session Tree Dump -```jsonl -{direct_session} -``` - -### SSE Events Captured -```text -{direct_sse} -``` - -### OTS Trajectory Summary -```json -{json.dumps(trajectories_summary, indent=2)} -``` - -### System Prompt Assembly -```text -{direct_prompt} -``` - -## Current Limitations -""" + "\n".join(f"- {item}" for item in limitations) + f""" - -## Reproduction Commands -```bash -python3 scripts/temper_agent_e2e_proof.py -cargo test --workspace -``` -""" - write_text(REPORT_PATH, report_text) - write_text(ARTIFACT_ROOT / "proof-summary.json", json.dumps(report, indent=2)) - print(json.dumps({"report": str(REPORT_PATH), "tenant": TENANT, "artifacts": str(ARTIFACT_ROOT)}, indent=2)) - return 0 - - -if __name__ == "__main__": - try: - raise SystemExit(main()) - except Exception as exc: - print(f"proof failed: {exc}", file=sys.stderr) - raise diff --git a/scripts/verify-all.sh b/scripts/verify-all.sh deleted file mode 100755 index b0f194c76..000000000 --- a/scripts/verify-all.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -set -euo pipefail -echo "=== Temper Full Verification Cascade ===" -echo "Running cargo test --workspace..." -cargo test --workspace -echo "" -echo "=== All tests passed ===" diff --git a/scripts/verify-latency-observability-package.sh b/scripts/verify-latency-observability-package.sh deleted file mode 100755 index a9c2a257e..000000000 --- a/scripts/verify-latency-observability-package.sh +++ /dev/null @@ -1,152 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -TEMPERPAW_WORKTREE="${TEMPERPAW_WORKTREE:-/Users/seshendranalla/Development/temperpaw-worktrees/latency-observability-program}" -MODE="${1:-quick}" - -if [[ "$MODE" != "quick" && "$MODE" != "full" ]]; then - echo "usage: $0 [quick|full]" >&2 - exit 2 -fi - -pass() { - printf 'ok: %s\n' "$1" -} - -fail() { - printf 'error: %s\n' "$1" >&2 - exit 1 -} - -require_file() { - local path="$1" - [[ -f "$path" ]] || fail "missing file: $path" - pass "found ${path#$ROOT/}" -} - -require_pattern() { - local path="$1" - local pattern="$2" - local label="$3" - rg -q --fixed-strings "$pattern" "$path" || fail "$label missing from ${path#$ROOT/}" - pass "$label" -} - -require_regex() { - local path="$1" - local pattern="$2" - local label="$3" - rg -q "$pattern" "$path" || fail "$label missing from ${path#$ROOT/}" - pass "$label" -} - -echo "== Latency/observability package preflight ==" -echo "Temper worktree: $ROOT" -echo "TemperPaw worktree: $TEMPERPAW_WORKTREE" -echo "Mode: $MODE" -echo - -require_file "$ROOT/docs/temper-latency-observability-report.html" -require_file "$ROOT/docs/adrs/0081-latency-observability-acceleration-program.md" -require_file "$ROOT/docs/adrs/0082-projection-correctness-observability.md" -require_file "$ROOT/docs/adrs/0083-trace-budget-and-fanout-summarization.md" -require_file "$ROOT/docs/adrs/0084-authz-latency-phase-instrumentation.md" -require_file "$ROOT/docs/runbooks/datadog-postgres-dbm.md" -require_file "$ROOT/docs/runbooks/latency-observability-release.md" -require_file "$ROOT/scripts/datadog-postgres-dbm-setup.sql" - -require_pattern "$ROOT/docs/temper-latency-observability-report.html" "Program Progress" "living dashboard progress section" -require_pattern "$ROOT/docs/temper-latency-observability-report.html" "OBS-005" "OBS-005 dashboard task" -require_pattern "$ROOT/docs/temper-latency-observability-report.html" "0083-trace-budget-and-fanout-summarization" "ADR-0083 dashboard link" -require_pattern "$ROOT/docs/temper-latency-observability-report.html" "0084-authz-latency-phase-instrumentation" "ADR-0084 dashboard link" -require_pattern "$ROOT/docs/temper-latency-observability-report.html" "Runtime deployment pending" "runtime deployment blocker recorded" - -require_pattern "$ROOT/crates/temper-server/src/profiling.rs" "TEMPER_PROFILING_CONTINUOUS" "profiler continuous gate" -require_pattern "$ROOT/crates/temper-server/src/profiling/metrics.rs" "datadog.profiling.rust.profiles_uploaded" "profiler upload metric" -require_pattern "$ROOT/crates/temper-store-postgres/src/metrics.rs" "temper_postgres_transaction_duration_ms" "Postgres transaction metric" -require_pattern "$ROOT/crates/temper-store-postgres/src/metrics.rs" "temper_postgres_pool_acquire_duration_ms" "Postgres pool metric" -require_pattern "$ROOT/crates/temper-server/src/query_projection_metrics.rs" "temper_query_projection_replay_parity_check_total" "projection replay parity metric" -require_pattern "$ROOT/crates/temper-server/src/odata/read_support/shadow.rs" "TEMPER_ODATA_CATALOG_SHADOW_READ_EVERY" "projection shadow-read gate" -require_regex "$ROOT/crates/temper-store-turso/src/schema/query_plane.rs" "fields[[:space:]]+TEXT NOT NULL DEFAULT '\\{\\}'" "Turso catalog fields column" -require_pattern "$ROOT/crates/temper-observe/src/otel/sampler.rs" "temper_trace_sampler_decisions_total" "trace sampler decision metric" -require_pattern "$ROOT/crates/temper-observe/src/otel/sampler.rs" "TEMPER_TRACE_DISPATCH_BACKGROUND_SAMPLE_PCT" "dispatch background trace budget flag" -require_pattern "$ROOT/crates/temper-server/src/trigger/dispatcher.rs" "reaction.fired_count" "reaction fanout summary span" -require_pattern "$ROOT/crates/temper-authz/src/metrics.rs" "temper_cedar_evaluation_duration_ms" "Cedar duration ms metric" -require_pattern "$ROOT/crates/temper-authz/src/metrics.rs" "temper_cedar_evaluation_phase_duration_ms" "Cedar phase duration metric" -require_pattern "$ROOT/crates/temper-authz/src/metrics.rs" "temper_cedar_request_attribute_count" "Cedar request shape metric" -require_pattern "$ROOT/crates/temper-authz/src/engine/mod.rs" "CedarEvaluationRecorder" "Cedar phase recorder" - -require_pattern "$ROOT/docs/runbooks/latency-observability-release.md" "TEMPER_TRACE_DISPATCH_BACKGROUND_SAMPLE_PCT=25" "trace budget runtime flag in runbook" -require_pattern "$ROOT/docs/runbooks/latency-observability-release.md" "temper_cedar_evaluation_phase_duration_ms" "Cedar phase metric in runbook" -require_pattern "$ROOT/docs/runbooks/latency-observability-release.md" "Datadog Metric Configuration" "Datadog metric config section" -require_pattern "$ROOT/docs/runbooks/latency-observability-release.md" "Trace budget" "trace budget live proof row" -require_pattern "$ROOT/docs/runbooks/latency-observability-release.md" "python3 scripts/temper_agent_e2e_proof.py" "e2e proof command in runbook" - -if [[ ! -d "$TEMPERPAW_WORKTREE" ]]; then - fail "TemperPaw worktree not found: $TEMPERPAW_WORKTREE" -fi - -require_file "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" -require_file "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" -require_file "$TEMPERPAW_WORKTREE/scripts/configure_metric_percentiles.py" -require_file "$TEMPERPAW_WORKTREE/scripts/read_datadog_snapshot.py" -python3 -m json.tool "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" >/dev/null -pass "TemperPaw dashboard JSON parses" -python3 -m json.tool "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" >/dev/null -pass "TemperPaw monitor JSON parses" -python3 -m py_compile "$TEMPERPAW_WORKTREE/scripts/configure_metric_percentiles.py" "$TEMPERPAW_WORKTREE/scripts/deploy_monitors.py" -pass "TemperPaw Datadog deploy helpers compile" -python3 -m py_compile "$TEMPERPAW_WORKTREE/scripts/read_datadog_snapshot.py" -pass "TemperPaw Datadog snapshot helper compiles" -require_pattern "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" "Trace Budget (ADR-0083)" "Trace Budget dashboard group" -require_pattern "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" "service:temperpaw" "Railway production service tag in dashboard" -require_pattern "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" "temper_trace_sampler_decisions_total" "trace sampler dashboard query" -require_pattern "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" "AuthZ Phase Breakdown (ADR-0084)" "AuthZ phase dashboard group" -require_pattern "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" "temper_cedar_evaluation_phase_duration_ms" "Cedar phase dashboard query" -if rg -q --fixed-strings "service:openpaw" "$TEMPERPAW_WORKTREE/dd-dashboards/temperpaw-overview.json" "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json"; then - fail "TemperPaw Datadog config still targets service:openpaw; Railway production exports DD_SERVICE=temperpaw" -fi -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[Temper] Trace Sampler Metrics Missing" "trace sampler missing monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "service:temperpaw" "Railway production service tag in monitors" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[Temper] Trace Sampler Delegated Volume Spike" "delegated volume monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[Temper] Dispatch Background Trace Budget Disabled" "trace budget disabled monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[Temper] Cedar Evaluation Duration Max Regression" "Cedar max fallback monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[Temper] Cedar AuthZ Phase Error" "Cedar phase error monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[TemperPaw] APM Request Rate Missing" "APM request rate monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[TemperPaw] APM HTTP 5xx Spike" "APM error monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[TemperPaw] APM Error Rate Spike" "APM error rate monitor" -require_pattern "$TEMPERPAW_WORKTREE/dd-monitors/temperpaw-monitors.json" "[TemperPaw] APM HTTP Duration p95 Regression" "APM duration monitor" -require_pattern "$TEMPERPAW_WORKTREE/scripts/configure_metric_percentiles.py" "include_percentiles" "Datadog percentile configuration helper" - -if [[ "$MODE" == "full" ]]; then - echo - echo "== Full focused verification ==" - ( - cd "$ROOT" - cargo fmt --check - cargo check -p temper-cli - cargo check -p temper-server - cargo test -p temper-authz -- --nocapture - cargo test -p temper-server profiling::tests --lib -- --nocapture - cargo test -p temper-observe otel --lib -- --nocapture - cargo test -p temper-store-postgres metrics --lib -- --nocapture - cargo test -p temper-server query_projection_metrics --lib -- --nocapture - cargo test -p temper-server odata::read_support --lib -- --nocapture - cargo test -p temper-server --test query_projection_backfill -- --nocapture - cargo test -p temper-store-turso load_entity_catalog_rows_returns_full_projected_fields --lib -- --nocapture - cargo test -p temper-store-turso export_query_projections_returns_all_fields_for_migration --lib -- --nocapture - git diff --check - ) - ( - cd "$TEMPERPAW_WORKTREE" - git diff --check - ) - pass "full focused verification passed" -else - echo - echo "quick mode skipped cargo tests; run '$0 full' before PR." -fi - -echo -pass "latency/observability package preflight passed" diff --git a/ui/landing/index.html.bak b/ui/landing/index.html.bak deleted file mode 100644 index b845b754b..000000000 --- a/ui/landing/index.html.bak +++ /dev/null @@ -1,947 +0,0 @@ - - - - - - Temper — The Operating System for Agents - - - - - - - - - - - - - - -
-
- - - - -
-
-
-
757 tests · 19 crates · Built with Rust
-
- - - Temper mascot - -
-

The operating system for agents

-

Temper is to agents what an OS is to processes. Every action is verified before it runs, authorized when it runs, and recorded after it runs.

-
- Get Started - Read the Paper -
-
-
- - -
-
-
-

What Agents Get

-

Everything an agent needs to operate safely

-

Instead of giving agents raw API keys and direct database access, you give them Temper.

-
-
-
-

Verified state

-

The agent describes what it needs. Temper generates a state machine, proves it correct across all reachable states, and deploys an API. Bad states are ruled out before the spec loads.

-
-
-
-

Governed access

-

Cedar authorization evaluates every action. Default-deny. When the agent tries something not yet permitted, the denial surfaces to the human for approval.

-
-
-
-

Complete audit trail

-

Every action carries agent identity, before/after state, and the authorization decision that governed it. Multiple agents sharing a Temper instance see each other's changes.

-
-
-
-

WASM integrations

-

External system calls execute through sandboxed WASM modules, gated by Cedar policies. No raw API keys. No direct network access.

-
-
-
-

Persistent state + event sourcing

-

State survives restarts. Every transition is an event in a durable journal. Long-running workflows pick up where they left off.

-
-
-
-

Self-describing API

-

The generated OData v4 API includes a $metadata endpoint. Agents discover the full surface without documentation.

-
-
-
-
- - -
-
-
-

How It Works

-

One interface. Complete governance.

-

Agents interact through a sandboxed Python REPL. Temper mediates all access.

- -
-
-

Agent

-

Claude Code, Cursor, OpenClaw, LangChain, ...

-
-
-
-
MCP (search + execute)
- -
-
-

Monty Sandbox

-

temper.start_server() · submit_specs() · action()

-
-
-
- -
-
-

Temper Core

-
-
Spec Parse
-
Verify L0-L3
-
Deploy
-
Cedar AuthZ
-
WASM
-
OData API
-
-
-
Event Sourcing
-
OTEL
-
Evolution Engine
-
-
-
-
- -
-
-

Storage

-

Postgres or Turso/libSQL

-
-
-
-
- - -
-
-
-

Verification

-

Four levels of proof before production

-

Every spec passes all four levels. No exceptions.

-
-
-

L0

-

Z3 SMT

-

Guards satisfiable, invariants inductive, no unreachable states.

-
-
-

L1

-

Stateright

-

Exhaustive state space exploration, safety + liveness properties.

-
-
-

L2

-

Simulation

-

Fault injection, message delays, drops, crashes. Reproducible via seeded PRNG.

-
-
-

L3

-

Proptest

-

Random action sequences with shrinking to minimal counterexamples.

-
-
-
- An agent can't ship an order without payment captured — not because someone reviewed the code, but because the invariant was proven to hold across all reachable states before the spec was loaded. -
-
-
- - -
-
-
-

Use Cases

-

Two modes, one platform

-

Same verification. Same governance. Same audit trail. What changes is who writes the specs and who sets the policies.

-
-
-

Agent OS

-

The agent builds and operates. The human sets policies. Cedar's default-deny posture means the agent can't exceed its authorization without explicit human approval.

-

Temper doesn't tell the agent what to do. It mediates the agent's access to state and external systems, ensures actions are authorized, and maintains a verifiable record of everything that happened.

-
-
-

Agent-built applications

-

Developer agents build full applications. User agents or humans consume them. When users encounter something the app can't do yet, the system captures it as an unmet intent.

-
User: "Split my order into two shipments" -App: (no matching action) → unmet intent - -→ Developer agent notified -→ Spec updated → verification cascade -→ Hot-deploy → next attempt succeeds
-
-
-
-
- - -
-
-
-

Quick Start

-

Agents write Python. Temper handles the rest.

-

The Monty sandbox exposes the full Temper API through a typed Python object.

-
-
- - monty-repl -
-
# 1. Start the server -await temper.start_server() - -# 2. Describe what you need — Temper verifies and deploys it -await temper.submit_specs("my-app", { - "Order.ioa.toml": order_spec, - "model.csdl.xml": data_model -}) -# → Verification cascade runs (L0-L3) -# → API is live if all levels pass - -# 3. Operate through the verified API -await temper.create("my-app", "Orders", {"title": "New order"}) -await temper.action("my-app", "Orders", "order-1", "SubmitOrder", {"items": 3}) - -# 4. If Cedar denies an action, wait for human approval -decision = await temper.poll_decision("my-app", "PD-abc123") -# → Human approves via Observe UI → retry succeeds
-
-
-
- - -
-
-
-

Positioning

-

How Temper is different

-

Temper often gets compared to things it isn't.

-
-
- Not an agent framework -

LangChain, CrewAI, OpenClaw, AutoGen build agents. Temper is the operating layer agents run on top of. You bring your own agent; Temper governs what it can do.

-
-
- Not a workflow engine -

Temporal and Inngest orchestrate steps. Temper verifies the state machine itself before it runs — proving invariants hold across all reachable states, not just the happy path.

-
-
- Not an API framework -

Rails, Django, FastAPI require controllers, routes, middleware. Temper generates the API from specifications. The spec is the app.

-
-
- Not a database -

Temper uses Postgres or Turso for persistence. It adds a verified state machine layer and event sourcing on top.

-
-
-
-
- - -
- - - - - - diff --git a/ui/landing/index.html.bak2 b/ui/landing/index.html.bak2 deleted file mode 100644 index 4c13eb085..000000000 --- a/ui/landing/index.html.bak2 +++ /dev/null @@ -1,930 +0,0 @@ - - - - - - Temper — The Operating System for Agents - - - - - - - - - - - - - - -
-
- - - - -
-
-
-
757 tests · 19 crates · Built with Rust
-
- - - Temper mascot - -
-

The operating system for agents

-

Temper is to agents what an OS is to processes. Every action is verified before it runs, authorized when it runs, and recorded after it runs.

-
- Get Started - Read the Paper -
-
-
- - -
-
-
-

What Agents Get

-

Everything an agent needs to operate safely

-

Instead of giving agents raw API keys and direct database access, you give them Temper.

-
-
-
-

Verified state

-

The agent describes what it needs. Temper generates a state machine, proves it correct across all reachable states, and deploys an API. Bad states are ruled out before the spec loads.

-
-
-
-

Governed access

-

Cedar authorization evaluates every action. Default-deny. When the agent tries something not yet permitted, the denial surfaces to the human for approval.

-
-
-
-

Complete audit trail

-

Every action carries agent identity, before/after state, and the authorization decision that governed it. Multiple agents sharing a Temper instance see each other's changes.

-
-
-
-

WASM integrations

-

External system calls execute through sandboxed WASM modules, gated by Cedar policies. No raw API keys. No direct network access.

-
-
-
-

Persistent state + event sourcing

-

State survives restarts. Every transition is an event in a durable journal. Long-running workflows pick up where they left off.

-
-
-
-

Self-describing API

-

The generated OData v4 API includes a $metadata endpoint. Agents discover the full surface without documentation.

-
-
-
-
- - -
-
-
-

How It Works

-

One interface. Complete governance.

-

Agents interact through a sandboxed Python REPL. Temper mediates all access.

- -
-
-

Agent

-

Claude Code, Cursor, OpenClaw, LangChain, ...

-
-
-
-
MCP (search + execute)
- -
-
-

Monty Sandbox

-

temper.start_server() · submit_specs() · action()

-
-
-
- -
-
-

Temper Core

-
-
Spec Parse
-
Verify L0-L3
-
Deploy
-
Cedar AuthZ
-
WASM
-
OData API
-
-
-
Event Sourcing
-
OTEL
-
Evolution Engine
-
-
-
-
- -
-
-

Storage

-

Postgres or Turso/libSQL

-
-
-
-
- - -
-
-
-

Verification

-

Four levels of proof before production

-

Every spec passes all four levels. No exceptions.

-
-
-

L0

-

Z3 SMT

-

Guards satisfiable, invariants inductive, no unreachable states.

-
-
-

L1

-

Stateright

-

Exhaustive state space exploration, safety + liveness properties.

-
-
-

L2

-

Simulation

-

Fault injection, message delays, drops, crashes. Reproducible via seeded PRNG.

-
-
-

L3

-

Proptest

-

Random action sequences with shrinking to minimal counterexamples.

-
-
-
- An agent can't ship an order without payment captured — not because someone reviewed the code, but because the invariant was proven to hold across all reachable states before the spec was loaded. -
-
-
- - -
-
-
-

Use Cases

-

Two modes, one platform

-

Same verification. Same governance. Same audit trail. What changes is who writes the specs and who sets the policies.

-
-
-

Agent OS

-

The agent builds and operates. The human sets policies. Cedar's default-deny posture means the agent can't exceed its authorization without explicit human approval.

-

Temper doesn't tell the agent what to do. It mediates the agent's access to state and external systems, ensures actions are authorized, and maintains a verifiable record of everything that happened.

-
-
-

Agent-built applications

-

Developer agents build full applications. User agents or humans consume them. When users encounter something the app can't do yet, the system captures it as an unmet intent.

-
User: "Split my order into two shipments" -App: (no matching action) → unmet intent - -→ Developer agent notified -→ Spec updated → verification cascade -→ Hot-deploy → next attempt succeeds
-
-
-
-
- - -
-
-
-

Quick Start

-

Agents write Python. Temper handles the rest.

-

The Monty sandbox exposes the full Temper API through a typed Python object.

-
-
- - monty-repl -
-
# 1. Start the server -await temper.start_server() - -# 2. Describe what you need — Temper verifies and deploys it -await temper.submit_specs("my-app", { - "Order.ioa.toml": order_spec, - "model.csdl.xml": data_model -}) -# → Verification cascade runs (L0-L3) -# → API is live if all levels pass - -# 3. Operate through the verified API -await temper.create("my-app", "Orders", {"title": "New order"}) -await temper.action("my-app", "Orders", "order-1", "SubmitOrder", {"items": 3}) - -# 4. If Cedar denies an action, wait for human approval -decision = await temper.poll_decision("my-app", "PD-abc123") -# → Human approves via Observe UI → retry succeeds
-
-
-
- - -
-
-
-

Positioning

-

How Temper is different

-

Temper often gets compared to things it isn't.

-
-
- Not an agent framework -

LangChain, CrewAI, OpenClaw, AutoGen build agents. Temper is the operating layer agents run on top of. You bring your own agent; Temper governs what it can do.

-
-
- Not a workflow engine -

Temporal and Inngest orchestrate steps. Temper verifies the state machine itself before it runs — proving invariants hold across all reachable states, not just the happy path.

-
-
- Not an API framework -

Rails, Django, FastAPI require controllers, routes, middleware. Temper generates the API from specifications. The spec is the app.

-
-
- Not a database -

Temper uses Postgres or Turso for persistence. It adds a verified state machine layer and event sourcing on top.

-
-
-
-
- - -
- - - - - - From 51ff0aec08503809b3861336add7274cdff74531 Mon Sep 17 00:00:00 2001 From: Rita Agafonova <36133358+rita-aga@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:06:41 -0400 Subject: [PATCH 4/5] feat: verify-temper feature map covers the enumerated surface (mcp-bridge, observe-ui added; unmapped verbs listed with reasons) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VLPhB9kjLeE48kLUyAXXq2 --- .agents/skills/verify-temper/features/README.md | 11 +++++++++++ .agents/skills/verify-temper/features/mcp-bridge.md | 13 +++++++++++++ .agents/skills/verify-temper/features/observe-ui.md | 13 +++++++++++++ 3 files changed, 37 insertions(+) create mode 100644 .agents/skills/verify-temper/features/mcp-bridge.md create mode 100644 .agents/skills/verify-temper/features/observe-ui.md diff --git a/.agents/skills/verify-temper/features/README.md b/.agents/skills/verify-temper/features/README.md index a70a56d93..1f9486fac 100644 --- a/.agents/skills/verify-temper/features/README.md +++ b/.agents/skills/verify-temper/features/README.md @@ -1,7 +1,18 @@ # Feature map +Surface enumeration (from `temper-cli` subcommands + served routes): Serve (OData API + Observe UI), Mcp (stdio bridge + REPL), Verify/VerifyIoa/VerifyRemote (cascade), DST suites, Init/Codegen (scaffolding), Install, Decide (approval CLI), MigrateTursoToPostgres (ops migration). + | Feature | File | Drive when you changed | |---|---|---| | Serve + OData | serve-and-odata.md | server, routes, stores, platform bootstrap | | Spec cascade | spec-cascade.md | any `.ioa.toml`, temper-spec, temper-verify | | DST proof | dst-proof.md | temper-runtime, temper-jit, temper-server sim paths | +| MCP bridge + REPL | mcp-bridge.md | temper-mcp, temper-sandbox, SDK surface | +| Observe UI + decisions | observe-ui.md | temper-observe, temper-authz, approval flow | + +## Not yet mapped + +- Init/Codegen - scaffolding verbs; drive = run them in a temp dir and build the output +- Install - app install flow; needs a target app checkout +- Decide (CLI) - covered indirectly by observe-ui.md's decision flow +- MigrateTursoToPostgres - one-way ops migration; drive only against scratch data diff --git a/.agents/skills/verify-temper/features/mcp-bridge.md b/.agents/skills/verify-temper/features/mcp-bridge.md new file mode 100644 index 000000000..065c51e4d --- /dev/null +++ b/.agents/skills/verify-temper/features/mcp-bridge.md @@ -0,0 +1,13 @@ +# MCP bridge and REPL + +## Sub-features +Stdio MCP server, sandboxed Python REPL, `temper.*` API (submit specs, create entities, invoke actions). + +## How to get to it (user POV) +Agent clients (Claude Code, Codex) connect over stdio: `cargo run -p temper-cli -- mcp` proxies to a running serve instance. + +## Driving it +Start serve first, then the bridge. Through the REPL: `await temper.specs("default")`, create an entity, invoke an action, read it back over OData. + +## Gotchas +The bridge proxies - it does not serve. A dead serve behind it turns every call into a transport error that looks like an auth failure. diff --git a/.agents/skills/verify-temper/features/observe-ui.md b/.agents/skills/verify-temper/features/observe-ui.md new file mode 100644 index 000000000..9def67c5e --- /dev/null +++ b/.agents/skills/verify-temper/features/observe-ui.md @@ -0,0 +1,13 @@ +# Observe UI + +## Sub-features +Web UI on the serve port: entity browser, pending Cedar decisions, approval flow (`temper decide` is the CLI equivalent). + +## How to get to it (user POV) +Browser at `http://localhost:/observe`. Auth-gated: unauthenticated requests 401 (that is fail-closed, not breakage). + +## Driving it +Browser tooling against /observe after authenticating; or drive the decision flow headlessly: trigger a Cedar-denied action over OData, list pending decisions, approve, re-invoke. + +## Gotchas +`/observe/health` is behind the same auth - use `/healthz` for liveness. A denial that never surfaces as a pending decision is a product finding, not a driver error. From 3450e61639307b39fb566cef8f38e69e1360dc0e Mon Sep 17 00:00:00 2001 From: Rita Agafonova <36133358+rita-aga@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:16:36 -0400 Subject: [PATCH 5/5] fix: verify-temper enumeration missed the /api governance routes and /_admin - now listed (route trees verified from server source) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VLPhB9kjLeE48kLUyAXXq2 --- .agents/skills/verify-temper/features/README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.agents/skills/verify-temper/features/README.md b/.agents/skills/verify-temper/features/README.md index 1f9486fac..234f8edf7 100644 --- a/.agents/skills/verify-temper/features/README.md +++ b/.agents/skills/verify-temper/features/README.md @@ -1,6 +1,10 @@ # Feature map -Surface enumeration (from `temper-cli` subcommands + served routes): Serve (OData API + Observe UI), Mcp (stdio bridge + REPL), Verify/VerifyIoa/VerifyRemote (cascade), DST suites, Init/Codegen (scaffolding), Install, Decide (approval CLI), MigrateTursoToPostgres (ops migration). +Surface enumeration. + +Served route trees (crates/temper-server/src): `/tdata` (OData), `/observe` (UI + health), `/api` (authorize, decisions, policies, audit, repl), `/_admin` (profiling), `/healthz`. +CLI verbs (temper-cli): Serve, Mcp, Verify, VerifyIoa, VerifyRemote, Init, Codegen, Install, Decide, MigrateTursoToPostgres. +Plus the DST suites (crates/temper-platform/tests). | Feature | File | Drive when you changed | |---|---|---| @@ -12,6 +16,9 @@ Surface enumeration (from `temper-cli` subcommands + served routes): Serve (ODat ## Not yet mapped +- `/api` governance routes (authorize, policies, audit) - the Cedar policy/audit surface; decisions is partially covered by observe-ui.md, the rest needs its own file +- `/_admin` profiling (cpu/wall) - ops-only; drive read-only + - Init/Codegen - scaffolding verbs; drive = run them in a temp dir and build the output - Install - app install flow; needs a target app checkout - Decide (CLI) - covered indirectly by observe-ui.md's decision flow