diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml deleted file mode 100644 index d34bcd4..0000000 --- a/.github/workflows/pages.yml +++ /dev/null @@ -1,49 +0,0 @@ -name: GitHub Pages demo - -on: - push: - branches: - - main - workflow_dispatch: - -concurrency: - group: pages - cancel-in-progress: false - -permissions: - contents: read - -jobs: - build: - runs-on: ubuntu-24.04 - steps: - - name: Checkout - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 - with: - persist-credentials: false - - name: Set up Node - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e - with: - node-version: "20.19.0" - - name: Build static demo - run: npm --prefix packages/orchestration run build:pages-demo - - name: Configure GitHub Pages - uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b - - name: Upload GitHub Pages artifact - uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b - with: - path: dist/pages-demo - - deploy: - environment: - name: github-pages - url: ${{ steps.deployment.outputs.page_url }} - runs-on: ubuntu-24.04 - needs: build - permissions: - pages: write - id-token: write - steps: - - name: Deploy GitHub Pages - id: deployment - uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e diff --git a/.gitignore b/.gitignore index 82d28e4..0e318b5 100644 --- a/.gitignore +++ b/.gitignore @@ -60,6 +60,7 @@ evals/results/verification-baseline/ .codex/ .claude/ .cursor/ +.continue/ .gemini/ .kilo/ /.codacy/ @@ -72,6 +73,7 @@ evals/results/verification-baseline/ /docs/agent/ /plan.md /status.md +/WIP.md /AUDIT*.md /CODEBASE_AUDIT*.md /PLAN*.md @@ -117,3 +119,7 @@ id_ed25519.* /.netrc /.npmrc /.pypirc +/.repowise/ +/.mcp.json +/work-in-progress/ +/wip/ diff --git a/AGENTS.md b/AGENTS.md index fcf1ccc..1da0d98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,3 +47,81 @@ Keep autonomous changes inside their approved worktree and plan-owned paths. Evaluator fixtures, judge code, policies, Git history, remotes, secrets, and publication surfaces remain outside model-controlled mutation. Follow `SECURITY.md` for private vulnerability reporting. + + +## Codebase Intelligence for rae (Repowise) + +Indexed by [Repowise](https://repowise.dev). Last indexed: 2026-08-03 (commit 3d4982e). Confidence: 100%. +The MCP tools below serve pre-verified docs, symbols, history, and health from that index. Every response carries `_meta` freshness fields; a `stale_warning` appears only when a file the response actually serves changed after indexing, so silence means current. + +### How to work in this repo + +- **Pre-edit phase** (locate, understand, assess) is where these tools win: `get_answer` for how/where/why, `search_codebase` to find, `get_context` for a file's map, `get_risk` before touching a hotspot. +- **Edit phase**: reading a file before you edit it is correct and expected. Use these tools to decide *which* files to read and edit, not to replace that read. +- **Noisy commands** (tests, builds, `git log`/`diff`, searches, listings): prefer `repowise distill `, the same command with its exit code preserved and errors-first compact output. A `[repowise#: N lines omitted]` marker is fully recoverable via `repowise expand ` (add `-q ` to filter); never re-run the command to see omitted output. + +### Trust protocol + +- `verified: true` means the served bytes were checked against the live tree. Never follow it with a re-read of the same lines. +- `get_answer` at `confidence: "high"` or `grounding: "extracted"` is content-grounded: cite it directly. `symbol_bodies`, `quotes`, and `code_rationale` entries are live source, so use them instead of opening the file. +- The **only** re-read triggers: `bounds: "approximate"`, `_meta.stale_warning`, `search_method: "bm25"`, `confidence: "low"`. `index_behind: true` alone is informational; the served content is unaffected by the drift. +- Not valid reasons to re-read: "just to be safe", "to see full context" (use the skeleton or a range read), "the file might have changed" (`verified` already checked). +- For exhaustive literal sweeps (rename every call site) plain text search is unbeatable, so use it. Reach for `get_context(include=["callers"])` when you need the `callers_total`/`callers_truncated` honesty signal instead of a maybe-incomplete grep. + +### Tools + +| Tool | When and why | +|------|--------------| +| `get_answer(question)` | First call for any how / where / why question. `confidence: "high"` or `grounding: "extracted"` is content-grounded — cite it directly. When the question names an indexed symbol, `symbol_bodies` carries its full live body (skip the `get_symbol` follow-up). Low confidence returns `best_guesses` with one-line justifications plus `code_rationale` (rationale comments mined live from candidate source). | +| `get_context(targets=[...])` | Triage card for files/modules/symbols: summary, signatures, `symbol_id`s, `hotspot` bit. File targets auto-serve a `verified` skeleton (every signature at a fraction of a full Read); `mostly_full` marks files where Read costs little more. Batch targets in one call. Opt-in blocks: `include=["callers"|"callees"|"ownership"|"decisions"|"metrics"]`. | +| `get_symbol(id)` | One verified body: `"path.py::Name"` (indexed symbol), `"path.py:140-180"` (live range read), or `"repowise#"` (omission ref). Source arrives in Read's numbered format — treat it as an already-performed Read. `truncated` responses carry a `continuation` naming the exact next range; ambiguous ids return every match in `candidates`. Index misses fall back to live-grep `fallback_lines`. | +| `search_codebase(query)` | Hybrid search, auto-routed by query shape: identifier → symbol hits (pipe `symbol_id` into `get_symbol`), path → file pages, prose → wiki-semantic. Force with `mode=symbol|path|concept|hybrid`. Concept hits carry a `sources` list; a hit whose sources are `[fts]` only is a keyword match with no semantic agreement — verify it. | +| `get_why(query, targets?)` | Why the code is shaped this way: decision records with evidence and supersession lineage, falling back to git archaeology and `code_rationale` comments. Call before refactors or pattern divergences. | +| `get_risk(targets, changed_files?)` | What history says about touching these files: churn, owners, co-change partners, blast radius. PR mode (`changed_files`) leads with a `directive` block — read `will_break` / `missing_cochanges` / `missing_tests` / `tests_to_run` first. `tests_to_run` is coverage-backed (the tests the per-test map proves exercise the changed files); empty means unknown, never no tests. To score a whole commit or diff range instead, use `get_change_risk`. | +| `get_change_risk(revspec, extensions?, exclude_patterns?)` | Pre-merge defect score for a whole commit or `base..head` range, computed from its diff shape on the live checkout (no index, no LLM). Lead with `risk_percentile` (this change ranked against sampled recent commits), summarized by `review_priority` and `classification`; `score` / `probability` / `level` are the corpus-calibrated fallback. Distinct from `get_risk`, which scores indexed files by path. A `warning` field flags an empty diff (bad revspec or over-tight extension / exclusion filters). | +| `get_health(targets?, include?)` | Health scores + findings on three dimensions (defect / maintainability / performance). Self-check the files you touched before finishing; `include=["biomarkers"|"refactoring"|"signals"]` for depth. | +| `get_dead_code()` | Confidence-tiered unreachable files / unused exports / zombie packages. For cleanup sweeps, not targeted fixes. | +| `get_overview()` | Architecture map + tool recipes. Call once, first, in an unfamiliar repo; skip it after that. | + +**Compose them:** low-confidence `get_answer` then read `best_guesses[0].file`; `get_context` shows `hotspot: true` then `get_risk` before editing; `decision_records` titles then `get_why(targets=[...])`; PR review then `get_risk(targets, changed_files)` and read `directive` first. A `tombstone` error means the file moved, so follow `successor_paths`. + +### Architecture +I’m using the RepoWise codebase-exploration workflow to ground the overview in the indexed architecture, then I’ll verify the maintained entry-point documentation and source contracts. RepoWise’s indexed overview call was unavailable in this session, so I’m falling back to the repository’s maintained documentation and narrowly scoped source reads, as the exploration workflow prescribes. RAE consumes repository-change tasks, committed Git state, validated workflow and policy definitions, and optional local graph context; processes them through isolated-worktree orchestration, typed agent and control nodes, schema gates, human checkpoints, verification, and bounded repair; and produces repository changes, immutable run artifacts, traces, benchmark evidence, and release-readiness reports for human review. RAE, or Reliable Agentic Engineering, is a source-distributed toolkit for controlled repository maintenance, autonomous code changes, audit and repair loops, and evaluation. + +### Key modules +- `packages/orchestration/scripts/pipeline/lib` — The pipeline execution-services layer consumes validated workflow contracts, configured agent invocations, autonomous runtime actions, and… +- `evals/scripts/lib` — The outcome-evaluation layer consumes trusted judge cases, candidate-runner argument vectors, trace resources, RAE execution evidence… +- `packages/orchestration/operator` — The Pipeline Operator Console is the local operator-facing application layer: it consumes authenticated requests, project and run data… +- `packages/orchestration/skills/dev-tools/multi-model-review/src` — The multi-model review orchestration layer consumes validated tool input and normalized findings from review-model adapters, then produces… +- `packages/orchestration/scripts` — The validation and evaluation support layer consumes runner manifests, command-line arguments, evaluation task sets, autonomous policy… +- `packages/orchestration/scripts/pipeline` — The autonomous pipeline control layer consumes pipeline invocations, workflow contracts and proposals, repository and artifact evidence… +- `packages/orchestration/skills/dev-tools/quality-gate/src` — The quality-gate evaluation layer is a deterministic decision boundary: it consumes tool input, artifacts, workspace-contained schemas, and… +- `packages/orchestration/skills/dev-tools/trace-collector/src` — The trace collection and profiling boundary consumes validated trace requests, pipeline event data, profile mutations, and repository… +- `evals/scripts` — The evaluation-harness operations layer consumes umbrella task specifications, sealed evidence, benchmark inputs, judge-calibration data… +- `packages/orchestration/skills/dev-tools/_shared/src` — The Safe Developer Tool Runtime is the shared execution boundary for development tools: it consumes schema-governed input and… + +### Entry points +- `packages/orchestration/operator/server.mjs` +- `packages/orchestration/operator/static/app.js` + +### Files that need care (bug-fix history first, then churn — check `get_risk` before editing) +- `evals/tests/test_benchmark_contracts_core_d.py` — 4 bug fixes, last fix 5 days ago (bug magnet); 7 commits/90d +- `evals/scripts/common.py` — 4 bug fixes, last fix 5 days ago (bug magnet); 7 commits/90d +- `packages/orchestration/skills/dev-tools/_shared/src/path-safety-helpers.ts` — 4 bug fixes, last fix 5 days ago (bug magnet); 4 commits/90d +- `packages/orchestration/skills/dev-tools/trace-collector/tests/unit/trace.test.ts` — 4 bug fixes, last fix 5 days ago (bug magnet); 5 commits/90d +- `evals/scripts/lib/release_gate_core.py` — 4 bug fixes, last fix 5 days ago (bug magnet); 5 commits/90d + +### Code health +Three co-equal signals: defect risk 8.74/10 avg, hotspot health 6.55/10 (stable), worst `packages/orchestration/scripts/pipeline/lib/runtime-state-guard.mjs` at 3.74/10 · maintainability 9.41/10 · performance risk 195 open static I/O-in-loop / N+1 findings. Detail: `get_health()`. + +Critical files: +- `packages/orchestration/scripts/verify.sh` — change entropy — impact −2.6 +- `evals/tests/test_benchmark_contracts_core_b.py` — prior defect — impact −2.0 +- `evals/tests/test_benchmark_contracts_core_c.py` — prior defect — impact −2.0 +- `packages/orchestration/scripts/lib/argv.mjs` — prior defect — impact −2.0 +- `packages/orchestration/scripts/pipeline/lib/commands.mjs` — prior defect — impact −2.0 + +### Commands +- Lint: `ruff check .` + + diff --git a/DESIGN.md b/DESIGN.md index 316e6b6..903661e 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -13,3 +13,7 @@ The workflow editor extends that system as a visual systems map: Canvas dragging is optional and never the only authoring path. Standard controls retain native keyboard behavior. Focus indicators remain visible, status is communicated with words and symbols as well as color, and motion is removed when the user requests reduced motion. Responsive layouts preserve the registry, editor, and validation order at narrow widths. This register describes the implemented design intent. It makes no formal accessibility-conformance claim. + +The experimental hosted platform has no implemented public console or hosted +product interface. Its current surface is a JSON HTTP and MCP compatibility +boundary; it does not extend the operator console's visual system. diff --git a/PRODUCT.md b/PRODUCT.md index 5ac9ef6..7f7f013 100644 --- a/PRODUCT.md +++ b/PRODUCT.md @@ -1,6 +1,10 @@ # Product register -RAE is a source-distributed toolkit for bounded repository change, repair, and evaluation. Its supported autonomous provider is Codex. It operates on a committed Git repository in an isolated worktree by default and does not commit, push, publish, deploy, or promote workflow revisions. +RAE is a source-distributed toolkit for bounded repository change, repair, and +evaluation. Codex is the default autonomous executor. OpenCode is available +only through an explicit route and the documented macOS containment backend. +RAE operates on a committed Git repository in an isolated worktree by default +and does not commit, push, publish, deploy, or promote workflow revisions. The graph-native runtime treats a versioned workflow as executable policy. Requirements, design criticism, planning, alignment, mutation, verification, and repair are arbitrary typed nodes rather than a fixed phase list. Each run owns an immutable workflow snapshot, node guidance, payload contracts, attempt envelopes, trace order, and evidence references. Existing v1 run requests remain linear when resumed. @@ -8,11 +12,23 @@ The operator is a loopback-only, bearer-authenticated local console. It is used The improvement campaign is evaluator-owned. It compares bounded workflow policy and topology candidates against frozen development and held-out matrices, records append-only lineage, and emits a recommendation. It never activates a candidate. +The experimental hosted-platform source is a separate control-plane and worker +slice. It stores control-plane state in PostgreSQL, protects routes with OIDC, +uses fenced worker leases, and can verify S3-compatible artifacts. The local +operator can proxy allowlisted remote routes, but the platform is not wired to +the umbrella CLI and it has no production +deployment or external integration evidence. Workflow 2.2 is likewise an +experimental local wait-and-signal contract. Its bounded context assembly does +not support a context-efficiency claim until the required 25 percent comparison +is recorded. + Primary users are maintainers who need inspectable repository automation, explicit ownership, conservative mutation, reproducible evidence, and human control over publication and policy changes. Product boundaries: - Runtime code, judges, fixtures, safety invariants, provider selection, model selection, tools, commands, and promotion rules are not candidate-editable. - Context graph memory is separately opt-in with `--graph-memory` and cannot authorize mutation. -- The supported execution sandbox is the repository's documented macOS evaluator backend. Unsupported hosts fail closed where that boundary applies. +- OpenCode writes require the macOS Seatbelt backend and an isolated worktree. + Codex retains its documented workspace sandbox requirements. Unsupported + hosts fail closed where the selected boundary applies. - Local evidence is not publication or release evidence. diff --git a/README.md b/README.md index 3f4dc3f..690d768 100644 --- a/README.md +++ b/README.md @@ -12,17 +12,15 @@ [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/sebastianspicker/rae/badge)](https://scorecard.dev/viewer/?uri=github.com/sebastianspicker/rae) [![License: MIT](https://img.shields.io/github/license/sebastianspicker/rae)](LICENSE) -RAE is a source-distributed toolkit for staged repository changes, bounded -audit and repair loops, repository maintenance, and evaluation. It combines a -thin command dispatcher with package-owned runtimes, JSON contracts, -verification scripts, benchmark fixtures, and evidence-oriented documentation. +RAE is a source-distributed toolkit for controlled repository changes. It +combines graph workflows, isolated Git worktrees, schema-validated artifacts, +human checkpoints, verification gates, and local run evidence. The repository is an alpha candidate. It does not publish a package, container, -hosted service, or stable API. See [Release Status](RELEASE_STATUS.md) for the -current release evidence. - -Explore the [static Evidence Dossier demo](https://sebastianspicker.github.io/rae/). -It uses sanitized fixture data, runs no command, and stores no state. +hosted service, or stable API. An experimental hosted-platform source slice is +present and the local operator can proxy allowlisted run routes in remote mode; +it is not wired into the umbrella CLI. See [Release +Status](RELEASE_STATUS.md) for the current release evidence. ## Capabilities and limitations @@ -30,17 +28,23 @@ RAE currently provides: - a graph-native orchestration runtime with typed nodes, joins, bounded repair loops, and immutable evidence envelopes - isolated Git worktrees for autonomous repository changes -- a loopback-only operator console for run status, checkpoints, resume, and stop +- a loopback-only operator console with synchronized Loop, Graph, Analyze, and JSON workflow views +- explicit Codex and OpenCode execution routes through operator-owned profiles - Ralph audit, linting, and story-scoped fixing modes - benchmark validation, execution, comparison, calibration, and release gates - opt-in local repository, workflow, evidence, and temporal-memory graph projections +- an experimental hosted control-plane and self-hosted worker source slice - a transactional Git co-author trailer cleaner - sanitized environment-profile templates and installers The following limits are part of the current implementation: -- autonomous execution supports the installed Codex CLI; other committed - adapters are guidance, not executable provider integrations +- Codex remains the default autonomous executor; OpenCode must be selected + explicitly and is never selected by `auto` +- OpenCode mutation is supported only on macOS, in an isolated RAE worktree, + under the system `sandbox-exec` boundary; OpenCode rejects `--in-place` +- OpenRouter models are supported only through OpenCode configuration; RAE does + not call the OpenRouter API directly - the custom command provider is an unsandboxed test surface and always fails `agent doctor` - RAE does not commit, push, publish, or deploy target-repository changes @@ -60,7 +64,9 @@ The following limits are part of the current implementation: - `git`, `jq`, `rg`, and `shellcheck` - Python dependencies from `requirements-ci.txt` or `requirements-macos.txt` -- Codex CLI only for provider-backed autonomous and Ralph runs +- Codex CLI for Codex-backed autonomous runs and Ralph +- OpenCode CLI only for explicitly selected OpenCode autonomous routes on + macOS - `git-filter-repo` only for the co-author trailer cleaner `./scripts/rae.sh doctor` checks the required runtime versions and executable @@ -145,9 +151,21 @@ future runs only. Workflow schema 2.1 supports bounded data-driven fan-out, item streams, deterministic transforms, threshold joins, until-dry discovery, and logical -execution tiers. Use `--execution-profile ` to supply the operator-owned -Codex model mapping. Use `graph workflow propose` for a read-only, draft-only -Codex topology proposal; it never activates or executes the candidate. +execution tiers. Execution profile 3.0 maps those tiers and optional node +overrides to explicit Codex or OpenCode routes. Use `graph workflow analyze` +for static topology and bound diagnostics. Use `graph workflow propose +--preview` for a validated, unsaved candidate. Proposals, saved revisions, and +exact-digest activation remain separate operations. + +The operator console presents the same workflow as four synchronized views. +Its guided editor targets workflow 2.1. Existing workflow 2.0 and experimental +2.2 revisions remain available through the JSON view. See [Graph Engineering +with RAE](docs/tutorials/graph-engineering-with-rae.md) and [Execution +Profile 3.0](docs/reference/contracts/execution-profile-v3.md). + +Workflow 2.2 is an experimental local wait-and-signal contract. It adds typed +wait signals and bounded context manifests without changing existing 2.0 or 2.1 +runs. See [Workflow 2.2 Contract](docs/reference/contracts/workflow-v2.2.md). Use `--through plan` to stop before repository mutation. The default worktree is stored under the target repository's Git metadata at @@ -158,7 +176,8 @@ Serve the local operator console for explicitly allowlisted repositories: ```bash ./scripts/rae.sh operator serve \ - --project /canonical/path/to/target-repository + --project /canonical/path/to/target-repository \ + --execution-profile /absolute/path/to/execution-profile.json ``` Run Ralph health checks or an audit: @@ -279,14 +298,16 @@ gate is: ``` A release consists of a reviewed source tag and optional source archive. The -repository contains no application deployment configuration. +repository contains no production application deployment configuration. The +experimental local development compose file is not a deployment artifact. ## Troubleshooting - `rae.sh doctor` reports the installed version and path for each required command. Install the missing command or select Python with `PYTHON_BIN`. -- `agent doctor` fails when the installed Codex CLI is unauthenticated or lacks - sandbox, JSON-schema output, event streaming, or ephemeral-session support. +- `agent doctor` without provider options checks Codex. For OpenCode, pass + `--provider opencode --model `; the diagnostic also verifies + the macOS sandbox and effective denied-by-default tool configuration. - A failed autonomous phase names its gate and run report. Correct the reported dependency or target issue, then use `agent resume` with the printed run ID and worktree path. diff --git a/RELEASE_STATUS.md b/RELEASE_STATUS.md index 54ddf33..9dd5843 100644 --- a/RELEASE_STATUS.md +++ b/RELEASE_STATUS.md @@ -1,112 +1,96 @@ # Release Status -Evidence cutoff: 2026-07-24 +Evidence cutoff: 2026-08-04 -Verdict: DOCUMENTATION AND AVAILABLE FOCUSED GATES PASS; NOT READY TO PUBLISH +Verdict: LOCAL IMPLEMENTATION GATES PASS; NOT READY TO PUBLISH -## Candidate identity +## Candidate scope - Proposed version: `v0.1.0-alpha.1` -- Branch: `docs-security-badges` -- Baseline HEAD: `b3a5b635032b996f943749583e93d714dc8e0ae3` -- Components: Ralph `0.3.0`; coauthor trailer cleaner `3.0.0` -- Candidate state: 309 modified, 27 deleted, and 154 untracked paths; - zero staged files; untagged, uncommitted, and unpublished -- Branch state: the configured upstream is gone; the cached `origin/main` is - 14 commits ahead of this baseline +- Distribution: reviewed source tag and optional source archive +- Published package, container, hosted service, or stable API: none +- Current working tree: uncommitted and unsuitable as a release artifact + +The candidate scope is the local source toolkit: graph-native repository +workflows, isolated worktrees, the loopback operator, evaluation tools, Ralph, +and repository-maintenance utilities. The hosted-platform package and workflow +2.2 remain experimental. + +## Implemented local surface + +- Workflow 2.1 supports typed nodes and edges, bounded fan-out, deterministic + transforms, first-success and quorum joins, checkpoints, and bounded cycles. +- The loopback operator provides synchronized Loop, Graph, Analyze, and JSON + views. Five guided templates compile directly to workflow 2.1. +- Workflow analysis reports schema and topology diagnostics, unreachable nodes, + writer and verification paths, bounded attempts and dynamic instances, + concurrency, and resolved execution routes. +- Execution profile 3.0 maps logical tiers and optional node overrides to named + Codex or OpenCode routes without adding provider configuration to workflows. +- Workflow proposals remain drafts. Preview, revision saving, validation, diff, + and exact-digest activation are separate operator actions. Activation affects + future runs only. +- OpenCode is explicit, never selected by `auto`, and supported only through the + documented macOS containment backend. OpenCode writes require an isolated RAE + worktree and reject `--in-place`. ## Verified local evidence -- `python3 -B scripts/verify_repo.py --skip-mkdocs` passes source headers, - obsolete-artifact checks, deterministic screenshots, brand assets, - frontmatter, local link paths, citation density, and evaluation metadata. -- `pyright --project pyrightconfig.json` reports zero errors, warnings, or - informational diagnostics. -- Existing candidate Bash entrypoints pass syntax checking and ShellCheck. - Candidate Python files compile with bytecode redirected outside the - repository. -- Ralph passes 63 of 63 tests in the current working tree. Its transaction tests - cover protected metadata placement, real Codex sandbox denial, native - no-clobber promotion and recovery, concurrent-entry preservation, read-only - directory subtrees, crash checkpoints, retained conflict evidence, - idempotent recovery, and partial terminal cleanup. -- The coauthor trailer cleaner passes 65 of 65 tests. The public profile - transaction suite, root runtime contract, and evaluation metadata validator - pass. -- The authenticated loopback operator console passes 25 of 25 control, - security, server, recovery, and UI-contract tests. -- Focused adversarial regressions reproduce the orchestration recovery race and - Ralph transaction overwrite and cleanup cases, and those regressions pass. - The remaining boundaries are documented in `SECURITY.md` and the package - security files. -- Orchestration's dependency-free skill validation, stale-reference check, - hygiene check, 11-file strict link check, adapter synchronization, package - integrity, operator suite, and JavaScript syntax checks pass. -- All 107 candidate JSON files and 18 YAML/CFF files parse. Every referenced - `src-*` bibliography key has a matching explicit bibliography anchor. All - versioned root-lock `node_modules` entries include `resolved` and `integrity`. -- The two deterministic SVG command captures are current, reproducible, and - free of private paths. The 1280 by 640 social preview is visually legible and - contains no slogan, credential, or private machine content. -- `./scripts/rae.sh doctor` passes with GNU Bash `5.3.15`, Python `3.14.6`, - Node.js `22.23.1`, Git, `rg`, npm, `jq`, ShellCheck, and git-filter-repo. - `./scripts/rae.sh agent doctor` confirms the installed Codex path is - authenticated and exposes the required sandbox, structured-output, event, - and ephemeral-session capabilities. - -## Verification limits and publication blockers - -- `./scripts/verify.sh --skip-install` stops because `ruff` is not installed. - The current environment also lacks `pytest`, `mkdocs`, and `lizard`, so the - complete Python suite, strict MkDocs build, complexity gate, and umbrella - verifier are not available. No dependency installation was authorized. -- `packages/orchestration/scripts/verify.sh --skip-install` passes its first six - dependency-free gates, then the `_shared` TypeScript build stops because the - local installation lacks `ajv`, `ajv-formats`, and Node type declarations. - `npm run test:runner` cannot find the local Vitest executable. A clean - `npm ci`, package build, and full package test lane remain required. -- The in-app browser has no available browser, and no Playwright installation - is present. The operator console therefore has contract-test coverage but no - final live render, console inspection, viewport review, interaction smoke, - or sanitized operator screenshot. -- No clean isolated installation, disposable real-provider outcome run, sealed - held-out evaluation, optimizer recommendation, or live release-candidate - browser smoke was performed. -- The project still lacks a private conduct-reporting address. GitHub's content - reporting controls cover conduct on GitHub, but a project-specific private - route is required before publication. -- The working-tree secret scan found only a synthetic operator test match. No - Gitleaks configuration or history scan is present, so this is working-tree - evidence rather than a complete secret-history audit. -- `python3 -B scripts/verify_repo.py --release-candidate` correctly rejects the - dirty worktree. GitHub CI, CodeQL, Scorecard, badges, external links, and the - public release page cannot be confirmed until an approved candidate commit - exists. -- The recorded test results describe this mutable working tree, not an immutable - release artifact. The final candidate commit and hosted checks must anchor the - publication evidence. -- The candidate must be reconciled onto refreshed `main`; the current branch - upstream is gone and cached `origin/main` is ahead. - -## Accepted alpha boundaries - -This source candidate implements local, experimental autonomous workflows. It -does not claim a stable API, remote operation, unsandboxed safety, universal -agent reliability, or provider-backed performance. The custom command provider -is an explicitly unsafe test surface. Ralph multi-path promotion is recoverable -but not globally atomic, and its concurrent-entry guarantee assumes stable -parent directories. These limits are acceptable only when they remain visible -in the alpha documentation and release notes. - -No file was staged, committed, tagged, pushed, released, or published during -this preparation pass. +- `packages/orchestration/scripts/verify.sh --skip-install` passes the package + builds, lint and format checks, runner, operator, shared-runtime, + quality-gate, review, and trace-collector suites. +- The pipeline runner passes 396 tests. The operator passes 42 tests. +- `python -m pytest evals/tests tests` passes 74 tests under Python 3.14.6. +- Ruff, Pyright, Lizard, the root runtime contract, evaluation validation, + profile installation, Ralph's 63 tests, and the co-author cleaner's 65 tests + pass in the current working tree. +- OpenCode doctor passes locally with OpenCode 1.18.11 and verifies the exact + denied-by-default tool surface under macOS Seatbelt. +- Real Seatbelt checks deny read-node writes and deny write-node access outside + the isolated workspace, including `.pipeline`. The verification broker runs + its approved Git check under a nested no-network sandbox. +- `git diff --check` passes. + +These results apply to the current mutable checkout. They are not evidence for +an immutable tag, hosted deployment, arbitrary repository, or provider-backed +task outcome. + +## Publication blockers + +- The root `./scripts/verify.sh --skip-install` gate stops at repository + validation because `docs/assets/screenshots/rae-agent-safety.svg` is stale + relative to its deterministic generator. The complete root gate therefore + does not pass. +- No authenticated OpenCode proposal or write run has captured a real provider + event stream and completed the full designer-to-activation acceptance path. +- No final browser render, responsive interaction, console, or screenshot smoke + was performed. The in-app browser was unavailable and Playwright is not + installed in the current environment. +- The working tree contains extensive uncommitted changes. Release-candidate + verification requires a reviewed, committed candidate with current hosted CI + and security checks. +- The project still needs a private conduct-reporting address before + publication. + +## Experimental boundaries + +The hosted control-plane and worker package is not deployed. Source-unit tests +do not establish PostgreSQL migration and reconciliation, OIDC issuer +interoperability, object-storage transfer, remote worker isolation, secret +handling, hosted recovery, or production operations. + +Workflow 2.2 implements local durable waits, typed signals, and bounded context +assembly. It has no context-efficiency result. A frozen comparison with the +predefined threshold remains required before making such a claim. + +OpenRouter models are supported only through OpenCode provider configuration. +RAE does not call the OpenRouter API directly. The OpenCode adapter is macOS +only in this candidate. ## Next gate -Before publication, provide a private conduct-reporting route, restore the -pinned Python and Node dependencies in an authorized isolated environment, and -run the complete commands in `RELEASING.md`. Then perform the browser and -provider smoke lanes, reconcile the reviewed candidate onto refreshed `main`, -create the approved candidate commit, run -`./scripts/verify.sh --release-candidate`, and confirm the hosted workflows. -Only then should the maintainer create `v0.1.0-alpha.1` and its release. +Regenerate and review the stale deterministic screenshot, complete the root +verification gate, and run the authenticated OpenCode and browser acceptance +lanes. Then review and commit the candidate, run +`./scripts/verify.sh --release-candidate`, and confirm the hosted checks against +that exact commit before creating a tag or release. diff --git a/SECURITY.md b/SECURITY.md index c986826..2ac291c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -56,8 +56,8 @@ Do not publish: ### Autonomous orchestration -- The supported Codex path requires workspace sandboxing, structured output, - JSON event streaming, and a fresh session for each phase. +- The Codex path requires workspace sandboxing, structured output, JSON event + streaming, and a fresh session for each phase. - Provider requests contain the task, phase guidance, and selected predecessor artifacts. Absolute POSIX, Windows, UNC, and `file:` URL path tokens are sanitized before request construction; the local Codex process still receives @@ -84,6 +84,37 @@ Do not publish: resume. A same-user command can escape RAE's filesystem controls or leave detached descendants. +#### OpenCode execution + +- OpenCode must be selected explicitly. `auto` never selects it. The current + adapter is supported only on macOS and requires a root-owned, non-symlink + `/usr/bin/sandbox-exec`. +- OpenCode write routes require RAE's isolated worktree and reject + `--in-place`. The Seatbelt profile denies writes to Git metadata, + `.pipeline`, the source checkout, home directories, workflow registries, + evaluator-owned paths, and other external locations. Read routes receive no + workspace write permission. +- Each attempt uses `--pure`, JSON event output, a fresh session directory, and + an inline denied-by-default permission configuration. RAE checks the merged + configuration before execution and rejects unknown tools, additional MCP + servers, project extensions, shell, web, external-directory, subagent, + skill, question, and plugin access. +- Verification runs through a RAE-owned broker that accepts opaque allowlisted + IDs. It executes fixed argument vectors without a shell under a nested + no-network sandbox and records bounded, redacted evidence independently of + model output. +- The pinned OpenCode process can read the configured OpenCode credential store + needed for provider authentication. RAE records only provider and source + metadata, not credential values or store contents. The operator profile API + never returns credential paths, environment values, or raw provider traces. +- Provider inference still requires outbound network access from the OpenCode + process. Denying web and MCP tools prevents model-directed network tools; it + does not make the provider process itself offline. RAE makes no provider-side + storage or retention claim. +- Resume requires the recorded workflow and execution-profile digests, resolved + routes, models, variants, OpenCode version, and executable digest. Drift fails + before provider execution. + ### Ralph fixing mode - Audit and linting modes are read-only. @@ -120,6 +151,26 @@ Do not publish: from jsDelivr. Building the documentation is local, but viewing pages that use that script makes a request to that third-party CDN. +### Experimental hosted platform + +- `packages/orchestration/platform/` is not a deployed service. The loopback + operator can proxy only allowlisted remote routes and keeps the upstream + token out of browser code. Insecure authentication and cleartext HTTP require + explicit development flags. +- Hosted configuration requires OIDC validation of exact issuer, audience, + JWKS URL, token type, bounded issue time, allowed asymmetric signing + algorithms, subject, and unexpired expiration. Route scopes and + project claims are enforced per request; worker identifiers must match token + subjects. +- Worker reports, heartbeats, and artifact finalization require an active + lease with the matching worker and fence value. Artifact verification hashes + the object and quarantines a mismatched upload. +- The checked-in compose file contains development credentials and cleartext + loopback ports. It is not production deployment configuration. HTTPS, + identity-provider interoperability, database and object-store operations, + worker isolation, secret handling, and incident recovery remain deployment + responsibilities without current integration evidence. + ### Repository and evaluation data - Public agent profiles must remain machine-agnostic. diff --git a/TESTING.md b/TESTING.md index 6716de1..28f94ba 100644 --- a/TESTING.md +++ b/TESTING.md @@ -29,7 +29,7 @@ The root gate runs: ## Test inventory -The current tree contains 129 executable test source files and 11 referenced +The current tree contains 146 executable test source files and 12 referenced runner, helper, fixture, or configuration files. | Classification | Paths | Count | Runner or owner | @@ -41,10 +41,11 @@ runner, helper, fixture, or configuration files. | Experimental | `evals/fixtures/autonomous-outcomes/*/tests/test_*.py` | 3 | Outcome evaluator fixture manifests | | Active | `packages/loops/ralph/tests/ralph_*_test.sh` | 63 | `packages/loops/ralph/scripts/run_tests.sh` | | Active support | Ralph test runner and `tests/lib/test_helpers.sh` | 2 | Ralph shell suite | -| Active | `packages/orchestration/operator/tests/*.test.mjs` | 5 | Node test runner | -| Active | `packages/orchestration/scripts/pipeline/tests/*.test.mjs` | 26 | Vitest | -| Active support | Pipeline Vitest config, test helper, and two fixture modules | 4 | Pipeline Vitest suite | -| Active | `packages/orchestration/skills/dev-tools/*/tests/unit/*.test.ts` | 17 | Package-local Vitest commands | +| Active | `packages/orchestration/operator/tests/*.test.mjs` | 7 | Node test runner | +| Active | `packages/orchestration/scripts/pipeline/tests/*.test.mjs` | 38 | Vitest | +| Experimental source unit | `packages/orchestration/platform/test/platform.test.mjs` | 1 | `npm --prefix packages/orchestration/platform test` | +| Active support | Pipeline Vitest config, test helper, and three fixture modules | 5 | Pipeline Vitest suite | +| Active | `packages/orchestration/skills/dev-tools/*/tests/unit/*.test.ts` | 19 | Package-local Vitest commands | | Active support | `trace-test-helpers.ts` | 1 | Trace collector tests | | Active | `profiles/agent-environments/tests/profile-installation.sh` | 1 | Root verifier | | Active | `tools/repo-hygiene/coauthor-trailer-cleaner/tests/test-*.sh` | 2 | Tool test runner | @@ -81,11 +82,32 @@ npm --prefix packages/orchestration run test:operator npm --prefix packages/orchestration run test:runner npm --prefix packages/orchestration run verify npm --prefix packages/orchestration run benchmark:workflow-topology +npm --prefix packages/orchestration/platform test ``` +Workflow Designer, execution-profile, and OpenCode boundaries: + +```bash +npm --prefix packages/orchestration run test:runner -- --run \ + tests/workflow-designer.test.mjs \ + tests/execution-profile-v3.test.mjs \ + tests/opencode-adapter.test.mjs \ + tests/verification-broker.test.mjs +``` + +These tests use controlled executables for event parsing, malformed output, +timeouts, route selection, and resume drift. The macOS integration cases also +exercise the real Seatbelt and broker boundaries. They do not replace an +authenticated provider run against a specific OpenCode version and account. + The topology benchmark is a deterministic scheduler fixture for event order, critical path, and barrier idle time. It does not measure model quality. +The platform source-unit suite uses the in-memory store. Docker, PostgreSQL, +OIDC, S3-compatible storage, and remote-worker integration remain separate +unrun evidence lanes. The workflow 2.2 Vitest coverage is local scheduler +evidence and does not establish hosted workflow execution. + Ralph: ```bash diff --git a/docs/INDEX.md b/docs/INDEX.md index 549b02f..8d68c03 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -1,7 +1,7 @@ --- status: stable owner: core -last_reviewed: 2026-07-24 +last_reviewed: 2026-08-04 source_of_truth: README.md evidence_links: reference/claims/evidence-index.md --- @@ -21,6 +21,7 @@ common commands, repository structure, and current limitations. ## Tutorials +- [Graph engineering with RAE](tutorials/graph-engineering-with-rae.md) - [First pipeline](tutorials/first-pipeline.md) - [Autonomous code change](tutorials/autonomous-code-change.md) - [First Ralph run](tutorials/first-ralph-run.md) @@ -42,6 +43,9 @@ common commands, repository structure, and current limitations. - [Repository hygiene CLI](reference/cli/repo-hygiene.md) - [Module boundaries](reference/architecture/module-boundaries.md) - [Artifact schemas](reference/contracts/artifact-schemas.md) +- [Execution profile 3.0](reference/contracts/execution-profile-v3.md) +- [Local graph and memory](reference/contracts/graph-memory.md) +- [Workflow 2.2](reference/contracts/workflow-v2.2.md) - [Quality gates](reference/contracts/quality-gates.md) - [Task specifications](reference/contracts/task-specs.md) - [Safety boundaries](reference/invariants/safety-boundaries.md) diff --git a/docs/assets/screenshots/rae-agent-safety.svg b/docs/assets/screenshots/rae-agent-safety.svg index b38569f..e57a380 100644 --- a/docs/assets/screenshots/rae-agent-safety.svg +++ b/docs/assets/screenshots/rae-agent-safety.svg @@ -1,12 +1,12 @@ - + RAE autonomous agent safety defaults Deterministic terminal capture generated from the live RAE CLI. - - + + @@ -16,47 +16,51 @@ RAE autonomous coding-agent orchestrator Usage: - node scripts/pipeline/autonomous.mjs doctor [--provider codex] + node scripts/pipeline/autonomous.mjs doctor [--provider codex|opencode] [--model <provider/model>] node scripts/pipeline/autonomous.mjs run --task <text> [options] node scripts/pipeline/autonomous.mjs resume --run-id <id> [options] node scripts/pipeline/autonomous.mjs status --project-root <workspace> --run-id <id> [--json] node scripts/pipeline/autonomous.mjs stop --project-root <workspace> --run-id <id> [--json] - node scripts/pipeline/autonomous.mjs resolve-checkpoint --project-root <workspace> --run-id <id> - --checkpoint-id <id> --decision <approved|rejected|escalated> - --decision-id <id> --actor <label> --rationale <text> [--json] - node scripts/pipeline/autonomous.mjs events --project-root <workspace> --run-id <id> - [--after-seq <n>] [--limit <1..1000>] [--json] - - Run options: - --project-root <path> Target Git repository (default: current directory) - --task <text> Work request - --task-file <path> Read a relative, non-symlink .md or .txt file under the project root - --provider <name> auto or codex (command is test-integration only) - --model <id> Optional Codex model override - --reasoning-effort <level> low, medium, high, or xhigh - --execution-profile <file> Operator-owned logical tier to Codex mapping - --timeout-seconds <n> Per-phase timeout (default: 1800) - --policy <path> Validated data-only autonomous policy JSON - --workflow <path> Explicit graph-native workflow JSON for a new run - --legacy-linear Start a temporary v1 ten-phase run - --checkpoint-policy <mode> Human pause mode: none, before-mutation, or before-mutation-and-ship - --graph-memory <mode> Local graph mode: off, read, or read-write (default: off) - --in-place Modify a clean target checkout directly - --through <node-id> Stop after one workflow node - --max-concurrency <n> Concurrent readers, from 1 to 4 (default: 4) - --max-repair-rounds <n> Repair iterations, from 1 to 5 (default: 5) - --run-id <id> Resume an existing run (resume command only) - --json Emit the final result as JSON - - Custom command-provider options: - --agent-command <path> Executable implementing the rae-agent-v1 stdin/stdout protocol - --agent-arg <value> Argument for the command; repeat as needed - --allow-unsafe-command-provider - Explicitly enable the unsandboxed test-integration provider - - Safety defaults: - - run creates an isolated Git worktree unless --in-place is explicit - - Codex phases use read-only or workspace-write sandbox modes as appropriate - - agents may not commit, push, publish, install dependencies, or use network infrastructure - - the custom command provider always fails doctor and cannot run without an unsafe opt-in + node scripts/pipeline/autonomous.mjs signal --project-root <workspace> --run-id <id> + --node-id <wait-node> --signal <name> --idempotency-key <key> [--payload-json <json>] [--json] + node scripts/pipeline/autonomous.mjs resolve-checkpoint --project-root <workspace> --run-id <id> + --checkpoint-id <id> --decision <approved|rejected|escalated> + --decision-id <id> --actor <label> --rationale <text> [--json] + node scripts/pipeline/autonomous.mjs events --project-root <workspace> --run-id <id> + [--after-seq <n>] [--limit <1..1000>] [--json] + + Run options: + --project-root <path> Target Git repository (default: current directory) + --task <text> Work request + --task-file <path> Read a relative, non-symlink .md or .txt file under the project root + --provider <name> auto, codex, or explicit opencode (command is test-integration only) + --model <id> Optional Codex model or required OpenCode provider/model + --reasoning-effort <level> Codex low, medium, high, or xhigh + --variant <name> Optional OpenCode model variant + --execution-profile <file> Operator-owned logical tiers and provider routes + --timeout-seconds <n> Per-phase timeout (default: 1800) + --policy <path> Validated data-only autonomous policy JSON + --workflow <path> Explicit graph-native workflow JSON for a new run + --legacy-linear Start a temporary v1 ten-phase run + --checkpoint-policy <mode> Human pause mode: none, before-mutation, or before-mutation-and-ship + --graph-memory <mode> Local graph mode: off, read, or read-write (default: off) + --in-place Modify a clean target checkout directly + --through <node-id> Stop after one workflow node + --max-concurrency <n> Concurrent readers, from 1 to 4 (default: 4) + --max-repair-rounds <n> Repair iterations, from 1 to 5 (default: 5) + --run-id <id> Resume an existing run (resume command only) + --json Emit the final result as JSON + + Custom command-provider options: + --agent-command <path> Executable implementing the rae-agent-v1 stdin/stdout protocol + --agent-arg <value> Argument for the command; repeat as needed + --allow-unsafe-command-provider + Explicitly enable the unsandboxed test-integration provider + + Safety defaults: + - run creates an isolated Git worktree unless --in-place is explicit + - Codex phases use read-only or workspace-write sandbox modes as appropriate + - OpenCode is explicit, macOS-contained, and never selected by auto + - agents may not commit, push, publish, install dependencies, or use network infrastructure + - the custom command provider always fails doctor and cannot run without an unsafe opt-in diff --git a/docs/governance/quality-policy.md b/docs/governance/quality-policy.md index 8fed2b7..6e20129 100644 --- a/docs/governance/quality-policy.md +++ b/docs/governance/quality-policy.md @@ -1,7 +1,7 @@ --- status: stable owner: core -last_reviewed: 2026-07-31 +last_reviewed: 2026-07-10 source_of_truth: ../reference/contracts/quality-gates.md evidence_links: ../reference/invariants/determinism-contracts.md --- @@ -35,10 +35,6 @@ exclusions. ## Exact analyzer exceptions -- Bandit `B101` remains enforced for production Python. Codacy findings under - `evals/tests/` and `tests/` are classified as `TestCode` because pytest - assertions are the executable test contract; Ruff independently enforces - `S101` outside those test paths. - Bandit `B404` is omitted because it reports imports rather than executable sinks; Bandit `B603` remains enabled for every subprocess call site. - OpenGrep's Python `dangerous-subprocess-use-audit` rule is omitted because diff --git a/docs/how-to/deploy-experimental-platform.md b/docs/how-to/deploy-experimental-platform.md new file mode 100644 index 0000000..98104cc --- /dev/null +++ b/docs/how-to/deploy-experimental-platform.md @@ -0,0 +1,69 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/platform +evidence_links: ../reference/claims/claims-ledger.md +--- + +# Deploy the Experimental Platform + +This is a local development procedure, not production deployment guidance. The +checked-in compose file deliberately uses loopback cleartext PostgreSQL and +MinIO ports plus development credentials. Do not publish it, expose it, or +reuse its credentials. + +## Prepare configuration + +Set `RAE_PLATFORM_CONFIG` to a TOML file containing at least a database URL. +For a local experiment, set `platform.development = true` before enabling +`platform.allowInsecureAuth` or `platform.allowInsecureHttp`. The checked-in +Compose file publishes those development services on loopback only. + +For any hosted experiment, omit `allowInsecureAuth` and configure `oidc` with +an exact issuer, audience, JWKS URL, and allowed asymmetric JWS algorithms. Put +the service behind HTTPS. +If artifacts are needed, configure an S3-compatible bucket, region, optional +endpoint, and optional path-style addressing. + +## Start the local development stack + +From `packages/orchestration/platform/`, review the checked-in development-only +`dev/platform.toml` configuration, set `RAE_DEV_MINIO_ACCESS_KEY` and +`RAE_DEV_MINIO_SECRET_KEY` to disposable local values, then run: + +```bash +docker compose -f compose.yaml up --build migrate control +``` + +The control process never applies migrations automatically. Verify that all +migrations completed before serving work: + +```bash +curl http://127.0.0.1:8080/readyz +``` + +The expected response is `{"status":"ready"}`. `GET /healthz` only proves +that the HTTP process answered. It does not prove schema readiness, OIDC, +object storage, or worker execution. + +## Required external evidence + +No current repository evidence proves a container deployment, PostgreSQL +integration, OIDC issuer interoperability, S3-compatible upload and +verification, worker execution, or operational recovery. A hosted trial must +record those results separately, including HTTPS termination, secret handling, +database backup and restore, object retention and quarantine handling, and +worker loss recovery. + +See [Use the Experimental Hosted API](hosted-api.md) for the route contract. + +## Source note + +- [NIST GenAI Profile](../reference/claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../reference/claims/bibliography.md#src-model-cards) +- [Datasheets](../reference/claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../reference/claims/bibliography.md#src-openai-evals) +- [PaperBench](../reference/claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../reference/claims/bibliography.md#src-ieee-1012) +- [Diataxis](../reference/claims/bibliography.md#src-diataxis) diff --git a/docs/how-to/hosted-api.md b/docs/how-to/hosted-api.md new file mode 100644 index 0000000..859a4a6 --- /dev/null +++ b/docs/how-to/hosted-api.md @@ -0,0 +1,79 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/platform/src/http.mjs +evidence_links: ../reference/claims/claims-ledger.md +--- + +# Use the Experimental Hosted API + +This API is an experimental control-plane surface. The loopback operator can +proxy its allowlisted run routes in remote mode, but it does not provide a +production service and must not be exposed with local insecure authentication. + +## Prerequisites + +Configure the control process with `RAE_PLATFORM_CONFIG` and apply migrations +before serving. Hosted callers need a bearer token accepted by the configured +OIDC issuer, audience, and signing-algorithm policy. The token must contain an +unexpired `exp`, a bounded `iat`, a subject, the required `rae.*` scope, and an authorized +`projects` or `project_ids` claim. + +The control process exposes unauthenticated `GET /healthz`, `GET /readyz`, and +`GET /metrics`. `GET /readyz` returns `503` until every checked-in migration is +recorded. `GET /.well-known/oauth-protected-resource` describes the MCP +resource when OIDC is configured. + +## Route groups + +| Route | Required scope | Notes | +| --- | --- | --- | +| `POST /api/v2/revisions`, activate, or diff | `rae.policy.write` | Uploads, validates, compares, or activates an exact immutable revision. Activation requires `Idempotency-Key`. | +| `POST /api/v2/runs` | `rae.run.submit` | Requires `Idempotency-Key`; the run envelope is limited to 256 KiB. | +| `GET /api/v2/runs/` and `/events` | `rae.run.read` | Reads an authorized run or its events; `?stream=true&from=` opens bounded SSE. | +| `POST /api/v2/runs//cancel` | `rae.run.cancel` | Requires `Idempotency-Key`. | +| `POST /api/v2/runs//signals` | `rae.run.signal` | Requires `Idempotency-Key`. | +| `POST /api/v2/runs//rebind` | `rae.run.cancel` | Requires `Idempotency-Key`, an operator decision, and matching digests. | +| Worker register, claim, and heartbeat | `rae.work.claim` | Worker subject must equal the supplied stable worker identifier. | +| Worker report, failure, and artifact upload | `rae.work.report` | Reports require the current fenced lease. | +| Artifact download | `rae.run.read` | Available only for authorized projects when storage is configured. | +| `POST /mcp` | Per-tool `rae.run.*` scope | Stateless Streamable HTTP MCP compatibility surface. | + +All listed routes are implementation references, not a stability commitment. +The request parser accepts JSON bodies up to 1,050,000 bytes. Mutating run +operations named above enforce `Idempotency-Key`; callers must provide one for +every mutation. Revision upload also computes and verifies the supplied digest. + +## Worker protocol + +Register first, then claim work. A worker uses HTTPS, long-polls for up to 25 +seconds, sends a heartbeat every 20 seconds, and reports success or failure +with the claim's node identifier and fence value. A lost heartbeat aborts the +current worker operation rather than reporting a stale result. + +The worker resolves the claim's logical project ID through its private +`RAE_PROJECT_MAP_FILE`, verifies the claim's profile digest against the local +execution-profile v2 snapshot, replaces all filesystem paths locally, and +runs the existing sandboxed workflow-agent child. The map must be an +owner-only regular file and each root must be a canonical Git top level. + +Each provider-node payload must contain `prompt`, `outputSchema`, +`profileDigest`, and a logical `tier`; it may contain a bounded +`timeoutSeconds`. The control plane supplies only the logical project, run, +attempt, and node identities plus read/write access. The worker derives the +workspace root, output paths, Codex model, reasoning effort, credentials, and +MCP allowlist from its local map and matching profile snapshot. + +For the deployment boundary, see +[Deploy the Experimental Platform](deploy-experimental-platform.md). + +## Source note + +- [NIST GenAI Profile](../reference/claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../reference/claims/bibliography.md#src-model-cards) +- [Datasheets](../reference/claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../reference/claims/bibliography.md#src-openai-evals) +- [PaperBench](../reference/claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../reference/claims/bibliography.md#src-ieee-1012) +- [Diataxis](../reference/claims/bibliography.md#src-diataxis) diff --git a/docs/how-to/recover-workflow-v2.2-wait.md b/docs/how-to/recover-workflow-v2.2-wait.md new file mode 100644 index 0000000..988cbab --- /dev/null +++ b/docs/how-to/recover-workflow-v2.2-wait.md @@ -0,0 +1,43 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/scripts/pipeline/lib/workflow-v22-reducer.mjs +evidence_links: ../reference/claims/claims-ledger.md +--- + +# Recover a Workflow 2.2 Wait + +Workflow 2.2 recovery is local and applies only to its durable wait state. It +does not recover a hosted worker, PostgreSQL lease, or object-store transfer. + +1. Keep the original workspace and run directory intact. The wait state is at + `.pipeline/runs//workflow/wait-state.json`. +2. Use the original run ID, workspace root, wait-node name, signal name, and + idempotency key when retrying a signal. Replaying the same node and key is a + no-op. +3. Resume the run with `./scripts/rae.sh agent resume --project-root + --run-id `. +4. Inspect the wait state and immutable node envelopes before another signal or + a retry. The scheduler only consumes accepted signals recorded on or before + the wait deadline. + +If the deadline passed, the resumed scheduler creates a failed wait envelope. +Recovery then follows the workflow's failure edges, if any. Do not edit the +wait-state file to force progress. The reducer checks the run and workflow +digest against the immutable snapshot and fails when the state is busy or does +not match. + +The tested recovery case is a local signal and resume flow. Container restart, +PostgreSQL failover, OIDC token refresh, S3 transfer recovery, and remote worker +recovery remain unverified integration work. + +## Source note + +- [NIST GenAI Profile](../reference/claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../reference/claims/bibliography.md#src-model-cards) +- [Datasheets](../reference/claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../reference/claims/bibliography.md#src-openai-evals) +- [PaperBench](../reference/claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../reference/claims/bibliography.md#src-ieee-1012) +- [Diataxis](../reference/claims/bibliography.md#src-diataxis) diff --git a/docs/how-to/run-workflow-v2.2-wait.md b/docs/how-to/run-workflow-v2.2-wait.md new file mode 100644 index 0000000..9409c5e --- /dev/null +++ b/docs/how-to/run-workflow-v2.2-wait.md @@ -0,0 +1,66 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/scripts/pipeline/autonomous.mjs +evidence_links: ../reference/claims/claims-ledger.md +--- + +# Run a Workflow 2.2 Wait + +Use this procedure only with a validated local workflow whose +`schema_version` is `2.2.0` and that defines a `wait` node and matching signal +contract. Workflow 2.2 is experimental and remains a local orchestration +surface. + +Start a run with the workflow path: + +```bash +./scripts/rae.sh agent run \ + --project-root /path/to/target-repository \ + --task "Perform the approved task" \ + --workflow /path/to/workflow-v2.2.json +``` + +When execution reaches a wait node, the scheduler persists state under +`.pipeline/runs//workflow/wait-state.json` and returns a waiting +result. No provider call is made for the open wait itself. + +Record an accepted signal with a stable retry key and a JSON payload matching +the workflow's signal contract: + +```bash +node packages/orchestration/scripts/pipeline/autonomous.mjs signal \ + --project-root /path/from/run-output \ + --run-id \ + --node-id \ + --signal \ + --idempotency-key \ + --payload-json '{"decision":"approve"}' \ + --json +``` + +Resume the same run using the workspace root printed by the original command: + +```bash +./scripts/rae.sh agent resume \ + --project-root /path/from/run-output \ + --run-id +``` + +The reducer consumes the earliest accepted unconsumed signal at or before the +deadline. A timeout writes a failed wait envelope, so define failure edges when +the workflow needs a controlled timeout path. + +This procedure does not activate a workflow revision, publish a change, or +connect the local run to the experimental hosted platform. + +## Source note + +- [NIST GenAI Profile](../reference/claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../reference/claims/bibliography.md#src-model-cards) +- [Datasheets](../reference/claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../reference/claims/bibliography.md#src-openai-evals) +- [PaperBench](../reference/claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../reference/claims/bibliography.md#src-ieee-1012) +- [Diataxis](../reference/claims/bibliography.md#src-diataxis) diff --git a/docs/how-to/test-experimental-hosted-platform.md b/docs/how-to/test-experimental-hosted-platform.md new file mode 100644 index 0000000..685d72a --- /dev/null +++ b/docs/how-to/test-experimental-hosted-platform.md @@ -0,0 +1,46 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/platform/test/platform.test.mjs +evidence_links: ../reference/claims/claims-ledger.md +--- + +# Test the Experimental Hosted Platform + +Run the focused source-unit suite from the platform package: + +```bash +npm --prefix packages/orchestration/platform test +``` + +The current suite uses the in-memory store. It verifies canonical revision +digests, digest mismatch rejection, idempotent run submission, the 256 KiB run +envelope limit, four-reader writer exclusion, authorization failure, and +traceparent format. + +Run the workflow 2.2 contract suite separately: + +```bash +npm --prefix packages/orchestration run test:runner -- workflow-v22.test.mjs +``` + +That suite covers bounded and ordered context, artifact references instead of +partial predecessor objects, fail-closed context overflow, idempotent signals, +local resume, and timeout routing. + +The focused tests do not start Docker, PostgreSQL, MinIO or another S3 service, +an OIDC issuer, or a remote worker. They do not prove hosted deployment, +database migration behavior, token interoperability, presigned URL transfer, +or end-to-end worker recovery. Record those as integration evidence before +describing the platform as deployable. + +## Source note + +- [NIST GenAI Profile](../reference/claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../reference/claims/bibliography.md#src-model-cards) +- [Datasheets](../reference/claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../reference/claims/bibliography.md#src-openai-evals) +- [PaperBench](../reference/claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../reference/claims/bibliography.md#src-ieee-1012) +- [Diataxis](../reference/claims/bibliography.md#src-diataxis) diff --git a/docs/reference/architecture/experimental-hosted-platform.md b/docs/reference/architecture/experimental-hosted-platform.md new file mode 100644 index 0000000..1dcccb9 --- /dev/null +++ b/docs/reference/architecture/experimental-hosted-platform.md @@ -0,0 +1,81 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/platform +evidence_links: ../claims/claims-ledger.md +--- + +# Experimental Hosted Platform + +The hosted platform is an experimental control-plane and worker boundary under +`packages/orchestration/platform/`. The loopback operator can proxy allowlisted +run routes in remote mode, but it is not a production deployment. + +## Components + +The control process loads TOML configuration, requires explicit database +migrations, checks readiness, and serves a raw Node HTTP surface. PostgreSQL +stores revisions, runs, nodes, leases, attempts, events, an outbox, workers, +and artifact metadata. The worker polls the control plane over HTTPS, renews +its lease, maps logical projects to private canonical Git roots, verifies +local profile digests, runs the sandboxed Codex child, and reports its result. + +Optional S3-compatible storage reserves immutable objects by SHA-256 key, +issues five-minute upload and download URLs, and verifies the object digest and +size before marking it verified. A checksum mismatch is copied to a quarantine +key and rejected. + +The platform also exposes a stateless Streamable HTTP MCP endpoint. It can +submit, read, signal, or cancel project-authorized runs and read run events. It +does not expose an arbitrary command, Git publication, or deployment action. + +## Execution and recovery boundaries + +Workers receive 60-second fenced leases and send heartbeats every 20 seconds. +The store accepts heartbeat, report, and artifact finalization only while the +lease is active and its worker and fence match. Expired leases are reconciled +back to queued state. Read nodes may run four-wide; a write node excludes other +active nodes in both the PostgreSQL and in-memory implementations. + +Run pinning and explicit rebind require matching repository and worktree +digests from the run and worker. This is an implementation boundary, not proof +that a worker is otherwise isolated or trusted. + +## Security boundary + +Hosted configuration requires OIDC unless explicit development, insecure-auth, +and insecure-HTTP flags are all set. OIDC validation checks an exact issuer, +audience, JWKS URL, configured asymmetric signing algorithms and token type, +subject, bounded issue time, and unexpired expiration. Routes also require a +named `rae.*` scope and a project claim. Worker identifiers must equal the +token subject. + +The development compose file is intentionally not a hosted deployment +configuration. It uses loopback cleartext ports and development credentials +for PostgreSQL and MinIO. The checked-in image runs as `node`, uses a read-only +application filesystem at runtime, and still requires deployment-specific +network, identity, database, object-store, and operational controls. + +## Evidence status + +Source-unit tests cover canonical revision digests, request idempotency and the +256 KiB run-envelope bound, reader and writer exclusion, fenced completion, +authorization failure, traceparent construction, worker URL validation, and +the two-failed-heartbeat stop rule. They do not prove PostgreSQL migrations, +container startup, OIDC issuer interoperability, S3 compatibility, worker +execution, or a complete hosted recovery path. + +See [Hosted API](../../how-to/hosted-api.md), +[experimental deployment](../../how-to/deploy-experimental-platform.md), and +[experimental platform testing](../../how-to/test-experimental-hosted-platform.md). + +## Source note + +- [NIST GenAI Profile](../claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../claims/bibliography.md#src-model-cards) +- [Datasheets](../claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../claims/bibliography.md#src-openai-evals) +- [PaperBench](../claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../claims/bibliography.md#src-ieee-1012) +- [Diataxis](../claims/bibliography.md#src-diataxis) diff --git a/docs/reference/architecture/system-overview.md b/docs/reference/architecture/system-overview.md index e55a1ca..efd8700 100644 --- a/docs/reference/architecture/system-overview.md +++ b/docs/reference/architecture/system-overview.md @@ -45,6 +45,10 @@ flowchart LR environments. The current committed surface defines policy and boundaries; sanitized payloads land here only after extraction. Manifest v2 uses no-follow filesystem operations and retained recovery evidence. +7. `packages/orchestration/platform/` + Experimental hosted control-plane and worker source. It is separate from + the loopback operator and umbrella CLI, and has no production deployment + evidence. ## Integration rule @@ -84,6 +88,9 @@ work is reliable. The important artifact families are: gate reports, and result ledgers. - `profiles/agent-environments/` Sanitized operator profile material and installation regression fixtures. +- `packages/orchestration/platform/` + Experimental PostgreSQL state, OIDC route checks, fenced worker leases, + S3-compatible artifact handling, and Streamable HTTP MCP compatibility. New code should either produce one of these artifacts, validate one of these artifacts, or stay inside the package-local runtime that owns the behavior. @@ -103,6 +110,8 @@ explicitly separated modules, shared publication doctrine, and traceable claims. - this diagram is explanatory architecture, not empirical proof of universal superiority over other designs +- the hosted-platform source is not evidence of a deployed service or external + integration ## Source note diff --git a/docs/reference/claims/bibliography.md b/docs/reference/claims/bibliography.md index 4f9c588..9fd0000 100644 --- a/docs/reference/claims/bibliography.md +++ b/docs/reference/claims/bibliography.md @@ -1,7 +1,7 @@ --- status: stable owner: science -last_reviewed: 2026-04-17 +last_reviewed: 2026-08-04 source_of_truth: editorial evidence_links: evidence-index.md --- @@ -274,6 +274,43 @@ https://arxiv.org/abs/2501.14050 Interactive Environments." 2026. https://arxiv.org/abs/2605.12493 +## Execution and containment sources + +### SRC-OPENCODE-CLI { #src-opencode-cli } + +OpenCode. "CLI." Accessed August 4, 2026. +https://dev.opencode.ai/docs/cli/ + +### SRC-OPENCODE-CONFIG { #src-opencode-config } + +OpenCode. "Config." Accessed August 4, 2026. +https://dev.opencode.ai/docs/config + +### SRC-OPENCODE-PERMISSIONS { #src-opencode-permissions } + +OpenCode. "Permissions." Accessed August 4, 2026. +https://dev.opencode.ai/docs/permissions/ + +### SRC-OPENCODE-TOOLS { #src-opencode-tools } + +OpenCode. "Tools." Accessed August 4, 2026. +https://dev.opencode.ai/docs/tools/ + +### SRC-OPENCODE-PROVIDERS { #src-opencode-providers } + +OpenCode. "Providers." Accessed August 4, 2026. +https://opencode.ai/docs/providers + +### SRC-JSON-SCHEMA-2020-12 { #src-json-schema-2020-12 } + +JSON Schema. "JSON Schema Draft 2020-12." Accessed August 4, 2026. +https://json-schema.org/draft/2020-12 + +### SRC-APPLE-APP-SANDBOX { #src-apple-app-sandbox } + +Apple. "Accessing files from the macOS App Sandbox." Accessed August 4, 2026. +https://developer.apple.com/documentation/security/accessing-files-from-the-macos-app-sandbox + ## Coverage note The bibliography is also a thesis-support surface for the documentation corpus. diff --git a/docs/reference/claims/claims-ledger.md b/docs/reference/claims/claims-ledger.md index c0e968b..8080b5c 100644 --- a/docs/reference/claims/claims-ledger.md +++ b/docs/reference/claims/claims-ledger.md @@ -47,6 +47,8 @@ evidence_links: evidence-index.md | CLM-020 | Failure analysis is more diagnostic when representation, inference, coordination, and governance failures are separated instead of collapsed into one label. | engineering_heuristic | adopted | [Evidence Index](evidence-index.md#clm-020) | [Dossier](dossiers/clm-020-layered-failure-model.md) | | CLM-021 | Negative results should be preserved as first-class evidence when they constrain interpretation, calibration, or future design. | governance_rule | adopted | [Evidence Index](evidence-index.md#clm-021) | [Dossier](dossiers/clm-021-negative-results.md) | | CLM-022 | Graph-informed repository context should remain experimental until it improves localization or reduces context under frozen held-out evaluation without reducing task passes or crossing repository and protected-path boundaries. | governance_rule | adopted | [Evidence Index](evidence-index.md#clm-022) | [Graph Contract](../contracts/graph-memory.md#experimental-status) | +| CLM-023 | The experimental hosted platform and local workflow 2.2 surfaces have implementation contracts, but require external integration and comparative evaluation before operational or context-efficiency claims. A predefined 25 percent comparison threshold remains required for any context-efficiency claim. | implementation_reference | provisional | `packages/orchestration/platform/`, `packages/orchestration/contracts/workflows/workflow-v2.2.schema.json` | None | +| CLM-024 | Workflow 2.1 remains provider-neutral while execution profile 3.0 resolves explicit Codex and OpenCode routes locally; OpenCode mutation requires an isolated worktree, an exact denied-by-default tool surface, and the macOS containment backend. | implementation_reference | provisional | `packages/orchestration/contracts/workflows/execution-profile-v3.schema.json`, `packages/orchestration/scripts/pipeline/lib/opencode-adapter.mjs` | [Execution Profile 3.0](../contracts/execution-profile-v3.md) | ## Status meanings diff --git a/docs/reference/claims/evidence-index.md b/docs/reference/claims/evidence-index.md index 1511d8e..8f4b019 100644 --- a/docs/reference/claims/evidence-index.md +++ b/docs/reference/claims/evidence-index.md @@ -61,7 +61,7 @@ evidence_links: bibliography.md - Dossier: [CLM-008 coordination topology](dossiers/clm-008-coordination-topology.md) - Internal anchor: `docs/explanation/science/coordination-cost.md` - Internal anchor: `packages/orchestration/docs/ORCHESTRATION_POLICY.md` -- Internal anchor: `docs/tutorials/graph-engineering-with-codex.md` +- Internal anchor: `docs/tutorials/graph-engineering-with-rae.md` - Internal anchor: `packages/orchestration/scripts/pipeline/lib/workflow-scheduler-v21.mjs` - External anchor: [Amdahl 1967](bibliography.md#src-amdahl-1967) - External anchor: [Conway 1968](bibliography.md#src-conway-1968) @@ -111,7 +111,7 @@ evidence_links: bibliography.md - Internal anchor: `docs/explanation/science/cognitive-tiering.md` - Internal anchor: `docs/explanation/supplementary/design-axioms.md` - Internal anchor: `packages/orchestration/contracts/workflows/execution-profile-v1.schema.json` -- Internal anchor: `docs/tutorials/graph-engineering-with-codex.md` +- Internal anchor: `docs/tutorials/graph-engineering-with-rae.md` - External anchor: [Kahneman](bibliography.md#src-kahneman-fast-slow) - External anchor: [Bainbridge automation](bibliography.md#src-bainbridge-automation) - External anchor: [Parasuraman and Riley](bibliography.md#src-parasuraman-riley) diff --git a/docs/reference/cli/umbrella.md b/docs/reference/cli/umbrella.md index 37569e1..6835222 100644 --- a/docs/reference/cli/umbrella.md +++ b/docs/reference/cli/umbrella.md @@ -1,7 +1,7 @@ --- status: stable owner: core -last_reviewed: 2026-07-24 +last_reviewed: 2026-08-04 source_of_truth: scripts/rae.sh evidence_links: ../claims/evidence-index.md --- @@ -69,8 +69,11 @@ Provider-backed autonomous work has a separate diagnostic: ./scripts/rae.sh agent doctor ``` -It requires authentication, workspace sandboxing, JSON-schema output, event -streaming, and ephemeral sessions. +Without provider options, the command checks Codex authentication, workspace +sandboxing, JSON-schema output, event streaming, and ephemeral sessions. Use +`agent doctor --provider opencode --model ` to check the exact +OpenCode binary, merged permission configuration, credential-store presence, +and macOS containment backend. ## Autonomous run @@ -111,9 +114,29 @@ tiers: --task "Implement and verify the requested change" ``` -`--execution-profile` is mutually exclusive with `--model` and -`--reasoning-effort`. The validated profile and canonical digest are stored in -the run request and remain immutable on resume. +`--execution-profile` is mutually exclusive with `--provider`, `--model`, +`--reasoning-effort`, and `--variant`. Execution profile 3.0 resolves logical +tiers and optional per-node overrides to named Codex or OpenCode routes. The +validated profile, canonical digest, resolved node routes, models, and exact +executor versions are stored in the run request and remain immutable on +resume. + +OpenCode is explicit: + +```bash +./scripts/rae.sh agent doctor \ + --provider opencode \ + --model opencode/example-model + +./scripts/rae.sh agent run \ + --project-root /path/to/target-repository \ + --provider opencode \ + --model openrouter/example-model \ + --task "Implement and verify the requested change" +``` + +OpenCode writes require the isolated macOS worktree backend and reject +`--in-place`. `auto` never selects OpenCode. ## Local graph and memory @@ -134,20 +157,32 @@ Workflow revisions use the same graph command family: ./scripts/rae.sh graph workflow list --project-root /path/to/target-repository ./scripts/rae.sh graph workflow validate --project-root /path/to/target-repository \ --workflow-file /absolute/path/to/workflow.json +./scripts/rae.sh graph workflow analyze \ + --workflow-file /absolute/path/to/workflow.json \ + --execution-profile /absolute/path/to/execution-profile.json ./scripts/rae.sh graph workflow propose --project-root /path/to/target-repository \ --task "Design a bounded topology" --base-workflow graph-native-default \ - --actor "operator-name" --rationale "Draft for review" + --actor "operator-name" --rationale "Draft for review" \ + --execution-profile /absolute/path/to/execution-profile.json --preview ``` +`analyze` reports schema and topology errors, unreachable nodes, writer and +verification paths, bounded attempts and instances, concurrency, and resolved +routes. It reports monetary cost as unavailable when provider usage data is not +present. + `propose` starts one read-only, ephemeral structured-output session and permits -one correction after local validation. It stores only a valid attributed draft. -It does not activate or execute the draft. +one correction after local validation. `--preview` returns a validated candidate +without saving it; omitting `--preview` stores a valid attributed draft. An +execution profile supplies the `judgment` route. Neither mode activates or +executes the result. ## Operator console ```bash ./scripts/rae.sh operator serve \ - --project /canonical/path/to/target-repository + --project /canonical/path/to/target-repository \ + --execution-profile /absolute/path/to/execution-profile.json ``` Repeat `--project` for additional allowlisted roots. The server binds to diff --git a/docs/reference/contracts/artifact-schemas.md b/docs/reference/contracts/artifact-schemas.md index 115bdba..da32d94 100644 --- a/docs/reference/contracts/artifact-schemas.md +++ b/docs/reference/contracts/artifact-schemas.md @@ -32,6 +32,7 @@ Current imported schema set includes: - graph-native workflow 2.0 and immutable node-result envelope 2.0 - graph-native workflow and node-instance envelope 2.1 - operator-owned execution profile with economy, standard, and judgment tiers +- graph-native workflow and immutable node-result envelope 2.2 Version 2.1 adds bounded maps, item streams, allowlisted transforms, threshold joins, typed failure collection, until-dry convergence, and immutable instance @@ -39,6 +40,11 @@ identity. Version 2.0 remains a separate accepted contract for existing run snapshots and locally activated revisions. RAE does not rewrite private registries or migrate stored runs automatically. +Version 2.2 adds local durable wait nodes, typed signal contracts, bounded +context manifests, and immutable references for predecessor records that do +not fit inline. It is experimental and does not connect the local scheduler to +the hosted platform. See [Workflow 2.2 Contract](workflow-v2.2.md). + Umbrella eval/runtime schemas additionally include: - `evals/schemas/task-spec.schema.json` diff --git a/docs/reference/contracts/execution-profile-v3.md b/docs/reference/contracts/execution-profile-v3.md new file mode 100644 index 0000000..0aef780 --- /dev/null +++ b/docs/reference/contracts/execution-profile-v3.md @@ -0,0 +1,79 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/contracts/workflows/execution-profile-v3.schema.json +evidence_links: ../claims/claims-ledger.md +--- + +# Execution profile 3.0 + +Execution profile schema `3.0.0` maps logical workflow tiers to named provider +routes. Workflow 2.1 remains provider-neutral. Provider selection is local run +configuration and is not stored in the workflow revision. + +Each route declares an `executor` and `model`. Codex routes declare +`reasoning_effort`; OpenCode routes may declare `variant`. The `tiers` object +maps `economy`, `standard`, and `judgment` to route IDs. `node_routes` may +override a route for a specific provider-backed workflow node. + +```json +{ + "schema_version": "3.0.0", + "profile_id": "local-mixed", + "routes": { + "routine": { + "executor": "codex", + "model": "gpt-5.6-terra", + "reasoning_effort": "medium" + }, + "review": { + "executor": "opencode", + "model": "openrouter/example-model", + "variant": "high" + } + }, + "tiers": { + "economy": "routine", + "standard": "routine", + "judgment": "review" + }, + "node_routes": { + "security-review": "review" + } +} +``` + +Profiles cannot contain credentials, commands, executable paths, tool grants, +or remote configuration references. RAE snapshots the canonical profile +digest and the resolved route, model, executor version, and executable digest +for every provider-backed node. Resume fails when this provenance drifts. + +OpenCode is explicit and is never selected by `--provider auto`. OpenCode write +routes require an isolated RAE worktree and the macOS `sandbox-exec` backend. +The launcher verifies OpenCode's merged configuration before execution and +admits only read, glob, grep, workspace edit when required, and RAE's opaque +verification broker. Shell, web, external-directory, subagent, skill, plugin, +question, and unapproved MCP access remain denied. + +The first OpenCode release supports provider models configured in OpenCode, +including `opencode/...` and `openrouter/...`. RAE does not call OpenRouter +directly. + +## Interpretation limits + +- the workflow remains provider-neutral, but selected prompts and repository + context still cross the configured provider boundary +- local doctor and containment checks do not prove provider-side storage, + retention, availability, or model behavior +- the current OpenCode adapter is a macOS-only execution surface + +## Source note + +- [OpenCode CLI](../claims/bibliography.md#src-opencode-cli) +- [OpenCode configuration](../claims/bibliography.md#src-opencode-config) +- [OpenCode permissions](../claims/bibliography.md#src-opencode-permissions) +- [OpenCode tools](../claims/bibliography.md#src-opencode-tools) +- [OpenCode providers](../claims/bibliography.md#src-opencode-providers) +- [JSON Schema 2020-12](../claims/bibliography.md#src-json-schema-2020-12) +- [Apple App Sandbox file access](../claims/bibliography.md#src-apple-app-sandbox) diff --git a/docs/reference/contracts/workflow-v2.2.md b/docs/reference/contracts/workflow-v2.2.md new file mode 100644 index 0000000..130ce75 --- /dev/null +++ b/docs/reference/contracts/workflow-v2.2.md @@ -0,0 +1,65 @@ +--- +status: experimental +owner: orchestration +last_reviewed: 2026-08-04 +source_of_truth: packages/orchestration/contracts/workflows/workflow-v2.2.schema.json +evidence_links: ../claims/claims-ledger.md +--- + +# Workflow 2.2 Contract + +Workflow 2.2 is an experimental local scheduler contract for durable waits and +operator signals. It is separate from the experimental hosted platform. The +local scheduler requires a durable run directory and is selected only for a +workflow whose `schema_version` is `2.2.0`. + +## Schema shape + +A workflow declares an identifier, revision, entry and terminal nodes, nodes, +edges, and at least one signal contract. Nodes may be `agent`, `join`, `gate`, +`checkpoint`, `wait`, or `terminal`. A wait node declares a timeout from 60 +seconds through 30 days, its accepted signal names, and the signal contract +that validates a recorded payload. + +Budgets constrain concurrency to 1 through 4, attempts per node to 1 through +3, and context to 16 KiB through 256 KiB. The default context cap is 128 KiB. +The immutable node envelope records input and output digests, an execution +tier, evidence fields, and a context manifest with included, omitted, inline, +and referenced inputs. + +## Context assembly + +The scheduler orders mandatory context as task, node guidance, mapped item, +and predecessor records. It either includes a complete predecessor object or +an immutable artifact reference. If mandatory material cannot fit, scheduling +fails before a provider call. Operational evidence, verified graph records, +and admitted memory are optional and require explicit policy permission and +budget. + +This is a bounded-context implementation property. It does not establish a +context-efficiency benefit. A frozen comparison with a predefined 25 percent +threshold remains required before any efficiency claim can be made. + +## Wait state + +Wait signals are persisted under +`.pipeline/runs//workflow/wait-state.json`. Signal recording is +idempotent per wait node and idempotency key. On resume, the reducer consumes +the earliest accepted unconsumed signal at or before the wait deadline. A wait +does not invoke a provider while it is open. A timed-out wait creates a failed +node envelope and follows failure edges when the workflow defines them. + +Workflow 2.2 does not migrate stored 2.0 or 2.1 runs or private registries. +Use [Run a Workflow 2.2 Wait](../../how-to/run-workflow-v2.2-wait.md) and +[Recover a Workflow 2.2 Wait](../../how-to/recover-workflow-v2.2-wait.md) for +local operations. + +## Source note + +- [NIST GenAI Profile](../claims/bibliography.md#src-nist-genai-profile) +- [Model Cards](../claims/bibliography.md#src-model-cards) +- [Datasheets](../claims/bibliography.md#src-datasheets) +- [OpenAI evals guidance](../claims/bibliography.md#src-openai-evals) +- [PaperBench](../claims/bibliography.md#src-openai-paperbench) +- [IEEE 1012](../claims/bibliography.md#src-ieee-1012) +- [Diataxis](../claims/bibliography.md#src-diataxis) diff --git a/docs/reference/terminology.md b/docs/reference/terminology.md index b3f721a..1f0eab0 100644 --- a/docs/reference/terminology.md +++ b/docs/reference/terminology.md @@ -1,7 +1,7 @@ --- status: stable owner: core -last_reviewed: 2026-04-12 +last_reviewed: 2026-08-04 source_of_truth: editorial evidence_links: claims/claims-ledger.md --- @@ -43,7 +43,12 @@ evidence_links: claims/claims-ledger.md stable mapped-item key. - `execution tier` A workflow-owned logical request for economy, standard, or judgment work. An - operator-owned execution profile resolves it to concrete Codex settings. + operator-owned execution profile resolves it to a named Codex or OpenCode + route without placing provider configuration in the workflow. +- `execution route` + An operator-owned executor and model mapping selected by a logical tier or a + node-specific override. Routes are stored in execution profile 3.0, not in + workflow revisions. ## Thesis validation diff --git a/docs/tutorials/graph-engineering-with-codex.md b/docs/tutorials/graph-engineering-with-rae.md similarity index 76% rename from docs/tutorials/graph-engineering-with-codex.md rename to docs/tutorials/graph-engineering-with-rae.md index 580764c..9a250f5 100644 --- a/docs/tutorials/graph-engineering-with-codex.md +++ b/docs/tutorials/graph-engineering-with-rae.md @@ -1,25 +1,25 @@ --- status: experimental owner: orchestration -last_reviewed: 2026-07-30 +last_reviewed: 2026-08-04 source_of_truth: packages/orchestration/contracts/workflows/workflow-v2.1.schema.json evidence_links: ../reference/claims/evidence-index.md --- -# Graph Engineering with Codex and RAE +# Graph Engineering with RAE This course has two layers. The topology layer explains what work can proceed and what must wait. The contract layer names the workflow fields that make that decision durable and reviewable. -RAE and Codex solve different coordination problems. Native Codex +RAE workflows are provider-neutral. Native Codex [subagents](https://learn.chatgpt.com/docs/agent-configuration/subagents) are -collaborators inside one Codex task. A RAE agent node starts a fresh, -ephemeral [`codex exec`](https://learn.chatgpt.com/docs/non-interactive-mode) -session and persists its validated result as workflow evidence. RAE also uses -isolated Git [worktrees](https://learn.chatgpt.com/docs/environments/git-worktrees) -for writer runs. It does not preserve a shared model conversation between -nodes. +collaborators inside one Codex task. A RAE agent node is a durable workflow +unit whose provider route is selected by the operator. Codex and OpenCode +attempts both start fresh sessions and persist only validated workflow output +and sanitized execution evidence. Writer runs use isolated Git +[worktrees](https://learn.chatgpt.com/docs/environments/git-worktrees). Nodes do +not share a model conversation. Every agent or mapped agent instance consumes model tokens. Scheduling, joins, condition evaluation, and allowlisted transforms do not call a model. A wider @@ -44,24 +44,40 @@ only typed predecessor envelopes and the mapped item, when present. A node can request a logical `economy`, `standard`, or `judgment` tier. It cannot name a provider, model, tool, command, environment value, or reasoning effort. -The operator owns concrete model selection in an execution profile: +The operator owns concrete provider and model selection in execution profile +3.0: ```json { - "schema_version": "1.0.0", - "profile_id": "local-codex-routing", + "schema_version": "3.0.0", + "profile_id": "local-routing", + "routes": { + "routine": { + "executor": "codex", + "model": "operator-codex-model", + "reasoning_effort": "medium" + }, + "review": { + "executor": "opencode", + "model": "opencode/example-model", + "variant": "high" + } + }, "tiers": { - "economy": { "model": "operator-economy-model", "reasoning_effort": "low" }, - "standard": { "model": "operator-standard-model", "reasoning_effort": "medium" }, - "judgment": { "model": "operator-judgment-model", "reasoning_effort": "high" } + "economy": "routine", + "standard": "routine", + "judgment": "review" } } ``` -Replace the three operator-owned model identifiers with models available in -your Codex installation. The profile is validated and snapshotted by digest. -`--execution-profile` cannot be combined with global `--model` or -`--reasoning-effort`, and resume uses the stored snapshot. +Replace the model identifiers with models available through the selected local +executors. The profile is validated and snapshotted by digest. +`--execution-profile` cannot be combined with global provider, model, +reasoning-effort, or variant overrides. Resume requires the stored profile, +resolved routes, models, and exact executor versions. See [Execution Profile +3.0](../reference/contracts/execution-profile-v3.md) for the full contract and +OpenCode platform limits. ## 3. Diamonds and deterministic reduction @@ -119,11 +135,39 @@ when a round yields no globally unseen stable key. The seen set includes every previous decision, including rejected findings. This prevents a rejected item from being rediscovered forever under a different round. -## 8. Proposal, validation, activation, execution, evidence +## 8. Design, analyze, propose, and activate ![Human-activated workflow lifecycle](../assets/diagrams/human-activated-workflow-lifecycle.svg) -Generate a draft: +Start the loopback operator to use the synchronized Loop, Graph, Analyze, and +JSON views: + +```bash +./scripts/rae.sh operator serve \ + --project /path/to/target-repository \ + --execution-profile /absolute/path/to/execution-profile.json +``` + +The guided editor provides single-agent verification, maker-checker repair, +parallel review with quorum, mapped work, and bounded until-dry templates. Each +template compiles to workflow 2.1. Node and edge controls are keyboard +operable; JSON remains available for existing workflow 2.0 and experimental +2.2 revisions. + +Analyze a workflow file without saving it: + +```bash +./scripts/rae.sh graph workflow analyze \ + --workflow-file /absolute/path/to/workflow.json \ + --execution-profile /absolute/path/to/execution-profile.json +``` + +The analyzer reports schema and topology diagnostics, unreachable nodes, +writer and verification paths, bounded attempt and instance estimates, +concurrency bounds, and resolved execution routes. It does not invent a cost +estimate when provider usage data is unavailable. + +Generate a validated candidate without saving it: ```bash ./scripts/rae.sh graph workflow propose \ @@ -131,12 +175,15 @@ Generate a draft: --task "Design a bounded review topology for this repository" \ --base-workflow graph-native-default \ --actor "operator-name" \ - --rationale "Draft for topology review" + --rationale "Draft for topology review" \ + --execution-profile /absolute/path/to/execution-profile.json \ + --preview ``` The proposal session is read-only and ephemeral. RAE permits one correction -attempt after local validation, then stores only a valid optimistic-lock draft. -The command does not activate or execute it. +attempt after local validation. `--preview` returns the candidate without +saving it; omitting `--preview` stores a valid optimistic-lock draft. Neither +mode activates or executes the result. Review and activate an exact revision: diff --git a/evals/scripts/lib/policy_optimizer.py b/evals/scripts/lib/policy_optimizer.py index 489713e..b711fa0 100644 --- a/evals/scripts/lib/policy_optimizer.py +++ b/evals/scripts/lib/policy_optimizer.py @@ -378,19 +378,6 @@ def _start_campaign( return state, output_dir / "lineage.jsonl" -def _record_blocked_iteration( - state: dict[str, Any], - lineage_path: pathlib.Path, - iteration: int, - reason: str, - candidate_id: str | None = None, -) -> None: - event: dict[str, Any] = {"iteration": iteration, "decision": "blocked", "reason": reason} - if candidate_id is not None: - event["candidate_id"] = candidate_id - _record_event(state["lineage"], lineage_path, event) - - def _run_iterations( state: dict[str, Any], lineage_path: pathlib.Path, @@ -398,7 +385,15 @@ def _run_iterations( ) -> None: for iteration in range(1, campaign.max_iterations + 1): if trusted_manifest(campaign.trusted_paths) != state["initial_manifest"]: - _record_blocked_iteration(state, lineage_path, iteration, "evaluator-integrity-drift") + _record_event( + state["lineage"], + lineage_path, + { + "iteration": iteration, + "decision": "blocked", + "reason": "evaluator-integrity-drift", + }, + ) break candidate, candidate_id = _candidate_or_rejection( campaign.proposer, @@ -415,12 +410,15 @@ def _run_iterations( state, candidate, campaign.evaluator, campaign.resource_budget ) if budget_blocked: - _record_blocked_iteration( - state, + _record_event( + state["lineage"], lineage_path, - iteration, - "budget-exceeded-or-incomplete-measurement", - candidate_id, + { + "iteration": iteration, + "candidate_id": candidate_id, + "decision": "blocked", + "reason": "budget-exceeded-or-incomplete-measurement", + }, ) dump_json(campaign.output_dir / "evaluations" / f"{candidate_id}.json", evaluation) break diff --git a/evals/scripts/lib/release_gate_core.py b/evals/scripts/lib/release_gate_core.py index dfe9edd..ea0a054 100644 --- a/evals/scripts/lib/release_gate_core.py +++ b/evals/scripts/lib/release_gate_core.py @@ -1,191 +1,79 @@ #!/usr/bin/env python3 -"""Release gate CLI and core checks.""" +"""Compose release-gate contract checks and retain their stable import surface.""" -import json import pathlib -from typing import Any, cast - -from common import ( - RESULTS_ROOT, - iso_timestamp, - load_json, - load_json_object, - path_exists, - repo_relpath, - resolve_metadata_path, +from typing import Any + +from common import iso_timestamp, path_exists, repo_relpath + +from lib.release_gate_evidence import ( + _ARTIFACT_READ_ERRORS, + REQUIRED_RUN_FIELDS, + _benchmark_reference_issues, + _calibration_issues, + _candidate_matches, + _checkpoint_gate_issues, + _declared_input_issues, + _ledger_registration_issues, + _load_ledger_entries, + _load_object, + _prior_gate_report_issues, + _provided_evidence_issues, + _provided_types_match, + _regression_status_issues, + _required_field_issues, + _result_artifact_issues, + _result_payload_issues, + _summary_evidence_issues, + cross_split_evidence_issues, + discover_release_gated_evidence, + verification_evidence_issues, ) - -from lib.release_gate_helpers import ( - path_under_results, - path_within_run_scope, - resolve_declared_results_path, - same_repo_path, - validate_checkpoint_statuses, - validate_verification_evidence_entry, +from lib.release_gate_helpers import path_under_results +from lib.release_gate_resources import ( + _resource_policy_has_limit, + _resource_usage_issues, + _resource_usage_state_issues, + _scalar_resource_issues, + _total_token_issues, ) -REQUIRED_RUN_FIELDS = ( - "benchmark_id", - "benchmark_version", - "split", - "judge_version", - "command", - "result_path", - "judge_calibration_report_path", - "ledger_path", - "regression_report_path", - "cost_usd", - "latency_seconds", +__all__ = ( + "REQUIRED_RUN_FIELDS", + "_ARTIFACT_READ_ERRORS", + "_benchmark_reference_issues", + "_build_gate_report", + "_calibration_issues", + "_candidate_matches", + "_checkpoint_gate_issues", + "_claim_link_issues", + "_declared_input_issues", + "_ledger_registration_issues", + "_load_ledger_entries", + "_load_object", + "_numeric_field_issues", + "_prior_gate_report_issues", + "_provided_evidence_issues", + "_provided_types_match", + "_regression_status_issues", + "_required_field_issues", + "_required_split_issues", + "_resource_policy_has_limit", + "_resource_usage_issues", + "_resource_usage_state_issues", + "_result_artifact_issues", + "_result_payload_issues", + "_scalar_resource_issues", + "_summary_evidence_issues", + "_total_token_issues", + "cross_split_evidence_issues", + "discover_release_gated_evidence", + "path_under_results", + "validate_run_card_contract", + "verification_evidence_issues", ) -def _load_object(path: pathlib.Path) -> dict[str, Any]: - return cast(dict[str, Any], load_json_object(path)) - - -def _required_field_issues(run_card: dict[str, Any]) -> list[str]: - return [ - f"run card missing required field: {field}" - for field in REQUIRED_RUN_FIELDS - if run_card.get(field) in (None, "") - ] - - -def _benchmark_reference_issues(benchmark: dict[str, Any], run_card: dict[str, Any]) -> list[str]: - issues: list[str] = [] - if run_card.get("benchmark_id") not in (None, benchmark["benchmark_id"]): - issues.append("run card benchmark_id does not match benchmark card") - if run_card.get("benchmark_version") not in (None, benchmark["version"]): - issues.append("run card benchmark_version does not match benchmark card") - return issues - - -def _checkpoint_gate_issues( - benchmark: dict[str, Any], - run_card: dict[str, Any], - run_card_path: pathlib.Path, -) -> list[str]: - states, issues = validate_checkpoint_statuses(run_card, run_card_path) - block_pending = benchmark.get("release_gate", {}).get("block_on_pending_checkpoints", True) - if block_pending: - issues.extend( - f"checkpoint not approved: {state}" for state in states if state != "approved" - ) - return issues - - -def _load_ledger_entries(path: pathlib.Path) -> list[dict[str, Any]]: - entries: list[dict[str, Any]] = [] - with path.open("r", encoding="utf-8") as handle: - for line in handle: - if not line.strip(): - continue - entry = json.loads(line) - if isinstance(entry, dict): - entries.append(cast(dict[str, Any], entry)) - return entries - - -def _ledger_registration_issues(ledger_path: pathlib.Path, run_id: object) -> list[str]: - try: - entries = _load_ledger_entries(ledger_path) - except Exception: - return ["result ledger could not be read"] - if not any(entry.get("run_id") == run_id for entry in entries): - return ["run_id missing from result ledger"] - return [] - - -def _regression_status_issues(path: pathlib.Path) -> list[str]: - try: - regression = _load_object(path) - except Exception: - return ["regression report could not be read"] - if regression.get("status") != "pass": - return ["regression report is not pass"] - return [] - - -def _prior_gate_report_issues(run_card: dict[str, Any]) -> list[str]: - if run_card.get("release_gate_status") is None: - return [] - gate_path, issues = resolve_declared_results_path( - run_card, "release_gate_report_path", "release gate report" - ) - if gate_path is None: - return issues - try: - gate_report = _load_object(gate_path) - except Exception: - return [*issues, "release gate report could not be read"] - if gate_report.get("run_id") != run_card.get("run_id"): - issues.append("release gate report run_id mismatch") - return issues - - -def cross_split_evidence_issues( - benchmark: dict[str, Any], - run_card: dict[str, Any], - run_card_path: pathlib.Path, -) -> list[str]: - """Validate non-circular evidence for an earlier required split.""" - regression_path, regression_issues = resolve_declared_results_path( - run_card, "regression_report_path", "regression report" - ) - ledger_path, ledger_issues = resolve_declared_results_path( - run_card, "ledger_path", "result ledger" - ) - issues = [*regression_issues, *ledger_issues] - if regression_path is None or ledger_path is None: - return issues - issues.extend(_required_field_issues(run_card)) - issues.extend(_benchmark_reference_issues(benchmark, run_card)) - issues.extend(_regression_status_issues(regression_path)) - issues.extend(verification_evidence_issues(run_card, run_card_path)) - issues.extend(_checkpoint_gate_issues(benchmark, run_card, run_card_path)) - issues.extend(_ledger_registration_issues(ledger_path, run_card.get("run_id"))) - issues.extend(_prior_gate_report_issues(run_card)) - return issues - - -def _candidate_matches( - candidate: dict[str, Any], - benchmark_id: str, - benchmark_version: str, - required_split: str, -) -> bool: - return all( - ( - candidate.get("evidence_type", "benchmark-run") == "benchmark-run", - candidate.get("benchmark_id") == benchmark_id, - candidate.get("benchmark_version") == benchmark_version, - candidate.get("split") == required_split, - ) - ) - - -def discover_release_gated_evidence( - benchmark: dict[str, Any], - benchmark_id: str, - benchmark_version: str, - required_split: str, - current_run_card: dict[str, Any], - current_run_contract_ok: bool, -) -> bool: - if current_run_card.get("split") == required_split and current_run_contract_ok: - return True - for path in RESULTS_ROOT.rglob("run-card-*.json"): - try: - candidate = _load_object(path) - except (json.JSONDecodeError, ValueError): - continue - if not _candidate_matches(candidate, benchmark_id, benchmark_version, required_split): - continue - if not cross_split_evidence_issues(benchmark, candidate, path): - return True - return False - - def _numeric_field_issues(run_card: dict[str, Any]) -> list[str]: issues: list[str] = [] for field in ("cost_usd", "latency_seconds"): @@ -195,177 +83,6 @@ def _numeric_field_issues(run_card: dict[str, Any]) -> list[str]: return issues -def _resource_policy_has_limit(policy: dict[str, Any]) -> bool: - """Report whether a policy needs a resource-usage payload to enforce a limit.""" - return any( - policy.get(field) is not None - for field in ( - "max_agent_duration_seconds", - "max_agent_calls", - "max_parallelism", - "max_total_tokens", - ) - ) - - -def _scalar_resource_issues(policy: dict[str, Any], usage: dict[str, Any]) -> list[str]: - """Compare independently measured scalar resources with their policy limits.""" - limits = ( - ("max_agent_duration_seconds", "agent_duration_seconds"), - ("max_agent_calls", "agent_calls"), - ("max_parallelism", "max_parallelism"), - ) - issues: list[str] = [] - for policy_field, usage_field in limits: - limit = policy.get(policy_field) - value = usage.get(usage_field) - if limit is not None and (not isinstance(value, (int, float)) or value > limit): - issues.append(f"run card exceeds resource policy {policy_field}") - return issues - - -def _total_token_issues(policy: dict[str, Any], usage: dict[str, Any]) -> list[str]: - """Require both token counters before enforcing a combined token limit.""" - total_limit = policy.get("max_total_tokens") - if total_limit is None: - return [] - input_tokens = usage.get("input_tokens") - output_tokens = usage.get("output_tokens") - if not isinstance(input_tokens, int) or not isinstance(output_tokens, int): - return ["run card lacks total-token measurement"] - if input_tokens + output_tokens > total_limit: - return ["run card exceeds resource policy max_total_tokens"] - return [] - - -def _resource_usage_state_issues(policy: dict[str, Any], usage: object) -> list[str]: - if not isinstance(usage, dict): - return ( - ["run card missing resource_usage"] - if policy.get("require_measurement") or _resource_policy_has_limit(policy) - else [] - ) - status = usage.get("measurement_status") - if status not in {"complete", "partial", "unavailable"}: - return ["run card has invalid resource_usage.measurement_status"] - if policy.get("require_measurement") and status != "complete": - return ["run card resource measurement is not complete"] - return [] - - -def _resource_usage_issues(benchmark: dict[str, Any], run_card: dict[str, Any]) -> list[str]: - """Enforce an optional resource policy without treating unknown usage as zero.""" - policy = benchmark.get("resource_policy") - if policy is None: - return [] - if not isinstance(policy, dict): - return ["benchmark resource_policy must be an object"] - usage = run_card.get("resource_usage") - issues = _resource_usage_state_issues(policy, usage) - if not isinstance(usage, dict): - return issues - issues.extend(_scalar_resource_issues(policy, usage)) - issues.extend(_total_token_issues(policy, usage)) - return issues - - -def _result_payload_issues( - result: object, - benchmark: dict[str, Any], - run_card: dict[str, Any], -) -> list[str]: - if not isinstance(result, dict): - return ["benchmark result must be a JSON object"] - checks = ( - ("run_id", run_card.get("run_id"), "benchmark result run_id mismatch"), - ( - "benchmark_id", - benchmark["benchmark_id"], - "benchmark result benchmark_id mismatch", - ), - ( - "benchmark_version", - benchmark["version"], - "benchmark result benchmark_version mismatch", - ), - ("split", run_card.get("split"), "benchmark result split mismatch"), - ) - for field, expected, message in checks: - if result.get(field) != expected: - return [message] - if "fail_count" not in result or "task_count" not in result: - return ["benchmark result missing failure summary"] - if "aggregate_metrics" not in result: - return ["benchmark result missing aggregate_metrics"] - return [] - - -def _result_artifact_issues( - benchmark: dict[str, Any], - run_card: dict[str, Any], - run_card_path: pathlib.Path, -) -> list[str]: - result_ref = run_card.get("result_path") - if not result_ref: - return [] - try: - result_path = resolve_metadata_path( - result_ref, label="run card result_path", contained_by=RESULTS_ROOT - ) - except ValueError as exc: - return [str(exc)] - if not path_within_run_scope(result_path, run_card_path): - return ["run card result_path is outside current run scope"] - if not result_path.exists(): - return ["benchmark result artifact missing"] - return _result_payload_issues(load_json(result_path), benchmark, run_card) - - -def _calibration_issues(run_card: dict[str, Any]) -> list[str]: - calibration_ref = run_card.get("judge_calibration_report_path") - if not calibration_ref: - return [] - try: - path = resolve_metadata_path( - calibration_ref, - label="judge calibration report", - contained_by=RESULTS_ROOT, - ) - except ValueError as exc: - return [str(exc)] - if not path.exists(): - return ["judge calibration report missing"] - calibration = load_json(path) - if not isinstance(calibration, dict): - return ["judge calibration report must be a JSON object"] - if calibration.get("calibration_case_count", 0) <= 0: - return ["judge calibration report has no calibration cases"] - rate = calibration.get("agreement_rate") - if not isinstance(rate, (int, float)): - return ["judge calibration report missing agreement_rate"] - if rate < 1.0: - return ["judge calibration agreement_rate is below required threshold"] - return [] - - -def _declared_input_issues( - run_card: dict[str, Any], - field: str, - provided_path: pathlib.Path, - mismatch_message: str, -) -> list[str]: - path_ref = run_card.get(field) - if not path_ref: - return [] - try: - declared = resolve_metadata_path( - path_ref, label=f"run card {field}", contained_by=RESULTS_ROOT - ) - except ValueError as exc: - return [str(exc)] - return [] if same_repo_path(declared, provided_path) else [mismatch_message] - - def validate_run_card_contract( benchmark: dict[str, Any], run_card: dict[str, Any], @@ -406,73 +123,6 @@ def validate_run_card_contract( return issues -def _summary_evidence_issues(summary: dict[str, Any]) -> list[str]: - issues: list[str] = [] - status = summary.get("status") - if status in {"partial", "missing"}: - issues.append(f"verification evidence incomplete: {status}") - missing_types = summary.get("missing_types", []) - if isinstance(missing_types, list): - issues.extend( - f"missing required verification evidence type: {evidence_type}" - for evidence_type in missing_types - if isinstance(evidence_type, str) and evidence_type - ) - return issues - - -def _provided_evidence_issues( - provided: list[object], - run_card: dict[str, Any], - run_card_path: pathlib.Path, -) -> tuple[list[str], set[str]]: - issues: list[str] = [] - validated_types: set[str] = set() - for index, entry in enumerate(provided): - if not isinstance(entry, dict): - issues.append(f"verification_evidence.provided[{index}] must be an object") - continue - entry_type = entry.get("type") - if isinstance(entry_type, str) and entry_type: - validated_types.add(entry_type) - issues.extend( - validate_verification_evidence_entry( - entry, index=index, run_card=run_card, run_card_path=run_card_path - ) - ) - return issues, validated_types - - -def _provided_types_match(summary: dict[str, Any], validated_types: set[str]) -> bool: - declared = summary.get("provided_types") - if not isinstance(declared, list): - return True - declared_types = {entry for entry in declared if isinstance(entry, str) and entry} - return declared_types == validated_types - - -def verification_evidence_issues( - run_card: dict[str, Any], run_card_path: pathlib.Path -) -> list[str]: - evidence = run_card.get("verification_evidence") - if evidence is None: - return [] - if not isinstance(evidence, dict): - return ["verification_evidence must be an object when present"] - summary = evidence.get("summary") - if not isinstance(summary, dict): - return ["verification_evidence.summary missing"] - issues = _summary_evidence_issues(summary) - provided = evidence.get("provided") - if not isinstance(provided, list): - return [*issues, "verification_evidence.provided missing"] - provided_issues, validated_types = _provided_evidence_issues(provided, run_card, run_card_path) - issues.extend(provided_issues) - if not _provided_types_match(summary, validated_types): - issues.append("verification evidence provided_types does not match provided entries") - return issues - - def _claim_link_issues(benchmark: dict[str, Any]) -> tuple[list[str], list[Any]]: claim_links = benchmark.get("claim_links", []) required = benchmark.get("release_gate", {}).get("required_claim_links", claim_links) diff --git a/evals/scripts/lib/release_gate_evidence.py b/evals/scripts/lib/release_gate_evidence.py new file mode 100644 index 0000000..48cd5be --- /dev/null +++ b/evals/scripts/lib/release_gate_evidence.py @@ -0,0 +1,345 @@ +"""Evidence and artifact validation for release-gated benchmark runs.""" + +import json +import pathlib +from typing import Any, cast + +from common import RESULTS_ROOT, load_json, load_json_object, resolve_metadata_path + +from lib.release_gate_helpers import ( + path_within_run_scope, + resolve_declared_results_path, + same_repo_path, + validate_checkpoint_statuses, + validate_verification_evidence_entry, +) + +REQUIRED_RUN_FIELDS = ( + "benchmark_id", + "benchmark_version", + "split", + "judge_version", + "command", + "result_path", + "judge_calibration_report_path", + "ledger_path", + "regression_report_path", + "cost_usd", + "latency_seconds", +) + +# Artifact parsing may fail because a run produced unreadable or malformed +# evidence. Preserve those gate failures while allowing implementation bugs +# to surface instead of being misreported as missing evidence. +_ARTIFACT_READ_ERRORS = (OSError, UnicodeDecodeError, ValueError) + + +def _load_object(path: pathlib.Path) -> dict[str, Any]: + return cast(dict[str, Any], load_json_object(path)) + + +def _required_field_issues(run_card: dict[str, Any]) -> list[str]: + return [ + f"run card missing required field: {field}" + for field in REQUIRED_RUN_FIELDS + if run_card.get(field) in (None, "") + ] + + +def _benchmark_reference_issues(benchmark: dict[str, Any], run_card: dict[str, Any]) -> list[str]: + issues: list[str] = [] + if run_card.get("benchmark_id") not in (None, benchmark["benchmark_id"]): + issues.append("run card benchmark_id does not match benchmark card") + if run_card.get("benchmark_version") not in (None, benchmark["version"]): + issues.append("run card benchmark_version does not match benchmark card") + return issues + + +def _checkpoint_gate_issues( + benchmark: dict[str, Any], + run_card: dict[str, Any], + run_card_path: pathlib.Path, +) -> list[str]: + states, issues = validate_checkpoint_statuses(run_card, run_card_path) + block_pending = benchmark.get("release_gate", {}).get("block_on_pending_checkpoints", True) + if block_pending: + issues.extend( + f"checkpoint not approved: {state}" for state in states if state != "approved" + ) + return issues + + +def _load_ledger_entries(path: pathlib.Path) -> list[dict[str, Any]]: + entries: list[dict[str, Any]] = [] + with path.open("r", encoding="utf-8") as handle: + for line in handle: + if not line.strip(): + continue + entry = json.loads(line) + if isinstance(entry, dict): + entries.append(cast(dict[str, Any], entry)) + return entries + + +def _ledger_registration_issues(ledger_path: pathlib.Path, run_id: object) -> list[str]: + try: + entries = _load_ledger_entries(ledger_path) + except _ARTIFACT_READ_ERRORS: + return ["result ledger could not be read"] + if not any(entry.get("run_id") == run_id for entry in entries): + return ["run_id missing from result ledger"] + return [] + + +def _regression_status_issues(path: pathlib.Path) -> list[str]: + try: + regression = _load_object(path) + except _ARTIFACT_READ_ERRORS: + return ["regression report could not be read"] + if regression.get("status") != "pass": + return ["regression report is not pass"] + return [] + + +def _prior_gate_report_issues(run_card: dict[str, Any]) -> list[str]: + if run_card.get("release_gate_status") is None: + return [] + gate_path, issues = resolve_declared_results_path( + run_card, "release_gate_report_path", "release gate report" + ) + if gate_path is None: + return issues + try: + gate_report = _load_object(gate_path) + except _ARTIFACT_READ_ERRORS: + return [*issues, "release gate report could not be read"] + if gate_report.get("run_id") != run_card.get("run_id"): + issues.append("release gate report run_id mismatch") + return issues + + +def _summary_evidence_issues(summary: dict[str, Any]) -> list[str]: + issues: list[str] = [] + status = summary.get("status") + if status in {"partial", "missing"}: + issues.append(f"verification evidence incomplete: {status}") + missing_types = summary.get("missing_types", []) + if isinstance(missing_types, list): + issues.extend( + f"missing required verification evidence type: {evidence_type}" + for evidence_type in missing_types + if isinstance(evidence_type, str) and evidence_type + ) + return issues + + +def _provided_evidence_issues( + provided: list[object], + run_card: dict[str, Any], + run_card_path: pathlib.Path, +) -> tuple[list[str], set[str]]: + issues: list[str] = [] + validated_types: set[str] = set() + for index, entry in enumerate(provided): + if not isinstance(entry, dict): + issues.append(f"verification_evidence.provided[{index}] must be an object") + continue + entry_type = entry.get("type") + if isinstance(entry_type, str) and entry_type: + validated_types.add(entry_type) + issues.extend( + validate_verification_evidence_entry( + entry, index=index, run_card=run_card, run_card_path=run_card_path + ) + ) + return issues, validated_types + + +def _provided_types_match(summary: dict[str, Any], validated_types: set[str]) -> bool: + declared = summary.get("provided_types") + if not isinstance(declared, list): + return True + declared_types = {entry for entry in declared if isinstance(entry, str) and entry} + return declared_types == validated_types + + +def verification_evidence_issues( + run_card: dict[str, Any], run_card_path: pathlib.Path +) -> list[str]: + evidence = run_card.get("verification_evidence") + if evidence is None: + return [] + if not isinstance(evidence, dict): + return ["verification_evidence must be an object when present"] + summary = evidence.get("summary") + if not isinstance(summary, dict): + return ["verification_evidence.summary missing"] + issues = _summary_evidence_issues(summary) + provided = evidence.get("provided") + if not isinstance(provided, list): + return [*issues, "verification_evidence.provided missing"] + provided_issues, validated_types = _provided_evidence_issues(provided, run_card, run_card_path) + issues.extend(provided_issues) + if not _provided_types_match(summary, validated_types): + issues.append("verification evidence provided_types does not match provided entries") + return issues + + +def cross_split_evidence_issues( + benchmark: dict[str, Any], + run_card: dict[str, Any], + run_card_path: pathlib.Path, +) -> list[str]: + """Validate non-circular evidence for an earlier required split.""" + regression_path, regression_issues = resolve_declared_results_path( + run_card, "regression_report_path", "regression report" + ) + ledger_path, ledger_issues = resolve_declared_results_path( + run_card, "ledger_path", "result ledger" + ) + issues = [*regression_issues, *ledger_issues] + if regression_path is None or ledger_path is None: + return issues + issues.extend(_required_field_issues(run_card)) + issues.extend(_benchmark_reference_issues(benchmark, run_card)) + issues.extend(_regression_status_issues(regression_path)) + issues.extend(verification_evidence_issues(run_card, run_card_path)) + issues.extend(_checkpoint_gate_issues(benchmark, run_card, run_card_path)) + issues.extend(_ledger_registration_issues(ledger_path, run_card.get("run_id"))) + issues.extend(_prior_gate_report_issues(run_card)) + return issues + + +def _candidate_matches( + candidate: dict[str, Any], + benchmark_id: str, + benchmark_version: str, + required_split: str, +) -> bool: + return all( + ( + candidate.get("evidence_type", "benchmark-run") == "benchmark-run", + candidate.get("benchmark_id") == benchmark_id, + candidate.get("benchmark_version") == benchmark_version, + candidate.get("split") == required_split, + ) + ) + + +def discover_release_gated_evidence( + benchmark: dict[str, Any], + benchmark_id: str, + benchmark_version: str, + required_split: str, + current_run_card: dict[str, Any], + current_run_contract_ok: bool, +) -> bool: + if current_run_card.get("split") == required_split and current_run_contract_ok: + return True + for path in RESULTS_ROOT.rglob("run-card-*.json"): + try: + candidate = _load_object(path) + except (json.JSONDecodeError, ValueError): + continue + if not _candidate_matches(candidate, benchmark_id, benchmark_version, required_split): + continue + if not cross_split_evidence_issues(benchmark, candidate, path): + return True + return False + + +def _result_payload_issues( + result: object, + benchmark: dict[str, Any], + run_card: dict[str, Any], +) -> list[str]: + if not isinstance(result, dict): + return ["benchmark result must be a JSON object"] + checks = ( + ("run_id", run_card.get("run_id"), "benchmark result run_id mismatch"), + ( + "benchmark_id", + benchmark["benchmark_id"], + "benchmark result benchmark_id mismatch", + ), + ( + "benchmark_version", + benchmark["version"], + "benchmark result benchmark_version mismatch", + ), + ("split", run_card.get("split"), "benchmark result split mismatch"), + ) + for field, expected, message in checks: + if result.get(field) != expected: + return [message] + if "fail_count" not in result or "task_count" not in result: + return ["benchmark result missing failure summary"] + if "aggregate_metrics" not in result: + return ["benchmark result missing aggregate_metrics"] + return [] + + +def _result_artifact_issues( + benchmark: dict[str, Any], + run_card: dict[str, Any], + run_card_path: pathlib.Path, +) -> list[str]: + result_ref = run_card.get("result_path") + if not result_ref: + return [] + try: + result_path = resolve_metadata_path( + result_ref, label="run card result_path", contained_by=RESULTS_ROOT + ) + except ValueError as exc: + return [str(exc)] + if not path_within_run_scope(result_path, run_card_path): + return ["run card result_path is outside current run scope"] + if not result_path.exists(): + return ["benchmark result artifact missing"] + return _result_payload_issues(load_json(result_path), benchmark, run_card) + + +def _calibration_issues(run_card: dict[str, Any]) -> list[str]: + calibration_ref = run_card.get("judge_calibration_report_path") + if not calibration_ref: + return [] + try: + path = resolve_metadata_path( + calibration_ref, + label="judge calibration report", + contained_by=RESULTS_ROOT, + ) + except ValueError as exc: + return [str(exc)] + if not path.exists(): + return ["judge calibration report missing"] + calibration = load_json(path) + if not isinstance(calibration, dict): + return ["judge calibration report must be a JSON object"] + if calibration.get("calibration_case_count", 0) <= 0: + return ["judge calibration report has no calibration cases"] + rate = calibration.get("agreement_rate") + if not isinstance(rate, (int, float)): + return ["judge calibration report missing agreement_rate"] + if rate < 1.0: + return ["judge calibration agreement_rate is below required threshold"] + return [] + + +def _declared_input_issues( + run_card: dict[str, Any], + field: str, + provided_path: pathlib.Path, + mismatch_message: str, +) -> list[str]: + path_ref = run_card.get(field) + if not path_ref: + return [] + try: + declared = resolve_metadata_path( + path_ref, label=f"run card {field}", contained_by=RESULTS_ROOT + ) + except ValueError as exc: + return [str(exc)] + return [] if same_repo_path(declared, provided_path) else [mismatch_message] diff --git a/evals/scripts/lib/release_gate_resources.py b/evals/scripts/lib/release_gate_resources.py new file mode 100644 index 0000000..aed5064 --- /dev/null +++ b/evals/scripts/lib/release_gate_resources.py @@ -0,0 +1,77 @@ +"""Resource-policy validation for release-gated benchmark runs.""" + +from typing import Any + + +def _resource_policy_has_limit(policy: dict[str, Any]) -> bool: + """Report whether a policy needs a resource-usage payload to enforce a limit.""" + return any( + policy.get(field) is not None + for field in ( + "max_agent_duration_seconds", + "max_agent_calls", + "max_parallelism", + "max_total_tokens", + ) + ) + + +def _scalar_resource_issues(policy: dict[str, Any], usage: dict[str, Any]) -> list[str]: + """Compare independently measured scalar resources with their policy limits.""" + limits = ( + ("max_agent_duration_seconds", "agent_duration_seconds"), + ("max_agent_calls", "agent_calls"), + ("max_parallelism", "max_parallelism"), + ) + issues: list[str] = [] + for policy_field, usage_field in limits: + limit = policy.get(policy_field) + value = usage.get(usage_field) + if limit is not None and (not isinstance(value, (int, float)) or value > limit): + issues.append(f"run card exceeds resource policy {policy_field}") + return issues + + +def _total_token_issues(policy: dict[str, Any], usage: dict[str, Any]) -> list[str]: + """Require both token counters before enforcing a combined token limit.""" + total_limit = policy.get("max_total_tokens") + if total_limit is None: + return [] + input_tokens = usage.get("input_tokens") + output_tokens = usage.get("output_tokens") + if not isinstance(input_tokens, int) or not isinstance(output_tokens, int): + return ["run card lacks total-token measurement"] + if input_tokens + output_tokens > total_limit: + return ["run card exceeds resource policy max_total_tokens"] + return [] + + +def _resource_usage_state_issues(policy: dict[str, Any], usage: object) -> list[str]: + if not isinstance(usage, dict): + return ( + ["run card missing resource_usage"] + if policy.get("require_measurement") or _resource_policy_has_limit(policy) + else [] + ) + status = usage.get("measurement_status") + if status not in {"complete", "partial", "unavailable"}: + return ["run card has invalid resource_usage.measurement_status"] + if policy.get("require_measurement") and status != "complete": + return ["run card resource measurement is not complete"] + return [] + + +def _resource_usage_issues(benchmark: dict[str, Any], run_card: dict[str, Any]) -> list[str]: + """Enforce an optional resource policy without treating unknown usage as zero.""" + policy = benchmark.get("resource_policy") + if policy is None: + return [] + if not isinstance(policy, dict): + return ["benchmark resource_policy must be an object"] + usage = run_card.get("resource_usage") + issues = _resource_usage_state_issues(policy, usage) + if not isinstance(usage, dict): + return issues + issues.extend(_scalar_resource_issues(policy, usage)) + issues.extend(_total_token_issues(policy, usage)) + return issues diff --git a/evals/scripts/lib/run_benchmark_artifacts.py b/evals/scripts/lib/run_benchmark_artifacts.py new file mode 100644 index 0000000..e8e247b --- /dev/null +++ b/evals/scripts/lib/run_benchmark_artifacts.py @@ -0,0 +1,288 @@ +"""Build benchmark evidence artifacts and finish release-gate reporting.""" + +import pathlib +import sys +from collections.abc import Callable +from dataclasses import dataclass +from typing import Any + +from common import ( + RESULTS_ROOT, + append_jsonl, + default_system_metadata, + dump_json, + iso_timestamp, + load_json, + metric_ratio, + repo_relpath, + today_iso, +) +from router import ROUTER_VERSION + +from lib.run_benchmark_evidence import aggregate_verification_evidence + + +@dataclass(frozen=True) +class BenchmarkRun: + """Result data shared by the benchmark evidence-writing stages.""" + + benchmark: dict[str, Any] + split: str + run_id: str + task_results: list[dict[str, Any]] + metrics: dict[str, float] + result_path: pathlib.Path + output_dir: pathlib.Path + + +@dataclass(frozen=True) +class BenchmarkArtifacts: + """Evidence paths consumed by the final release-gate stage.""" + + run_card_path: pathlib.Path + regression_path: pathlib.Path + ledger_path: pathlib.Path + calibration_path: pathlib.Path + + +def _count_passes(task_results: list[dict[str, Any]]) -> int: + return sum(1 for result in task_results if result["judge"]["verdict"] == "pass") + + +def _count_judge_successes(task_results: list[dict[str, Any]], field: str) -> int: + return sum(1 for result in task_results if result["judge"][field]) + + +def _count_checkpoint_tasks(task_results: list[dict[str, Any]]) -> int: + return sum(1 for result in task_results if result["checkpoint_paths"]) + + +def _count_approved_checkpoint_tasks( + task_results: list[dict[str, Any]], +) -> int: + return sum( + 1 + for result in task_results + if result["checkpoint_paths"] and result["judge"]["checkpoint_ok"] + ) + + +def aggregate_results(task_results: list[dict[str, Any]]) -> dict[str, float]: + """Aggregate task-level benchmark results into release-gate metrics.""" + total = len(task_results) + return { + "success_rate": metric_ratio(_count_passes(task_results), total), + "route_accuracy": metric_ratio(_count_judge_successes(task_results, "route_ok"), total), + "artifact_completeness": metric_ratio( + _count_judge_successes(task_results, "artifacts_ok"), total + ), + "checkpoint_compliance": metric_ratio( + _count_approved_checkpoint_tasks(task_results), + _count_checkpoint_tasks(task_results), + ), + } + + +def _regression_issues( + metrics: dict[str, float], + baseline_metrics: dict[str, Any], + minimum_metrics: dict[str, Any], + max_negative_delta: float, +) -> list[str]: + issues: list[str] = [] + for metric, value in metrics.items(): + baseline_value = float(baseline_metrics.get(metric, value)) + if value + max_negative_delta < baseline_value: + issues.append(f"{metric} regressed below baseline") + if metric in minimum_metrics and value < float(minimum_metrics[metric]): + issues.append(f"{metric} below minimum") + return issues + + +def write_regression_report( + run: BenchmarkRun, + *, + resolve_repo_child_path: Callable[[Any, pathlib.Path, str], pathlib.Path], +) -> pathlib.Path: + """Write the regression comparison for one completed benchmark run.""" + regression_policy = run.benchmark.get("regression_policy", {}) + baselines = regression_policy.get("baseline_results", {}) + baseline_value = baselines.get(run.split, "") + baseline_path = ( + resolve_repo_child_path(baseline_value, RESULTS_ROOT, "baseline result path") + if baseline_value + else None + ) + if baseline_path is not None and baseline_path.exists(): + baseline = load_json(baseline_path) + baseline_metrics = baseline.get("aggregate_metrics", {}) + else: + baseline_metrics = {} + max_negative_delta = float(regression_policy.get("max_negative_delta", 0.0)) + minimum_metrics = regression_policy.get("minimum_metrics", {}) + regressions = _regression_issues( + run.metrics, baseline_metrics, minimum_metrics, max_negative_delta + ) + + report = { + "report_id": f"regression-{run.run_id}", + "benchmark_id": run.benchmark["benchmark_id"], + "split": run.split, + "generated_at": iso_timestamp(), + "status": "pass" if not regressions else "fail", + "baseline_result_path": repo_relpath(baseline_path) + if baseline_path is not None and baseline_path.exists() + else "", + "aggregate_metrics": run.metrics, + "baseline_metrics": baseline_metrics, + "issues": regressions, + } + output_path = run.output_dir / ( + f"regression-{run.benchmark['benchmark_id']}-{run.split}-{run.run_id}.json" + ) + dump_json(output_path, report) + return output_path + + +def write_result_ledger(run: BenchmarkRun, ledger_path: pathlib.Path) -> None: + """Append aggregate and task-result entries for one benchmark run.""" + aggregate_entry = { + "entry_id": f"{run.run_id}-aggregate", + "kind": "benchmark-run", + "timestamp": iso_timestamp(), + "benchmark_id": run.benchmark["benchmark_id"], + "run_id": run.run_id, + "split": run.split, + "result_path": repo_relpath(run.result_path), + "claim_links": run.benchmark.get("claim_links", []), + "aggregate_metrics": run.metrics, + } + append_jsonl(ledger_path, aggregate_entry) + for result in run.task_results: + append_jsonl( + ledger_path, + { + "entry_id": f"{run.run_id}-{result['task_id']}", + "kind": "task-result", + "timestamp": iso_timestamp(), + "benchmark_id": run.benchmark["benchmark_id"], + "run_id": run.run_id, + "task_id": result["task_id"], + "split": result["split"], + "routed_runtime": result["routed_runtime"], + "trace_paths": result["trace_paths"], + "artifact_paths": result["artifact_paths"], + "checkpoint_paths": result["checkpoint_paths"], + "claim_links": result["claim_links"], + "judge_verdict": result["judge"]["verdict"], + }, + ) + + +def _unique_result_paths(task_results: list[dict[str, Any]], field: str) -> list[str]: + paths: set[str] = set() + for result in task_results: + paths.update(result[field]) + return sorted(paths) + + +def _all_result_paths(task_results: list[dict[str, Any]], field: str) -> list[str]: + paths: list[str] = [] + for result in task_results: + paths.extend(result[field]) + return paths + + +def _all_tasks_passed(task_results: list[dict[str, Any]]) -> bool: + return all(result["judge"]["verdict"] == "pass" for result in task_results) + + +def _run_card_aggregate_fields( + task_results: list[dict[str, Any]], +) -> dict[str, Any]: + duration = round( + sum(result["command_result"]["duration_seconds"] for result in task_results), 4 + ) + return { + "status": "pass" if _all_tasks_passed(task_results) else "fail", + "trace_paths": _unique_result_paths(task_results, "trace_paths"), + "artifact_paths": _unique_result_paths(task_results, "artifact_paths"), + "checkpoint_paths": _all_result_paths(task_results, "checkpoint_paths"), + "latency_seconds": duration, + } + + +def _build_run_card(run: BenchmarkRun, artifacts: BenchmarkArtifacts) -> dict[str, Any]: + aggregate_fields = _run_card_aggregate_fields(run.task_results) + return { + "run_id": run.run_id, + "evidence_type": "benchmark-run", + "benchmark_id": run.benchmark["benchmark_id"], + "benchmark_version": run.benchmark["version"], + "date": today_iso(), + "split": run.split, + "system": default_system_metadata("umbrella-benchmark-runner"), + "judge_version": run.benchmark["judge_version"], + "command": "python3 evals/scripts/run_benchmark.py", + "result_path": repo_relpath(run.result_path), + "status": aggregate_fields["status"], + "task_spec_path": run.benchmark["task_specs_path"], + "routed_runtime": "mixed", + "router": {"version": ROUTER_VERSION, "decision_mode": "per-task"}, + "trace_paths": aggregate_fields["trace_paths"], + "artifact_paths": aggregate_fields["artifact_paths"], + "checkpoint_paths": aggregate_fields["checkpoint_paths"], + "verification_evidence": aggregate_verification_evidence(run.task_results), + "claim_links": run.benchmark.get("claim_links", []), + "ledger_path": repo_relpath(artifacts.ledger_path), + "regression_report_path": repo_relpath(artifacts.regression_path), + "judge_calibration_report_path": repo_relpath(artifacts.calibration_path), + "cost_usd": 0.0, + "latency_seconds": aggregate_fields["latency_seconds"], + "notes": f"Executed {len(run.task_results)} task(s) for split {run.split}.", + } + + +def _write_run_artifacts( + run: BenchmarkRun, + calibration_path: pathlib.Path, + *, + resolve_repo_child_path: Callable[[Any, pathlib.Path, str], pathlib.Path], +) -> BenchmarkArtifacts: + regression_path = write_regression_report(run, resolve_repo_child_path=resolve_repo_child_path) + ledger_path = run.output_dir / "result-ledger.jsonl" + write_result_ledger(run, ledger_path) + run_card_path = run.output_dir / ( + f"run-card-{run.benchmark['benchmark_id']}-{run.split}-{run.run_id}.json" + ) + artifacts = BenchmarkArtifacts( + run_card_path=run_card_path, + regression_path=regression_path, + ledger_path=ledger_path, + calibration_path=calibration_path, + ) + dump_json(run_card_path, _build_run_card(run, artifacts)) + return artifacts + + +def _finish_release_gate( + benchmark_path: pathlib.Path, + run: BenchmarkRun, + artifacts: BenchmarkArtifacts, + *, + run_release_gate: Callable[[pathlib.Path, BenchmarkArtifacts, pathlib.Path], dict[str, Any]], + failure_report_message: Callable[[pathlib.Path | None], str], + report_child_failure: Callable[..., int], +) -> int: + output = run.output_dir / ( + f"release-gate-{run.benchmark['benchmark_id']}-{run.split}-{run.run_id}.json" + ) + result = run_release_gate(benchmark_path, artifacts, output) + if result["returncode"] != 0: + report_message = failure_report_message(output) + if report_message: + print(report_message, file=sys.stderr) + return result["returncode"] + return report_child_failure(result, fallback="release gate failed", report_path=output) + print(repo_relpath(artifacts.run_card_path)) + return 0 diff --git a/evals/scripts/lib/run_benchmark_report.py b/evals/scripts/lib/run_benchmark_report.py index 06be876..d72f49a 100644 --- a/evals/scripts/lib/run_benchmark_report.py +++ b/evals/scripts/lib/run_benchmark_report.py @@ -4,27 +4,29 @@ import pathlib import re import sys +from dataclasses import dataclass from typing import Any from common import ( RESULTS_ROOT, ROOT, - append_jsonl, - default_system_metadata, dump_json, is_within_directory, iso_timestamp, load_json, - metric_ratio, new_run_id, repo_relpath, run_command, - today_iso, ) -from router import ROUTER_VERSION +from lib.run_benchmark_artifacts import ( + BenchmarkArtifacts, + BenchmarkRun, + _finish_release_gate, + _write_run_artifacts, + aggregate_results, +) from lib.run_benchmark_evidence import ( - aggregate_verification_evidence, load_optional_json_artifact, validate_artifact_id, ) @@ -33,6 +35,17 @@ REPO_RELATIVE_PATH_RE = re.compile(r"^[A-Za-z0-9._/-]+$") +@dataclass(frozen=True) +class BenchmarkExecutionRequest: + """Inputs required to execute one benchmark split.""" + + benchmark: dict[str, Any] + tasks: list[dict[str, Any]] + split: str + output_dir: pathlib.Path + checkpoint_mode: str + + def resolve_repo_child_path(path_str: Any, base: pathlib.Path, label: str) -> pathlib.Path: """Reject paths that could make a benchmark read or write outside its allowed root.""" if not isinstance(path_str, str) or not path_str: @@ -82,146 +95,6 @@ def validate_benchmark_inputs(benchmark: dict[str, Any], task_bundle: dict[str, _validate_task_entries(task_bundle) -def _count_passes(task_results: list[dict[str, Any]]) -> int: - return sum(1 for result in task_results if result["judge"]["verdict"] == "pass") - - -def _count_judge_successes(task_results: list[dict[str, Any]], field: str) -> int: - return sum(1 for result in task_results if result["judge"][field]) - - -def _count_checkpoint_tasks(task_results: list[dict[str, Any]]) -> int: - return sum(1 for result in task_results if result["checkpoint_paths"]) - - -def _count_approved_checkpoint_tasks( - task_results: list[dict[str, Any]], -) -> int: - return sum( - 1 - for result in task_results - if result["checkpoint_paths"] and result["judge"]["checkpoint_ok"] - ) - - -def aggregate_results(task_results: list[dict[str, Any]]) -> dict[str, float]: - total = len(task_results) - return { - "success_rate": metric_ratio(_count_passes(task_results), total), - "route_accuracy": metric_ratio(_count_judge_successes(task_results, "route_ok"), total), - "artifact_completeness": metric_ratio( - _count_judge_successes(task_results, "artifacts_ok"), total - ), - "checkpoint_compliance": metric_ratio( - _count_approved_checkpoint_tasks(task_results), - _count_checkpoint_tasks(task_results), - ), - } - - -def _regression_issues( - metrics: dict[str, float], - baseline_metrics: dict[str, Any], - minimum_metrics: dict[str, Any], - max_negative_delta: float, -) -> list[str]: - issues: list[str] = [] - for metric, value in metrics.items(): - baseline_value = float(baseline_metrics.get(metric, value)) - if value + max_negative_delta < baseline_value: - issues.append(f"{metric} regressed below baseline") - if metric in minimum_metrics and value < float(minimum_metrics[metric]): - issues.append(f"{metric} below minimum") - return issues - - -def write_regression_report( - benchmark: dict[str, Any], - split: str, - aggregate_metrics: dict[str, float], - output_dir: pathlib.Path, - run_id: str, -) -> pathlib.Path: - regression_policy = benchmark.get("regression_policy", {}) - baselines = regression_policy.get("baseline_results", {}) - baseline_value = baselines.get(split, "") - baseline_path = ( - resolve_repo_child_path(baseline_value, RESULTS_ROOT, "baseline result path") - if baseline_value - else None - ) - if baseline_path is not None and baseline_path.exists(): - baseline = load_json(baseline_path) - baseline_metrics = baseline.get("aggregate_metrics", {}) - else: - baseline = {} - baseline_metrics = {} - max_negative_delta = float(regression_policy.get("max_negative_delta", 0.0)) - minimum_metrics = regression_policy.get("minimum_metrics", {}) - regressions = _regression_issues( - aggregate_metrics, baseline_metrics, minimum_metrics, max_negative_delta - ) - - report = { - "report_id": f"regression-{run_id}", - "benchmark_id": benchmark["benchmark_id"], - "split": split, - "generated_at": iso_timestamp(), - "status": "pass" if not regressions else "fail", - "baseline_result_path": repo_relpath(baseline_path) - if baseline_path is not None and baseline_path.exists() - else "", - "aggregate_metrics": aggregate_metrics, - "baseline_metrics": baseline_metrics, - "issues": regressions, - } - output_path = output_dir / f"regression-{benchmark['benchmark_id']}-{split}-{run_id}.json" - dump_json(output_path, report) - return output_path - - -def write_result_ledger( - benchmark: dict[str, Any], - run_id: str, - split: str, - task_results: list[dict[str, Any]], - aggregate_metrics: dict[str, float], - result_path: pathlib.Path, - ledger_path: pathlib.Path, -) -> None: - aggregate_entry = { - "entry_id": f"{run_id}-aggregate", - "kind": "benchmark-run", - "timestamp": iso_timestamp(), - "benchmark_id": benchmark["benchmark_id"], - "run_id": run_id, - "split": split, - "result_path": repo_relpath(result_path), - "claim_links": benchmark.get("claim_links", []), - "aggregate_metrics": aggregate_metrics, - } - append_jsonl(ledger_path, aggregate_entry) - for result in task_results: - append_jsonl( - ledger_path, - { - "entry_id": f"{run_id}-{result['task_id']}", - "kind": "task-result", - "timestamp": iso_timestamp(), - "benchmark_id": benchmark["benchmark_id"], - "run_id": run_id, - "task_id": result["task_id"], - "split": result["split"], - "routed_runtime": result["routed_runtime"], - "trace_paths": result["trace_paths"], - "artifact_paths": result["artifact_paths"], - "checkpoint_paths": result["checkpoint_paths"], - "claim_links": result["claim_links"], - "judge_verdict": result["judge"]["verdict"], - }, - ) - - def _parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( description="Run a benchmark split through the umbrella harness." @@ -259,23 +132,24 @@ def _load_run_context( return benchmark_card_path, benchmark, tasks, output_dir -def _execute_benchmark( - benchmark: dict[str, Any], - tasks: list[dict[str, Any]], - split: str, - output_dir: pathlib.Path, - checkpoint_mode: str, -) -> tuple[str, list[dict[str, Any]], dict[str, float], pathlib.Path]: - run_id = new_run_id(f"{benchmark['benchmark_id']}-{split}") +def _execute_benchmark(request: BenchmarkExecutionRequest) -> BenchmarkRun: + run_id = new_run_id(f"{request.benchmark['benchmark_id']}-{request.split}") task_results = [ - execute_task(task, output_dir, run_id, checkpoint_mode, benchmark) for task in tasks + execute_task( + task, + request.output_dir, + run_id, + request.checkpoint_mode, + request.benchmark, + ) + for task in request.tasks ] aggregate_metrics = aggregate_results(task_results) result = { "run_id": run_id, - "benchmark_id": benchmark["benchmark_id"], - "benchmark_version": benchmark["version"], - "split": split, + "benchmark_id": request.benchmark["benchmark_id"], + "benchmark_version": request.benchmark["version"], + "split": request.split, "executed_at": iso_timestamp(), "task_count": len(task_results), "pass_count": sum(1 for result in task_results if result["judge"]["verdict"] == "pass"), @@ -283,21 +157,31 @@ def _execute_benchmark( "aggregate_metrics": aggregate_metrics, "task_results": task_results, } - result_path = output_dir / f"result-{benchmark['benchmark_id']}-{split}-{run_id}.json" + result_path = request.output_dir / ( + f"result-{request.benchmark['benchmark_id']}-{request.split}-{run_id}.json" + ) dump_json(result_path, result) - return run_id, task_results, aggregate_metrics, result_path + return BenchmarkRun( + benchmark=request.benchmark, + split=request.split, + run_id=run_id, + task_results=task_results, + metrics=aggregate_metrics, + result_path=result_path, + output_dir=request.output_dir, + ) -def _run_calibration( - benchmark: dict[str, Any], output_dir: pathlib.Path, run_id: str -) -> tuple[pathlib.Path, dict[str, Any]]: - calibration_path = output_dir / f"judge-calibration-{benchmark['benchmark_id']}-{run_id}.json" +def _run_calibration(run: BenchmarkRun) -> tuple[pathlib.Path, dict[str, Any]]: + calibration_path = run.output_dir / ( + f"judge-calibration-{run.benchmark['benchmark_id']}-{run.run_id}.json" + ) result = run_command( [ sys.executable, str(ROOT / "evals/scripts/judge_calibration.py"), "--judge-config", - str(ROOT / benchmark["judge_path"]), + str(ROOT / run.benchmark["judge_path"]), "--output", str(calibration_path), ] @@ -305,84 +189,9 @@ def _run_calibration( return calibration_path, result -def _unique_result_paths(task_results: list[dict[str, Any]], field: str) -> list[str]: - paths: set[str] = set() - for result in task_results: - paths.update(result[field]) - return sorted(paths) - - -def _all_result_paths(task_results: list[dict[str, Any]], field: str) -> list[str]: - paths: list[str] = [] - for result in task_results: - paths.extend(result[field]) - return paths - - -def _all_tasks_passed(task_results: list[dict[str, Any]]) -> bool: - return all(result["judge"]["verdict"] == "pass" for result in task_results) - - -def _run_card_aggregate_fields( - task_results: list[dict[str, Any]], -) -> dict[str, Any]: - duration = round( - sum(result["command_result"]["duration_seconds"] for result in task_results), 4 - ) - return { - "status": "pass" if _all_tasks_passed(task_results) else "fail", - "trace_paths": _unique_result_paths(task_results, "trace_paths"), - "artifact_paths": _unique_result_paths(task_results, "artifact_paths"), - "checkpoint_paths": _all_result_paths(task_results, "checkpoint_paths"), - "latency_seconds": duration, - } - - -def _build_run_card( - benchmark: dict[str, Any], - split: str, - task_results: list[dict[str, Any]], - run_id: str, - result_path: pathlib.Path, - ledger_path: pathlib.Path, - regression_path: pathlib.Path, - calibration_path: pathlib.Path, -) -> dict[str, Any]: - aggregate_fields = _run_card_aggregate_fields(task_results) - return { - "run_id": run_id, - "evidence_type": "benchmark-run", - "benchmark_id": benchmark["benchmark_id"], - "benchmark_version": benchmark["version"], - "date": today_iso(), - "split": split, - "system": default_system_metadata("umbrella-benchmark-runner"), - "judge_version": benchmark["judge_version"], - "command": "python3 evals/scripts/run_benchmark.py", - "result_path": repo_relpath(result_path), - "status": aggregate_fields["status"], - "task_spec_path": benchmark["task_specs_path"], - "routed_runtime": "mixed", - "router": {"version": ROUTER_VERSION, "decision_mode": "per-task"}, - "trace_paths": aggregate_fields["trace_paths"], - "artifact_paths": aggregate_fields["artifact_paths"], - "checkpoint_paths": aggregate_fields["checkpoint_paths"], - "verification_evidence": aggregate_verification_evidence(task_results), - "claim_links": benchmark.get("claim_links", []), - "ledger_path": repo_relpath(ledger_path), - "regression_report_path": repo_relpath(regression_path), - "judge_calibration_report_path": repo_relpath(calibration_path), - "cost_usd": 0.0, - "latency_seconds": aggregate_fields["latency_seconds"], - "notes": f"Executed {len(task_results)} task(s) for split {split}.", - } - - def _run_release_gate( benchmark_card_path: pathlib.Path, - run_card_path: pathlib.Path, - regression_path: pathlib.Path, - ledger_path: pathlib.Path, + artifacts: BenchmarkArtifacts, output_path: pathlib.Path, ) -> dict[str, Any]: return run_command( @@ -392,11 +201,11 @@ def _run_release_gate( "--benchmark-card", str(benchmark_card_path), "--run-card", - str(run_card_path), + str(artifacts.run_card_path), "--regression-report", - str(regression_path), + str(artifacts.regression_path), "--ledger", - str(ledger_path), + str(artifacts.ledger_path), "--output", str(output_path), ] @@ -426,84 +235,33 @@ def _failure_report_message(report_path: pathlib.Path | None) -> str: return "\n".join(str(issue) for issue in issues) -def _write_run_artifacts( - benchmark: dict[str, Any], - split: str, - task_results: list[dict[str, Any]], - run_id: str, - metrics: dict[str, float], - result_path: pathlib.Path, - output_dir: pathlib.Path, - calibration_path: pathlib.Path, -) -> tuple[pathlib.Path, pathlib.Path, pathlib.Path]: - regression_path = write_regression_report(benchmark, split, metrics, output_dir, run_id) - ledger_path = output_dir / "result-ledger.jsonl" - write_result_ledger(benchmark, run_id, split, task_results, metrics, result_path, ledger_path) - run_card = _build_run_card( - benchmark, - split, - task_results, - run_id, - result_path, - ledger_path, - regression_path, - calibration_path, - ) - run_card_path = output_dir / f"run-card-{benchmark['benchmark_id']}-{split}-{run_id}.json" - dump_json(run_card_path, run_card) - return run_card_path, regression_path, ledger_path - - -def _finish_release_gate( - benchmark_path: pathlib.Path, - benchmark: dict[str, Any], - split: str, - run_id: str, - output_dir: pathlib.Path, - run_card_path: pathlib.Path, - regression_path: pathlib.Path, - ledger_path: pathlib.Path, -) -> int: - output = output_dir / f"release-gate-{benchmark['benchmark_id']}-{split}-{run_id}.json" - result = _run_release_gate(benchmark_path, run_card_path, regression_path, ledger_path, output) - if result["returncode"] != 0: - report_message = _failure_report_message(output) - if report_message: - print(report_message, file=sys.stderr) - return result["returncode"] - return _report_child_failure(result, fallback="release gate failed", report_path=output) - print(repo_relpath(run_card_path)) - return 0 - - def main() -> int: args = _parse_args() benchmark_path, benchmark, tasks, output_dir = _load_run_context(args) - run_id, task_results, metrics, result_path = _execute_benchmark( - benchmark, tasks, args.split, output_dir, args.checkpoint_mode + run = _execute_benchmark( + BenchmarkExecutionRequest( + benchmark=benchmark, + tasks=tasks, + split=args.split, + output_dir=output_dir, + checkpoint_mode=args.checkpoint_mode, + ) ) - calibration_path, calibration_result = _run_calibration(benchmark, output_dir, run_id) + calibration_path, calibration_result = _run_calibration(run) if calibration_result["returncode"] != 0 or not calibration_path.exists(): return _report_child_failure(calibration_result, fallback="judge calibration failed") - run_card_path, regression_path, ledger_path = _write_run_artifacts( - benchmark, - args.split, - task_results, - run_id, - metrics, - result_path, - output_dir, + artifacts = _write_run_artifacts( + run, calibration_path, + resolve_repo_child_path=resolve_repo_child_path, ) return _finish_release_gate( benchmark_path, - benchmark, - args.split, - run_id, - output_dir, - run_card_path, - regression_path, - ledger_path, + run, + artifacts, + run_release_gate=_run_release_gate, + failure_report_message=_failure_report_message, + report_child_failure=_report_child_failure, ) diff --git a/evals/tests/test_benchmark_contracts_core_a.py b/evals/tests/test_benchmark_contracts_core_a.py index dbc1505..4baaa6d 100644 --- a/evals/tests/test_benchmark_contracts_core_a.py +++ b/evals/tests/test_benchmark_contracts_core_a.py @@ -3,7 +3,6 @@ import json import os import pathlib -import shutil import subprocess import sys import tempfile @@ -20,16 +19,6 @@ ) -def _trusted_bash() -> str: - bash = shutil.which("bash") - if bash is None: - raise RuntimeError("bash is required for the doctor contract test") - resolved = pathlib.Path(bash).resolve() - if not resolved.is_absolute() or not resolved.is_file(): - raise RuntimeError("resolved bash executable is not a regular absolute path") - return str(resolved) - - def test_run_benchmark_rejects_output_dir_outside_results_root() -> None: with tempfile.TemporaryDirectory(prefix="rae-benchmark-outside-") as tmp: output_dir = pathlib.Path(tmp) @@ -209,7 +198,7 @@ def test_rae_doctor_reports_missing_rg_dependency() -> None: install_path_mirror(bin_dir) # B603 rationale: fixed Bash executable and repository test entrypoint. completed = subprocess.run( # nosec B603 - [_trusted_bash(), str(ROOT / "scripts/rae.sh"), "doctor"], + ["bash", str(ROOT / "scripts/rae.sh"), "doctor"], cwd=ROOT, text=True, capture_output=True, @@ -226,7 +215,7 @@ def test_rae_doctor_reports_missing_rg_dependency() -> None: install_path_mirror(bin_dir, exclude={"rg"}) # B603 rationale: fixed Bash executable and repository test entrypoint. completed = subprocess.run( # nosec B603 - [_trusted_bash(), str(ROOT / "scripts/rae.sh"), "doctor"], + ["bash", str(ROOT / "scripts/rae.sh"), "doctor"], cwd=ROOT, text=True, capture_output=True, @@ -242,7 +231,7 @@ def test_rae_doctor_reports_missing_rg_dependency() -> None: def test_rae_worktree_help_lists_supervision_commands() -> None: # B603 rationale: fixed Bash executable and repository test entrypoint. completed = subprocess.run( # nosec B603 - [_trusted_bash(), str(ROOT / "scripts/rae.sh"), "worktree", "help"], + ["bash", str(ROOT / "scripts/rae.sh"), "worktree", "help"], cwd=ROOT, text=True, capture_output=True, diff --git a/evals/tests/test_benchmark_contracts_core_b.py b/evals/tests/test_benchmark_contracts_core_b.py index 9fbf2cd..d16ea67 100644 --- a/evals/tests/test_benchmark_contracts_core_b.py +++ b/evals/tests/test_benchmark_contracts_core_b.py @@ -30,36 +30,6 @@ def _remove_gate_report_path(run_card_path: pathlib.Path) -> None: write_json(run_card_path, payload) -def _stale_required_split_benchmark() -> tuple[dict, pathlib.Path]: - benchmark_path = ROOT / "evals/benchmarks/tool-selection-core.benchmark-card.json" - benchmark = json.loads(benchmark_path.read_text(encoding="utf-8")) - benchmark = { - **benchmark, - "benchmark_id": "tool-selection-core-stale-required-split", - "version": "1.0.2", - } - path = RESULTS_ROOT / ".tmp-tool-selection-core-stale-required-split.benchmark-card.json" - write_json(path, benchmark) - return benchmark, path - - -def _write_stale_dev_fixture(output_dir: pathlib.Path, benchmark: dict) -> None: - _, _, _, _, run_card_path = write_release_gate_fixture( - output_dir / "dev-stale", - split="dev", - run_id="tool-selection-core-stale-required-split-dev", - benchmark=benchmark, - calibration_payload={ - "judge_id": "router", - "agreement_rate": 1.0, - "calibration_case_count": 4, - "status": "pass", - }, - release_gate_status="pass", - ) - _remove_gate_report_path(run_card_path) - - def test_validate_eval_metadata_discovers_generated_run_card_names() -> None: benchmark_path = ROOT / "evals/benchmarks/tool-selection-core.benchmark-card.json" benchmark = json.loads(benchmark_path.read_text(encoding="utf-8")) @@ -124,14 +94,38 @@ def test_release_gate_ignores_stale_passing_run_cards_for_required_split() -> No # The held-out gate checks for dev evidence (prior required split). A dev # run-card that declares release_gate_status: pass but is missing its gate # report file is stale and must be rejected. - benchmark, temp_benchmark_path = _stale_required_split_benchmark() + benchmark_path = ROOT / "evals/benchmarks/tool-selection-core.benchmark-card.json" + benchmark = json.loads(benchmark_path.read_text(encoding="utf-8")) + benchmark = { + **benchmark, + "benchmark_id": "tool-selection-core-stale-required-split", + "version": "1.0.2", + } + temp_benchmark_path = ( + RESULTS_ROOT / ".tmp-tool-selection-core-stale-required-split.benchmark-card.json" + ) + write_json(temp_benchmark_path, benchmark) with tempfile.TemporaryDirectory( dir=RESULTS_ROOT, prefix="release-gate-stale-required-split-" ) as tmp: output_dir = pathlib.Path(tmp) # Create the dev fixture then corrupt it by removing its gate report path. - _write_stale_dev_fixture(output_dir, benchmark) + dev_dir = output_dir / "dev-stale" + _, _, _, _, dev_run_card_path = write_release_gate_fixture( + dev_dir, + split="dev", + run_id="tool-selection-core-stale-required-split-dev", + benchmark=benchmark, + calibration_payload={ + "judge_id": "router", + "agreement_rate": 1.0, + "calibration_case_count": 4, + "status": "pass", + }, + release_gate_status="pass", + ) + _remove_gate_report_path(dev_run_card_path) # Create the held-out fixture; this is the run being gated. _, regression_path, ledger_path, _, run_card_path = write_release_gate_fixture( diff --git a/evals/tests/test_benchmark_contracts_core_d.py b/evals/tests/test_benchmark_contracts_core_d.py index f1b3aeb..3e2181f 100644 --- a/evals/tests/test_benchmark_contracts_core_d.py +++ b/evals/tests/test_benchmark_contracts_core_d.py @@ -120,6 +120,38 @@ def test_release_gate_fails_when_calibration_report_is_missing() -> None: assert "judge calibration report missing" in gate_report["issues"] +def test_release_gate_fails_when_result_ledger_is_malformed() -> None: + benchmark_path = ROOT / "evals/benchmarks/tool-selection-core.benchmark-card.json" + benchmark = json.loads(benchmark_path.read_text(encoding="utf-8")) + with tempfile.TemporaryDirectory(dir=RESULTS_ROOT, prefix="release-gate-bad-ledger-") as tmp: + output_dir = pathlib.Path(tmp) + _, regression_path, ledger_path, _, run_card_path = write_release_gate_fixture( + output_dir, + split="dev", + run_id="tool-selection-core-dev-malformed-ledger", + benchmark=benchmark, + calibration_payload=calibration_payload(), + ) + ledger_path.write_text("{not-json}\n", encoding="utf-8") + write_release_gate_fixture( + output_dir / "held-out-pass", + split="held-out", + run_id="tool-selection-core-held-out-malformed-ledger", + benchmark=benchmark, + calibration_payload=calibration_payload(), + release_gate_status="pass", + ) + gate_output_path = output_dir / "release-gate-tool-selection-core-dev-malformed-ledger.json" + + completed = run_release_gate( + benchmark_path, run_card_path, regression_path, ledger_path, gate_output_path + ) + + assert completed.returncode != 0 + gate_report = json.loads(gate_output_path.read_text(encoding="utf-8")) + assert "result ledger could not be read" in gate_report["issues"] + + def test_validate_eval_metadata_rejects_invalid_workflow_verb_in_task_bundle() -> None: with tempfile.TemporaryDirectory( dir=RESULTS_ROOT, prefix="validate-invalid-workflow-verb-" diff --git a/mkdocs.yml b/mkdocs.yml index 184f6ec..4c063a9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,7 +41,7 @@ exclude_docs: | nav: - Home: INDEX.md - Tutorials: - - Graph Engineering with Codex and RAE: tutorials/graph-engineering-with-codex.md + - Graph Engineering with RAE: tutorials/graph-engineering-with-rae.md - Autonomous Code Change: tutorials/autonomous-code-change.md - First Pipeline: tutorials/first-pipeline.md - First Ralph Run: tutorials/first-ralph-run.md @@ -53,6 +53,11 @@ nav: - Add a Tool: how-to/add-a-tool.md - Publish a Sanitized Profile: how-to/publish-a-sanitized-profile.md - Reproduce a Result: how-to/reproduce-a-result.md + - Use the Experimental Hosted API: how-to/hosted-api.md + - Run a Workflow 2.2 Wait: how-to/run-workflow-v2.2-wait.md + - Deploy the Experimental Platform: how-to/deploy-experimental-platform.md + - Recover a Workflow 2.2 Wait: how-to/recover-workflow-v2.2-wait.md + - Test the Experimental Hosted Platform: how-to/test-experimental-hosted-platform.md - Reference: - Workflow Rubric: reference/workflow-rubric.md - Repo Map: reference/repo-map.md @@ -60,14 +65,17 @@ nav: - Architecture: - System Overview: reference/architecture/system-overview.md - Module Boundaries: reference/architecture/module-boundaries.md + - Experimental Hosted Platform: reference/architecture/experimental-hosted-platform.md - Contracts: - Artifact Schemas: reference/contracts/artifact-schemas.md + - Execution Profile 3.0: reference/contracts/execution-profile-v3.md - Task Specs: reference/contracts/task-specs.md - Human Checkpoints: reference/contracts/human-checkpoints.md - Result Ledger: reference/contracts/result-ledger.md - Quality Gates: reference/contracts/quality-gates.md - Report Types: reference/contracts/report-types.md - Local Graph and Memory: reference/contracts/graph-memory.md + - Workflow 2.2: reference/contracts/workflow-v2.2.md - CLI: - Umbrella CLI: reference/cli/umbrella.md - Orchestration CLI: reference/cli/orchestration.md diff --git a/packages/loops/ralph/scripts/ralph_fs_txn.py b/packages/loops/ralph/scripts/ralph_fs_txn.py index 8063fbf..aa8e8de 100644 --- a/packages/loops/ralph/scripts/ralph_fs_txn.py +++ b/packages/loops/ralph/scripts/ralph_fs_txn.py @@ -882,10 +882,26 @@ def prepared_change_unit( if encoded in changed: return True before, after = baseline.get(encoded), prepared.get(encoded) - if before is None or after is None or before["kind"] != "dir" or after["kind"] != "dir": + if not prepared_directory_pair(before, after): return False - rel = decode_path(encoded) - return any(decode_path(item) != rel and path_under(decode_path(item), rel) for item in changed) + return changed_descendant(encoded, changed) + + +def prepared_directory_pair(before: dict[str, Any] | None, after: dict[str, Any] | None) -> bool: + if before is None or after is None: + return False + return before["kind"] == "dir" and after["kind"] == "dir" + + +def changed_descendant(encoded: str, changed: list[str]) -> bool: + relative = decode_path(encoded) + for item in changed: + child = decode_path(item) + if child == relative: + continue + if path_under(child, relative): + return True + return False def remove_tree(path: str | bytes | Path) -> None: @@ -983,53 +999,6 @@ def pointer_path_command(args: argparse.Namespace) -> int: return 0 -def _create_mirror_journal( - root: bytes, - runtime: bytes, - metadata_root: bytes, - transaction_dir: Path, - provider_directory: Path, -) -> tuple[Path, dict[str, Any]]: - baseline_path = transaction_dir / "baseline" - baseline_path.mkdir(mode=0o700) - mirror_path = provider_directory / WORKSPACE_NAME - mirror_path.mkdir(mode=0o700) - mirror = os.fsencode(mirror_path) - baseline_store = os.fsencode(baseline_path) - manifest = make_manifest(root, runtime) - copy_manifest(root, baseline_store, manifest) - copy_manifest(root, mirror, manifest) - transaction_id = str(uuid.uuid4()) - quarantine_root = os.path.join(runtime, b".fixing-quarantine", os.fsencode(transaction_id)) - journal = { - "format": FORMAT_VERSION, - "id": transaction_id, - "state": "mirrored", - "root": os.fsdecode(root), - "runtime": os.fsdecode(runtime), - "metadata_root": os.fsdecode(metadata_root), - "mirror": os.fsdecode(mirror), - "baseline_store": os.fsdecode(baseline_store), - "quarantine_root": os.fsdecode(quarantine_root), - "root_identity": identity(os.lstat(root)), - "runtime_identity": identity(os.lstat(runtime)), - "metadata_root_identity": identity(os.lstat(metadata_root)), - "mirror_identity": identity(os.lstat(mirror)), - "provider_directory_identity": identity(os.lstat(provider_directory)), - "baseline_store_identity": identity(os.lstat(baseline_store)), - "baseline": manifest, - "prepared": None, - "changed": [], - "promoted": [], - "active": None, - "active_started": False, - "evidence": [], - } - journal_path = transaction_dir / "journal.json" - json_dump_atomic(journal_path, journal) - return journal_path, journal - - def mirror_command(args: argparse.Namespace) -> int: root, runtime, metadata_root, pointer = caller_identity(args) if os.path.lexists(pointer): @@ -1043,12 +1012,46 @@ def mirror_command(args: argparse.Namespace) -> int: os.path.realpath(tempfile.mkdtemp(prefix="txn-", dir=transaction_parent)) ) provider_directory = Path(os.path.realpath(tempfile.mkdtemp(prefix=PROVIDER_DIRECTORY_PREFIX))) + mirror_path = provider_directory / WORKSPACE_NAME + mirror_path.mkdir(mode=0o700) os.chmod(transaction_dir, 0o700) # nosemgrep os.chmod(provider_directory, 0o700) # nosemgrep try: - journal_path, journal = _create_mirror_journal( - root, runtime, metadata_root, transaction_dir, provider_directory - ) + baseline_path = transaction_dir / "baseline" + baseline_path.mkdir(mode=0o700) + mirror = os.fsencode(mirror_path) + baseline_store = os.fsencode(baseline_path) + manifest = make_manifest(root, runtime) + copy_manifest(root, baseline_store, manifest) + copy_manifest(root, mirror, manifest) + transaction_id = str(uuid.uuid4()) + quarantine_root = os.path.join(runtime, b".fixing-quarantine", os.fsencode(transaction_id)) + journal = { + "format": FORMAT_VERSION, + "id": transaction_id, + "state": "mirrored", + "root": os.fsdecode(root), + "runtime": os.fsdecode(runtime), + "metadata_root": os.fsdecode(metadata_root), + "mirror": os.fsdecode(mirror), + "baseline_store": os.fsdecode(baseline_store), + "quarantine_root": os.fsdecode(quarantine_root), + "root_identity": identity(os.lstat(root)), + "runtime_identity": identity(os.lstat(runtime)), + "metadata_root_identity": identity(os.lstat(metadata_root)), + "mirror_identity": identity(os.lstat(mirror)), + "provider_directory_identity": identity(os.lstat(provider_directory)), + "baseline_store_identity": identity(os.lstat(baseline_store)), + "baseline": manifest, + "prepared": None, + "changed": [], + "promoted": [], + "active": None, + "active_started": False, + "evidence": [], + } + journal_path = transaction_dir / "journal.json" + json_dump_atomic(journal_path, journal) json_dump_atomic( pointer, { @@ -1292,20 +1295,6 @@ def conflict_message(encoded: str, context: str) -> str: return f"live checkout changed during {context}: {os.fsdecode(decode_path(encoded))}" -def _quarantine_matches_expected( - quarantine: bytes, - encoded: str, - expected: dict[str, Any], - expected_subtree: list[dict[str, Any]] | None, -) -> bool: - if entry_at_absolute(quarantine, encoded) != expected: - return False - return ( - expected_subtree is None - or subtree_manifest_absolute(quarantine, encoded) == expected_subtree - ) - - def quarantine_live_entry( path: Path, journal: dict[str, Any], @@ -1342,7 +1331,12 @@ def quarantine_live_entry( raise TransactionConflict(conflict_message(encoded, context)) from error item["state"] = "quarantined" write_evidence(path, journal) - if _quarantine_matches_expected(quarantine, encoded, expected, expected_subtree): + entry_matches = entry_at_absolute(quarantine, encoded) == expected + subtree_matches = ( + expected_subtree is None + or subtree_manifest_absolute(quarantine, encoded) == expected_subtree + ) + if entry_matches and subtree_matches: return item try: @@ -2003,9 +1997,15 @@ def quarantined_directory( return None -def _recovery_candidate( - journal: dict[str, Any], entry: dict[str, Any], subtree: bool -) -> bytes | None: +def restore_baseline_entry( + path: Path, + journal: dict[str, Any], + root: bytes, + baseline_store: bytes, + entry: dict[str, Any], + context: str, + subtree: bool = False, +) -> None: candidate = quarantined_baseline(journal, entry["path"], entry) if candidate is None and entry["kind"] == "dir" and not subtree: candidate = quarantined_directory(journal, entry["path"]) @@ -2015,43 +2015,38 @@ def _recovery_candidate( raise TransactionConflict( conflict_message(entry["path"], "recovery directory staging") ) - return candidate - - -def _stage_recovery_entry( - path: Path, - journal: dict[str, Any], - root: bytes, - baseline_store: bytes, - entry: dict[str, Any], - context: str, - subtree: bool, - candidate: bytes | None, -) -> dict[str, Any]: - if candidate is not None: - return stage_existing_entry( - path, journal, entry["path"], candidate, context, "recovery", subtree - ) - if subtree and entry["kind"] == "dir": - return stage_directory_subtree( - path, journal, root, baseline_store, journal["baseline"], entry, context, "recovery" + if candidate is None: + if subtree and entry["kind"] == "dir": + item = stage_directory_subtree( + path, + journal, + root, + baseline_store, + journal["baseline"], + entry, + context, + "recovery", + ) + else: + item = stage_manifest_entry( + path, + journal, + root, + baseline_store, + entry, + context, + "recovery", + ) + else: + item = stage_existing_entry( + path, + journal, + entry["path"], + candidate, + context, + "recovery", + subtree, ) - return stage_manifest_entry(path, journal, root, baseline_store, entry, context, "recovery") - - -def restore_baseline_entry( - path: Path, - journal: dict[str, Any], - root: bytes, - baseline_store: bytes, - entry: dict[str, Any], - context: str, - subtree: bool = False, -) -> None: - candidate = _recovery_candidate(journal, entry, subtree) - item = _stage_recovery_entry( - path, journal, root, baseline_store, entry, context, subtree, candidate - ) expected_subtree = manifest_subtree(journal["baseline"], entry["path"]) if subtree else None install_staged_entry( path, @@ -2203,11 +2198,7 @@ def restore_directory( "recovery directory restoration", subtree, ) - elif ( - subtree - and current["kind"] == "dir" - and subtree_manifest(root, encoded) == manifest_subtree(journal["prepared"] or [], encoded) - ): + elif current_matches_prepared_subtree(journal, root, encoded, current, subtree): replace_recovery_subtree(path, journal, root, context["baseline_store"], entry, current) elif current["kind"] == "dir" and current in directory_prepared_entries( context["prepared"], encoded @@ -2219,6 +2210,18 @@ def restore_directory( raise TransactionConflict(conflict_message(encoded, "recovery directory restoration")) +def current_matches_prepared_subtree( + journal: dict[str, Any], + root: bytes, + encoded: str, + current: dict[str, Any], + subtree: bool, +) -> bool: + if not subtree or current["kind"] != "dir": + return False + return subtree_manifest(root, encoded) == manifest_subtree(journal["prepared"] or [], encoded) + + def replace_recovery_subtree( path: Path, journal: dict[str, Any], @@ -2359,20 +2362,23 @@ def discard_command(args: argparse.Namespace) -> int: return 0 -def _cleanup_missing_terminal_journal( - pointer: Path, journal_path: Path, data: dict[str, Any] -) -> None: - if data.get("terminal") not in ("committed", "recovered", "discarded"): - raise RuntimeError("nonterminal transaction journal is missing") - if os.path.lexists(journal_path.parent): - owned_private_directory(journal_path.parent, "transaction directory") - remove_tree(journal_path.parent) - unlink_pointer_durable(pointer) - - -def _recover_loaded_journal(args: argparse.Namespace, root: bytes) -> None: +def recover_command(args: argparse.Namespace) -> int: + root, runtime, metadata_root, pointer = caller_identity(args) + if not os.path.lexists(pointer): + return 0 + data = pointer_data(pointer) + journal_path = pointer_journal_path(data, root, runtime, metadata_root) + if not os.path.lexists(journal_path): + if data.get("terminal") not in ("committed", "recovered", "discarded"): + raise RuntimeError("nonterminal transaction journal is missing") + if os.path.lexists(journal_path.parent): + owned_private_directory(journal_path.parent, "transaction directory") + remove_tree(journal_path.parent) + unlink_pointer_durable(pointer) + return 0 path, journal, bound_root, _, mirror, bound_pointer = load_bound_journal( - args, allow_terminal_cleanup=True + args, + allow_terminal_cleanup=True, ) if bound_root != root: raise RuntimeError("transaction recovery root mismatch") @@ -2392,18 +2398,6 @@ def _recover_loaded_journal(args: argparse.Namespace, root: bytes) -> None: journal["evidence"], cleanup_state, ) - - -def recover_command(args: argparse.Namespace) -> int: - root, runtime, metadata_root, pointer = caller_identity(args) - if not os.path.lexists(pointer): - return 0 - data = pointer_data(pointer) - journal_path = pointer_journal_path(data, root, runtime, metadata_root) - if not os.path.lexists(journal_path): - _cleanup_missing_terminal_journal(pointer, journal_path, data) - return 0 - _recover_loaded_journal(args, root) return 0 diff --git a/packages/orchestration/README.md b/packages/orchestration/README.md index 6ec8290..7d77a11 100644 --- a/packages/orchestration/README.md +++ b/packages/orchestration/README.md @@ -50,7 +50,9 @@ report. - Node.js `>=20.19.0 <21`, `>=22.12.0 <23`, or `>=24.0.0` - npm - `git` and `rg` -- Codex CLI for provider-backed autonomous runs +- Codex CLI for Codex-backed autonomous runs +- OpenCode CLI for explicit OpenCode routes on the supported macOS containment + backend - a target Git repository with at least one commit and usable `HEAD` and current-branch reflogs @@ -97,8 +99,9 @@ Useful options: - `--workflow ` selects a validated graph-native workflow for a new run - `--execution-profile ` snapshots an operator-owned mapping from logical - economy, standard, and judgment tiers to Codex model settings; it is mutually - exclusive with global model and reasoning overrides + economy, standard, and judgment tiers to named Codex or OpenCode routes; it is + mutually exclusive with global provider, model, reasoning, and variant + overrides - `--through ` stops after the selected workflow node - `--max-concurrency <1..4>` caps concurrent read-only nodes - `--max-repair-rounds <1..5>` tightens the workflow repair bound @@ -133,7 +136,7 @@ It has no filesystem or network sandbox, always fails `agent doctor`, and requires `--allow-unsafe-command-provider` on every run and resume. Do not use it as an operational backend. -## Operator console +## Operator console and workflow designer From the repository root: @@ -144,10 +147,21 @@ From the repository root: Repeat `--project` to allowlist more than one Git root. The console binds only to loopback and prints a URL with an ephemeral bearer token in the fragment. +Repeat `--execution-profile` to preload server-owned profile files. The browser +receives only profile IDs, route metadata, models, and readiness. It never +receives profile paths, credentials, environment values, or raw provider +events. It exposes status, projected events, stop, interrupt, resume, checkpoint, and fail-closed cleanup controls. It does not expose in-place execution, arbitrary commands, environment overrides, Git publication, or deployment. +The workflow workspace keeps Loop, Graph, Analyze, and JSON views synchronized. +Five guided templates compile directly to workflow 2.1. Structured node and +edge controls remain keyboard operable, while the JSON view retains access to +existing 2.0 and experimental 2.2 revisions. Analysis and proposal results stay +unsaved until an operator creates a revision. Activation remains a separate, +exact-digest action. + See [`operator/README.md`](operator/README.md) for the HTTP and event contract. ## Local graph projections @@ -179,7 +193,15 @@ until-dry convergence, and logical execution tiers. Stored 2.0 runs and active 2.0 registry revisions keep their original executor. RAE does not migrate a private registry automatically. -Create a draft-only Codex proposal with: +Workflow 2.2 is an experimental local scheduler for durable wait nodes and +typed signals. It writes wait state under +`.pipeline/runs//workflow/wait-state.json`, consumes accepted signals +idempotently on resume, and fails a wait on timeout. Its bounded context +assembly is not a context-efficiency result. Existing 2.0 and 2.1 runs remain +on their original schedulers. See +[`docs/reference/contracts/workflow-v2.2.md`](../../docs/reference/contracts/workflow-v2.2.md). + +Create a proposal with: ```bash ./scripts/rae.sh graph workflow propose \ @@ -190,8 +212,10 @@ Create a draft-only Codex proposal with: --rationale "Draft for review" ``` -The proposal uses a read-only ephemeral session, validates locally, permits one -correction, and never activates or executes its output. +Without `--preview`, the command stores a validated draft revision. Add +`--preview` to return a validated candidate without saving it. When an +execution profile is supplied, proposal generation uses its `judgment` route. +Neither mode activates or executes the result. ## Low-level pipeline API @@ -250,9 +274,10 @@ python3 scripts/adapters/generate_adapters.py python3 scripts/adapters/generate_adapters.py --check ``` -Committed adapters exist for Codex, Cursor, Claude, Gemini, and Kilo. Only -Codex currently has a supported autonomous CLI executor. The other adapters -are portable guidance and must not be interpreted as executable integrations. +Committed guidance adapters exist for Codex, Cursor, Claude, Gemini, and Kilo. +The autonomous runtime has executable adapters for Codex and explicit OpenCode +routes. The other adapters are portable guidance and must not be interpreted +as executable integrations. ## Repository structure @@ -267,6 +292,7 @@ are portable guidance and must not be interpreted as executable integrations. | `orchestrators/` | Stage instructions consumed by the runtime | | `skills/dev-tools/` | Quality-gate, review, and trace packages | | `docs/` | Package runbook, platform notes, policy, and repository map | +| `platform/` | Experimental PostgreSQL control plane, OIDC API, fenced worker lease, artifact, and MCP source | ## Security and data handling @@ -288,6 +314,13 @@ The runner: roots during writable stages - rejects protected Git-state changes on supported provider runs +OpenCode write routes add a macOS Seatbelt boundary around the isolated +worktree. RAE verifies the effective OpenCode configuration before execution, +denies shell, web, external-directory, plugin, skill, subagent, question, and +unapproved MCP access, and exposes only an opaque allowlisted verification +broker. The pinned OpenCode process can read its configured credential store; +credential contents are not copied into run artifacts or operator responses. + The provider process still receives the working directory and schema paths needed for execution. Consult the provider's data controls for storage and retention behavior. @@ -331,6 +364,12 @@ Operational recovery and troubleshooting are documented in [`docs/RUNBOOK.md`](docs/RUNBOOK.md). Platform support is documented in [`docs/PLATFORMS.md`](docs/PLATFORMS.md). +The `platform/` package is an experimental vertical slice connected to the +operator only through remote-mode proxy routes. Its source-unit test can be +run with `npm --prefix platform test` from this directory. PostgreSQL, +container, OIDC, S3-compatible storage, and remote worker execution remain +integration evidence lanes. + ## Limitations - Worktree isolation depends on Git reflogs and repository identity checks. @@ -338,6 +377,10 @@ Operational recovery and troubleshooting are documented in creates a new POSIX session. - Guard recovery fails closed while ownership or repository identity is uncertain. +- OpenCode write routes are supported only on macOS, require the isolated + worktree, and reject `--in-place`. +- A real provider run is still required before treating fake-executable event + tests as evidence for a specific OpenCode release or provider account. - Deterministic fixtures and committed baselines are test evidence, not proof of behavior on arbitrary repositories. - The low-level stage runner validates pipeline contracts; it is not a diff --git a/packages/orchestration/contracts/workflows/execution-profile-v2.schema.json b/packages/orchestration/contracts/workflows/execution-profile-v2.schema.json new file mode 100644 index 0000000..02a8f0b --- /dev/null +++ b/packages/orchestration/contracts/workflows/execution-profile-v2.schema.json @@ -0,0 +1,92 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://rae.local/contracts/workflows/execution-profile-v2.schema.json", + "title": "RAE operator-owned Codex execution and capability profile", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "profile_id", + "tiers", + "capability_sets", + "default_capability_set", + "node_capability_sets" + ], + "properties": { + "schema_version": { "const": "2.0.0" }, + "profile_id": { "type": "string", "pattern": "^[a-z][a-z0-9-]{2,63}$" }, + "tiers": { + "type": "object", + "additionalProperties": false, + "required": ["economy", "standard", "judgment"], + "properties": { + "economy": { "$ref": "#/$defs/model_mapping" }, + "standard": { "$ref": "#/$defs/model_mapping" }, + "judgment": { "$ref": "#/$defs/model_mapping" } + } + }, + "capability_sets": { + "type": "object", + "minProperties": 1, + "propertyNames": { "$ref": "#/$defs/name" }, + "additionalProperties": { "$ref": "#/$defs/capability_set" } + }, + "default_capability_set": { "$ref": "#/$defs/name" }, + "node_capability_sets": { + "type": "object", + "minProperties": 1, + "propertyNames": { "$ref": "#/$defs/node_id" }, + "additionalProperties": { "$ref": "#/$defs/name" } + } + }, + "$defs": { + "name": { "type": "string", "pattern": "^[a-z][a-z0-9-]{1,63}$" }, + "node_id": { "type": "string", "pattern": "^[a-z][a-z0-9-]{0,63}$" }, + "model_mapping": { + "type": "object", + "additionalProperties": false, + "required": ["model", "reasoning_effort"], + "properties": { + "model": { "type": "string", "minLength": 1, "maxLength": 128 }, + "reasoning_effort": { "enum": ["low", "medium", "high", "xhigh"] } + } + }, + "capability_set": { + "type": "object", + "additionalProperties": false, + "required": ["web_search", "mcp_servers", "credential_env_vars"], + "properties": { + "web_search": { "const": "disabled" }, + "mcp_servers": { + "type": "array", + "maxItems": 16, + "items": { "$ref": "#/$defs/mcp_server" } + }, + "credential_env_vars": { + "type": "array", + "maxItems": 32, + "uniqueItems": true, + "items": { "type": "string", "pattern": "^[A-Z][A-Z0-9_]{1,127}$" } + } + } + }, + "mcp_server": { + "type": "object", + "additionalProperties": false, + "required": ["name", "transport", "url", "enabled_tools", "token_env_var"], + "properties": { + "name": { "$ref": "#/$defs/name" }, + "transport": { "const": "streamable-http" }, + "url": { "type": "string", "format": "uri", "pattern": "^https://" }, + "enabled_tools": { + "type": "array", + "minItems": 1, + "maxItems": 128, + "uniqueItems": true, + "items": { "type": "string", "minLength": 1, "maxLength": 128 } + }, + "token_env_var": { "type": "string", "pattern": "^[A-Z][A-Z0-9_]{1,127}$" } + } + } + } +} diff --git a/packages/orchestration/contracts/workflows/execution-profile-v3.schema.json b/packages/orchestration/contracts/workflows/execution-profile-v3.schema.json new file mode 100644 index 0000000..022ead6 --- /dev/null +++ b/packages/orchestration/contracts/workflows/execution-profile-v3.schema.json @@ -0,0 +1,69 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://rae.local/contracts/workflows/execution-profile-v3.schema.json", + "title": "RAE provider-neutral execution route profile", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "profile_id", "routes", "tiers"], + "properties": { + "schema_version": { "const": "3.0.0" }, + "profile_id": { "type": "string", "pattern": "^[a-z][a-z0-9-]{2,63}$" }, + "routes": { + "type": "object", + "minProperties": 1, + "maxProperties": 32, + "propertyNames": { "$ref": "#/$defs/route_id" }, + "additionalProperties": { "$ref": "#/$defs/route" } + }, + "tiers": { + "type": "object", + "additionalProperties": false, + "required": ["economy", "standard", "judgment"], + "properties": { + "economy": { "$ref": "#/$defs/route_id" }, + "standard": { "$ref": "#/$defs/route_id" }, + "judgment": { "$ref": "#/$defs/route_id" } + } + }, + "node_routes": { + "type": "object", + "maxProperties": 64, + "propertyNames": { "$ref": "#/$defs/node_id" }, + "additionalProperties": { "$ref": "#/$defs/route_id" } + } + }, + "$defs": { + "route_id": { "type": "string", "pattern": "^[a-z][a-z0-9-]{1,63}$" }, + "node_id": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "model": { + "type": "string", + "minLength": 3, + "maxLength": 160, + "pattern": "^[A-Za-z0-9][A-Za-z0-9._/-]*$" + }, + "route": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": ["executor", "model", "reasoning_effort"], + "properties": { + "executor": { "const": "codex" }, + "model": { "$ref": "#/$defs/model" }, + "reasoning_effort": { "enum": ["low", "medium", "high", "xhigh"] } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": ["executor", "model"], + "properties": { + "executor": { "const": "opencode" }, + "model": { "$ref": "#/$defs/model" }, + "variant": { "type": "string", "minLength": 1, "maxLength": 64, "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$" } + } + } + ] + } + } +} diff --git a/packages/orchestration/contracts/workflows/node-envelope-v2.2.schema.json b/packages/orchestration/contracts/workflows/node-envelope-v2.2.schema.json new file mode 100644 index 0000000..aff38fb --- /dev/null +++ b/packages/orchestration/contracts/workflows/node-envelope-v2.2.schema.json @@ -0,0 +1,42 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://rae.local/contracts/workflows/node-envelope-v2.2.schema.json", + "title": "Immutable workflow node result envelope v2.2", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "run_id", "workflow_digest", "node_id", "instance_id", "attempt", "status", "payload", "findings", "evidence_refs", "ownership", "changed_paths", "command_evidence", "resource_usage", "input_digest", "output_digest", "execution_tier", "context_manifest"], + "properties": { + "schema_version": { "const": "2.2.0" }, + "run_id": { "type": "string", "minLength": 1 }, + "workflow_digest": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "node_id": { "type": "string", "minLength": 1 }, + "instance_id": { "type": "string", "pattern": "^[a-zA-Z0-9._:-]{1,192}$" }, + "attempt": { "type": "integer", "minimum": 1, "maximum": 3 }, + "status": { "enum": ["passed", "failed", "blocked", "stopped", "skipped"] }, + "failure": { "type": ["object", "null"] }, + "payload": {}, + "findings": { "type": "array", "maxItems": 1024, "items": { "type": "object" } }, + "evidence_refs": { "type": "array", "maxItems": 1024, "items": { "type": "string" } }, + "ownership": { "type": "object" }, + "changed_paths": { "type": "array", "maxItems": 4096, "uniqueItems": true, "items": { "type": "string" } }, + "command_evidence": { "type": "array", "maxItems": 1024, "items": { "type": "object" } }, + "resource_usage": { "type": "object" }, + "input_digest": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "output_digest": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, + "execution_tier": { "enum": ["economy", "standard", "judgment", "runtime"] }, + "context_manifest": { + "type": "object", "additionalProperties": false, + "required": ["cap_bytes", "assembled_bytes", "artifact_refs", "inline_artifacts", "mandatory_budget_bytes", "optional_budget_bytes", "included", "omitted"], + "properties": { + "cap_bytes": { "type": "integer", "minimum": 16384, "maximum": 262144 }, + "assembled_bytes": { "type": "integer", "minimum": 0, "maximum": 262144 }, + "mandatory_budget_bytes": { "type": "integer", "minimum": 16384, "maximum": 262144 }, + "optional_budget_bytes": { "type": "integer", "minimum": 0, "maximum": 262144 }, + "included": { "type": "array", "items": { "type": "object" } }, + "omitted": { "type": "array", "items": { "type": "object" } }, + "artifact_refs": { "type": "array", "items": { "type": "object" } }, + "inline_artifacts": { "type": "array", "items": { "type": "object" } } + } + } + } +} diff --git a/packages/orchestration/contracts/workflows/workflow-v2.2.schema.json b/packages/orchestration/contracts/workflows/workflow-v2.2.schema.json new file mode 100644 index 0000000..6d7ba3f --- /dev/null +++ b/packages/orchestration/contracts/workflows/workflow-v2.2.schema.json @@ -0,0 +1,69 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://rae.local/contracts/workflows/workflow-v2.2.schema.json", + "title": "RAE graph-native workflow v2.2", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "workflow_id", "revision", "entry_node", "terminal_node", "nodes", "edges", "signal_contracts"], + "properties": { + "schema_version": { "const": "2.2.0" }, + "workflow_id": { "type": "string", "pattern": "^[a-z][a-z0-9-]{2,63}$" }, + "revision": { "type": "integer", "minimum": 1 }, + "title": { "type": "string", "minLength": 1, "maxLength": 160 }, + "entry_node": { "$ref": "#/$defs/nodeId" }, + "terminal_node": { "$ref": "#/$defs/nodeId" }, + "nodes": { "type": "array", "minItems": 2, "maxItems": 64, "items": { "$ref": "#/$defs/node" } }, + "edges": { "type": "array", "minItems": 1, "maxItems": 256, "items": { "$ref": "#/$defs/edge" } }, + "payload_contracts": { "type": "object", "maxProperties": 64, "additionalProperties": { "type": "object" } }, + "signal_contracts": { "type": "object", "minProperties": 1, "maxProperties": 32, "additionalProperties": { "type": "object" } }, + "budgets": { + "type": "object", "additionalProperties": false, + "properties": { + "max_concurrency": { "type": "integer", "minimum": 1, "maximum": 4 }, + "max_attempts_per_node": { "type": "integer", "minimum": 1, "maximum": 3 }, + "max_context_bytes": { "type": "integer", "minimum": 16384, "maximum": 262144 } + } + } + }, + "$defs": { + "nodeId": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "signalName": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "wait": { + "type": "object", "additionalProperties": false, "required": ["timeout_seconds", "signals", "signal_contract"], + "properties": { + "timeout_seconds": { "type": "integer", "minimum": 60, "maximum": 2592000 }, + "signals": { "type": "array", "minItems": 1, "maxItems": 32, "uniqueItems": true, "items": { "$ref": "#/$defs/signalName" } }, + "signal_contract": { "type": "string", "pattern": "^[A-Za-z][A-Za-z0-9._-]{0,63}$" } + } + }, + "node": { + "type": "object", "additionalProperties": false, "required": ["id", "kind", "access", "guidance"], + "properties": { + "id": { "$ref": "#/$defs/nodeId" }, + "kind": { "enum": ["agent", "join", "gate", "checkpoint", "wait", "terminal"] }, + "access": { "enum": ["read", "write", "control"] }, + "guidance": { "type": "string", "minLength": 1, "maxLength": 12000 }, + "role": { "type": "string", "maxLength": 128 }, + "payload_contract": { "type": "string", "pattern": "^[A-Za-z][A-Za-z0-9._-]{0,63}$" }, + "tier": { "enum": ["economy", "standard", "judgment"] }, + "join": { "enum": ["all", "any"] }, + "resource": { "type": "string", "maxLength": 128 }, + "ownership_plan": { "type": "boolean" }, + "mutation_checkpoint": { "type": "boolean" }, + "verification": { "type": "boolean" }, + "context": { "type": "object", "additionalProperties": false, "properties": { "include_operational_evidence": { "type": "boolean" } } }, + "wait": { "$ref": "#/$defs/wait" } + } + }, + "edge": { + "type": "object", "additionalProperties": false, "required": ["from", "to", "type"], + "properties": { + "from": { "$ref": "#/$defs/nodeId" }, + "to": { "$ref": "#/$defs/nodeId" }, + "type": { "enum": ["sequence", "artifact", "condition"] }, + "artifact": { "type": "string", "maxLength": 128 }, + "condition": { "enum": ["success", "failure", "blocking-findings", "budget-available"] } + } + } + } +} diff --git a/packages/orchestration/docs/ROADMAP.md b/packages/orchestration/docs/ROADMAP.md index a923c43..89d0409 100644 --- a/packages/orchestration/docs/ROADMAP.md +++ b/packages/orchestration/docs/ROADMAP.md @@ -5,16 +5,32 @@ It is not a release schedule. ## Provider execution -- Codex is the only supported autonomous CLI integration. +- Codex remains the default autonomous CLI integration. +- OpenCode is supported only when selected explicitly. The current adapter + requires macOS Seatbelt containment, an isolated RAE worktree for writes, and + rejects `--in-place`. - Cursor, Claude, Gemini, and Kilo have synchronized guidance adapters but no autonomous executor. - The command provider remains an unsandboxed test interface and cannot pass operational diagnostics. +Remaining provider work includes a containment backend for other operating +systems, an authenticated OpenCode acceptance run against a real provider event +stream, and broader provider-version coverage. Direct OpenRouter API execution +is not implemented; OpenRouter models are available only through OpenCode +configuration. + Any additional executor must provide workspace isolation, structured artifact output, event streaming, fresh sessions, child-environment filtering, deadline handling, and protected Git-state enforcement before it can be supported. +## Workflow designer + +The loopback operator implements synchronized Loop, Graph, Analyze, and JSON +views for workflow 2.1. Remaining acceptance work is a rendered browser smoke +and one authenticated proposal-to-activation run. Workflow 2.0 and experimental +2.2 remain expert JSON surfaces rather than guided-editor targets. + ## Process containment The operator console uses POSIX process groups for interruption. It cannot prove diff --git a/packages/orchestration/operator/README.md b/packages/orchestration/operator/README.md index 67e82b0..c02c9c2 100644 --- a/packages/orchestration/operator/README.md +++ b/packages/orchestration/operator/README.md @@ -15,6 +15,16 @@ evidence from a real run: ![Evidence Dossier mobile](docs/screenshots/evidence-dossier-mobile.png) +Regenerate both captures from the current operator UI and the sanitized graph +fixture: + +```bash +node packages/orchestration/operator/scripts/capture-docs-screenshots.mjs +``` + +The capture script requires a local Chrome or Chromium installation. It starts +an ephemeral loopback fixture server and does not read repository run state. + Start it with one or more canonical Git roots: ```bash @@ -28,16 +38,50 @@ The supported umbrella form is: ./scripts/rae.sh operator serve --project /absolute/path/to/repository ``` +Preload one or more server-owned execution profiles with repeatable +`--execution-profile ` arguments. Profile paths and credentials stay on +the server. The browser receives only sanitized IDs, routes, models, and +readiness. + The server prints one URL. Its 256-bit session token appears only in the URL fragment. The app removes the fragment from browser history and keeps the token in memory for bearer-authenticated API and event-stream requests. +## Remote upstream mode + +To use the same local console with a separately hosted operator API, keep the +browser session on loopback and configure an upstream origin and an owner-only +credential file: + +```bash +./scripts/rae.sh operator serve \ + --remote-url https://operator.example \ + --token-file /absolute/path/to/operator-token +``` + +Remote mode cannot be combined with `--project`. The browser still reaches only +the ephemeral local URL and sends only its local session bearer. The server +reads the upstream bearer token from `--token-file` for every forwarded request; +it is never included in browser JavaScript, local API responses, or errors. + +`--remote-url` must be an origin-only HTTPS URL. HTTP is accepted only for an +explicit loopback development origin. The token file must be a regular file +owned by the current user, with no group or world permissions; symlinks and +unsafe files are rejected. Token rotation therefore takes effect on the next +request without restarting the console. + +Remote mode is a fixed API relay, not a general proxy. It rejects redirects and +forwards only the `/api/v1` methods used by this console, including the listed +run, event, control, and workflow-editor routes. Request bodies are limited to +64 KiB and upstream responses to 1 MiB. + ## API All `/api/v1` requests require the session bearer token and an exact loopback `Host`. State-changing requests also require the exact loopback `Origin`. - `GET /api/v1/projects` +- `GET /api/v1/projects/:projectId/execution-profiles` - `GET|POST /api/v1/projects/:projectId/runs` - `GET /api/v1/projects/:projectId/runs/:runId` - `GET /api/v1/projects/:projectId/runs/:runId/events` @@ -47,29 +91,47 @@ All `/api/v1` requests require the session bearer token and an exact loopback - `POST .../:runId/interrupt` - `POST .../:runId/checkpoint-decision` - `POST .../:runId/cleanup` - -Start accepts only `task` and `checkpoint_policy`. Interrupt and cleanup require -`confirm_run_id` to exactly match the selected run. A checkpoint decision -requires its opaque `checkpoint_id`, an opaque `decision_id`, one of `approve`, -`reject`, or `escalate`, and a non-empty `rationale`. The server records the -actor as `rae-loopback-operator`. +- `GET /api/v1/projects/:projectId/workflows` +- `GET /api/v1/projects/:projectId/workflows/:workflowId` +- `GET|POST /api/v1/projects/:projectId/workflows/templates` +- `POST /api/v1/projects/:projectId/workflows/:workflowId/analysis` +- `POST /api/v1/projects/:projectId/workflows/:workflowId/proposals` +- `GET /api/v1/projects/:projectId/workflows/:workflowId/proposals/:jobId` +- `POST /api/v1/projects/:projectId/workflows/:workflowId/drafts` +- `GET /api/v1/projects/:projectId/workflows/:workflowId/diff` +- `POST /api/v1/projects/:projectId/workflows/:workflowId/revisions/:revision/validate` +- `POST /api/v1/projects/:projectId/workflows/:workflowId/revisions/:revision/activate` + +Start accepts `task`, `checkpoint_policy`, and an optional preloaded +`execution_profile_id`. It never accepts a profile path. Interrupt and cleanup +require `confirm_run_id` to exactly match the selected run. A checkpoint +decision requires its opaque `checkpoint_id`, an opaque `decision_id`, one of +`approve`, `reject`, or `escalate`, and a non-empty `rationale`. The server +records the actor as `rae-loopback-operator`. The console never accepts in-place execution, command providers, environment overrides, raw trace access, forced cleanup, commit, push, or publish controls. Cleanup delegates to the pipeline's ownership- and dirty-state-validating worktree cleanup operation. -The workflow editor lists immutable revisions, renders synchronized SVG and -structured node/edge views, validates drafts, compares revisions, displays -budgets and activation history, and activates only after exact digest -confirmation. Native forms and the structured list provide every authoring -operation; canvas dragging is not required. Registry mutations are rejected -while any allowlisted project run is active. +The workflow editor provides synchronized Loop, Graph, Analyze, and JSON views. +It compiles five guided templates to workflow 2.1, exposes keyboard-operable +node and edge controls, analyzes unsaved revisions, and loads validated proposal +jobs into the unsaved editor. Saving a revision and activating its exact digest +remain separate human actions. Workflow 2.0 and experimental 2.2 stay available +through the expert JSON view. Registry mutations are rejected while any +allowlisted project run is active. + +Proposal creation is asynchronous. A request accepts only `task`, optional +`base_revision`, and optional `execution_profile_id`; task text is limited to +32 KiB and the in-memory queue holds at most 12 jobs. The result is validated +before it is returned to the editor. The proposal endpoint does not save a +revision, activate a digest, or start a run. Run projections include bounded graph health counts when a projection exists: availability, validation state, node and edge counts, stale-source count, and -stale-memory and unresolved-conflict counts. The API does not expose raw graph records, absolute -paths, prompts, provider metadata, or untrusted memory text. +stale-memory and unresolved-conflict counts. The API does not expose raw graph +records, absolute paths, prompts, provider metadata, or untrusted memory text. Only one process started by a server instance may be active at once. Interrupt signals that owned process group, records `interrupted` after it exits, and diff --git a/packages/orchestration/operator/demo/README.md b/packages/orchestration/operator/demo/README.md deleted file mode 100644 index a089f70..0000000 --- a/packages/orchestration/operator/demo/README.md +++ /dev/null @@ -1,18 +0,0 @@ -# Static Evidence Dossier demo - -The GitHub Pages demo is derived from the maintained operator console at build -time. It copies the canonical HTML, CSS, and JavaScript modules, then adds only -this directory's simulation notice, sanitized fixture adapter, and visible -action labels. - -The demo runs no command, contacts no operator API, and stores no run state. -Every interaction changes only the in-memory fixture until the page is reloaded. -It is a product walkthrough, not evidence from an autonomous run. - -Build it from the repository root with: - -```bash -npm --prefix packages/orchestration run build:pages-demo -``` - -The generated site is written to the ignored `dist/pages-demo/` directory. diff --git a/packages/orchestration/operator/demo/demo.css.txt b/packages/orchestration/operator/demo/demo.css.txt deleted file mode 100644 index 2dfe823..0000000 --- a/packages/orchestration/operator/demo/demo.css.txt +++ /dev/null @@ -1,96 +0,0 @@ -/* Keeps the static simulation unmistakable while preserving the canonical Evidence Dossier UI. */ - -:root { - --demo-notice-height: 2.75rem; -} - -.demo-notice { - position: sticky; - top: 0; - z-index: 50; - min-height: var(--demo-notice-height); - display: flex; - align-items: center; - justify-content: center; - gap: 0.65rem; - padding: 0.55rem 1rem; - border-bottom: 1px solid var(--ink); - background: var(--trace); - color: var(--sheet); - font-size: 0.76rem; - line-height: 1.25; - text-align: center; -} - -.demo-notice strong { - font-family: var(--mono); - font-size: 0.7rem; - letter-spacing: 0.08em; - text-transform: uppercase; -} - -.demo-notice a { - color: inherit; - text-underline-offset: 0.18em; -} - -.head { - top: var(--demo-notice-height); -} - -.skip { - position: fixed; - top: 0.5rem; - left: 0.5rem; - z-index: 60; - padding: 0.45rem 0.65rem; - background: var(--sheet); - color: var(--ink); - transform: translateY(-200%); -} - -.skip:focus { - transform: translateY(0); -} - -.simulated-label { - display: inline-block; - margin-left: 0.45rem; - padding-left: 0.45rem; - border-left: 1px solid currentColor; - font: 600 0.58rem/1 var(--mono); - letter-spacing: 0.04em; - text-transform: uppercase; - opacity: 0.78; -} - -.decision-action .simulated-label, -.run-controls .simulated-label { - font-size: 0.55rem; -} - -.primary-action .simulated-label { - display: inline-block; -} - -@media (max-width: 640px) { - .demo-notice { - position: static; - justify-content: flex-start; - flex-wrap: wrap; - text-align: left; - } - - .head { - top: 0; - } - - .head__sep, - .head__surface { - display: none; - } - - .demo-notice a { - width: 100%; - } -} diff --git a/packages/orchestration/operator/demo/mock-api.js b/packages/orchestration/operator/demo/mock-api.js deleted file mode 100644 index 8291cf5..0000000 --- a/packages/orchestration/operator/demo/mock-api.js +++ /dev/null @@ -1,420 +0,0 @@ -/** Supplies sanitized, in-memory API responses for the static operator-console demonstration. */ - -const PHASES = [ - "arm", - "design", - "adversarial-review", - "plan", - "pmatch", - "build", - "quality-static", - "quality-tests", - "post-build", - "release-readiness", -]; - -const SIMULATED_ACTION_IDS = [ - "new-run-button", - "start-submit", - "stop-button", - "interrupt-button", - "resume-button", - "cleanup-button", - "confirm-submit", -]; - -const baseRun = { - phase_order: PHASES, - workspace_mode: "worktree", - evidence: { present: 6 }, - resources: { agent_calls: 11, input: 184220, output: 28310, cost: null }, - graph_health: { - available: false, - valid: false, - node_count: 0, - edge_count: 0, - stale_sources: 0, - stale_memory: 0, - unresolved_conflicts: 0, - }, -}; - -const runs = [ - { - ...baseRun, - id: "run-7f3a2c91", - task: "Add a tested health endpoint and document the public behavior.", - branch: "pipeline/run-7f3a2c91", - workspace_label: ".git/rae-worktrees/run-7f3a2c91", - status: "awaiting", - current_phase: "build", - started_at: "2026-07-23T14:02:00.000Z", - updated_at: "2026-07-23T14:08:00.000Z", - completed_gates: [ - "arm-gate", - "design-gate", - "adversarial-review-gate", - "plan-gate", - "pmatch-gate", - ], - gates: [ - { gate_id: "arm-gate", phase: "arm", status: "pass", artifact_ref: "brief · a91f" }, - { gate_id: "design-gate", phase: "design", status: "pass", artifact_ref: "design · 3c20" }, - { - gate_id: "adversarial-review-gate", - phase: "adversarial-review", - status: "pass", - artifact_ref: "review · 88e1", - }, - { gate_id: "plan-gate", phase: "plan", status: "pass", artifact_ref: "plan · b7d4" }, - { gate_id: "pmatch-gate", phase: "pmatch", status: "pass", artifact_ref: "drift · 0f2a" }, - { - gate_id: "build-gate", - phase: "build", - status: "pending", - artifact_ref: "build · 7c…e19", - }, - ], - checkpoints: [ - { - checkpoint_id: "cp-4b91-build", - purpose: "mutation", - phase: "build", - status: "pending", - message: - "Plan-owned implementation is staged. Gate policy before-mutation-and-ship requires an operator record before quality-static runs.", - requested_at: "2026-07-23T14:08:00.000Z", - }, - ], - controls: { stop: true, interrupt: true, resume: false, cleanup: false }, - }, - { - ...baseRun, - id: "run-91bc08d2", - task: "Harden report path confinement for Ralph fixing transactions.", - branch: "pipeline/run-91bc08d2", - workspace_label: ".git/rae-worktrees/run-91bc08d2", - status: "completed", - current_phase: "release-readiness", - started_at: "2026-07-23T12:42:00.000Z", - updated_at: "2026-07-23T13:12:00.000Z", - completed_gates: PHASES.map((phase) => `${phase}-gate`), - gates: PHASES.map((phase, index) => ({ - gate_id: `${phase}-gate`, - phase, - status: "pass", - artifact_ref: `evidence · ${String(index + 1).padStart(2, "0")}`, - })), - checkpoints: [], - controls: { stop: false, interrupt: false, resume: false, cleanup: true }, - }, - { - ...baseRun, - id: "run-2e11d4a0", - task: "Correct a scoped documentation claim without widening the change set.", - branch: "pipeline/run-2e11d4a0", - workspace_label: ".git/rae-worktrees/run-2e11d4a0", - status: "blocked", - current_phase: "pmatch", - started_at: "2026-07-23T11:20:00.000Z", - updated_at: "2026-07-23T11:58:00.000Z", - completed_gates: ["arm-gate", "design-gate", "adversarial-review-gate", "plan-gate"], - gates: [ - { gate_id: "arm-gate", phase: "arm", status: "pass", artifact_ref: "brief · d102" }, - { gate_id: "design-gate", phase: "design", status: "pass", artifact_ref: "design · 73a4" }, - { - gate_id: "adversarial-review-gate", - phase: "adversarial-review", - status: "pass", - artifact_ref: "review · 220c", - }, - { gate_id: "plan-gate", phase: "plan", status: "pass", artifact_ref: "plan · c814" }, - { gate_id: "pmatch-gate", phase: "pmatch", status: "failed", artifact_ref: "drift · 91ff" }, - ], - checkpoints: [], - controls: { stop: false, interrupt: false, resume: true, cleanup: true }, - }, -]; - -const eventsByRun = new Map( - Object.entries({ - "run-7f3a2c91": [ - { - seq: 1, - ts: "2026-07-23T14:02:05.000Z", - phase: "arm", - event: "artifact_recorded", - artifact_ref: "brief · a91f", - status: "pass", - tier: "local", - }, - { - seq: 2, - ts: "2026-07-23T14:03:18.000Z", - phase: "design", - event: "gate_completed", - gate_id: "design-gate", - status: "pass", - tier: "local", - }, - { - seq: 3, - ts: "2026-07-23T14:04:42.000Z", - phase: "adversarial-review", - event: "review_completed", - artifact_ref: "review · 88e1", - status: "pass", - tier: "local", - }, - { - seq: 4, - ts: "2026-07-23T14:06:09.000Z", - phase: "plan", - event: "plan_validated", - artifact_ref: "plan · b7d4", - status: "pass", - tier: "local", - }, - { - seq: 5, - ts: "2026-07-23T14:07:31.000Z", - phase: "pmatch", - event: "drift_check_completed", - gate_id: "pmatch-gate", - status: "pass", - tier: "local", - }, - { - seq: 6, - ts: "2026-07-23T14:08:00.000Z", - phase: "build", - event: "checkpoint_requested", - event_id: "cp-4b91-build", - status: "pending", - tier: "human", - }, - ], - "run-91bc08d2": [ - { - seq: 1, - ts: "2026-07-23T12:42:03.000Z", - phase: "arm", - event: "run_started", - event_id: "evt-01", - status: "pass", - tier: "local", - }, - { - seq: 2, - ts: "2026-07-23T13:12:00.000Z", - phase: "release-readiness", - event: "release_gate_completed", - gate_id: "release-readiness-gate", - status: "pass", - tier: "local", - }, - ], - "run-2e11d4a0": [ - { - seq: 1, - ts: "2026-07-23T11:20:02.000Z", - phase: "arm", - event: "run_started", - event_id: "evt-01", - status: "pass", - tier: "local", - }, - { - seq: 2, - ts: "2026-07-23T11:58:00.000Z", - phase: "pmatch", - event: "drift_detected", - gate_id: "pmatch-gate", - status: "failed", - tier: "local", - }, - ], - }), -); - -function json(payload, status = 200) { - return new Response(JSON.stringify(payload), { - status, - headers: { "content-type": "application/json; charset=utf-8" }, - }); -} - -function bodyOf(options) { - return options.body ? JSON.parse(options.body) : {}; -} - -function updateRun(run, changes) { - Object.assign(run, changes, { updated_at: "2026-07-23T14:24:00.000Z" }); -} - -function applyCheckpointDecision(run, body) { - const checkpoint = run.checkpoints.find((item) => item.checkpoint_id === body.checkpoint_id); - if (!checkpoint) return json({ error: { message: "Fixture checkpoint not found." } }, 404); - checkpoint.status = body.decision; - if (body.decision === "approve") { - const buildGate = run.gates.find((gate) => gate.gate_id === "build-gate"); - buildGate.status = "pass"; - if (!run.completed_gates.includes("build-gate")) run.completed_gates.push("build-gate"); - updateRun(run, { status: "running", current_phase: "quality-static" }); - } else { - updateRun(run, { status: body.decision === "reject" ? "blocked" : "awaiting" }); - } - const events = eventsByRun.get(run.id); - events.push({ - seq: events.length + 1, - ts: run.updated_at, - phase: checkpoint.phase, - event: `checkpoint_${body.decision}`, - event_id: checkpoint.checkpoint_id, - status: body.decision, - tier: "human", - }); - return json({ ok: true, simulated: true }); -} - -function createRun(body) { - const id = `run-demo-${String(runs.length + 1).padStart(2, "0")}`; - const run = { - ...structuredClone(baseRun), - id, - task: body.task, - branch: `pipeline/${id}`, - workspace_label: `.git/rae-worktrees/${id}`, - status: "awaiting", - current_phase: "arm", - started_at: "2026-07-23T14:24:00.000Z", - updated_at: "2026-07-23T14:24:00.000Z", - completed_gates: [], - gates: [ - { gate_id: "arm-gate", phase: "arm", status: "pending", artifact_ref: "brief · fixture" }, - ], - checkpoints: [ - { - checkpoint_id: `${id}-arm`, - purpose: "mutation", - phase: "arm", - status: "pending", - message: "This simulated run is waiting at its first fixture checkpoint.", - requested_at: "2026-07-23T14:24:00.000Z", - }, - ], - controls: { stop: true, interrupt: true, resume: false, cleanup: false }, - }; - runs.unshift(run); - eventsByRun.set(id, [ - { - seq: 1, - ts: run.started_at, - phase: "arm", - event: "fixture_run_created", - event_id: `${id}-created`, - status: "pending", - tier: "simulation", - }, - ]); - return json({ run_id: id, simulated: true }, 202); -} - -function streamResponse(signal) { - const stream = new ReadableStream({ - start(controller) { - if (signal?.aborted) { - controller.close(); - return; - } - signal?.addEventListener("abort", () => controller.close(), { once: true }); - }, - }); - return new Response(stream, { - status: 200, - headers: { "content-type": "application/x-ndjson" }, - }); -} - -function getResponse(path, options) { - if (path === "/projects") { - return json({ - projects: [{ id: "project_fixture", label: "sebastianspicker/rae · fixture" }], - }); - } - if (path.endsWith("/events/stream")) return streamResponse(options.signal); - - const eventsMatch = path.match(/^\/projects\/[^/]+\/runs\/([^/]+)\/events$/); - if (eventsMatch) { - const events = structuredClone(eventsByRun.get(decodeURIComponent(eventsMatch[1])) || []); - return json({ events, next_after: events.at(-1)?.seq || 0 }); - } - - if (/^\/projects\/[^/]+\/runs$/.test(path)) return json({ runs: structuredClone(runs) }); - return null; -} - -function actionStatus(action) { - if (action === "stop") return "stopping"; - if (action === "resume") return "running"; - return "interrupted"; -} - -function postAction(actionMatch, options) { - const runId = decodeURIComponent(actionMatch[1]); - const action = actionMatch[2]; - const run = runs.find((item) => item.id === runId); - if (!run) return json({ error: { message: "Fixture run not found." } }, 404); - if (action === "checkpoint-decision") return applyCheckpointDecision(run, bodyOf(options)); - if (action === "cleanup") { - runs.splice(runs.indexOf(run), 1); - return json({ ok: true, simulated: true }); - } - updateRun(run, { status: actionStatus(action) }); - if (action === "interrupt") { - run.controls = { stop: false, interrupt: false, resume: true, cleanup: true }; - } - return json({ ok: true, simulated: true }); -} - -function postResponse(path, options) { - if (/^\/projects\/[^/]+\/runs$/.test(path)) return createRun(bodyOf(options)); - - const actionMatch = path.match( - /^\/projects\/[^/]+\/runs\/([^/]+)\/(stop|resume|interrupt|cleanup|checkpoint-decision)$/, - ); - return actionMatch ? postAction(actionMatch, options) : null; -} - -function demoFetch(input, options = {}) { - const url = new URL(typeof input === "string" ? input : input.url, location.href); - if (!url.pathname.startsWith("/api/v1/")) { - throw new Error("The static simulation does not permit network requests."); - } - - const path = url.pathname.slice("/api/v1".length); - const method = String(options.method || "GET").toUpperCase(); - const response = method === "GET" ? getResponse(path, options) : postResponse(path, options); - return response || json({ error: { message: "Unsupported static-demo request." } }, 404); -} - -function markSimulatedControls() { - const controls = [ - ...SIMULATED_ACTION_IDS.map((id) => document.getElementById(id)), - ...document.querySelectorAll("[data-decision]"), - ]; - for (const control of controls) { - if (!control || control.querySelector(".simulated-label")) continue; - const marker = document.createElement("span"); - marker.className = "simulated-label"; - marker.textContent = "Simulated"; - marker.setAttribute("aria-hidden", "true"); - control.append(marker); - control.setAttribute("aria-label", `${control.textContent.trim()} (simulated)`); - } -} - -history.replaceState(null, "", `${location.pathname}${location.search}#token=static-demo`); -window.fetch = demoFetch; -markSimulatedControls(); -await import("../app.js"); diff --git a/packages/orchestration/operator/docs/screenshots/evidence-dossier-desktop.png b/packages/orchestration/operator/docs/screenshots/evidence-dossier-desktop.png index ae8b9b3..b9cd967 100644 Binary files a/packages/orchestration/operator/docs/screenshots/evidence-dossier-desktop.png and b/packages/orchestration/operator/docs/screenshots/evidence-dossier-desktop.png differ diff --git a/packages/orchestration/operator/docs/screenshots/evidence-dossier-mobile.png b/packages/orchestration/operator/docs/screenshots/evidence-dossier-mobile.png index 21cdf00..27b6d1f 100644 Binary files a/packages/orchestration/operator/docs/screenshots/evidence-dossier-mobile.png and b/packages/orchestration/operator/docs/screenshots/evidence-dossier-mobile.png differ diff --git a/packages/orchestration/operator/lib/control.mjs b/packages/orchestration/operator/lib/control.mjs index bf0f2f4..90cac1e 100644 --- a/packages/orchestration/operator/lib/control.mjs +++ b/packages/orchestration/operator/lib/control.mjs @@ -26,7 +26,7 @@ function httpError(status, message) { } function assertAllowedStartFields(body) { - const allowed = new Set(["task", "checkpoint_policy"]); + const allowed = new Set(["task", "checkpoint_policy", "execution_profile_id"]); for (const key of Object.keys(body)) { if (!allowed.has(key)) throw httpError(400, `unsupported start field: ${key}`); } @@ -48,11 +48,15 @@ function startCheckpointPolicy(value) { return policy; } -export function validateStartInput(body) { +export function validateStartInput(body, executionProfile = null) { assertAllowedStartFields(body); + if (body.execution_profile_id !== undefined && !executionProfile?.source) { + throw httpError(400, "execution_profile_id must name a preloaded execution profile"); + } return { task: normalizedStartTask(body.task), checkpointPolicy: startCheckpointPolicy(body.checkpoint_policy), + ...(executionProfile ? { executionProfile } : {}), }; } @@ -199,8 +203,8 @@ export class RunController { return this.ownedRunId; } - start(project, body) { - const { task, checkpointPolicy } = validateStartInput(body); + start(project, body, executionProfile = null) { + const { task, checkpointPolicy } = validateStartInput(body, executionProfile); const baselineIds = new Set(this.discoverRunsFn(project).map((run) => run.id)); this.#spawn( project, @@ -214,6 +218,7 @@ export class RunController { "codex", "--checkpoint-policy", checkpointPolicy, + ...(executionProfile ? ["--execution-profile", executionProfile.source] : []), "--json", ], baselineIds, diff --git a/packages/orchestration/operator/lib/profiles.mjs b/packages/orchestration/operator/lib/profiles.mjs new file mode 100644 index 0000000..446401a --- /dev/null +++ b/packages/orchestration/operator/lib/profiles.mjs @@ -0,0 +1,86 @@ +/** Keeps execution-profile files server-side and exposes a deliberately small public projection. */ +import { existsSync } from "node:fs"; +import { pathToFileURL } from "node:url"; +import { resolve } from "node:path"; + +const executionProfilePath = resolve( + import.meta.dirname, + "../../scripts/pipeline/lib/execution-profile.mjs", +); + +function unavailable(message) { + throw Object.assign(new Error(message), { status: 503 }); +} + +function publicProfile({ profile }) { + const routeRecords = + profile.schema_version === "3.0.0" + ? Object.entries(profile.routes).map(([id, route]) => ({ id, ...route })) + : Object.entries(profile.tiers ?? {}).map(([id, route]) => ({ + id, + executor: "codex", + ...route, + })); + const models = + profile.schema_version === "3.0.0" + ? Object.fromEntries( + Object.entries(profile.tiers).map(([tier, routeId]) => [ + tier, + profile.routes[routeId].model, + ]), + ) + : Object.fromEntries( + Object.entries(profile.tiers ?? {}).map(([tier, mapping]) => [tier, mapping.model]), + ); + return Object.freeze({ + id: profile.profile_id, + routes: routeRecords.sort((left, right) => left.id.localeCompare(right.id)), + models, + readiness: "loaded", + }); +} + +/** + * Loads each explicitly supplied profile once at server startup. Source paths, + * environment names, capabilities, and profile contents never leave this map. + */ +export async function loadOperatorProfiles(paths = []) { + if (!Array.isArray(paths)) throw new Error("execution profiles must be an array"); + if (paths.length > 16) throw new Error("at most 16 execution profiles may be loaded"); + if (paths.length === 0) return new OperatorProfiles(); + if (!existsSync(executionProfilePath)) unavailable("execution profile support is unavailable"); + const module = await import(pathToFileURL(executionProfilePath).href); + if (typeof module.loadExecutionProfile !== "function") { + unavailable("execution profile support is unavailable"); + } + const loaded = paths.map((pathValue) => module.loadExecutionProfile(pathValue)); + return new OperatorProfiles(loaded); +} + +export class OperatorProfiles { + constructor(loaded = []) { + this.records = new Map(); + for (const record of loaded) { + const id = record?.profile?.profile_id; + if (typeof id !== "string" || !id) throw new Error("invalid execution profile"); + if (this.records.has(id)) throw new Error(`duplicate execution profile id: ${id}`); + this.records.set(id, Object.freeze(record)); + } + } + + list() { + return [...this.records.values()] + .map(publicProfile) + .sort((left, right) => left.id.localeCompare(right.id)); + } + + resolve(id) { + if (id === undefined || id === null || id === "") return null; + if (typeof id !== "string" || !/^[a-z][a-z0-9-]{2,63}$/.test(id)) { + throw Object.assign(new Error("invalid execution_profile_id"), { status: 400 }); + } + const record = this.records.get(id); + if (!record) throw Object.assign(new Error("unknown execution_profile_id"), { status: 400 }); + return record; + } +} diff --git a/packages/orchestration/operator/lib/proposals.mjs b/packages/orchestration/operator/lib/proposals.mjs new file mode 100644 index 0000000..b62ccf8 --- /dev/null +++ b/packages/orchestration/operator/lib/proposals.mjs @@ -0,0 +1,135 @@ +/** Bounded, ephemeral workflow-proposal jobs. Candidates are never saved or activated here. */ +import { randomUUID } from "node:crypto"; +import { existsSync } from "node:fs"; +import { resolve } from "node:path"; +import { pathToFileURL } from "node:url"; + +const proposalPath = resolve( + import.meta.dirname, + "../../scripts/pipeline/lib/workflow-proposal.mjs", +); +const workflowContractPath = resolve( + import.meta.dirname, + "../../scripts/pipeline/lib/workflow-contract.mjs", +); +const MAX_JOBS = 12; +const MAX_TASK_BYTES = 32 * 1024; +const PROPOSAL_FIELDS = new Set(["task", "base_revision", "execution_profile_id"]); + +function httpError(status, message) { + return Object.assign(new Error(message), { status }); +} + +/** Validates the enclosing proposal object before field-specific checks. */ +export function validateProposalBody(body) { + if (!body || typeof body !== "object" || Array.isArray(body)) + throw httpError(400, "proposal body is required"); + for (const key of Object.keys(body)) + if (!PROPOSAL_FIELDS.has(key)) throw httpError(400, `unsupported proposal field: ${key}`); + return body; +} + +/** Validates and normalizes the proposal fields accepted by the job queue. */ +export function validateProposalFields(body) { + if (typeof body.task !== "string" || body.task.trim().length === 0) + throw httpError(400, "proposal task is required"); + if (Buffer.byteLength(body.task, "utf8") > MAX_TASK_BYTES) + throw httpError(413, "proposal task exceeds 32768 bytes"); + if (body.base_revision !== undefined && !/^[0-9]{1,9}$/.test(String(body.base_revision))) { + throw httpError(400, "invalid base_revision"); + } + return { + task: body.task.trim(), + baseRevision: body.base_revision ?? null, + executionProfileId: body.execution_profile_id ?? null, + }; +} + +function requestInput(body) { + return validateProposalFields(validateProposalBody(body)); +} + +async function defaultCandidateRunner(input) { + if (!existsSync(proposalPath)) throw httpError(503, "workflow proposal support is unavailable"); + const module = await import(pathToFileURL(proposalPath).href); + // The legacy `proposeWorkflow` persists a draft and is deliberately never called. + if (typeof module.proposeWorkflowCandidate !== "function") { + throw httpError(503, "unsaved workflow proposal support is unavailable"); + } + return module.proposeWorkflowCandidate(input); +} + +async function validateCandidate(candidate) { + if (!existsSync(workflowContractPath)) + throw httpError(503, "workflow validation support is unavailable"); + const module = await import(pathToFileURL(workflowContractPath).href); + if (typeof module.validateWorkflow !== "function") { + throw httpError(503, "workflow validation support is unavailable"); + } + return module.validateWorkflow(candidate); +} + +function publicJob(job) { + return { + id: job.id, + workflow_id: job.workflowId, + state: job.state, + created_at: job.createdAt, + completed_at: job.completedAt ?? null, + ...(job.error ? { error: job.error } : {}), + ...(job.candidate ? { candidate: job.candidate } : {}), + }; +} + +export class WorkflowProposalJobs { + constructor({ candidateRunner = defaultCandidateRunner, maxJobs = MAX_JOBS } = {}) { + this.candidateRunner = candidateRunner; + this.maxJobs = maxJobs; + this.jobs = new Map(); + } + + submit({ project, workflowId, body, executionProfile = null }) { + const input = requestInput(body); + if (input.executionProfileId && !executionProfile?.source) { + throw httpError(400, "execution_profile_id must name a preloaded execution profile"); + } + if (this.jobs.size >= this.maxJobs) throw httpError(429, "workflow proposal queue is full"); + const job = { + id: `proposal-${randomUUID()}`, + workflowId, + state: "queued", + createdAt: new Date().toISOString(), + }; + this.jobs.set(job.id, job); + queueMicrotask(async () => { + job.state = "running"; + try { + const candidate = await this.candidateRunner({ + projectRoot: project.root, + workflowId, + task: input.task, + baseRevision: input.baseRevision, + ...(executionProfile ? { executionProfile: executionProfile.source } : {}), + }); + if (!candidate || typeof candidate !== "object" || Array.isArray(candidate)) { + throw new Error("proposal runner returned no workflow candidate"); + } + job.candidate = structuredClone(await validateCandidate(candidate)); + job.state = "completed"; + } catch (error) { + job.error = error?.status >= 500 ? error.message : "proposal could not be generated"; + job.state = "failed"; + } finally { + job.completedAt = new Date().toISOString(); + } + }); + return publicJob(job); + } + + get(id, workflowId = null) { + const job = this.jobs.get(id); + if (!job) throw httpError(404, "proposal job not found"); + if (workflowId && job.workflowId !== workflowId) throw httpError(404, "proposal job not found"); + return publicJob(job); + } +} diff --git a/packages/orchestration/operator/lib/remote.mjs b/packages/orchestration/operator/lib/remote.mjs new file mode 100644 index 0000000..956f109 --- /dev/null +++ b/packages/orchestration/operator/lib/remote.mjs @@ -0,0 +1,324 @@ +/** Constrains remote operator API forwarding to the console's known REST surface. */ +import { closeSync, constants, fstatSync, lstatSync, openSync, readFileSync } from "node:fs"; + +export const MAX_REMOTE_RESPONSE_BYTES = 1024 * 1024; +const MAX_REQUEST_BYTES = 64 * 1024; +const SAFE_PROJECT_ID = /^[A-Za-z0-9_-]{8,64}$/; +const SAFE_RUN_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; +const SAFE_WORKFLOW_ID = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; +const SAFE_REVISION = /^[1-9][0-9]{0,8}$/; +const SAFE_PROPOSAL_JOB_ID = /^proposal-[a-f0-9-]{36}$/; +const SAFE_QUERY_VALUE = /^[A-Za-z0-9._-]{1,128}$/; + +const SEGMENTS = Object.freeze({ + projectId: SAFE_PROJECT_ID, + proposalJobId: SAFE_PROPOSAL_JOB_ID, + revision: SAFE_REVISION, + runId: SAFE_RUN_ID, + workflowId: SAFE_WORKFLOW_ID, +}); + +/** + * Fixed remote API contract. Keep each reachable console route explicit so an + * upstream credential cannot become a general-purpose proxy capability. + */ +const REMOTE_ROUTE_SPECS = Object.freeze([ + { method: "GET", path: ["api", "v1", "projects"], query: [] }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "execution-profiles"], + query: [], + }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "runs"], + query: ["cursor", "limit"], + }, + { + method: "POST", + path: ["api", "v1", "projects", SEGMENTS.projectId, "runs"], + query: ["cursor", "limit"], + }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "runs", SEGMENTS.runId], + query: [], + }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "runs", SEGMENTS.runId, "events"], + query: ["after", "limit"], + }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "runs", SEGMENTS.runId, "events", "stream"], + query: ["after"], + }, + ...["stop", "resume", "interrupt", "checkpoint-decision", "cleanup"].map((action) => ({ + method: "POST", + path: ["api", "v1", "projects", SEGMENTS.projectId, "runs", SEGMENTS.runId, action], + query: [], + })), + { method: "GET", path: ["api", "v1", "projects", SEGMENTS.projectId, "workflows"], query: [] }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "workflows", "templates"], + query: [], + }, + { + method: "POST", + path: ["api", "v1", "projects", SEGMENTS.projectId, "workflows", "templates"], + query: [], + }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "workflows", SEGMENTS.workflowId], + query: [], + }, + { + method: "POST", + path: [ + "api", + "v1", + "projects", + SEGMENTS.projectId, + "workflows", + SEGMENTS.workflowId, + "analysis", + ], + query: [], + }, + { + method: "POST", + path: [ + "api", + "v1", + "projects", + SEGMENTS.projectId, + "workflows", + SEGMENTS.workflowId, + "proposals", + ], + query: [], + }, + { + method: "GET", + path: [ + "api", + "v1", + "projects", + SEGMENTS.projectId, + "workflows", + SEGMENTS.workflowId, + "proposals", + SEGMENTS.proposalJobId, + ], + query: [], + }, + { + method: "POST", + path: ["api", "v1", "projects", SEGMENTS.projectId, "workflows", SEGMENTS.workflowId, "drafts"], + query: [], + }, + { + method: "GET", + path: ["api", "v1", "projects", SEGMENTS.projectId, "workflows", SEGMENTS.workflowId, "diff"], + query: ["from", "to"], + }, + ...["validate", "activate"].map((action) => ({ + method: "POST", + path: [ + "api", + "v1", + "projects", + SEGMENTS.projectId, + "workflows", + SEGMENTS.workflowId, + "revisions", + SEGMENTS.revision, + action, + ], + query: [], + })), +]); + +function remoteError(message, status = 502) { + return Object.assign(new Error(message), { status }); +} + +function isLoopbackHost(hostname) { + return hostname === "127.0.0.1" || hostname === "::1"; +} + +/** Validates the single upstream origin permitted for a remote console session. */ +export function parseRemoteUrl(value) { + let url; + try { + url = new URL(value); + } catch { + throw new Error("--remote-url must be an absolute HTTPS URL"); + } + if (url.username || url.password || url.search || url.hash || url.pathname !== "/") { + throw new Error("--remote-url must contain only an origin"); + } + if (url.protocol !== "https:" && !(url.protocol === "http:" && isLoopbackHost(url.hostname))) { + throw new Error("--remote-url must use HTTPS (HTTP is limited to loopback development)"); + } + return url; +} + +function validateTokenFileStat(stat) { + if (!stat.isFile() || stat.isSymbolicLink() || (stat.mode & 0o077) !== 0) { + throw remoteError("upstream token file is unsafe"); + } + if (typeof process.getuid === "function" && stat.uid !== process.getuid()) { + throw remoteError("upstream token file is unsafe"); + } +} + +/** Reads one owner-only bearer token without following a symlink. */ +export function readRemoteTokenFile(tokenFile) { + let descriptor; + try { + validateTokenFileStat(lstatSync(tokenFile)); + const noFollow = constants.O_NOFOLLOW ?? 0; + descriptor = openSync(tokenFile, constants.O_RDONLY | noFollow); + validateTokenFileStat(fstatSync(descriptor)); + const token = readFileSync(descriptor, "utf8").trim(); + if (!/^[\x21-\x7e]{1,8192}$/.test(token)) throw remoteError("upstream token file is invalid"); + return token; + } catch (error) { + if (error?.status) throw error; + throw remoteError("upstream token file is unavailable"); + } finally { + if (descriptor !== undefined) closeSync(descriptor); + } +} + +function decodeParts(pathname) { + try { + return pathname + .split("/") + .filter(Boolean) + .map((part) => decodeURIComponent(part)); + } catch { + throw remoteError("remote operator path is not allowed", 404); + } +} + +function hasAllowedQuery(searchParams, allowed) { + const seen = new Set(); + for (const [key, value] of searchParams) { + if (!allowed.has(key) || seen.has(key) || !SAFE_QUERY_VALUE.test(value)) { + return false; + } + seen.add(key); + } + return true; +} + +function matchesRoutePart(expected, actual) { + return typeof expected === "string" ? expected === actual : expected.test(actual); +} + +function matchesRoute(spec, method, parts, searchParams) { + return ( + spec.method === method && + spec.path.length === parts.length && + spec.path.every((expected, index) => matchesRoutePart(expected, parts[index])) && + hasAllowedQuery(searchParams, new Set(spec.query)) + ); +} + +/** Returns true only for routes implemented by the local console UI. */ +export function isAllowedRemoteRequest(method, pathname, searchParams) { + const parts = decodeParts(pathname); + return REMOTE_ROUTE_SPECS.some((spec) => matchesRoute(spec, method, parts, searchParams)); +} + +async function readRequestBody(req) { + const declared = Number(req.headers["content-length"] ?? 0); + if (!Number.isFinite(declared) || declared < 0 || declared > MAX_REQUEST_BYTES) { + throw remoteError("request body exceeds 65536 bytes", 413); + } + let size = 0; + const chunks = []; + for await (const chunk of req) { + size += chunk.length; + if (size > MAX_REQUEST_BYTES) throw remoteError("request body exceeds 65536 bytes", 413); + chunks.push(chunk); + } + return Buffer.concat(chunks); +} + +async function readResponseBody(response) { + assertResponseLength(response); + const reader = response.body?.getReader(); + if (!reader) return Buffer.alloc(0); + let size = 0; + const chunks = []; + for (;;) { + const { value, done } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > MAX_REMOTE_RESPONSE_BYTES) { + await reader.cancel(); + throw remoteError("remote operator response exceeds size limit"); + } + chunks.push(Buffer.from(value)); + } + return Buffer.concat(chunks); +} + +function assertResponseLength(response) { + const declared = Number(response.headers.get("content-length") ?? 0); + if (!Number.isFinite(declared) || declared < 0 || declared > MAX_REMOTE_RESPONSE_BYTES) { + throw remoteError("remote operator response exceeds size limit"); + } +} + +/** Creates the server-side-only upstream credential and strict REST forwarder. */ +export function createRemoteOperatorProxy({ remoteUrl, tokenFile, fetchImpl = globalThis.fetch }) { + if (typeof tokenFile !== "string" || tokenFile.length === 0) { + throw new Error("--token-file is required with --remote-url"); + } + if (typeof fetchImpl !== "function") throw new Error("remote fetch is unavailable"); + const upstream = parseRemoteUrl(remoteUrl); + return { + async forward(req, url) { + if (!isAllowedRemoteRequest(req.method, url.pathname, url.searchParams)) { + throw remoteError("remote operator path is not allowed", 404); + } + const target = new URL(`${url.pathname}${url.search}`, upstream); + if (target.origin !== upstream.origin) + throw remoteError("remote operator path is not allowed", 404); + const body = ["POST", "PUT", "PATCH"].includes(req.method) + ? await readRequestBody(req) + : null; + const response = await fetchImpl(target, { + method: req.method, + headers: { + authorization: `Bearer ${readRemoteTokenFile(tokenFile)}`, + ...(body ? { "content-type": "application/json" } : {}), + }, + body: body?.length ? body : undefined, + redirect: "manual", + signal: AbortSignal.timeout(15_000), + }); + if (response.status >= 300 && response.status < 400) { + throw remoteError("remote operator redirect rejected"); + } + const contentType = response.headers.get("content-type") ?? "application/octet-stream"; + if (url.pathname.endsWith("/events/stream")) { + assertResponseLength(response); + return { status: response.status, contentType, stream: response.body }; + } + const responseBody = await readResponseBody(response); + return { + status: response.status, + contentType, + body: responseBody, + }; + }, + }; +} diff --git a/packages/orchestration/operator/lib/runs.mjs b/packages/orchestration/operator/lib/runs.mjs index 2011c87..f6f3561 100644 --- a/packages/orchestration/operator/lib/runs.mjs +++ b/packages/orchestration/operator/lib/runs.mjs @@ -106,17 +106,11 @@ function workspaceRuns(project, workspaceRoot, projectRoot) { function guardedWorkspaceRun(project, workspaceRoot) { const guard = inspectRuntimeStateGuard(workspaceRoot); - if (!guard.found || guardIsReadableInactive(guard, workspaceRoot)) return null; - return activeGuardedWorkspaceRun(project, workspaceRoot, guard); -} - -function guardIsReadableInactive(guard, workspaceRoot) { - if (guard.ownerActive) return false; - ensureRuntimeStateReadable(workspaceRoot, { expectedRunId: guard.runId }); - return true; -} - -function activeGuardedWorkspaceRun(project, workspaceRoot, guard) { + if (!guard.found) return null; + if (!guard.ownerActive) { + ensureRuntimeStateReadable(workspaceRoot, { expectedRunId: guard.runId }); + return null; + } return { id: guard.runId, project_id: project.id, @@ -187,31 +181,20 @@ function projectedEvents(run, runDir) { } function runTiming(request, events, runDir) { - const startedAt = runStartTime(request, events, runDir); + const startedAt = + request.requested_at ?? + events[0]?.ts ?? + readJson(join(runDir, "operator-control.json"))?.updated_at ?? + null; return { startedAt, - updatedAt: runUpdatedTime(events, runDir, startedAt), + updatedAt: + readOperatorControl(basename(runDir), resolve(runDir, "../../..")).updated_at ?? + events.at(-1)?.ts ?? + startedAt, }; } -function runStartTime(request, events, runDir) { - if (request.requested_at) return request.requested_at; - return eventOrControlStartTime(events, runDir); -} - -function eventOrControlStartTime(events, runDir) { - if (events[0]?.ts) return events[0].ts; - return readJson(join(runDir, "operator-control.json"))?.updated_at ?? null; -} - -function runUpdatedTime(events, runDir, startedAt) { - return ( - readOperatorControl(basename(runDir), resolve(runDir, "../../..")).updated_at ?? - events.at(-1)?.ts ?? - startedAt - ); -} - function runResources(progress, events) { return { input: progress.cost_summary?.total_tokens_in ?? null, diff --git a/packages/orchestration/operator/lib/workflows.mjs b/packages/orchestration/operator/lib/workflows.mjs index cdeea66..a37c49d 100644 --- a/packages/orchestration/operator/lib/workflows.mjs +++ b/packages/orchestration/operator/lib/workflows.mjs @@ -7,6 +7,10 @@ const registryPath = resolve( import.meta.dirname, "../../scripts/pipeline/lib/workflow-registry.mjs", ); +const workflowDesignerPath = resolve( + import.meta.dirname, + "../../scripts/pipeline/lib/workflow-designer.mjs", +); function unavailable() { throw Object.assign(new Error("workflow registry is unavailable"), { status: 503 }); @@ -46,3 +50,30 @@ export function assertRegistryMethod(registry, name) { if (!registry || typeof registry[name] !== "function") unavailable(); return registry[name].bind(registry); } + +/** Returns pipeline-owned static workflow analysis when that optional export exists. */ +export async function analyzeWorkflowFor(workflow) { + if (!existsSync(workflowDesignerPath)) + return { available: false, reason: "workflow analysis is unavailable" }; + const module = await import(pathToFileURL(workflowDesignerPath).href); + if (typeof module.analyzeWorkflow !== "function") { + return { available: false, reason: "workflow analysis is unavailable" }; + } + return { available: true, analysis: await module.analyzeWorkflow(workflow) }; +} + +/** Lists the pipeline-owned v2.1 guided templates. */ +export async function workflowTemplates() { + if (!existsSync(workflowDesignerPath)) unavailable(); + const module = await import(pathToFileURL(workflowDesignerPath).href); + if (typeof module.listWorkflowTemplates !== "function") unavailable(); + return module.listWorkflowTemplates(); +} + +/** Compiles a guided template to the unchanged workflow v2.1 contract. */ +export async function compileWorkflowTemplateFor(templateId, options) { + if (!existsSync(workflowDesignerPath)) unavailable(); + const module = await import(pathToFileURL(workflowDesignerPath).href); + if (typeof module.compileWorkflowTemplate !== "function") unavailable(); + return module.compileWorkflowTemplate(templateId, options); +} diff --git a/packages/orchestration/operator/scripts/capture-docs-screenshots.mjs b/packages/orchestration/operator/scripts/capture-docs-screenshots.mjs new file mode 100644 index 0000000..cc1f7b8 --- /dev/null +++ b/packages/orchestration/operator/scripts/capture-docs-screenshots.mjs @@ -0,0 +1,387 @@ +/** Captures operator documentation screenshots from current UI code and sanitized fixtures. */ +import { spawn } from "node:child_process"; +import { createServer } from "node:http"; +import { existsSync, readFileSync } from "node:fs"; +import { extname, resolve, sep } from "node:path"; + +import { + compileWorkflowTemplate, + listWorkflowTemplates, +} from "../../scripts/pipeline/lib/workflow-designer.mjs"; +import { workflowDigest } from "../../scripts/pipeline/lib/workflow-contract.mjs"; + +const OPERATOR_ROOT = resolve(import.meta.dirname, ".."); +const STATIC_ROOT = resolve(OPERATOR_ROOT, "static"); +const SCREENSHOT_ROOT = resolve(OPERATOR_ROOT, "docs", "screenshots"); +const TOKEN = "rae-docs-capture-token"; +const PROJECT_ID = "project_docs_fixture"; +const WORKFLOW = compileWorkflowTemplate("bounded-until-dry-loop", { + workflow_id: "release-discovery", + revision: 3, + title: "Bounded release discovery", + max_repair_rounds: 3, +}); +const WORKFLOW_DIGEST = workflowDigest(WORKFLOW); + +const RUN = Object.freeze({ + id: "run-2026-08-05-graph", + task: "Update the operator graph designer and verify the public documentation.", + branch: "pipeline/operator-graph-docs", + workspace_mode: "isolated-worktree", + workspace_label: "rae-worktree", + status: "completed", + runtime_active: false, + current_phase: "release-readiness", + phase_order: ["arm", "plan", "build", "quality-tests", "release-readiness"], + completed_gates: ["arm-gate", "plan-gate", "build-gate", "quality-tests-gate"], + started_at: "2026-08-05T08:32:00.000Z", + updated_at: "2026-08-05T08:47:00.000Z", + controls: { stop: false, interrupt: false, resume: false, cleanup: true }, + checkpoints: [], + gates: [ + { gate_id: "arm-gate", phase: "arm", status: "pass", artifact_ref: "brief.json" }, + { gate_id: "plan-gate", phase: "plan", status: "pass", artifact_ref: "plan.json" }, + { gate_id: "build-gate", phase: "build", status: "pass", artifact_ref: "changes.json" }, + { + gate_id: "quality-tests-gate", + phase: "quality-tests", + status: "pass", + artifact_ref: "verification.json", + }, + { + gate_id: "release-readiness-gate", + phase: "release-readiness", + status: "pass", + artifact_ref: "release.json", + }, + ], + evidence: { present: 12 }, + resources: { agent_calls: 8, input: 42816, output: 9312, cost: null }, + graph_health: { + available: true, + valid: true, + node_count: WORKFLOW.nodes.length, + edge_count: WORKFLOW.edges.length, + stale_sources: 0, + stale_memory: 0, + unresolved_conflicts: 0, + }, + workflow: { + workflow_id: WORKFLOW.workflow_id, + revision: WORKFLOW.revision, + digest: WORKFLOW_DIGEST, + budgets: WORKFLOW.budgets, + instances: [ + instance("discovery-loop", "passed", 2, "control", { convergence: { dry: true } }), + instance("discover", "passed", 2, "economy"), + instance("assess", "passed", 2, "judgment"), + instance("verify", "passed", 1, "control"), + instance("complete", "passed", 1, "control"), + ], + }, +}); + +const EVENTS = Object.freeze([ + event(1, "plan", "artifact_validated", "pass", "plan.json"), + event(2, "build", "workspace_changed", "pass", "changes.json"), + event(3, "quality-tests", "verification_completed", "pass", "verification.json"), + event(4, "release-readiness", "workflow_completed", "pass", "release.json"), +]); + +const CAPTURE_STYLE = ` + +`; +const CAPTURE_PROBE = ` + +`; + +const MIME_TYPES = new Map([ + [".css", "text/css; charset=utf-8"], + [".html", "text/html; charset=utf-8"], + [".js", "text/javascript; charset=utf-8"], + [".svg", "image/svg+xml"], +]); + +function instance(nodeId, status, attempt, executionTier, extra = {}) { + return { + instance_id: nodeId, + node_id: nodeId, + parent_node: null, + item_key: null, + item_digest: null, + status, + attempt, + execution_tier: executionTier, + selection: null, + quorum: null, + convergence: null, + ...extra, + }; +} + +function event(seq, phase, name, status, artifactRef) { + return { + seq, + event: name, + phase, + status, + artifact_ref: artifactRef, + ts: `2026-08-05T08:${String(35 + seq * 3).padStart(2, "0")}:00.000Z`, + }; +} + +function json(response, value, status = 200) { + response.writeHead(status, { + "cache-control": "no-store", + "content-type": "application/json; charset=utf-8", + }); + response.end(`${JSON.stringify(value)}\n`); +} + +function workflowRecord() { + return { + workflow_id: WORKFLOW.workflow_id, + active: { + workflow_id: WORKFLOW.workflow_id, + revision: WORKFLOW.revision, + digest: WORKFLOW_DIGEST, + }, + revisions: [ + { + revision: WORKFLOW.revision, + digest: WORKFLOW_DIGEST, + workflow: WORKFLOW, + }, + ], + workflow: WORKFLOW, + digest: WORKFLOW_DIGEST, + activation_history: [{ revision: WORKFLOW.revision, activated_at: "2026-08-05T08:31:00.000Z" }], + }; +} + +function apiResponse(pathname) { + if (pathname === "/api/v1/projects") { + return { projects: [{ id: PROJECT_ID, label: "sebastianspicker/rae" }] }; + } + if (pathname === `/api/v1/projects/${PROJECT_ID}/execution-profiles`) { + return { + profiles: [ + { + id: "local-mixed", + readiness: "ready", + models: { + economy: "openrouter/qwen3-coder", + standard: "opencode/gpt-5.2-codex", + judgment: "gpt-5.3-codex", + }, + }, + ], + }; + } + if (pathname === `/api/v1/projects/${PROJECT_ID}/runs`) return { runs: [RUN] }; + if (pathname.endsWith(`/runs/${RUN.id}/events`)) { + return { events: EVENTS, next_after: EVENTS.at(-1).seq }; + } + if (pathname === `/api/v1/projects/${PROJECT_ID}/workflows`) { + return { + workflows: [ + { + workflow_id: WORKFLOW.workflow_id, + latest_revision: WORKFLOW.revision, + latest_digest: WORKFLOW_DIGEST, + active: true, + }, + ], + }; + } + if (pathname === `/api/v1/projects/${PROJECT_ID}/workflows/templates`) { + return { templates: listWorkflowTemplates() }; + } + if (pathname === `/api/v1/projects/${PROJECT_ID}/workflows/${WORKFLOW.workflow_id}`) { + return { workflow: workflowRecord() }; + } + return null; +} + +function staticResponse(pathname) { + const relative = pathname === "/" ? "index.html" : pathname.slice(1); + const target = resolve(STATIC_ROOT, relative); + if (target !== STATIC_ROOT && !target.startsWith(`${STATIC_ROOT}${sep}`)) return null; + if (!existsSync(target)) return null; + const contentType = MIME_TYPES.get(extname(target)) ?? "application/octet-stream"; + let body = readFileSync(target); + if (target === resolve(STATIC_ROOT, "index.html")) { + body = Buffer.from( + body.toString("utf8").replace("", `${CAPTURE_STYLE}${CAPTURE_PROBE}`), + ); + } + return { body, contentType }; +} + +function createFixtureServer() { + return createServer((request, response) => { + const url = new URL(request.url ?? "/", "http://127.0.0.1"); + if (url.pathname.startsWith("/api/v1/")) { + if (request.headers.authorization !== `Bearer ${TOKEN}`) { + json(response, { error: { message: "unauthorized" } }, 401); + return; + } + if (url.pathname.endsWith("/events/stream")) { + response.writeHead(200, { + "cache-control": "no-store", + "content-type": "text/event-stream; charset=utf-8", + }); + response.end(); + return; + } + const value = apiResponse(url.pathname); + if (value) json(response, value); + else json(response, { error: { message: "fixture route not found" } }, 404); + return; + } + const file = staticResponse(url.pathname); + if (!file) { + response.writeHead(404).end("Not found\n"); + return; + } + response.writeHead(200, { + "cache-control": "no-store", + "content-type": file.contentType, + }); + response.end(file.body); + }); +} + +function browserPath() { + const candidates = [ + "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", + "/Applications/Chromium.app/Contents/MacOS/Chromium", + "/usr/bin/google-chrome", + "/usr/bin/chromium", + "/usr/bin/chromium-browser", + ]; + const found = candidates.find((candidate) => existsSync(candidate)); + if (!found) throw new Error("Chrome or Chromium is required to capture operator screenshots"); + return found; +} + +function capture(browser, url, filename, width, height) { + return new Promise((resolvePromise, reject) => { + const child = spawn( + browser, + [ + "--headless=new", + "--disable-gpu", + "--hide-scrollbars", + "--no-first-run", + "--no-default-browser-check", + "--force-device-scale-factor=1", + "--run-all-compositor-stages-before-draw", + "--virtual-time-budget=1800", + `--window-size=${width},${height}`, + `--screenshot=${resolve(SCREENSHOT_ROOT, filename)}`, + url, + ], + { stdio: ["ignore", "ignore", "pipe"] }, + ); + let errorOutput = ""; + child.stderr.setEncoding("utf8"); + child.stderr.on("data", (chunk) => { + errorOutput += chunk; + }); + child.on("error", reject); + child.on("exit", (code) => { + if (code === 0) resolvePromise(); + else reject(new Error(`browser capture failed (${code}): ${errorOutput.trim()}`)); + }); + }); +} + +function probe(browser, url) { + return new Promise((resolvePromise, reject) => { + const child = spawn( + browser, + [ + "--headless=new", + "--disable-gpu", + "--no-first-run", + "--no-default-browser-check", + "--run-all-compositor-stages-before-draw", + "--virtual-time-budget=1800", + "--dump-dom", + url, + ], + { stdio: ["ignore", "pipe", "pipe"] }, + ); + let output = ""; + let errorOutput = ""; + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + output += chunk; + }); + child.stderr.on("data", (chunk) => { + errorOutput += chunk; + }); + child.on("error", reject); + child.on("exit", (code) => { + if (code !== 0) { + reject(new Error(`browser probe failed (${code}): ${errorOutput.trim()}`)); + return; + } + if (!output.includes('data-capture-ready="true"')) { + reject(new Error("operator fixture did not reach the connected Graph view")); + return; + } + if (output.includes("data-capture-error=")) { + reject(new Error("operator fixture reported a browser error")); + return; + } + resolvePromise(); + }); + }); +} + +async function main() { + const server = createFixtureServer(); + await new Promise((resolvePromise, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", resolvePromise); + }); + try { + const address = server.address(); + const url = `http://127.0.0.1:${address.port}/#token=${TOKEN}`; + const browser = browserPath(); + await probe(browser, url); + await capture(browser, url, "evidence-dossier-desktop.png", 1360, 1600); + await capture(browser, url, "evidence-dossier-mobile.png", 390, 1400); + } finally { + await new Promise((resolvePromise) => server.close(resolvePromise)); + } +} + +await main(); diff --git a/packages/orchestration/operator/server.mjs b/packages/orchestration/operator/server.mjs index 2af9bb1..ec4b105 100644 --- a/packages/orchestration/operator/server.mjs +++ b/packages/orchestration/operator/server.mjs @@ -1,6 +1,7 @@ #!/usr/bin/env node /** Serves the authenticated loopback-only operator API and static console. */ import { createServer } from "node:http"; +import { readFileSync } from "node:fs"; import { dirname, extname, resolve } from "node:path"; import { fileURLToPath } from "node:url"; import { RunController } from "./lib/control.mjs"; @@ -15,7 +16,16 @@ import { validateRunId, } from "./lib/security.mjs"; import { discoverRuns, locateRun, paginatedEvents, publicRun } from "./lib/runs.mjs"; -import { assertRegistryMethod, workflowRegistryFor } from "./lib/workflows.mjs"; +import { + analyzeWorkflowFor, + assertRegistryMethod, + compileWorkflowTemplateFor, + workflowRegistryFor, + workflowTemplates, +} from "./lib/workflows.mjs"; +import { OperatorProfiles, loadOperatorProfiles } from "./lib/profiles.mjs"; +import { WorkflowProposalJobs } from "./lib/proposals.mjs"; +import { createRemoteOperatorProxy, MAX_REMOTE_RESPONSE_BYTES } from "./lib/remote.mjs"; import { assertSupportedNodeRuntime } from "../scripts/lib/node-runtime.mjs"; assertSupportedNodeRuntime(); @@ -32,7 +42,6 @@ const STATIC_ROOT_FILES = new Map([ ["/index.html", "index.html"], ]); const API_PREFIX = "/api/v1"; -const { readFileSync } = process.getBuiltinModule("node:fs"); function securityHeaders() { return { @@ -55,6 +64,42 @@ function sendJson(res, status, value) { res.end(`${JSON.stringify(value)}\n`); } +async function sendRemoteResponse(req, res, upstream) { + res.writeHead(upstream.status, { + ...securityHeaders(), + "content-type": upstream.contentType, + ...(upstream.body ? { "content-length": upstream.body.length } : {}), + }); + if (!upstream.stream) { + res.end(upstream.body); + return; + } + const reader = upstream.stream.getReader(); + let size = 0; + let closed = false; + const finish = async () => { + if (closed) return; + closed = true; + await reader.cancel().catch(() => {}); + res.end(); + }; + const timeout = setTimeout(() => void finish(), 15_000); + timeout.unref?.(); + req.once("close", () => void finish()); + try { + while (!closed) { + const { value, done } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > MAX_REMOTE_RESPONSE_BYTES) break; + res.write(Buffer.from(value)); + } + } finally { + clearTimeout(timeout); + await finish(); + } +} + function errorResponse(res, error) { const status = Number.isInteger(error?.status) ? error.status : 500; const message = status >= 500 ? "internal server error" : error.message; @@ -149,7 +194,7 @@ function streamEvents(req, res, run, after) { } async function routeApi(req, res, url, context) { - const { projects, token, controller, host, origin } = context; + const { projects, token, controller, host, origin, remote } = context; const loopback = validateLoopbackRequest(req, { host, origin, @@ -163,88 +208,70 @@ async function routeApi(req, res, url, context) { sendJson(res, 401, { error: { status: 401, message: "bearer authentication required" } }); return; } - return routeAuthorizedApi(req, res, url, context, projects, controller); -} + if (remote) { + const upstream = await remote.forward(req, url); + await sendRemoteResponse(req, res, upstream); + return; + } -async function routeAuthorizedApi(req, res, url, context, projects, controller) { const parts = splitPath(url.pathname); - assertApiPath(parts); - if (isProjectsEndpoint(parts)) { - return routeProjects(req, res, projects, controller); - } - const project = projectForRoute(projects, parts); - if (parts[4] === "workflows") { - return routeWorkflows(req, res, url, context, project, parts.slice(5)); + if (parts[0] !== "api" || parts[1] !== "v1") + throw Object.assign(new Error("not found"), { status: 404 }); + if (parts.length === 3 && parts[2] === "projects") { + requireMethod(req, "GET"); + sendJson(res, 200, { + projects: projects.map(({ id, label }) => ({ id, label })), + active_run_id: controller.refreshOwnership(), + }); + return; } - if (parts[4] !== "runs") throw Object.assign(new Error("not found"), { status: 404 }); - return routeRuns(req, res, url, project, controller, parts); -} - -function assertApiPath(parts) { - if (parts[0] === "api" && parts[1] === "v1") return; - throw Object.assign(new Error("not found"), { status: 404 }); -} - -function isProjectsEndpoint(parts) { - return parts.length === 3 && parts[2] === "projects"; -} - -function projectForRoute(projects, parts) { if (parts[2] !== "projects" || !parts[3]) throw Object.assign(new Error("not found"), { status: 404 }); const project = findProject(projects, parts[3]); if (!project) throw Object.assign(new Error("project not found"), { status: 404 }); - return project; -} - -function routeProjects(req, res, projects, controller) { - requireMethod(req, "GET"); - sendJson(res, 200, { - projects: projects.map(({ id, label }) => ({ id, label })), - active_run_id: controller.refreshOwnership(), - }); -} + if (parts[4] === "execution-profiles" && parts.length === 5) { + requireMethod(req, "GET"); + sendJson(res, 200, { profiles: (context.profiles ?? new OperatorProfiles()).list() }); + return; + } + if (parts[4] === "workflows") { + await routeWorkflows(req, res, url, context, project, parts.slice(5)); + return; + } + if (parts[4] !== "runs") throw Object.assign(new Error("not found"), { status: 404 }); -async function routeRuns(req, res, url, project, controller, parts) { if (parts.length === 5) { - return routeRunCollection(req, res, url, project, controller); + if (req.method === "GET") { + const cursor = positiveInteger(url.searchParams.get("cursor"), 0, 1_000_000); + const limit = positiveInteger(url.searchParams.get("limit"), 30, 100); + controller.refreshOwnership(); + const all = discoverRuns(project); + const page = all + .slice(cursor, cursor + limit) + .map((run) => publicRun(run, controller.ownedRunId)); + sendJson(res, 200, { + runs: page, + next_cursor: cursor + page.length < all.length ? cursor + page.length : null, + }); + return; + } + requireMethod(req, "POST"); + const body = await readJsonBody(req); + const executionProfile = (context.profiles ?? new OperatorProfiles()).resolve( + body.execution_profile_id, + ); + sendJson(res, 202, controller.start(project, body, executionProfile)); + return; } + const runId = validateRunId(parts[5]); if (parts.length === 6) { - return routeRunDetail(req, res, project, controller, runId); + requireMethod(req, "GET"); + controller.refreshOwnership(); + sendJson(res, 200, { run: publicRun(locateRun(project, runId), controller.ownedRunId) }); + return; } - return routeRunAction(req, res, url, project, controller, runId, parts); -} - -function runPage(url, project, controller) { - const cursor = positiveInteger(url.searchParams.get("cursor"), 0, 1_000_000); - const limit = positiveInteger(url.searchParams.get("limit"), 30, 100); - controller.refreshOwnership(); - const all = discoverRuns(project); - const runs = all - .slice(cursor, cursor + limit) - .map((run) => publicRun(run, controller.ownedRunId)); - const nextCursor = cursor + runs.length < all.length ? cursor + runs.length : null; - return { runs, next_cursor: nextCursor }; -} - -async function routeRunCollection(req, res, url, project, controller) { - if (req.method === "GET") return sendJson(res, 200, runPage(url, project, controller)); - return startRun(req, res, project, controller); -} - -async function startRun(req, res, project, controller) { - requireMethod(req, "POST"); - sendJson(res, 202, controller.start(project, await readJsonBody(req))); -} -function routeRunDetail(req, res, project, controller, runId) { - requireMethod(req, "GET"); - controller.refreshOwnership(); - sendJson(res, 200, { run: publicRun(locateRun(project, runId), controller.ownedRunId) }); -} - -async function routeRunAction(req, res, url, project, controller, runId, parts) { const action = parts[6]; if (action === "events" && parts.length === 7) { requireMethod(req, "GET"); @@ -262,23 +289,18 @@ async function routeRunAction(req, res, url, project, controller, runId, parts) requireMethod(req, "POST"); if (parts.length !== 7) throw Object.assign(new Error("not found"), { status: 404 }); const body = await readJsonBody(req); - return routeRunControlAction(res, action, project, controller, runId, body); -} - -function routeRunControlAction(res, action, project, controller, runId, body) { - switch (action) { - case "stop": - return sendJson(res, 200, { control: controller.stop(project, runId) }); - case "resume": - return sendJson(res, 202, controller.resume(project, runId)); - case "interrupt": - return sendJson(res, 202, controller.interrupt(project, runId, body)); - case "checkpoint-decision": - return sendJson(res, 200, { checkpoint: controller.decideCheckpoint(project, runId, body) }); - case "cleanup": - return sendJson(res, 202, controller.cleanup(project, runId, body)); - default: - throw Object.assign(new Error("not found"), { status: 404 }); + if (action === "stop") { + sendJson(res, 200, { control: controller.stop(project, runId) }); + } else if (action === "resume") { + sendJson(res, 202, controller.resume(project, runId)); + } else if (action === "interrupt") { + sendJson(res, 202, controller.interrupt(project, runId, body)); + } else if (action === "checkpoint-decision") { + sendJson(res, 200, { checkpoint: controller.decideCheckpoint(project, runId, body) }); + } else if (action === "cleanup") { + sendJson(res, 202, controller.cleanup(project, runId, body)); + } else { + throw Object.assign(new Error("not found"), { status: 404 }); } } @@ -309,6 +331,31 @@ async function routeWorkflows(req, res, url, context, project, tail) { sendJson(res, 200, { workflows: await assertRegistryMethod(registry, "list")() }); return; } + if (tail[0] === "templates" && tail.length === 1) { + if (req.method === "GET") { + sendJson(res, 200, { templates: await workflowTemplates() }); + return; + } + requireMethod(req, "POST"); + const body = await readJsonBody(req); + const allowed = new Set(["template_id", "workflow_id", "revision", "title"]); + if ( + !body || + typeof body !== "object" || + Array.isArray(body) || + Object.keys(body).some((key) => !allowed.has(key)) + ) { + throw Object.assign(new Error("invalid workflow template request"), { status: 400 }); + } + sendJson(res, 200, { + workflow: await compileWorkflowTemplateFor(body.template_id, { + workflow_id: body.workflow_id, + revision: body.revision, + ...(body.title ? { title: body.title } : {}), + }), + }); + return; + } const workflowId = tail[0]; if (!/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/.test(workflowId)) { throw Object.assign(new Error("invalid workflow id"), { status: 400 }); @@ -318,6 +365,48 @@ async function routeWorkflows(req, res, url, context, project, tail) { sendJson(res, 200, { workflow: await assertRegistryMethod(registry, "show")(workflowId) }); return; } + if (tail[1] === "analysis" && tail.length === 2) { + requireMethod(req, "POST"); + const body = await readJsonBody(req); + if ( + !body || + typeof body !== "object" || + Array.isArray(body) || + Object.keys(body).some((key) => key !== "workflow") + ) { + throw Object.assign(new Error("analysis accepts only a workflow object"), { status: 400 }); + } + sendJson(res, 200, await analyzeWorkflowFor(body.workflow)); + return; + } + if (tail[1] === "proposals" && tail.length === 2) { + requireMethod(req, "POST"); + const body = await readJsonBody(req); + const executionProfile = (context.profiles ?? new OperatorProfiles()).resolve( + body.execution_profile_id, + ); + sendJson( + res, + 202, + (context.proposalJobs ?? new WorkflowProposalJobs()).submit({ + project, + workflowId, + body, + executionProfile, + }), + ); + return; + } + if (tail[1] === "proposals" && tail.length === 3) { + requireMethod(req, "GET"); + if (!/^proposal-[a-f0-9-]{36}$/.test(tail[2])) { + throw Object.assign(new Error("invalid proposal job id"), { status: 400 }); + } + sendJson(res, 200, { + proposal: (context.proposalJobs ?? new WorkflowProposalJobs()).get(tail[2], workflowId), + }); + return; + } if (tail[1] === "drafts" && tail.length === 2) { requireMethod(req, "POST"); workflowMutationAllowed(project, context.controller); @@ -383,11 +472,26 @@ export async function handleOperatorRequest(req, res, context) { } export function createOperatorServer({ - projects, + projects = [], token = createSessionToken(), controller = new RunController(), + remoteUrl = null, + tokenFile = null, + fetchImpl, + profiles = new OperatorProfiles(), + proposalJobs = new WorkflowProposalJobs(), }) { - if (!Array.isArray(projects) || projects.length === 0) throw new Error("projects are required"); + if (!Array.isArray(projects)) throw new Error("projects must be an array"); + if (remoteUrl && projects.length) + throw new Error("--remote-url cannot be combined with --project"); + if (!remoteUrl && tokenFile) throw new Error("--token-file requires --remote-url"); + if (!remoteUrl && projects.length === 0) throw new Error("projects are required"); + if (!profiles || typeof profiles.list !== "function" || typeof profiles.resolve !== "function") { + throw new Error("profiles must be an OperatorProfiles instance"); + } + const remote = remoteUrl + ? createRemoteOperatorProxy({ remoteUrl, tokenFile, ...(fetchImpl ? { fetchImpl } : {}) }) + : null; const server = createServer(async (req, res) => { const address = server.address(); if (!address || typeof address === "string") { @@ -399,6 +503,9 @@ export function createOperatorServer({ projects, token, controller, + profiles, + proposalJobs, + remote, host, origin: `http://${host}`, }); @@ -409,35 +516,61 @@ export function createOperatorServer({ function parseCli(argv) { const paths = []; let port = 0; - const args = argv.values(); - for (let arg = args.next(); !arg.done; arg = args.next()) { - if (arg.value === "--project") { - const value = args.next().value; + let remoteUrl = null; + let tokenFile = null; + const executionProfiles = []; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--project") { + const value = argv[index + 1]; if (!value) throw new Error("--project requires a path"); paths.push(value); - } else if (arg.value === "--port") { - port = Number(args.next().value); + index += 1; + } else if (arg === "--port") { + port = Number(argv[index + 1]); if (!Number.isInteger(port) || port < 0 || port > 65535) throw new Error("invalid --port"); - } else if (arg.value === "--help" || arg.value === "-h") { - writeUsage(); + index += 1; + } else if (arg === "--remote-url") { + const value = argv[index + 1]; + if (!value) throw new Error("--remote-url requires a URL"); + remoteUrl = value; + index += 1; + } else if (arg === "--token-file") { + const value = argv[index + 1]; + if (!value) throw new Error("--token-file requires a path"); + tokenFile = value; + index += 1; + } else if (arg === "--execution-profile") { + const value = argv[index + 1]; + if (!value) throw new Error("--execution-profile requires a path"); + executionProfiles.push(value); + index += 1; + } else if (arg === "--help" || arg === "-h") { + process.stdout.write( + "Usage: node operator/server.mjs (--project [--project ] | --remote-url --token-file ) [--execution-profile ] [--port 0]\n", + ); return null; } else { - throw new Error(`unknown argument: ${arg.value}`); + throw new Error(`unknown argument: ${arg}`); } } - return { projects: createProjectRegistry(paths), port }; -} - -function writeUsage() { - process.stdout.write( - "Usage: node operator/server.mjs --project PROJECT_ROOT [--project PROJECT_ROOT] [--port PORT]\n", - ); + if (remoteUrl && paths.length) throw new Error("--remote-url cannot be combined with --project"); + if (remoteUrl && !tokenFile) throw new Error("--token-file is required with --remote-url"); + if (!remoteUrl && tokenFile) throw new Error("--token-file requires --remote-url"); + return { + projects: remoteUrl ? [] : createProjectRegistry(paths), + port, + remoteUrl, + tokenFile, + executionProfiles, + }; } async function main() { const options = parseCli(process.argv.slice(2)); if (!options) return; - const instance = createOperatorServer({ projects: options.projects }); + const profiles = await loadOperatorProfiles(options.executionProfiles); + const instance = createOperatorServer({ ...options, profiles }); await new Promise((resolveListen, reject) => { instance.server.once("error", reject); instance.server.listen(options.port, "127.0.0.1", resolveListen); diff --git a/packages/orchestration/operator/static/css/shell.css b/packages/orchestration/operator/static/css/shell.css index 57dc1da..575dfef 100644 --- a/packages/orchestration/operator/static/css/shell.css +++ b/packages/orchestration/operator/static/css/shell.css @@ -2,6 +2,22 @@ * Shell: centered case frame, head bar, project switcher, primary actions. */ +.skip { + position: fixed; + top: 0.5rem; + left: 0.5rem; + z-index: 40; + padding: 0.45rem 0.65rem; + border: 1px solid var(--ink); + background: var(--sheet); + color: var(--ink); + transform: translateY(calc(-100% - 0.75rem)); +} + +.skip:focus-visible { + transform: translateY(0); +} + .runboard, .frame.runboard, .runboard.frame, diff --git a/packages/orchestration/operator/static/css/workflows.css b/packages/orchestration/operator/static/css/workflows.css index 80b7a13..1c90dcd 100644 --- a/packages/orchestration/operator/static/css/workflows.css +++ b/packages/orchestration/operator/static/css/workflows.css @@ -7,6 +7,7 @@ .workflow-rail h3,.workflow-inspector h3 { margin-top:0; } .workflow-canvas { min-width:0; } #workflow-graph { display:block; width:100%; min-height:14rem; border:1px solid var(--line, #c7d0d4); background:#f7fafb; } +#workflow-graph-panel { overflow-x:auto; } .workflow-node { fill:#fff; stroke:#17324d; stroke-width:2; rx:6; } .workflow-node--join,.workflow-node--fan-out { fill:#e7f3f6; } .workflow-node--gate,.workflow-node--checkpoint { fill:#fff4dc; } @@ -19,12 +20,27 @@ .workflow-edge-label { fill:#435466; font:10px ui-monospace,monospace; text-anchor:middle; paint-order:stroke; stroke:#f7fafb; stroke-width:3px; } .workflow-node-badge { fill:#506274; font:10px ui-monospace,monospace; } .workflow-form { display:grid; gap:.55rem; margin-top:.75rem; } +.workflow-views,.workflow-control-row,.workflow-checks { display:flex; flex-wrap:wrap; gap:.45rem; margin:.75rem 0; } +.workflow-views [role="tab"][aria-selected="true"] { background:#17324d; color:#fff; } +.workflow-view { min-width:0; } +.workflow-structured-controls { display:grid; gap:.45rem; padding:.75rem; border:1px solid var(--line, #c7d0d4); } +.workflow-structured-controls h3 { margin:0; } +.workflow-inspector-grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:.5rem; } +.workflow-inspector-grid label { display:grid; gap:.2rem; min-width:0; } +.workflow-inspector input:not([type="checkbox"]),.workflow-inspector select,.workflow-inspector textarea { width:100%; min-width:0; } +.workflow-checks label { display:flex; align-items:center; gap:.35rem; } +.workflow-checks input { flex:none; } +.workflow-analysis-output { max-height:18rem; overflow:auto; } +#workflow-analysis-output { min-height:7rem; white-space:pre-wrap; border:1px solid var(--line, #c7d0d4); padding:.5rem; } .workflow-form textarea,#workflow-diff-output { width:100%; box-sizing:border-box; font: .82rem ui-monospace, monospace; } .workflow-form textarea { min-height:9rem; } .workflow-rail button { display:block; width:100%; margin:.25rem 0; text-align:left; } #workflow-history { padding-left:1.2rem; } +.workflow-inspector dd { overflow-wrap:anywhere; } .workflow-structure { display:grid; gap:.65rem; max-height:22rem; overflow:auto; } .workflow-structure table { width:100%; border-collapse:collapse; font-size:.82rem; } .workflow-structure th,.workflow-structure td { padding:.35rem; border-bottom:1px solid var(--line, #c7d0d4); text-align:left; vertical-align:top; } #workflow-diff-output { min-height:5rem; white-space:pre-wrap; border:1px solid var(--line, #c7d0d4); padding:.5rem; } -@media (max-width: 900px) { .workflow-editor__grid { grid-template-columns:1fr; } .workflow-rail { max-height:12rem; overflow:auto; } } +@media (prefers-reduced-motion: reduce) { .workflow-editor * { scroll-behavior:auto; transition:none !important; } } +@media (max-width: 900px) { .workflow-editor { max-width:100%; } .workflow-editor__grid { grid-template-columns:1fr; min-width:0; max-width:100%; } .workflow-rail { max-height:12rem; overflow:auto; } } +@media (max-width: 640px) { .workflow-editor__head { align-items:flex-start; flex-direction:column; } .workflow-inspector-grid { grid-template-columns:1fr; } .workflow-canvas input,.workflow-canvas select,.workflow-canvas textarea { width:100%; max-width:100%; } .workflow-control-row button { flex:1 1 9rem; } } diff --git a/packages/orchestration/operator/static/index.html b/packages/orchestration/operator/static/index.html index ce78d0e..5aef099 100644 --- a/packages/orchestration/operator/static/index.html +++ b/packages/orchestration/operator/static/index.html @@ -297,15 +297,43 @@

Registry

Loading workflows…

+
+ + + + +
+ +
Workflow stage map A topologically layered workflow map. Color and line style distinguish data, stream, condition, sequence, and loop-back edges. The tables after the map provide the complete equivalent structure and live instance state. +
+
- - + +
+

Structured authoring

+ + +
+ + + + + +
@@ -320,6 +348,11 @@

Structured nodes and edges