diff --git a/AGENTS.md b/AGENTS.md index 37b9b5a..b0443e3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,9 +8,13 @@ deployment. Daimon owns only the per-agent runtime boundary. ## Structure +- `docs/` explains the current package surface and runtime behavior. +- `archive/` preserves historical plans and audits that are no longer current + operating documentation. - `src/core/` defines per-agent harness contracts. - `src/pi/` implements the contract using Pi's SDK. - `src/observability/` records local agent/org activity traces. +- `src/runtime/` owns the organization-runtime host and durable control plane. - `src/examples/` contains runnable local examples and E2E checks. ## Rules diff --git a/README.md b/README.md index 4661f7e..60cf74b 100644 --- a/README.md +++ b/README.md @@ -342,14 +342,13 @@ The Pi E2E uses the local Codex CLI subscription auth file to seed an ignored Pi `auth.json` under `.runtime/`. These are live runs: they spend real tokens and require local engine auth -(`~/.codex/auth.json` for Pi/Codex; mixed-engine and triad additionally need +(`~/.codex/auth.json` for Pi/Codex; triad additionally needs authenticated `grok` and `agy` CLIs on PATH). They are not part of `npm test`. ```bash npm install npm run e2e:pi-agent npm run e2e:pi-memory-org -npm run e2e:mixed-engine-org npm run e2e:jungian-play-org npm run e2e:jungian-triad-org ``` @@ -381,14 +380,15 @@ speaks through a Pi agent seeded from local Codex subscription auth. All three selves carry the same full archetype set, rotated through the run so every archetype gets consulted. -## Design Notes - -- `MEMORY-SYSTEM.md` describes the implemented scoped memory runtime. -- `ENGINE-SYSTEM.md` describes the engine abstraction plan: Pi, local/API - model providers, and CLI-backed engines such as `codex`, `claude`, `grok`, - and `agy`. -- Mneme is a sibling package, `@noopolis/mneme`, published separately and used - by Daimon in-process for Pi agents. Other runtimes can use Mneme through its - MCP server. The agent-facing tools stay named `memory_search`, - `memory_register`, and `memory.*` at the protocol boundary because those names - are clearer to agents. +## Docs + +Start with `docs/README.md` for the current documentation map: + +- `docs/runtime.md` explains the organization-runtime config, control API, + schedules, durable wake acceptance, and attention. +- `docs/engines.md` explains Pi and CLI engine boundaries. +- `docs/memory.md` explains Daimon's Mneme adapter. +- `docs/observability.md` explains causal events, raw capture, and world + trajectories. + +Historical migration plans and audits live under `archive/`. diff --git a/archive/README.md b/archive/README.md new file mode 100644 index 0000000..67980ea --- /dev/null +++ b/archive/README.md @@ -0,0 +1,22 @@ +# Archive + +This directory preserves historical design plans, audits, and migration notes +that are no longer current operating documentation. + +Current Daimon docs live in: + +- `docs/README.md` for navigation. +- `docs/runtime.md` for the organization runtime and control API. +- `docs/engines.md` for Pi, Codex, Grok, and AGY engine boundaries. +- `docs/memory.md` for Mneme integration. +- `docs/observability.md` for causal events, raw capture, and world trajectories. + +Archived files are kept verbatim except for their path in the repository. Treat +them as historical evidence, not current authority. + +## repository-layout-cleanup + +- `MIGRATION_PLAN.md` - historical migration plan. +- `MEMORY-SYSTEM.md` - historical memory-system design note. +- `ENGINE-SYSTEM.md` - historical engine-system plan. +- `docs-audit.md` - historical documentation audit. diff --git a/ENGINE-SYSTEM.md b/archive/repository-layout-cleanup/ENGINE-SYSTEM.md similarity index 100% rename from ENGINE-SYSTEM.md rename to archive/repository-layout-cleanup/ENGINE-SYSTEM.md diff --git a/MEMORY-SYSTEM.md b/archive/repository-layout-cleanup/MEMORY-SYSTEM.md similarity index 100% rename from MEMORY-SYSTEM.md rename to archive/repository-layout-cleanup/MEMORY-SYSTEM.md diff --git a/MIGRATION_PLAN.md b/archive/repository-layout-cleanup/MIGRATION_PLAN.md similarity index 100% rename from MIGRATION_PLAN.md rename to archive/repository-layout-cleanup/MIGRATION_PLAN.md diff --git a/docs-audit.md b/archive/repository-layout-cleanup/docs-audit.md similarity index 100% rename from docs-audit.md rename to archive/repository-layout-cleanup/docs-audit.md diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..7ecd4bd --- /dev/null +++ b/docs/README.md @@ -0,0 +1,38 @@ +# Daimon Docs + +Daimon is the per-agent runtime harness for Noopolis. Each harness runs +one prepared agent; the optional runtime host routes wakes to several harnesses +with separate agent homes and queues. + +Current docs: + +- [runtime.md](runtime.md) - runtime config, control endpoints, schedules, durable wake + acceptance, and attention. +- [engines.md](engines.md) - Pi harnessing, CLI engine boundaries, auth storage, and live + checks. +- [memory.md](memory.md) - how Daimon adapts Mneme without owning memory semantics. +- [observability.md](observability.md) - causal events, raw training capture, and redacted world + trajectories. + +Historical plans and audits were moved to the [archive](../archive/). +They are preserved for context, but the files above and the source contracts are +the current guide. + +Source map: + +- [`src/core/`](../src/core/) - runtime-neutral harness contracts. +- [`src/pi/`](../src/pi/) - Pi adapter, prompts, world tools, memory tools, wake handling, + and engine-session adapters. +- [`src/runtime/`](../src/runtime/) - organization-runtime config, HTTP control process, durable + wake acceptance, schedules, attention, production tool mounting, and engine + readiness. +- [`src/contracts/`](../src/contracts/) - data-only JSON Schema and contract constants. +- [`src/observability/`](../src/observability/) - causal event helpers and org observation. +- [`src/mcp/`](../src/mcp/) - tool server used by production MCP mounts. + +Public package entrypoints: + +- `@noopolis/daimon` +- `@noopolis/daimon/pi` +- `@noopolis/daimon/runtime` +- `@noopolis/daimon/observability` diff --git a/docs/engines.md b/docs/engines.md new file mode 100644 index 0000000..313ef4f --- /dev/null +++ b/docs/engines.md @@ -0,0 +1,88 @@ +# Engines + +[Documentation index](README.md) + +Daimon has two layers: + +- The runtime-neutral harness contract in `src/core`. +- Engine adapters under `src/pi` and `src/runtime`. + +The simple one-agent contract is: + +```ts +import type { AgentHarnessAdapter, WakeEvent } from "@noopolis/daimon"; +``` + +`AgentHarnessAdapter.startAgent()` receives one agent's id, name, +instructions, workspace path, runtime home, and optional tool names. The +returned handle accepts one `WakeEvent` at a time and returns text, duration, +and status. + +## Pi + +`@noopolis/daimon/pi` exports `PiHarnessAdapter` and helpers for Pi auth, +models, world tools, and traces: + +```ts +import { PiHarnessAdapter } from "@noopolis/daimon/pi"; +``` + +The Pi adapter creates the runtime-home subdirectories it needs, creates the +workspace if absent, resolves a Pi model, mounts optional Mneme and world tools, +and creates awake or dream sessions. Caller instructions are included in the +system prompt; SOUL or identity-like standing instructions belong in the +agent's `instructions` field supplied by the caller. + +Model intents support built-in Pi providers and explicit OpenAI-compatible or +Anthropic-compatible endpoints. Endpoint auth is limited to `none` or +`api_key`; Codex and Claude subscription helpers write Pi auth storage instead. + +Auth helpers: + +- `seedPiOpenAICodexAuthFromCodex` +- `seedPiAnthropicAuthFromClaudeCode` +- `seedPiApiKeyAuth` +- `createPiOpenAICodexAuthFromCodexToken` + +## CLI Engines + +The organization runtime accepts `codex`, `grok`, and `agy` engine intents. It +does not accept arbitrary commands or environment maps in config. + +At startup and before each wake, Daimon resolves the executable from a safe +`PATH`, pins its file identity, probes `--version`, and verifies the selected +auth boundary. + +Codex uses a private `.codex/auth.json` under each agent runtime home. Optional +Codex config fields are `model`, `reasoningEffort`, and the fixed no-network +workspace sandbox policy. + +The production Grok path uses an external Daimon engine broker with one durable +subscription credential authority. Agent workers receive scoped capabilities; +the broker owns refresh and stale-credential recovery. The runtime checks broker +readiness before admitting Grok agents and verifies their sandbox policy before +turns. The older credential-lease helper is not the production host path. + +AGY uses OS-native secure storage through one private D-Bus and Secret Service +realm. Enroll it once with: + +```bash +daimon-runtime auth agy login --config /runtime/daimon-runtime.json +``` + +Daimon does not accept portable AGY token files, API keys, ADC credentials, or +ambient D-Bus sessions. + +## Live Checks + +`npm test`, `npm run typecheck`, and `npm run build` do not call model +providers. + +Live scripts and examples spend real tokens and require local subscription +auth: + +- `npm run live:grok-broker` +- `npm run e2e:pi-agent` +- `npm run e2e:pi-memory-org` +- `npm run e2e:jungian-play-org` +- `npm run e2e:jungian-triad-org` diff --git a/docs/memory.md b/docs/memory.md new file mode 100644 index 0000000..7a29768 --- /dev/null +++ b/docs/memory.md @@ -0,0 +1,61 @@ +# Memory + +[Documentation index](README.md) + +Daimon adapts Mneme into agent turns; it does not own memory storage, policy, +recall, redaction, or MCP. Those semantics live in `@noopolis/mneme`. + +For Pi agents, enable memory on `PiHarnessAdapter`: + +```ts +import { PiHarnessAdapter } from "@noopolis/daimon/pi"; + +const adapter = new PiHarnessAdapter({ + authPath: "/runtime/pi-auth/auth.json", + memory: { + runtimeHomePath: "/runtime/memory/writer", + source: "daimon", + tokenBudget: 20000 + } +}); +``` + +If `memory.runtimeHomePath` is omitted, the adapter uses the agent's +`runtimeHomePath`. Supplying an explicit memory runtime home lets several Pi +or CLI sessions share one Mneme bank while keeping their engine homes separate. + +Per turn, Daimon asks Mneme to prepare the memory context and then exposes only +Mneme's model-facing tools for the active trusted context. The context is +checked against: + +- the current agent id; +- the Mneme authority bank id; +- the active principal scope; +- a bounded set of canonical allowed scopes. + +Tool arguments are validated against Mneme's tool schemas before execution. +Unexpected top-level fields are rejected. + +Awake wakes use normal working memory behavior. Dream wakes use a fresh +one-off session under `sessions/dream/-` and prepend dream +guidance for consolidation work. + +Daimon does not automatically write every turn into memory. Agents persist +memory only by calling Mneme tools such as `memory_register`, +`memory_summarize`, and `memory_forget`. + +Organization-runtime config can declare memory per agent: + +```json +{ + "memory": { + "runtimeHomePath": "/runtime/memory/writer", + "source": "daimon", + "tokenBudget": 20000 + } +} +``` + +In production CLI engines, the runtime passes this memory config into the same +Pi harness layer that mounts production tools. The memory runtime remains +in-process for Daimon; other runtimes can use Mneme through its own MCP server. diff --git a/docs/WORLD_TRAJECTORIES.md b/docs/observability.md similarity index 74% rename from docs/WORLD_TRAJECTORIES.md rename to docs/observability.md index 26da46b..71e1a5c 100644 --- a/docs/WORLD_TRAJECTORIES.md +++ b/docs/observability.md @@ -1,4 +1,21 @@ -# World trajectories +# Observability + +[Documentation index](README.md) + +Daimon records runtime evidence at several layers. The important boundary is +privacy: full cognition capture is private and opt-in, while causal events and +world trajectories are minimized surfaces for auditing and joins. + +## Causal events + +`@noopolis/daimon/observability` exports helpers for Noopolis causal events and +the org observer. Pi turns also stamp input and output causal events from +`src/pi/turnCausal.ts`. + +The causal layer records event identity, emitter, principal, causes, type, and +payload. It is the join surface for Moltnet, Mneme, Simfile, and Daimon traces. + +## Capture surfaces Two deliberately separate capture surfaces exist: @@ -15,7 +32,7 @@ is stored under `private-training/pi/raw/turns/` with `0700` directories and `0600` files; it is outside ordinary telemetry and is never exported by default. -The capture reuses Pi rather than building a parallel cognition recorder: +Raw capture reuses Pi rather than building a parallel cognition recorder: - `pi-session.jsonl` is copied byte-for-byte from Pi's native `SessionManager`. @@ -37,9 +54,9 @@ four files and permissions are complete; a failed publication is not retried against the same immutable turn path. Retention deletes the oldest per-turn capture after the configured maximum. -Stable run/tick/wake identifiers are recorded only as join metadata. +Stable run, tick, and wake identifiers are recorded only as join metadata. Authoritative post-action physics outcomes remain Simfile-owned and are joined -separately; the raw artifact never becomes simulation authority. +separately. The raw artifact never becomes simulation authority. ## Redacted world trajectory @@ -66,3 +83,8 @@ Pi also cannot observe later mechanical effects that happen after an action receipt. Simfile may join public contact, kick, goal, score, or next-state facts through the exported receipt identifiers. Until that join exists, `outcome.status` is `pending_world_join`; Daimon does not invent a reward. + +The schema constant is `daimon.world_trajectory.v1`. The implementation lives +in `src/pi/worldTrajectory.ts`; world tool protocol and authority handling live +beside it in `src/pi/worldTools.ts`, `src/pi/worldToolProtocol.ts`, and +`src/pi/worldNudge.ts`. diff --git a/docs/runtime.md b/docs/runtime.md new file mode 100644 index 0000000..9574a5a --- /dev/null +++ b/docs/runtime.md @@ -0,0 +1,89 @@ +# Runtime + +[Documentation index](README.md) + +Daimon's organization runtime hosts isolated agents from one strict config. It +does not know team structure, org graphs, Moltnet room policy, deployment, or +Spawnfile compilation. + +Use the package subpath: + +```ts +import { + createOrganizationRuntimeHost, + parseOrganizationRuntimeConfig +} from "@noopolis/daimon/runtime"; +``` + +The CLI entrypoint is: + +```bash +daimon-runtime run --config /runtime/daimon-runtime.json +``` + +The config parser accepts `noopolis.daimon.organization-runtime.v1` and `v2`. +Both require a host and one to 32 agents. Each agent has: + +- `id`, `name`, and `instructions`. +- absolute `workspacePath` and `runtimeHomePath`. +- `engine.kind` of `codex`, `grok`, or `agy`. +- optional `mcp`, `moltnet`, `memory`, and `attention`. +- for v2 only, one `schedule`. + +Configuration must not contain credentials, arbitrary environment maps, command +arrays for engines, process handles, roles, teams, or parent/member links. The +parser normalizes paths and rejects overlapping workspace/runtime-home roots. +Before starting agents, the host verifies caller-created directories are real, +current-user-owned, and not unsafe through symlinks or writable permissions. + +The control token is named by `host.controlTokenEnv`; only the variable name is +stored in config. HTTP requests use `Authorization: Bearer `. + +Runtime endpoints: + +- `GET /healthz` is unauthenticated process health. +- `POST /v1/wake` runs one synchronous wake for non-attention agents. +- `GET /v1/health` returns host and agent health. +- `GET /v1/activity` returns bounded activity. + +Set `DAIMON_RUNTIME_ACCEPTANCE_STORE` to enable the durable v2 control plane: + +- `POST /v2/wakes` fsyncs an accepted wake before execution. +- `GET /v2/wake-receipts/` returns redacted lifecycle status. +- `GET /v2/activity` includes durable receipts and active executions. +- `GET /v2/availability` reports running, pending, deferred, and budget state. + +Equal delivery retries return the original acceptance. A changed payload for the +same delivery id is rejected. Accepted delivery is durable at-least-once turn +delivery; destinations that require exactly-once effects must deduplicate their +own external side effects. + +Attention is opt-in per agent: + +```json +{ + "attention": { + "maxBatchMessages": 8, + "maxBatchBytes": 12000, + "maxExecutions": 30, + "maxTokens": 3000000 + } +} +``` + +Attention agents require the durable v2 route. They receive `daimon_inbox` and +`daimon_inbox_disposition`; reading does not complete a message, and unmarked +or deferred deliveries stay pending. + +Version 2 schedules are normalized on the agent: + +- `{ "kind": "disabled" }` +- `{ "kind": "every", "interval_ms": 60000, "prompt": "..." }` +- `{ "kind": "cron", "cron": "0 9 * * 1", "timezone": "Europe/Berlin", "prompt": "..." }` + +Optional `jitter_seconds` is bounded to one hour. Schedules are runtime-native +delivery sources; they do not give Daimon org-graph authority. + +`reconcileOfflineWakeTransition` is a library operation for deployment-admin +recovery of a blocked durable store. It is not an HTTP endpoint or normal host +action. diff --git a/package.json b/package.json index 12614df..dc42793 100644 --- a/package.json +++ b/package.json @@ -39,19 +39,20 @@ "access": "public" }, "scripts": { - "build": "rm -rf dist && tsc --project tsconfig.build.json && node --import tsx scripts/emitRuntimeContractManifest.mjs && node src/runtime/native/copyArtifact.mjs", + "build": "rm -rf dist && tsc --project tsconfig.build.json && node --import tsx scripts/emitRuntimeContractManifest.ts && node --experimental-strip-types src/runtime/native/copyArtifact.ts", "build:test-runtime": "rm -rf dist-test-runtime && tsc --project tsconfig.test-runtime.json", "clean": "rm -rf dist", "prepack": "npm run build", "prepublishOnly": "npm run typecheck && npm test", - "typecheck": "tsc --project tsconfig.json --noEmit", - "verify:contract-manifest": "node --import tsx scripts/emitRuntimeContractManifest.mjs --check", - "verify:production-closure": "node scripts/verifyProductionClosure.mjs", - "test": "node --import tsx --test \"src/**/*.test.ts\"", + "typecheck": "tsc --project tsconfig.json --noEmit && tsc --project scripts/tsconfig.json --noEmit", + "verify:contract-manifest": "node --import tsx scripts/emitRuntimeContractManifest.ts --check", + "verify:native": "node --experimental-strip-types src/runtime/native/verifyArtifacts.ts", + "verify:production-closure": "node --experimental-strip-types scripts/verifyProductionClosure.ts", + "test": "node --import tsx --test \"src/**/*.test.ts\" scripts/scriptSourcePolicy.test.ts", "emit-causal-fixture": "tsx src/observability/emitCausalFixture.ts", "emit-causal-fixture:spoof": "tsx src/observability/emitCausalFixture.ts --spoof", - "live:codex-session": "node --import tsx scripts/liveCodexSession.mjs", - "live:grok-broker": "node --import tsx scripts/liveGrokBrokerSession.mjs", + "live:codex-session": "node --import tsx scripts/liveCodexSession.ts", + "live:grok-broker": "node --import tsx scripts/liveGrokBrokerSession.ts", "e2e:pi-agent": "tsx src/examples/pi-agent.ts", "e2e:pi-memory-org": "tsx src/examples/pi-memory-org.ts", "e2e:jungian-play-org": "tsx src/examples/jungian-play-org.ts", diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index e9f343e..cf97bfd 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -5,3 +5,10 @@ Daimon against external runtimes. They are not part of the automated test suite and must not embed credentials or override the production command path unless a script's purpose explicitly requires that behavior. +Maintained scripts are TypeScript sources. Use `node --import tsx` for scripts +that import the repository source graph, because source imports may use emitted +`.js` specifiers that native strip-types does not resolve. Use +`node --experimental-strip-types` only for standalone builtin-only helpers. + +The live Codex and Grok scripts spend provider quota and may read local +operator auth. Typecheck them, but do not add them to automated tests. diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..fe7895b --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,15 @@ +# Daimon Scripts + +Maintained scripts in this folder are TypeScript sources. + +| Script | Caller | Purpose | +|---|---|---| +| `emitRuntimeContractManifest.ts` | `npm run build`, `npm run verify:contract-manifest`, `src/runtime/contractManifestArtifacts.test.ts` | Emits or checks packaged runtime contract manifest artifacts. | +| `src/runtime/native/verifyArtifacts.ts` | `npm run verify:native`, `npm run build` through `copyArtifact.ts` | Verifies checked-in native broker artifacts and provenance. | +| `verifyProductionClosure.ts` | `npm run verify:production-closure` | Checks built `dist/` for explicit-test runtime leakage. | +| `liveCodexSession.ts` | `npm run live:codex-session` | Manual live Codex tool-call probe. Spends provider quota. | +| `liveGrokBrokerSession.ts` | `npm run live:grok-broker` | Manual live Grok broker auth probe. Reads local Grok auth and spends provider quota. | + +`emitRuntimeContractManifest.ts` and the live probes use `node --import tsx` +because they import the repository source graph. Builtin-only helpers can run +with native `node --experimental-strip-types`. diff --git a/scripts/emitRuntimeContractManifest.mjs b/scripts/emitRuntimeContractManifest.ts similarity index 82% rename from scripts/emitRuntimeContractManifest.mjs rename to scripts/emitRuntimeContractManifest.ts index 7e931d3..2da9965 100644 --- a/scripts/emitRuntimeContractManifest.mjs +++ b/scripts/emitRuntimeContractManifest.ts @@ -10,9 +10,14 @@ const runtimeDist = fileURLToPath(new URL("../dist/runtime/", import.meta.url)); const manifestName = "contract-manifest.json"; const digestName = "contract-manifest.sha256"; -export const contractManifestArtifacts = () => { +type ContractManifestArtifacts = Readonly<{ + digest: Buffer; + manifest: Buffer; +}>; + +export const contractManifestArtifacts = (): ContractManifestArtifacts => { const canonical = canonicalJson(RUNTIME_CONTRACT_MANIFEST); - const parsed = JSON.parse(canonical); + const parsed = JSON.parse(canonical) as unknown; if (canonicalJson(parsed) !== canonical) throw new Error("contract manifest failed canonical round trip"); const manifest = Buffer.from(`${canonical}\n`, "utf8"); const hash = createHash("sha256").update(manifest).digest("hex"); @@ -20,35 +25,35 @@ export const contractManifestArtifacts = () => { return Object.freeze({ manifest, digest: Buffer.from(`sha256:${hash}\n`, "ascii") }); }; -const assertRuntimeDist = async (outputDirectory) => { +const assertRuntimeDist = async (outputDirectory: string): Promise => { const entry = await stat(outputDirectory); if (!entry.isDirectory()) throw new Error("compiled runtime output is not a directory"); }; -export const emitContractManifestArtifacts = async (outputDirectory = runtimeDist) => { +export const emitContractManifestArtifacts = async (outputDirectory = runtimeDist): Promise => { await assertRuntimeDist(outputDirectory); const artifacts = contractManifestArtifacts(); await writeFile(path.join(outputDirectory, manifestName), artifacts.manifest, { flag: "w" }); await writeFile(path.join(outputDirectory, digestName), artifacts.digest, { flag: "w" }); }; -export const verifyContractManifestArtifacts = async (outputDirectory = runtimeDist) => { +export const verifyContractManifestArtifacts = async (outputDirectory = runtimeDist): Promise => { await assertRuntimeDist(outputDirectory); const expected = contractManifestArtifacts(); const [manifest, digest] = await Promise.all([ readFile(path.join(outputDirectory, manifestName)), - readFile(path.join(outputDirectory, digestName)) + readFile(path.join(outputDirectory, digestName)), ]); if (!manifest.equals(expected.manifest) || !digest.equals(expected.digest)) { throw new Error("emitted runtime contract artifacts drift from source constants"); } }; -const main = async () => { +const main = async (): Promise => { const args = process.argv.slice(2); if (args.length === 0) return emitContractManifestArtifacts(); if (args.length === 1 && args[0] === "--check") return verifyContractManifestArtifacts(); - throw new Error("usage: emitRuntimeContractManifest.mjs [--check]"); + throw new Error("usage: emitRuntimeContractManifest.ts [--check]"); }; const invoked = process.argv[1]; diff --git a/scripts/liveCodexSession.mjs b/scripts/liveCodexSession.ts similarity index 75% rename from scripts/liveCodexSession.mjs rename to scripts/liveCodexSession.ts index 19d000c..96c98e2 100644 --- a/scripts/liveCodexSession.mjs +++ b/scripts/liveCodexSession.ts @@ -7,6 +7,15 @@ import { defineTool } from "@earendil-works/pi-coding-agent"; import { createCliSessionFactory } from "../src/pi/cliSession.ts"; +type TextContent = { text: string; type: "text" }; + +const isTextContent = (value: unknown): value is TextContent => ( + value !== null + && typeof value === "object" + && (value as { type?: unknown }).type === "text" + && typeof (value as { text?: unknown }).text === "string" +); + const workspacePath = await mkdtemp(path.join(os.tmpdir(), "daimon-live-codex-")); let toolInvoked = false; const lookup = defineTool({ @@ -16,7 +25,7 @@ const lookup = defineTool({ parameters: Type.Object({ question: Type.String({ description: "The verification question." }) }, { additionalProperties: false }), - async execute(_toolCallId, params) { + async execute(_toolCallId, params: { question: string }) { toolInvoked = true; return { content: [{ type: "text", text: `The verified answer is PINEAPPLE. Question: ${params.question}` }], @@ -30,13 +39,14 @@ try { engine: "codex", maxToolTurns: 3, timeoutMs: 120_000, - onToolsMounted: (tools) => process.stderr.write(`mounted tools: ${tools.map((tool) => tool.name).join(", ")}\n`) + onToolsMounted: (tools) => process.stderr.write(`mounted tools: ${tools.map((tool) => tool.name).join(", ")}\n`), })({ cwd: workspacePath, customTools: [lookup] }); let finalText = ""; const unsubscribe = session.subscribe((event) => { if (event.type !== "turn_end") return; + if (!("content" in event.message)) return; finalText = Array.isArray(event.message.content) - ? event.message.content.filter((entry) => entry.type === "text").map((entry) => entry.text).join("") + ? event.message.content.filter(isTextContent).map((entry) => entry.text).join("") : event.message.content; }); await session.prompt("You must call the live_lookup tool before answering. Then reply with the verified answer and nothing else."); diff --git a/scripts/liveGrokBrokerSession.mjs b/scripts/liveGrokBrokerSession.ts similarity index 90% rename from scripts/liveGrokBrokerSession.mjs rename to scripts/liveGrokBrokerSession.ts index f1e59d5..092bb9b 100644 --- a/scripts/liveGrokBrokerSession.mjs +++ b/scripts/liveGrokBrokerSession.ts @@ -17,11 +17,11 @@ const authFile = path.join(process.env.GROK_HOME ?? path.join(os.homedir(), ".gr const root = await mkdtemp(path.join(os.tmpdir(), "daimon-live-grok-")); const sentinel = "GROK_DAIMON_AUTH_OK"; const authority = { - async accessToken(forceRefresh) { + async accessToken(forceRefresh: boolean): Promise { if (forceRefresh) throw new Error("Live probe requires a current Grok login"); return (await readGrokBrokerCredential(authFile)).accessToken; }, - async markRejected() { throw new Error("Live probe requires a current Grok login"); } + async markRejected(): Promise { throw new Error("Live probe requires a current Grok login"); }, }; let stage = "credential preflight"; @@ -47,21 +47,22 @@ try { const child = trackCliChild(spawn("grok", [ "--sandbox", "strict", "--prompt-file", prompt, "--no-memory", "--no-subagents", "--disable-web-search", "--max-turns", "1", "--permission-mode", "dontAsk", - "--model", "daimon-broker-grok", "--output-format", "streaming-messages-json" + "--model", "daimon-broker-grok", "--output-format", "streaming-messages-json", ], { cwd: home, detached: process.platform !== "win32", env: { PATH: process.env.PATH, HOME: home, GROK_HOME: home, LANG: "C", LC_ALL: "C", TZ: "UTC" }, - stdio: ["ignore", "pipe", "pipe"] + stdio: ["ignore", "pipe", "pipe"], })); - let output; + let output: string; try { output = await readChild(child, 90_000, [capability], { retainStdoutTail: true }); } finally { await terminateChild(child); } stage = `terminal verification ${round}`; const turn = decodeGrokHeadlessTurn(output); assert.equal(turn.text, sentinel); - assert.ok(turn.usage?.total > 0, "Expected real provider usage"); + const usage = turn.usage; + assert.ok(usage !== undefined && usage.total > 0, "Expected real provider usage"); await assert.rejects(access(path.join(home, "auth.json")), { code: "ENOENT" }); - process.stdout.write(`${JSON.stringify({ round, reply: sentinel, usage: turn.usage })}\n`); + process.stdout.write(`${JSON.stringify({ round, reply: sentinel, usage })}\n`); } finally { await proxy.close(); } } process.stdout.write("GROK_BROKER_AUTH_RESTART_OK\n"); diff --git a/scripts/scriptSourcePolicy.test.ts b/scripts/scriptSourcePolicy.test.ts new file mode 100644 index 0000000..1a73f68 --- /dev/null +++ b/scripts/scriptSourcePolicy.test.ts @@ -0,0 +1,39 @@ +import assert from "node:assert/strict"; +import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; + +const maintainedRoots = ["scripts", "src/runtime/native"]; + +const maintainedJavaScriptSources = async (roots: string[]): Promise => { + const results: string[] = []; + const walk = async (directory: string): Promise => { + for (const entry of await readdir(directory, { withFileTypes: true })) { + const entryPath = path.join(directory, entry.name); + if (entry.isDirectory()) { + await walk(entryPath); + } else if (entry.name.endsWith(".mjs")) { + results.push(entryPath); + } + } + }; + await Promise.all(roots.map(walk)); + return results.sort(); +}; + +test("maintained scripts contain no JavaScript source files", async () => { + assert.deepEqual(await maintainedJavaScriptSources(maintainedRoots), []); +}); + +test("script source policy detects a maintained JavaScript regression", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "daimon-script-policy-")); + try { + const scripts = path.join(root, "scripts"); + await mkdir(scripts); + await writeFile(path.join(scripts, "regression.mjs"), "export {};\n"); + assert.deepEqual(await maintainedJavaScriptSources([scripts]), [path.join(scripts, "regression.mjs")]); + } finally { + await rm(root, { force: true, recursive: true }); + } +}); diff --git a/scripts/tsconfig.json b/scripts/tsconfig.json new file mode 100644 index 0000000..42997de --- /dev/null +++ b/scripts/tsconfig.json @@ -0,0 +1,16 @@ +{ + "extends": "../tsconfig.json", + "compilerOptions": { + "allowImportingTsExtensions": true, + "declaration": false, + "declarationMap": false, + "noEmit": true, + "rootDir": ".." + }, + "include": [ + "*.ts", + "../src/runtime/native/build.ts", + "../src/runtime/native/copyArtifact.ts", + "../src/runtime/native/verifyArtifacts.ts" + ] +} diff --git a/scripts/verifyProductionClosure.mjs b/scripts/verifyProductionClosure.ts similarity index 91% rename from scripts/verifyProductionClosure.mjs rename to scripts/verifyProductionClosure.ts index c9e476d..f3551ba 100644 --- a/scripts/verifyProductionClosure.mjs +++ b/scripts/verifyProductionClosure.ts @@ -3,8 +3,9 @@ import { readdir, readFile } from "node:fs/promises"; import path from "node:path"; const root = path.resolve("dist"); -const files = []; -async function walk(directory) { +const files: string[] = []; + +async function walk(directory: string): Promise { for (const entry of await readdir(directory, { withFileTypes: true })) { const file = path.join(directory, entry.name); if (entry.isDirectory()) await walk(file); else files.push(file); diff --git a/src/pi/worldTrajectory.ts b/src/pi/worldTrajectory.ts index a598949..93d59df 100644 --- a/src/pi/worldTrajectory.ts +++ b/src/pi/worldTrajectory.ts @@ -12,7 +12,7 @@ export const WORLD_TRAJECTORY_SCHEMA = "daimon.world_trajectory.v1" as const; * Pi's SessionManager remains the private raw session recorder. This module * derives a minimized public/evaluation projection from the same subscribed * session events. Raw training capture is deliberately separate; see - * docs/WORLD_TRAJECTORIES.md. + * docs/observability.md. */ export interface PiWorldTrajectoryToolCall { diff --git a/src/runtime/AGENTS.md b/src/runtime/AGENTS.md index 265d06e..93723c1 100644 --- a/src/runtime/AGENTS.md +++ b/src/runtime/AGENTS.md @@ -115,6 +115,8 @@ declared networks and room/DM surfaces. Optional stdio MCP calls consume only a Spawnfile-compiled, digest-attested test artifact and enforce its agent/server/tool allowlist. These modules build only into `dist-test-runtime`, never production `dist`, and remain inert unless the fixed test-mode environment gate is present. +The optional container fixture for that explicit test runtime lives at +`src/runtime/fixtures/Dockerfile.test-runtime`. Every agent-facing tool in `productionAgentTools.ts` must return its payload in `details`, not only in `content`. The MCP mount lowers `details` to diff --git a/src/runtime/contractManifestArtifacts.test.ts b/src/runtime/contractManifestArtifacts.test.ts index f66a263..caaf1f0 100644 --- a/src/runtime/contractManifestArtifacts.test.ts +++ b/src/runtime/contractManifestArtifacts.test.ts @@ -11,7 +11,7 @@ import { canonicalRuntimeContractManifest } from "./contractManifest.js"; const exec = promisify(execFile); const manifestPath = path.resolve("dist/runtime/contract-manifest.json"); const digestPath = path.resolve("dist/runtime/contract-manifest.sha256"); -const emitterPath = path.resolve("scripts/emitRuntimeContractManifest.mjs"); +const emitterPath = path.resolve("scripts/emitRuntimeContractManifest.ts"); const npm = async (...args: string[]): Promise => { const result = await exec("npm", args, { diff --git a/Dockerfile.test-runtime b/src/runtime/fixtures/Dockerfile.test-runtime similarity index 100% rename from Dockerfile.test-runtime rename to src/runtime/fixtures/Dockerfile.test-runtime diff --git a/src/runtime/native/build.mjs b/src/runtime/native/build.ts similarity index 98% rename from src/runtime/native/build.mjs rename to src/runtime/native/build.ts index 68300f4..e6144bd 100644 --- a/src/runtime/native/build.mjs +++ b/src/runtime/native/build.ts @@ -12,7 +12,7 @@ const sourceHash = createHash('sha256'); for (const file of sourceFiles) sourceHash.update(readFileSync(path.join(root, file))); const sourceSha256 = `sha256:${sourceHash.digest('hex')}`; mkdirSync(artifacts, { recursive: true }); -for (const [architecture, platform] of [['x64', 'amd64'], ['arm64', 'arm64']]) { +for (const [architecture, platform] of [['x64', 'amd64'], ['arm64', 'arm64']] as const) { const nonce = randomUUID(), image = `daimon-engine-broker:${nonce}`, container = `daimon-engine-broker-${nonce}`; try { execFileSync('docker', ['build', '--network=none', '--platform', `linux/${platform}`, '-t', image, root], { stdio: 'inherit' }); diff --git a/src/runtime/native/copyArtifact.mjs b/src/runtime/native/copyArtifact.ts similarity index 94% rename from src/runtime/native/copyArtifact.mjs rename to src/runtime/native/copyArtifact.ts index 8cd2c3d..1166aa6 100644 --- a/src/runtime/native/copyArtifact.mjs +++ b/src/runtime/native/copyArtifact.ts @@ -2,7 +2,7 @@ import { chmod, copyFile, mkdir } from 'node:fs/promises'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; -await import('./verifyArtifacts.mjs'); +await import('./verifyArtifacts.ts'); const root = path.dirname(fileURLToPath(import.meta.url)); const architecture = process.arch; if (!['x64', 'arm64'].includes(architecture) || process.platform !== 'linux') { diff --git a/src/runtime/native/verifyArtifacts.mjs b/src/runtime/native/verifyArtifacts.mjs deleted file mode 100644 index 66beba8..0000000 --- a/src/runtime/native/verifyArtifacts.mjs +++ /dev/null @@ -1,18 +0,0 @@ -import { createHash } from 'node:crypto'; -import { lstat, readFile } from 'node:fs/promises'; -import path from 'node:path'; -import { fileURLToPath } from 'node:url'; - -const root = path.dirname(fileURLToPath(import.meta.url)); -const sourceFiles = ['engineBrokerLauncher.c', 'engineBrokerLauncher.h', 'engineBrokerLauncherCore.inc', 'engineBrokerLauncherServer.inc', 'engineBrokerLauncherModes.inc', 'engineBrokerLauncherMain.inc']; -const sourceHash = createHash('sha256'); -for (const file of sourceFiles) sourceHash.update(await readFile(path.join(root, file))); -const sourceSha256 = `sha256:${sourceHash.digest('hex')}`; -for (const architecture of ['x64', 'arm64']) { - const binary = path.join(root, 'artifacts', `daimon-engine-broker-${architecture}`), stat = await lstat(binary); - if (!stat.isFile() || (stat.mode & 0o111) === 0) throw new Error('native broker artifact is not executable'); - const bytes = await readFile(binary), provenance = JSON.parse(await readFile(`${binary}.provenance.json`, 'utf8')); - const digest = `sha256:${createHash('sha256').update(bytes).digest('hex')}`; - if (provenance.version !== 'daimon.engine-broker-native-build.v1' || provenance.architecture !== architecture || provenance.source_sha256 !== sourceSha256 || provenance.binary_sha256 !== digest || provenance.install_path !== '/opt/daimon/bin/daimon-engine-broker') throw new Error('native broker provenance mismatch'); -} -console.log('native engine broker artifacts verified'); diff --git a/src/runtime/native/verifyArtifacts.ts b/src/runtime/native/verifyArtifacts.ts new file mode 100644 index 0000000..0ed76e7 --- /dev/null +++ b/src/runtime/native/verifyArtifacts.ts @@ -0,0 +1,38 @@ +import { createHash } from 'node:crypto'; +import { lstat, readFile } from 'node:fs/promises'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const root = path.dirname(fileURLToPath(import.meta.url)); +const sourceFiles = ['engineBrokerLauncher.c', 'engineBrokerLauncher.h', 'engineBrokerLauncherCore.inc', 'engineBrokerLauncherServer.inc', 'engineBrokerLauncherModes.inc', 'engineBrokerLauncherMain.inc']; +const sourceHash = createHash('sha256'); +for (const file of sourceFiles) sourceHash.update(await readFile(path.join(root, file))); +const sourceSha256 = `sha256:${sourceHash.digest('hex')}`; +type NativeArchitecture = 'x64' | 'arm64'; +type Provenance = { + architecture: NativeArchitecture; + binary_sha256: string; + install_path: string; + source_sha256: string; + version: 'daimon.engine-broker-native-build.v1'; +}; +const isProvenance = (value: unknown): value is Provenance => ( + value !== null + && typeof value === 'object' + && !Array.isArray(value) + && (value as { version?: unknown }).version === 'daimon.engine-broker-native-build.v1' + && typeof (value as { architecture?: unknown }).architecture === 'string' + && typeof (value as { source_sha256?: unknown }).source_sha256 === 'string' + && typeof (value as { binary_sha256?: unknown }).binary_sha256 === 'string' + && typeof (value as { install_path?: unknown }).install_path === 'string' +); + +for (const architecture of ['x64', 'arm64'] as const) { + const binary = path.join(root, 'artifacts', `daimon-engine-broker-${architecture}`), stat = await lstat(binary); + if (!stat.isFile() || (stat.mode & 0o111) === 0) throw new Error('native broker artifact is not executable'); + const bytes = await readFile(binary); + const provenance = JSON.parse(await readFile(`${binary}.provenance.json`, 'utf8')) as unknown; + const digest = `sha256:${createHash('sha256').update(bytes).digest('hex')}`; + if (!isProvenance(provenance) || provenance.architecture !== architecture || provenance.source_sha256 !== sourceSha256 || provenance.binary_sha256 !== digest || provenance.install_path !== '/opt/daimon/bin/daimon-engine-broker') throw new Error('native broker provenance mismatch'); +} +console.log('native engine broker artifacts verified'); diff --git a/tsconfig.build.json b/tsconfig.build.json index 843391e..759ccc0 100644 --- a/tsconfig.build.json +++ b/tsconfig.build.json @@ -5,6 +5,9 @@ "src/runtime/testRuntimeMoltnetActions.ts", "src/runtime/testRuntimeMcpActions.ts", "src/runtime/fixtures/testMcpServer.mjs", - "src/runtime/testRuntimeSubprocess.ts" + "src/runtime/testRuntimeSubprocess.ts", + "src/runtime/native/build.ts", + "src/runtime/native/copyArtifact.ts", + "src/runtime/native/verifyArtifacts.ts" ] } diff --git a/tsconfig.json b/tsconfig.json index 688c4c2..4283cb5 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -13,5 +13,10 @@ "rootDir": "src", "types": ["node"] }, - "include": ["src/**/*.ts"] + "include": ["src/**/*.ts"], + "exclude": [ + "src/runtime/native/build.ts", + "src/runtime/native/copyArtifact.ts", + "src/runtime/native/verifyArtifacts.ts" + ] } diff --git a/tsconfig.test-runtime.json b/tsconfig.test-runtime.json index bd56e32..571680d 100644 --- a/tsconfig.test-runtime.json +++ b/tsconfig.test-runtime.json @@ -1,5 +1,10 @@ { "extends": "./tsconfig.json", "compilerOptions": { "declaration": false, "declarationMap": false, "outDir": "dist-test-runtime", "sourceMap": false }, - "exclude": ["src/**/*.test.ts"] + "exclude": [ + "src/**/*.test.ts", + "src/runtime/native/build.ts", + "src/runtime/native/copyArtifact.ts", + "src/runtime/native/verifyArtifacts.ts" + ] }