feat(runtime)!: make capture retention and workload budgets explicit

[morluto]

Summary

Make capture retention explicit and keep completed native evidence recoverable when immediate analysis fails.

Fixes #470, fixes #471, fixes #472, fixes #473, fixes #474, fixes #475.

Problem and expected behavior

Real Slime investigations exposed captures lost to decoder limits, verbose console output treated as a universal disk-storage requirement, delayed MCP cancellation, and incorrect provider coverage or execution-success reporting. A completed capture should remain available for analysis retry; bounded diagnostics must disclose omissions rather than imply complete logs.

Change

• Default to bounded in-memory console diagnostics with observed/retained/omitted byte counts. Retain full disk-backed console streams only for process-output evidence, oracle inputs, or explicit selection.
• Separate optional workload time/RSS budgets from decoder limits. Keep cancellation, output/storage protection and finite cleanup.
• Return failed immediate analysis as an MCP error with partial evidence, a preservable session handle and an evidence resource link when preserved. Retain ordered sources for retry without recapturing.
• Record rejected native bundles while preserving diagnostics and existing broker failure attribution; constrain rejected-bundle cleanup to request-owned scratch.
• Move blocking MCP work into request-owned, serialized worker phases while allowing independent capture processes to overlap.
• Batch Memray aggregation, preserve large counters and remove the manual extractor revision; correct Perfetto edge coverage, Compute Sanitizer error collection, writable-growth observation and zero-exit failure classification.
• Add py-spy caller capture, native Triton autotune-cache capture, explicit SARIF source roots and bounded text-fragment previews; repair pytest fixture capture ownership.

Suggested review order

1. Public contracts and capture orchestration: runtime_contracts.py, stateless.py.
2. Process ownership, output retention and cleanup: execution.py.
3. Immutable failure evidence and MCP projections: evidence_models.py, repository.py, mcp/server.py.
4. Provider changes and behavioral regressions. Memray and Perfetto fixes have separate commits.

Contract and boundary impact

• Semantic owners: capture admission/finalization and the subprocess broker; transports remain thin.
• CLI/MCP: target.console_output, optional target.budget, CLI --console-output/--workload-budget/--limits; 49 generated tools.
• Storage/provenance: optional diagnostic and rejection metadata; failed captures can preserve without native inputs. Native bytes remain content-addressed; no automatic store migration.
• Compatibility: Triton native cache support targets 3.7.x; CPU sampled edges are not invocation counts.
• Concurrency/security: request-owned cancellation and cleanup; trusted local execution is not sandboxing. Resource projections omit paths, argv, environment and diagnostic text.
• Claims: capture outcome remains distinct from analysis success; omitted console bytes cannot be recovered by later preservation.

Example target options:

{"console_output":"full","budget":{"timeout_seconds":600,"max_memory_bytes":8589934592}}

Both budget fields otherwise default to null. A decoder-limit failure now returns isError: true, code: LIMIT_EXCEEDED and details.partial_evidence, rather than losing the completed capture.

Evidence and regression coverage

Executed real Slime scheduling, CUDA correctness, profiling and recovery workloads on Linux. A 1 KiB analysis-input limit reproduced capture loss before the recovery fix; afterward the native Memray profile survived and a fresh server reanalyzed it without rerunning Slime. Coverage.py regressions additionally check exact preserved bytes and source mappings.

Tests cover diagnostic omissions, full-output/oracle retention, cancellation, concurrent requests, explicit budgets, native rejection and external symlink sentinel survival. Existing investigation records distinguish measured observations from inferred conclusions; native workload artifacts remain local and are not included in this PR.

Remaining proof gaps: representative inference-server exports, broader platform/GPU coverage and a dedicated special-file rejection workload. Shared storage admission and native-artifact budget separation remain unfinished; this PR does not claim the investigation is exhausted.

• Observed, derived and inferred claims remain distinguishable.
• Inputs, versions and provenance remain bound.
• Compatibility and incompleteness remain visible.

Validation

On 689414a:

• uv run pytest -q — 349 passed, 192 deselected.
• uv run pytest tests/test_native_capture_recovery.py tests/test_workload_budgets.py tests/test_cli_stateless.py -q -o addopts='' — 50 passed.
• uv run ruff check src tests tools — passed.
• uv run mypy src tests tools — passed, 134 files.
• uv run ruff format --check src tests tools — passed, 134 files.
• git diff origin/main...HEAD --check — passed.

Before the release-metadata-only merge, the broader capture/runtime/CLI run passed 183 tests with one skip and unraisable warnings treated as errors. Source and test contents are unchanged by that merge.

Compatibility and safety

Behavior changes are intentional: callers needing previous workload ceilings must explicitly set a budget; callers needing complete console evidence must select full retention unless the tool or oracle requires it. New failure manifests may require this reader version. Existing native storage safeguards remain; there is no claim of unlimited artifact storage, strict filesystem quotas or complete cross-platform validation.

Review checklist

• Related issues and concrete behavior changes are described.
• Behavioral success and failure regressions are included; proof gaps are stated.
• Existing owning-contract documentation and representative interface examples are included.
• Final diff excludes local artifacts, private host paths and unrelated release changes.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T04:26:25.483998Z	689414a	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 689414a36d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Cap console hashing at the settled byte count

When a workload leaves a detached process running, that process can derive the request directory from a provider artifact path and continuously append to console-*/stdout after the broker closes its sink. This unbounded sha256_file(path) then follows the growing file indefinitely, outside the workload deadline and while run_in_request holds the runtime-wide lock, blocking every subsequent MCP operation. Hash only up to expected_size (and reject any extra byte) so settled console validation remains bounded.

AGENTS.md reference: AGENTS.md:L33-L41

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Project the target execution limit alongside oracle limits

For a preserved capture whose target hits the new RSS budget, the MCP evidence resource exposes failure_code: LIMIT_EXCEEDED and the requested budget but omits the target execution's limit receipt, even though this projection now retains that field for semantic oracles. After restart, an agent therefore cannot distinguish a memory-budget termination from output, writable-growth, or storage-reserve termination or inspect the observed value without leaving MCP and reading the private manifest. Include the top-level limit in this safe projection so the preserved execution provenance remains actionable.

AGENTS.md reference: AGENTS.md:L33-L34

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Exclude the copied pytest plugin from artifact growth

When pytest capture is run with a lowered max_output_bytes (for example 1 KiB), this 5.5 KiB Flameox-owned plugin is copied into the capture's monitored writable root after the broker records its baseline. The resource observer consequently attributes the internal support file to workload artifact growth and can terminate an otherwise tiny pytest capture with WRITABLE_LIMIT_EXCEEDED. Materialize the plugin before the baseline is measured, or place it outside the provider output root, so request-lowered limits apply only to generated capture artifacts.

AGENTS.md reference: AGENTS.md:L46-L48

Useful? React with 👍 / 👎.