Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docs/adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,12 @@ Nsight Compute capture requires both the `ncu` executable and NVIDIA's vendor-sh
`extras/python/ncu_report.py` reader. Flameox resolves both before workload execution and returns
external setup guidance if the installation is incomplete.

Provider ownership is explicit during capture. External collectors installed
with Flameox, such as py-spy, execute from the launched uvx environment rather
than ambient request `PATH`. In-process collectors, including coverage.py and
Provider ownership is explicit during capture. py-spy executes from a verified session binding
prepared independently with `uvx --from py-spy==0.4.2`, or from beside the server interpreter.
Preparation checks the collector version and binds its executable identity; later capture rechecks
that identity. A removed or changed uv cache entry requires preparation again. Capture never
silently installs dependencies or resolves managed collectors from request `PATH`.
In-process collectors, including coverage.py, PyTorch, and
Memray, must be installed in the declared workload interpreter; capture probes
that interpreter before execution and never substitutes Flameox's interpreter.

Expand Down
10 changes: 7 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,13 @@ capability selector or a free-form analysis argument object.
Capture-provider contracts supply the discriminated provider variants for each compatible capture
tool. Missing packages, executables, permissions, versions, or platforms do not change the catalog;
the attempted tool returns typed remediation. The separately invoked CLI setup command or MCP
`prepare_providers` tool may prepare the exact version-pinned uvx environment named by an explicit
Python provider set; neither creates project state nor owns a durable operation or provider
inventory. Host profilers, drivers, and permissions remain external and receive guidance only.
`prepare_providers` tool resolves dependencies according to where they execute. CLI setup prepares
the complete version-pinned server environment. MCP preparation can activate a pinned standalone
py-spy collector in the existing session; server-import dependencies are checked against the active
release's complete requested dependency contract. Neither creates project state nor owns a durable
operation or provider inventory. Host profilers, drivers, and permissions remain external.
`providers/preparation.py` owns bounded preparation and verified session executable bindings;
transports do not infer readiness from installation receipts.

Request validation is also a normalization boundary. Runtime admission constructs the typed
capability and provider models once; execution, provenance, limitations, and result descriptions
Expand Down
64 changes: 43 additions & 21 deletions docs/interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,39 +93,50 @@ but never serialize arbitrary exception text: filesystem exceptions may contain
dependency exceptions may contain argv or environment-derived values. Unexpected failures use a
stable operation-specific summary; cancellation remains a control path and is re-raised.

`prepare_providers` and the capture tools are open-world. Preparation resolves the exact package
requirement through `uvx`, verifies that environment by running Flameox's version command, and
returns the same requirement in a global, version-pinned MCP launcher. Capture tools execute an
explicit caller-supplied target that may itself access external services. The request names the
complete managed
provider set; Flameox keeps no installed-provider inventory or setup receipt. System profilers,
drivers, device access, and OS permissions remain external requirements. Flameox returns guidance
for them but does not invoke a system package manager or elevate privileges. Preparation creates no
project state,
durable job, or plan, and it cannot add packages to the currently running server. A result with
a non-null `next_action` identifies the typed `reconnect_mcp` handoff, explains that the current
server is unchanged, and directs the agent to reconnect with the returned launcher before retrying
the capture. Host-only preparation returns `next_action: null`. Managed provider IDs are `aiperf`,
`memray`, `otlp`, `perfetto`, `py-spy`, and `torch`; every call declares the complete desired set. A
provider such as Perfetto may be both prepared Python support and an external host Trace Processor
requirement. Preparation waits up to 1,800 seconds by default; callers may set `timeout_seconds`
from 1 through 3,600. A uvx failure returns its complete stderr in `SETUP_FAILURE`.
`prepare_providers` and capture tools are open-world. Preparation uses request-owned bounded
subprocesses and an overall deadline; cancellation settles the installer and its descendants.
py-spy is prepared as a pinned standalone collector and becomes available in the same live session.
Bindings are activated only after the complete request succeeds, including any server preparation;
failure, timeout, or cancellation leaves prior session bindings unchanged. Preparation forwards
the safe uv controls `UV_OFFLINE`, `UV_CACHE_DIR`, `UV_PYTHON_DOWNLOADS`, and `UV_NO_CONFIG`.
Repeating preparation reuses its verified binding without installation. The returned launcher still
names the complete requested server provider set; preparing another provider does not remove
existing session bindings. No durable provider inventory, project state, or job is created.

Server-import requirements are compared with the active release and installed dependency versions.
`activation_status` is `ready`, `restart_required`, `unknown`, or `not_applicable`. `next_action` is
null for ready or host-only requests. Otherwise `reconnect_mcp` explicitly says whether reconnection
is required or conditional and tells the caller to preserve needed session analyses first. An
unknown identity does not establish a required restart. A prepared server environment is verified
with its version command; preparation does not replace active imports.

Managed IDs remain `aiperf`, `memray`, `otlp`, `perfetto`, `py-spy`, and `torch`. External host tools,
drivers, permissions, and workload-interpreter requirements are reported separately and remain
unverified by preparation. Both structured results and text summaries carry those handoffs. No
system package manager or privilege elevation is invoked. Perfetto still requires an externally
installed Trace Processor. The deadline defaults to 1,800 seconds and accepts 1 through 3,600;
MCP preparation failures use bounded path-free diagnostics, while CLI setup retains local stderr.

## Sources and limits

The strict source union is:

```text
PathSource {kind: "path", path, format?, producer?, expected_sha256?}
EvidenceSource {kind: "evidence", evidence_id, artifact_role?}
EvidenceSource {kind: "evidence", evidence_id, artifact_role? OR artifact_selector?}
```

Continuations are opaque integrity cursors bound to the request and exact input
digests. They contain no authority, credentials, or artifact data and are not
an authentication boundary: a caller already authorized to submit the analysis
can choose which of its rows to request. They can cross process boundaries, so
a CLI invocation can resume a previous page. A changed input cannot reuse a
continuation.
a CLI invocation can resume a previous page. Tokens bind ordered content digests, formats,
producer identities, arguments, and limits, independently of storage paths and publication roles.
After preserving a capture, repeat the original options and limits with the evidence resource's
ordered `analysis_sources` and the returned continuation. Scratch can be released immediately.
A changed input cannot reuse a continuation. Tokens issued by older path-bound implementations
must be restarted with a fresh analysis. Preview `offset` counts logical rows: text lines, JSONL
records, CSV data records, Parquet records, and projected JSON entries.

Decoded offsets must be integers within the available bounded population. Negative offsets and
offsets at or beyond the end fail with `INVALID_INPUT`; they never use Python slicing semantics or
Expand Down Expand Up @@ -174,6 +185,17 @@ eligible blocks and a deterministic percentile interval when at least three
blocks survive capture/oracle validation, and classifies the effect against the
declared threshold. Work is not detached; the request receives progress and owns
cancellation.
The experiment's `point_estimate_classification` is descriptive; its `decision_basis` is explicit
on the metrics block. It does not claim confidence-qualified improvement or equivalence.

Capture `outcome` is computed from every execution before diagnostic compaction and retains exact
success/failure counts. MCP error classification consumes that outcome even when no execution
diagnostics fit inline. Each execution identifies whether `returncode` belongs to the workload or
collector, retains the invoked executable SHA-256, and leaves `workload_returncode` null for wrapped
captures. Exit ownership is declared by each invocation builder: self-reporting workloads retain
their observed exit even when they use a provider other than `direct`. A usable profile does not
prove workload success. Preserved stdout, stderr, and profiles
are individually selectable from the evidence resource.
The first declared case is the baseline. A case inherits the target argv when it omits `argv`, and
its environment overrides the target environment. Each block randomizes case order from the
declared seed. The semantic oracle runs after every successful capture in that case environment;
Expand Down Expand Up @@ -205,7 +227,7 @@ flameox setup
flameox mcp serve|inspect
flameox analyze [--continuation TOKEN] [--preserve]
flameox capture [--experiment JSON] [--preserve] -- <argv...>
flameox evidence query|show
flameox evidence query|show|location
```

`setup` detects supported coding agents and uses one multi-select prompt to choose which global MCP
Expand Down
8 changes: 6 additions & 2 deletions docs/investigations.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,12 @@ The 0.2 runtime accepts `wall_time_ns` and paired `median_difference` or
`mean_difference`. Each non-baseline case is compared with the first declared
case within the same blocks. Failed or oracle-invalid pairs are excluded and
reported as limitations; fewer than three eligible pairs produce a descriptive
estimate without a confidence interval. The practical-threshold decision is
returned as typed comparison evidence, not retained only as request metadata.
estimate without a confidence interval. `point_estimate_classification` describes only the
observed estimate against the practical margin, with `decision_basis=descriptive_point_estimate`
on the experiment metrics block. It replaces the ambiguous `decision` field. `within_threshold`
does not establish equivalence; a wide interval may still span material improvement and regression.
The deterministic percentile interval is reported separately and is not a calibrated equivalence
test. Semantic correctness still requires the declared oracle.

Artifact comparison is separate from that experiment result. Capture the representative baseline
and candidate summaries independently, preserve them if they must survive the session, then submit
Expand Down
19 changes: 19 additions & 0 deletions docs/storage-and-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,25 @@ Missing or corrupt resources are MCP resource errors. Native payload bytes are
not exposed through resources; their digest and role remain visible in the
manifest.

Agent projections expose an opaque `source` accepted unchanged by analysis tools for each artifact,
plus `logical_sources` for directory bundles and ordered `analysis_sources` for the original
analysis. Selectors address immutable manifest positions, not hashes of private roles that could
be checked against guessed filenames. File selectors always select exact members, even when a
filename contains the directory-role delimiter. New manifests include `source_layout`: each source
declares its file/directory kind, exact artifact indices, and identity, with an ordered mapping for
the original analysis inputs. Empty directories retain their metadata without inventing a native
payload, and re-preserving a selected member retains its file identity. Readers validate membership,
digests, sizes, and analysis mappings. Existing manifests without this optional field remain
readable using their original role-based bundle convention. Missing or
ambiguous selectors point back to the evidence resource for enumeration.

Corruption remains fail-closed. Runtime errors include the selected configuration source, a
path-free store identifier, and recovery instructions. `flameox evidence location` prints the
resolved directory locally without reading or initializing the repository. Restore the original
store from a known-good backup, or select a distinct empty store with `FLAMEOX_DATA_DIR` and
restart/reconnect. Switching stores does not recover old evidence. Do not delete existing data or
synthesize replacement metadata.

## Format evolution

This is repository format `1`. Unsupported repository or manifest versions
Expand Down
12 changes: 12 additions & 0 deletions src/flameox/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -426,6 +426,18 @@ def evidence_query(
runtime.close()


@evidence_app.command("location")
def evidence_location() -> None:
"""Show the selected local evidence directory without opening or modifying it."""
runtime = _runtime()
try:
_write(
{"directory": str(runtime.repository.root), "environment_variable": "FLAMEOX_DATA_DIR"}
)
finally:
runtime.close()


@evidence_app.command("show")
def evidence_show(
evidence_id: Annotated[str, typer.Argument()],
Expand Down
4 changes: 4 additions & 0 deletions src/flameox/execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@
"UV_EXTRA_INDEX_URL",
"UV_INDEX",
"UV_NATIVE_TLS",
"UV_OFFLINE",
"UV_CACHE_DIR",
"UV_PYTHON_DOWNLOADS",
"UV_NO_CONFIG",
"PIP_INDEX_URL",
"PIP_EXTRA_INDEX_URL",
)
Expand Down
58 changes: 46 additions & 12 deletions src/flameox/mcp/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
from pathlib import Path
from typing import Annotated, Any, Literal, cast

import anyio
from mcp.server import MCPServer
from mcp.server.mcpserver import Context
from mcp_types import CallToolResult, ContentBlock, ResourceLink, TextContent, ToolAnnotations
Expand All @@ -25,7 +24,7 @@
)
from pydantic.json_schema import JsonSchemaValue

import flameox.setup as provider_setup
import flameox.providers.environment as provider_setup
from flameox import __version__
from flameox.mcp.capability_tools import (
Execution,
Expand Down Expand Up @@ -119,7 +118,10 @@ class LauncherEnvelope(BaseModel):

class ReconnectActionEnvelope(BaseModel):
kind: Literal["reconnect_mcp"] = Field(description="Reconnect the MCP server process.")
message: str = Field(description="Required handoff before retrying with the prepared provider.")
message: str = Field(
description="Handoff conditions and session-evidence preservation guidance."
)
necessity: Literal["required", "conditional"]


class PreparationEnvelope(_Envelope):
Expand All @@ -139,7 +141,13 @@ class PreparationEnvelope(_Envelope):
description="Version-pinned launcher for the requested provider set."
)
next_action: ReconnectActionEnvelope | None = Field(
description="Required reconnection action, or null when the current server can continue."
description=(
"Reconnection handoff with explicit necessity; null for ready or host-only preparation."
)
)
activation_status: Literal["ready", "restart_required", "unknown", "not_applicable"]
workload_requirements: list[ExternalRequirementEnvelope] = Field(
description="Requirements to verify in the exact workload environment before capture."
)


Expand Down Expand Up @@ -223,6 +231,21 @@ def _success_summary(value: dict[str, Any], *, resource: ResourceLink | None) ->
action = (
"reconnect using the returned launcher" if preparation_action else "continue capture"
)
if (
isinstance(preparation_action, dict)
and preparation_action.get("necessity") == "conditional"
):
action = (
"verify the active environment; reconnect only if the requested support is absent"
)
if value.get("external_requirements"):
action = (
(action + "; " if preparation_action else "")
+ "verify or satisfy the listed external requirements before capture; "
"host readiness has not been verified"
)
if value.get("workload_requirements"):
action += "; verify the listed requirements in the exact workload interpreter"
return (
f"Provider preparation completed; next: {action}. "
"Full details are in structuredContent."
Expand Down Expand Up @@ -297,6 +320,7 @@ async def lifespan(_: MCPServer[AnalysisRuntime]) -> AsyncIterator[AnalysisRunti

@server.tool(annotations=PREPARE, structured_output=True)
async def prepare_providers(
ctx: Context[AnalysisRuntime],
provider_ids: Annotated[
list[str],
Field(
Expand All @@ -317,21 +341,25 @@ async def prepare_providers(
"""Prepare managed providers and return any required MCP reconnection action."""

try:
preparation = await anyio.to_thread.run_sync(
provider_setup.prepare_providers, provider_ids, timeout_seconds
)
preparation = await runtime(ctx).dependencies.prepare(provider_ids, timeout_seconds)
except provider_setup.ProviderSelectionFailure as error:
return _failure(RuntimeFailure("INVALID_INPUT", str(error)))
except provider_setup.SetupFailure as error:
return _failure(RuntimeFailure("SETUP_FAILURE", str(error)))

next_action = None
if preparation.restart_required:
if preparation.restart_required is not False:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Align server guidance with live py-spy activation

When prepare_providers is called for only py-spy, this new logic correctly leaves next_action null because the collector is activated in the current session, but the MCP server instructions at lines 308–310 still direct the agent to reconnect after every missing managed provider. Following that higher-level guidance can unnecessarily terminate the session and lose unpreserved analyses, so make it tell callers to follow the returned activation status instead.

AGENTS.md reference: AGENTS.md:L44-L48

Useful? React with 👍 / 👎.

next_action = {
"kind": "reconnect_mcp",
"necessity": "required" if preparation.restart_required else "conditional",
"message": (
"Reconnect Flameox with the returned launcher before retrying the capture; "
"the current server process is unchanged."
"The active server does not satisfy the requested dependency contract. "
if preparation.restart_required
else "The active server dependency identity could not be verified. "
)
+ (
"Preserve needed session analyses before reconnecting with the complete "
"returned launcher. Reconnection ends session-local evidence access."
),
}
return _success(
Expand All @@ -351,6 +379,11 @@ async def prepare_providers(
"args": preparation.launcher_args,
},
"next_action": next_action,
"activation_status": preparation.activation_status,
"workload_requirements": [
{"provider_id": item.provider_id, "guidance": item.guidance}
for item in preparation.workload_requirements
],
}
)

Expand Down Expand Up @@ -485,11 +518,12 @@ async def progress(current: int, total: int, message: str) -> None:
failed = [
item for item in value["capture"]["executions"] if item["status"] != "succeeded"
]
if failed:
if value["capture"]["outcome"]["status"] != "succeeded":
return _failure(
RuntimeFailure(
"EXECUTION_FAILURE",
"One or more captured targets exited unsuccessfully.",
"One or more capture executions failed; consult exit attribution "
"and preserved diagnostics before inferring workload failure.",
details={"partial_evidence": value, "failed_executions": failed},
)
)
Expand Down
Loading
Loading