Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
7b0d05a
test(memory): bind a fake driver instead of an in-process engine
YellowSnnowmann Sep 9, 2026
747f564
test(memory): drop the integration targets that drive the engine dire…
YellowSnnowmann Sep 9, 2026
cc99ba9
build(memory): cut the engine out of the test build
YellowSnnowmann Sep 9, 2026
7e87e9c
test(memory): restore the host tests deleted for an incidental engine…
YellowSnnowmann Sep 9, 2026
16421b3
chore(memory): clear the cfg gates and imports the shed left behind
YellowSnnowmann Sep 9, 2026
99db92b
Merge remote-tracking branch 'upstream/main' into feat/6161-drop-memo…
YellowSnnowmann Sep 9, 2026
9ff8333
fix(memory): bind the test driver once, not once per caller
YellowSnnowmann Sep 9, 2026
61e503c
test(memory): stop asserting one engine's behaviour from handler tests
YellowSnnowmann Sep 9, 2026
7573232
chore(vendor): bump tinymemory to 5c55431 (tinymemory#151)
YellowSnnowmann Sep 9, 2026
fbf5a3b
fix(memory): give the auto-save tests a store that retains
YellowSnnowmann Sep 9, 2026
1645d97
chore(ci): declare the tinymemory pin drift, and correct the memory R…
YellowSnnowmann Sep 9, 2026
21effaf
revert(bench): keep the profiling binaries, which never measured the …
YellowSnnowmann Sep 9, 2026
f5fa3cc
test(memory): restore the flows guards, and drop the seam installers'…
YellowSnnowmann Sep 9, 2026
c8d9aa9
docs(bench): finish the scenario cleanup the restore left half-done
YellowSnnowmann Sep 9, 2026
7853943
fix(prompts): restore the profiling UTC shortcut with its feature
YellowSnnowmann Sep 9, 2026
b494fac
docs(memory): stop describing host_impls.rs in the present tense
YellowSnnowmann Sep 9, 2026
a59075d
test(memory): restore host coverage for the entity read-RPCs
YellowSnnowmann Sep 9, 2026
6c8448a
chore(memory): retire the contacts gate's engine test and its allowli…
YellowSnnowmann Sep 9, 2026
67996bd
test(memory): make the entity-RPC assertions able to fail
YellowSnnowmann Sep 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/ci-lite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -675,8 +675,6 @@ jobs:
openhuman/flows/mod.rs
openhuman/mcp/server/resources.rs
openhuman/mcp/server/tools/mod.rs
openhuman/memory/people/mod.rs
openhuman/memory/people/mod_contacts_gate_tests_tests.rs
openhuman/platform/socket/event_handlers.rs
openhuman/platform/socket/ops.rs
openhuman/skills/mod.rs
Expand Down
93 changes: 2 additions & 91 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

236 changes: 56 additions & 180 deletions Cargo.toml

Large diffs are not rendered by default.

36 changes: 22 additions & 14 deletions docs/library-benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,14 @@ subconscious pass, a memory ingest, a bare embed) that each have their own
startup cost, steady-state footprint, and growth curve.

This document describes the benchmark environment built to measure that: a
pinned `library-profile` binary with eight scenarios, four driver scripts
pinned `library-profile` binary with six scenarios, four driver scripts

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

priority medium critique confident

Add missing scenarios to the scenario table

The diff says "six scenarios" but the table below it lists only four (agent-turn, long-agent, workflow, fleet). Meanwhile every command example in the document — library-bench.sh --scenarios, library-cpu.sh, the OPENHUMAN_PROFILE_HOLD_SECS invocation, and the OPENHUMAN_PROFILE_PREWARM_SUBAGENTS env-var row — references subagent-storm, which is not in the table. subagent-storm replaced subagents in the examples, so it is clearly a live scenario, but it was never added to the table. Add subagent-storm (and the remaining missing scenario) to the table, or correct the count to match what is listed.

[RULE] doc-internal-consistency ·

under `scripts/profile/`, and the comparison point the team cares about
(ZeroClaw). It builds on the manual investigation in
[`docs/resource-profiling-session-2026-07-21.md`](resource-profiling-session-2026-07-21.md);
read that document for the deep memory/CPU attribution work. This document is
about running repeatable benchmarks, not re-deriving those findings.

## The eight scenarios
## The scenarios

All scenarios run in `target/release/library-profile <scenario>`, replace
network inference with a deterministic provider (`rss-bench` feature), and
Expand All @@ -27,15 +27,23 @@ stderr). Each models a distinct embedding use case:

| Scenario | Models |
| --- | --- |
| `memory-ingest` | Canonicalizing and ingesting a batch of chat messages through the real extraction/admission/tree-queue pipeline. |
| `subagents` | A delegation turn: an orchestrator session spawns real subagents via `spawn_parallel_agents` and merges their findings. |
| `agent-turn` | The minimal embed case: one agent, one turn, no delegation, no workflow. The smallest useful "hello world" for a host that just wants a single reply. |
| `long-agent` | A long-running agent loop (`OPENHUMAN_PROFILE_TURNS`, default 25) in one process, to see whether RSS plateaus or grows per turn. |
| `workflow` | A saved automation run (`flows_create` + `flows_run`), representing the flows/automation embedding path rather than ad hoc chat. |
| `subconscious` | A background subconscious turn (the always-on reflective pass), distinct from an interactive chat turn. |
| `cold-phases` | Bootstrap attribution: per-phase checkpoints (config load, registry init, agent build, memory construction, first turn) so cold-start cost can be attributed to a phase instead of one lump sum. |
| `fleet` | N concurrent live agents with latency-realistic mock inference — the "100-1000 agents in a 2 GB / 2 vCPU server" question. See [below](#the-2-gb--2-vcpu-server-budget). |

Four scenarios that this table used to list are gone, for two different
reasons. `memory-ingest` and `cold-phases` measured the memory engine embedded
in this process — one drained its queue, the other checkpointed its bootstrap
through `tinymemory_core::store::MemoryClient` — and the binary no longer links
one (openhuman#6161). Measuring the memory *module* over the bus instead is a
different scenario and wants its own design, not a revived file. `subagents`
and `subconscious` had already stopped existing before that: the subconscious
domain was removed from the product outright, and both were still named in
`library-bench.sh`'s sweep list, where `dispatch` answered "unknown scenario".
Comment thread
coderabbitai[bot] marked this conversation as resolved.
The measurement tables further down are left as recorded — they are a log of
what was measured when, not a description of what runs today.

## How to run

Six scripts under `scripts/profile/` (each has `-h`/`--help`):
Expand All @@ -47,15 +55,15 @@ Six scripts under `scripts/profile/` (each has `-h`/`--help`):
```bash
./scripts/profile/library-bench.sh # default build, all scenarios
./scripts/profile/library-bench.sh --slim # --no-default-features recipe
./scripts/profile/library-bench.sh --scenarios "long-agent,subagents" --turns 50 --warm
./scripts/profile/library-bench.sh --scenarios "long-agent,subagent-storm" --turns 50 --warm
```

- **`library-cpu.sh`** — a `samply` wrapper for one scenario's CPU profile,
isolated from persistence/timezone noise by default.

```bash
./scripts/profile/library-cpu.sh subagents
samply load target/profile/rust-library/subagents-cpu.json.gz
./scripts/profile/library-cpu.sh subagent-storm
samply load target/profile/rust-library/subagent-storm-cpu.json.gz
```

- **`library-heap.sh`** — builds the `rss-bench-dhat` variant and runs a
Expand All @@ -64,8 +72,8 @@ Six scripts under `scripts/profile/` (each has `-h`/`--help`):
`library-bench.sh` output.

```bash
./scripts/profile/library-heap.sh memory-ingest
# load target/profile/rust-library/dhat-memory-ingest.json at
./scripts/profile/library-heap.sh agent-turn
# load target/profile/rust-library/dhat-agent-turn.json at
# https://nnethercote.github.io/dh_view/dh_view.html
```

Expand Down Expand Up @@ -118,7 +126,7 @@ behavior, not linked code size.
| Variable | Effect |
| --- | --- |
| `OPENHUMAN_PROFILE_TURNS` | Turn count for `long-agent` (default 25). |
| `OPENHUMAN_PROFILE_PREWARM_SUBAGENTS=1` | Run one warm-up turn before measuring (`subagents`/`subconscious`), isolating first-use cost from steady state. |
| `OPENHUMAN_PROFILE_PREWARM_SUBAGENTS=1` | Run one warm-up turn before measuring (`subagent-storm`), isolating first-use cost from steady state. |
| `OPENHUMAN_PROFILE_DISABLE_MEMORY_WRITES=1` | Disable `memory.auto_save` and episodic capture, isolating orchestration from persistence. |
| `OPENHUMAN_PROFILE_FORCE_UTC=1` | Skip `iana_time_zone`/CoreFoundation timezone resolution. |
| `OPENHUMAN_PROFILE_HOLD_SECS` / `HOLD_BEFORE_SECS` | Pause the process at settled/baseline state for external inspection (`vmmap`, `heap`, `malloc_history`, Instruments). |
Expand Down Expand Up @@ -264,12 +272,12 @@ Start cheap, escalate only as needed:
4. **Instruments / `vmmap` / `heap` / `malloc_history`** — deepest macOS-native attribution, using the `OPENHUMAN_PROFILE_HOLD_SECS` / `HOLD_BEFORE_SECS` hooks to pause the process at baseline or settled state:

```bash
OPENHUMAN_PROFILE_HOLD_SECS=120 target/release/library-profile subagents &
OPENHUMAN_PROFILE_HOLD_SECS=120 target/release/library-profile subagent-storm &
vmmap -summary <pid>
heap -sH <pid>

MallocStackLogging=1 OPENHUMAN_PROFILE_HOLD_SECS=120 \
target/release/library-profile subagents &
target/release/library-profile subagent-storm &
malloc_history <pid> -allBySize
```

Expand Down
8 changes: 7 additions & 1 deletion scripts/ci/module-pin-exemptions.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"is a deliberate edit to this file, which is a reviewable diff.",
"",
"Delete an entry the moment the pins are reconciled. An exemption that has",
"stopped being true fails the gate too `expect` must still match."
"stopped being true fails the gate too \u2014 `expect` must still match."
],
"exemptions": [
{
Expand All @@ -29,6 +29,12 @@
"submodule": "vendor/tinymcp",
"expect": "v0.3.2-2-g8b0627d",
"reason": "The host compiles the MCP contract against tinyhumansai/tinymcp#13 (Supervisor::tick returns a TickReport; needed by openhuman#5931), merged to tinymcp main but not yet in a tagged release, while the registry keeps the published v0.3.2 artifact. The drift is compile-only: the tinymcp module is registry-entered but not wired (AGENTS.md, 'step two of the extraction'), so no build downloads or loads that artifact. Delete this entry when tinymcp cuts its next release and the registry pin moves onto it."
},
{
"id": "tinymemory",
"submodule": "vendor/tinymemory",
"expect": "v1.15.2-15-g5c55431",
"reason": "The host compiles the memory contract against tinymemory main (tinyhumansai/tinymemory#148, #150 and #151 \u2014 the engine-free conformance driver openhuman#6161 binds in place of an in-process engine), merged upstream but not yet in a tagged release, while the registry keeps the published v1.15.2 artifact. The drift cannot reach a runtime mismatch, and that was checked rather than assumed: the entire v1.15.2..5c55431 delta to the crates this build ships is (a) one additive re-export, `pub use tinymemory_bus::chrono` in tinymemory-api, so a driver crate depending on the contract alone can name the `DateTime<Utc>` two MemoryTree methods already take, (b) a `#[cfg(test)] mod summarise_tests;` line in tinymemory-core, and (c) 31 lines of doc comment on MemoryDocuments::list_documents and delete_document. Zero removals, no method added or changed, no wire slot moved, CONTRACT_VERSION untouched \u2014 so the v1.15.2 artifact serves exactly the contract compiled here. Everything else in the range is tinymemory-conformance, a test-only crate that is a dev-dependency here and is not in the module artifact at all. Delete this entry when tinymemory cuts its next release and the registry pin moves onto it."
}
]
}
1 change: 0 additions & 1 deletion scripts/lib/feature-forwarding.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ export const INTENTIONALLY_NOT_FORWARDED = {
// 'some-gate': 'Reason it must not ship in the desktop build.',
tui: 'Terminal UI subcommand (openhuman tui/chat); the desktop app ships its own Tauri UI and never runs the ratatui terminal front-end. NOTE: `tui` is also default-OFF, so it is in NEITHER the contributor nor the product set and no ordinary lane compiles it — the feature-gate-smoke lane checks it explicitly. Any future entry here in the same position needs the same treatment.',
medulla: 'Medulla orchestration-backend client; the desktop app is OpenHuman\'s own product and never dials a Medulla backend. Consumed by the Medulla TUI, which embeds this crate directly.',
'memory-engine-seams': 'Compiles `memory::host_impls` — the seven host seams for an IN-PROCESS `tinymemory-core` — and turns that optional engine dependency on. Forwarding it would undo openhuman#5560 exactly: the shipped app reaches memory through the loaded tinymemory TinyBus module over `tinymemory-api`, and a second in-process engine over the same `memory.db` is the duplicate this shed removed. It is in `default` (not the product set) because `tests/*.rs` integration targets link this lib as a NORMAL dependency, where `#[cfg(test)]` is false and the module would be invisible however the engine is declared — several of them do drive a real engine and fail with "no EmbeddingHost installed" without it.',
};

/**
Expand Down
16 changes: 8 additions & 8 deletions scripts/profile/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ process, and aggregates median/min/max duration, settled RSS, retained delta,
and peak delta into `summary.json` + `summary.md`.

```bash
./scripts/profile/library-bench.sh # all 7 scenarios, default build, 5 repeats
./scripts/profile/library-bench.sh # all 6 scenarios, default build, 5 repeats
./scripts/profile/library-bench.sh --slim --repeat 7 # slim (no-default-features) build
./scripts/profile/library-bench.sh --scenarios "long-agent,subagents" --turns 50 --warm
./scripts/profile/library-bench.sh --scenarios "long-agent,subagent-storm" --turns 50 --warm
```

Results land in `target/profile/rust-library/bench-<timestamp>/` (or `--out DIR`).
Expand All @@ -33,9 +33,9 @@ Wraps `samply record` around one scenario, isolated from persistence/timezone
noise by default (matching the documented cold-path CPU recipe).

```bash
./scripts/profile/library-cpu.sh subagents
./scripts/profile/library-cpu.sh subagent-storm
./scripts/profile/library-cpu.sh long-agent -- OPENHUMAN_PROFILE_TURNS=50
samply load target/profile/rust-library/subagents-cpu.json.gz
samply load target/profile/rust-library/subagent-storm-cpu.json.gz
```

### `library-heap.sh` — live heap attribution via dhat
Expand All @@ -45,9 +45,9 @@ timing numbers from this build are perturbed by instrumentation; use it only
for allocation-site/retained-bytes attribution, not for RSS comparisons.

```bash
./scripts/profile/library-heap.sh memory-ingest
./scripts/profile/library-heap.sh agent-turn
# open https://nnethercote.github.io/dh_view/dh_view.html and load
# target/profile/rust-library/dhat-memory-ingest.json
# target/profile/rust-library/dhat-agent-turn.json
```

### `library-fleet.sh` — fleet sweep + 2 GB / 2 vCPU budget gate
Expand Down Expand Up @@ -118,10 +118,10 @@ runners don't false-fail.
./scripts/profile/library-bench.sh

# 2. CPU attribution for the slowest/most interesting scenario
./scripts/profile/library-cpu.sh subagents
./scripts/profile/library-cpu.sh subagent-storm

# 3. If a scenario's RSS looks off, drill into live heap
./scripts/profile/library-heap.sh subagents
./scripts/profile/library-heap.sh subagent-storm
```

All scripts require `jq` for JSON parsing/aggregation; `library-cpu.sh` also
Expand Down
18 changes: 14 additions & 4 deletions scripts/profile/library-bench.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,32 @@
# --scenarios "a,b,c" Comma-separated scenario list (default: all seven)
# --turns N OPENHUMAN_PROFILE_TURNS for long-agent (default binary default: 25)
# --skip-build Reuse the existing target/release binaries
# --warm Also run PREWARM_SUBAGENTS=1 variants for subagents + subconscious
# --warm Also run PREWARM_SUBAGENTS=1 variants for subagent-storm
# --out DIR Output directory (default: target/profile/rust-library/bench-<timestamp>)
# -h, --help Show this help
#
# Examples:
# ./scripts/profile/library-bench.sh
# ./scripts/profile/library-bench.sh --slim --repeat 7
# ./scripts/profile/library-bench.sh --scenarios "long-agent,subagents" --turns 50 --warm
# ./scripts/profile/library-bench.sh --scenarios "long-agent,subagent-storm" --turns 50 --warm

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"

ALL_SCENARIOS="memory-ingest,subagents,agent-turn,long-agent,workflow,subconscious,cold-phases"
WARM_ELIGIBLE=("subagents" "subconscious")
# The scenarios `library-profile` actually dispatches — keep in sync with
# `src/bin/library_profile/scenarios/mod.rs`, which is the source of truth.
#
# This list had drifted before openhuman#6161 touched it: `subagents` and
# `subconscious` are named here and neither has been a scenario for some time
# (the subconscious domain was removed from the product outright). `dispatch`
# answers "unknown scenario: …" for those, so a full sweep exited non-zero on
# two entries that could never run. `memory-ingest` and `cold-phases` leave now
# for a different reason: both measured the in-process memory engine, which
# this binary no longer links.
ALL_SCENARIOS="agent-turn,long-agent,workflow,fleet,skill-run,subagent-storm"
WARM_ELIGIBLE=("subagent-storm")

SLIM=0
REPEAT=5
Expand Down
10 changes: 4 additions & 6 deletions src/bin/library_profile/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,16 @@
//! with network inference replaced by a deterministic provider.
//!
//! Scenarios (`library-profile <scenario>`):
//! - `memory-ingest` — ingest 100 chat messages, drain the memory queue.
//! - `agent-turn` — a single cold agent turn (minimal library unit).
//! - `long-agent` — N warmed sequential turns with a per-turn checkpoint series.
//! - `workflow` — a real flows trigger->transform->agent graph, end to end.
//! - `cold-phases` — per-phase checkpoints of the cold bootstrap in one region.
//! - `fleet` — N live agents: marginal RSS, idle CPU, fd/thread growth, turn latency.
//! - `skill-run` — a skill step executing on a real `node` child: process-tree RSS.
//! - `subagent-storm`— K parallel researcher subagents in one instance: marginal RSS per subagent.
//!
//! `memory-ingest` and `cold-phases` were removed with the in-process memory
//! engine (openhuman#6161); see `scenarios/mod.rs`.
//!
//! stdout is ALWAYS a single pretty JSON object (the pinned schema in
//! `harness::ProfileResult`); every diagnostic goes to stderr with the stable
//! `[library-profile]` prefix.
Expand Down Expand Up @@ -52,11 +53,9 @@ fn start_dhat(scenario: &str) -> Result<dhat::Profiler> {

async fn dispatch(scenario: &str) -> Result<ProfileResult> {
match scenario {
"memory-ingest" => scenarios::memory_ingest::run().await,
"agent-turn" => scenarios::agent_turn::run().await,
"long-agent" => scenarios::long_agent::run().await,
"workflow" => scenarios::workflow::run().await,
"cold-phases" => scenarios::cold_phases::run().await,
"fleet" => scenarios::fleet::run().await,
"skill-run" => scenarios::skill_run::run().await,
"subagent-storm" => scenarios::subagent_storm::run().await,
Expand Down Expand Up @@ -86,8 +85,7 @@ fn main() -> Result<()> {
// can size the worker pool (the `fleet` scenario simulates the 2 vCPU box).
let scenario = std::env::args().nth(1).context(
"usage: library-profile \
<memory-ingest|agent-turn|long-agent|workflow|cold-phases|fleet|\
skill-run|subagent-storm>",
<agent-turn|long-agent|workflow|fleet|skill-run|subagent-storm>",
)?;

// Profiler must outlive the whole run + the JSON print so its Drop writes
Expand Down
Loading
Loading