Pi-Bench Memory Benchmark Runner

Reproducible experiment infrastructure for comparing external memory backends in proactive personal-assistant agents

This repository packages the model configurations, runtime integrations, Docker entry points, and analysis tools used to run Pi-Bench with Mem0 and LightMem. The focus is research engineering: keeping model and memory conditions comparable, making long-running jobs resumable, preserving trace provenance, and separating credentials and generated outputs from source control.

Engineering highlights

• Controlled experiment matrix: paired configurations compare Mem0 and LightMem across GPT and Qwen model routes.
• Memory-provider isolation: external memory is enabled through model configuration while the default benchmark behavior remains available.
• Container-ready execution: a dedicated Dockerfile and entrypoint reproduce runtime setup across personas and model conditions.
• Offline model preparation: LightMem's LLMLingua dependency can be downloaded once and mounted read-only to avoid per-task network variation.
• Sequential and resumable orchestration: shell runners execute full persona matrices while preserving independent output directories.
• Trace recovery: successful interaction histories can be reconstructed into evaluation-ready traces after interrupted or partial runs.
• Operational observability: scripts aggregate token usage, API latency, end-to-end elapsed time, completion status, and persona-level scores.
• Publication hygiene: API keys, .env files, model caches, third-party checkouts, and benchmark outputs are excluded.

flowchart LR
    A[Model and memory YAML] --> B[Sequential launcher]
    B --> C[Pi-Bench Docker runtime]
    C --> D[Proactive assistant agent]
    D --> E{External memory}
    E --> F[Mem0]
    E --> G[LightMem]
    F --> H[Task histories and logs]
    G --> H
    H --> I[Trace reconstruction]
    H --> J[Usage accounting]
    H --> K[Persona score reports]
    I --> L[Evaluation artifacts]
    J --> L
    K --> L

Loading

Experiment configurations

The repository includes paired memory conditions for:

config/models/gpt-5-mini.mem0.yaml
config/models/gpt-5-mini.lightmem.yaml
config/models/qwen3-8b.mem0.yaml
config/models/qwen3-8b.lightmem.yaml

Keeping model routes and run structure explicit makes backend comparisons auditable instead of relying on hidden notebook state.

Repository guide

Path	Role
src/	benchmark runtime, channels, evaluation, user agent, and configuration
memory/	Mem0 integration layer
config/models/	model × memory experiment definitions
docker/Dockerfile.memory	memory-enabled benchmark image
scripts/run_memory_pibench.sh	one model/persona subset run
scripts/run_memory_full_sequential.sh	full sequential experiment matrix
scripts/smoke_memory_backends.py	provider initialization check
scripts/reconstruct_trace_from_history.py	recover traces from successful histories
scripts/collect_run_usage.py	token, call, and timing aggregation
scripts/report_persona_scores.py	checklist and proactiveness summaries
scripts/trace_viewer.py	HTML trace inspection

Environment

Use Python 3.11 or later. Keep all real credentials in the local environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
cp env.example.sh .env

Model routes:

export OPENAI_BASE_URL="..."
export OPENAI_API_KEY="..."
export QWEN_BASE_URL="..."
export QWEN_API_KEY="..."

Memory services:

export MEMORY_LLM_MODEL="..."
export MEMORY_LLM_BASE_URL="..."
export MEMORY_LLM_API_KEY="..."
export MEMORY_EMBEDDING_MODEL="..."
export MEMORY_EMBEDDING_BASE_URL="..."
export MEMORY_EMBEDDING_API_KEY="..."
export LIGHTMEM_LLMLINGUA_MODEL_PATH="/path/to/local/llmlingua-2"

Never commit populated environment files.

Run

Prepare the local LightMem compression model once:

python scripts/download_lightmem_model.py

Run one condition or the full sequential matrix:

bash scripts/run_memory_pibench.sh \
  config/models/gpt-5-mini.lightmem.yaml \
  researcher \
  outputs-smoke

bash scripts/run_memory_full_sequential.sh outputs-full-memory

Inspect resource use and scores:

python scripts/collect_run_usage.py outputs-full-memory
python scripts/report_persona_scores.py outputs-full-memory

Reproducibility boundary

This is a runner and instrumentation snapshot, not a bundle of completed benchmark claims. Full execution depends on Pi-Bench/AppWorld assets, provider credentials, model availability, and external-memory dependencies. Generated results are intentionally excluded, so performance should be reported only from a named run directory with its configuration and trace artifacts.

Pi-Bench originates from the upstream project and remains subject to its license and citation requirements. This repository's engineering focus is the memory-backend experiment layer and supporting operational tooling.