feat: Memory optimization to prevent OOM in multi-session scenarios

[oyi77]

Summary

Addresses Out of Memory (OOM) errors when running multiple OpenClaude sessions concurrently. On a 32GB system, 4 instances can exhaust all available RAM. This PR adds memory-pressure-aware conversation compaction, automatic paste store cleanup, configurable per-session memory budgets, and concurrent session budget scaling.

Changes

New Files

• src/utils/memoryPressure.ts — Memory pressure monitor that watches RSS every 30s with two thresholds:
  • elevated (80% of budget): triggers conversation compaction
  • critical (90% of budget): forces aggressive compaction + cache pruning
• src/utils/memoryCompaction.ts — Connects pressure levels to compaction actions

Modified Files

• src/utils/backgroundHousekeeping.ts — Wired cleanupOldPastes() into background housekeeping (previously existed but was never auto-triggered)
• src/main.tsx — Start memory pressure monitor on initialization
• src/QueryEngine.ts — Added OPENCLAUDE_MAX_ACTIVE_MESSAGES check (default: 200) to trigger compaction when message count exceeds limit
• bin/openclaude — Added --max-memory=MB CLI flag that maps to existing V8 heap limit
• src/utils/concurrentSessions.ts — Added calculatePerSessionMemoryBudget() that auto-adjusts based on concurrent session count

Environment Variables

Variable	Default	Description
OPENCLAUDE_MAX_MEMORY_MB	1536	Per-session memory budget in MB
OPENCLAUDE_MAX_ACTIVE_MESSAGES	200	Max messages before compaction triggers
--max-memory=MB	8192	V8 heap limit (CLI flag)

Key Findings

During analysis, several existing mechanisms were discovered that complement this work:

• fileStateCache already uses LRU with 25MB limit (well-designed)
• Auto-compact already exists in QueryEngine with snip compaction
• Heap dump service already triggers at 1.5GB for diagnostics
• bin/openclaude already has --max-old-space-size (default 8GB)

The main gaps were: no memory pressure detection, paste cleanup never called automatically, no message count limits, and no concurrent session budget scaling.

Testing

• Build: ✅ bun run build passes
• Tests: ✅ 3020/3021 pass (1 pre-existing failure unrelated to these changes)
• No breaking changes — all behavior is additive with sensible defaults

How It Works

1. On startup, memoryPressure monitor begins watching RSS every 30s
2. When RSS exceeds 80% of the per-session budget (default 1.5GB), it fires elevated pressure
3. The compaction trigger requests conversation compaction from QueryEngine
4. If RSS exceeds 90%, it forces aggressive compaction and clears caches
5. Paste store cleanup runs automatically during background housekeeping (7-day TTL)
6. Concurrent sessions auto-adjust their memory budgets (e.g., 4 sessions on 32GB = ~7GB each)

Related

• Addresses the OOM issue reported with 4 sessions on 32GB RAM
• Complements existing heap dump diagnostics at 1.5GB