A beginner-friendly guide to understanding Agent Memory, context compression, and memory architectures for long-running AI agents.
agent memory manages short-term context window limits while persisting key decisions, learnings, and state across iterations
- The Simple Idea
- Short-Term vs Long-Term Memory
- Context Window Management
- Memory Summarization & Pruning
- My Learning Notes
- Common Misunderstandings
- Related Concepts
Large Language Models have finite context windows. As an agent performs dozens of tool calls, raw conversation logs quickly overflow the window or degrade reasoning accuracy.
Agent Memory decouples transient execution steps from long-term knowledge:
Transient Logs (Raw Tool Calls) -> Summarizer -> Long-Term Memory (State & Learnings)
The agent keeps key facts, architectural rules, and user preferences in memory while pruning verbose raw logs.
| Memory Tier | Storage Scope | Contents | Persistence |
|---|---|---|---|
| Short-Term (Context Window) | Current session | Active prompt, recent tool calls & results | Volatile / reset per task |
| Working Memory | Task lifetime | Current plan, open questions, intermediate variables | Task duration |
| Long-Term Memory | Global / Multi-session | User preferences, codebase rules (AGENTS.md), Knowledge Items |
Permanent |
Strategies for preserving context quality:
- Sliding Window Pruning: Truncating old tool responses while keeping system prompts and recent steps intact.
- Knowledge Items (KIs): Storing structured markdown snapshots of codebase patterns and past learnings.
- Structured Summaries: Summarizing completed subtasks into 2-line summaries before proceeding.
Raw Tool Output (500 lines) -> Summarizer -> "go test passed across 60 packages" (1 line)
By condensing tool outputs into concise observations, the agent preserves context space for reasoning about future steps.
Early agent designs saved every single tool output into context, causing model slowdowns and hallucinations.
Structured memory management is essential:
keep context clean — store details in persistent files and feed only relevant summaries back to the prompt
"A larger context window removes the need for agent memory."
No. Larger windows increase token latency and cost, and can lead to attention drift ("lost in the middle"). Memory management improves accuracy regardless of context size.
Previous: tool-calling
Next: prompt-routing
Related: multi-agent