AI Eng Prep — Batch 04: Memory & context

AI-Engineering-Preparation/CHANGELOG.md

@@ -5,6 +5,8 @@ All notable changes to the AI Engineering Preparation track are recorded here. T
 ## [Unreleased]
 
 ### Added
+
+- **📦 Batch 04 — Memory & context.** Completes Path 03. Six concept notes: three under `concepts/memory/` (state vs. memory; short-term, long-term, and external memory; the memory lifecycle) and three under `concepts/context/` (context engineering; context strategies — retrieval, compaction, note-taking, isolation; context rot and failure modes). One runnable, offline, deterministic lab — `labs/04-memory-and-context/` — building a checkpointed state-vs-memory agent (`checkpoints.py`: rollback re-runs only affected steps, memory persists) and a context-budget assembler (`context_budget.py`: select by priority, compact the overflow), each with a `--self-test`, plus scaffold and solution notebooks. One diagram (`diagrams/agent-state-and-memory.md`). Navigation, glossary, references, and counts updated; Path 03 is now complete.
 - **📦 Batch 03 — Retrieval.** Delivers the retrieval half of Path 03. Six concept notes: three under `concepts/rag/` (RAG end-to-end; chunking and retrieval; reranking and citation) and three under `concepts/vector-db/` (similarity and approximate nearest neighbors; HNSW; IVF and quantization). One runnable, offline, deterministic lab — `labs/03-rag-and-ann/` — building a minimal RAG pipeline (`rag.py`: TF-IDF retrieval, grounded answer, citation, abstention) and an exact-vs-IVF nearest-neighbor tradeoff study (`ann.py`: recall vs. vectors scanned), each with a `--self-test`, plus scaffold and solution notebooks. One math page (`math-foundations/03-nearest-neighbor-search.md`) on the k-NN problem, exact cost, recall@k, and the IVF tradeoff. One diagram (`diagrams/rag-pipeline.md`). Navigation, glossary, and counts updated; the `rag` and `vector-db` areas now link the delivered notes and lab. Memory and context engineering complete Path 03 next.
 
 - **📦 Batch 02 — ML & RL fundamentals.** Fills in Path 02. Six concept notes (the ML lifecycle; feature stores and training/serving skew; monitoring, drift, and retraining; RL primitives; policy gradients and PPO; RLHF from preferences to policy), a runnable RL lab (`labs/02-rl-from-scratch/`: Q-learning and a Bradley-Terry reward model), a math page (`math-foundations/02-rl-objectives.md`), and a diagram (`diagrams/ml-lifecycle.md`).
@@ -13,4 +15,12 @@ All notable changes to the AI Engineering Preparation track are recorded here. T
 
 - **🧱 Batch 00 — scaffold.** Establishes the AI Engineering Preparation sub-folder inside the parent repository: landing README, changelog, dual-license note, and style rules; five learning-path index pages, twelve concept-area index pages, five project specs, and stubs for labs, math-foundations, diagrams, the glossary, and a seeded canonical references list.
 
-  Curriculum scope was mapped from a set of external topic guides used only for sequencing; all wording is original and technical claims point to canonical sources (see `references/references.md`).
+- **📦 Batch 03 — Retrieval.** Delivered the retrieval half of Path 03: six concept notes (RAG end-to-end; chunking and retrieval; reranking and citation; similarity and ANN; HNSW; IVF and quantization), a RAG-and-ANN lab (`labs/03-rag-and-ann/`), a math page (`math-foundations/03-nearest-neighbor-search.md`), and a diagram (`diagrams/rag-pipeline.md`).
+
+- **📦 Batch 02 — ML & RL fundamentals.** Filled in Path 02: six concept notes (ML lifecycle; feature stores; monitoring/drift; RL primitives; policy gradients; RLHF), an RL lab (`labs/02-rl-from-scratch/`), a math page (`math-foundations/02-rl-objectives.md`), and a diagram (`diagrams/ml-lifecycle.md`).
+
+- **📦 Batch 01 — LLM foundations.** Filled in Path 01: six concept notes under `concepts/llm/`, a tokenization/embeddings lab (`labs/01-tokenization-and-embeddings/`), a math page (`math-foundations/01-embeddings-and-similarity.md`), and a diagram (`diagrams/text-to-prediction.md`).
+
+- **🧱 Batch 00 — scaffold.** Established the AI Engineering Preparation sub-folder inside the parent repository: landing README, changelog, dual-license note, and style rules; five learning-path index pages, twelve concept-area index pages, five project specs, and stubs for labs, math-foundations, diagrams, the glossary, and a seeded canonical references list.
+
+Curriculum scope was mapped from a set of external topic guides used only for sequencing; all wording is original and technical claims point to canonical sources (see `references/references.md`).

AI-Engineering-Preparation/README.md

@@ -2,6 +2,7 @@
 
 A structured, open preparation hub for **AI engineering** roles and system design — built inside the [Agentic AI Engineer](../) repository as a companion track. It turns the moving pieces of modern LLM and agent systems into a navigable curriculum: learning paths, modular concept notes, runnable labs, capstone projects, diagrams, a glossary, and a curated set of canonical references.
 
+> Status: **Batch 04 — Memory & context.** Paths 01–03 are complete: LLM foundations; ML & RL fundamentals; and retrieval & memory (RAG, vector databases, agent state vs. memory, the memory lifecycle, and context engineering), with four runnable labs. Paths 04 and 05 fill in next.
 > Status: **Batch 03 — Retrieval.** Paths 01 and 02 are complete; Path 03's retrieval half (RAG end-to-end and vector databases / ANN) is delivered, with a from-scratch RAG-and-ANN lab. Memory and context engineering complete Path 03 in the next batch.
 
 ## Learning paths
@@ -34,6 +35,8 @@ AI-Engineering-Preparation/
 |---|---|---|
 | Learning paths | 5 | 5 |
 | Concept areas | 12 | 12 |
+| Concept notes | 24 | grows per batch |
+| Labs | 4 | grows per batch |
 | Concept notes | 18 | grows per batch |
 | Labs | 3 | grows per batch |
 | Projects (specs) | 5 | 5, then built out |

AI-Engineering-Preparation/concepts/context/README.md

@@ -2,19 +2,18 @@
 
 Treating the context window as a working set to be budgeted: what competes for space, how to retrieve only what the current step needs, and how to summarize and drop stale material.
 
-> Batch 00: area scaffold. The notes below are planned; they land in later batches.
+> Batch 04: notes delivered. Runnable companion: [`labs/04-memory-and-context/`](../../labs/04-memory-and-context/) (`context_budget.py`).
 
-## Planned notes
+## Notes
 
-- What fills the context window.
-- Retrieval scoped to the current step.
-- Summarization and dropping stale checkpoints.
-- Keeping memory outside the prompt.
-- Cost, latency, and reliability tradeoffs.
+1. [Context engineering](./context-engineering.md) — designing what the model sees each call; write/select/compress/isolate.
+2. [Context strategies](./context-strategies.md) — just-in-time retrieval; compaction vs. summarization; note-taking; sub-agent isolation.
+3. [Context rot and failure modes](./context-rot-and-failure-modes.md) — why long contexts degrade, and what to do about it.
 
 ## Key references
 
-- Effective context engineering for AI agents (official engineering guidance).
+- Effective context engineering for AI agents — Anthropic (2025).
+- Context Rot — Chroma technical report (2025).
 - Lost in the Middle — arXiv:2307.03172.
 
 See the full list in [`../../references/references.md`](../../references/references.md). All explanations are original; sources are cited, not reproduced ([`STYLE.md`](../../STYLE.md)).

AI-Engineering-Preparation/concepts/context/context-engineering.md

@@ -0,0 +1,35 @@
+# Context engineering
+
+> Concept note. ~9 min. Builds on [the context window](../llm/context-window.md). Runnable companion: [`labs/04-memory-and-context/`](../../labs/04-memory-and-context/) (`context_budget.py`).
+
+**Context engineering** is the practice of deliberately designing what a model sees on every inference call. It is the discipline that grew up once builders realized that for agents, the wording of a prompt was no longer the bottleneck — the bottleneck was assembling the right instructions, the right tools, the right slice of history, and the right retrieved facts into a finite window, turn after turn, without it collapsing under its own weight.
+
+## Beyond prompt engineering
+
+Prompt engineering optimizes a single, mostly static instruction. Context engineering optimizes a *system* that builds the context dynamically, every call, from many moving sources: system and developer instructions, the conversation so far, retrieved knowledge, [memory](../memory/), tool definitions, and tool outputs. In an agent that runs for dozens of steps, what fills the window changes constantly, and managing that flow — not phrasing one prompt — is where reliability is won or lost.
+
+## Four moves: write, select, compress, isolate
+
+A useful way to organize the techniques is by what they do to information relative to the window:
+
+- **Write** — put information *outside* the context window so it persists without consuming tokens: a scratchpad of working notes, or durable [memory](../memory/). The agent writes results down and reads them back when needed.
+- **Select** — pull *into* the window only what the current step needs. This is [retrieval](../rag/), and increasingly **just-in-time retrieval**: hold lightweight identifiers (file paths, queries) and load the underlying content at runtime rather than front-loading everything.
+- **Compress** — reduce what is already in the window when it grows long: summarize history, or strip content that is recoverable from the environment (see [strategies](./context-strategies.md)).
+- **Isolate** — split context across boundaries so one subtask does not pollute another, typically by giving sub-agents their own windows and passing back only summaries.
+
+The single principle under all four is signal-to-noise: every token in the window should earn its place, because tokens that do not help actively hurt (see [context rot](./context-rot-and-failure-modes.md)).
+
+## Why it is a discipline, not a trick
+
+It would be convenient if a large enough window made this unnecessary. It does not: windows have grown to millions of tokens and the problem remains, because models use long contexts unevenly and degrade as they fill. So context engineering is ongoing work — measuring what is in the window, deciding what to keep, and building the machinery to assemble it well on every call. The lab builds the smallest version of that machinery: an assembler that selects by priority and compresses the overflow to stay within a budget.
+
+## What to remember
+
+- Context engineering is designing what the model sees each call — a dynamic system, not a static prompt.
+- Organize the techniques as write, select, compress, isolate; the unifying goal is signal-to-noise in a finite window.
+- A bigger window does not retire the discipline, because long contexts are used unevenly.
+
+## References
+
+- Anthropic (2025). *Effective context engineering for AI agents* — official engineering guidance.
+- Hong, K., Troynikov, A., Huber, J. (2025). *Context Rot: How Increasing Input Tokens Impacts LLM Performance.* Chroma technical report. See [`../../references/references.md`](../../references/references.md).

AI-Engineering-Preparation/concepts/context/context-rot-and-failure-modes.md

@@ -0,0 +1,32 @@
+# Context rot and failure modes
+
+> Concept note. ~8 min. Builds on [context engineering](./context-engineering.md); revisits [the context window](../llm/context-window.md).
+
+The reason context engineering exists is that long contexts fail in measurable ways. Understanding *how* they fail is what justifies the effort of keeping a window small and clean rather than just making it bigger.
+
+## Context rot
+
+A natural assumption is that a model treats its 100,000th token as reliably as its 100th. It does not. **Context rot** is the measured degradation in output quality as input length grows — and it shows up even when the window is nowhere near full, and even on simple tasks. Controlled evaluation across many frontier models found that performance becomes less reliable as inputs lengthen, for every model tested. The practical lesson is blunt: "the window is big enough" is not the same as "the model will use it well." More tokens in can mean worse output out.
+
+## The mechanisms
+
+Three compounding effects drive context rot:
+
+- **Lost in the middle.** Models attend best to the start and end of the context and worst to the material in between, so a fact buried in the middle of a long context is often missed even though it is present. Accuracy can drop sharply for mid-context information.
+- **Attention dilution.** [Attention](../llm/attention.md) compares every token with every other, so the relationships to track grow with the square of the length. As the window fills, the signal for any one relevant token is spread thinner.
+- **Distractor interference.** Content that is semantically similar to the query but irrelevant does not just take up space — it actively misleads, pulling the model toward plausible-but-wrong material.
+
+## What this means for design
+
+These failure modes turn "add more context" from a safe default into a risk. The responses are exactly the [strategies](./context-strategies.md) of the previous note: retrieve only what the step needs so distractors never enter; compact and summarize so the window stays short; place the most important material where attention is strongest (near the start or end), not buried in the middle; and isolate subtasks so their clutter does not dilute the main thread. The unifying aim is to keep the signal-to-noise ratio high, because the model's effective use of context — not the window's advertised size — is what bounds performance.
+
+## What to remember
+
+- Context rot is the measured drop in quality as input grows, present even below the window limit and on simple tasks.
+- It is driven by lost-in-the-middle positioning, attention dilution, and distractor interference.
+- The fix is selection and compression, and placing key material where the model attends best — not a bigger window.
+
+## References
+
+- Hong, K., Troynikov, A., Huber, J. (2025). *Context Rot: How Increasing Input Tokens Impacts LLM Performance.* Chroma technical report.
+- Liu, N. F., et al. (2023). *Lost in the Middle: How Language Models Use Long Contexts.* arXiv:2307.03172. See [`../../references/references.md`](../../references/references.md).

AI-Engineering-Preparation/concepts/context/context-strategies.md

@@ -0,0 +1,36 @@
+# Context strategies: retrieval, compaction, note-taking, isolation
+
+> Concept note. ~9 min. Builds on [context engineering](./context-engineering.md).
+
+The four moves — write, select, compress, isolate — show up in practice as a handful of concrete techniques. These are the levers you actually pull to keep an agent's window clean over a long task.
+
+## Just-in-time retrieval (select)
+
+Instead of front-loading everything an agent might need, keep lightweight references — file paths, identifiers, queries — and load the underlying content only when a step needs it. A coding agent does not hold every file in context; it holds the paths and reads a file when it acts on it. This keeps the window small and current, and mirrors how a person works from references rather than memorizing the whole codebase. It is [retrieval](../rag/) applied to the agent's own working set.
+
+## Compaction vs. summarization (compress)
+
+When history grows long, you reduce it — but there are two distinct ways, and the difference matters.
+
+- **Compaction (reversible).** Strip content that is recoverable from the environment, replacing it with a pointer. If a tool returned a 500-line file earlier, the history does not need the full contents — it needs the path, because the agent can re-read the file if required. Nothing is truly lost; it is offloaded. This is the safest, lightest-touch reduction, and clearing old raw tool results is the low-hanging fruit.
+- **Summarization (lossy).** Use a model to summarize a stretch of history into a shorter form, typically when the window nears a threshold, then continue from the summary. This recovers a lot of room but discards detail permanently, so the art is choosing what survives — over-aggressive summarization drops a fact whose importance only surfaces later.
+
+The guidance is to prefer reversibility where you can (offload, keep a pointer) and reserve lossy summarization for when you must.
+
+## Structured note-taking (write)
+
+Have the agent write durable notes *outside* the context window — a scratchpad, a running plan, a file — and read them back later, rather than holding everything in the conversation. This persists progress across compaction and even across sessions, and it gives the agent an explicit, inspectable working memory instead of an implicit one buried in chat history.
+
+## Sub-agent isolation (isolate)
+
+For a complex task, give a subtask its own context window via a sub-agent: it does focused work in isolation and returns only a concise result to the main agent. The main thread never sees the subtask's intermediate clutter, which keeps its window clean and its attention on the overall goal. The cost is coordination and the risk of losing detail in the handoff, so isolation pays off when subtasks are genuinely separable.
+
+## What to remember
+
+- Just-in-time retrieval loads content on demand from lightweight references instead of front-loading it.
+- Prefer reversible compaction (offload, keep a pointer) over lossy summarization; summarize only when you must.
+- Structured note-taking externalizes working memory; sub-agent isolation keeps subtask clutter out of the main window.
+
+## References
+
+- Anthropic (2025). *Effective context engineering for AI agents* — official engineering guidance. See [`../../references/references.md`](../../references/references.md).

AI-Engineering-Preparation/concepts/memory/README.md

@@ -2,21 +2,18 @@
 
 Why long-running agents need more than a model call: the difference between state (the current task) and memory (what carries across tasks), the three memory kinds, and the lifecycle that keeps memory from rotting.
 
-> Batch 00: area scaffold. The notes below are planned; they land in later batches.
+> Batch 04: notes delivered. Runnable companion: [`labs/04-memory-and-context/`](../../labs/04-memory-and-context/).
 
-## Planned notes
+## Notes
 
-- State vs. memory.
-- Short-term, long-term, and external memory.
-- The memory lifecycle: create, update, summarize, delete.
-- Checkpointing and rollback.
-- Consistency under change: react fast with state, learn slowly with memory.
-- Memory failure modes.
+1. [State vs. memory](./state-vs-memory.md) — rewindable task state vs. durable carry-across; react fast with state, learn slowly with memory.
+2. [Short-term, long-term, and external memory](./memory-types.md) — the three tiers and the RAM/disk analogy.
+3. [The memory lifecycle](./memory-lifecycle.md) — create, read, update, delete; consolidation; consistency under change.
 
 ## Key references
 
-- MemGPT: Towards LLMs as Operating Systems — arXiv:2310.08560.
-- Mem0: production-ready long-term memory — arXiv:2504.19413.
-- LangGraph persistence (official docs).
+- MemGPT — arXiv:2310.08560.
+- Mem0 — arXiv:2504.19413.
+- LangGraph persistence (checkpointers) — official docs.
 
 See the full list in [`../../references/references.md`](../../references/references.md). All explanations are original; sources are cited, not reproduced ([`STYLE.md`](../../STYLE.md)).