fix(labs): clean Batch 83 lint issues

.github/workflows/backend-integration.yml

@@ -0,0 +1,72 @@
+name: Backend integration (Lab 57)
+
+# Runs the skip-unless-live integration test from labs/57-real-backends-integration against real
+# Redis and LocalStack (SQS) service containers. The same test skips locally when REDIS_URL /
+# AWS_ENDPOINT_URL are unset, so this workflow is the "live" half of that contract.
+
+on:
+  push:
+    branches: [main]
+    paths: ["labs/57-real-backends-integration/**", ".github/workflows/backend-integration.yml"]
+  pull_request:
+    paths: ["labs/57-real-backends-integration/**", ".github/workflows/backend-integration.yml"]
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  integration:
+    name: Real Redis + LocalStack
+    runs-on: ubuntu-latest
+
+    services:
+      redis:
+        image: redis:7
+        ports: ["6379:6379"]
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 5
+      localstack:
+        image: localstack/localstack:3
+        ports: ["4566:4566"]
+        env:
+          SERVICES: sqs
+        options: >-
+          --health-cmd "curl -sf http://localhost:4566/_localstack/health || exit 1"
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 10
+
+    env:
+      REDIS_URL: redis://localhost:6379
+      AWS_ENDPOINT_URL: http://localhost:4566
+      AWS_ACCESS_KEY_ID: test
+      AWS_SECRET_ACCESS_KEY: test
+      AWS_DEFAULT_REGION: us-east-1
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install redis boto3 pytest
+
+      - name: Self-test (offline, via fakeredis + moto)
+        working-directory: labs/57-real-backends-integration
+        run: |
+          pip install fakeredis "moto[sqs]"
+          python backends.py --self-test
+
+      - name: Integration test (live Redis + LocalStack)
+        working-directory: labs/57-real-backends-integration
+        run: pytest test_integration.py -v

README.md

@@ -17,9 +17,9 @@
 <table>
 <tr>
 <td align="center"><b>9</b><br><sub>Learning paths<br>(all substantial)</sub></td>
-<td align="center"><b>57</b><br><sub>Hands-on labs<br>(notebooks)</sub></td>
+<td align="center"><b>59</b><br><sub>Hands-on labs<br>(notebooks)</sub></td>
 <td align="center"><b>8</b><br><sub>Capstone<br>projects</sub></td>
-<td align="center"><b>15</b><br><sub>Math foundations<br>(with code)</sub></td>
+<td align="center"><b>16</b><br><sub>Math foundations<br>(with code)</sub></td>
 </tr>
 <tr>
 <td align="center"><b>12</b><br><sub>Architecture<br>patterns</sub></td>

concepts/observability/README.md

@@ -8,6 +8,7 @@ In 2026, agent observability became a discipline of its own: agents run for hour
 
 | Page | Read time | Covers |
 |------|-----------|--------|
+| 📖 [from-offline-mocks-to-live-ci.md](./from-offline-mocks-to-live-ci.md) | ~9 min | The last gap: running the real-client code against live infrastructure - a test pyramid (offline self-tests on every push, live Redis/LocalStack integration in CI, scheduled red-team generation), a real OTLP collector pipeline, and what stays unchanged across the move. With CI and trace-pipeline diagrams. |
 | 📖 [from-stand-ins-to-production.md](./from-stand-ins-to-production.md) | ~9 min | The move from each observability stand-in (file queue, fixed threshold, additive shift, hand-built data, keyword detector) to its production form, with a pipeline diagram and the one rule that survives every swap: code to the contract, calibrate on held-out data, keep a human where the automated grade is only a floor. |
 | 📖 [observability-for-agent-pms.md](./observability-for-agent-pms.md) | ~18 min | Why observability is the priority for agent teams, the four pillars (traces, evals, alerts, red teaming) and how they form one loop, and where the PM's job ends and engineering's begins — with the metrics a PM should own. |
 

concepts/observability/from-offline-mocks-to-live-ci.md

@@ -0,0 +1,56 @@
+# From offline mocks to live CI
+
+> Concept note. About 9 minutes to read. Companion to [`from-stand-ins-to-production.md`](./from-stand-ins-to-production.md) and Module 27 of the [Agentic RAG path](../../learning-paths/02-agentic-rag/).
+
+Module 26 made the durable backends, the trace pipeline, and the red-team scorer use the *real* client libraries, verified offline against `fakeredis` / `moto` / an in-memory exporter. This note is about the last gap: those offline mocks mirror documented behavior, not every server edge case, so the same code has to also run against live infrastructure - in CI, on a schedule, and against a real collector and a real model. The principle is a test pyramid for systems with external dependencies: fast offline tests on every change, slower live tests where the mocks stop being faithful.
+
+## The test pyramid for real backends
+
+```mermaid
+flowchart TD
+    A[Every push / PR] --> B[Offline self-tests<br/>fakeredis · moto · in-memory exporter]
+    B -->|fast, deterministic, no services| C{green?}
+    C -->|yes| D[Live integration job<br/>Redis + LocalStack service containers]
+    D -->|skip-unless-live test, now live| E{green?}
+    E -->|yes| F[Merge]
+    C -->|no| X[Fail fast]
+    E -->|no| X
+    G[Nightly / scheduled] --> H[Red-team generation + LLM judge<br/>garak · PyRIT · AgentDojo + real model]
+    H --> I[Gate on pass rate]
+```
+
+The bottom of the pyramid is the offline self-test: `backends.py --self-test` runs the real `redis-py` and `boto3` code through `fakeredis` and `moto` in milliseconds, on every push, with no services to stand up. The middle is the live integration job: the *same* `test_integration.py` that skips locally now runs against Redis and LocalStack service containers (see [`.github/workflows/backend-integration.yml`](../../.github/workflows/backend-integration.yml)), catching what the mocks can't - real consumer-group lag, real visibility-timeout timing, real redrive. The top is expensive and adversarial: red-team generation with a real model judge, run on a schedule rather than per-commit because it costs tokens and time.
+
+The reason to keep all three, rather than only the live tests, is feedback speed. A broken contract should fail in seconds on the developer's machine; a server-specific bug can wait for the CI job. Pushing everything to live tests makes the suite slow, flaky (services time out), and expensive.
+
+## Item 1: backend integration in CI
+
+The skip-unless-live design from [Lab 57](../../labs/57-real-backends-integration/) is what makes this clean. The test reads `REDIS_URL` / `AWS_ENDPOINT_URL`; locally they're unset and it skips, so `pytest` is safe to run anywhere. In CI, service containers set those variables and the identical test runs live. Nothing in the test branches on "am I in CI" - the environment supplies the endpoint or it doesn't. Use unique stream/queue names per run and tear them down, so reruns don't collide.
+
+## Item 3: a real collector and trace backend
+
+```mermaid
+flowchart LR
+    A[Agent] -->|OTLP gRPC/HTTP<br/>gen_ai.* spans| C[OTel Collector]
+    C -->|batch · filter genai · sample| B[(Trace backend<br/>Tempo / Jaeger / vendor)]
+    B -->|query API| E[Eval service<br/>cost + quality loops]
+    E --> G{Gates}
+```
+
+[Lab 56](../../labs/56-production-traces-routing/)'s `eval_service.py` reads spans from a JSONL stand-in. The [`ops/otel-collector/`](../../ops/otel-collector/) bundle replaces it with a real pipeline: the agent swaps its in-memory exporter for an OTLP exporter pointed at the collector, the collector batches and filters to `gen_ai.chat` spans and forwards to a backend, and the eval service queries that backend. The span shape - the `gen_ai.*` semantic-convention attributes - never changes, so the reconstruction logic in `eval_service.py` is untouched; only its *source* moves from a file to a backend query. The collector is what decouples the agent from the backend: batching, filtering, sampling, and fan-out to multiple backends live in collector config, changeable without redeploying the agent.
+
+## Item 4: red-team generation and the judge in CI
+
+[Lab 52](../../labs/52-red-teaming-trajectories/)'s `redteam_adapters.py` turns garak / PyRIT / AgentDojo output into the trajectory schema, and `AnthropicJudge` backs the LLM-judge hook with a real (injectable) model. In CI this is a scheduled job, not a per-commit one: generate trajectories from the tools, score them with the keyword detectors plus the LLM judge, and gate on the per-category pass rate. The judge call needs an API key (a CI secret) and costs tokens, which is why it runs nightly rather than on every push - and why the keyword detectors, which are free and deterministic, run in the fast offline tier and the judge only re-grades the cases they flag as ambiguous.
+
+## What does not change
+
+- **The code under test.** The offline self-test and the live integration test run the *same* backend code; the only difference is the injected client. That is the whole point of coding to the contract.
+- **The span shape.** Moving from an in-memory exporter to a collector changes the transport, not the `gen_ai.*` attributes, so the eval loop is portable.
+- **The discipline.** Fast deterministic tests on every change; slow, costly, or flaky tests (live services, real models) on a schedule or behind a path filter. Match the tier to the cost and the failure mode.
+
+## See also
+
+- 📖 [From stand-ins to production](./from-stand-ins-to-production.md) — the swaps this puts under CI.
+- 🧪 [Lab 57](../../labs/57-real-backends-integration/) (integration test), [Lab 56](../../labs/56-production-traces-routing/) (traces), [Lab 52](../../labs/52-red-teaming-trajectories/) (red-team adapters + judge).
+- ⚙️ [`.github/workflows/backend-integration.yml`](../../.github/workflows/backend-integration.yml) and [`ops/otel-collector/`](../../ops/otel-collector/).

glossary/terms.md

@@ -140,6 +140,8 @@ Where a term has no dedicated repo page yet, the entry links to a canonical exte
 
 **Cosine similarity.** $\cos(\mathbf{u}, \mathbf{v}) = \mathbf{u} \cdot \mathbf{v} / (\lVert \mathbf{u} \rVert \lVert \mathbf{v} \rVert)$. The angle-only similarity used in nearly every vector search. → [`math-foundations/02-embeddings-vector-similarity.md`](../math-foundations/02-embeddings-vector-similarity.md).
 
+**Covariate shift.** Distribution shift where the inputs change but the input-to-label relationship does not; a threshold or model fit on the old input distribution may no longer be calibrated. → [`math-foundations/16-threshold-selection-under-shift.md`](../math-foundations/16-threshold-selection-under-shift.md).
+
 **CrewAI.** Agent framework emphasizing role-based crews — specialist agents with assigned roles, goals, and backstories — coordinated through tasks. → [crewai.com](https://www.crewai.com/) (needs manual verification).
 
 **Critic agent.** Second-pass agent that reviews another agent's output and provides structured feedback. Core to generator-critic patterns and reflection. → [`patterns/07-reflection.md`](../patterns/07-reflection.md).
@@ -148,6 +150,8 @@ Where a term has no dedicated repo page yet, the entry links to a canonical exte
 
 **Cross-encoder.** Model that scores a query-document *pair* directly, rather than embedding each independently. Used for reranking: slower than dual-encoder retrieval, more accurate. → Path 02.
 
+**Cross-validation.** Estimating held-out performance by partitioning the data into k folds and, for each, fitting on the other k-1 and evaluating on the held-out fold; lower variance than a single split when data is scarce. → [`math-foundations/16-threshold-selection-under-shift.md`](../math-foundations/16-threshold-selection-under-shift.md).
+
 ## D
 
 **DAG (Directed Acyclic Graph).** Graph with directed edges and no cycles. The natural shape for plan-and-execute workflows, since acyclicity bounds runtime to at most $\|V\|$ steps. → [`math-foundations/10-multi-agent-coordination.md`](../math-foundations/10-multi-agent-coordination.md).
@@ -286,6 +290,8 @@ Where a term has no dedicated repo page yet, the entry links to a canonical exte
 
 **Idempotent (tool).** Tool whose effect on the world is the same whether called once or multiple times. Property that makes retry-on-failure safe. → [`concepts/tools/`](../concepts/tools/).
 
+**In-sample optimism.** The upward bias from tuning a threshold (or any selected parameter) and reporting its accuracy on the same data; scales with the number of candidates and shrinks like 1/sqrt(n). → [`labs/59-retuning-on-held-out-pairs/`](../labs/59-retuning-on-held-out-pairs/).
+
 **Indirect prompt injection.** Attack where the malicious instruction lives inside content the agent retrieves (a web page, a PDF, a tool result) rather than in the user's message. Harder to defend than direct injection. → [`security/`](../security/) + Path 07 Module 4.
 
 **Inference.** Running a trained model to produce output. Distinct from training. The unit of cost in production. → [`production/`](../production/).
@@ -432,6 +438,8 @@ Where a term has no dedicated repo page yet, the entry links to a canonical exte
 
 **Orchestration.** The layer that decides which agent or tool runs next and routes data between them. Examples: LangGraph, AutoGen, CrewAI, custom Python. → Path 03.
 
+**OTLP collector.** A process that receives OpenTelemetry spans over OTLP, batches/filters/samples them, and forwards to one or more trace backends - decoupling the agent (which only speaks OTLP) from the backend. → [`ops/otel-collector/`](../ops/otel-collector/).
+
 ## P
 
 **Parallel tool calls.** Agent capability to invoke multiple tools simultaneously in one step, with the model emitting all calls together and the runtime executing them concurrently. Major latency win when tools are independent. → [OpenAI docs](https://platform.openai.com/docs/guides/function-calling) (needs manual verification).
@@ -550,8 +558,12 @@ Where a term has no dedicated repo page yet, the entry links to a canonical exte
 
 **Sentence-transformer.** Family of dense text embedding models (e.g. all-MiniLM-L6-v2) producing normalized sentence vectors for semantic similarity; the production embedder behind the change-detection fingerprint. → [`labs/55-calibrated-detection-judgment/`](../labs/55-calibrated-detection-judgment/).
 
+**Service container (CI).** A dependency (Redis, LocalStack, a database) that a CI job starts alongside the test runner, so a skip-unless-live integration test runs against real infrastructure in CI. → [`labs/57-real-backends-integration/`](../labs/57-real-backends-integration/).
+
 **SFT (Supervised Fine-Tuning).** Fine-tuning a base model on input-output pairs to teach a specific task. The step before RLHF or DPO in modern alignment pipelines. → Path 06.
 
+**Shared-space embedding.** A multimodal embedding (CLIP/SigLIP) that maps images and text into one vector space so a text query can retrieve an image directly; strong on appearance, blind to dense in-image text. → [`labs/60-multimodal-rag-runnable/`](../labs/60-multimodal-rag-runnable/).
+
 **SigLIP.** Sigmoid-loss image-text embedding model, a successor to CLIP with stronger zero-shot retrieval; a shared-space option for multimodal RAG. → [`concepts/rag/multimodal-rag.md`](../concepts/rag/multimodal-rag.md).
 
 **SKILL.md.** Convention (Anthropic Claude Code, Claude.ai) where a folder contains a SKILL.md file teaching the agent how to perform a category of tasks. Used by Claude's file-creation and document-editing skills. → [Anthropic docs](https://docs.claude.com/en/docs/claude-code/skills) (needs manual verification).

labs/59-retuning-on-held-out-pairs/README.md

@@ -0,0 +1,57 @@
+# Lab 59: Re-tuning the threshold on held-out pairs
+
+> 🔴 Advanced · ⏱ ~70–85 min · 📚 Builds on Lab 55 · Module 27
+
+## 🎯 Goal
+
+When you swap in a real sentence-transformer ([Lab 55](../55-calibrated-detection-judgment/) `embedders.py`) and re-tune the change-detection threshold, the question is how to tune *out-of-sample* and how many labeled pairs you need. In-sample tuning — pick the threshold that maximizes accuracy on a set of pairs, then report that accuracy — is optimistically biased. This lab measures that optimism as a function of sample size and shows the held-out (cross-validation) fix.
+
+By the end you should be able to:
+
+- Explain why in-sample threshold selection inflates the reported accuracy.
+- Use k-fold cross-validation to get an held-out estimate and a stable threshold.
+- Read the optimism curve to decide how many reflow/edit pairs to label.
+
+## 📋 Prerequisites
+
+- 🧪 [Lab 55](../55-calibrated-detection-judgment/) (the threshold and the swappable embedder).
+- 📐 [math-foundations/16](../../math-foundations/16-threshold-selection-under-shift.md) — in-sample optimism and held-out validation.
+- **Assumed background:** ROC/Youden's J, cross-validation, the bias–variance idea.
+
+**Setup:** Python 3.11+, standard library. `--real-embedder` uses a sentence-transformer when installed; the char-trigram stand-in is the offline default.
+
+## 🛠 Module
+
+| Component | Notes |
+|---|---|
+| `retune.py` | `make_pairs` (overlapping hard cases), `tune_threshold`, `cv_accuracy`, `optimism_curve` (`--self-test`) |
+
+## What the numbers say
+
+| n (pairs) | in-sample acc | held-out acc | optimism |
+|---|---|---|---|
+| 16 | 0.881 | 0.850 | +0.031 |
+| 40 | 0.841 | 0.823 | +0.019 |
+| 80 | 0.811 | 0.809 | +0.002 |
+| 160 | 0.823 | 0.823 | +0.000 |
+
+## Design choices and tradeoffs
+
+- **A threshold is a fitted parameter.** It earns the same train/validation discipline as any model. Tuning and reporting on the same pairs is data leakage; the inflation is real and one-sided.
+- **Optimism shrinks with n.** The bias is large when the sample is small relative to the threshold's freedom to chase noise, and vanishes as n grows — so the curve is a label-budget tool, not just a warning.
+- **Cross-validate for the estimate, refit on all data for the deploy threshold.** Use CV to estimate unbiased accuracy and threshold stability; fit the threshold you actually ship on all your labeled pairs.
+
+## Common gotchas
+
+- **Don't report in-sample accuracy.** If you tuned the threshold on a set, that set's accuracy is optimistic — quote the held-out number.
+- **Hard cases matter.** Paraphrases (meaning preserved, low cosine) and negations (meaning changed, high cosine) are where the threshold actually lives; a clean, separable sample hides the problem and the optimism.
+- **Re-tune on change.** A threshold fit for the char-trigram embedder doesn't transfer to a sentence-transformer — re-run the whole procedure when the embedder or data distribution changes.
+
+## 🧮 Going deeper
+
+- 📐 [math-foundations/16](../../math-foundations/16-threshold-selection-under-shift.md) — the bias of in-sample selection and how it scales with n.
+- 🧪 [Lab 55](../55-calibrated-detection-judgment/) — the threshold and the swappable embedder.
+
+## What comes next
+
+- 🧪 [Lab 60: Multimodal RAG, runnable](../60-multimodal-rag-runnable/) — the other retrieval-science gap: retrieval over images and tables.