diff --git a/authbridge/README.md b/authbridge/README.md index feee2482..44383c57 100644 --- a/authbridge/README.md +++ b/authbridge/README.md @@ -387,6 +387,7 @@ The easiest way to get all prerequisites is to use the [Rossoctl Quickstart](htt - [Manual deployment](./demos/github-issue/demo-manual.md) — deploy everything via `kubectl` and YAML manifests - [UI deployment](./demos/github-issue/demo-ui.md) — import agent and tool via the Rossoctl dashboard - **[Token-Exchange Routes](./demos/token-exchange-routes/README.md)** - Configuration reference for the `authproxy-routes` ConfigMap; covers single-target (one route) and multi-target (one agent → many tools) patterns +- **[Lineage Demo](./demos/lineage/README.md)** - Per-request data lineage from the sidecar on the Weather Agent pair, attached to the running Deployments with the [lineage attach kit](./lineage-attach/README.md); shows the same turn fragmented and then as one trace (no Keycloak involved) All demos cover configuring Keycloak, deploying, and testing. diff --git a/authbridge/demos/README.md b/authbridge/demos/README.md index 3646a186..79bf8593 100644 --- a/authbridge/demos/README.md +++ b/authbridge/demos/README.md @@ -24,6 +24,8 @@ more AuthBridge capabilities. | **[abctl Walkthrough](weather-agent/demo-with-abctl.md)** | Reference | Watch the AuthBridge plugin pipeline live with the `abctl` TUI | Tooling only | | **[IBAC](ibac/README.md)** | Intermediate | Intent-Based Access Control: LLM judge denies outbound HTTP that doesn't align with the user's recorded intent. Reproduces the email-poison / prompt-injection attack from `huang195/ibac`; chat with the agent through the rossoctl UI and see the exfiltration blocked, then `make show-result` for a pipeline-level forensic | UI + kubectl | | **[SPARC (finance)](finance-sparc/README.md)** | Intermediate | SPARC pre-tool reflection: the `sparc` plugin blocks a hallucinated/ungrounded tool argument (an invented transaction id) before it executes and transparently asks the user to clarify, then approves the corrected call. Complements IBAC — SPARC verifies argument grounding, IBAC verifies intent alignment | UI + kubectl | +| **[Lineage](lineage/README.md)** | Intermediate | Per-request lineage on the Weather Agent pair: attach the sidecar with the lineage attach kit and see one turn first as 35 separate traces (the app forwards no `traceparent`), then as one trace of 70 spans with one root once the app's own propagation is switched on. Deploys the two stock images plain; edits nothing else | kubectl + scripts | +| **[Lineage attach kit](../lineage-attach/README.md)** | Reference | Attach per-request lineage to any existing Deployment: enable the `lineage-telemetry` plugin and every HTTP exchange becomes two facts-only spans (`request` + `response`, paired by `lineage.exchange.id`) sent to **any** OTLP consumer. A strategic-merge patch + ConfigMap, generated and validated; a propagate-only OTel shim for uninstrumented Python apps, activated by one env var | kubectl + scripts | | **[CPEX Bridge (HR)](hr-cpex/README.md)** | Advanced | CPEX/APL declarative policy: one route chains a coarse APL predicate, an embedded Cedar PDP, RFC 8693 token exchange with a post-check, PII redaction and audit plugins. Same request, different data per caller (Bob sees an SSN, Eve gets it redacted). Self-contained: its own kind cluster + namespace, deployed via `make` rather than operator injection | [kubectl (make)](hr-cpex/README.md#quick-start) | ## Recommended Path diff --git a/authbridge/demos/lineage/README.md b/authbridge/demos/lineage/README.md new file mode 100644 index 00000000..daa231db --- /dev/null +++ b/authbridge/demos/lineage/README.md @@ -0,0 +1,186 @@ +# Lineage demo — the weather agent and its tool, with per-request lineage + +The [Weather Agent](../weather-agent/demo-ui.md) pair from `rossoctl/examples` +— an A2A agent that asks an LLM and calls one MCP tool — deployed plain, then +given per-request lineage with the [lineage attach kit](../../lineage-attach/README.md) +and nothing else. Six steps. You will see the same turn twice: first as +**19 separate traces** (the entry alone and each of the app's 18 calls in a +trace of its own, because the app does not carry `traceparent`), then as +**one trace of 70 spans** with one root, after the +app's own propagation is switched on. Nothing about the app is edited except +one environment variable that the app itself defines. + +Read [the kit's README](../../lineage-attach/README.md) for what the spans +carry and [DESIGN](../../lineage-attach/DESIGN.md) for why propagation is the +app's job; this page is only the walk-through. + +## Prerequisites + +- The rossoctl platform on kind (`rossoctl` cluster), namespace `team1` with + its platform-rendered `envoy-config` ConfigMap, and the platform collector + (`deploy/otel-collector` in `rossoctl-system`, stock `debug` exporter). +- A sidecar image that carries `lineage-telemetry` (cortex #761): until a release does, + [RECIPE step 1](../../lineage-attach/RECIPE.md#1-a-sidecar-image-that-carries-the-plugin-once-per-cluster-until-a-release-does) + builds and loads it. Then, for the whole session: + + ```sh + export SIDECAR_IMAGE=docker.io/library/authbridge-envoy:latest PROXY_INIT_IMAGE=docker.io/library/proxy-init:latest + ``` +- An LLM the agent can reach over **plaintext HTTP** (an HTTPS LLM is TLS + passthrough: the sidecar records no hop for it). Default: Ollama on the host + with `qwen2.5:7b` (`ollama pull qwen2.5:7b`); edit `k8s/weather.yaml`'s + `weather-llm` ConfigMap for anything else. +- Egress from the cluster to `https://wttr.in`, which the tool queries. +- `kubectl` and `python3` on the host (`ask.sh`, `show-trace.py`). + +Run everything from this directory. `KIT=../../lineage-attach`. + +## 1. Deploy the pair, plain + +```sh +kubectl apply -f k8s/weather.yaml +kubectl -n team1 rollout status deploy/weather-tool && kubectl -n team1 rollout status deploy/weather-service +./ask.sh "What is the weather in Paris?" +``` + +`k8s/weather.yaml` is the two stock images (`weather_service`, `weather_tool`) +as two Deployments and two Services — no `AgentRuntime`, no platform sidecar, +no auth. `ask.sh` sends one A2A `message/send` from a pod inside the cluster +(a port-forward would bypass the sidecar) with a `traceparent` whose trace id +it prints. Pass: an `answer:` line with the weather. Nothing is captured yet. + +## 2. Attach lineage (capture) + +```sh +for d in weather-tool weather-service; do NAMESPACE=team1 DEPLOY=$d CAPTURE_IO=true $KIT/sidecar-patch.sh; done +``` + +Pass — each ends with: + +``` +>> back out: kubectl -n team1 rollout undo deploy/ --to-revision= && kubectl -n team1 delete cm authbridge-lineage-config- +deployment "" successfully rolled out +>> lineage sidecar attached to deploy/ (self_id=, ns=team1) +``` + +That is the whole attachment: a ConfigMap and a strategic-merge patch per +Deployment, both generated by the kit. Both pods are now `2/2`. `CAPTURE_IO=true` +is the demo's choice, not the kit's default: the spans then carry the question, +the tool arguments and the prompts, so the trace reads as a story. Keep the +back-out lines; step 6 uses them. + +## 3. One turn — every hop alone + +```sh +./ask.sh "What is the weather in Paris?" # prints: trace id: +./show-trace.py +``` + +`show-trace.py` reads the collector's log and lists the sidecar spans of one +trace. Measured: + +``` +2 sidecar spans, 1 exchanges: 1 inbound a2a +parent.source: 1 wire, 0 tracestate, 0 none +traces begun by an unparented outbound hop while this one was in flight: 18 +shape: ENTRY ONLY — nothing the app called landed here; its calls are the stray traces above +``` + +The sidecar saw everything the turn did — 35 exchanges: the A2A entry, 16 MCP +exchanges to the tool (session handshakes, tool listing, the call), 2 LLM +calls, and the tool's 16 inbound sides — and recorded all 70 spans. But the +app forwarded no `traceparent`, so the entry is alone in your trace and each +of the app's 18 calls started a trace of its own: its sidecar found nothing on +the wire to parent on (`parent.source=none`), forwarded a `traceparent` of its +own making, and the tool's side of each MCP call joined *that* trace — 19 +traces, each internally consistent and each useless, because nothing links a +call to the question that caused it. This is the case DESIGN calls *the one +that looks fine and is not*: count spans and it passes; read the shape and it +fails. + +## 4. Switch the app's propagation on + +The weather agent ships its own OpenTelemetry setup, activated by one +variable it defines: when `OTEL_EXPORTER_OTLP_ENDPOINT` is set it extracts the +inbound `traceparent` and instruments `httpx`, so its LLM and tool calls carry +it. Point it at the platform collector's OTLP/HTTP receiver — port 8335 on the +stock chart, not 4318: + +```sh +kubectl -n team1 set env deploy/weather-service OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.rossoctl-system.svc.cluster.local:8335 +kubectl -n team1 rollout status deploy/weather-service +``` + +> **Why not the kit's shim here?** Because the interlock refuses this image, +> correctly: `$KIT/build-otel-shim.sh ghcr.io/rossoctl/examples/weather_service:latest` +> exits 3 with `REFUSING to bake …: it already instruments httpx`. An app that +> brings its own instrumentation gets its own switch; the shim is for the app +> that brings none ([RECIPE step 2](../../lineage-attach/RECIPE.md#2-bake-the-propagation-shim-onto-the-app-image-once-per-image)). +> The tool image bakes, but it does not need to: its one call that matters is +> HTTPS to `wttr.in`, which the sidecar passes through unseen, and its other +> outbound — its own OTLP export to the collector — is on the plugin's default +> `bypass_hosts` list, so capture is all it needs. + +## 5. The same turn — one trace + +```sh +./ask.sh "What is the weather in Paris?" +./show-trace.py +``` + +Measured: + +``` +70 sidecar spans, 35 exchanges: 1 inbound a2a, 16 inbound mcp, 2 outbound inference, 16 outbound mcp +parent.source: 1 wire, 34 tracestate, 0 none +traces begun by an unparented outbound hop while this one was in flight: 0 +shape: OK — one root, unstamped only at the entry, the app's calls are in this trace +``` + +The table above those lines is the turn, hop by hop, in time order: the A2A +entry (`wire` — the caller minted the trace), then each MCP exchange seen +twice (outbound at the agent, inbound at the tool, both `tracestate`), the two +LLM calls (`inference`, peer `host.containers.internal:11434`), and the A2A +response last. The apps' own spans arrive in the same trace too — the agent's +A2A server, LangChain and `openai.chat` spans and its `httpx` `POST`s, the +tool's `tools/list` and `tools/call` — 82 of them on this turn; the sidecar's +are the ones with `lineage.*` attributes. + +## 6. Back out + +The kit's attachment is one revision per Deployment and one ConfigMap each, +and `sidecar-patch.sh` printed the revision to return to. The tool is one +revision above plain; the agent is two, because step 4 rolled it again — a +bare `rollout undo` there would land on the attached spec whose ConfigMap is +about to go, and the pod would hang on the missing mount. Name the revision: + +```sh +kubectl -n team1 rollout undo deploy/weather-tool --to-revision=1 && kubectl -n team1 delete cm authbridge-lineage-config-weather-tool +kubectl -n team1 rollout undo deploy/weather-service --to-revision=1 && kubectl -n team1 delete cm authbridge-lineage-config-weather-service +``` + +Pass: `kubectl -n team1 get pods` shows both pods `1/1` again. Then the app: +`kubectl delete -f k8s/weather.yaml`. + +## Files + +| file | what | +|---|---| +| `k8s/weather.yaml` | the pair, plain: two Deployments, two Services, one ConfigMap for the LLM | +| `ask.sh` | one A2A turn from inside the cluster with a chosen trace id | +| `show-trace.py` | the shape of one trace from the collector log, with a verdict; exit 0 only for one root with the app's calls inside it | + +Everything that attaches lineage is the kit's; this directory holds only the +application and the two readers. + +## If it does not work + +| symptom | cause | +|---|---| +| `ask.sh` prints no answer, or the agent logs `Cannot connect to MCP` | the tool is not ready, or `MCP_URL` in `k8s/weather.yaml` does not match the Service name | +| the answer is an LLM error | `weather-llm` ConfigMap: the base URL is not reachable from a pod (podman kind: `host.containers.internal`; docker kind: `host.docker.internal`), or the model is not pulled | +| `show-trace.py` finds no spans | the sidecar image predates the plugin, or the collector was restarted — `kubectl -n team1 logs deploy/weather-service -c envoy-proxy` | +| step 5 still says FRAGMENTED | the agent did not restart with the variable — `kubectl -n team1 logs deploy/weather-service -c agent \| grep 'httpx instrumented'` | +| the agent logs export failures after step 4 | wrong collector port: the stock chart serves OTLP/HTTP on `8335`, not `4318`; propagation works regardless, but the agent's own spans do not arrive | +| the answer is a tool error about `wttr.in` | the tool needs egress to `https://wttr.in`; the sidecar passes HTTPS through, so this is cluster egress, not lineage | +| anything about the attachment itself | the kit's [Troubleshooting](../../lineage-attach/README.md#troubleshooting) | diff --git a/authbridge/demos/lineage/ask.sh b/authbridge/demos/lineage/ask.sh new file mode 100755 index 00000000..37406b11 --- /dev/null +++ b/authbridge/demos/lineage/ask.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# ask.sh — one A2A turn to the weather agent, sent from inside the cluster (a +# port-forward would bypass the sidecar) with a trace id of our choosing, so +# the spans it produces can be found by that id. +# +# The request is the A2A 0.3 JSON-RPC shape (`message/send`, `parts: [{kind: +# text}]`); the agent runs a2a-sdk 1.x and answers it through its 0.3 +# compatibility routes. The result is a Task (answer in status.message or the +# artifacts) or a Message (answer in parts); both are read. +# +# Usage: ./ask.sh ["What is the weather in Paris?"] +# NS=team1 SVC=weather-service PORT=8080 override the target. +set -euo pipefail +NS="${NS:-team1}"; SVC="${SVC:-weather-service}"; PORT="${PORT:-8080}" +question="${1:-What is the weather in Paris?}" +trace_id="$(python3 -c 'import secrets; print(secrets.token_hex(16))')" +body="$(python3 -c 'import json, sys, uuid +print(json.dumps({"jsonrpc": "2.0", "id": "1", "method": "message/send", "params": {"message": { + "role": "user", "messageId": uuid.uuid4().hex, "parts": [{"kind": "text", "text": sys.argv[1]}]}}}))' "$question")" +echo "trace id: ${trace_id}" +kubectl -n "$NS" run "ask-${trace_id:0:12}" --rm -i --quiet --restart=Never --image=curlimages/curl:8.11.1 -- \ + curl -sS --max-time 300 -H 'content-type: application/json' \ + -H "traceparent: 00-${trace_id}-0000000000000001-01" \ + -d "$body" "http://${SVC}:${PORT}/" \ + | python3 -c 'import json, sys +r = json.load(sys.stdin) +res = r.get("result", r) +parts = (res.get("status", {}).get("message", {}).get("parts", []) + or [p for a in res.get("artifacts", []) for p in a.get("parts", [])] + or res.get("parts", [])) +print("answer:", " ".join(p.get("text", "") for p in parts) or json.dumps(r)[:300])' +echo "trace id: ${trace_id} (kubectl -n rossoctl-system logs deploy/otel-collector | grep -c ${trace_id})" diff --git a/authbridge/demos/lineage/k8s/weather.yaml b/authbridge/demos/lineage/k8s/weather.yaml new file mode 100644 index 00000000..c8b15d1e --- /dev/null +++ b/authbridge/demos/lineage/k8s/weather.yaml @@ -0,0 +1,194 @@ +# The weather pair from rossoctl/examples, deployed plain: no AgentRuntime, no +# platform sidecar, no auth — exactly the shape the lineage attach kit adopts. +# Two stock images, mirroring demos/weather-agent/k8s/*-advanced.yaml minus the +# enrollment labels and the auth ConfigMaps. One deliberate difference: the +# stock agent annotates its LLM port (11434) out of the sidecar's redirect; +# here it is left in, because the LLM hop is exactly what lineage should see. +# Point the agent at your LLM in the ConfigMap (host Ollama by default; +# qwen2.5:7b answers the demo's question with one tool call). +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: weather-llm + namespace: team1 +data: + # podman-backed kind reaches the host as host.containers.internal; + # docker-backed kind as host.docker.internal. OpenAI: https://api.openai.com/v1 + # (an HTTPS LLM is TLS passthrough — the sidecar records no hop for it). + LLM_API_BASE: "http://host.containers.internal:11434/v1" + LLM_API_KEY: "ollama" + LLM_MODEL: "qwen2.5:7b" +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: weather-service + namespace: team1 + labels: + app.kubernetes.io/name: weather-service +spec: + # Cold pulls on kind can exceed the 600s default (as the stock manifests note). + progressDeadlineSeconds: 1800 + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: weather-service + template: + metadata: + labels: + app.kubernetes.io/name: weather-service + spec: + # The image runs as 1001 (its Dockerfile); match it at pod level. + securityContext: + runAsNonRoot: true + runAsUser: 1001 + runAsGroup: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: agent + image: ghcr.io/rossoctl/examples/weather_service:latest + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + ports: + - containerPort: 8000 + name: http + env: + - name: PORT + value: "8000" + - name: HOST + value: "0.0.0.0" + - name: UV_CACHE_DIR + value: "/app/.cache" + - name: MCP_URL + value: "http://weather-tool-mcp:8000/mcp" + - name: LOG_LEVEL + value: "INFO" + envFrom: + - configMapRef: + name: weather-llm + volumeMounts: + - name: cache + mountPath: /app/.cache + - name: marvin + mountPath: /.marvin + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 10 + periodSeconds: 5 + failureThreshold: 30 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: cache + emptyDir: {} + - name: marvin + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: weather-service + namespace: team1 +spec: + selector: + app.kubernetes.io/name: weather-service + ports: + - name: http + port: 8080 + targetPort: 8000 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: weather-tool + namespace: team1 + labels: + app.kubernetes.io/name: weather-tool +spec: + progressDeadlineSeconds: 1800 + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app.kubernetes.io/name: weather-tool + template: + metadata: + labels: + app.kubernetes.io/name: weather-tool + spec: + securityContext: + runAsNonRoot: true + runAsUser: 1001 + runAsGroup: 1001 + fsGroup: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: mcp + image: ghcr.io/rossoctl/examples/weather_tool:latest + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + ports: + - containerPort: 8000 + name: http + env: + - name: PORT + value: "8000" + - name: HOST + value: "0.0.0.0" + - name: UV_NO_CACHE + value: "1" + volumeMounts: + - name: cache + mountPath: /app/.cache + - name: tmp + mountPath: /tmp + readinessProbe: + tcpSocket: + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 24 + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + volumes: + - name: cache + emptyDir: {} + - name: tmp + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: weather-tool-mcp + namespace: team1 +spec: + selector: + app.kubernetes.io/name: weather-tool + ports: + - name: http + port: 8000 + targetPort: 8000 diff --git a/authbridge/demos/lineage/show-trace.py b/authbridge/demos/lineage/show-trace.py new file mode 100755 index 00000000..f41178cd --- /dev/null +++ b/authbridge/demos/lineage/show-trace.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Print the shape of one trace from the platform collector's debug log. + +The stock collector prints every span it receives (debug exporter, +verbosity: detailed). This reads that log, keeps the sidecar's spans for one +trace id and lists them in time order — self id, direction, protocol, role, +where the parent came from, the peer — then says whether the shape is right: +one unstamped hop at the entry (an inbound: `wire` when the caller sent a +`traceparent`, `none` when it sent nothing), `tracestate` everywhere else, and +the app's own outbound calls *in this trace*. An entry alone is not a good +shape: it means the app's calls went to traces of their own, and those are +counted too. + +Usage: ./show-trace.py [--since 10m] +""" + +import argparse +import collections +import re +import subprocess +import sys + +ATTR = re.compile(r"-> ([\w.]+): Str\((.*)\)$", re.M) +TRACE = re.compile(r"Trace ID\s*:\s*(\w+)") +START = re.compile(r"Start time\s*:\s*(\S+ \S+)") + + +def sidecar_blocks(log: str): + for block in re.split(r"\n(?=Span #\d+)", log): + if "lineage.role" in block: + yield block + + +def stray_outbound_traces(log: str, trace_id: str, first: str, last: str) -> int: + """Traces other than trace_id that begin with an outbound hop that had no + stamp to parent on (`wire` or `none`) — an app call that started a trace + of its own — started while trace_id was in flight (between its first and + last span).""" + strays = set() + for block in sidecar_blocks(log): + attrs = dict(ATTR.findall(block)) + tid = TRACE.search(block) + start = START.search(block) + if ( + tid + and start + and tid.group(1) != trace_id + and first <= start.group(1) <= last + and attrs.get("lineage.role") == "request" + and attrs.get("lineage.direction") == "outbound" + and attrs.get("lineage.parent.source") in ("wire", "none") + ): + strays.add(tid.group(1)) + return len(strays) + + +def spans_for(log: str, trace_id: str): + rows = [] + for block in sidecar_blocks(log): + tid = TRACE.search(block) + if not tid or tid.group(1) != trace_id: + continue + attrs = dict(ATTR.findall(block)) + start = START.search(block) + rows.append( + ( + start.group(1) if start else "", # full timestamp: sorts across midnight + attrs.get("lineage.self.id", ""), + attrs.get("lineage.direction", ""), + attrs.get("lineage.protocol", ""), + attrs.get("lineage.role", ""), + attrs.get("lineage.parent.source", ""), + attrs.get("lineage.peer.host", "")[:30], + attrs.get("lineage.outcome", ""), + ) + ) + rows.sort() + return rows + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("trace_id") + ap.add_argument("--since", default="10m", help="collector log window (kubectl --since)") + ap.add_argument("--namespace", default="rossoctl-system") + args = ap.parse_args() + log = subprocess.run( # noqa: S603 — fixed argv, no shell + ["kubectl", "-n", args.namespace, "logs", "deploy/otel-collector", "--since", args.since], + check=True, + capture_output=True, + text=True, + ).stdout + rows = spans_for(log, args.trace_id) + if not rows: + print(f"no sidecar spans for {args.trace_id} in the last {args.since}", file=sys.stderr) + return 1 + print(f"{'time':<13}{'self':<17}{'dir':<10}{'proto':<11}{'role':<10}{'parent':<12}{'peer':<31}outcome") + for r in rows: + print(f"{r[0][11:23]:<13}{r[1]:<17}{r[2]:<10}{r[3]:<11}{r[4]:<10}{r[5]:<12}{r[6]:<31}{r[7]}") + requests = [r for r in rows if r[4] == "request"] + parents = collections.Counter(r[5] for r in requests) + by_proto = collections.Counter(f"{r[2]} {r[3]}" for r in requests) + strays = stray_outbound_traces(log, args.trace_id, rows[0][0], rows[-1][0]) + print() + mix = ", ".join(f"{n} {k}" for k, n in sorted(by_proto.items())) + print(f"{len(rows)} sidecar spans, {len(requests)} exchanges: {mix}") + wire, stamped, none = parents.get("wire", 0), parents.get("tracestate", 0), parents.get("none", 0) + print(f"parent.source: {wire} wire, {stamped} tracestate, {none} none") + print(f"traces begun by an unparented outbound hop while this one was in flight: {strays}") + has_outbound = any(r[2] == "outbound" for r in requests) + # The one unstamped hop must be the first request AND an inbound: an + # unstamped outbound root is a stray trace, whatever else it holds. + unstamped = parents.get("wire", 0) + parents.get("none", 0) + one_root = unstamped == 1 and requests[0][5] in ("wire", "none") and requests[0][2] == "inbound" + if one_root and has_outbound and strays == 0: + print("shape: OK — one root, unstamped only at the entry, the app's calls are in this trace") + return 0 + if not has_outbound: + print("shape: ENTRY ONLY — nothing the app called landed here; its calls are the stray traces above") + else: + print("shape: FRAGMENTED — an unstamped non-entry hop, or stray traces, mark un-propagated calls") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/authbridge/lineage-attach/DESIGN.md b/authbridge/lineage-attach/DESIGN.md new file mode 100644 index 00000000..284eb54d --- /dev/null +++ b/authbridge/lineage-attach/DESIGN.md @@ -0,0 +1,246 @@ +# Design — why the attachment works, and where it stops + +The [README](README.md) says what to do. This is why it is built this way, what +it depends on, and the cases that look fine and are not. Nothing here is needed +to attach; all of it is needed to trust the result. + +## Two halves, one insight + +Every HTTP exchange a workload takes part in crosses its pod's network +boundary, and a sidecar at that boundary sees all of it: the request as it +arrives or leaves, the response as it ends, the protocol, the peer, the +outcome. Those are facts, and they can be recorded without the application +knowing. That is the first half — **capture** — and it is pure configuration: +a plugin entry in the sidecar's pipeline. + +What the boundary cannot see is *causality inside the pod*: which inbound +request made the app issue which outbound call. The sidecar lives outside the +app's execution context and has no way to know which coroutine did what. To +attribute an outbound hop to the inbound that caused it, something must carry +a token **with the execution scope through the app** — which is exactly what +the W3C `traceparent` header is for, and only code running *inside* the +request's context can copy it from the inbound request onto the outbound +ones. That is the second half — **propagation** — and it is the only thing the +app must do. + +The shim does it for the app: stock OpenTelemetry auto-instrumentation, +exporting nothing, baked as an inert layer. Nothing about what the app *does* +enters into it — the same layer serves an agent, a tool, a relay — and its +activation is one environment variable. Everything else the sidecar records +from the outside. + +## How a hop finds its parent + +Each sidecar stamps the id of the span it just emitted into the request's +`tracestate` header before forwarding; the next sidecar parents on that stamp +(`lineage.parent.source=tracestate`) and re-stamps its own. When no stamp is +present the hop is parented on the `traceparent` the wire carried and records +`lineage.parent.source=wire`; when the wire carried nothing at all it roots a +trace of its own and records `none`. The forwarded `traceparent` itself is +never modified — when none arrived the sidecar forwards one naming its own +request span, so the next hop has a context to extract and the stamp has a +header to ride on — and nothing guesses: a hop without context is *visibly* +unparented rather than attached to a plausible caller. + +So there is exactly one unstamped hop (`wire`, or `none` for a caller that +sent no `traceparent`) per well-propagated trace, at the entry, where +the caller supplied the first `traceparent`. Every other unstamped hop marks a +pod that did not carry the context through — `none` when its app sent no +`traceparent` at all, `wire` when it forwarded one without the stamp. The +subtree beneath it fragments out of its caller's +trace — *visibly absent* lineage rather than wrong lineage — and attribution is +lost for everything downstream of that pod. + +> In-process `traceparent` propagation is mandatory, and the sidecar cannot do +> it from outside the app. + +## Why the shim is needed, and what it installs + +The shim supplies propagation with stock OpenTelemetry auto-instrumentation and +**exports nothing**: + +- **server side** (`starlette` / `asgi` / `fastapi`) — extract the inbound + `traceparent`, make it the active context. +- **client side** (`httpx` / `requests` / `aiohttp-client` / `urllib3`) — inject + `traceparent` on outbound calls. `urllib3` is what covers `boto3`/`botocore`: + an S3 read from a tool otherwise escapes into a trace of its own (seen live). +- **`threading`** — carry the active context across `Thread.start` / + `ThreadPoolExecutor.submit`. **Load-bearing**: frameworks that run the LLM + call in a worker thread (anything using `loop.run_in_executor`) otherwise + lose the context at the thread boundary and those legs silently fragment. +- **exporters pinned to `none`, and no `OTEL_EXPORTER_OTLP_ENDPOINT`** — the + activation hook sets all three signal exporters to `none` as environment + defaults the moment it wakes, so the shim's instrumentation generates spans + that go nowhere. Your telemetry backend sees only what the sidecar emits. + +### The activation hook + +Activation is a baked, env-gated site hook (`lineage-propagate-hook.py`, +installed into site-packages as a `.pth` + module pair): every Python process +of the image checks one variable at startup and either initializes stock +auto-instrumentation (`LINEAGE_PROPAGATE=1`) or does nothing at all. The +image's ENTRYPOINT/CMD is never rewritten, there is no launcher process, and +an unactivated `-otel` image runs the app exactly as its base does, with no +OpenTelemetry module loaded — +`build-otel-shim.sh` attests both halves of that claim on every bake, before +the image is ever loaded. It is the same attach shape as the Java agent +(`JAVA_TOOL_OPTIONS`) and Node (`NODE_OPTIONS`): the switch lives in the +Deployment, next to the image reference, and nowhere in the app. + +A hook failure cannot take the app down: the module catches everything and +logs; propagation is then off and the trace shows it (`none` on the pod's +outbound hops), which is the +honest failure mode. Read the hook's docstring for the full contract. + +### An app that already configures OpenTelemetry itself + +That app is outside the shim's envelope, and the bake interlock cannot see it +(it probes for instrumentors, not for SDK use). The hook's `initialize()` +installs the global `TracerProvider` first, so a provider the app sets in code +is rejected (`Overriding of current TracerProvider is not allowed`) and the +app's own exporter goes silent. An `OTEL_*_EXPORTER` the app sets in its +environment wins over the hook's `none` (`setdefault`), so the shim's own +server and client spans then flow to the app's backend — and `otlp` needs an +exporter package the shim does not install, in which case the hook fails at +startup and propagation stays off. For such an app attach capture only (no +`APP_CONTAINER`) and let its own instrumentation carry `traceparent`. + +## The envelope + +The shim is generic across the mainstream Python stack, not universal: + +- **Python**, in any of the common layouts — a venv (declared via + `$VIRTUAL_ENV` or at the usual paths) or a plain pip/system python; + `build-otel-shim.sh` probes the image for the interpreter the app runs and + refuses when it finds none (an explicit argument overrides the probe). +- Server is **ASGI / Starlette / FastAPI**; HTTP client is **httpx**, + **requests**, **aiohttp** or **urllib3** (which is how `boto3` talks). +- A caller that sends no `traceparent` gets one minted by the entry sidecar + (`parent.source=none` on that hop, contract v1.6); the shim itself seeds + nothing. +- **Not covered:** work handed to a `multiprocessing` child or a `subprocess`, + threads started at app boot rather than inside a request, and an app that + configures the OpenTelemetry SDK itself — in code or through `OTEL_*` + environment variables (above). +- The base image has a shell and coreutils (`Dockerfile.otel-shim` runs one + `RUN` step to place the hook); a distroless base fails that step. + +Outside the envelope, attach capture only (no `APP_CONTAINER`): every hop is +still recorded. Whether those hops are *correctly attributed* depends entirely +on the app carrying the trace context (`traceparent`, and the stamp in +`tracestate`) from its inbound to its outbound calls itself — and that is a +stronger condition than it sounds. + +### Half-instrumented apps: the case that looks fine and is not + +Propagation needs **two** halves: something that extracts the inbound +`traceparent` into an active context (`starlette` / `asgi` / `fastapi`), and +something that injects it on the way out (`httpx` / `requests` / `aiohttp` / +`urllib3`). An app carrying only the client half can inject, but has nothing +to inject *from*. + +That app is in a corner this kit cannot get you out of: + +- **The shim refuses it**, correctly — its client instrumentor is one this shim + installs, so wrapping would stack a second one on the same library. +- **Capture-only does not save it** — with no server-side extraction, every + outbound call starts a *fresh* trace. + +Measured on exactly such an app: it served requests, called its LLM +successfully, and produced 26 outbound exchanges — **every one of them alone +in its own trace, none sharing a trace with the inbound that caused it.** No +correlation survives at all, and every per-trace consistency check stays green +while it happens. + +**Check the shape, not just the count.** A concurrency check that only asks "is +each trace internally consistent?" will pass this app perfectly: a trace holding +one lonely inbound is clean by every structural measure. What exposes it is +expecting a *number of hops* per trace and finding one. If your app should make +three outbound calls per request, assert that three appear in the same trace — +otherwise a total attribution failure reads as a clean run. The same test +caught the shim's own gap once: a tool reading S3 through `boto3` escaped into +its own trace on a run whose every functional check passed, because the +recipe lacked the `urllib3` instrumentor. One unstamped hop in the wrong place +is the whole signal. + +If you own the image, the fix is to add the missing server-side instrumentor +(or activate stock auto-instrumentation, which brings both halves — that is +what the shim's hook does). If you do not, the image reference is the only +lever left (`SELF_ACTIVATE=1`, README "The propagation half") — and only if +the shim's interlock lets the image through. + +### Baking is not purely additive + +The shim installs one pinned OpenTelemetry contrib release +(`OTEL_CONTRIB_VERSION` in `Dockerfile.otel-shim`), and the resolver brings +the SDK that release requires — on an image that already carries an older SDK, +the bake *upgrades* it (observed: `opentelemetry-sdk 1.42.1 → 1.44.0`, semconv +`0.63b1 → 0.65b0`). Harmless for an app that does not pin, but an app pinning +an older SDK can be affected by being wrapped. Check the build output if your +app is sensitive to those versions. + +### Refuse rather than guess + +`build-otel-shim.sh` probes the image and stops if it finds no runnable Python, +or if any of the eight instrumentors the shim installs is already present +(wrapping would stack a second instrumentor on the same library), or if the +image already carries the shim's own hook. It does **not** refuse on the mere +presence of the `opentelemetry.instrumentation` namespace or of a dormant SDK +— an image can carry those as transitive dependencies and still need the shim. +`FORCE_BAKE=1` overrides, which makes the human judgment explicit and +greppable. + +## What the sidecar can and cannot see + +- **Plaintext HTTP only.** An HTTPS destination is TLS passthrough — no hop is + recorded for it. Inside a cluster that is usually every hop that matters; + an external HTTPS LLM is the notable exception, and it is invisible. +- **Non-HTTP protocols cannot pass through the outbound listener at all.** + `proxy-init` redirects every outbound TCP connection to the sidecar, whose + outbound listener hands each non-TLS connection to its HTTP codec. A + Postgres or SMTP connection is closed on arrival (seen live as `psycopg` + "server terminated abnormally"). `OUTBOUND_PORTS_EXCLUDE` keeps such ports + out of the redirect; those hops carry no HTTP facts and are invisible to + lineage either way. S3 is HTTP and stays in. The redirect is symmetric: + every inbound TCP port of the pod is sent to the inbound listener too, and + this kit exposes no inbound exclusion (`proxy-init` supports + `INBOUND_PORTS_EXCLUDE`; the generator does not offer it). An app that + *serves* a plaintext non-HTTP protocol cannot be adopted as is. +- **Content is captured through parsers, not off the wire.** With + `capture_io: true` the A2A, MCP and inference parsers attach the parsed + message as `input.value` / `output.value`; a plain HTTP exchange has no + parser and is recorded bodyless — who called whom, with what outcome, but + not what was said. Payloads are capped (`max_payload_bytes`, 4096 by + default) with a visible truncation marker. + +## Where you see the spans, on the rossoctl platform + +The platform's `rossoctl-deps` chart +([`charts/rossoctl-deps/values.yaml`](https://github.com/rossoctl/rossoctl/blob/main/charts/rossoctl-deps/values.yaml), +`otel.collector`) runs an OTel collector (`deploy/otel-collector` in +`rossoctl-system`) whose base `traces` pipeline exports to `debug` at +`verbosity: detailed` — span attributes included — and does **not** install +Phoenix (`components.phoenix.enabled` defaults to `false`). Spans arriving at +the default endpoint are printed to the collector's log and stored nowhere +queryable: + +```sh +kubectl logs -n rossoctl-system deploy/otel-collector | grep lineage.exchange.id +``` + +That is enough to prove the plugin works. For a store, add an exporter and a +pipeline to the collector's ConfigMap (Phoenix with +`--set components.phoenix.enabled=true`, Jaeger, any OTLP store of your own), or point +`OTEL_ENDPOINT` straight at a sink of your own. Nothing in this kit depends +on which you choose. + +## Limits, in one place + +- **The sidecar sees plaintext HTTP only.** +- **Trace-context propagation is the app's job** — the shim does it for the + in-envelope stack, and nothing else can do it from outside the process. +- **The patch path is not durable**: the owner still owns the Deployment, and + a platform-side rewrite (an operator reconcile, a chart upgrade, a UI + redeploy) silently drops the sidecar. Keep the attachment in the manifests + when you own them (README "Bring your own manifests"). +- **A workload whose image *and* env you cannot influence stays capture-only.** diff --git a/authbridge/lineage-attach/Dockerfile.otel-shim b/authbridge/lineage-attach/Dockerfile.otel-shim new file mode 100644 index 00000000..e021f630 --- /dev/null +++ b/authbridge/lineage-attach/Dockerfile.otel-shim @@ -0,0 +1,68 @@ +# Propagate-only OpenTelemetry shim for an uninstrumented Python app image. +# +# Auto-instrumentation is layered ON TOP of the app image so the app carries +# the W3C traceparent through itself: extract on the inbound ASGI/Starlette/ +# FastAPI request, inject on the outbound httpx/requests/aiohttp call. Nothing +# is exported (the hook pins every exporter to `none`); the sidecar does the +# capture. Neither the app's source nor its command is touched. +# +# Activation is environment, not command: a site-packages hook (`.pth` + +# `_lineage_propagate.py`, see lineage-propagate-hook.py) runs stock +# auto-instrumentation at interpreter start ONLY when LINEAGE_PROPAGATE=1 is +# set — the same attach shape as JAVA_TOOL_OPTIONS / NODE_OPTIONS. Without it +# the app runs exactly as on its base: no OpenTelemetry module loads. +# +# One recipe for every app: instrumentors target libraries, and only the ones +# the app imports activate. +# +# Build with build-otel-shim.sh (detects the build-args, refuses images it +# cannot safely wrap, attests the result, kind-loads it). Direct: +# podman build -f Dockerfile.otel-shim --build-arg BASE_IMAGE= -t -otel:latest . +# +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +# The python the app runs in, and the uid:gid it runs as — detected from the +# image by build-otel-shim.sh; the defaults match the common uv-built layout. +ARG VENV_PYTHON=/app/.venv/bin/python +ARG APP_UID=1001 +ARG APP_GID=0 + +USER root +# Our own static uv, pinned by digest: a builder-stage image may have neither +# uv nor pip in its runtime layer. +COPY --from=ghcr.io/astral-sh/uv:0.9.24@sha256:816fdce3387ed2142e37d2e56e1b1b97ccc1ea87731ba199dc8a25c04e4997c5 /uv /usr/local/bin/uv +# One contrib release for everything, so a bake never silently upgrades the +# OTel set under an app. -distro is the SDK; the eight others — server +# (starlette/asgi/fastapi), client (httpx/requests/aiohttp/urllib3) and +# -threading — are the instrumentors build-otel-shim.sh's interlock probes +# for; keep the two lists in sync. +# `-urllib3` covers clients that bypass `requests`: boto3/botocore (S3 and +# every AWS-style store) drive urllib3 directly. Without it an object read +# lands in a trace of its own (seen live: a MinIO GetObject escaping into a +# trace of its own). Inside a `requests` call urllib3 is suppressed, so +# the two never stack. +# `-threading` is load-bearing: OTel context is contextvars-based and does not +# cross run_in_executor / ThreadPoolExecutor on its own; without it the LLM or +# tool call made from a worker thread fragments out of its caller's trace. +# `--system --break-system-packages` lets the install land in a non-venv +# python too; both flags are inert when --python is a venv. +ARG OTEL_CONTRIB_VERSION=0.65b0 +RUN uv pip install --no-cache --python ${VENV_PYTHON} --system --break-system-packages \ + "opentelemetry-distro==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-starlette==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-asgi==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-fastapi==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-httpx==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-requests==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-aiohttp-client==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-urllib3==${OTEL_CONTRIB_VERSION}" \ + "opentelemetry-instrumentation-threading==${OTEL_CONTRIB_VERSION}" +# The hook, into the same environment: `site` imports every .pth at +# interpreter start; the module no-ops unless LINEAGE_PROPAGATE=1. +COPY lineage-propagate-hook.py /tmp/lineage-propagate-hook.py +RUN sp="$(${VENV_PYTHON} -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')" \ + && install -m 0644 /tmp/lineage-propagate-hook.py "${sp}/_lineage_propagate.py" \ + && printf 'import _lineage_propagate\n' > "${sp}/zz-lineage-propagate.pth" \ + && rm /tmp/lineage-propagate-hook.py +USER ${APP_UID}:${APP_GID} diff --git a/authbridge/lineage-attach/README.md b/authbridge/lineage-attach/README.md new file mode 100644 index 00000000..7ff3ec70 --- /dev/null +++ b/authbridge/lineage-attach/README.md @@ -0,0 +1,252 @@ +# Lineage — per-request data lineage from the AuthBridge sidecar + +Every HTTP exchange a workload takes part in crosses its pod's network +boundary. A sidecar at that boundary sees all of it — who called whom, over +which protocol, with what outcome, and — if you switch it on — what was said. +Attach the AuthBridge sidecar to a Deployment that is **already running**, +switch on the `lineage-telemetry` plugin, and each exchange becomes **two +OTLP spans**, request and response, sent to any OTLP consumer. The application +is not asked to do anything, and nothing about what it does enters into it. + +One thing the boundary cannot see is which inbound request caused which +outbound call. Only code running inside the request can carry that, and it +does so by forwarding two headers, `traceparent` and `tracestate`. So the app image gets one +inert layer — stock OpenTelemetry auto-instrumentation with every exporter +off — that wakes on a single environment variable and forwards the header. +The app's source, command and manifests stay as its owner wrote them. + +That is the whole attachment: a ConfigMap and a strategic-merge patch on the +cluster side, an image reference and one variable on the app side. It is +additive, it is reversible with one `rollout undo`, and it is the same for +every workload in the fleet — an agent, a tool, a relay, a service nobody +remembers writing. + +> **Until a release carries the plugin, build the sidecar yourself.** The +> published `ghcr.io/rossoctl/cortex/authbridge-envoy` image boots without +> `lineage-telemetry` and logs `unknown plugin`. The plugin is cortex #761; +> [RECIPE.md](RECIPE.md) step 1 builds and loads `authbridge-envoy` + `proxy-init` +> from a tree that carries it. + +**Start here:** [RECIPE.md](RECIPE.md) — six steps, expected output, back +out. **Why it works and where it stops:** [DESIGN.md](DESIGN.md). **See it +run:** the [lineage demo](../demos/lineage/README.md) on the Weather Agent pair. + +--- + +## What you get + +Per HTTP exchange, two spans. The request span is named +`{self_id} {protocol} {operation}`, the response span appends ` response`; the +pair is joined by `lineage.exchange.id` (the request span's own id): + +| attribute | what it records | +|---|---| +| `lineage.exchange.id` · `lineage.role` | pairs the two spans of one exchange; `request` / `response` | +| `lineage.direction` · `lineage.self.id` · `lineage.peer.host` | `inbound` / `outbound`; this workload's stable id; the other end | +| `lineage.protocol` | `a2a` / `mcp` / `inference` / `http` | +| `lineage.outcome`, `lineage.denied_by` | how the exchange ended | +| `lineage.principal.sub`, `lineage.principal.client` | the caller's identity when a validated token carried one (the generated pipeline runs no `jwt-validation`, so not with the generated ConfigMap) | +| `lineage.parent.source` | `tracestate` — parented on the previous sidecar's stamp; `wire` — on a `traceparent` that arrived without a stamp; `none` — nothing valid arrived, so this hop roots a trace and a `traceparent` is minted for the next | +| `input.value` / `output.value` | with `capture_io: true`: the parsed A2A message, MCP arguments or LLM prompt, cut at `max_payload_bytes` (4096 unless `MAX_PAYLOAD_BYTES` says otherwise) with a visible marker | + +The attributes are the plugin's: [`plugin-catalog.md`](../docs/plugin-catalog.md#lineage-telemetry) +lists its knobs, [`lineage-wire-contract.md`](../docs/lineage-wire-contract.md) the wire format. + +A well-propagated trace has exactly one unstamped hop, at the entry: `wire` +when the caller sent a `traceparent`, `none` when it sent nothing. Every other +unstamped hop marks a pod that did not carry the context through — `none` when +its app sent no `traceparent` at all, `wire` when it forwarded one without the +stamp — and the subtree beneath it is visibly its own trace rather than +silently misattributed. The spans say +what happened on the wire; whatever consumes them decides what it means. + +--- + +## How to attach + +The target is a Deployment that already exists. `attach-lineage.sh` generates +exactly two things — the per-app plugin ConfigMap (`EMIT=cm`) and a +strategic-merge patch with the sidecar pieces (`EMIT=patch`, the default) — +and there are two ways to consume them. + +### Adopt a live Deployment + +```sh +DEPLOY= [APP_CONTAINER= APP_IMAGE=docker.io/library/-otel:latest] ./sidecar-patch.sh +``` + +`sidecar-patch.sh` checks the preconditions (the Deployment exists, the +platform-rendered `envoy-config` ConfigMap is in the namespace, no container +already named `envoy-proxy`/`proxy-init`, no port collision, `APP_CONTAINER` +names a real container), applies the ConfigMap, +patches the Deployment, and waits for the rollout. The patch only *adds*: +lists merge by name, so everything the owner wrote stays as written. To back +out, `kubectl rollout undo deploy/ --to-revision=` with the revision +the script printed (the patch is one revision), then delete +`authbridge-lineage-config-`. + +Two limits. **The target must not already carry an AuthBridge sidecar** — a +platform-enrolled workload (an `AgentRuntime` CR) has an injected one, also +named `envoy-proxy`, so the script refuses rather than merge into it; see the +namespace route below. **A patch is not durable** — the owner still owns the +Deployment, and a platform-side rewrite (operator reconcile, chart upgrade, UI +redeploy) silently drops the sidecar. Re-run after such a change, or keep the +attachment in the manifests. + +### Bring your own manifests + +If you own the app's manifests, make lineage part of them instead — the +attachment then survives every re-deploy: + +```sh +NAME=my-agent EMIT=cm ./attach-lineage.sh > lineage-cm.yaml +NAME=my-agent EMIT=patch APP_CONTAINER=agent APP_IMAGE=docker.io/library/my-agent-otel:latest \ + ./attach-lineage.sh > lineage-patch.yaml +``` + +```yaml +# kustomization.yaml +resources: [deployment.yaml, service.yaml, lineage-cm.yaml] # yours untouched, plus the generated cm +patches: + - path: lineage-patch.yaml + target: { kind: Deployment, name: my-agent } +``` + +### The propagation half + +Both routes attach **capture**. For *attribution* — outbound hops landing in +the trace of the inbound that caused them — the app must forward +`traceparent`: its own instrumentation, or the shim: + +```sh +./build-otel-shim.sh :latest # -> -otel:latest, attested, kind-loaded +``` + +then hand the patch the container to switch on: `APP_CONTAINER=` adds +`LINEAGE_PROPAGATE=1` to that container's env (merged by name — nothing else +in the container changes) and `APP_IMAGE=…-otel:latest` points it at the baked +image. Without `APP_CONTAINER` the app container is not touched at all. + +The `-otel` image must be resolvable the way the base is: the patch swaps +`image` and leaves `imagePullPolicy` alone, so a kind-loaded image needs +`IfNotPresent` and a registry-pulled base needs the `-otel` tag pushed beside +it. A Deployment whose env you cannot touch at all has one lever left, the +image reference: `SELF_ACTIVATE=1 ./build-otel-shim.sh` bakes the switch in. + +### Enrolled workloads: the namespace-ConfigMap route + +When the platform injects its own AuthBridge sidecar (an `AgentRuntime` CR), +lineage is enabled for the whole namespace by adding the three parsers + +`lineage-telemetry` to both directions of the operator-rendered +`authbridge-runtime-config` ConfigMap. Leave `self_id` unset — each pod +resolves its identity from the operator-mounted credential. The propagation +half is unchanged, and a platform upgrade re-renders the ConfigMap; re-apply +after one. This route is described here, not exercised: nothing in this kit +generates that edit. + +--- + +## What it costs, per workload + +Deploying the app is your work and stays your work. Attaching lineage adds +two commands and no YAML: + +| step | command | per | done for you | +|---|---|---|---| +| bake | `./build-otel-shim.sh ` | image | interpreter and uid detection, the interlock, the attestation, the kind load | +| attach | `DEPLOY= APP_CONTAINER= APP_IMAGE=…-otel:latest ./sidecar-patch.sh` | Deployment | the ConfigMap, the patch, five preconditions, the rollout wait | + +Capture only is one command, the attach without `APP_CONTAINER`. A fleet is +two loops (RECIPE "A fleet"). Then read the *shape* of one trace: one root per +turn, unstamped only at the entry. + +--- + +## Prerequisites and configuration + +- A cluster with the platform installed and the platform-rendered + **`envoy-config` ConfigMap** in the target namespace (the sidecar mounts it). +- Sidecar images resolvable from the cluster: `SIDECAR_IMAGE` / + `PROXY_INIT_IMAGE`, defaulting to the published + `ghcr.io/rossoctl/cortex/{authbridge-envoy,proxy-init}:latest` — see the + caveat at the top. +- An **OTLP/gRPC endpoint**, `OTEL_ENDPOINT`, default + `otel-collector.rossoctl-system.svc.cluster.local:4317`. Nothing downstream + is assumed; DESIGN "Where you see the spans" covers the platform collector. +- **Know what leaves the pod.** Content capture is off by default, as in the + plugin. `CAPTURE_IO=true` makes the spans carry parsed content: LLM prompts, + MCP arguments, A2A messages, cut at 4,096 bytes (`MAX_PAYLOAD_BYTES`; + `-1` keeps them whole). That content is + PII-bearing. It travels to `OTEL_ENDPOINT` as plain gRPC unless the endpoint + starts with `https://`, and on the stock platform the collector prints every + attribute into its own pod log. + +The generated ConfigMap's plugin entry: + +```yaml +- name: lineage-telemetry + config: + otel_endpoint: "otel-collector.rossoctl-system.svc.cluster.local:4317" # host:port; https:// prefix turns on TLS + capture_io: false # the plugin's default; CAPTURE_IO=true attaches the parsed content — PII lives in it + self_id: "" # falls back to self_id_file (the operator-mounted credential) + # max_payload_bytes: 4096 — the plugin's default cap on a captured value (MAX_PAYLOAD_BYTES); bypass_paths / bypass_hosts as below +``` + +`bypass_paths` / `bypass_hosts` keep infrastructure noise out (agent-card +discovery, health probes, telemetry backends). `OUTBOUND_PORTS_EXCLUDE` keeps +a port out of the iptables redirect: an app's own telemetry export port, or a +**plaintext non-HTTP store** it talks to (Postgres 5432, SMTP 1025, Redis 6379 +— the outbound listener's HTTP codec would close them; DESIGN "What the +sidecar can and cannot see"). Never exclude LLM, tool, peer or S3 ports. +`NO_EMIT=1` keeps the sidecar as a pure proxy — a clean A/B baseline. +Script knobs: `NAME`/`DEPLOY`, `NAMESPACE`, `SELF_ID`, `OTEL_ENDPOINT`, +`CAPTURE_IO`, `MAX_PAYLOAD_BYTES`, `APP_CONTAINER`, `APP_IMAGE`, `OUTBOUND_PORTS_EXCLUDE`, `SIDECAR_IMAGE`, +`PROXY_INIT_IMAGE`, `NO_EMIT`, `EMIT`; each script's header documents its own. + +--- + +## Files + +Two moments, eight files. The bake happens once per image, on a laptop; the +attach once per Deployment, against the cluster. The only thing that crosses +between them is an image reference. + +``` +BAKE — once per app image ATTACH — once per Deployment + build-otel-shim.sh :latest sidecar-patch.sh DEPLOY= [APP_CONTAINER= APP_IMAGE=] + ├─ container-runtime.sh podman/docker, kind load ├─ checks Deployment · envoy-config · names · ports · container + ├─ Dockerfile.otel-shim + lineage-propagate-hook.py ├─ attach-lineage.sh EMIT=cm → kubectl apply + └─ -otel:latest inert until LINEAGE_PROPAGATE=1 ├─ attach-lineage.sh EMIT=patch → kubectl patch + └─ kubectl rollout status +``` + +| file | what it is | +|---|---| +| `RECIPE.md` · `DESIGN.md` | the step-by-step; the reasoning, envelope and limits | +| `attach-lineage.sh` | **the one generator** — every YAML byte of the attachment, `EMIT=patch` / `EMIT=cm`, env-driven, stdout only, every input validated or refused | +| `sidecar-patch.sh` | the live applier: preconditions, then ConfigMap + patch + rollout wait; owns no YAML | +| `Dockerfile.otel-shim` | the propagate-only layer, one recipe for every in-envelope app, instrumentors pinned to one contrib release | +| `build-otel-shim.sh` | bakes, attests (gate off: nothing OTel-shaped loads; gate on: a `traceparent` is injected), kind-loads; refuses images it cannot safely wrap | +| `lineage-propagate-hook.py` | the env-gated site hook the Dockerfile installs (`.pth` + module); read its docstring for the contract | +| `container-runtime.sh` | sourced helper: docker vs podman, kind load either way | + +--- + +## Troubleshooting + +| symptom | cause | +|---|---| +| No spans at all | Wrong `OTEL_ENDPOINT`, or the sidecar image predates the plugin — read the `envoy-proxy` container's log. | +| `envoy-proxy` restarts with `unknown plugin "lineage-telemetry"` | The published image, until a release carries the plugin. Build from this repo (RECIPE step 1); `rollout undo` meanwhile. The patch pulls `IfNotPresent`, so a node that cached an older `:latest` keeps it. | +| Only inbound hops, never outbound | `proxy-init` did not install its iptables rules — its log. | +| Outbound hops fragment (`lineage.parent.source=none` on the pod's outbound hops) | `traceparent` not propagating: the app container lacks `LINEAGE_PROPAGATE=1` (the patch sets it with `APP_CONTAINER`; an operator-owned Deployment needs `SELF_ACTIVATE=1`), or the call runs in a worker thread (the `threading` instrumentor is bundled), or the client library is outside the envelope. Only the entry hop dangling is expected. | +| The app cannot reach its database / mail server after the patch | A plaintext non-HTTP port went through the outbound HTTP codec — `OUTBOUND_PORTS_EXCLUDE` it. | +| A non-HTTP port the app *serves* stops answering after the patch | Inbound is redirected too, and there is no inbound exclusion knob; the app cannot be adopted as is (DESIGN "What the sidecar can and cannot see"). `rollout undo`. | +| Nothing captured when testing | `kubectl port-forward` reaches the app on loopback and bypasses the sidecar. Drive from inside the cluster. | +| `kind load` fails under podman | `container-runtime.sh` saves + loads an archive for podman v5; `CONTAINER_TOOL` forces a runtime, `KIND_CLUSTER_NAME` the cluster. | +| Sidecar `ImagePullBackOff` | `SIDECAR_IMAGE` / `PROXY_INIT_IMAGE` unresolvable from the cluster. | +| App container `ErrImagePull` after the patch | `APP_IMAGE` unresolvable under the container's own `imagePullPolicy` ("The propagation half"). `rollout undo`. | +| Pod stuck `ContainerCreating`, `configmap "envoy-config" not found` | Not a platform-set-up namespace (`sidecar-patch.sh` checks; the manifests route cannot). | +| `sidecar-patch.sh` refuses: "already has a container named …" | The target carries an `envoy-proxy` container or a `proxy-init` init container — enrolled workload, another mesh, or an earlier attach. Namespace route for the first; `rollout undo` the last. | +| `sidecar-patch.sh` refuses: "already declares containerPort …" | An undeclared sidecar, or the app itself listens on 9090/15123/15124; the latter cannot be adopted. | +| `sidecar-patch.sh` refuses: "has no container named …" | `APP_CONTAINER` matches nothing; a strategic merge would otherwise *add* a stub container by that name. | diff --git a/authbridge/lineage-attach/RECIPE.md b/authbridge/lineage-attach/RECIPE.md new file mode 100644 index 00000000..639ed319 --- /dev/null +++ b/authbridge/lineage-attach/RECIPE.md @@ -0,0 +1,118 @@ +# Recipe — attach lineage to a running Deployment + +A step-by-step for an operator or a coding agent. Every step is one command, +the one line of output that means it worked, and what to do when it did not. +Run from this directory. The explanations are in [README.md](README.md); the +reasons behind them in [DESIGN.md](DESIGN.md). + +**Inputs.** `NS` — the namespace · `DEPLOY` — the Deployment · `CONTAINER` — +the name of its app container (`kubectl -n $NS get deploy/$DEPLOY -o jsonpath='{.spec.template.spec.containers[*].name}'`) +· `IMAGE` — that container's image, present locally under the name the engine knows it by (a podman build of +`my-agent:latest` is `localhost/my-agent:latest`; a bare name is asked of the engine first, then taken as `docker.io/library/`). + +## 0. Preconditions (read-only) + +| check | command | pass | +|---|---|---| +| the Deployment exists and is not platform-enrolled | `kubectl -n $NS get deploy $DEPLOY -o jsonpath='{.spec.template.spec.initContainers[*].name} {.spec.template.spec.containers[*].name}'` | prints the app container(s) only — no `proxy-init`, no `envoy-proxy` | +| the platform rendered the sidecar's config here | `kubectl -n $NS get cm envoy-config` | found | +| the app's image is local (for the bake) | `podman image exists $IMAGE` (docker: `docker image inspect $IMAGE >/dev/null`) | exit 0 | +| an OTLP/gRPC collector is reachable in-cluster | `kubectl -n rossoctl-system get deploy otel-collector` | found (else set `OTEL_ENDPOINT` in step 3) | + +Enrolled workloads (an `AgentRuntime` CR) already carry a sidecar: this recipe +refuses them by design; see README "Enrolled workloads". + +## 1. A sidecar image that carries the plugin (once per cluster, until a release does) + +The plugin is cortex #761: build from a tree that carries `authbridge/authlib/plugins/lineage/` +(`main` once #761 has merged; the #761 branch until then). + +```sh +( cd .. && podman build -f cmd/authbridge-envoy/Dockerfile -t docker.io/library/authbridge-envoy:latest . \ + && podman build -f proxy-init/Dockerfile.init -t docker.io/library/proxy-init:latest proxy-init/ ) +for ref in authbridge-envoy proxy-init; do podman save docker.io/library/$ref:latest -o /tmp/$ref.tar \ + && KIND_EXPERIMENTAL_PROVIDER=podman kind load image-archive /tmp/$ref.tar --name rossoctl; rm -f /tmp/$ref.tar; done +export SIDECAR_IMAGE=docker.io/library/authbridge-envoy:latest PROXY_INIT_IMAGE=docker.io/library/proxy-init:latest +``` + +Pass: `podman exec crictl images | grep -E 'library/(authbridge-envoy|proxy-init)'` lists both. +Docker hosts: `docker build` with the same `-f`/`-t`, then `kind load docker-image --name rossoctl`. +Skip this step once the published `ghcr.io/rossoctl/cortex/authbridge-envoy` carries `lineage-telemetry`; +the symptom of skipping it too early is the sidecar crash-looping with `unknown plugin "lineage-telemetry"`. + +## 2. Bake the propagation shim onto the app image (once per image) + +```sh +KIND_CLUSTER_NAME=rossoctl ./build-otel-shim.sh $IMAGE +``` + +Pass: last line `>> loaded docker.io/library/-otel:latest into kind cluster rossoctl` +(exit 0; the attestation runs before the load and prints nothing when it passes). +Fail `REFUSING to bake … already instruments …` (exit 3): the app instruments itself — go to step 3 **without** `APP_CONTAINER`/`APP_IMAGE` (capture only). +Fail `REFUSING to bake … no runnable Python found` (exit 3): outside the shim's envelope (DESIGN "The envelope") — same, capture only; or pass the interpreter as arg 3 if you know it. +Fail `REFUSING to bake … is not present locally` (exit 3): wrong `IMAGE` — see Inputs; nothing was built. +Fail `ATTESTATION FAILED` (exit 4): the bake itself is broken (the image was not loaded) — read the assertion it prints; not an app property. + +## 3. Attach (once per Deployment) + +```sh +NAMESPACE=$NS DEPLOY=$DEPLOY APP_CONTAINER=$CONTAINER APP_IMAGE=docker.io/library/-otel:latest ./sidecar-patch.sh +``` + +Add `OUTBOUND_PORTS_EXCLUDE=5432,1025` (comma-separated) if the app speaks a plaintext **non-HTTP** +protocol to a store — Postgres, SMTP, Redis. Never exclude LLM, tool, peer or S3 ports. +Set `OTEL_ENDPOINT=host:port` for a collector other than the platform's. + +Pass — the output ends with: +``` +>> back out: kubectl -n rollout undo deploy/ --to-revision= && kubectl -n delete cm authbridge-lineage-config- +deployment "" successfully rolled out +>> lineage sidecar attached to deploy/ (self_id=, ns=) +``` +(preceded by `configmap/authbridge-lineage-config- created` and `deployment.apps/ patched`; +the back-out line is printed before the rollout wait so it is there even when the wait fails). +Add `CAPTURE_IO=true` to attach the parsed content — prompts, tool arguments, messages — to the spans (off by default; PII). +Fail `already has a container named` / `already declares containerPort` → enrolled or colliding workload (README "How to attach"). +Fail `has no container named` → wrong `APP_CONTAINER`; nothing was applied. +Rollout stuck → run the back-out line printed above, then read the sidecar log (step 4). + +## 4. Verify + +```sh +kubectl -n $NS logs deploy/$DEPLOY -c envoy-proxy | grep 'lineage-telemetry: initialized' +``` +Pass: `… endpoint= self_id=`. + +Then one request **from inside the cluster** (a port-forward bypasses the sidecar) with a trace id you choose: +```sh +T=$(python3 -c 'import secrets;print(secrets.token_hex(16))') +kubectl -n $NS run drive --rm -i --restart=Never --image=curlimages/curl:8.11.1 -- \ + curl -s -H "traceparent: 00-$T-0000000000000001-01" http://:/ # any request the app answers +kubectl -n rossoctl-system logs deploy/otel-collector | grep -c "$T" +``` +Pass: a count ≥ 2 (one request span + one response span per exchange the app took part in). +Attribution check, when the app calls out: every hop after the entry must show +`lineage.parent.source: tracestate`; a `none` (or `wire`) on a non-entry hop is an un-propagated call +(DESIGN "Why the shim is needed"). + +## 5. Back out + +```sh +kubectl -n $NS rollout undo deploy/$DEPLOY --to-revision= && kubectl -n $NS delete cm authbridge-lineage-config-$DEPLOY +``` +`` is the revision step 3 printed in its last line — the spec as it was before the attach. A bare +`rollout undo` goes one step back, which is that spec only if nothing rolled the Deployment since; +`kubectl -n $NS rollout history deploy/$DEPLOY` lists the revisions if the line is gone. Delete the +ConfigMap after the undo, not before: a revision that still mounts it cannot start without it. + +## A fleet + +Bake once per image, attach once per Deployment; two loops. Then check the **shape** of one trace +(one root, unstamped only at the entry), not just that spans arrived. + +```sh +for img in agent-a agent-b tool-x; do KIND_CLUSTER_NAME=rossoctl ./build-otel-shim.sh docker.io/library/$img:latest; done +for d in agent-a agent-b tool-x; do + NAMESPACE=$NS DEPLOY=$d APP_CONTAINER=app APP_IMAGE=docker.io/library/$d-otel:latest ./sidecar-patch.sh +done +``` diff --git a/authbridge/lineage-attach/attach-lineage.sh b/authbridge/lineage-attach/attach-lineage.sh new file mode 100755 index 00000000..514235a3 --- /dev/null +++ b/authbridge/lineage-attach/attach-lineage.sh @@ -0,0 +1,352 @@ +#!/usr/bin/env bash +# attach-lineage.sh — the one generator. Given a Deployment's NAME (and, +# optionally, which of its containers to switch propagation on and with which +# image), print ONE of the two objects that attach lineage to it: +# +# EMIT=patch (default) strategic-merge patch adding the sidecar pieces — +# proxy-init initContainer, envoy-proxy container, two +# config volumes — and, with APP_CONTAINER, the +# propagation switch on the app's own container. +# EMIT=cm the per-app plugin ConfigMap the sidecar mounts +# (parser chain + lineage-telemetry entry). +# +# Apply both and the app's traffic flows through the lineage plugin. The app +# itself is never described: no Deployment, no Service, no app config — lists +# merge by name, so nothing the owner wrote changes. Stdout only; this script +# never touches the cluster. +# +# Consumed by sidecar-patch.sh (live: checks, applies both, waits) or by your +# own kustomization (the cm under resources:, the patch under patches: with +# target kind Deployment / name NAME) — see README.md "How to attach". +# +# Propagation: capture alone cannot attribute an app's outbound calls to the +# inbound that caused them. The sidecar forwards the trace context (a +# traceparent when none arrived, plus its own tracestate stamp), but only code +# running inside the request can carry it from the inbound to the outbound +# calls it causes — the app must do that itself. For an +# uninstrumented Python app, bake the shim first (build-otel-shim.sh), then +# pass APP_CONTAINER (+ APP_IMAGE): the patch sets LINEAGE_PROPAGATE=1 on that +# container, which wakes the baked hook. Without APP_CONTAINER the patch is +# capture-only and the app container is not touched. +# +# Usage: +# NAME=echo-upstream ./attach-lineage.sh # the patch +# NAME=echo-upstream EMIT=cm ./attach-lineage.sh # the ConfigMap +# NAME=my-agent APP_CONTAINER=agent \ +# APP_IMAGE=docker.io/library/my-agent-otel:latest ./attach-lineage.sh +# +# Variables: +# NAME (required) the target Deployment; also names the ConfigMap +# (authbridge-lineage-config-NAME) and defaults SELF_ID +# NAMESPACE default team1 +# SELF_ID lineage identity on every span (default NAME) +# APP_CONTAINER the app container to set LINEAGE_PROPAGATE=1 on. MUST name +# an existing container: a strategic merge ADDS a stub for an +# unknown name. Checked by sidecar-patch.sh, not here. +# APP_IMAGE needs APP_CONTAINER: the -otel image to set on it +# OTEL_ENDPOINT OTLP/gRPC target for the plugin's spans (default: the +# platform collector). Any OTLP consumer works. Plain gRPC +# unless it starts with https://. +# CAPTURE_IO false (default, the plugin's own) | true — whether the spans +# carry the parsed content (input.value / output.value: +# prompts, tool arguments, messages). PII-bearing: opt in. +# MAX_PAYLOAD_BYTES cap on a captured value (plugin default 4096 when unset; +# -1 = attach whole). Only meaningful with CAPTURE_IO=true. +# OUTBOUND_PORTS_EXCLUDE ports proxy-init must not intercept — an app's OWN +# telemetry export port (e.g. 4317), or a plaintext non-HTTP +# store (Postgres 5432, SMTP 1025: the outbound HTTP codec +# would close them). Never LLM/tool/S3 ports. +# SIDECAR_IMAGE default ghcr.io/rossoctl/cortex/authbridge-envoy:latest — +# UNTIL A RELEASE CARRIES lineage-telemetry (cortex #761) it +# boots without the plugin; build from a tree that has it and +# point this at your tag (RECIPE.md step 1) +# PROXY_INIT_IMAGE default ghcr.io/rossoctl/cortex/proxy-init:latest +# NO_EMIT=1 omit the plugin entry: the sidecar proxies, emits nothing +# (parsers alone are legal). The A/B baseline. +# EMIT patch (default) | cm +# +# Structure: parse_inputs validates EVERY knob (all refusals live there); +# build_* each assemble one optional YAML fragment into a global; the three +# fragment functions are the single source of the sidecar YAML; emit() +# dispatches to the two emitters. +set -euo pipefail + +# Every free-form value lands in a double-quoted YAML scalar, where only '"' +# and '\' are special — refusing those two is exactly sufficient. Whitespace +# and control characters are refused as well: never part of an endpoint or +# image ref, and a control character is illegal in YAML even inside quotes. +yaml_safe() { # $1 = what it is (for the error), $2 = the value + local unsafe=$'"\\' + case "$2" in + *["$unsafe"]*|*[[:space:][:cntrl:]]*) + printf "error: %s '%s' contains whitespace, a control character or one of %s, which this script cannot quote safely\n" "$1" "$2" "$unsafe" >&2 + exit 2 ;; + esac +} + +parse_inputs() { + # Knobs of the removed app-deployment mode (EMIT=manifest): a caller passing + # one is running a stale recipe — refuse loudly rather than ignore. + local stale + for stale in APP_PORT SVC_PORT ENV_VARS APP_COMMAND APP_RESOURCES \ + PVC_NAME PVC_MOUNT WORKLOAD_TYPE WORKLOAD_PROTOCOL LABEL_PREFIX \ + NO_PROPAGATE APP_ENTRYPOINT; do + if [ -n "${!stale:-}" ]; then + echo "error: $stale is not a knob — it was removed with the app-deployment (EMIT=manifest) mode." >&2 + echo " This script only attaches lineage to an EXISTING Deployment; the app itself" >&2 + echo " is deployed by its owner. See the header and README.md." >&2 + exit 2 + fi + done + + EMIT="${EMIT:-patch}" + NAME="${NAME:?set NAME}" + NAMESPACE="${NAMESPACE:-team1}" + # NAME (RFC 1123 subdomain) and NAMESPACE (DNS label) are interpolated bare. + # 227 = 253 minus the ConfigMap name's prefix (authbridge-lineage-config-). + if [ "${#NAME}" -gt 227 ] \ + || ! [[ "$NAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ ]]; then + echo "error: NAME='$NAME' must be a lowercase RFC 1123 subdomain of at most 227 chars (253 minus the ConfigMap prefix)" >&2; exit 2 + fi + if ! [[ "$NAMESPACE" =~ ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ ]]; then + echo "error: NAMESPACE='$NAMESPACE' is not a DNS label (lowercase alphanumerics and '-', max 63)" >&2; exit 2 + fi + case "$EMIT" in + patch|cm) ;; + *) echo "error: EMIT must be patch|cm (got '$EMIT')" >&2; exit 2 ;; + esac + SELF_ID="${SELF_ID:-$NAME}" + APP_CONTAINER="${APP_CONTAINER:-}" + APP_IMAGE="${APP_IMAGE:-}" + OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}" + OTEL_ENDPOINT="${OTEL_ENDPOINT:-otel-collector.rossoctl-system.svc.cluster.local:4317}" + CAPTURE_IO="${CAPTURE_IO:-false}" + case "$CAPTURE_IO" in + true|false) ;; + *) echo "error: CAPTURE_IO must be true|false (got '$CAPTURE_IO')" >&2; exit 2 ;; + esac + MAX_PAYLOAD_BYTES="${MAX_PAYLOAD_BYTES:-}" + [[ "$MAX_PAYLOAD_BYTES" =~ ^(-1|[1-9][0-9]*)?$ ]] \ + || { echo "error: MAX_PAYLOAD_BYTES must be a positive integer or -1 (got '$MAX_PAYLOAD_BYTES')" >&2; exit 2; } + # Published images by default; point at local tags when building from source. + SIDECAR_IMAGE="${SIDECAR_IMAGE:-ghcr.io/rossoctl/cortex/authbridge-envoy:latest}" + PROXY_INIT_IMAGE="${PROXY_INIT_IMAGE:-ghcr.io/rossoctl/cortex/proxy-init:latest}" + NO_EMIT="${NO_EMIT:-0}" + case "$NO_EMIT" in + 0|1) ;; + *) echo "error: NO_EMIT must be 0|1 (got '$NO_EMIT')" >&2; exit 2 ;; + esac + + local v + for v in SELF_ID OTEL_ENDPOINT APP_IMAGE SIDECAR_IMAGE PROXY_INIT_IMAGE; do + yaml_safe "$v" "${!v}" + done + # A container name is a DNS label; it is interpolated bare. + if [ -n "$APP_CONTAINER" ] && ! [[ "$APP_CONTAINER" =~ ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ ]]; then + echo "error: APP_CONTAINER='$APP_CONTAINER' is not a valid container name (DNS label)" >&2; exit 2 + fi + if [ -n "$APP_IMAGE" ] && [ -z "$APP_CONTAINER" ]; then + echo "error: APP_IMAGE needs APP_CONTAINER — the image lands on a container the patch must name" >&2; exit 2 + fi + # Ports exactly as proxy-init hands them to iptables: no leading zero (iptables + # reads `010` as octal and refuses `0080`), at most five digits (a longer + # number overflows the `-gt` test below), then the range. + [[ "$OUTBOUND_PORTS_EXCLUDE" =~ ^([1-9][0-9]{0,4}(,[1-9][0-9]{0,4})*)?$ ]] \ + || { echo "error: OUTBOUND_PORTS_EXCLUDE='$OUTBOUND_PORTS_EXCLUDE' is not a comma-separated list of ports (1-65535, no leading zeros)" >&2; exit 2; } + local port + for port in ${OUTBOUND_PORTS_EXCLUDE//,/ }; do + if [ "$port" -gt 65535 ]; then + echo "error: OUTBOUND_PORTS_EXCLUDE has '$port', which is not a port (1-65535)" >&2; exit 2 + fi + done +} + +build_proxy_env() { + # proxy-init OUTBOUND_PORTS_EXCLUDE env (only if set) + exclude_env="" + if [ -n "$OUTBOUND_PORTS_EXCLUDE" ]; then + exclude_env=" + - name: OUTBOUND_PORTS_EXCLUDE + value: \"${OUTBOUND_PORTS_EXCLUDE}\"" + fi +} + +build_plugin_entry() { + # The plugin entry, one variable so NO_EMIT is a single point. Empty → the + # parsers stay and the sidecar is a pure proxy that emits nothing. + lineage_plugin="" + if [ "$NO_EMIT" != "1" ]; then + lineage_plugin=' + - name: lineage-telemetry + config: + otel_endpoint: "'"${OTEL_ENDPOINT}"'" + capture_io: '"${CAPTURE_IO}"' + self_id: "'"${SELF_ID}"'"' + # The plugin's own default applies when unset; an explicit value is emitted. + [ -z "$MAX_PAYLOAD_BYTES" ] || lineage_plugin="${lineage_plugin} + max_payload_bytes: ${MAX_PAYLOAD_BYTES}" + fi +} + +build_app_patch() { + # The propagation switch. `containers` and `env` both merge by name, so this + # sets exactly LINEAGE_PROPAGATE (and the image, if given) on the owner's + # container and nothing else. One env var is enough: it wakes the baked hook, + # which sets the propagate-only posture itself as env defaults. + app_patch="" + if [ -n "$APP_CONTAINER" ]; then + app_patch=" + - name: ${APP_CONTAINER}" + if [ -n "$APP_IMAGE" ]; then + app_patch="${app_patch} + image: \"${APP_IMAGE}\"" + fi + app_patch="${app_patch} + env: + - { name: LINEAGE_PROPAGATE, value: \"1\" }" + fi +} + +# ---- the sidecar fragments (the single source of every sidecar YAML byte) ---- + +sidecar_container() { # the envoy-proxy container (8-space list-item indent) + cat < mcp > inference). + pipeline: + inbound: + plugins: + - name: a2a-parser + - name: mcp-parser + - name: inference-parser${lineage_plugin} + outbound: + plugins: + - name: a2a-parser + - name: mcp-parser + - name: inference-parser${lineage_plugin} +EOF +} + +emit_patch() { # strategic merge: lists merge by name — the owner's spec is untouched + # apiVersion/kind/metadata make it a complete resource, which kustomize + # requires of a patch file; kubectl patch merges them harmlessly. + cat <&2; exit 2; } + parse_inputs # every knob: read, default, validate — all refusals live here + build_proxy_env # optional OUTBOUND_PORTS_EXCLUDE env for proxy-init + build_plugin_entry # the lineage-telemetry pipeline entry (empty under NO_EMIT=1) + build_app_patch # optional propagation switch on the app's own container + emit # dispatch: patch | cm +} +main "$@" diff --git a/authbridge/lineage-attach/build-otel-shim.sh b/authbridge/lineage-attach/build-otel-shim.sh new file mode 100755 index 00000000..8df8e64c --- /dev/null +++ b/authbridge/lineage-attach/build-otel-shim.sh @@ -0,0 +1,251 @@ +#!/usr/bin/env bash +# build-otel-shim.sh — bake the propagate-only OTel shim onto a Python app +# image and load it into kind. Only the base image changes per app: the +# interpreter and uid:gid are DETECTED from the image, and the app's command is +# never touched — the shim activates through the environment +# (LINEAGE_PROPAGATE=1, see Dockerfile.otel-shim). +# +# Usage: +# ./build-otel-shim.sh [wrapper-tag] [venv-python] [app-uid[:gid]] +# venv-python / app-uid are escape hatches for when detection picks wrong; +# an explicit value is used as-is. Give app-uid as uid:gid — a bare uid +# still takes the gid the base image's own user resolves to. +# +# Env: +# FORCE_BAKE=1 override the refuse-to-bake interlock +# SELF_ACTIVATE=1 bake LINEAGE_PROPAGATE=1 INTO the image — for workloads +# whose Deployment you cannot edit. Default images stay +# inert and are activated by the Deployment env. +# NO_KIND_LOAD=1 build + attest only (CI / offline) +# +# Every probe runs the (unaudited) app image with --network=none. +# container-runtime.sh picks podman or docker (override: CONTAINER_TOOL). +# +# Structure: main() at the bottom is the pipeline; each phase is a function +# and a failed phase exits (3 = image refused, 4 = attestation failed). +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +. "${SCRIPT_DIR}/container-runtime.sh" + +parse_args() { + BASE_IMAGE="${1:?usage: build-otel-shim.sh [wrapper-tag] [venv-python] [app-uid[:gid]]}" + # default wrapper tag: -otel:latest (tag or digest stripped) + local base_short base_name + base_short="${BASE_IMAGE##*/}"; base_name="${base_short%%[:@]*}" + WRAPPER_TAG="${2:-${base_name}-otel:latest}" + case "${WRAPPER_TAG##*/}" in *:*) ;; *) WRAPPER_TAG="${WRAPPER_TAG}:latest" ;; esac + VENV_PYTHON="${3:-}" + APP_UID="${4:-}" + APP_GID="${APP_UID#*:}"; [ "$APP_GID" = "$APP_UID" ] && APP_GID="" + APP_UID="${APP_UID%%:*}" + FORCE_BAKE="${FORCE_BAKE:-0}" + SELF_ACTIVATE="${SELF_ACTIVATE:-0}" + NO_KIND_LOAD="${NO_KIND_LOAD:-0}" + + # A registry-qualified ref is used as is. A bare name is asked of the engine + # first — podman keeps a local build as localhost/, docker as + # docker.io/library/ — and only a name the engine does not have is + # taken to mean docker.io/library/. + case "$BASE_IMAGE" in + */*) base_ref="$BASE_IMAGE" ;; + *) if "$CONTAINER_TOOL" image inspect "$BASE_IMAGE" >/dev/null 2>&1; then + base_ref="$BASE_IMAGE" + else + base_ref="docker.io/library/${BASE_IMAGE}" + fi ;; + esac + # Every probe below inspects or runs the image; say so once instead of + # surfacing the engine's own "no such object". + if ! "$CONTAINER_TOOL" image inspect "$base_ref" >/dev/null 2>&1; then + echo "REFUSING to bake ${BASE_IMAGE}: ${base_ref} is not present locally (${CONTAINER_TOOL})." >&2 + echo " Pull or build it first, and give the ref the engine knows it by" >&2 + echo " (a podman build of my-agent:latest is localhost/my-agent:latest)." >&2 + exit 3 + fi +} + +# The two per-image build inputs are IN the image — probe it, never transcribe. +runs_python() { # $1 = candidate interpreter path/name + "$CONTAINER_TOOL" run --rm --network=none --entrypoint "$1" "$base_ref" -c 'import sys' >/dev/null 2>&1 +} + +detect_python() { # sets VENV_PYTHON (validates it when given explicitly) + if [ -z "$VENV_PYTHON" ]; then + # The env the image declares, then the common venv layouts, then PATH. + local candidates virtual_env c + candidates=() + virtual_env="$("$CONTAINER_TOOL" inspect --format '{{range .Config.Env}}{{println .}}{{end}}' "$base_ref" \ + | sed -n 's/^VIRTUAL_ENV=//p' | head -1)" + [ -n "$virtual_env" ] && candidates+=("${virtual_env}/bin/python") + candidates+=(/app/.venv/bin/python /opt/venv/bin/python python3) + for c in "${candidates[@]}"; do + if runs_python "$c"; then VENV_PYTHON="$c"; break; fi + done + if [ -z "$VENV_PYTHON" ]; then + echo "REFUSING to bake ${base_ref}: no runnable Python found." >&2 + echo " Probed: ${candidates[*]}" >&2 + echo " The image is outside the shim envelope (non-Python, or an unusual layout)." >&2 + echo " -> pass the interpreter explicitly as arg 3, or attach the sidecar only:" >&2 + echo " DEPLOY= ./sidecar-patch.sh (still captures every HTTP hop," >&2 + echo " but pairing under concurrency needs the app to propagate on its own)" >&2 + exit 3 + fi + echo ">> detected app python: ${VENV_PYTHON}" + elif ! runs_python "$VENV_PYTHON"; then + echo "REFUSING to bake ${base_ref}: no runnable Python at ${VENV_PYTHON} (explicit arg)." >&2 + exit 3 + fi +} + +detect_user() { # sets APP_UID and APP_GID (either may be given explicitly) + local config_user_full="" + if [ -z "$APP_UID" ]; then + # Exactly the user the base declared (root when empty); a named user is + # resolved to its uid inside the image. + local config_user + config_user_full="$("$CONTAINER_TOOL" inspect --format '{{.Config.User}}' "$base_ref")" + config_user="${config_user_full%%:*}" + case "$config_user" in + "") APP_UID=0 ;; + *[!0-9]*) APP_UID="$("$CONTAINER_TOOL" run --rm --network=none --entrypoint id "$base_ref" -u 2>/dev/null)" || { + echo "REFUSING to bake ${base_ref}: cannot resolve user '${config_user}' to a uid" >&2 + echo " (no 'id' binary in the image?) -> pass the uid explicitly as arg 4." >&2 + exit 3 + } ;; + *) APP_UID="$config_user" ;; + esac + echo ">> detected app user: uid=${APP_UID} (Config.User='${config_user_full:-}')" + fi + if [ -z "$APP_GID" ]; then + # An explicit `USER uid:gid`, else the primary gid the runtime gives that + # user (0 without a passwd entry — which is what the base ran as). + case "$config_user_full" in + *:*) APP_GID="${config_user_full#*:}" ;; + *) APP_GID="$("$CONTAINER_TOOL" run --rm --network=none --entrypoint id "$base_ref" -g 2>/dev/null)" || APP_GID=0 ;; + esac + echo ">> detected app group: gid=${APP_GID}" + fi +} + +# The interlock asks exactly "would wrapping DOUBLE-instrument?": is any of the +# eight instrumentors this shim installs already present? Not the +# `opentelemetry.instrumentation` namespace (a transitive dep of anything +# OTel-adjacent, no library instrumentation in it) and not a dormant SDK +# (a2a-sdk ships one on every stock agent) — neither is a refusal signal. An +# app that activates its SDK in code is not statically detectable. +refuse_already_instrumented() { + [ "$FORCE_BAKE" = "1" ] && return 0 + # keep in sync with Dockerfile.otel-shim's install RUN (all but -distro) + local already + if already=$("$CONTAINER_TOOL" run --rm --network=none --entrypoint "$VENV_PYTHON" "$base_ref" -c ' +import importlib.util as u +mods = ["starlette", "asgi", "fastapi", "httpx", "requests", "aiohttp_client", "urllib3", "threading"] +found = [m for m in mods if u.find_spec("opentelemetry.instrumentation." + m)] +print(",".join(found)) +raise SystemExit(0 if found else 1)' 2>/dev/null); then + echo "REFUSING to bake ${base_ref}: it already instruments ${already}" >&2 + echo " Those are instrumentors this shim installs, so wrapping would stack a" >&2 + echo " second one on the same library (an -otel image, or an app that bundles" >&2 + echo " its own instrumentation)." >&2 + echo " -> if this is a stock app image, point me at the un-shimmed base." >&2 + echo " -> FORCE_BAKE=1 overrides if you know the wrap is safe." >&2 + exit 3 + fi + # An already-baked -otel image carries the hook; refuse to bake it twice. + if "$CONTAINER_TOOL" run --rm --network=none --entrypoint "$VENV_PYTHON" "$base_ref" -c ' +import importlib.util as u +raise SystemExit(0 if u.find_spec("_lineage_propagate") else 1)' >/dev/null 2>&1; then + echo "REFUSING to bake ${base_ref}: it already carries the lineage propagate hook" >&2 + echo " (an already-baked -otel image). -> point me at the un-shimmed base." >&2 + echo " -> FORCE_BAKE=1 overrides if you know the wrap is safe." >&2 + exit 3 + fi +} + +build_image() { + echo ">> building shim ${WRAPPER_TAG} FROM ${base_ref} (${CONTAINER_TOOL}, python=${VENV_PYTHON}, uid=${APP_UID})" + "$CONTAINER_TOOL" build -f "${SCRIPT_DIR}/Dockerfile.otel-shim" \ + --build-arg "BASE_IMAGE=${base_ref}" \ + --build-arg "VENV_PYTHON=${VENV_PYTHON}" \ + --build-arg "APP_UID=${APP_UID}" \ + --build-arg "APP_GID=${APP_GID}" \ + -t "${WRAPPER_TAG}" "${SCRIPT_DIR}" +} + +# Attestation: the bake proves itself before anything is loaded. Gate off, not +# one opentelemetry module may load; gate on, the hook ran, the exporter +# selection is explicit, and the propagator injects a traceparent. +verify_inert() { + if ! "$CONTAINER_TOOL" run --rm --network=none --entrypoint "$VENV_PYTHON" "$WRAPPER_TAG" -c ' +import sys +loaded = sorted(m for m in sys.modules if m.startswith("opentelemetry")) +raise SystemExit("gate off, yet otel loaded: %s" % loaded if loaded else 0)'; then + echo "ATTESTATION FAILED for ${WRAPPER_TAG}: image is not inert with the gate off." >&2 + exit 4 + fi +} + +verify_propagates() { + if ! "$CONTAINER_TOOL" run --rm --network=none -e LINEAGE_PROPAGATE=1 --entrypoint "$VENV_PYTHON" "$WRAPPER_TAG" -c ' +import os, sys +assert "opentelemetry.instrumentation.auto_instrumentation" in sys.modules, "hook did not run" +assert os.environ.get("OTEL_TRACES_EXPORTER") is not None, "exporter selection not pinned" +from opentelemetry import trace +from opentelemetry.propagate import inject +with trace.get_tracer("attest").start_as_current_span("attest"): + carrier = {} + inject(carrier) +assert "traceparent" in carrier, "propagator injects nothing: %r" % carrier'; then + echo "ATTESTATION FAILED for ${WRAPPER_TAG}: gate on, but the hook did not come up." >&2 + exit 4 + fi +} + +self_activate() { # optional: bake the activation in + [ "$SELF_ACTIVATE" = "1" ] || return 0 + echo ">> baking LINEAGE_PROPAGATE=1 into ${WRAPPER_TAG} (SELF_ACTIVATE=1)" + printf 'FROM %s\nENV LINEAGE_PROPAGATE=1\n' "$WRAPPER_TAG" \ + | "$CONTAINER_TOOL" build -t "$WRAPPER_TAG" - +} + +publish() { + # A bare tag is aliased under docker.io/library/ so kind resolves it; + # a registry-qualified one is used as is. + local alias_ref + case "$WRAPPER_TAG" in + */*) alias_ref="$WRAPPER_TAG" ;; + *) alias_ref="docker.io/library/${WRAPPER_TAG}" + "$CONTAINER_TOOL" tag "${WRAPPER_TAG}" "${alias_ref}" ;; + esac + + if [ "$NO_KIND_LOAD" = "1" ]; then + echo ">> built + attested ${alias_ref} (kind load skipped: NO_KIND_LOAD=1)" + else + kind_load "${alias_ref}" + echo ">> loaded ${alias_ref} into kind cluster ${KIND_CLUSTER_NAME}" + fi + if [ "$SELF_ACTIVATE" = "1" ]; then + echo ">> NOTE: this image is SELF-ACTIVATING (LINEAGE_PROPAGATE=1 baked in) — any" + echo ">> deployment of it propagates. Meant for workloads whose Deployment you" + echo ">> cannot edit; everywhere else prefer the inert default." + else + echo ">> NOTE: the -otel image is INERT — it runs exactly like its base until a" + echo ">> Deployment sets LINEAGE_PROPAGATE=1 in the app container's env" + echo ">> (attach-lineage.sh does this). Deploy it without that env and you" + echo ">> simply get the base image's behavior: no propagation, and the trace" + echo ">> fragments at this pod, visibly (lineage.parent.source=none on its outbound hops)." + fi +} + +main() { + parse_args "$@" # args + env → globals + detect_python # the app's interpreter, probed (or arg 3) + detect_user # uid:gid, probed (or arg 4) + refuse_already_instrumented # would double-instrument, or already baked → exit 3 + build_image # Dockerfile.otel-shim with the detected build-args + verify_inert # gate off: nothing OTel loads → else exit 4 + verify_propagates # gate on: hook up, traceparent injected → else exit 4 + self_activate # SELF_ACTIVATE=1 only + publish # alias, kind-load, print the NOTE +} +main "$@" diff --git a/authbridge/lineage-attach/container-runtime.sh b/authbridge/lineage-attach/container-runtime.sh new file mode 100644 index 00000000..0e2c15b9 --- /dev/null +++ b/authbridge/lineage-attach/container-runtime.sh @@ -0,0 +1,38 @@ +# shellcheck shell=bash +# container-runtime.sh — sourced by build-otel-shim.sh: picks the container +# engine (CONTAINER_TOOL) and loads an image into kind the way that engine +# needs (kind_load). Everything else uses $CONTAINER_TOOL directly. +# KIND_CLUSTER_NAME: target cluster (default rossoctl). + +# Podman is checked FIRST: on podman hosts `docker` is often a compat client, +# and `kind load docker-image` through it is exactly the breakage to avoid. +container_tool() { + [ -n "${CONTAINER_TOOL:-}" ] && return 0 + if command -v podman >/dev/null 2>&1; then CONTAINER_TOOL=podman + elif command -v docker >/dev/null 2>&1; then CONTAINER_TOOL=docker + else + echo "error: neither docker nor podman on PATH (set CONTAINER_TOOL)" >&2 + return 1 + fi +} + +# `kind load docker-image` misbehaves under podman v5; save + image-archive. +kind_load_podman() { + local ref="$1" tar rc=0 + tar="$(mktemp "${TMPDIR:-/tmp}/kind-load.XXXXXX")" + podman save -o "$tar" "$ref" \ + && KIND_EXPERIMENTAL_PROVIDER=podman kind load image-archive "$tar" --name "$KIND_CLUSTER_NAME" \ + || rc=$? + rm -f "$tar" # on success and failure alike + return "$rc" +} + +kind_load_docker() { + kind load docker-image "$1" --name "$KIND_CLUSTER_NAME" +} + +kind_load() { "kind_load_${CONTAINER_TOOL}" "$@"; } + +# `return`, not `exit`: sourced — the caller's `set -e` handles it. +container_tool || return 1 +KIND_CLUSTER_NAME="${KIND_CLUSTER_NAME:-rossoctl}" diff --git a/authbridge/lineage-attach/lineage-propagate-hook.py b/authbridge/lineage-attach/lineage-propagate-hook.py new file mode 100644 index 00000000..194ee0ee --- /dev/null +++ b/authbridge/lineage-attach/lineage-propagate-hook.py @@ -0,0 +1,40 @@ +"""Env-gated activation hook for the propagate-only OTel shim. + +Dockerfile.otel-shim installs this as ``_lineage_propagate.py`` in the app +environment's site-packages next to a one-line ``.pth``, so ``site`` imports +it at every interpreter start, before any app code. The shim thus attaches +through the environment (like ``JAVA_TOOL_OPTIONS`` / ``NODE_OPTIONS``); the +container's command is never rewritten. + +Inert unless ``LINEAGE_PROPAGATE=1``. When active: pin the propagate-only +posture as env *defaults* (every exporter ``none``, ``tracecontext,baggage`` +propagators — ``setdefault``, so a deliberate override still wins), then run +stock auto-instrumentation via ``initialize()``; which instrumentors activate +depends on what the app imports. + +Failure policy: never take the app down. ``initialize()`` swallows its own +exceptions, the guard below covers the rest, and ``site`` itself survives a +broken ``.pth`` line. A hook failure therefore means propagation is OFF and the +trace fragments at this pod, visibly (``parent.source=none`` on its outbound +hops) — absent lineage, +never wrong lineage. +""" + +import os + +if os.environ.get("LINEAGE_PROPAGATE") == "1": + os.environ.setdefault("OTEL_TRACES_EXPORTER", "none") + os.environ.setdefault("OTEL_METRICS_EXPORTER", "none") + os.environ.setdefault("OTEL_LOGS_EXPORTER", "none") + os.environ.setdefault("OTEL_PROPAGATORS", "tracecontext,baggage") + try: + from opentelemetry.instrumentation.auto_instrumentation import initialize + + initialize() + # Deliberately broad: never break the app this hook rides in. + except Exception: # noqa: BLE001 + import logging + + logging.getLogger(__name__).exception( + "lineage propagate hook failed to initialize; propagation is OFF for this process" + ) diff --git a/authbridge/lineage-attach/sidecar-patch.sh b/authbridge/lineage-attach/sidecar-patch.sh new file mode 100755 index 00000000..8dd88f82 --- /dev/null +++ b/authbridge/lineage-attach/sidecar-patch.sh @@ -0,0 +1,176 @@ +#!/usr/bin/env bash +# sidecar-patch.sh — attach the lineage sidecar to an EXISTING Deployment. +# +# The live applier. The Deployment is deployed and owned by someone else; this +# script only ADDS the lineage pieces through a strategic-merge patch (lists +# merge by name, so nothing the owner wrote changes; the app container is +# touched only when APP_CONTAINER opts it in). Every YAML byte comes from +# attach-lineage.sh: EMIT=cm (the plugin ConfigMap), EMIT=patch (the sidecar). +# This script checks, applies both, and waits for the rollout. +# +# Not durable: the owner keeps owning the object, and a platform rewrite (an +# operator reconcile, a UI redeploy) silently drops the patch — observed live +# when an operator reconciled a patched Deployment. Re-run after any +# platform-side change, or keep the attachment in your own manifests instead +# (README.md "Bring your own manifests"). To back out: the `rollout undo +# --to-revision` line this script prints last (a bare undo is right only +# until the Deployment rolls again for another reason), then delete the CM. +# +# Refused: a target that already carries a container named `envoy-proxy` or an +# init container named `proxy-init` (the operator's sidecar, another mesh's +# init, a leftover of an earlier attach — the merge would silently take it +# over rather than sit beside it), or one that declares 9090, 15123 or 15124 +# (an undeclared sidecar, or the app itself on a port the sidecar binds). +# +# Propagation: an uninstrumented app also needs the shim — bake it with +# build-otel-shim.sh, then pass APP_CONTAINER (+ APP_IMAGE) so the patch +# flips LINEAGE_PROPAGATE=1 on the app's own container. Without it this is +# capture only (README.md "The propagation half"). +# +# Usage: +# DEPLOY=echo-upstream ./sidecar-patch.sh +# DEPLOY=my-agent APP_CONTAINER=agent \ +# APP_IMAGE=docker.io/library/my-agent-otel:latest ./sidecar-patch.sh +# +# Env — read here: +# DEPLOY target Deployment (required) +# NAMESPACE default team1 +# SELF_ID lineage identity (default: DEPLOY) +# APP_CONTAINER the app container to switch propagation on (optional) +# Env — inherited by attach-lineage.sh and validated there (see its header): +# APP_IMAGE, OTEL_ENDPOINT, CAPTURE_IO (content capture, off unless true), +# MAX_PAYLOAD_BYTES (cap on a captured value; plugin default 4096, -1 = whole), +# SIDECAR_IMAGE, PROXY_INIT_IMAGE, NO_EMIT, +# OUTBOUND_PORTS_EXCLUDE (an app's OWN telemetry port, or a plaintext non-HTTP store +# port such as Postgres/SMTP — never LLM/tool/S3 ports). +# +# Requires in the namespace: the platform's `envoy-config` ConfigMap; the +# sidecar + proxy-init images resolvable from the cluster (README "Prerequisites and configuration"). +# +# Structure: read_inputs → preconditions (five, read-only; each returns or exits) → +# note_capture_only → apply (the only cluster writes). gen() is the one bridge +# to the generator. +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + +read_inputs() { + DEPLOY="${DEPLOY:?usage: DEPLOY= [NAMESPACE=team1] [SELF_ID=] [APP_CONTAINER= [APP_IMAGE=]] [SIDECAR_IMAGE= PROXY_INIT_IMAGE=] [OUTBOUND_PORTS_EXCLUDE=ports] sidecar-patch.sh}" + NAMESPACE="${NAMESPACE:-team1}" + SELF_ID="${SELF_ID:-$DEPLOY}" + APP_CONTAINER="${APP_CONTAINER:-}" +} + +require_deployment() { + kubectl get deploy -n "$NAMESPACE" "$DEPLOY" >/dev/null +} + +require_envoy_config() { # the patch mounts it; missing → the pod never starts + kubectl get cm -n "$NAMESPACE" envoy-config >/dev/null || { + echo "error: ConfigMap envoy-config missing in $NAMESPACE (rendered by the platform chart)" >&2 + exit 1 + } +} + +refuse_name_collision() { + # Lists merge by NAME: a target already carrying either name is merged over, + # not added beside. Init containers included — that is where proxy-init + # lands. (A native sidecar, an initContainer with restartPolicy Always, is + # covered by name here but not by the port check below.) + local names n + names="$(kubectl get deploy -n "$NAMESPACE" "$DEPLOY" \ + -o jsonpath='{range .spec.template.spec.initContainers[*]}{.name}{" "}{end}{range .spec.template.spec.containers[*]}{.name}{" "}{end}')" + for n in envoy-proxy proxy-init; do + case " $names " in + *" $n "*) + echo "error: $DEPLOY already has a container named $n — the patch would merge over it, not add beside it" >&2 + echo " (an operator-injected sidecar, another mesh's init, or an earlier attach) — refusing" >&2 + exit 1 ;; + esac + done +} + +refuse_port_collision() { + # An existing sidecar or the app on a sidecar port — see the header. Ports are + # what a Deployment declares; an undeclared app port cannot be seen from here. + local declared_ports p + declared_ports="$(kubectl get deploy -n "$NAMESPACE" "$DEPLOY" \ + -o jsonpath='{range .spec.template.spec.containers[*].ports[*]}{.containerPort}{" "}{end}')" + for p in 15124 15123 9090; do + case " $declared_ports " in + *" $p "*) + echo "error: $DEPLOY already declares containerPort $p, which the lineage sidecar binds" >&2 + echo " (an operator-injected sidecar, or the app itself on that port) — refusing to patch over it" >&2 + exit 1 ;; + esac + done +} + +require_app_container() { + # A strategic merge ADDS a stub container for an unknown name instead of + # failing — so the name must exist. The generator cannot check this. + [ -n "$APP_CONTAINER" ] || return 0 + local containers + containers="$(kubectl get deploy -n "$NAMESPACE" "$DEPLOY" \ + -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{end}')" + case " $containers " in + *" $APP_CONTAINER "*) ;; + *) + echo "error: deploy/$DEPLOY has no container named '$APP_CONTAINER' (it has: ${containers% })" >&2 + echo " — refusing: the patch would ADD a stub container by that name instead of failing" >&2 + exit 1 ;; + esac +} + +note_capture_only() { + # Whether the app propagates traceparent is a property of the app that + # nothing here can see — say so once instead of guessing from its env. + [ -z "$APP_CONTAINER" ] || return 0 + echo "NOTE: the sidecar records every hop; whether $DEPLOY's outbound hops attribute to" >&2 + echo " their inbound depends on the app carrying the trace context (traceparent +" >&2 + echo " tracestate) from inbound to outbound itself (its own instrumentation, or the" >&2 + echo " baked shim + APP_CONTAINER=). Verify pairing" >&2 + echo " under concurrency before relying on it (DESIGN.md, 'The envelope')." >&2 +} + +gen() { # $1 = EMIT mode; the other knobs reach the generator through the environment + EMIT="$1" NAME="$DEPLOY" SELF_ID="$SELF_ID" NAMESPACE="$NAMESPACE" \ + "${SCRIPT_DIR}/attach-lineage.sh" +} + +apply() { + # Both objects are generated before the first write, so a generator refusal + # stops the script with nothing applied. ConfigMap first — the patch's + # volume names it. + local cm patch before + cm="$(gen cm)" + patch="$(gen patch)" + # The revision to return to. A bare `rollout undo` goes one step back, which + # is this one only until the Deployment rolls again for any other reason. + before="$(kubectl get deploy -n "$NAMESPACE" "$DEPLOY" \ + -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}')" + kubectl apply -f - <<<"$cm" + kubectl patch deploy "$DEPLOY" -n "$NAMESPACE" --type strategic --patch "$patch" || { + kubectl delete cm -n "$NAMESPACE" "authbridge-lineage-config-$DEPLOY" # nothing else was written + exit 1 + } + # Said before the wait: a rollout that never completes still needs this line. + echo ">> back out: kubectl -n $NAMESPACE rollout undo deploy/$DEPLOY --to-revision=${before:-?} && kubectl -n $NAMESPACE delete cm authbridge-lineage-config-$DEPLOY" + kubectl rollout status -n "$NAMESPACE" "deploy/$DEPLOY" --timeout=180s + echo ">> lineage sidecar attached to deploy/$DEPLOY (self_id=$SELF_ID, ns=$NAMESPACE)" +} + +preconditions() { # read-only: each returns or exits — nothing is applied yet + require_deployment + require_envoy_config + refuse_name_collision + refuse_port_collision + require_app_container +} + +main() { + read_inputs # DEPLOY required; the rest defaulted or inherited + preconditions # five checks that can only stop the script + note_capture_only # no APP_CONTAINER → say what that means, once + apply # generate both, then the only cluster writes: cm → patch → rollout +} +main "$@"