Feat: Add the lineage demo on the Weather Agent pair #853
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,186 @@ | ||
| # Lineage demo — the weather agent and its tool, with per-request lineage | ||
|
|
||
| The [Weather Agent](../weather-agent/demo-ui.md) pair from `rossoctl/examples` | ||
| — an A2A agent that asks an LLM and calls one MCP tool — deployed plain, then | ||
| given per-request lineage with the [lineage attach kit](../../lineage-attach/README.md) | ||
| and nothing else. Six steps. You will see the same turn twice: first as | ||
| **19 separate traces** (the entry alone and each of the app's 18 calls in a | ||
| trace of its own, because the app does not carry `traceparent`), then as | ||
| **one trace of 70 spans** with one root, after the | ||
| app's own propagation is switched on. Nothing about the app is edited except | ||
| one environment variable that the app itself defines. | ||
|
|
||
| Read [the kit's README](../../lineage-attach/README.md) for what the spans | ||
| carry and [DESIGN](../../lineage-attach/DESIGN.md) for why propagation is the | ||
| app's job; this page is only the walk-through. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - The rossoctl platform on kind (`rossoctl` cluster), namespace `team1` with | ||
| its platform-rendered `envoy-config` ConfigMap, and the platform collector | ||
| (`deploy/otel-collector` in `rossoctl-system`, stock `debug` exporter). | ||
| - A sidecar image that carries `lineage-telemetry` (cortex #761): until a release does, | ||
| [RECIPE step 1](../../lineage-attach/RECIPE.md#1-a-sidecar-image-that-carries-the-plugin-once-per-cluster-until-a-release-does) | ||
| builds and loads it. Then, for the whole session: | ||
|
|
||
| ```sh | ||
| export SIDECAR_IMAGE=docker.io/library/authbridge-envoy:latest PROXY_INIT_IMAGE=docker.io/library/proxy-init:latest | ||
| ``` | ||
| - An LLM the agent can reach over **plaintext HTTP** (an HTTPS LLM is TLS | ||
| passthrough: the sidecar records no hop for it). Default: Ollama on the host | ||
| with `qwen2.5:7b` (`ollama pull qwen2.5:7b`); edit `k8s/weather.yaml`'s | ||
| `weather-llm` ConfigMap for anything else. | ||
| - Egress from the cluster to `https://wttr.in`, which the tool queries. | ||
| - `kubectl` and `python3` on the host (`ask.sh`, `show-trace.py`). | ||
|
Member
There was a problem hiding this comment. nit — the host-tool line is right and closes a gap the kit's own prerequisites still have, but the permissions are unstated: |
||
|
|
||
| Run everything from this directory. `KIT=../../lineage-attach`. | ||
|
Member
There was a problem hiding this comment. must-fix — for d in weather-tool weather-service; do NAMESPACE=team1 DEPLOY=$d CAPTURE_IO=true $KIT/sidecar-patch.sh; doneWith Everything else about that invocation is correct: |
||
|
|
||
| ## 1. Deploy the pair, plain | ||
|
|
||
| ```sh | ||
| kubectl apply -f k8s/weather.yaml | ||
| kubectl -n team1 rollout status deploy/weather-tool && kubectl -n team1 rollout status deploy/weather-service | ||
| ./ask.sh "What is the weather in Paris?" | ||
| ``` | ||
|
|
||
| `k8s/weather.yaml` is the two stock images (`weather_service`, `weather_tool`) | ||
| as two Deployments and two Services — no `AgentRuntime`, no platform sidecar, | ||
| no auth. `ask.sh` sends one A2A `message/send` from a pod inside the cluster | ||
| (a port-forward would bypass the sidecar) with a `traceparent` whose trace id | ||
| it prints. Pass: an `answer:` line with the weather. Nothing is captured yet. | ||
|
|
||
| ## 2. Attach lineage (capture) | ||
|
|
||
| ```sh | ||
| for d in weather-tool weather-service; do NAMESPACE=team1 DEPLOY=$d CAPTURE_IO=true $KIT/sidecar-patch.sh; done | ||
| ``` | ||
|
|
||
| Pass — each ends with: | ||
|
|
||
| ``` | ||
| >> back out: kubectl -n team1 rollout undo deploy/<name> --to-revision=<n> && kubectl -n team1 delete cm authbridge-lineage-config-<name> | ||
| deployment "<name>" successfully rolled out | ||
| >> lineage sidecar attached to deploy/<name> (self_id=<name>, ns=team1) | ||
| ``` | ||
|
|
||
| That is the whole attachment: a ConfigMap and a strategic-merge patch per | ||
| Deployment, both generated by the kit. Both pods are now `2/2`. `CAPTURE_IO=true` | ||
|
Member
There was a problem hiding this comment. suggestion — worth stating where the captured content lands. With That is the right choice for this demo and the trade-off is already half-stated ("the demo's choice, not the kit's default"). But the demo is the artefact people copy into their own clusters, and #761 gates |
||
| is the demo's choice, not the kit's default: the spans then carry the question, | ||
| the tool arguments and the prompts, so the trace reads as a story. Keep the | ||
| back-out lines; step 6 uses them. | ||
|
|
||
| ## 3. One turn — every hop alone | ||
|
|
||
| ```sh | ||
| ./ask.sh "What is the weather in Paris?" # prints: trace id: <id> | ||
| ./show-trace.py <id> | ||
| ``` | ||
|
|
||
| `show-trace.py` reads the collector's log and lists the sidecar spans of one | ||
| trace. Measured: | ||
|
|
||
| ``` | ||
| 2 sidecar spans, 1 exchanges: 1 inbound a2a | ||
| parent.source: 1 wire, 0 tracestate, 0 none | ||
| traces begun by an unparented outbound hop while this one was in flight: 18 | ||
| shape: ENTRY ONLY — nothing the app called landed here; its calls are the stray traces above | ||
| ``` | ||
|
|
||
| The sidecar saw everything the turn did — 35 exchanges: the A2A entry, 16 MCP | ||
| exchanges to the tool (session handshakes, tool listing, the call), 2 LLM | ||
| calls, and the tool's 16 inbound sides — and recorded all 70 spans. But the | ||
| app forwarded no `traceparent`, so the entry is alone in your trace and each | ||
| of the app's 18 calls started a trace of its own: its sidecar found nothing on | ||
| the wire to parent on (`parent.source=none`), forwarded a `traceparent` of its | ||
| own making, and the tool's side of each MCP call joined *that* trace — 19 | ||
| traces, each internally consistent and each useless, because nothing links a | ||
| call to the question that caused it. This is the case DESIGN calls *the one | ||
| that looks fine and is not*: count spans and it passes; read the shape and it | ||
| fails. | ||
|
|
||
| ## 4. Switch the app's propagation on | ||
|
|
||
| The weather agent ships its own OpenTelemetry setup, activated by one | ||
| variable it defines: when `OTEL_EXPORTER_OTLP_ENDPOINT` is set it extracts the | ||
| inbound `traceparent` and instruments `httpx`, so its LLM and tool calls carry | ||
| it. Point it at the platform collector's OTLP/HTTP receiver — port 8335 on the | ||
| stock chart, not 4318: | ||
|
|
||
| ```sh | ||
| kubectl -n team1 set env deploy/weather-service OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.rossoctl-system.svc.cluster.local:8335 | ||
|
Member
There was a problem hiding this comment. suggestion — Related, while you are in this step: the agent's own OTLP export now goes out through its lineage sidecar with no |
||
| kubectl -n team1 rollout status deploy/weather-service | ||
| ``` | ||
|
|
||
| > **Why not the kit's shim here?** Because the interlock refuses this image, | ||
| > correctly: `$KIT/build-otel-shim.sh ghcr.io/rossoctl/examples/weather_service:latest` | ||
| > exits 3 with `REFUSING to bake …: it already instruments httpx`. An app that | ||
| > brings its own instrumentation gets its own switch; the shim is for the app | ||
| > that brings none ([RECIPE step 2](../../lineage-attach/RECIPE.md#2-bake-the-propagation-shim-onto-the-app-image-once-per-image)). | ||
| > The tool image bakes, but it does not need to: its one call that matters is | ||
| > HTTPS to `wttr.in`, which the sidecar passes through unseen, and its other | ||
| > outbound — its own OTLP export to the collector — is on the plugin's default | ||
| > `bypass_hosts` list, so capture is all it needs. | ||
|
|
||
| ## 5. The same turn — one trace | ||
|
|
||
| ```sh | ||
| ./ask.sh "What is the weather in Paris?" | ||
| ./show-trace.py <id> | ||
| ``` | ||
|
|
||
| Measured: | ||
|
|
||
| ``` | ||
| 70 sidecar spans, 35 exchanges: 1 inbound a2a, 16 inbound mcp, 2 outbound inference, 16 outbound mcp | ||
| parent.source: 1 wire, 34 tracestate, 0 none | ||
| traces begun by an unparented outbound hop while this one was in flight: 0 | ||
| shape: OK — one root, unstamped only at the entry, the app's calls are in this trace | ||
| ``` | ||
|
|
||
| The table above those lines is the turn, hop by hop, in time order: the A2A | ||
| entry (`wire` — the caller minted the trace), then each MCP exchange seen | ||
| twice (outbound at the agent, inbound at the tool, both `tracestate`), the two | ||
| LLM calls (`inference`, peer `host.containers.internal:11434`), and the A2A | ||
| response last. The apps' own spans arrive in the same trace too — the agent's | ||
| A2A server, LangChain and `openai.chat` spans and its `httpx` `POST`s, the | ||
| tool's `tools/list` and `tools/call` — 82 of them on this turn; the sidecar's | ||
| are the ones with `lineage.*` attributes. | ||
|
|
||
| ## 6. Back out | ||
|
|
||
| The kit's attachment is one revision per Deployment and one ConfigMap each, | ||
| and `sidecar-patch.sh` printed the revision to return to. The tool is one | ||
| revision above plain; the agent is two, because step 4 rolled it again — a | ||
| bare `rollout undo` there would land on the attached spec whose ConfigMap is | ||
| about to go, and the pod would hang on the missing mount. Name the revision: | ||
|
|
||
| ```sh | ||
| kubectl -n team1 rollout undo deploy/weather-tool --to-revision=1 && kubectl -n team1 delete cm authbridge-lineage-config-weather-tool | ||
|
Member
There was a problem hiding this comment. nit — step 2 tells the reader to "Keep the back-out lines; step 6 uses them" ( |
||
| kubectl -n team1 rollout undo deploy/weather-service --to-revision=1 && kubectl -n team1 delete cm authbridge-lineage-config-weather-service | ||
| ``` | ||
|
|
||
| Pass: `kubectl -n team1 get pods` shows both pods `1/1` again. Then the app: | ||
| `kubectl delete -f k8s/weather.yaml`. | ||
|
|
||
| ## Files | ||
|
|
||
| | file | what | | ||
| |---|---| | ||
| | `k8s/weather.yaml` | the pair, plain: two Deployments, two Services, one ConfigMap for the LLM | | ||
| | `ask.sh` | one A2A turn from inside the cluster with a chosen trace id | | ||
| | `show-trace.py` | the shape of one trace from the collector log, with a verdict; exit 0 only for one root with the app's calls inside it | | ||
|
|
||
| Everything that attaches lineage is the kit's; this directory holds only the | ||
| application and the two readers. | ||
|
|
||
| ## If it does not work | ||
|
|
||
| | symptom | cause | | ||
| |---|---| | ||
| | `ask.sh` prints no answer, or the agent logs `Cannot connect to MCP` | the tool is not ready, or `MCP_URL` in `k8s/weather.yaml` does not match the Service name | | ||
| | the answer is an LLM error | `weather-llm` ConfigMap: the base URL is not reachable from a pod (podman kind: `host.containers.internal`; docker kind: `host.docker.internal`), or the model is not pulled | | ||
| | `show-trace.py` finds no spans | the sidecar image predates the plugin, or the collector was restarted — `kubectl -n team1 logs deploy/weather-service -c envoy-proxy` | | ||
| | step 5 still says FRAGMENTED | the agent did not restart with the variable — `kubectl -n team1 logs deploy/weather-service -c agent \| grep 'httpx instrumented'` | | ||
|
Member
There was a problem hiding this comment. nit — this row promises a verdict the code will not print for that cause. If the agent did not restart with the variable, its outbound calls carry nothing, so the queried trace holds the entry alone and While in this table: the likeliest real failure is missing from it — |
||
| | the agent logs export failures after step 4 | wrong collector port: the stock chart serves OTLP/HTTP on `8335`, not `4318`; propagation works regardless, but the agent's own spans do not arrive | | ||
| | the answer is a tool error about `wttr.in` | the tool needs egress to `https://wttr.in`; the sidecar passes HTTPS through, so this is cluster egress, not lineage | | ||
| | anything about the attachment itself | the kit's [Troubleshooting](../../lineage-attach/README.md#troubleshooting) | | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| #!/usr/bin/env bash | ||
| # ask.sh — one A2A turn to the weather agent, sent from inside the cluster (a | ||
| # port-forward would bypass the sidecar) with a trace id of our choosing, so | ||
| # the spans it produces can be found by that id. | ||
| # | ||
| # The request is the A2A 0.3 JSON-RPC shape (`message/send`, `parts: [{kind: | ||
| # text}]`); the agent runs a2a-sdk 1.x and answers it through its 0.3 | ||
| # compatibility routes. The result is a Task (answer in status.message or the | ||
| # artifacts) or a Message (answer in parts); both are read. | ||
| # | ||
| # Usage: ./ask.sh ["What is the weather in Paris?"] | ||
| # NS=team1 SVC=weather-service PORT=8080 override the target. | ||
| set -euo pipefail | ||
| NS="${NS:-team1}"; SVC="${SVC:-weather-service}"; PORT="${PORT:-8080}" | ||
| question="${1:-What is the weather in Paris?}" | ||
| trace_id="$(python3 -c 'import secrets; print(secrets.token_hex(16))')" | ||
| body="$(python3 -c 'import json, sys, uuid | ||
| print(json.dumps({"jsonrpc": "2.0", "id": "1", "method": "message/send", "params": {"message": { | ||
| "role": "user", "messageId": uuid.uuid4().hex, "parts": [{"kind": "text", "text": sys.argv[1]}]}}}))' "$question")" | ||
| echo "trace id: ${trace_id}" | ||
| kubectl -n "$NS" run "ask-${trace_id:0:12}" --rm -i --quiet --restart=Never --image=curlimages/curl:8.11.1 -- \ | ||
| curl -sS --max-time 300 -H 'content-type: application/json' \ | ||
| -H "traceparent: 00-${trace_id}-0000000000000001-01" \ | ||
| -d "$body" "http://${SVC}:${PORT}/" \ | ||
| | python3 -c 'import json, sys | ||
| r = json.load(sys.stdin) | ||
| res = r.get("result", r) | ||
| parts = (res.get("status", {}).get("message", {}).get("parts", []) | ||
| or [p for a in res.get("artifacts", []) for p in a.get("parts", [])] | ||
| or res.get("parts", [])) | ||
| print("answer:", " ".join(p.get("text", "") for p in parts) or json.dumps(r)[:300])' | ||
| echo "trace id: ${trace_id} (kubectl -n rossoctl-system logs deploy/otel-collector | grep -c ${trace_id})" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggestion — this row says "first as 35 separate traces", but the demo's own README says 19 (
demos/lineage/README.md:7), and 19 is the number the tooling actually produces: 18 strays plus the queried trace (show-trace.py:103,109). 35 is the exchange count, not the trace count — the two numbers appear a few lines apart in the demo README and got crossed here. Worth fixing since the index row is what most people read first.