feat: add OpenTelemetry tracing

.github/bonk_reviewer.md

@@ -29,10 +29,10 @@ You have write access to PR comments via the `gh` CLI. **Prefer the batch review
 
 ### Batch review (recommended)
 
-Write a JSON file and submit it as a review:
+Submit the JSON review directly over standard input. Do not write it to `/tmp` or another file:
 
 ````
-cat > /tmp/review.json << 'REVIEW'
+gh api repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER/reviews --input - << 'REVIEW'
 {
   "event": "COMMENT",
   "body": "Review summary here.",
@@ -46,7 +46,6 @@ cat > /tmp/review.json << 'REVIEW'
   ]
 }
 REVIEW
-gh api repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER/reviews --input /tmp/review.json
 ````
 
 Each comment needs `path`, `line`, `side`, and `body`. Use `suggestion` fences in `body` for applicable changes.

.github/workflows/bonk.yml

@@ -33,16 +33,16 @@ jobs:
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CF_GATEWAY_BASE_URL: https://gateway.ai.cloudflare.com/v1/${{ vars.CLOUDFLARE_ACCOUNT_ID }}/${{ vars.CLOUDFLARE_GATEWAY_ID }}/compat
-          # Must stay below the job's timeout-minutes. The harness defaults to 45m, so 25m gives opencode enough time to retry
-          OPENCODE_TIMEOUT: 25m
-          OPENCODE_PRINT_LOGS: '1'
+          # Match the job timeout; the harness defaults to 45m.
+          OPENCODE_TIMEOUT: 30m
+          OPENCODE_PRINT_LOGS: "1"
           OPENCODE_LOG_LEVEL: INFO
         with:
           oidc_base_url: https://ask-bonk.cloudflare-exponent.workers.dev/auth
-          model: 'cf-gateway/nemotron-3-120b-a12b'
-          mentions: '/bonk,@ask-bonk'
-          forks: 'false'
+          model: "cf-gateway/deepseek-v4-flash-0731"
+          mentions: "/bonk,@ask-bonk"
+          forks: "false"
           permissions: write
-          opencode_version: '1.18.13'
+          opencode_version: "1.18.13"
           # token_permissions defaults to WRITE so bonk can push commits
           # when asked via /bonk.

.github/workflows/new-pr-review.yml

@@ -39,16 +39,16 @@ jobs:
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CF_GATEWAY_BASE_URL: https://gateway.ai.cloudflare.com/v1/${{ vars.CLOUDFLARE_ACCOUNT_ID }}/${{ vars.CLOUDFLARE_GATEWAY_ID }}/compat
-          # Must stay below the job's timeout-minutes. The harness defaults to 45m, so 25m gives opencode enough time to retry
-          OPENCODE_TIMEOUT: 25m
-          OPENCODE_PRINT_LOGS: '1'
+          # Match the job timeout; the harness defaults to 45m.
+          OPENCODE_TIMEOUT: 30m
+          OPENCODE_PRINT_LOGS: "1"
           OPENCODE_LOG_LEVEL: INFO
         with:
           oidc_base_url: https://ask-bonk.cloudflare-exponent.workers.dev/auth
-          model: 'cf-gateway/nemotron-3-120b-a12b'
-          forks: 'false'
+          model: "cf-gateway/deepseek-v4-flash-0731"
+          forks: "false"
           permissions: write
-          opencode_version: '1.18.13'
+          opencode_version: "1.18.13"
           prompt: ${{ steps.prompt.outputs.value }}
           # The auto-reviewer must never push to PR branches.
-          token_permissions: 'NO_PUSH'
+          token_permissions: "NO_PUSH"

.opencode/agents/reviewer.md

@@ -1,7 +1,7 @@
 ---
 description: Read-only code reviewer for pull requests
 mode: primary
-model: cf-gateway/nemotron-3-120b-a12b
+model: cf-gateway/deepseek-v4-flash-0731
 temperature: 0.1
 permission:
   edit: deny

architecture/01-model-source.md

@@ -10,6 +10,7 @@ A Cog model consists of:
 my-model/
 ├── cog.yaml          # Environment configuration
 ├── run.py            # Runner class
+├── telemetry.py      # Optional Python observability configuration
 └── weights/          # Model weights (optional, can be downloaded)
 ```
 
@@ -44,6 +45,7 @@ concurrency:
 | `build.run`             | Arbitrary shell commands during build       |
 | `run`                   | Path to runner class (`module:ClassName`)   |
 | `concurrency.max`       | Max concurrent predictions (requires async) |
+| `observability.config`  | Optional Python tracer-provider factory     |
 
 The [Build System](./05-build-system.md) uses this configuration to produce an image containing all necessary dependencies, libraries, and the correct Python/CUDA versions.
 

architecture/04-container-runtime.md

@@ -385,6 +385,18 @@ Models can record custom metrics via `self.record_metric(name, value, mode)` in
 
 Metrics appear in the prediction response's `metrics` object alongside the built-in `predict_time`.
 
+## Distributed Tracing
+
+Opt-in OpenTelemetry tracing uses a provider in the parent process, a provider in the worker process, and a Python provider installed before predictor import. The parent and worker exchange an optional W3C carrier alongside prediction IPC. Transport context remains separate from the user-owned request `context` map.
+
+Models may provide `observability.config`, which Cog validates and stages at a fixed image path. During worker setup, Cog loads that module, installs the `TracerProvider` returned by `create_tracer_provider()`, and then calls optional `configure_instrumentation()` before predictor import. Cog owns provider flush and shutdown. Configuration failures stop model setup because the user explicitly selected that module.
+
+The framework trace covers HTTP handling, validation, the logical prediction lifetime, worker execution, input preparation, predictor invocation, output upload, and setup. Model code creates ordinary child spans through `opentelemetry.trace`.
+
+The `Prediction` state object owns the logical prediction span so asynchronous and SSE requests can return before the span reaches a terminal state. Signed output uploads never receive trace headers. Webhooks receive the active prediction context.
+
+Framework tracing is inert unless the image enables it and the runtime supplies a collector endpoint. A configured Python provider may run without that endpoint, but it then emits model spans without framework parents. The disabled path creates no provider, exporter, background telemetry thread, connection, or real framework span.
+
 ## User-Defined Healthchecks
 
 Models can implement a custom healthcheck that runs alongside the built-in health state machine. The parent sends `Healthcheck { id }` on the control channel; the worker runs the user's healthcheck and responds with `HealthcheckResult { id, status, error }`.
@@ -399,6 +411,8 @@ If the healthcheck fails, the HTTP `/health-check` endpoint returns `UNHEALTHY`
 | `COG_LOG_LEVEL`                  | INFO    | Logging verbosity (ignored if `RUST_LOG` is set) |
 | `COG_MAX_CONCURRENCY`            | 1       | Number of concurrent prediction slots            |
 | `COG_SETUP_TIMEOUT`              | none    | Setup timeout in seconds (0 is ignored)          |
+| `COG_TRACE_ENABLED`              | false   | Runtime tracing switch for an opted-in image     |
+| `COG_OBSERVABILITY_CONFIG`       | none    | Internal path to staged Python telemetry config  |
 | `COG_THROTTLE_RESPONSE_INTERVAL` | 0.5s    | Webhook response throttling interval             |
 | `LOG_FORMAT`                     | json    | Set to `console` for human-readable log output   |
 

architecture/05-build-system.md

@@ -60,6 +60,7 @@ Reads `cog.yaml` and validates/completes the configuration:
 - Validates Python version (3.10-3.13)
 - Auto-detects CUDA version from PyTorch/TensorFlow requirements
 - Resolves package versions against compatibility matrix
+- Validates optional Python observability configuration inside the project
 
 ```mermaid
 flowchart LR
@@ -85,6 +86,8 @@ flowchart LR
 
 The generator produces a Dockerfile from the validated config.
 
+When `observability.config` is set, build orchestration validates the project-local file and stages it in the private `cog_build` context. Generated and custom-Dockerfile wrapper layers copy that artifact to `/.cog/telemetry.py`; the original user path is never exposed to runtime path resolution.
+
 #### Generated Dockerfile Sections
 
 ```dockerfile