From 1721bf1ac85e7a0f9705bae509ea3c678a9bbf2d Mon Sep 17 00:00:00 2001 From: Saurabh Jain Date: Fri, 8 May 2026 12:44:04 +0200 Subject: [PATCH] feat(telemetry): adopt v1 schema (telemetry_type, profile, classifiers) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aligns the anonymous 7-day heartbeat with the v1 telemetry schema locked at axonflow-enterprise#2004 (#2008 plugin umbrella). Bundled into the same minor as the V1.1 list-recent-decisions skill from the prior unreleased section so users only see one minor transition. New emitted fields on the heartbeat: - telemetry_type: "plugin" — discriminator the receiver uses to route plugin pings vs SDK / platform / synthetic on the v1 schema - endpoint_type — localhost | private_network | remote | unknown; mirrors SDK ClassifyEndpoint - profile — sourced from AXONFLOW_PROFILE; reports "unknown" when unset deployment_mode enum normalised to v1 (`self_hosted | community_saas | unknown`). Detection moves off AXONFLOW_MODE / AXONFLOW_AUTH onto endpoint host + AXONFLOW_TRY=1 (community-saas behind custom hostname). The prior `production`/`development`/`community-saas` values are removed — analytics queries that match those must update. Heartbeat real-stack harness updated to assert all four v1 payload fields against a real network round-trip; runtime-e2e/v1_telemetry_schema/ delegates to it for the definition-of-done mechanical gate. Bumps 1.3.0 → 1.4.0 in .codex-plugin/plugin.json + CHANGELOG. Signed-off-by: Saurabh Jain --- .codex-plugin/marketplace.json | 4 +- .codex-plugin/plugin.json | 2 +- CHANGELOG.md | 13 +++- runtime-e2e/v1_telemetry_schema/README.md | 25 +++++++ runtime-e2e/v1_telemetry_schema/test.sh | 49 +++++++++++++ scripts/telemetry-ping.sh | 75 ++++++++++++++++++-- tests/heartbeat-real-stack/run_real_stack.sh | 31 ++++++-- 7 files changed, 183 insertions(+), 16 deletions(-) create mode 100644 runtime-e2e/v1_telemetry_schema/README.md create mode 100755 runtime-e2e/v1_telemetry_schema/test.sh diff --git a/.codex-plugin/marketplace.json b/.codex-plugin/marketplace.json index bb743a3..039b461 100644 --- a/.codex-plugin/marketplace.json +++ b/.codex-plugin/marketplace.json @@ -7,14 +7,14 @@ }, "metadata": { "description": "Runtime governance for OpenAI Codex. Policy enforcement on terminal commands, advisory governance via skills, PII detection, audit trails, and compliance-grade decision records.", - "version": "1.3.0" + "version": "1.4.0" }, "plugins": [ { "name": "axonflow", "source": "./", "description": "Policy enforcement, PII detection, and audit trails for OpenAI Codex. Hybrid governance — enforces policies on terminal commands (exec_command) via hooks, provides advisory governance for other tools via implicit-activation skills, and records compliance-grade audit trails. Self-hosted via Docker — all data stays on your infrastructure.", - "version": "1.3.0", + "version": "1.4.0", "author": { "name": "AxonFlow", "email": "hello@getaxonflow.com", diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index e5eafc5..0270066 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "axonflow", "displayName": "AxonFlow Governance", "description": "Policy enforcement, PII detection, and audit trails for OpenAI Codex. Hybrid governance — enforces policies on terminal commands (exec_command) via hooks, provides advisory governance for other tools via implicit-activation skills, and records compliance-grade audit trails. Self-hosted via Docker — all data stays on your infrastructure.", - "version": "1.3.0", + "version": "1.4.0", "author": { "name": "AxonFlow", "email": "hello@getaxonflow.com", diff --git a/CHANGELOG.md b/CHANGELOG.md index 44aade4..41d1195 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,18 @@ ## [Unreleased] +## [1.4.0] - 2026-05-08 + ### Added -- **`list-recent-decisions` skill** — V1.1 (axonflow-enterprise#1982). Codex agents drive the new `list_recent_decisions` MCP tool to surface "what just got blocked" UX, appeal flows, and decision-history forensics. Tier-throttled per the platform's Free/Pro window+limit; the skill explicitly instructs the LLM to render the V1 upgrade envelope verbatim on cap-hit (locks in `feedback_429_no_upgrade_hint_is_conversion_gap.md`). -- `runtime-e2e/list-recent-decisions/` — wire-level test asserting (a) MCP server advertises the tool, (b) happy-path returns the decisions shape, (c) cap-hit returns the wrapped V1 upgrade envelope with `upgrade.compare_url` + `upgrade.buy_url` intact. -- `tests/e2e/runtime-mcp-tools.sh` extended with a 7th scenario covering the over-cap envelope at the wire level. +- **V1.1 `list-recent-decisions` skill** (axonflow-enterprise#1982). Drives the new `list_recent_decisions` MCP tool from Codex; Free-tier cap-hits render the V1 upgrade envelope verbatim. Plus `runtime-e2e/list-recent-decisions/` and a 7th over-cap scenario in `tests/e2e/runtime-mcp-tools.sh`. +- **v1 telemetry-schema** (axonflow-enterprise#2008): heartbeat now emits `telemetry_type: "plugin"`, `endpoint_type` (`localhost | private_network | remote | unknown`), and `profile` from `AXONFLOW_PROFILE`. +- `AXONFLOW_PROFILE` env var: drives the new `profile` payload field; reports `unknown` when unset. +- `AXONFLOW_TRY=1` env var: forces `deployment_mode=community_saas` for tenants behind custom hostnames proxying try.getaxonflow.com. + +### Changed + +- `deployment_mode` allowlist normalised to `self_hosted | community_saas | unknown` (was `production`/`development`/`community-saas`). Detection now derives from endpoint host + `AXONFLOW_TRY=1`. Analytics queries on the legacy values must update. ## [1.3.0] - 2026-05-07 — V1 Plugin Pro upgrade-prompt envelope + 5 new MCP tools surfaced diff --git a/runtime-e2e/v1_telemetry_schema/README.md b/runtime-e2e/v1_telemetry_schema/README.md new file mode 100644 index 0000000..9fd72cb --- /dev/null +++ b/runtime-e2e/v1_telemetry_schema/README.md @@ -0,0 +1,25 @@ +# Runtime E2E — v1 telemetry-schema heartbeat (#2008) + +Drives the plugin's `pre-tool-check.sh` hook against a real local +checkpoint server, captures the actual heartbeat payload off the wire, +and asserts the four v1-schema fields: `telemetry_type`, +`deployment_mode`, `endpoint_type`, `profile`. + +This wrapper delegates to `tests/heartbeat-real-stack/run_real_stack.sh` +which is the canonical telemetry runtime-proof harness — it predates +the `runtime-e2e/` directory convention but exercises exactly the same +real-runtime path the gate is asking for. + +## Prereqs + +- `bash`, `jq`, `curl`, `python3` on `$PATH` + +## Run + +```bash +./runtime-e2e/v1_telemetry_schema/test.sh +``` + +The harness binds the receiver to a free 127.0.0.1 port, exercises a +cold-start (15 assertions including v1 field shape) and a warm-cache +(stamp-gate suppression) scenario, then exits non-zero on any failure. diff --git a/runtime-e2e/v1_telemetry_schema/test.sh b/runtime-e2e/v1_telemetry_schema/test.sh new file mode 100755 index 0000000..c0e5e54 --- /dev/null +++ b/runtime-e2e/v1_telemetry_schema/test.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Runtime proof for the v1 telemetry-schema heartbeat payload (#2008). +# +# The plugin's anonymous heartbeat now carries four v1-schema fields: +# telemetry_type, deployment_mode, endpoint_type, profile. The +# canonical wire-shape proof for these lives at +# tests/heartbeat-real-stack/run_real_stack.sh — that harness drives +# the public pre-tool-check.sh hook against a Python fake checkpoint +# server bound to 127.0.0.1, captures the actual ping payload off the +# wire, and asserts the four field contracts. +# +# This runtime-e2e/ wrapper exists so the definition-of-done.yml +# mechanical gate (which only inspects runtime-e2e/) sees the runtime +# proof for this PR. It runs the same harness — no mocks, no stubs. +# +# Exit codes: +# 0 PASS — all 15 cold-start + warm-cache assertions pass +# 1 FAIL — any assertion failed +# 0 SKIP — required tools missing (bash, jq, curl, python3) + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PLUGIN_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)" +HARNESS="$PLUGIN_DIR/tests/heartbeat-real-stack/run_real_stack.sh" + +for tool in bash jq curl python3; do + if ! command -v "$tool" >/dev/null 2>&1; then + echo "SKIP: $tool not on PATH" + exit 0 + fi +done + +if [ ! -f "$HARNESS" ]; then + echo "FAIL: harness missing at $HARNESS" + exit 1 +fi + +echo "==> Running heartbeat-real-stack harness against v1 telemetry payload" +( cd "$PLUGIN_DIR" && bash "$HARNESS" ) +RC=$? + +if [ $RC -eq 0 ]; then + echo "PASS: v1 telemetry-schema fields verified on the wire" + exit 0 +fi + +echo "FAIL: heartbeat-real-stack harness reported assertion failures (rc=$RC)" +exit 1 diff --git a/scripts/telemetry-ping.sh b/scripts/telemetry-ping.sh index 1ad7f60..4100173 100755 --- a/scripts/telemetry-ping.sh +++ b/scripts/telemetry-ping.sh @@ -173,12 +173,69 @@ else PLATFORM_VERSION="\"${PLATFORM_VERSION}\"" fi -if [ "${AXONFLOW_MODE:-}" = "community-saas" ]; then - DEPLOYMENT_MODE="community-saas" -elif [ -n "${AXONFLOW_AUTH:-}" ]; then - DEPLOYMENT_MODE="production" +# v1 telemetry-schema (#2008) classifiers. deployment_mode is derived from +# the configured endpoint host plus the AXONFLOW_TRY=1 explicit override; +# endpoint_type follows the SDK ClassifyEndpoint shape so cross-client +# analytics dimensions stay consistent. Both functions return one of the +# fixed v1 allowlist values; "unknown" is preferred over silent fallback +# when input is missing/unparseable. +classify_deployment_mode() { + local endpoint="$1" + if [ "${AXONFLOW_TRY:-}" = "1" ]; then + printf 'community_saas' + return + fi + if [ -z "$endpoint" ]; then + printf 'unknown' + return + fi + local host + host=$(printf '%s' "$endpoint" | sed -nE 's#^[a-zA-Z][a-zA-Z0-9+.-]*://([^/:?#]+).*#\1#p' | tr '[:upper:]' '[:lower:]') + if [ -z "$host" ]; then + printf 'unknown' + return + fi + case "$host" in + try.getaxonflow.com|*.try.getaxonflow.com) + printf 'community_saas' ;; + *) + printf 'self_hosted' ;; + esac +} + +classify_endpoint_type() { + local endpoint="$1" + if [ -z "$endpoint" ]; then + printf 'unknown' + return + fi + local host + host=$(printf '%s' "$endpoint" | sed -nE 's#^[a-zA-Z][a-zA-Z0-9+.-]*://([^/:?#]+).*#\1#p' | tr '[:upper:]' '[:lower:]') + if [ -z "$host" ]; then + printf 'unknown' + return + fi + case "$host" in + localhost|127.0.0.1|::1|0.0.0.0|*.localhost) + printf 'localhost'; return ;; + *.local|*.internal|*.lan|*.intranet) + printf 'private_network'; return ;; + esac + case "$host" in + 10.*|192.168.*) printf 'private_network'; return ;; + 172.1[6-9].*|172.2[0-9].*|172.3[01].*) printf 'private_network'; return ;; + esac + printf 'remote' +} + +DEPLOYMENT_MODE=$(classify_deployment_mode "$ENDPOINT") +ENDPOINT_TYPE=$(classify_endpoint_type "$ENDPOINT") + +PROFILE_RAW="${AXONFLOW_PROFILE:-}" +if [ -z "$PROFILE_RAW" ]; then + PROFILE="unknown" else - DEPLOYMENT_MODE="development" + PROFILE="$PROFILE_RAW" fi HOOK_COUNT=0 @@ -188,16 +245,20 @@ if [ -f "$HOOKS_FILE" ]; then fi PAYLOAD=$(jq -n \ + --arg telemetry_type "plugin" \ --arg sdk "codex-plugin" \ --arg sdk_version "$SDK_VERSION" \ --arg os "$(uname -s)" \ --arg arch "$(uname -m)" \ --arg runtime_version "${BASH_VERSION:-unknown}" \ --arg deployment_mode "$DEPLOYMENT_MODE" \ + --arg endpoint_type "$ENDPOINT_TYPE" \ + --arg profile "$PROFILE" \ --arg instance_id "$INSTANCE_ID" \ --argjson hook_count "$HOOK_COUNT" \ --argjson platform_version "$PLATFORM_VERSION" \ '{ + telemetry_type: $telemetry_type, sdk: $sdk, sdk_version: $sdk_version, platform_version: $platform_version, @@ -205,8 +266,10 @@ PAYLOAD=$(jq -n \ arch: $arch, runtime_version: $runtime_version, deployment_mode: $deployment_mode, + endpoint_type: $endpoint_type, features: ["hooks:\($hook_count)"], - instance_id: $instance_id + instance_id: $instance_id, + profile: $profile }' 2>/dev/null) if [ -z "$PAYLOAD" ]; then diff --git a/tests/heartbeat-real-stack/run_real_stack.sh b/tests/heartbeat-real-stack/run_real_stack.sh index 0b80492..08c4fc4 100755 --- a/tests/heartbeat-real-stack/run_real_stack.sh +++ b/tests/heartbeat-real-stack/run_real_stack.sh @@ -179,13 +179,36 @@ else fail "telemetry counter is $COLD_COUNTER (expected 1)" fi -# Assertion 5: deployment_mode=community-saas in the captured ping. +# Assertion 5: v1 telemetry-schema fields in the captured ping (#2008). +# In this harness the heartbeat endpoint is the 127.0.0.1 fake (set via +# AXONFLOW_HARNESS_AGENT_ENDPOINT), so the v1 classifier returns +# endpoint_type=localhost and deployment_mode=self_hosted — the canary +# log on stderr ("mode=community-saas") describes a separate +# bootstrap-side mode dimension and is asserted independently above. if [ -f "$WORK_DIR/_pings.jsonl" ]; then + COLD_TT=$(jq -r '.telemetry_type' "$WORK_DIR/_pings.jsonl" | head -1) + if [ "$COLD_TT" = "plugin" ]; then + pass "ping telemetry_type=plugin" + else + fail "ping telemetry_type=$COLD_TT (expected plugin)" + fi + COLD_ET=$(jq -r '.endpoint_type' "$WORK_DIR/_pings.jsonl" | head -1) + if [ "$COLD_ET" = "localhost" ]; then + pass "ping endpoint_type=localhost (harness binds to 127.0.0.1)" + else + fail "ping endpoint_type=$COLD_ET (expected localhost)" + fi + COLD_PROFILE=$(jq -r '.profile' "$WORK_DIR/_pings.jsonl" | head -1) + if [ "$COLD_PROFILE" = "unknown" ]; then + pass "ping profile=unknown (AXONFLOW_PROFILE unset)" + else + fail "ping profile=$COLD_PROFILE (expected unknown)" + fi COLD_MODE=$(jq -r '.deployment_mode' "$WORK_DIR/_pings.jsonl" | head -1) - if [ "$COLD_MODE" = "community-saas" ]; then - pass "ping deployment_mode=community-saas" + if [ "$COLD_MODE" = "self_hosted" ]; then + pass "ping deployment_mode=self_hosted (harness endpoint is 127.0.0.1)" else - fail "ping deployment_mode=$COLD_MODE (expected community-saas)" + fail "ping deployment_mode=$COLD_MODE (expected self_hosted)" fi fi