From ddb4b427bb66d50917dfe5fc9212ea9d65d3f8ac Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Tue, 11 Aug 2026 14:15:17 +0300 Subject: [PATCH 1/4] docs: add RFD for local telemetry recording Define a consented, local-only telemetry contract covering resolution, sessions, commands, hook reliability, and Claude skill activation. Document privacy boundaries, storage behavior, user controls, extensibility, and the staged implementation plan. --- md/SUMMARY.md | 4 + md/rfds/telemetry-recording/README.md | 427 ++++++++++++++++++ .../proposed-configuration-telemetry.md | 41 ++ .../proposed-data-collected.md | 333 ++++++++++++++ .../proposed-reference-telemetry.md | 149 ++++++ 5 files changed, 954 insertions(+) create mode 100644 md/rfds/telemetry-recording/README.md create mode 100644 md/rfds/telemetry-recording/proposed-configuration-telemetry.md create mode 100644 md/rfds/telemetry-recording/proposed-data-collected.md create mode 100644 md/rfds/telemetry-recording/proposed-reference-telemetry.md diff --git a/md/SUMMARY.md b/md/SUMMARY.md index 0ed860b4..a0681170 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -96,6 +96,10 @@ - [Discovery & sync](./rfds/registry-centric-plugins/discovery-sync/README.md) - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) + - [Telemetry: recording events](./rfds/telemetry-recording/README.md) + - [Proposed: What Symposium records](./rfds/telemetry-recording/proposed-data-collected.md) + - [Proposed: `cargo agents telemetry`](./rfds/telemetry-recording/proposed-reference-telemetry.md) + - [Proposed: Telemetry configuration](./rfds/telemetry-recording/proposed-configuration-telemetry.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/telemetry-recording/README.md b/md/rfds/telemetry-recording/README.md new file mode 100644 index 00000000..115b45c1 --- /dev/null +++ b/md/rfds/telemetry-recording/README.md @@ -0,0 +1,427 @@ +# Telemetry: recording events + +## TL;DR + +- Replace the experimental event list with a question-driven, closed schema for local telemetry. +- Keep recording off by default, per-user, local only, and gated by versioned consent. Existing unversioned opt-ins must consent again. +- Record observed sessions, configured agents, public package/extension resolution, aggregate Claude skill invocation, aggregate hook reliability, command use, and known storage gaps. +- Never record individual prompt/tool activity rows, prompt or tool details, paths, private names, dependency snapshots, or a global installation/workspace id. +- Use purpose-scoped pseudonyms and non-waiting, best-effort recording. Telemetry failure must not disrupt hooks, sync, or commands. +- Defer upload, server-side handling, and subjective feedback to separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). + +Supporting pages: [data contract and exclusions](./proposed-data-collected.md), [telemetry command reference and consent disclosure](./proposed-reference-telemetry.md), and [configuration and consent states](./proposed-configuration-telemetry.md). + +## Motivation + +Symposium has no production evidence about which integrations people reach, which public packages resolve to plugins and skills, whether Claude activates those skills, or whether hooks are slow or failing. The existing experimental telemetry was never wired into production and would record one row per prompt or tool call: high-volume activity that does not answer those questions. + +We need a low-volume, inspectable record of reach, resolution, actual skill activation, and reliability so the team can prioritize agent support, improve recommendations, and detect unacceptable hook cost. The schema must be agreed before collection begins so every field has a stated use and privacy boundary. + +This telemetry can show that a public skill resolved and Claude activated it. It cannot show that Claude followed the skill or that the skill improved the task outcome. Controlled evaluation and explicit feedback remain separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). + +## Change in a nutshell + +Recording uses purpose-shaped JSONL. A full sync emits one summary plus safe package and extension relationships; hook and Claude skill observations update bounded daily snapshots: + +```text +observed session -> session_start +full sync -> resolution_summary + -> package_resolution* + -> extension_resolution* +completed hook -> hook_metrics snapshot + -> plugin_hook_metrics snapshot +Claude Skill -> extension_invocation_metrics snapshot +command -> command +``` + +The [exhaustive data contract](./proposed-data-collected.md) owns exact fields, enums, exclusions, and complete JSONL examples. The main design has these invariants: + +- Only `Enabled` recording can create telemetry state or files. +- The producer accepts closed typed data, never arbitrary metadata or raw errors. +- Public names require allowlisted provenance and validation; everything else is unnamed or opaque. +- Pseudonyms are scoped to one measurement and normally rotate every 30 days. +- Routine hook and skill invocations update aggregates; they never become activity rows. +- A non-waiting process lock may drop a whole batch or observation but never delays another recorder. +- Files remain through D30 and first become eligible for lazy deletion on D31. +- Nothing is uploaded by this RFD. + +## Detailed plans + +### Measurement questions + +All measures describe opted-in installations, not the whole user population. Reports must state that selection bias. + + +| # | Question | Operational definition | +| --- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| Q1 | Do opted-in installations return after first observed use? | Deduplicate `session_start` by `retention_subject` and measure observed cohort days D1, D7, and D30. | +| Q2 | Which plugins and skills resolve? | Count `extension_resolution` occurrences and scoped subjects by public extension and safe witnessed path. | +| Q3 | Which public packages and versions occur, and what do they resolve? | Count `package_resolution` by public coordinate and `extension_match`; use paths carried by `extension_resolution`. | +| Q4 | Are Symposium and plugin hooks failing or slow? | Use daily invocation/attempt counters, outcomes, fixed latency histograms, and complete identified-session impact counts. | +| Q5 | Which agents, versions, and platforms have reach? | Keep configured-agent observations separate from observed hook sessions. | +| Q6 | Which command surfaces are used? | Count completed built-ins and eligible public plugin commands without arguments. | +| Q7 | Which resolved public skills does Claude actually activate? | Count completed `extension_invocation_metrics` and complete identified-session counts by public skill and safe resolution subject. | + + +Q1 measures continued installation, not continued value: session-start runs automatically once installed. Q2 proves resolution, not activation. Q7 proves activation, not that Claude followed the skill or completed the task better. Q3 records relationship edges, not a complete dependency set. Q4 counts only completed observations, so host termination can be invisible. + +### Scope and boundaries + +This RFD specifies the complete scope of local recording: consent, provenance, resolution evidence, event families, identifiers, agent capability gaps, storage, controls, rollout, documentation, and tests. + +It covers [#242](https://github.com/symposium-dev/symposium/issues/242) and the activity-metric portion of [#243](https://github.com/symposium-dev/symposium/issues/243). It does not cover #243's experimental outcome signals, [#244 uploading](https://github.com/symposium-dev/symposium/issues/244), or [#245 feedback collection](https://github.com/symposium-dev/symposium/issues/245). + +It also does not specify new host-hook timeouts, public-identity inference from arbitrary git URLs, private-source names, or a fix for concurrent full-sync mutation of installed skills. Telemetry locking protects telemetry only. + +### Vocabulary + +**Discovery** finds candidate packages, plugins, and skills before consent is known. It never records telemetry. + +**Resolution** selects the packages and extensions that actually apply. Resolution events describe the final set and the evidence that selected it. + +An **observed session** is one whose registered `SessionStart` hook ran. It is narrower than a configured agent. + +A **skill activation** is a completed, structured agent invocation of a particular installed skill. It does not mean the agent followed the skill or that the skill improved the result. + +### Consent and activation + +Telemetry remains a per-user setting: + +```toml +[telemetry] +enabled = true +consent-version = 1 +``` + + +| Effective state | Condition | Recording | +| ----------------- | ---------------------------------------- | --------------------------------------------- | +| `Disabled` | `enabled` absent or false | None; create no telemetry directory or state. | +| `ConsentRequired` | enabled but consent version absent/stale | None until current disclosure is accepted. | +| `Enabled` | enabled with current consent version | Events defined by this contract. | + + +Existing users with an unversioned `enabled = true` must consent again. Interactive `init` and `telemetry enable` present the same disclosure and default to no. Non-interactive calls require an explicit acknowledgement; editing the boolean alone cannot upgrade consent. + +The [consent version 1 disclosure in the proposed telemetry command reference](./proposed-reference-telemetry.md#enable) is authoritative and is shared by interactive `init` and `telemetry enable`. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [proposed telemetry configuration page](./proposed-configuration-telemetry.md) is authoritative for effective-state semantics. + +Adding an agent enum value creates a new schema version for each affected event kind because existing typed readers cannot parse that value. It does not by itself require renewed consent when the fields, categories, timestamp precision, and correlation boundaries remain inside the accepted disclosure. A new field, hook surface, or linkage for that agent does require a consent-version increase. + +No production event is wired until the typed catalogue, privacy contract, controls, documentation, and current consent flow all land. Earlier implementation steps remain inert behind `ConsentRequired`. + +### Event contract + +The schema is closed and typed: fixed variants, enums, counters, and bounded structures. It rejects arbitrary metadata maps, raw PM/agent payloads, errors, debug strings, command arguments, and sanitization fallbacks. Counters use checked unsigned arithmetic; overflow drops the batch or observation. + +Every row has a per-kind schema version, fixed kind, random row id, UTC day, and Symposium version. Only completed `session_start` and `command` rows carry a wall-clock timestamp, truncated to one second. Millisecond durations remain because Q4 needs mergeable distributions. + + +| Kind | Purpose and emission | +| ------------------------------ | -------------------------------------------------------------------------------------------------------------------- | +| `session_start` | A registered session-start hook completed; optional agent-scoped session id and D0-D30 return subject. | +| `agent_configuration` | Daily configured/not-configured observation for each supported agent. | +| `resolution_summary` | One completed full-sync result with public/unnamed inputs, reason counts, resolved artifacts, changes, and duration. | +| `package_resolution` | One eligible public input coordinate with `public`, `unnamed_only`, or `none` extension match. | +| `extension_resolution` | One public plugin/skill and one bounded safe path that selected it. | +| `hook_metrics` | Cumulative daily counters and latency histogram per agent, hook, and identifier epoch. | +| `plugin_hook_metrics` | Cumulative daily counters/histograms per agent, hook, epoch, and bounded plugin bucket. | +| `extension_invocation_metrics` | Cumulative daily Claude skill-attempt, completion, and failure counters per bounded public or unnamed bucket. | +| `command` | One completed eligible top-level command without arguments. | +| `storage_limit` | At most one daily marker naming the low-volume operation whose whole batch did not fit. | + + +The `session_start` event is authoritative for observed-session and return measurements (Q1 and Q5). A `hook_metrics` row whose hook is `session_start` measures only that hook surface's reliability and latency for Q4; its invocation count is not a session count. + +Ordinary hook lookup emits no resolution summary. A structured Claude `Skill` observation may update `extension_invocation_metrics` without emitting a resolution event. Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./proposed-data-collected.md) for exact fields and invariants. + +### Event producers + +Producers return typed observations or reports; they never serialize telemetry or write files. The `Recorder` checks consent, applies public-identity and correlation rules, derives identifiers, and sends an accepted event batch or aggregate update to storage. + +| Rows | Producer boundary | +| --- | --- | +| `session_start` | The outer hook wrapper, after a registered `SessionStart` completes successfully. | +| `agent_configuration` | The first non-telemetry recording-capable invocation each UTC day, as one all-agent snapshot derived from Symposium's per-user agent list. | +| `resolution_summary`, `package_resolution`, `extension_resolution` | One structured full-sync report assembled while resolving and installing; telemetry does not rerun predicates or rediscover relationships. | +| `hook_metrics` | The outer hook wrapper, after the hook's final outcome and duration are known. | +| `plugin_hook_metrics` | The plugin dispatcher, around each applicable plugin's preparation and execution. | +| `extension_invocation_metrics` | The Claude `Skill` adapter normalizes targeted pre/success/failure hooks, then the generated installation index supplies safe attribution. | +| `command` | The top-level CLI dispatcher, after an eligible built-in or public plugin command reaches an outcome. | +| `storage_limit` | The JSONL sink itself, when a complete low-volume batch cannot fit. | + +Agent and package-manager payloads therefore provide input to existing Symposium operations; they do not emit arbitrary telemetry. If a producer cannot construct its complete typed observation, it records nothing for that observation. + +### Identifiers and correlation boundaries + +On first enabled recording, telemetry atomically creates a random 32-byte identity key in `state.toml`. It derives the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension: + +```text +HMAC(key, "telemetry::v1\0" || window || "\0" || dimension) +``` + +The dimension is included so each identifier represents one installation for one package, agent, or command dimension, never the installation globally. + + +| Identifier | Scope | +| ------------------- | ------------------------------------------------------------------------------------------------ | +| `event_id` | One row; random rather than derived. | +| `session_id` | Agent + vendor session id + 30-day window; optional. | +| `retention_subject` | One observed-session D0-D30 cohort; `session_start` only. | +| `agent_subject` | One agent + 30-day window. | +| `package_subject` | One safe public package coordinate + 30-day window. | +| `extension_subject` | One safe target/path + 30-day window; shared by its resolution and public invocation aggregates. | +| `hook_subject` | One agent/hook surface + 30-day window. | +| `plugin_subject` | One safe public plugin + 30-day window. | +| `command_subject` | One safe command coordinate + 30-day window. | + + +The return subject is the sole cross-agent exception: it deduplicates Q1 but cannot link to other event kinds. A cohort remains stable through D30; the next observed session starts a new cohort. Accepting new consent or resetting identifiers rotates the key and starts another cohort. + +`session_id` is absent when the agent supplies none, including Copilot. Raw vendor ids never enter events or an unkeyed hash. There is no global installation/workspace id, and future analysis or upload must not reconstruct one. Missing identity state is created only when enabled; malformed existing state stops recording until explicit identifier reset. + +### Public package and extension identity + +Each package manager must return typed provenance with the resolved coordinate: registry URL, git, path, workspace, or unknown. Core, rather than the PM, maps allowlisted public registries to stable ecosystem labels. Raw URLs and provenance never enter an event; non-allowlisted sources are private/unnamed by default. + +A named package requires allowlisted provenance plus a valid public name and exact resolved version. Other inputs increment `unnamed_packages` and exactly one of `private_registry`, `git`, `path`, `workspace`, `unknown_source`, or `invalid_coordinate`. Source reasons take precedence; `invalid_coordinate` applies only to an otherwise public source with an invalid coordinate. Exact versions let the team isolate version-specific recommendation gaps. + +`package_resolution.extension_match` distinguishes: + +- `public`: at least one eligible public extension matched; +- `unnamed_only`: extension content matched, but none was safe to name; +- `none`: no resolved extension matched. + +Plugin, skill, and command names follow the same public-by-allowlist rule. This requires extending the current PM representation, which loses source provenance after loading metadata. + +### Complete safe resolution paths + +`extension_resolution` records the actual successful package-to-plugin-to-skill path. Predicate evaluation produces evidence in its original pass; telemetry never re-evaluates a predicate. + +Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason (`private_source`, `non_package_predicate`, or `limit`). Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path. + +Paths are bounded to 8 levels, 16 evidence leaves, and 4 KiB; an over-limit subtree becomes `opaque: limit`. Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled; only an enabled recorder serializes that evidence as telemetry. Existing cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`, but caching an entire `PredicateSet` would lose successful branches and witnesses and requires revisiting this design. + +This preserves actionable resolution relationships without recording a complete dependency snapshot. It proves that an extension resolved. A matching `extension_invocation_metrics` row separately proves that Claude activated the installed skill. + +### Hook aggregation and agent capability + +Each completed hook merges into one daily `hook_metrics` row per agent, surface, and active identifier epoch. Per-plugin preparation/execution merges into `plugin_hook_metrics`. Resets can create another epoch row on the same day. Fixed histogram bounds are `[5, 10, 25, 50, 100, 250, 500, 1000]` milliseconds so rows remain mergeable. + +Outcome counters and histograms have exact sum invariants defined in the data contract. Identified-session counts are all-or-nothing: every observation must supply an id, and each keyed set is limited to 256. On a missing id, overflow, or state/snapshot mismatch, sets are discarded and the row remains incomplete for that day rather than publishing a plausible partial count. + +Private plugins merge into unnamed buckets. At most 128 public-plugin rows are named per UTC day across agents, hooks, and identifier epochs; later names merge into overflow buckets. Extension invocation metrics have a separate 128-public-skill daily limit. Each named subset is first-observed, so earlier-in-day plugins or skills are overrepresented when its limit is reached. Analysis must report overflow and must not treat named rows as a random sample. The combined aggregate snapshot may occupy at most 512 KiB of the shared daily allowance. + +Hook rows disclose exact daily counts. `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` therefore approximate daily tool/prompt activity even without individual records. Extension invocation rows disclose exact daily skill-attempt, completion, and failure counts. The disclosure states both explicitly. + +This matrix reports current adapter capability and test coverage, not telemetry priority. The producer contract is agent-neutral; unavailable values remain optional or unsupported as shown. + + +| Agent | Configuration | `SessionStart` | Session id | Fresh/resume | `Stop` | Skill invocation | +| -------------- | ------------- | -------------- | ---------- | ------------ | ------ | --------------------------- | +| Claude Code | yes | yes | yes | yes | yes | attempted/completed/failed | +| Codex CLI | yes | yes | yes | yes | no | unsupported | +| GitHub Copilot | yes | yes | no | no | no | unsupported | +| Gemini CLI | yes | yes | yes | no | no | unsupported | +| Kiro | yes | yes | yes | no | no | unsupported | +| OpenCode | yes | no | n/a | n/a | no | unsupported | +| Goose | yes | no | n/a | n/a | no | unsupported | + + +Configured reach covers all seven agents; observed-session measures cover only registered hook integrations. `Stop` and structured skill invocation remain Claude-only in version 1. An unsupported agent produces no invocation row, which means unknown rather than zero. `Stop` is not required by Q1-Q7. + +### Skill invocation attribution + +Agent-specific parsing ends at a normalized extension-use observation. The Claude adapter accepts only a fixture-tested `Skill` tool shape and produces a skill identifier plus one phase: `attempted`, `completed`, or `failed`. Raw Claude input and ephemeral invocation ids do not enter the recorder. + +Full sync writes a versioned installation index under the agent skills parent at `.symposium/index-v1.json`. This generated, gitignored installation state is separate from configuration and telemetry. It maps the actual agent-facing identifier to the installed directory, marker fingerprint, eligibility, and safe public coordinate/path when one exists. Private identifiers may remain in this local index because they already exist in installed skill content; they are never serialized as telemetry. + +The index is atomically replaced after sync determines which installations succeeded. Before naming a public invocation, lookup verifies the Symposium marker and fingerprint. A hook sees an old or new complete index; a missing, corrupt, or stale mapping never falls back to a path or content guess. + +Public matches reuse the `extension_subject` derived for the selected safe resolution path. Other observations merge into `unnamed` rows with one fixed reason: `ineligible`, `not_discovered`, `attribution_unavailable`, `ambiguous`, or `invalid_signal`. After 128 named public-skill rows in one UTC day, further public matches merge into `overflow`. + +Claude's existing `PreToolUse` and `PostToolUse` hooks increment attempts and completions. A `PostToolUseFailure` registration matched only to `Skill` increments failures and does not create generic failure-surface metrics. Each phase update is an independent lower bound: loss of one update means completed plus failed need not equal attempted and may exceed it. + +The same all-or-nothing 256-id rule applies to attempted and completed distinct-session sets. A missing id, overflow, or state/snapshot mismatch makes both counts incomplete for that row and day. Completion proves activation only; it does not show whether Claude followed the instructions or improved the result. + +### Recording architecture + + +| Unit | Responsibility | +| ------------------------- | --------------------------------------------------------------------------------- | +| `Recorder` | Enforce consent, buffer typed batches, and isolate recording failures. | +| `IdentityDeriver` | Own identity state and derive scoped pseudonyms. | +| `PublicIdentityPolicy` | Convert PM provenance into safe coordinates or unnamed counts. | +| `ResolutionWitness` | Carry evidence through one-pass resolution. | +| `InstalledExtensionIndex` | Persist actual agent-facing skill attribution after successful installation. | +| `ObservationRouter` | Normalize agent-native extension-use signals and send them only to active sinks. | +| `MetricAggregator` | Merge hook, plugin-hook, and extension-invocation observations into bounded rows. | +| `JsonlSink` | Lock, cap, retain, append/replace, inspect, and clear files. | + + +Only an enabled `Recorder` owns telemetry sink and identity components; call sites cannot write directly. The generated installation index is functional sync state and exists independently of telemetry consent. A sync returns a structured report with provenance and witnesses, which installs skills, atomically replaces that index, and lets the recorder sanitize one buffered relationship batch when enabled. + +A hook prepares its agent response, then converts timings/outcomes into aggregate updates where possible. The observation router does not load the index or construct an extension-use observation when telemetry is disabled and no other sink is active. Commands are measured once at top-level dispatch. Raw errors never enter telemetry. + +### Storage, concurrency, and retention + +Low-volume rows append to `events-YYYY-MM-DD.jsonl`. Current daily hook, plugin-hook, and extension-invocation aggregates live in a bounded, atomically replaced `metrics-YYYY-MM-DD.jsonl` snapshot. `state.toml` holds the identity key, cohort/cleanup metadata, marker state, and temporary keyed session-count sets; these sets are never emitted and expire at day rollover. + +Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement; no `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state/snapshot divergence and permanently mark affected daily session counts incomplete. + +The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events; an ordinary batch that cannot fit is replaced by the daily marker and ordinary recording stops for that day. Relationship batches are never split. + +Aggregate counters are lower bounds. No durable counter can quantify observations lost to lock contention, process termination, or I/O failure because those conditions can also prevent writing the counter; a cap-only counter would not measure total loss. + +Files survive D30 and become eligible for lazy deletion when `current_day - file_day > 30`, first on D31. `clear` deletes event/metric files and pending count sets but preserves consent and identity/cohort state. `reset-identifiers` rotates future identifiers without rewriting old files. `disable` stops recording but keeps files by default; uninstall also leaves them. Exact command behavior belongs to the [telemetry CLI reference](./proposed-reference-telemetry.md). + +Concurrent agents produce separate session and agent/hook rows. The same public package/path derives the same dimension subject for unique-install deduplication, but no project id links the agents. A simultaneous flush may be dropped; concurrent mutation of installed skills remains a separate problem. + +### Schema evolution and limitations + +Schema versions are per event kind. Semantic or correlation changes create a new version; privacy expansions also require a new consent version. Readers retain malformed and unknown lines, count them separately, and exclude them from typed analysis; raw `show` preserves their bytes. + +The ["What is never recorded" section of the proposed data contract](./proposed-data-collected.md#what-is-never-recorded) is a producer rule. In summary: no prompt/tool content or per-invocation rows, raw errors/payloads, paths or workspace identity, environment/machine/account values, private-source names, arbitrary URLs, global identifiers, or timestamps finer than one second. + +Here, the per-invocation exclusion includes individual skill activations and raw agent-facing skill identifiers. + +Scoped pseudonyms do not make a local directory anonymous. File/day/order and one buffered sync can expose co-occurrence; unusual public versions, public skill-use counts, agent/platform combinations, and exact counts can fingerprint an installation. Missing data is also non-random: busy multi-agent sessions contend more, terminated hooks lose final observations, and unsupported agents have configuration but not session or skill-invocation observations. Reports must state these limitations. + +### Extensibility + +The event catalogue is closed for consent version 1, but new measurements can be added as typed, versioned event families. Every family goes through `Recorder` and remains subject to consent, public-identity rules, storage caps, retention, failure behavior, and the never-record exclusions. + +Agent-native extension-use payloads remain behind the normalized observation boundary. Version 1 implements only the Claude `Skill` adapter. Supporting another agent requires a thin adapter, native-signal conformance fixtures, capability documentation, and an event schema update; it does not duplicate attribution, aggregation, or storage. + +The same model can later support eligible public plugins. A plugin would declare bounded feature names in its manifest and report only core-defined aggregates. Symposium would own and validate the schema, record and use the resulting data, and optionally provide aggregate reports to plugin authors. Plugins could not add arbitrary fields, bypass consent, or write directly to storage. This RFD implements only the event families listed above; plugin reporting requires a separate implementation and disclosure before collection begins. + +### Boundary with controlled evaluation + +Production telemetry and a future evaluation harness may consume the same normalized extension-use observation but must use separate sinks and contracts. The telemetry sink writes only the daily aggregates defined here. An explicitly started harness may retain detailed per-run evidence in its isolated evaluation workspace without reading telemetry JSONL or relying on rotating telemetry identifiers. + +This RFD preserves that internal seam but does not define a harness, scenario format, trace format, outcome grader, or token accounting. Controlled with/without comparisons answer whether a skill improved a task; production telemetry does not. + +### Boundary with future upload + +Accepting this RFD is not consent to upload. A future RFD must define transport/authentication, renewed consent and scheduling, retry/idempotency, server retention/access/deletion, reporting thresholds, and incident handling. + +Upload may use only accepted local fields and must preserve scoped-correlation boundaries. It cannot create a global subject from identifiers, file order, batch membership, request grouping, or transport metadata. `event_id` and per-kind versions enable retry and mixed-version handling; they pre-approve no transport. + +### Proposed documentation + +- [What Symposium records](./proposed-data-collected.md): normative fields, enums, examples, and exclusions. +- [`cargo agents telemetry`](./proposed-reference-telemetry.md): controls, files, inspection, and concurrency. +- [Telemetry configuration](./proposed-configuration-telemetry.md): consent and effective-state semantics. + +These remain proposed pages until implementation lands; shipped design/reference chapters continue to describe the current binary. + +## Frequently asked questions + +### What does pseudonymous mean here? + +A scoped identifier is derived from a random local secret rather than machine identity, but it still links observations inside one stated purpose and window. Rotation and domain separation limit that linkage; they do not make the local files anonymous. + +### Why retain scoped identifiers, provenance, and witnesses before upload? + +They already serve local measurement and privacy. Q1-Q7 need narrow deduplication and relationship evidence; provenance prevents private coordinates from reaching an inspectable/shareable telemetry directory. Recording plain coordinates now would create weaker local files and permanently lose the evidence needed by those questions. `event_id` also gives cumulative metric rows stable identity across snapshot replacement and reset epochs. + +### Why include exact package versions? + +A package name alone cannot distinguish a missing recommendation from a version-specific resolution gap or regression. Exact versions provide that diagnostic only after the PM proves an allowlisted public origin and validates the coordinate; private and local versions remain unnamed. + +### Why aggregate hooks but keep resolution events? + +Hooks are high-volume; Q4 needs rates and distributions, not traces. Resolution edges are low-volume and Q2/Q3 need the actual safe package-to-plugin-to-skill relationship. Aggregating those edges would erase the product signal. + +### Does a completed skill activation mean the skill helped? + +No. It means Claude successfully activated the installed skill. It does not show whether Claude followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill. + +### Why not record a complete dependency snapshot? + +Per-package frequency and safe extension paths answer Q3 without an explicit project fingerprint. Same-file/day/batch order can still reveal co-occurrence locally, but a dependency-set or resolution id would make that linkage direct and persistent. + +### Does recording delay or disrupt hooks? + +It performs bounded in-process work and one non-waiting lock attempt, so it does not promise zero latency. It never waits for another recorder, changes the user operation's result, or `fsync`s. Disabled consent, contention, caps, I/O failure, or termination can omit data; absence never proves that an action did not occur. + +### Why JSONL instead of SQLite? + +Low-volume events append, while hook metrics form a small bounded daily snapshot. JSONL keeps both byte-inspectable, preserves unknown versions and malformed lines, and needs no database. The process lock plus atomic snapshot replacement supplies the required concurrency behavior. + +## Implementation plan and status + +The seven steps below are PR-sized. Steps 1-6 keep production collection inert; Step 7 activates the current consent version only after every producer and control is present. Tests and benchmarks may construct an enabled recorder only through a test-only API bound to a temporary telemetry home. There is no runtime environment-variable or configuration bypass. + +```text +1. Contract and identity + | +2. Storage and controls + / \ +3. PM provenance 6. Reach/commands + | +4. Resolution witnesses/index/events + | +5. Hook and extension-use metrics + \ / + 7. Consent and activation +``` + +After Step 2, Steps 3 and 6 may proceed in parallel. Step 4 follows Step 3, Step 5 follows Step 4, and Step 7 waits for Steps 5 and 6. Step 3 isolates the risky core PM type seam from telemetry instrumentation. Steps 4 and 5 remain end-to-end measurement slices, and the plan remains seven PRs. + +### Step 1: Telemetry contract and identity + +Replace the dormant event types with the closed producer schema: common fields, bounded witnesses, fixed outcomes/histograms, extension-invocation counters and buckets, and per-kind version dispatch. Add telemetry-owned `state.toml`, atomic key creation, scoped HMAC derivation, local 30-day windows, D0-D30 cohorts, and reset primitives. Do not add storage or emission. + +Verify all contract examples round-trip; bounds and unknown versions behave as specified; arbitrary/private data cannot serialize; identities separate domains, dimensions, agents, and windows; D30 rollover/reset works; and disabled paths create nothing. + +- [ ] PR: telemetry contract and scoped identity + +### Step 2: Storage and local controls + +Add whole-batch event appends, non-waiting process locking, atomically replaced aggregate snapshots, daily caps/reservations, `storage_limit`, and lazy D31 cleanup. Add typed `status`, byte-preserving `show`, `clear`, and `reset-identifiers`. Expose a test-only enabled recorder bound to a caller-supplied temporary telemetry home for integration tests and benchmarks. No production caller writes through the sink yet, and no runtime bypass is added. + +Verify concurrent complete lines, old-or-new snapshots, whole-operation drops, cap/marker accounting, D30/D31 cleanup, malformed/unknown inspection, clear/reset semantics, test-only recorder isolation, and that management commands never record themselves. + +- [ ] PR: telemetry storage and local controls + +### Step 3: PM provenance and public-identity policy + +Extend every PM result with typed provenance and exact coordinates; add core-owned public allowlists and coordinate validation. If an out-of-process PM protocol lands first, carry provenance through it as part of this step. All supported PMs must provide provenance before events are wired. This step emits no telemetry. + +Verify every PM and source class, malformed and wildcard coordinates, public allowlist behavior, unnamed-reason precedence, and that raw URLs and private/local coordinates cannot become public identities. + +- [ ] PR: package provenance and public identity policy + +### Step 4: Resolution witnesses, attribution index, and recording + +Return safe evidence from the original predicate evaluation, preserving short-circuit and accepted cache behavior; whole-`PredicateSet` caching remains disallowed. Extend evidence through plugin/skill selection, return one structured full-sync report, and construct coherent `resolution_summary`, `package_resolution`, and `extension_resolution` batches. After installation, atomically replace the generated agent-facing attribution index with entries for successful managed skills and marker fingerprints. Ordinary read-only plugin lookup remains silent. + +Verify `all`/`any`/`not` and opaque/limit paths, cache hits without reevaluation, every `extension_match` case, complete package-to-plugin-to-skill paths, whole-batch failure, successful/failed installation indexing, atomic old-or-new index reads, stale/corrupt/fingerprint-mismatch handling, collisions, and exclusion of raw paths, private names, and dependency snapshots from telemetry. + +- [ ] PR: resolution witnesses, installed attribution, and recording + +### Step 5: Hook and extension-invocation telemetry + +Measure completed Symposium and plugin-hook handling. Merge top-level observations into `hook_metrics` and plugin preparation/execution into `plugin_hook_metrics`, with fixed outcomes/histograms, scoped subjects, public/unnamed/overflow buckets, the 128-named-row limit, and all-or-nothing identified-session counts. + +Add the sink-neutral extension-use observation and Claude `Skill` adapter. Existing pre/post-tool hooks and a targeted failure hook update `extension_invocation_metrics` through the installed attribution index, with independent phase counters, fixed unnamed reasons, a separate 128-public-skill limit, and all-or-nothing attempted/completed session counts. Never emit a per-invocation row. + +Measure the hook path before and after. Verify sanitized automatic/manual activation and lifecycle fixtures, unknown schemas, public/private/missing/ambiguous/stale attribution, marker validation, raw-input exclusion, all counter/histogram/cross-row invariants, independently dropped phases, boundary buckets, process merging, rollover/reset, 500-lifecycle boundedness, private/overflow behavior, missing/mixed/over-256 session ids, crash recovery, a worst-case snapshot within 512 KiB, integer overflow drops, a fake second sink, disabled-path short-circuiting, and unchanged agent output on recording failure. + +- [ ] PR: bounded hook and extension-invocation telemetry + +### Step 6: Session, configuration, and command telemetry + +Add capability-aware `session_start`, daily `agent_configuration`, and eligible top-level `command` rows over the shared recorder. The first non-telemetry recording-capable invocation of each UTC day attempts one all-agent configuration batch from the per-user agent list; it does not inspect agent-owned files. Exclude hook internals, telemetry controls, arguments, and unsafe plugin command names. + +Verify the capability matrix, Copilot without a session id, Claude-only `Stop`, configuration-list semantics, same-day configuration deduplication, retry after a dropped configuration batch, fixed command vocabulary/public eligibility, and failures before or after command dispatch. + +- [ ] PR: session reach and command telemetry + +### Step 7: Consent, activation, and documentation + +Add `consent-version`, one shared literal disclosure for `init` and `telemetry enable`, re-consent, explicit non-interactive acknowledgement, `enable`/`disable`, and final production wiring for Steps 4-6. The disclosure names exact daily skill activation and unnamed counts. There is no event-file migration because the dormant recorder was never called. Publish the proposed pages and update current design/flow chapters and `md/SUMMARY.md`. + +Verify new/existing configuration states, a snapshot of the identical versioned disclosure used by `init` and `telemetry enable`, interactive/non-interactive flows, no partial or disabled collection, absence of a runtime consent bypass, raw inspection/expiry, the full CLI/integration suite, hook-path benchmark, formatting, clippy, workspace tests, mdBook, and orphan checks. + +- [ ] PR: telemetry consent and recording activation diff --git a/md/rfds/telemetry-recording/proposed-configuration-telemetry.md b/md/rfds/telemetry-recording/proposed-configuration-telemetry.md new file mode 100644 index 00000000..4cd76a98 --- /dev/null +++ b/md/rfds/telemetry-recording/proposed-configuration-telemetry.md @@ -0,0 +1,41 @@ +# Telemetry configuration + +Telemetry consent is a per-user setting in `~/.symposium/config.toml`. Symposium reads it only from the user configuration file; project configuration cannot enable, disable, or grant consent for telemetry. + +```toml +[telemetry] +enabled = true +consent-version = 1 +``` + + +| Key | Type | Default | Meaning | +| ----------------- | ---------------- | ------- | --------------------------------------------------- | +| `enabled` | boolean | `false` | The user's stored choice to permit local recording. | +| `consent-version` | positive integer | absent | Disclosure version the user acknowledged. | + + +Both values must permit collection. For a binary whose current disclosure is +version 1: + + +| Configuration | Effective state | +| -------------------------------------------- | -------------------------------- | +| absent section or `enabled = false` | Disabled | +| `enabled = true` with absent/non-`1` version | Consent required; record nothing | +| `enabled = true` and `consent-version = 1` | Enabled | + + +The current version is owned by the binary, not accepted as an arbitrary configuration value. The [consent version 1 disclosure in the proposed telemetry command reference](./proposed-reference-telemetry.md#enable) is authoritative; interactive `init` presents the same text. + +A future release may require a higher version after a change to collected categories, linkability, timestamp precision, public-name eligibility, retention, or a normative exclusion. It records nothing under an older acknowledgement until you consent again. + +Adding an agent enum value creates a new schema version for each affected event kind so typed readers do not reinterpret old schemas. It does not by itself require renewed consent when the recorded fields, categories, timestamp precision, and correlation boundaries remain unchanged. New fields, hook surfaces, or linkage for that agent do require a higher consent version. + +Accepting a newer consent version rotates the telemetry identity key and starts a new D0-D30 return cohort. Existing event and aggregate-metric files are neither rewritten nor deleted, but their scoped identifiers cannot link to later rows. + +`cargo agents telemetry enable` presents the disclosure for the current version before writing these values. Interactive `cargo agents init` presents the same disclosure for new users and existing unversioned opt-ins, defaulting to no. Non-interactive `init` does not grant or upgrade consent. + +`cargo agents telemetry disable` sets `enabled = false` without deleting existing telemetry data files. Consent configuration is separate from the random identity key in `~/.symposium/telemetry/state.toml`. The state file is not configuration, and Symposium never reads it from project configuration. + +Symposium-managed project skills may also have a generated `/.symposium/index-v1.json` installation index. It associates the agent-facing skill identifier with the installation Discovery selected. This file is gitignored installation state, not project configuration or telemetry; changing it cannot grant consent, and telemetry commands do not inspect or delete it. diff --git a/md/rfds/telemetry-recording/proposed-data-collected.md b/md/rfds/telemetry-recording/proposed-data-collected.md new file mode 100644 index 00000000..626dffaa --- /dev/null +++ b/md/rfds/telemetry-recording/proposed-data-collected.md @@ -0,0 +1,333 @@ +# What Symposium records + +Telemetry is off by default, per-user, and stored only on your machine. Nothing described on this page is uploaded. `cargo agents telemetry show` exposes the exact stored bytes, and `cargo agents telemetry clear` deletes event and aggregate-metric files. The [never-record list](#what-is-never-recorded) summarizes the exclusions before the field-by-field contract. + +This page is the exhaustive producer contract. If a field is not listed here, Symposium does not write it as telemetry under consent version 1. + +## Common fields + +Every JSONL row has: + + +| Field | Example | Meaning | +| ----------- | --------------- | --------------------------------------- | +| `v` | `1` | Schema version for this row kind. | +| `kind` | `session_start` | Row kind from the list below. | +| `event_id` | `9f2c41b6-...` | Identifier for this row; random when minted. | +| `day` | `2026-08-03` | UTC calendar day. | +| `symposium` | `0.4.0` | Symposium version that wrote the event. | + + +Completed operational events (`session_start` and `command`) also have `at`, an RFC3339 UTC timestamp truncated to one second. Resolution, configuration, and aggregate metric rows have only `day`. Counters and durations are non-negative JSON integers fitting an unsigned 64-bit value. Arithmetic is checked; an overflowing batch or observation is dropped rather than wrapped. + +`event_id` exists to deduplicate a future retry of the same event or identify one cumulative metric row. A metric row keeps the `event_id` minted when its dimension first appears that UTC day as the row is rewritten. It is not an installation, session, account, or project identifier. + +## Example JSONL for every row kind + +These are independent row-shape examples, not one coherent operation or batch. The identifiers and coordinates are illustrative. An actual day may contain repeated low-volume events and one cumulative row per aggregate-metric dimension. + +```jsonl +{"v":1,"kind":"session_start","event_id":"9f2c41b6-495e-4c88-a22b-c597f8102aed","day":"2026-08-03","at":"2026-08-03T09:14:02Z","symposium":"0.4.0","agent":"claude","os":"linux","arch":"x86_64","start":"fresh","session_id":"sess_31d8b1916028f65a0c0521dc1f4c86fb","retention_subject":"ret_74ddf26f80ad8b58de7f03e6c632e654","cohort_day":0} +{"v":1,"kind":"agent_configuration","event_id":"1a77f7c8-2c45-4de2-8cc6-b507ac3605f8","day":"2026-08-03","symposium":"0.4.0","agent":"claude","configured":true,"os":"linux","arch":"x86_64","agent_subject":"agt_9255770e1679cb789796a9f9e86325c5"} +{"v":1,"kind":"resolution_summary","event_id":"6c3ef7a1-f782-4b47-9a25-829dd64e1ba2","day":"2026-08-03","symposium":"0.4.0","trigger":"session_start","outcome":"ok","duration_ms":142,"public_packages":2,"unnamed_packages":1,"unnamed_package_reasons":{"private_registry":1,"git":0,"path":0,"workspace":0,"unknown_source":0,"invalid_coordinate":0},"plugins":1,"skills":1,"installed":2,"updated":0,"reaped":0,"session_id":"sess_31d8b1916028f65a0c0521dc1f4c86fb"} +{"v":1,"kind":"package_resolution","event_id":"c03e7390-52b1-4e11-b7e7-9573c84e555a","day":"2026-08-03","symposium":"0.4.0","package":{"ecosystem":"cargo","name":"example-runtime","version":"1.2.3"},"extension_match":"public","package_subject":"pkg_f6db813c87209816ae4896f3e60dd774"} +{"v":1,"kind":"extension_resolution","event_id":"fa21a1d7-64a7-4e3e-9f6a-4572cf8f1939","day":"2026-08-03","symposium":"0.4.0","target":{"type":"skill","source":"symposium-recommendations","name":"example-debugging"},"path":[{"type":"package","ecosystem":"cargo","name":"example-runtime","version":"1.2.3"},{"type":"extension","extension_type":"plugin","source":"symposium-recommendations","name":"example-tools"},{"type":"extension","extension_type":"skill","source":"symposium-recommendations","name":"example-debugging"}],"extension_subject":"ext_6e68a75b9701bbad86cf32cd876e994a"} +{"v":1,"kind":"hook_metrics","event_id":"b563dd02-0301-4e2c-aac4-2e0d5dfaa977","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","invocations":500,"outcomes":{"ok":498,"blocked":1,"plugin_error":1,"internal_error":0},"plugins_attempted":500,"plugins_completed":500,"duration_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[40,80,180,130,55,12,2,1,0]},"identified_sessions":1,"identified_sessions_non_ok":1,"session_counts_complete":true,"hook_subject":"hok_51f4f4143ce32704e96086899dc66a27"} +{"v":1,"kind":"plugin_hook_metrics","event_id":"70dcad2a-fd19-4b5d-97cb-a7c7b52a81f1","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","plugin_scope":"public","plugin":{"source":"symposium-recommendations","name":"example-tools"},"attempts":500,"executions":500,"outcomes":{"ok":499,"blocked":0,"error":1},"prepare_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[400,90,10,0,0,0,0,0,0]},"execute_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[10,50,200,180,50,8,2,0,0]},"identified_sessions":1,"identified_sessions_non_ok":1,"session_counts_complete":true,"plugin_subject":"plg_d42e3f1fbac5bb001e1258a1e70b0d77"} +{"v":1,"kind":"command","event_id":"5d18caa8-84f7-4aa3-846c-99ea810ccd85","day":"2026-08-03","at":"2026-08-03T10:02:11Z","symposium":"0.4.0","command":{"type":"builtin","name":"use"},"duration_ms":820,"outcome":"ok","command_subject":"cmd_adf0c14ddc35b97762b5daae6f4119ce"} +{"v":1,"kind":"storage_limit","event_id":"8430f7f3-7ec5-4ca5-9c65-8f6e83eaa3de","day":"2026-08-03","symposium":"0.4.0","dropped_operation":"manual_sync"} +{"v":1,"kind":"extension_invocation_metrics","event_id":"7b3bda33-1fd7-4657-9637-4057268370dc","day":"2026-08-03","symposium":"0.4.0","agent":"claude","target_scope":"public","target":{"type":"skill","source":"symposium-recommendations","name":"example-debugging"},"attempted":14,"completed":12,"failed":2,"session_counts_complete":true,"identified_sessions":4,"identified_sessions_completed":3,"extension_subject":"ext_6e68a75b9701bbad86cf32cd876e994a"} +``` + +## Scoped identifiers + +Symposium stores a random secret key in `~/.symposium/telemetry/state.toml` while telemetry is enabled. It derives keyed identifiers for narrow purposes. The key is not written into events, printed by telemetry commands, or derived from your machine. + + +| Identifier | What it can link | Rotation | +| ------------------- | -------------------------------------------------------------- | --------------------------------- | +| `session_id` | Events carrying the same vendor session id for one agent | 30 days; omitted when unavailable | +| `retention_subject` | Observed session days in one D0-D30 cohort | After cohort day 30 | +| `agent_subject` | Repeated configuration observations for one agent | 30 days | +| `package_subject` | Repeated observations of one public package/version | 30 days | +| `extension_subject` | Resolution and public invocation aggregates for one safe path | 30 days | +| `hook_subject` | Daily hook aggregates for one agent and hook surface | 30 days | +| `plugin_subject` | Daily hook aggregates for one eligible public plugin | 30 days | +| `command_subject` | Repeated use of one eligible command | 30 days | + + +These values are pseudonymous, not anonymous: they deliberately permit limited linking inside the stated scope. `retention_subject` can link observed sessions across agents for D0-D30; this is the single exception needed for return measurement. No identifier links that cohort, a package subject, and a command subject, and there is no workspace id. However, all lines in your local telemetry directory come from your Symposium home; file order and same-day events can still suggest which observations happened together. + +## Version 1 public identity allowlists + +Only the following stable labels can make package, plugin, skill, or plugin-command names eligible for recording under consent version 1: + +| Enum | Version 1 values | Used by | +| --- | --- | --- | +| Package ecosystem | `cargo` | `package.ecosystem` and package path nodes. | +| Public extension source | `symposium-recommendations`, `crates-io` | Resolution/invocation `target.source`, extension path nodes, `plugin.source`, and plugin-command `source`. | + +`crates-io` applies only when core proves allowlisted crates.io provenance; `symposium-recommendations` identifies the built-in registry. `user-plugins`, configured registries, paths, workspaces, and arbitrary git sources remain unnamed. Raw registry names and URLs are never enum values. Adding an ecosystem or public-source label expands name eligibility and requires a new consent version. + +## Event kinds + +### `session_start` + +A registered Symposium session-start hook completed. + + +| Field | Values | Meaning | +| ------------------- | ---------------------------------------------- | ------------------------------------------------------- | +| `at` | UTC second | When Symposium completed the session-start handling. | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro` | Agent that invoked the registered hook. | +| `os` | `linux`, `macos`, `windows`, `other` | OS class for the running Symposium build. | +| `arch` | `x86_64`, `aarch64`, `other` | Architecture class for the running build. | +| `start` | `fresh`, `resumed`, `unknown` | Agent-supplied lifecycle classification when available. | +| `session_id` | scoped id, optional | Omitted when the agent supplies no session id. | +| `retention_subject` | scoped id | Deduplicates this D0-D30 observed-session cohort. | +| `cohort_day` | integer `0` through `30` | UTC days since the cohort's first observed session. | + + +GitHub Copilot does not currently supply a session id. OpenCode and Goose do not currently call Symposium through a registered session-start hook, so they do not produce this event. + +These rows, not `hook_metrics` rows whose `hook` is `session_start`, are authoritative for observed-session and return measurements. The aggregate hook rows measure only session-start hook reliability and latency. + +### `agent_configuration` + +A daily observation of whether a supported agent is configured for Symposium. + + +| Field | Values | Meaning | +| --------------- | ------------------------------------------------------------------- | -------------------------------------------------------- | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro`, `opencode`, `goose` | Agent being checked. | +| `configured` | boolean | Whether the agent is listed in per-user Symposium configuration. | +| `os` | `linux`, `macos`, `windows`, `other` | OS class for the running Symposium build. | +| `arch` | `x86_64`, `aarch64`, `other` | Architecture class for the running build. | +| `agent_subject` | scoped id | Deduplicates one installation for this agent and window. | + + +On the first non-telemetry recording-capable invocation each UTC day, Symposium attempts one whole batch containing all seven agents. `configured` means the agent is present in Symposium's per-user configuration at that moment; Symposium does not inspect agent-owned configuration files. A successful batch is the day's snapshot, so later configuration changes appear on the next UTC day's snapshot. If locking, storage, or I/O drops the batch, it remains outstanding and a later eligible invocation retries it. + +`configured: true` does not claim that an agent session ran or that the agent-owned integration files remain intact. + +### `resolution_summary` + +A full sync reached an observed result after session start, manual sync, `use`, or removal. Package/plugin/skill counts are distinct coordinates within the sync, not duplicate declarations. + + +| Field | Values | Meaning | +| ------------------ | ----------------------------------------------- | ----------------------------------------------------------------- | +| `trigger` | `session_start`, `manual_sync`, `use`, `remove` | Why full sync ran. | +| `outcome` | `ok`, `partial`, `error` | Closed result classification. | +| `duration_ms` | integer | Full resolution/sync duration. | +| `public_packages` | integer | Eligible public resolution-input packages. | +| `unnamed_packages` | integer | Private, local, unknown, invalid, or otherwise ineligible inputs. | +| `unnamed_package_reasons` | fixed counters | Mutually exclusive reasons that sum to `unnamed_packages`. | +| `plugins` | integer | Plugins in the final resolved set. | +| `skills` | integer | Skills in the final resolved set. | +| `installed` | integer | Artifacts newly installed by this sync. | +| `updated` | integer | Existing artifacts changed by this sync. | +| `reaped` | integer | Obsolete artifacts removed by this sync. | +| `session_id` | scoped id, optional | Present only for a sync inside an identified agent session. | + +`unnamed_package_reasons` has exactly `private_registry`, `git`, `path`, `workspace`, `unknown_source`, and `invalid_coordinate` counters. Each unnamed package increments exactly one. Source provenance takes precedence; `invalid_coordinate` is used only for an allowlisted public source with a malformed name or a missing, wildcard, or invalid exact version. The counters sum to `unnamed_packages`. + + +Read-only extension lookup on ordinary hook calls does not produce this event. + +### `package_resolution` + +One eligible public resolution-input package observed during a full sync. + + +| Field | Values | Meaning | +| ------------------- | ----------------------- | -------------------------------------------------------------------- | +| `package.ecosystem` | `cargo` | Stable public ecosystem label. | +| `package.name` | validated string | Public package name. | +| `package.version` | validated exact version | Exact resolved public version, never a range or `*`. | +| `extension_match` | `public`, `unnamed_only`, `none` | What kind of resolved extension, if any, the package contributed to. | +| `package_subject` | scoped id | Deduplicates this exact coordinate for 30 days. | + + +A package is named only when its package manager reports provenance matching a reviewed public-registry allowlist. Registry URLs themselves are not recorded. `extension_match: public` means at least one eligible public extension matched, including when unnamed content also matched. `unnamed_only` means extension content matched but none was eligible to name. `none` means no resolved extension matched. + +Events measure packages independently. Symposium does not record a workspace/resolution id, but the summary and contiguous relationship batch can still make the public package set for one sync recoverable, especially when only one sync occurred that day. Review the whole file before sharing it. + +### `extension_resolution` + +One public plugin or skill and one safe path that selected it. + + +| Field | Values | Meaning | +| ------------------- | --------------------------- | --------------------------------------------------- | +| `target.type` | `plugin`, `skill` | Resolved extension type. | +| `target.source` | `symposium-recommendations`, `crates-io` | Stable label, never a configured URL or local name. | +| `target.name` | validated string | Name defined by eligible public content. | +| `path` | bounded typed nodes | Actual safe package/predicate/extension chain. | +| `extension_subject` | scoped id | Deduplicates this safe target/path for 30 days. | + + +Path nodes are limited to: + + +| Node `type` | Recorded content | +| ----------- | -------------------------------------------------------------------- | +| `package` | Eligible public ecosystem, name, and exact version. | +| `extension` | Eligible public plugin/skill source and name. | +| `all` | All safe children that contributed to success. | +| `any` | The branch that actually made the expression succeed. | +| `not` | Marker only; the child is not recorded. | +| `opaque` | Fixed reason: `private_source`, `non_package_predicate`, or `limit`. | + + +Shell commands, paths, environment variables, custom predicate names/arguments, and private package/extension names never enter a path. Their position can be represented by an opaque marker. Paths are limited to 8 nesting levels, 16 evidence leaves, and 4 KiB; an over-limit subtree becomes `opaque: limit`. + +This event says an extension resolved. It does not say that an agent read or used the extension; a matching `extension_invocation_metrics` aggregate separately reports observed Claude activation. + +### `hook_metrics` + +The cumulative aggregate for one UTC day, agent, hook surface, and active identifier epoch. An epoch normally lasts 30 days but ends early on identifier reset or renewed consent. A reset can therefore leave two rows for the same agent/hook/day with different `hook_subject` values. A completed hook updates the current row in place; Symposium does not append one telemetry row per invocation. + + +| Field | Values | Meaning | +| --------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro` | Invoking agent. | +| `hook` | `pre_tool_use`, `post_tool_use`, `user_prompt_submit`, `session_start`, `stop` | Symposium hook surface. | +| `invocations` | integer | Completed hook observations merged into the row. | +| `outcomes` | hook outcome counters | Exact counters named `ok`, `blocked`, `plugin_error`, and `internal_error`. | +| `plugins_attempted` | integer | Plugin hooks whose preparation began. | +| `plugins_completed` | integer | Plugin hooks with an observed terminal result. | +| `duration_ms` | latency histogram | Parsed-input-to-response-ready latency; telemetry update time is excluded. | +| `session_counts_complete` | boolean | Whether the two distinct identified-session counts are complete for every observation. | +| `identified_sessions` | integer, optional | Distinct identified sessions represented; present only when `session_counts_complete=true`. | +| `identified_sessions_non_ok` | integer, optional | Those sessions with at least one non-`ok` observation; present only when counts are complete. | +| `hook_subject` | scoped id | Links this agent/hook dimension inside its 30-day identifier window. | + + +`outcomes` and the duration histogram each sum to `invocations`. The top-level outcome precedence is `internal_error`, then `blocked`, then `plugin_error`, then `ok`; exactly one counter advances per completed observation. Session counts remain complete only if every contributing observation supplied a session id and neither of the two sets exceeded 256 distinct ids for this row. On the first missing id or overflow, Symposium discards both sets, writes `session_counts_complete: false`, and omits both counts for the rest of that day. The raw and keyed session ids are never written into the aggregate file. + +### `plugin_hook_metrics` + +The cumulative aggregate for one UTC day, agent, hook surface, active identifier epoch, and bounded plugin bucket. A reset can leave multiple otherwise identical `unnamed` or `overflow` rows distinguished only by `event_id`; public rows also receive a new `plugin_subject`. + + +| Field | Values | Meaning | +| ----------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro` | Invoking agent. | +| `hook` | `pre_tool_use`, `post_tool_use`, `user_prompt_submit`, `session_start`, `stop` | Symposium hook surface. | +| `plugin_scope` | `public`, `unnamed`, `overflow` | Whether the bucket names an eligible public plugin. | +| `plugin.source` | `symposium-recommendations`, `crates-io`, conditional | Present only when `plugin_scope=public`. | +| `plugin.name` | validated string, conditional | Present only when `plugin_scope=public`. | +| `attempts` | integer | Plugin-hook attempts that reached an observed terminal result. | +| `executions` | integer | Those completed attempts that reached child execution. | +| `outcomes` | plugin outcome counters | Exact counters named `ok`, `blocked`, and `error`. | +| `prepare_ms` | latency histogram | Preparation time for every attempt. | +| `execute_ms` | latency histogram | Child execution time for attempts counted by `executions`. | +| `session_counts_complete` | boolean | Whether the two distinct identified-session counts are complete for every attempt. | +| `identified_sessions` | integer, optional | Distinct identified sessions represented; present only when `session_counts_complete=true`. | +| `identified_sessions_non_ok` | integer, optional | Those sessions with at least one non-`ok` attempt; present only when counts are complete. | +| `plugin_subject` | scoped id, conditional | Present only with an eligible public plugin. | + + +`attempts` counts only terminal results and `executions` counts the subset that started child execution. A preparation failure is terminal `error` with no execution. `outcomes` and `prepare_ms` each sum to `attempts`; `execute_ms` sums to `executions`. Across every epoch and bucket for one agent/hook/day, plugin `attempts` sum to the corresponding top-level `plugins_completed`. `blocked` means the plugin requested a block, `error` covers any closed preparation/execution failure, and `ok` is every other completed attempt. The same 256-id all-or-nothing session rule applies. Eligible public plugins use `{source, name}` coordinates from the reviewed allowlist. Private, local, invalid, or otherwise ineligible plugins merge into one `unnamed` row per agent/hook/epoch/day and expose no identity. At most 128 named public-plugin rows may appear across all agents, hooks, and identifier epochs in one UTC day; attempts for later rows merge into `overflow` rows per agent/hook/epoch and expose no identity. Identifier reset does not reset this daily limit. Named rows are first-observed, not sampled: earlier-in-day public plugins are overrepresented when the limit is reached. Analysis must report overflow and must not treat the named subset as random. + +A latency histogram is exactly: + +```json +{"bounds":[5,10,25,50,100,250,500,1000],"counts":[0,0,0,0,0,0,0,0,0]} +``` + +The nine non-overlapping millisecond buckets mean `<=5`, `(5,10]`, `(10,25]`, `(25,50]`, `(50,100]`, `(100,250]`, `(250,500]`, `(500,1000]`, and `>1000`. Bounds and count-array length cannot vary. Fixed histograms can be merged correctly across installations; locally calculated percentiles cannot. + +Neither aggregate kind has `at`, an invocation id, a raw or scoped session id, or an individual invocation's time or outcome. No hook input, output, stdout, stderr, error text, or numeric exit detail is recorded. Symposium adds no timeout as part of telemetry. A host termination, lock conflict, storage limit, or I/O failure can omit an observation. Aggregate counts are therefore lower bounds. No durable dropped-update counter can cover every loss mode because lock contention, termination, and I/O failure can also prevent writing that counter; counting only cap rejections would imply false completeness. + +These rows do reveal exact daily counts for each hook surface. In particular, `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` counts are proxies for daily tool and prompt activity even though no individual activity record exists. + +### `extension_invocation_metrics` + +The cumulative aggregate for one UTC day, Claude Code, active identifier epoch, and bounded skill bucket. Version 1 accepts only a fixture-tested Claude `Skill` signal. Symposium does not append one row per invocation. + +| Field | Values | Meaning | +| ------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| `agent` | `claude` | The only version 1 agent with a supported structured skill-invocation signal. | +| `target_scope` | `public`, `unnamed`, `overflow` | Whether the bucket names an eligible public skill. | +| `target.type` | `skill`, conditional | Present only when `target_scope=public`. | +| `target.source` | `symposium-recommendations`, `crates-io`, conditional | Reviewed public source; present only when `target_scope=public`. | +| `target.name` | validated string, conditional | Public skill name; present only when `target_scope=public`. | +| `unnamed_reason` | `ineligible`, `not_discovered`, `attribution_unavailable`, `ambiguous`, `invalid_signal`, conditional | Present only when `target_scope=unnamed`. | +| `attempted` | integer | Valid Claude `PreToolUse:Skill` observations merged into the row. | +| `completed` | integer | Successful Claude `PostToolUse:Skill` observations merged into the row. | +| `failed` | integer | Terminal Claude `PostToolUseFailure:Skill` observations merged into the row. | +| `session_counts_complete` | boolean | Whether both distinct identified-session counts are complete for all contributing observations. | +| `identified_sessions` | integer, optional | Distinct sessions with an attempted observation; present only when session counts are complete. | +| `identified_sessions_completed` | integer, optional | Distinct sessions with a completed observation; present only when session counts are complete. | +| `extension_subject` | scoped id, conditional | Present only for a public bucket; matches the selected safe resolution path for 30 days. | + +`attempted`, `completed`, and `failed` are independent lower bounds. Each hook phase updates the snapshot separately, so loss of one update means completed plus failed need not equal attempted and may exceed it. The two session counts have the same independence. `failed` advances only when Claude emits the targeted failure event; denial, termination, or a missing terminal observation is not inferred as failure. Completion means Claude successfully activated the skill; it does not show whether Claude followed its instructions or improved the task result. + +Public attribution comes only from the generated installation index and a matching Symposium marker fingerprint. `ineligible` combines private, local, invalid, and otherwise unsafe-to-name installed skills. `not_discovered` means the agent-facing identifier had no installed Discovery entry. `attribution_unavailable` means the index was missing, corrupt, or stale. `ambiguous` means more than one entry matched. `invalid_signal` means the Claude payload did not match the validated schema. No reason exposes the raw identifier. + +At most 128 public-skill rows may be named across identifier epochs in one UTC day. Later public skills merge into one `overflow` row per agent and epoch. Each fixed unnamed reason produces at most one row per agent and epoch. The public subset is first-observed, not sampled; analysis must report overflow and include unnamed counts when interpreting the public denominator. + +The same 256-id all-or-nothing rule applies to the attempted and completed session sets. On the first missing id, overflow, or state/snapshot mismatch, Symposium discards both sets, writes `session_counts_complete: false`, and omits both counts for the rest of that row and day. + +This aggregate has no `at`, duration, raw or scoped session id, invocation id, tool name, tool input/output, prompt, transcript, or individual outcome row. Unsupported agents emit no row, which means unknown rather than zero. + +### `command` + +One eligible top-level user command completed. + + +| Field | Values | Meaning | +| ----------------- | ---------------- | ------------------------------------------------------------- | +| `at` | UTC second | Completion time. | +| `command` | typed coordinate | Fixed built-in, or eligible public plugin command coordinate. | +| `duration_ms` | integer | Top-level command duration. | +| `outcome` | `ok`, `error` | Closed result. | +| `command_subject` | scoped id | Deduplicates this command for 30 days. | + + +Arguments are never recorded. Internal `hook`, all `telemetry` commands, and ineligible external/plugin commands do not produce command events. + +Built-in names are `init`, `sync`, `search`, `use`, `status`, `plugin_sync`, `plugin_list`, `plugin_show`, `plugin_validate`, `self_update`, and `crate_info`: + +```json +{"type":"builtin","name":"use"} +``` + +An eligible plugin command contains only its reviewed public-source label, public plugin name, and declared command name: + +```json +{"type":"plugin","source":"symposium-recommendations","plugin":"example-tools","name":"example-check"} +``` + +### `storage_limit` + +The next complete low-volume event batch did not fit in the shared daily 8 MiB allowance. In addition to the common fields, `dropped_operation` is `session_start`, `manual_sync`, `use`, `remove`, `init`, `configuration`, or `command`, identifying the top-level operation whose batch was rejected. This event appears at most once per UTC day and the marker itself counts toward 8 MiB. An aggregate-metric update that would exceed its separate 512 KiB maximum or the remaining shared allowance is dropped without this marker and does not stop low-volume recording. The marker does not report lock-contention, aggregate-update, or crash losses. + +## What is never recorded + +- Individual prompt/tool activity records, per-invocation hook records, or individual skill-invocation records. +- Prompt text or any substring of it. +- Tool names, arguments, responses, or results. +- Shell commands, file contents, patches, or URLs from user/agent data. +- Error messages, debug strings, hook stdout/stderr, numeric exit details, or raw agent/package-manager payloads. +- Raw agent-facing or private skill identifiers and ephemeral invocation identifiers. +- File paths, workspace roots, project names, repository ids, or dependency-set snapshots. +- Environment names/values, hostname, username, home directory, IP address, model name, or account/vendor identifiers. +- Names or versions from private registries, git/path/workspace sources, or sources whose public identity is not allowlisted. +- Raw configured registry names/URLs or arbitrary git URLs. +- Raw vendor session ids, a global installation id, a workspace id, or one identifier shared across measurement purposes. +- Wall-clock timestamps more precise than one second. + +## Storage and expiry + +Low-volume events are appended as JSON lines in `events-YYYY-MM-DD.jsonl` under `~/.symposium/telemetry/`. Current cumulative hook, plugin-hook, and extension-invocation aggregates are JSON lines in `metrics-YYYY-MM-DD.jsonl`; the bounded snapshot is rewritten atomically after a merge. A process uses a non-waiting telemetry lock and may drop its complete buffered event batch or aggregate observation rather than delay your hook or command. Recording failures never change the user operation's result. + +The event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB daily allowance. Aggregate metrics may use at most 512 KiB, so high-volume hook and skill activity cannot consume the allowance reserved for resolution and configuration events. A file is eligible for deletion only when `current_utc_day - file_utc_day > 30`: a D0 file remains throughout D30 and is first eligible on D31. Cleanup is lazy, so an old file remains until a recording-capable invocation or telemetry command runs. + +The state file temporarily holds the bounded keyed session sets and snapshot contribution counts used to calculate complete distinct-session counts. They are not printed or copied into metric rows, are discarded at UTC-day rollover, and are removed by `telemetry clear` or `telemetry reset-identifiers`. State is atomically replaced before the corresponding metric snapshot; if a later snapshot write fails, a contribution-count mismatch on the next update discards the sets and permanently marks the row's session counts incomplete for that day. `telemetry clear` deletes event and aggregate-metric files but preserves identity/cohort state. `telemetry reset-identifiers` rotates future identifiers and starts a new retention cohort. `telemetry disable` stops recording; existing files remain unless its interactive clear offer is accepted or `telemetry clear` removes them later. + +The generated `/.symposium/index-v1.json` file is installation state, not telemetry. It may contain local agent-facing identifiers, installed-directory fingerprints, eligibility, and safe public attribution needed to identify the skill Symposium installed. It is gitignored, atomically replaced after sync, excluded from `telemetry show` and `telemetry clear`, and not uploaded by this RFD. + +Public names/versions and exact counts can be identifying when they are unusual. Exact daily hook counts disclose approximate prompt/tool activity by surface, and extension-invocation rows disclose exact skill-attempt, completion, and failure counts. Lock contention, process termination, full files, and I/O failures can make the log incomplete. The records are therefore pseudonymous, locally associated, and best-effort; they must not be described as anonymous or as proof that an unrecorded action did not happen. diff --git a/md/rfds/telemetry-recording/proposed-reference-telemetry.md b/md/rfds/telemetry-recording/proposed-reference-telemetry.md new file mode 100644 index 00000000..e12c680d --- /dev/null +++ b/md/rfds/telemetry-recording/proposed-reference-telemetry.md @@ -0,0 +1,149 @@ +# `cargo agents telemetry` + +Manage opt-in, per-user local telemetry. See [What Symposium records](./proposed-data-collected.md) for the exhaustive field list and [Telemetry configuration](./proposed-configuration-telemetry.md) for consent semantics. + +Telemetry is off by default. Nothing is uploaded. + +## Usage + +```text +cargo agents telemetry [status] +cargo agents telemetry enable [--acknowledge] +cargo agents telemetry disable +cargo agents telemetry show [--count N] +cargo agents telemetry clear +cargo agents telemetry reset-identifiers +``` + +Telemetry management commands are never recorded as command telemetry. + +## `status` + +`status` shows the effective consent state and a physical/typed summary of local event and aggregate-metric files: + +```console +$ cargo agents telemetry status +Telemetry: enabled (consent version 1) + data directory: ~/.symposium/telemetry/ + stored: 3 file(s), 28.4 KiB + physical lines: 71 + supported rows: 68 + unknown schemas: 2 + malformed lines: 1 + range: 2026-08-01 through 2026-08-03 +``` + +Possible states are `disabled`, `consent required`, and `enabled`. `Consent required` means the config contains an earlier or unversioned opt-in; Symposium records nothing until you accept the current disclosure. + +Stored files/bytes and physical lines cover daily event and aggregate-metric files; they exclude `.lock`, `state.toml`, and temporary files. Supported rows are lines the current binary recognizes by kind and schema version. Unknown and malformed lines stay on disk and remain visible through `show`. `status` never prints the secret identity key or pending keyed session sets. + +## `enable` + +`enable` presents the exact disclosure below. Interactive `cargo agents init` uses the same text. Both default to no. + +```text +Symposium telemetry is off by default. If enabled, Symposium records: +- observed session starts and configured agents, with fresh/resumed status when + available, Symposium version, operating-system class, and architecture; +- public package names and exact versions, public plugin/skill resolution paths, + aggregate sync results, reasons packages remain unnamed, and whether only + unnamed extensions matched; +- exact daily Claude skill-activation attempts, completions, and failures, + public skill names, distinct-session counts when complete, fixed unnamed + reasons, and overflow counts; +- exact daily hook and plugin-hook counts, outcomes, and latency histograms; +- completed built-in and eligible public plugin commands, with outcome and + duration but without arguments; +- storage-limit markers naming only the affected operation; and +- purpose-scoped pseudonymous identifiers that link repeated observations for + up to 30 days, plus one D0-D30 observed-session cohort across agents. + +Counts for pre-tool-use, post-tool-use, and user-prompt-submit approximate daily +tool and prompt activity. Session starts and commands include UTC timestamps +truncated to one second; other rows include only the UTC day. + +Symposium never records prompt or tool content, tool names or arguments, file +paths, project or workspace identity, environment values, hostname, username, +model/account/vendor identifiers, raw errors, private package, plugin, or skill +names, raw agent-facing skill identifiers, individual hook-invocation rows, or +individual skill-invocation rows. + +Data stays on this machine in ~/.symposium/telemetry/. Nothing is uploaded. +Files remain through day 30 and become eligible for deletion on day 31. Inspect +them with cargo agents telemetry show and delete them with +cargo agents telemetry clear. + +Enable telemetry under consent version 1? [y/N] +``` + +The [exhaustive field list](./proposed-data-collected.md) and [never-record list](./proposed-data-collected.md#what-is-never-recorded) define the corresponding producer contract. + +In a non-interactive environment, `enable` does not change config unless `--acknowledge` is supplied explicitly. This prevents scripts or a manually retained unversioned boolean from upgrading consent silently. + +Enabling telemetry does not rewrite or assign new identifiers to old stored lines. Accepting a new consent version rotates the secret identity key and starts a new retention cohort, severing old and new scoped identifiers. + +## `disable` + +`disable` stops future recording by setting `enabled = false`. Existing event and aggregate-metric files remain on disk. In an interactive terminal, `disable` offers to clear them and defaults to keeping them. In a non-interactive environment, it prints the `clear` command instead of deleting data. + +## `show` + +`show` prints stored event and current aggregate-snapshot JSONL lines in deterministic storage order. UTC days are ascending; within a day, append-only event lines come first in physical order and aggregate rows follow in kind, agent, hook or target scope, public source/name or unnamed reason, and event-id order. The event id is a tie-breaker when an identifier reset creates two aggregate epochs in one day. `--count N` returns the last `N` lines of that ordering: + +```console +$ cargo agents telemetry show --count 2 +{"v":1,"kind":"command","event_id":"5d18caa8-84f7-4aa3-846c-99ea810ccd85","day":"2026-08-03","at":"2026-08-03T10:02:11Z","symposium":"0.4.0","command":{"type":"builtin","name":"use"},"duration_ms":820,"outcome":"ok","command_subject":"cmd_adf0c14ddc35b97762b5daae6f4119ce"} +{"v":1,"kind":"hook_metrics","event_id":"b563dd02-0301-4e2c-aac4-2e0d5dfaa977","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","invocations":500,"outcomes":{"ok":496,"blocked":1,"plugin_error":3,"internal_error":0},"plugins_attempted":500,"plugins_completed":500,"duration_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[8,17,76,144,181,68,6,0,0]},"session_counts_complete":true,"identified_sessions":4,"identified_sessions_non_ok":2,"hook_subject":"hok_b5b707841de7695912bec9b8bca382e8"} +``` + +The command copies stored line bytes; it does not parse, normalize, repair, or pretty-print them. Unknown-version and malformed lines are shown as stored. The current day's aggregate file is a cumulative snapshot, not a history of its replaced versions. Aggregate rows have no `at`, so this storage order must not be interpreted as chronology. Lines in the output came from the same Symposium home, and their order or day can expose co-occurrence even though the rows have no global installation or workspace id. Review the complete output before sharing it. The output can be redirected to create a local copy: + +```bash +cargo agents telemetry show --count 100000 > telemetry.jsonl +``` + +No separate export command is part of this RFD. + +## `clear` + +`clear` acquires the telemetry lock, deletes every `events-YYYY-MM-DD.jsonl` and `metrics-YYYY-MM-DD.jsonl` file, and discards pending aggregate session-count sets: + +```console +$ cargo agents telemetry clear +Deleted 12 telemetry data file(s) from ~/.symposium/telemetry/. +``` + +`clear` preserves the telemetry directory, lock, identity/cohort state, identity key, and consent setting. New rows in the same identifier window can therefore carry the same scoped subjects as cleared rows. Severing that future linkage also requires `reset-identifiers`. + +## `reset-identifiers` + +`reset-identifiers` acquires the telemetry lock, replaces the secret identity key, discards pending aggregate session-count sets, and starts a new retention cohort for future rows: + +```console +$ cargo agents telemetry reset-identifiers +Telemetry identifiers reset. Existing telemetry data files were not changed. +``` + +The command neither deletes nor rewrites old event or aggregate-metric rows. Identifiers before and after the reset cannot be derived into each other from the data files. + +If no identity state exists, the command reports that there is nothing to reset instead of creating a key. If existing state is unreadable or malformed, recording remains stopped until this command explicitly replaces it. + +## Files, concurrency, and expiry + +```text +~/.symposium/telemetry/ +|-- .lock +|-- state.toml +|-- metrics-2026-08-03.jsonl +`-- events-2026-08-03.jsonl +``` + +Each project skills parent may also contain a generated `.symposium/index-v1.json` installation index. It maps agent-facing skill identifiers to Symposium-managed installations so a later hook can attribute a skill activation. The index is gitignored installation state, not telemetry: `show`, `clear`, retention, and identifier reset do not read or delete it, and this RFD does not upload it. + +`state.toml` contains the secret identity key, rotation/cleanup state, and bounded keyed session sets plus contribution counts used for complete aggregate session counts. Do not publish it. The sets and contribution counts are never shown or copied into metric rows; they are discarded at day rollover or by `clear`/`reset-identifiers`. `show` reads event and aggregate-metric files only. `show` and `status` do not lock writers, so their multi-file view is not an atomic snapshot. + +Recorders make one non-waiting lock attempt. On contention they drop the entire event batch or aggregate observation rather than delay the agent or command. Event batches are appended. Hook, plugin-hook, and extension-invocation observations are merged into a bounded, canonically ordered snapshot by a same-directory temporary write and atomic replace; a crash leaves either the old or new complete snapshot, while abandoned temporary files are ignored and cleaned lazily. Session-count state is atomically replaced first and carries the snapshot contribution count; a mismatch after a failed snapshot write discards the sets and makes the row's session counts incomplete for that day. Management commands can wait for the lock. A crash can still lose the last batch or metric update, or leave a partial final event line; `status` reports that line as malformed and `show` preserves it. + +Hook and extension-invocation counts are lower bounds. There is no durable all-cause dropped-update counter because contention, termination, and I/O failure can also prevent writing that counter. + +Each UTC day's event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB allowance. Aggregate metrics may use at most 512 KiB; an aggregate update that would exceed that maximum or the remaining shared allowance is dropped without stopping low-volume event recording. Telemetry data files remain through D30 and become eligible for deletion on D31, when `current_utc_day - file_utc_day > 30`. Cleanup runs lazily, at most once per day, when a recording-capable or telemetry command next runs. Uninstalling Symposium does not delete these files. From 0fb0aca580bfc0c4df83718dc95c29fdca581f32 Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Mon, 17 Aug 2026 12:29:47 +0300 Subject: [PATCH 2/4] docs(rfd): refine telemetry recording design Clarify consent coverage, agent-neutral skill activation attribution terminology, private state handling, retention and implementation boundaries. Co-authored-by: Codex --- md/SUMMARY.md | 6 +- md/rfds/telemetry-recording/README.md | 122 +++++++++++------- .../proposed-configuration-telemetry.md | 8 +- .../proposed-data-collected.md | 24 ++-- .../proposed-reference-telemetry.md | 56 +++++--- 5 files changed, 132 insertions(+), 84 deletions(-) diff --git a/md/SUMMARY.md b/md/SUMMARY.md index a0681170..f1e51cc6 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -97,9 +97,9 @@ - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) - [Telemetry: recording events](./rfds/telemetry-recording/README.md) - - [Proposed: What Symposium records](./rfds/telemetry-recording/proposed-data-collected.md) - - [Proposed: `cargo agents telemetry`](./rfds/telemetry-recording/proposed-reference-telemetry.md) - - [Proposed: Telemetry configuration](./rfds/telemetry-recording/proposed-configuration-telemetry.md) + - [What Symposium records](./rfds/telemetry-recording/proposed-data-collected.md) + - [`cargo agents telemetry`](./rfds/telemetry-recording/proposed-reference-telemetry.md) + - [Telemetry configuration](./rfds/telemetry-recording/proposed-configuration-telemetry.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/telemetry-recording/README.md b/md/rfds/telemetry-recording/README.md index 115b45c1..fcc085f7 100644 --- a/md/rfds/telemetry-recording/README.md +++ b/md/rfds/telemetry-recording/README.md @@ -4,7 +4,7 @@ - Replace the experimental event list with a question-driven, closed schema for local telemetry. - Keep recording off by default, per-user, local only, and gated by versioned consent. Existing unversioned opt-ins must consent again. -- Record observed sessions, configured agents, public package/extension resolution, aggregate Claude skill invocation, aggregate hook reliability, command use, and known storage gaps. +- Record observed sessions, configured agents, public package/extension resolution, aggregate agent skill invocation, aggregate hook reliability, command use, and known storage gaps. - Never record individual prompt/tool activity rows, prompt or tool details, paths, private names, dependency snapshots, or a global installation/workspace id. - Use purpose-scoped pseudonyms and non-waiting, best-effort recording. Telemetry failure must not disrupt hooks, sync, or commands. - Defer upload, server-side handling, and subjective feedback to separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). @@ -13,15 +13,15 @@ Supporting pages: [data contract and exclusions](./proposed-data-collected.md), ## Motivation -Symposium has no production evidence about which integrations people reach, which public packages resolve to plugins and skills, whether Claude activates those skills, or whether hooks are slow or failing. The existing experimental telemetry was never wired into production and would record one row per prompt or tool call: high-volume activity that does not answer those questions. +Symposium has no production evidence about which integrations people reach, which public packages resolve to plugins and skills, whether agents activate those skills, or whether hooks are slow or failing. The existing experimental telemetry was never wired into production and would record one row per prompt or tool call: high-volume activity that does not answer those questions. We need a low-volume, inspectable record of reach, resolution, actual skill activation, and reliability so the team can prioritize agent support, improve recommendations, and detect unacceptable hook cost. The schema must be agreed before collection begins so every field has a stated use and privacy boundary. -This telemetry can show that a public skill resolved and Claude activated it. It cannot show that Claude followed the skill or that the skill improved the task outcome. Controlled evaluation and explicit feedback remain separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). +This telemetry can show that a public skill resolved and an agent activated it. It cannot show that the agent followed the skill or that the skill improved the task outcome. Version 1 observes structured skill activation only for Claude, but the event model and attribution boundary are agent-neutral. Controlled evaluation and explicit feedback remain separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). ## Change in a nutshell -Recording uses purpose-shaped JSONL. A full sync emits one summary plus safe package and extension relationships; hook and Claude skill observations update bounded daily snapshots: +Recording uses purpose-shaped JSONL. A full sync emits one summary plus safe package and extension relationships; hook and agent skill observations update bounded daily snapshots: ```text observed session -> session_start @@ -30,7 +30,7 @@ full sync -> resolution_summary -> extension_resolution* completed hook -> hook_metrics snapshot -> plugin_hook_metrics snapshot -Claude Skill -> extension_invocation_metrics snapshot +agent skill use -> extension_invocation_metrics snapshot command -> command ``` @@ -45,25 +45,29 @@ The [exhaustive data contract](./proposed-data-collected.md) owns exact fields, - Files remain through D30 and first become eligible for lazy deletion on D31. - Nothing is uploaded by this RFD. + + ## Detailed plans + + ### Measurement questions -All measures describe opted-in installations, not the whole user population. Reports must state that selection bias. +All measures describe opted-in installations, not the whole user population. Reports must state that selection bias. This RFD proposes the questions below to make the activity goals in [#246](https://github.com/symposium-dev/symposium/issues/246) and [#243](https://github.com/symposium-dev/symposium/issues/243) measurable. | # | Question | Operational definition | | --- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -| Q1 | Do opted-in installations return after first observed use? | Deduplicate `session_start` by `retention_subject` and measure observed cohort days D1, D7, and D30. | +| Q1 | Is another session observed after an installation's first session? | Deduplicate `session_start` by `retention_subject` and measure observed cohort days D1, D7, and D30. | | Q2 | Which plugins and skills resolve? | Count `extension_resolution` occurrences and scoped subjects by public extension and safe witnessed path. | | Q3 | Which public packages and versions occur, and what do they resolve? | Count `package_resolution` by public coordinate and `extension_match`; use paths carried by `extension_resolution`. | | Q4 | Are Symposium and plugin hooks failing or slow? | Use daily invocation/attempt counters, outcomes, fixed latency histograms, and complete identified-session impact counts. | | Q5 | Which agents, versions, and platforms have reach? | Keep configured-agent observations separate from observed hook sessions. | | Q6 | Which command surfaces are used? | Count completed built-ins and eligible public plugin commands without arguments. | -| Q7 | Which resolved public skills does Claude actually activate? | Count completed `extension_invocation_metrics` and complete identified-session counts by public skill and safe resolution subject. | +| Q7 | Which resolved public skills do agents actually activate? | Count completed `extension_invocation_metrics` and complete identified-session counts by public skill and safe resolution subject. | -Q1 measures continued installation, not continued value: session-start runs automatically once installed. Q2 proves resolution, not activation. Q7 proves activation, not that Claude followed the skill or completed the task better. Q3 records relationship edges, not a complete dependency set. Q4 counts only completed observations, so host termination can be invisible. +For each `retention_subject`, the first observed `session_start` establishes D0. D1, D7, or D30 is present when at least one later session start is observed on that cohort day, from the same or a different agent; multiple sessions on one day count once. Q1 therefore measures later-session retention, not one long session or continued value: session start runs automatically once Symposium is installed. Q2 proves resolution, not activation. Q7 proves activation, not that the agent followed the skill or completed the task better; version 1 can answer Q7 only for Claude. Q3 records relationship edges, not a complete dependency set. Q4 counts only completed observations, so host termination can be invisible. ### Scope and boundaries @@ -101,9 +105,9 @@ consent-version = 1 | `Enabled` | enabled with current consent version | Events defined by this contract. | -Existing users with an unversioned `enabled = true` must consent again. Interactive `init` and `telemetry enable` present the same disclosure and default to no. Non-interactive calls require an explicit acknowledgement; editing the boolean alone cannot upgrade consent. +Existing users with an unversioned `enabled = true` must consent again. Interactive `init` and `telemetry enable` present the same team-approved disclosure and default to no. Non-interactive calls require an explicit acknowledgement; editing the boolean alone cannot upgrade consent. -The [consent version 1 disclosure in the proposed telemetry command reference](./proposed-reference-telemetry.md#enable) is authoritative and is shared by interactive `init` and `telemetry enable`. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [proposed telemetry configuration page](./proposed-configuration-telemetry.md) is authoritative for effective-state semantics. +The [version 1 disclosure requirements](./proposed-reference-telemetry.md#disclosure-requirements) are authoritative. The command reference also gives a complete example for review, but does not pin its wording as implementation text. The team approves the final wording before recording is activated; `init` and `telemetry enable` then use the same snapshot-tested string. Editorial changes that preserve the required coverage and meaning do not require renewed consent. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [proposed telemetry configuration page](./proposed-configuration-telemetry.md) is authoritative for effective-state semantics. Adding an agent enum value creates a new schema version for each affected event kind because existing typed readers cannot parse that value. It does not by itself require renewed consent when the fields, categories, timestamp precision, and correlation boundaries remain inside the accepted disclosure. A new field, hook surface, or linkage for that agent does require a consent-version increase. @@ -125,35 +129,37 @@ Every row has a per-kind schema version, fixed kind, random row id, UTC day, and | `extension_resolution` | One public plugin/skill and one bounded safe path that selected it. | | `hook_metrics` | Cumulative daily counters and latency histogram per agent, hook, and identifier epoch. | | `plugin_hook_metrics` | Cumulative daily counters/histograms per agent, hook, epoch, and bounded plugin bucket. | -| `extension_invocation_metrics` | Cumulative daily Claude skill-attempt, completion, and failure counters per bounded public or unnamed bucket. | +| `extension_invocation_metrics` | Cumulative daily agent skill-attempt, completion, and failure counters per bounded public or unnamed bucket. | | `command` | One completed eligible top-level command without arguments. | | `storage_limit` | At most one daily marker naming the low-volume operation whose whole batch did not fit. | The `session_start` event is authoritative for observed-session and return measurements (Q1 and Q5). A `hook_metrics` row whose hook is `session_start` measures only that hook surface's reliability and latency for Q4; its invocation count is not a session count. -Ordinary hook lookup emits no resolution summary. A structured Claude `Skill` observation may update `extension_invocation_metrics` without emitting a resolution event. Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./proposed-data-collected.md) for exact fields and invariants. +Ordinary hook lookup emits no resolution summary. A structured agent skill-use observation may update `extension_invocation_metrics` without emitting a resolution event; version 1 obtains that observation from Claude's `Skill` signal. Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./proposed-data-collected.md) for exact fields and invariants. ### Event producers Producers return typed observations or reports; they never serialize telemetry or write files. The `Recorder` checks consent, applies public-identity and correlation rules, derives identifiers, and sends an accepted event batch or aggregate update to storage. -| Rows | Producer boundary | -| --- | --- | -| `session_start` | The outer hook wrapper, after a registered `SessionStart` completes successfully. | -| `agent_configuration` | The first non-telemetry recording-capable invocation each UTC day, as one all-agent snapshot derived from Symposium's per-user agent list. | + +| Rows | Producer boundary | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `session_start` | The outer hook wrapper, after a registered `SessionStart` completes successfully. | +| `agent_configuration` | The first non-telemetry recording-capable invocation each UTC day, as one all-agent snapshot derived from Symposium's per-user agent list. | | `resolution_summary`, `package_resolution`, `extension_resolution` | One structured full-sync report assembled while resolving and installing; telemetry does not rerun predicates or rediscover relationships. | -| `hook_metrics` | The outer hook wrapper, after the hook's final outcome and duration are known. | -| `plugin_hook_metrics` | The plugin dispatcher, around each applicable plugin's preparation and execution. | -| `extension_invocation_metrics` | The Claude `Skill` adapter normalizes targeted pre/success/failure hooks, then the generated installation index supplies safe attribution. | -| `command` | The top-level CLI dispatcher, after an eligible built-in or public plugin command reaches an outcome. | -| `storage_limit` | The JSONL sink itself, when a complete low-volume batch cannot fit. | +| `hook_metrics` | The outer hook wrapper, after the hook's final outcome and duration are known. | +| `plugin_hook_metrics` | The plugin dispatcher, around each applicable plugin's preparation and execution. | +| `extension_invocation_metrics` | The Claude `Skill` adapter normalizes targeted pre/success/failure hooks, then the generated installation index supplies safe attribution. | +| `command` | The top-level CLI dispatcher, after an eligible built-in or public plugin command reaches an outcome. | +| `storage_limit` | The JSONL sink itself, when a complete low-volume batch cannot fit. | + Agent and package-manager payloads therefore provide input to existing Symposium operations; they do not emit arbitrary telemetry. If a producer cannot construct its complete typed observation, it records nothing for that observation. ### Identifiers and correlation boundaries -On first enabled recording, telemetry atomically creates a random 32-byte identity key in `state.toml`. It derives the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension: +On first enabled recording, telemetry atomically creates a random 32-byte identity key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`), separate from the inspectable `/telemetry/` data directory. Symposium creates and replaces this state with owner-only permissions where the platform supports them. It derives the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension: ```text HMAC(key, "telemetry::v1\0" || window || "\0" || dimension) @@ -161,6 +167,10 @@ HMAC(key, "telemetry::v1\0" || window || "\0" || dimension) The dimension is included so each identifier represents one installation for one package, agent, or command dimension, never the installation globally. +Private state persists the identity key together with the current identifier-window and return-cohort anchors. Every recorder reads that state under the telemetry lock, so the same domain, window, and dimension produce the same subject across processes and restarts. Normal 30-day rollover changes the window input rather than replacing the key. `disable` and `clear` preserve the key and anchors; renewed consent and `reset-identifiers` replace the key and start a new cohort. + +The key is a secret, not anonymized telemetry. Possession permits candidate identifiers to be recomputed, so telemetry commands never print it and it remains outside the inspectable telemetry data directory. + | Identifier | Scope | | ------------------- | ------------------------------------------------------------------------------------------------ | @@ -199,11 +209,11 @@ Plugin, skill, and command names follow the same public-by-allowlist rule. This Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason (`private_source`, `non_package_predicate`, or `limit`). Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path. -Paths are bounded to 8 levels, 16 evidence leaves, and 4 KiB; an over-limit subtree becomes `opaque: limit`. Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled; only an enabled recorder serializes that evidence as telemetry. Existing cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`, but caching an entire `PredicateSet` would lose successful branches and witnesses and requires revisiting this design. +Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`; the complete path is also bounded to 16 evidence leaves and 4 KiB. This limit does not refer to filesystem path components; filesystem paths are never recorded. Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled; only an enabled recorder serializes that evidence as telemetry. Existing cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`, but caching an entire `PredicateSet` would lose successful branches and witnesses and requires revisiting this design. -This preserves actionable resolution relationships without recording a complete dependency snapshot. It proves that an extension resolved. A matching `extension_invocation_metrics` row separately proves that Claude activated the installed skill. +This preserves actionable resolution relationships without recording a complete dependency snapshot. It proves that an extension resolved. A matching `extension_invocation_metrics` row separately proves that an agent activated the installed skill; version 1 can produce that row only for Claude. -### Hook aggregation and agent capability +### Hook aggregation and version 1 agent coverage Each completed hook merges into one daily `hook_metrics` row per agent, surface, and active identifier epoch. Per-plugin preparation/execution merges into `plugin_hook_metrics`. Resets can create another epoch row on the same day. Fixed histogram bounds are `[5, 10, 25, 50, 100, 250, 500, 1000]` milliseconds so rows remain mergeable. @@ -213,18 +223,18 @@ Private plugins merge into unnamed buckets. At most 128 public-plugin rows are n Hook rows disclose exact daily counts. `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` therefore approximate daily tool/prompt activity even without individual records. Extension invocation rows disclose exact daily skill-attempt, completion, and failure counts. The disclosure states both explicitly. -This matrix reports current adapter capability and test coverage, not telemetry priority. The producer contract is agent-neutral; unavailable values remain optional or unsupported as shown. +This matrix defines which agent signals version 1 records. It is part of the producer contract, not implementation status or telemetry priority. Unsupported absence means unknown, not zero. -| Agent | Configuration | `SessionStart` | Session id | Fresh/resume | `Stop` | Skill invocation | -| -------------- | ------------- | -------------- | ---------- | ------------ | ------ | --------------------------- | +| Agent | Configuration | `SessionStart` | Session id | Fresh/resume | `Stop` | Skill invocation | +| -------------- | ------------- | -------------- | ---------- | ------------ | ------ | -------------------------- | | Claude Code | yes | yes | yes | yes | yes | attempted/completed/failed | -| Codex CLI | yes | yes | yes | yes | no | unsupported | -| GitHub Copilot | yes | yes | no | no | no | unsupported | -| Gemini CLI | yes | yes | yes | no | no | unsupported | -| Kiro | yes | yes | yes | no | no | unsupported | -| OpenCode | yes | no | n/a | n/a | no | unsupported | -| Goose | yes | no | n/a | n/a | no | unsupported | +| Codex CLI | yes | yes | yes | yes | no | unsupported | +| GitHub Copilot | yes | yes | no | no | no | unsupported | +| Gemini CLI | yes | yes | yes | no | no | unsupported | +| Kiro | yes | yes | yes | no | no | unsupported | +| OpenCode | yes | no | n/a | n/a | no | unsupported | +| Goose | yes | no | n/a | n/a | no | unsupported | Configured reach covers all seven agents; observed-session measures cover only registered hook integrations. `Stop` and structured skill invocation remain Claude-only in version 1. An unsupported agent produces no invocation row, which means unknown rather than zero. `Stop` is not required by Q1-Q7. @@ -237,11 +247,11 @@ Full sync writes a versioned installation index under the agent skills parent at The index is atomically replaced after sync determines which installations succeeded. Before naming a public invocation, lookup verifies the Symposium marker and fingerprint. A hook sees an old or new complete index; a missing, corrupt, or stale mapping never falls back to a path or content guess. -Public matches reuse the `extension_subject` derived for the selected safe resolution path. Other observations merge into `unnamed` rows with one fixed reason: `ineligible`, `not_discovered`, `attribution_unavailable`, `ambiguous`, or `invalid_signal`. After 128 named public-skill rows in one UTC day, further public matches merge into `overflow`. +Public matches reuse the `extension_subject` derived for the selected safe resolution path. Other observations merge into `unnamed` rows with one fixed reason: `ineligible`, `not_indexed`, `attribution_unavailable`, `ambiguous`, or `invalid_signal`. `not_indexed` means a valid attribution index has no matching agent-facing identifier; `attribution_unavailable` means the index is missing, corrupt, or stale. After 128 named public-skill rows in one UTC day, further public matches merge into `overflow`. Claude's existing `PreToolUse` and `PostToolUse` hooks increment attempts and completions. A `PostToolUseFailure` registration matched only to `Skill` increments failures and does not create generic failure-surface metrics. Each phase update is an independent lower bound: loss of one update means completed plus failed need not equal attempted and may exceed it. -The same all-or-nothing 256-id rule applies to attempted and completed distinct-session sets. A missing id, overflow, or state/snapshot mismatch makes both counts incomplete for that row and day. Completion proves activation only; it does not show whether Claude followed the instructions or improved the result. +The same all-or-nothing 256-id rule applies to attempted and completed distinct-session sets. A missing id, overflow, or state/snapshot mismatch makes both counts incomplete for that row and day. Completion proves activation only; it does not show whether the agent followed the instructions or improved the result. ### Recording architecture @@ -264,11 +274,11 @@ A hook prepares its agent response, then converts timings/outcomes into aggregat ### Storage, concurrency, and retention -Low-volume rows append to `events-YYYY-MM-DD.jsonl`. Current daily hook, plugin-hook, and extension-invocation aggregates live in a bounded, atomically replaced `metrics-YYYY-MM-DD.jsonl` snapshot. `state.toml` holds the identity key, cohort/cleanup metadata, marker state, and temporary keyed session-count sets; these sets are never emitted and expire at day rollover. +Low-volume rows append to `events-YYYY-MM-DD.jsonl`. Current daily hook, plugin-hook, and extension-invocation aggregates live in a bounded, atomically replaced `metrics-YYYY-MM-DD.jsonl` snapshot under the inspectable telemetry data directory. The sibling private `telemetry-state.toml` holds the identity key, cohort/cleanup metadata, marker state, and temporary keyed session-count sets; these sets are never emitted and expire at day rollover. The telemetry lock remains in the data directory and guards both data and private state mutations. -Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement; no `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state/snapshot divergence and permanently mark affected daily session counts incomplete. +Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement. Private-state replacement likewise uses a temporary file beside `telemetry-state.toml` in the config directory; abandoned state and snapshot temporaries are ignored and cleaned lazily under the telemetry lock. No `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state/snapshot divergence and permanently mark affected daily session counts incomplete. -The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events; an ordinary batch that cannot fit is replaced by the daily marker and ordinary recording stops for that day. Relationship batches are never split. +The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. This is a safety ceiling, not expected volume or preallocation: it bounds damage from a producer bug or unexpectedly large resolution batch, while normal recording should remain well below it. Together with D31 expiry, it bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events; an ordinary batch that cannot fit is replaced by the daily marker and ordinary recording stops for that day. Relationship batches are never split. Aggregate counters are lower bounds. No durable counter can quantify observations lost to lock contention, process termination, or I/O failure because those conditions can also prevent writing the counter; a cap-only counter would not measure total loss. @@ -309,16 +319,18 @@ Upload may use only accepted local fields and must preserve scoped-correlation b ### Proposed documentation - [What Symposium records](./proposed-data-collected.md): normative fields, enums, examples, and exclusions. -- [`cargo agents telemetry`](./proposed-reference-telemetry.md): controls, files, inspection, and concurrency. +- `[cargo agents telemetry](./proposed-reference-telemetry.md)`: controls, files, inspection, and concurrency. - [Telemetry configuration](./proposed-configuration-telemetry.md): consent and effective-state semantics. These remain proposed pages until implementation lands; shipped design/reference chapters continue to describe the current binary. ## Frequently asked questions + + ### What does pseudonymous mean here? -A scoped identifier is derived from a random local secret rather than machine identity, but it still links observations inside one stated purpose and window. Rotation and domain separation limit that linkage; they do not make the local files anonymous. +A scoped identifier is derived from a random local secret rather than machine identity, but it still links observations inside one stated purpose and window. Rotation and domain separation limit that linkage; they do not make the local files anonymous. The secret key is not anonymized or included in telemetry; it remains private because possessing it permits candidate identifiers to be recomputed. ### Why retain scoped identifiers, provenance, and witnesses before upload? @@ -334,7 +346,7 @@ Hooks are high-volume; Q4 needs rates and distributions, not traces. Resolution ### Does a completed skill activation mean the skill helped? -No. It means Claude successfully activated the installed skill. It does not show whether Claude followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill. +No. It means an agent successfully activated the installed skill; version 1 can observe this only for Claude. It does not show whether the agent followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill. ### Why not record a complete dependency snapshot? @@ -370,20 +382,24 @@ After Step 2, Steps 3 and 6 may proceed in parallel. Step 4 follows Step 3, Step ### Step 1: Telemetry contract and identity -Replace the dormant event types with the closed producer schema: common fields, bounded witnesses, fixed outcomes/histograms, extension-invocation counters and buckets, and per-kind version dispatch. Add telemetry-owned `state.toml`, atomic key creation, scoped HMAC derivation, local 30-day windows, D0-D30 cohorts, and reset primitives. Do not add storage or emission. +Replace the dormant event types with the closed producer schema: common fields, bounded witnesses, fixed outcomes/histograms, extension-invocation counters and buckets, and per-kind version dispatch. Add private telemetry-owned `/telemetry-state.toml`, atomic owner-only key creation where supported, scoped HMAC derivation, local 30-day windows, D0-D30 cohorts, and reset primitives. Keep this state outside the inspectable telemetry data directory. Do not add storage or emission. -Verify all contract examples round-trip; bounds and unknown versions behave as specified; arbitrary/private data cannot serialize; identities separate domains, dimensions, agents, and windows; D30 rollover/reset works; and disabled paths create nothing. +Verify all contract examples round-trip; bounds and unknown versions behave as specified; arbitrary/private data cannot serialize; identities separate domains, dimensions, agents, and windows; the same active-window inputs remain stable across recorders and restarts; normal rollover changes subjects without replacing the key; `disable`/`clear` preserve identity state; renewed consent/reset rotate it; and disabled paths create nothing. - [ ] PR: telemetry contract and scoped identity + + ### Step 2: Storage and local controls Add whole-batch event appends, non-waiting process locking, atomically replaced aggregate snapshots, daily caps/reservations, `storage_limit`, and lazy D31 cleanup. Add typed `status`, byte-preserving `show`, `clear`, and `reset-identifiers`. Expose a test-only enabled recorder bound to a caller-supplied temporary telemetry home for integration tests and benchmarks. No production caller writes through the sink yet, and no runtime bypass is added. -Verify concurrent complete lines, old-or-new snapshots, whole-operation drops, cap/marker accounting, D30/D31 cleanup, malformed/unknown inspection, clear/reset semantics, test-only recorder isolation, and that management commands never record themselves. +Verify concurrent complete lines, old-or-new snapshots, whole-operation drops, cap/marker accounting, D30/D31 cleanup, malformed/unknown inspection, private-state permissions and separation, abandoned state/snapshot temporary cleanup, clear/reset semantics, test-only recorder isolation, and that management commands never record themselves. - [ ] PR: telemetry storage and local controls + + ### Step 3: PM provenance and public-identity policy Extend every PM result with typed provenance and exact coordinates; add core-owned public allowlists and coordinate validation. If an out-of-process PM protocol lands first, carry provenance through it as part of this step. All supported PMs must provide provenance before events are wired. This step emits no telemetry. @@ -392,6 +408,8 @@ Verify every PM and source class, malformed and wildcard coordinates, public all - [ ] PR: package provenance and public identity policy + + ### Step 4: Resolution witnesses, attribution index, and recording Return safe evidence from the original predicate evaluation, preserving short-circuit and accepted cache behavior; whole-`PredicateSet` caching remains disallowed. Extend evidence through plugin/skill selection, return one structured full-sync report, and construct coherent `resolution_summary`, `package_resolution`, and `extension_resolution` batches. After installation, atomically replace the generated agent-facing attribution index with entries for successful managed skills and marker fingerprints. Ordinary read-only plugin lookup remains silent. @@ -400,6 +418,8 @@ Verify `all`/`any`/`not` and opaque/limit paths, cache hits without reevaluation - [ ] PR: resolution witnesses, installed attribution, and recording + + ### Step 5: Hook and extension-invocation telemetry Measure completed Symposium and plugin-hook handling. Merge top-level observations into `hook_metrics` and plugin preparation/execution into `plugin_hook_metrics`, with fixed outcomes/histograms, scoped subjects, public/unnamed/overflow buckets, the 128-named-row limit, and all-or-nothing identified-session counts. @@ -410,6 +430,8 @@ Measure the hook path before and after. Verify sanitized automatic/manual activa - [ ] PR: bounded hook and extension-invocation telemetry + + ### Step 6: Session, configuration, and command telemetry Add capability-aware `session_start`, daily `agent_configuration`, and eligible top-level `command` rows over the shared recorder. The first non-telemetry recording-capable invocation of each UTC day attempts one all-agent configuration batch from the per-user agent list; it does not inspect agent-owned files. Exclude hook internals, telemetry controls, arguments, and unsafe plugin command names. @@ -418,10 +440,12 @@ Verify the capability matrix, Copilot without a session id, Claude-only `Stop`, - [ ] PR: session reach and command telemetry + + ### Step 7: Consent, activation, and documentation -Add `consent-version`, one shared literal disclosure for `init` and `telemetry enable`, re-consent, explicit non-interactive acknowledgement, `enable`/`disable`, and final production wiring for Steps 4-6. The disclosure names exact daily skill activation and unnamed counts. There is no event-file migration because the dormant recorder was never called. Publish the proposed pages and update current design/flow chapters and `md/SUMMARY.md`. +Add `consent-version`, one shared team-approved disclosure for `init` and `telemetry enable`, re-consent, explicit non-interactive acknowledgement, `enable`/`disable`, and final production wiring for Steps 4-6. Review the final wording against the version 1 coverage requirements; the proposed prompt is a complete example, not the implementation string. There is no event-file migration because the dormant recorder was never called. Publish the proposed pages and update current design/flow chapters and `md/SUMMARY.md`. -Verify new/existing configuration states, a snapshot of the identical versioned disclosure used by `init` and `telemetry enable`, interactive/non-interactive flows, no partial or disabled collection, absence of a runtime consent bypass, raw inspection/expiry, the full CLI/integration suite, hook-path benchmark, formatting, clippy, workspace tests, mdBook, and orphan checks. +Verify new/existing configuration states, documented review of the final disclosure against every required coverage point, a snapshot proving `init` and `telemetry enable` use the identical approved text, interactive/non-interactive flows, no partial or disabled collection, absence of a runtime consent bypass, raw inspection/expiry, the full CLI/integration suite, hook-path benchmark, formatting, clippy, workspace tests, mdBook, and orphan checks. -- [ ] PR: telemetry consent and recording activation +- [ ] PR: telemetry consent and recording activation \ No newline at end of file diff --git a/md/rfds/telemetry-recording/proposed-configuration-telemetry.md b/md/rfds/telemetry-recording/proposed-configuration-telemetry.md index 4cd76a98..e7000e71 100644 --- a/md/rfds/telemetry-recording/proposed-configuration-telemetry.md +++ b/md/rfds/telemetry-recording/proposed-configuration-telemetry.md @@ -26,7 +26,7 @@ version 1: | `enabled = true` and `consent-version = 1` | Enabled | -The current version is owned by the binary, not accepted as an arbitrary configuration value. The [consent version 1 disclosure in the proposed telemetry command reference](./proposed-reference-telemetry.md#enable) is authoritative; interactive `init` presents the same text. +The current version is owned by the binary, not accepted as an arbitrary configuration value. The [version 1 disclosure requirements](./proposed-reference-telemetry.md#disclosure-requirements) are authoritative; the full prompt on that page is a non-normative example. Interactive `init` and `telemetry enable` present the same team-approved final text. A future release may require a higher version after a change to collected categories, linkability, timestamp precision, public-name eligibility, retention, or a normative exclusion. It records nothing under an older acknowledgement until you consent again. @@ -34,8 +34,8 @@ Adding an agent enum value creates a new schema version for each affected event Accepting a newer consent version rotates the telemetry identity key and starts a new D0-D30 return cohort. Existing event and aggregate-metric files are neither rewritten nor deleted, but their scoped identifiers cannot link to later rows. -`cargo agents telemetry enable` presents the disclosure for the current version before writing these values. Interactive `cargo agents init` presents the same disclosure for new users and existing unversioned opt-ins, defaulting to no. Non-interactive `init` does not grant or upgrade consent. +`cargo agents telemetry enable` presents the team-approved disclosure for the current version before writing these values. Interactive `cargo agents init` presents the same text for new users and existing unversioned opt-ins, defaulting to no. Non-interactive `init` does not grant or upgrade consent. -`cargo agents telemetry disable` sets `enabled = false` without deleting existing telemetry data files. Consent configuration is separate from the random identity key in `~/.symposium/telemetry/state.toml`. The state file is not configuration, and Symposium never reads it from project configuration. +`cargo agents telemetry disable` sets `enabled = false` without deleting existing telemetry data files. Consent configuration is separate from the random identity key and current identifier-window/cohort anchors in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). This state is stored outside the inspectable `/telemetry/` data directory and uses owner-only permissions where supported. It persists across `disable` and `clear` so identifiers remain consistent inside the active window; it is not configuration, and Symposium never reads it from project configuration. -Symposium-managed project skills may also have a generated `/.symposium/index-v1.json` installation index. It associates the agent-facing skill identifier with the installation Discovery selected. This file is gitignored installation state, not project configuration or telemetry; changing it cannot grant consent, and telemetry commands do not inspect or delete it. +Symposium-managed project skills may also have a generated `/.symposium/index-v1.json` installation index. It associates the agent-facing skill identifier with the resolved installation that sync produced. This file is gitignored installation state, not project configuration or telemetry; changing it cannot grant consent, and telemetry commands do not inspect or delete it. diff --git a/md/rfds/telemetry-recording/proposed-data-collected.md b/md/rfds/telemetry-recording/proposed-data-collected.md index 626dffaa..3f916e38 100644 --- a/md/rfds/telemetry-recording/proposed-data-collected.md +++ b/md/rfds/telemetry-recording/proposed-data-collected.md @@ -41,7 +41,9 @@ These are independent row-shape examples, not one coherent operation or batch. T ## Scoped identifiers -Symposium stores a random secret key in `~/.symposium/telemetry/state.toml` while telemetry is enabled. It derives keyed identifiers for narrow purposes. The key is not written into events, printed by telemetry commands, or derived from your machine. +When enabled recording first needs identity state, Symposium stores a random secret key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). The same state persists the current identifier-window and return-cohort anchors. Every recorder reads this state under the telemetry lock, so identical domain, window, and dimension inputs produce the same subject across processes and restarts. Normal 30-day rollover changes the window input without replacing the key; renewed consent or `telemetry reset-identifiers` replaces it. `telemetry disable` and `telemetry clear` preserve the key and anchors. + +This file is separate from the inspectable `/telemetry/` data directory and has owner-only permissions where the platform supports them. Symposium derives keyed identifiers for narrow purposes. The key is a secret rather than anonymized telemetry: it is not written into events, printed by telemetry commands, or derived from your machine, and possession permits candidate identifiers to be recomputed. | Identifier | What it can link | Rotation | @@ -90,7 +92,7 @@ A registered Symposium session-start hook completed. GitHub Copilot does not currently supply a session id. OpenCode and Goose do not currently call Symposium through a registered session-start hook, so they do not produce this event. -These rows, not `hook_metrics` rows whose `hook` is `session_start`, are authoritative for observed-session and return measurements. The aggregate hook rows measure only session-start hook reliability and latency. +These rows, not `hook_metrics` rows whose `hook` is `session_start`, are authoritative for observed-session and return measurements. For each `retention_subject`, the first row establishes D0; D1, D7, or D30 is present when at least one later session-start row has that `cohort_day`, regardless of agent or vendor session id. Multiple rows on the same cohort day count once. The aggregate hook rows measure only session-start hook reliability and latency. ### `agent_configuration` @@ -180,9 +182,9 @@ Path nodes are limited to: | `opaque` | Fixed reason: `private_source`, `non_package_predicate`, or `limit`. | -Shell commands, paths, environment variables, custom predicate names/arguments, and private package/extension names never enter a path. Their position can be represented by an opaque marker. Paths are limited to 8 nesting levels, 16 evidence leaves, and 4 KiB; an over-limit subtree becomes `opaque: limit`. +Shell commands, paths, environment variables, custom predicate names/arguments, and private package/extension names never enter a path. Their position can be represented by an opaque marker. Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`; the complete path is also limited to 16 evidence leaves and 4 KiB. This limit does not refer to filesystem path components; filesystem paths are never recorded. -This event says an extension resolved. It does not say that an agent read or used the extension; a matching `extension_invocation_metrics` aggregate separately reports observed Claude activation. +This event says an extension resolved. It does not say that an agent read or used the extension; a matching `extension_invocation_metrics` aggregate separately reports observed agent activation. Version 1 can produce that aggregate only for Claude. ### `hook_metrics` @@ -245,7 +247,7 @@ These rows do reveal exact daily counts for each hook surface. In particular, `p ### `extension_invocation_metrics` -The cumulative aggregate for one UTC day, Claude Code, active identifier epoch, and bounded skill bucket. Version 1 accepts only a fixture-tested Claude `Skill` signal. Symposium does not append one row per invocation. +The cumulative aggregate for one UTC day, supported agent, active identifier epoch, and bounded skill bucket. Version 1 supports only Claude Code and accepts only its fixture-tested `Skill` signal. Symposium does not append one row per invocation. | Field | Values | Meaning | | ------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | @@ -254,7 +256,7 @@ The cumulative aggregate for one UTC day, Claude Code, active identifier epoch, | `target.type` | `skill`, conditional | Present only when `target_scope=public`. | | `target.source` | `symposium-recommendations`, `crates-io`, conditional | Reviewed public source; present only when `target_scope=public`. | | `target.name` | validated string, conditional | Public skill name; present only when `target_scope=public`. | -| `unnamed_reason` | `ineligible`, `not_discovered`, `attribution_unavailable`, `ambiguous`, `invalid_signal`, conditional | Present only when `target_scope=unnamed`. | +| `unnamed_reason` | `ineligible`, `not_indexed`, `attribution_unavailable`, `ambiguous`, `invalid_signal`, conditional | Present only when `target_scope=unnamed`. | | `attempted` | integer | Valid Claude `PreToolUse:Skill` observations merged into the row. | | `completed` | integer | Successful Claude `PostToolUse:Skill` observations merged into the row. | | `failed` | integer | Terminal Claude `PostToolUseFailure:Skill` observations merged into the row. | @@ -263,9 +265,9 @@ The cumulative aggregate for one UTC day, Claude Code, active identifier epoch, | `identified_sessions_completed` | integer, optional | Distinct sessions with a completed observation; present only when session counts are complete. | | `extension_subject` | scoped id, conditional | Present only for a public bucket; matches the selected safe resolution path for 30 days. | -`attempted`, `completed`, and `failed` are independent lower bounds. Each hook phase updates the snapshot separately, so loss of one update means completed plus failed need not equal attempted and may exceed it. The two session counts have the same independence. `failed` advances only when Claude emits the targeted failure event; denial, termination, or a missing terminal observation is not inferred as failure. Completion means Claude successfully activated the skill; it does not show whether Claude followed its instructions or improved the task result. +`attempted`, `completed`, and `failed` are independent lower bounds. Each hook phase updates the snapshot separately, so loss of one update means completed plus failed need not equal attempted and may exceed it. The two session counts have the same independence. `failed` advances only when the supported agent emits its targeted failure signal; for version 1, that is Claude's `PostToolUseFailure:Skill` event. Denial, termination, or a missing terminal observation is not inferred as failure. Completion means the agent successfully activated the skill; it does not show whether the agent followed its instructions or improved the task result. -Public attribution comes only from the generated installation index and a matching Symposium marker fingerprint. `ineligible` combines private, local, invalid, and otherwise unsafe-to-name installed skills. `not_discovered` means the agent-facing identifier had no installed Discovery entry. `attribution_unavailable` means the index was missing, corrupt, or stale. `ambiguous` means more than one entry matched. `invalid_signal` means the Claude payload did not match the validated schema. No reason exposes the raw identifier. +Public attribution comes only from the generated installation index and a matching Symposium marker fingerprint. `ineligible` combines private, local, invalid, and otherwise unsafe-to-name installed skills. `not_indexed` means the index was valid and readable but contained no entry matching the agent-facing identifier. `attribution_unavailable` means the index was missing, corrupt, or stale. `ambiguous` means more than one entry matched. `invalid_signal` means the Claude payload did not match the validated schema. No reason exposes the raw identifier. At most 128 public-skill rows may be named across identifier epochs in one UTC day. Later public skills merge into one `overflow` row per agent and epoch. Each fixed unnamed reason produces at most one row per agent and epoch. The public subset is first-observed, not sampled; analysis must report overflow and include unnamed counts when interpreting the public denominator. @@ -322,11 +324,11 @@ The next complete low-volume event batch did not fit in the shared daily 8 MiB a ## Storage and expiry -Low-volume events are appended as JSON lines in `events-YYYY-MM-DD.jsonl` under `~/.symposium/telemetry/`. Current cumulative hook, plugin-hook, and extension-invocation aggregates are JSON lines in `metrics-YYYY-MM-DD.jsonl`; the bounded snapshot is rewritten atomically after a merge. A process uses a non-waiting telemetry lock and may drop its complete buffered event batch or aggregate observation rather than delay your hook or command. Recording failures never change the user operation's result. +Low-volume events are appended as JSON lines in `events-YYYY-MM-DD.jsonl` under the inspectable `/telemetry/` data directory (default `~/.symposium/telemetry/`). Current cumulative hook, plugin-hook, and extension-invocation aggregates are JSON lines in `metrics-YYYY-MM-DD.jsonl`; the bounded snapshot is rewritten atomically after a merge. The lock in this directory also guards sibling private state. A process uses one non-waiting lock attempt and may drop its complete buffered event batch or aggregate observation rather than delay your hook or command. Recording failures never change the user operation's result. -The event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB daily allowance. Aggregate metrics may use at most 512 KiB, so high-volume hook and skill activity cannot consume the allowance reserved for resolution and configuration events. A file is eligible for deletion only when `current_utc_day - file_utc_day > 30`: a D0 file remains throughout D30 and is first eligible on D31. Cleanup is lazy, so an old file remains until a recording-capable invocation or telemetry command runs. +The event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB daily allowance. The allowance is a safety ceiling, not expected volume or preallocation: it bounds damage from a producer bug or unexpectedly large resolution batch. Together with D31 expiry, it bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics may use at most 512 KiB, so high-volume hook and skill activity cannot consume the allowance reserved for resolution and configuration events. A file is eligible for deletion only when `current_utc_day - file_utc_day > 30`: a D0 file remains throughout D30 and is first eligible on D31. Cleanup is lazy, so an old file remains until a recording-capable invocation or telemetry command runs. -The state file temporarily holds the bounded keyed session sets and snapshot contribution counts used to calculate complete distinct-session counts. They are not printed or copied into metric rows, are discarded at UTC-day rollover, and are removed by `telemetry clear` or `telemetry reset-identifiers`. State is atomically replaced before the corresponding metric snapshot; if a later snapshot write fails, a contribution-count mismatch on the next update discards the sets and permanently marks the row's session counts incomplete for that day. `telemetry clear` deletes event and aggregate-metric files but preserves identity/cohort state. `telemetry reset-identifiers` rotates future identifiers and starts a new retention cohort. `telemetry disable` stops recording; existing files remain unless its interactive clear offer is accepted or `telemetry clear` removes them later. +The sibling private `/telemetry-state.toml` holds the identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, bounded keyed session sets, and snapshot contribution counts used to calculate complete distinct-session counts. It is atomically created and replaced with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. The sets are not printed or copied into metric rows, are discarded at UTC-day rollover, and are removed by `telemetry clear` or `telemetry reset-identifiers`. State is atomically replaced before the corresponding metric snapshot; if a later snapshot write fails, a contribution-count mismatch on the next update discards the sets and permanently marks the row's session counts incomplete for that day. `telemetry clear` deletes event and aggregate-metric files and atomically rewrites private state to remove pending sets, while preserving the identity key and current anchors. `telemetry reset-identifiers` rotates future identifiers and starts a new retention cohort. `telemetry disable` stops recording; existing files remain unless its interactive clear offer is accepted or `telemetry clear` removes them later. The generated `/.symposium/index-v1.json` file is installation state, not telemetry. It may contain local agent-facing identifiers, installed-directory fingerprints, eligibility, and safe public attribution needed to identify the skill Symposium installed. It is gitignored, atomically replaced after sync, excluded from `telemetry show` and `telemetry clear`, and not uploaded by this RFD. diff --git a/md/rfds/telemetry-recording/proposed-reference-telemetry.md b/md/rfds/telemetry-recording/proposed-reference-telemetry.md index e12c680d..97a7076d 100644 --- a/md/rfds/telemetry-recording/proposed-reference-telemetry.md +++ b/md/rfds/telemetry-recording/proposed-reference-telemetry.md @@ -35,11 +35,26 @@ Telemetry: enabled (consent version 1) Possible states are `disabled`, `consent required`, and `enabled`. `Consent required` means the config contains an earlier or unversioned opt-in; Symposium records nothing until you accept the current disclosure. -Stored files/bytes and physical lines cover daily event and aggregate-metric files; they exclude `.lock`, `state.toml`, and temporary files. Supported rows are lines the current binary recognizes by kind and schema version. Unknown and malformed lines stay on disk and remain visible through `show`. `status` never prints the secret identity key or pending keyed session sets. +Stored files/bytes and physical lines cover daily event and aggregate-metric files; they exclude `.lock`, temporary files, and sibling private `telemetry-state.toml`. Supported rows are lines the current binary recognizes by kind and schema version. Unknown and malformed lines stay on disk and remain visible through `show`. `status` never prints the secret identity key or pending keyed session sets. ## `enable` -`enable` presents the exact disclosure below. Interactive `cargo agents init` uses the same text. Both default to no. +### Disclosure requirements + +Before asking for consent, the team-approved version 1 disclosure must make these points clear: + +- Telemetry is off by default, requires an explicit opt-in, and defaults to no in both interactive entry points. +- Recorded categories include observed sessions and configured agents; Symposium version, agent labels, and platform classes; public package names and exact versions; public resolution relationships and aggregate sync results; exact daily hook, plugin-hook, and agent skill-activation metrics, with structured skill activation available only for Claude in version 1; completed eligible commands without arguments; and storage-limit markers. +- Exact hook-surface counts can approximate daily prompt/tool activity. The disclosure distinguishes second-precision session/command timestamps from day-only aggregate and resolution rows. +- Purpose-scoped pseudonymous identifiers permit the stated links for up to 30 days, including one D0-D30 observed-session cohort across agents; they are not anonymous or a global project/workspace identity. +- The main exclusions are stated plainly, including prompt/tool content, paths, project/workspace identity, environment and machine identity, raw errors and agent/package-manager payloads, private package/plugin/skill names, and individual hook or skill-invocation rows. +- Recorded data and private identifier/counting state remain local in their stated locations, nothing is uploaded, data remains through D30 and is eligible for deletion on D31, and the disclosure names the inspection and deletion commands. + +The [exhaustive field list](./proposed-data-collected.md) and [never-record list](./proposed-data-collected.md#what-is-never-recorded) define the details behind these requirements. The team may revise structure, tone, and wording before implementation, provided the final disclosure preserves every required point and its meaning. `cargo agents init` and `cargo agents telemetry enable` must use the same approved text and both default to no. + +### Example version 1 disclosure + +The following non-normative example demonstrates complete coverage. It is a review aid and starting point for the final product copy, not the string the implementation must reproduce. ```text Symposium telemetry is off by default. If enabled, Symposium records: @@ -61,14 +76,20 @@ Symposium telemetry is off by default. If enabled, Symposium records: Counts for pre-tool-use, post-tool-use, and user-prompt-submit approximate daily tool and prompt activity. Session starts and commands include UTC timestamps truncated to one second; other rows include only the UTC day. +Pseudonymous does not mean anonymous: the scoped identifiers permit only the +links described above, and no identifier links all telemetry or identifies a +project or workspace. Symposium never records prompt or tool content, tool names or arguments, file paths, project or workspace identity, environment values, hostname, username, -model/account/vendor identifiers, raw errors, private package, plugin, or skill -names, raw agent-facing skill identifiers, individual hook-invocation rows, or -individual skill-invocation rows. - -Data stays on this machine in ~/.symposium/telemetry/. Nothing is uploaded. +model/account/vendor identifiers, raw errors or agent/package-manager payloads, +private package, plugin, or skill names, raw agent-facing skill identifiers, +individual hook-invocation rows, or individual skill-invocation rows. + +Recorded data stays on this machine in ~/.symposium/telemetry/. A private +identifier key and bounded counting state stay in +~/.symposium/telemetry-state.toml; telemetry commands never print them. +Nothing is uploaded. Files remain through day 30 and become eligible for deletion on day 31. Inspect them with cargo agents telemetry show and delete them with cargo agents telemetry clear. @@ -76,8 +97,6 @@ cargo agents telemetry clear. Enable telemetry under consent version 1? [y/N] ``` -The [exhaustive field list](./proposed-data-collected.md) and [never-record list](./proposed-data-collected.md#what-is-never-recorded) define the corresponding producer contract. - In a non-interactive environment, `enable` does not change config unless `--acknowledge` is supplied explicitly. This prevents scripts or a manually retained unversioned boolean from upgrading consent silently. Enabling telemetry does not rewrite or assign new identifiers to old stored lines. Accepting a new consent version rotates the secret identity key and starts a new retention cohort, severing old and new scoped identifiers. @@ -131,19 +150,22 @@ If no identity state exists, the command reports that there is nothing to reset ## Files, concurrency, and expiry ```text -~/.symposium/telemetry/ -|-- .lock -|-- state.toml -|-- metrics-2026-08-03.jsonl -`-- events-2026-08-03.jsonl +~/.symposium/ +|-- telemetry-state.toml +`-- telemetry/ + |-- .lock + |-- metrics-2026-08-03.jsonl + `-- events-2026-08-03.jsonl ``` Each project skills parent may also contain a generated `.symposium/index-v1.json` installation index. It maps agent-facing skill identifiers to Symposium-managed installations so a later hook can attribute a skill activation. The index is gitignored installation state, not telemetry: `show`, `clear`, retention, and identifier reset do not read or delete it, and this RFD does not upload it. -`state.toml` contains the secret identity key, rotation/cleanup state, and bounded keyed session sets plus contribution counts used for complete aggregate session counts. Do not publish it. The sets and contribution counts are never shown or copied into metric rows; they are discarded at day rollover or by `clear`/`reset-identifiers`. `show` reads event and aggregate-metric files only. `show` and `status` do not lock writers, so their multi-file view is not an atomic snapshot. +`telemetry-state.toml` is private Symposium state outside the inspectable telemetry data directory. It contains the secret identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, and bounded keyed session sets plus contribution counts used for complete aggregate session counts. All recorders read this state under the telemetry lock. Normal 30-day rollover changes the window anchor without replacing the key; renewed consent or `reset-identifiers` replaces it, while `disable` and `clear` preserve it. + +Symposium atomically creates and replaces the file with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. `show`, `status`, data retention, and `clear` do not expose or delete the state file. `clear` atomically rewrites it only to remove pending sets, preserving the key and current anchors. The sets and contribution counts are never copied into metric rows and are discarded at day rollover or by `clear`/`reset-identifiers`. `show` and `status` do not lock writers, so their multi-file view is not an atomic snapshot. -Recorders make one non-waiting lock attempt. On contention they drop the entire event batch or aggregate observation rather than delay the agent or command. Event batches are appended. Hook, plugin-hook, and extension-invocation observations are merged into a bounded, canonically ordered snapshot by a same-directory temporary write and atomic replace; a crash leaves either the old or new complete snapshot, while abandoned temporary files are ignored and cleaned lazily. Session-count state is atomically replaced first and carries the snapshot contribution count; a mismatch after a failed snapshot write discards the sets and makes the row's session counts incomplete for that day. Management commands can wait for the lock. A crash can still lose the last batch or metric update, or leave a partial final event line; `status` reports that line as malformed and `show` preserves it. +Recorders make one non-waiting attempt on the lock in the telemetry data directory; that lock guards both data and private state mutations. On contention they drop the entire event batch or aggregate observation rather than delay the agent or command. Event batches are appended. Hook, plugin-hook, and extension-invocation observations are merged into a bounded, canonically ordered snapshot by a same-directory temporary write and atomic replace; a crash leaves either the old or new complete snapshot, while abandoned temporary files are ignored and cleaned lazily. Session-count state is atomically replaced first and carries the snapshot contribution count; a mismatch after a failed snapshot write discards the sets and makes the row's session counts incomplete for that day. Management commands can wait for the lock. A crash can still lose the last batch or metric update, or leave a partial final event line; `status` reports that line as malformed and `show` preserves it. Hook and extension-invocation counts are lower bounds. There is no durable all-cause dropped-update counter because contention, termination, and I/O failure can also prevent writing that counter. -Each UTC day's event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB allowance. Aggregate metrics may use at most 512 KiB; an aggregate update that would exceed that maximum or the remaining shared allowance is dropped without stopping low-volume event recording. Telemetry data files remain through D30 and become eligible for deletion on D31, when `current_utc_day - file_utc_day > 30`. Cleanup runs lazily, at most once per day, when a recording-capable or telemetry command next runs. Uninstalling Symposium does not delete these files. +Each UTC day's event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB allowance. This is a safety ceiling, not expected volume or preallocation: it bounds damage from a producer bug or unexpectedly large resolution batch. Together with D31 expiry, it bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics may use at most 512 KiB; an aggregate update that would exceed that maximum or the remaining shared allowance is dropped without stopping low-volume event recording. Telemetry data files remain through D30 and become eligible for deletion on D31, when `current_utc_day - file_utc_day > 30`. Cleanup runs lazily, at most once per day, when a recording-capable or telemetry command next runs. Uninstalling Symposium does not delete these files. From 3b6f9c38495dcca6e83fcd3de07d7b628d0cb0c8 Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Sun, 23 Aug 2026 00:03:29 +0300 Subject: [PATCH 3/4] Improve file names and structuring into focused directory --- md/SUMMARY.md | 6 +++--- md/rfds/telemetry-recording/README.md | 20 +++++++++---------- .../recorded-data.md} | 0 .../configuration.md} | 2 +- .../telemetry-command.md} | 4 ++-- 5 files changed, 16 insertions(+), 16 deletions(-) rename md/rfds/telemetry-recording/{proposed-data-collected.md => contract/recorded-data.md} (100%) rename md/rfds/telemetry-recording/{proposed-configuration-telemetry.md => reference/configuration.md} (94%) rename md/rfds/telemetry-recording/{proposed-reference-telemetry.md => reference/telemetry-command.md} (97%) diff --git a/md/SUMMARY.md b/md/SUMMARY.md index f1e51cc6..e42cd1cc 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -97,9 +97,9 @@ - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) - [Telemetry: recording events](./rfds/telemetry-recording/README.md) - - [What Symposium records](./rfds/telemetry-recording/proposed-data-collected.md) - - [`cargo agents telemetry`](./rfds/telemetry-recording/proposed-reference-telemetry.md) - - [Telemetry configuration](./rfds/telemetry-recording/proposed-configuration-telemetry.md) + - [What Symposium records](./rfds/telemetry-recording/contract/recorded-data.md) + - [`cargo agents telemetry`](./rfds/telemetry-recording/reference/telemetry-command.md) + - [Telemetry configuration](./rfds/telemetry-recording/reference/configuration.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/telemetry-recording/README.md b/md/rfds/telemetry-recording/README.md index fcc085f7..15208512 100644 --- a/md/rfds/telemetry-recording/README.md +++ b/md/rfds/telemetry-recording/README.md @@ -9,7 +9,7 @@ - Use purpose-scoped pseudonyms and non-waiting, best-effort recording. Telemetry failure must not disrupt hooks, sync, or commands. - Defer upload, server-side handling, and subjective feedback to separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). -Supporting pages: [data contract and exclusions](./proposed-data-collected.md), [telemetry command reference and consent disclosure](./proposed-reference-telemetry.md), and [configuration and consent states](./proposed-configuration-telemetry.md). +Supporting pages: [data contract and exclusions](./contract/recorded-data.md), [telemetry command reference and consent disclosure](./reference/telemetry-command.md), and [configuration and consent states](./reference/configuration.md). ## Motivation @@ -34,7 +34,7 @@ agent skill use -> extension_invocation_metrics snapshot command -> command ``` -The [exhaustive data contract](./proposed-data-collected.md) owns exact fields, enums, exclusions, and complete JSONL examples. The main design has these invariants: +The [exhaustive data contract](./contract/recorded-data.md) owns exact fields, enums, exclusions, and complete JSONL examples. The main design has these invariants: - Only `Enabled` recording can create telemetry state or files. - The producer accepts closed typed data, never arbitrary metadata or raw errors. @@ -107,7 +107,7 @@ consent-version = 1 Existing users with an unversioned `enabled = true` must consent again. Interactive `init` and `telemetry enable` present the same team-approved disclosure and default to no. Non-interactive calls require an explicit acknowledgement; editing the boolean alone cannot upgrade consent. -The [version 1 disclosure requirements](./proposed-reference-telemetry.md#disclosure-requirements) are authoritative. The command reference also gives a complete example for review, but does not pin its wording as implementation text. The team approves the final wording before recording is activated; `init` and `telemetry enable` then use the same snapshot-tested string. Editorial changes that preserve the required coverage and meaning do not require renewed consent. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [proposed telemetry configuration page](./proposed-configuration-telemetry.md) is authoritative for effective-state semantics. +The [version 1 disclosure requirements](./reference/telemetry-command.md#disclosure-requirements) are authoritative. The command reference also gives a complete example for review, but does not pin its wording as implementation text. The team approves the final wording before recording is activated; `init` and `telemetry enable` then use the same snapshot-tested string. Editorial changes that preserve the required coverage and meaning do not require renewed consent. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [telemetry configuration reference](./reference/configuration.md) is authoritative for effective-state semantics. Adding an agent enum value creates a new schema version for each affected event kind because existing typed readers cannot parse that value. It does not by itself require renewed consent when the fields, categories, timestamp precision, and correlation boundaries remain inside the accepted disclosure. A new field, hook surface, or linkage for that agent does require a consent-version increase. @@ -136,7 +136,7 @@ Every row has a per-kind schema version, fixed kind, random row id, UTC day, and The `session_start` event is authoritative for observed-session and return measurements (Q1 and Q5). A `hook_metrics` row whose hook is `session_start` measures only that hook surface's reliability and latency for Q4; its invocation count is not a session count. -Ordinary hook lookup emits no resolution summary. A structured agent skill-use observation may update `extension_invocation_metrics` without emitting a resolution event; version 1 obtains that observation from Claude's `Skill` signal. Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./proposed-data-collected.md) for exact fields and invariants. +Ordinary hook lookup emits no resolution summary. A structured agent skill-use observation may update `extension_invocation_metrics` without emitting a resolution event; version 1 obtains that observation from Claude's `Skill` signal. Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./contract/recorded-data.md) for exact fields and invariants. ### Event producers @@ -282,7 +282,7 @@ The event file, aggregate snapshot, and reserved maximum-size `storage_limit` ro Aggregate counters are lower bounds. No durable counter can quantify observations lost to lock contention, process termination, or I/O failure because those conditions can also prevent writing the counter; a cap-only counter would not measure total loss. -Files survive D30 and become eligible for lazy deletion when `current_day - file_day > 30`, first on D31. `clear` deletes event/metric files and pending count sets but preserves consent and identity/cohort state. `reset-identifiers` rotates future identifiers without rewriting old files. `disable` stops recording but keeps files by default; uninstall also leaves them. Exact command behavior belongs to the [telemetry CLI reference](./proposed-reference-telemetry.md). +Files survive D30 and become eligible for lazy deletion when `current_day - file_day > 30`, first on D31. `clear` deletes event/metric files and pending count sets but preserves consent and identity/cohort state. `reset-identifiers` rotates future identifiers without rewriting old files. `disable` stops recording but keeps files by default; uninstall also leaves them. Exact command behavior belongs to the [telemetry CLI reference](./reference/telemetry-command.md). Concurrent agents produce separate session and agent/hook rows. The same public package/path derives the same dimension subject for unique-install deduplication, but no project id links the agents. A simultaneous flush may be dropped; concurrent mutation of installed skills remains a separate problem. @@ -290,7 +290,7 @@ Concurrent agents produce separate session and agent/hook rows. The same public Schema versions are per event kind. Semantic or correlation changes create a new version; privacy expansions also require a new consent version. Readers retain malformed and unknown lines, count them separately, and exclude them from typed analysis; raw `show` preserves their bytes. -The ["What is never recorded" section of the proposed data contract](./proposed-data-collected.md#what-is-never-recorded) is a producer rule. In summary: no prompt/tool content or per-invocation rows, raw errors/payloads, paths or workspace identity, environment/machine/account values, private-source names, arbitrary URLs, global identifiers, or timestamps finer than one second. +The ["What is never recorded" section of the data contract](./contract/recorded-data.md#what-is-never-recorded) is a producer rule. In summary: no prompt/tool content or per-invocation rows, raw errors/payloads, paths or workspace identity, environment/machine/account values, private-source names, arbitrary URLs, global identifiers, or timestamps finer than one second. Here, the per-invocation exclusion includes individual skill activations and raw agent-facing skill identifiers. @@ -318,9 +318,9 @@ Upload may use only accepted local fields and must preserve scoped-correlation b ### Proposed documentation -- [What Symposium records](./proposed-data-collected.md): normative fields, enums, examples, and exclusions. -- `[cargo agents telemetry](./proposed-reference-telemetry.md)`: controls, files, inspection, and concurrency. -- [Telemetry configuration](./proposed-configuration-telemetry.md): consent and effective-state semantics. +- [What Symposium records](./contract/recorded-data.md): normative fields, enums, examples, and exclusions. +- [`cargo agents telemetry`](./reference/telemetry-command.md): controls, files, inspection, and concurrency. +- [Telemetry configuration](./reference/configuration.md): consent and effective-state semantics. These remain proposed pages until implementation lands; shipped design/reference chapters continue to describe the current binary. @@ -448,4 +448,4 @@ Add `consent-version`, one shared team-approved disclosure for `init` and `telem Verify new/existing configuration states, documented review of the final disclosure against every required coverage point, a snapshot proving `init` and `telemetry enable` use the identical approved text, interactive/non-interactive flows, no partial or disabled collection, absence of a runtime consent bypass, raw inspection/expiry, the full CLI/integration suite, hook-path benchmark, formatting, clippy, workspace tests, mdBook, and orphan checks. -- [ ] PR: telemetry consent and recording activation \ No newline at end of file +- [ ] PR: telemetry consent and recording activation diff --git a/md/rfds/telemetry-recording/proposed-data-collected.md b/md/rfds/telemetry-recording/contract/recorded-data.md similarity index 100% rename from md/rfds/telemetry-recording/proposed-data-collected.md rename to md/rfds/telemetry-recording/contract/recorded-data.md diff --git a/md/rfds/telemetry-recording/proposed-configuration-telemetry.md b/md/rfds/telemetry-recording/reference/configuration.md similarity index 94% rename from md/rfds/telemetry-recording/proposed-configuration-telemetry.md rename to md/rfds/telemetry-recording/reference/configuration.md index e7000e71..d0af558c 100644 --- a/md/rfds/telemetry-recording/proposed-configuration-telemetry.md +++ b/md/rfds/telemetry-recording/reference/configuration.md @@ -26,7 +26,7 @@ version 1: | `enabled = true` and `consent-version = 1` | Enabled | -The current version is owned by the binary, not accepted as an arbitrary configuration value. The [version 1 disclosure requirements](./proposed-reference-telemetry.md#disclosure-requirements) are authoritative; the full prompt on that page is a non-normative example. Interactive `init` and `telemetry enable` present the same team-approved final text. +The current version is owned by the binary, not accepted as an arbitrary configuration value. The [version 1 disclosure requirements](./telemetry-command.md#disclosure-requirements) are authoritative; the full prompt on that page is a non-normative example. Interactive `init` and `telemetry enable` present the same team-approved final text. A future release may require a higher version after a change to collected categories, linkability, timestamp precision, public-name eligibility, retention, or a normative exclusion. It records nothing under an older acknowledgement until you consent again. diff --git a/md/rfds/telemetry-recording/proposed-reference-telemetry.md b/md/rfds/telemetry-recording/reference/telemetry-command.md similarity index 97% rename from md/rfds/telemetry-recording/proposed-reference-telemetry.md rename to md/rfds/telemetry-recording/reference/telemetry-command.md index 97a7076d..47444719 100644 --- a/md/rfds/telemetry-recording/proposed-reference-telemetry.md +++ b/md/rfds/telemetry-recording/reference/telemetry-command.md @@ -1,6 +1,6 @@ # `cargo agents telemetry` -Manage opt-in, per-user local telemetry. See [What Symposium records](./proposed-data-collected.md) for the exhaustive field list and [Telemetry configuration](./proposed-configuration-telemetry.md) for consent semantics. +Manage opt-in, per-user local telemetry. See [What Symposium records](../contract/recorded-data.md) for the exhaustive field list and [Telemetry configuration](./configuration.md) for consent semantics. Telemetry is off by default. Nothing is uploaded. @@ -50,7 +50,7 @@ Before asking for consent, the team-approved version 1 disclosure must make thes - The main exclusions are stated plainly, including prompt/tool content, paths, project/workspace identity, environment and machine identity, raw errors and agent/package-manager payloads, private package/plugin/skill names, and individual hook or skill-invocation rows. - Recorded data and private identifier/counting state remain local in their stated locations, nothing is uploaded, data remains through D30 and is eligible for deletion on D31, and the disclosure names the inspection and deletion commands. -The [exhaustive field list](./proposed-data-collected.md) and [never-record list](./proposed-data-collected.md#what-is-never-recorded) define the details behind these requirements. The team may revise structure, tone, and wording before implementation, provided the final disclosure preserves every required point and its meaning. `cargo agents init` and `cargo agents telemetry enable` must use the same approved text and both default to no. +The [exhaustive field list](../contract/recorded-data.md) and [never-record list](../contract/recorded-data.md#what-is-never-recorded) define the details behind these requirements. The team may revise structure, tone, and wording before implementation, provided the final disclosure preserves every required point and its meaning. `cargo agents init` and `cargo agents telemetry enable` must use the same approved text and both default to no. ### Example version 1 disclosure From 2eb9ed3bd394a481947e17d27527f33e340625bc Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Sun, 23 Aug 2026 00:34:57 +0300 Subject: [PATCH 4/4] docs: refine telemetry RFD prose --- md/rfds/telemetry-recording/README.md | 337 ++++++++++++------ .../contract/recorded-data.md | 148 +++++--- .../reference/configuration.md | 27 +- .../reference/telemetry-command.md | 68 +++- 4 files changed, 407 insertions(+), 173 deletions(-) diff --git a/md/rfds/telemetry-recording/README.md b/md/rfds/telemetry-recording/README.md index 15208512..6d596dfb 100644 --- a/md/rfds/telemetry-recording/README.md +++ b/md/rfds/telemetry-recording/README.md @@ -13,15 +13,27 @@ Supporting pages: [data contract and exclusions](./contract/recorded-data.md), [ ## Motivation -Symposium has no production evidence about which integrations people reach, which public packages resolve to plugins and skills, whether agents activate those skills, or whether hooks are slow or failing. The existing experimental telemetry was never wired into production and would record one row per prompt or tool call: high-volume activity that does not answer those questions. +Symposium has no production evidence about which integrations people reach, which public packages resolve to plugins and skills, whether agents activate those skills, or whether hooks are slow or failing. -We need a low-volume, inspectable record of reach, resolution, actual skill activation, and reliability so the team can prioritize agent support, improve recommendations, and detect unacceptable hook cost. The schema must be agreed before collection begins so every field has a stated use and privacy boundary. +The existing experimental telemetry was never wired into production. Its proposed prompt and tool rows would be high-volume without answering those questions. -This telemetry can show that a public skill resolved and an agent activated it. It cannot show that the agent followed the skill or that the skill improved the task outcome. Version 1 observes structured skill activation only for Claude, but the event model and attribution boundary are agent-neutral. Controlled evaluation and explicit feedback remain separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). +The team instead needs a low-volume, inspectable record of reach, resolution, skill activation, and reliability. That evidence can guide agent support, recommendation work, and investigation of hook cost. The schema must be agreed before collection begins so every field has a stated use and privacy boundary. + +This telemetry can show that a public skill resolved and that an agent activated it. It cannot show that the agent followed the skill or that the skill improved the task outcome. Version 1 observes structured skill activation only for Claude, but the event model and attribution boundary are agent-neutral. + +Controlled evaluation and explicit feedback remain separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). ## Change in a nutshell -Recording uses purpose-shaped JSONL. A full sync emits one summary plus safe package and extension relationships; hook and agent skill observations update bounded daily snapshots: +From the user's perspective, telemetry follows one inspectable lifecycle: + +1. Telemetry begins disabled. `cargo agents init` and `cargo agents telemetry enable` present the same team-approved disclosure and default to no. +2. After versioned consent, recording-capable Symposium operations write only the disclosed local events and aggregates. +3. `cargo agents telemetry status` explains the effective state, while `show` exposes the stored bytes. +4. `disable` stops future recording without deleting existing files; `clear` removes recorded data, and `reset-identifiers` rotates future identifiers. +5. Nothing is uploaded under this RFD. + +Recording uses a JSONL schema built around those questions. A full sync emits one summary plus safe package and extension relationships. Hook and agent skill observations update bounded daily snapshots: ```text observed session -> session_start @@ -34,7 +46,7 @@ agent skill use -> extension_invocation_metrics snapshot command -> command ``` -The [exhaustive data contract](./contract/recorded-data.md) owns exact fields, enums, exclusions, and complete JSONL examples. The main design has these invariants: +The [exhaustive data contract](./contract/recorded-data.md) defines the fields, enums, exclusions, and complete JSONL examples. The main design has these invariants: - Only `Enabled` recording can create telemetry state or files. - The producer accepts closed typed data, never arbitrary metadata or raw errors. @@ -45,17 +57,12 @@ The [exhaustive data contract](./contract/recorded-data.md) owns exact fields, e - Files remain through D30 and first become eligible for lazy deletion on D31. - Nothing is uploaded by this RFD. - - ## Detailed plans - - ### Measurement questions All measures describe opted-in installations, not the whole user population. Reports must state that selection bias. This RFD proposes the questions below to make the activity goals in [#246](https://github.com/symposium-dev/symposium/issues/246) and [#243](https://github.com/symposium-dev/symposium/issues/243) measurable. - | # | Question | Operational definition | | --- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Q1 | Is another session observed after an installation's first session? | Deduplicate `session_start` by `retention_subject` and measure observed cohort days D1, D7, and D30. | @@ -66,16 +73,21 @@ All measures describe opted-in installations, not the whole user population. Rep | Q6 | Which command surfaces are used? | Count completed built-ins and eligible public plugin commands without arguments. | | Q7 | Which resolved public skills do agents actually activate? | Count completed `extension_invocation_metrics` and complete identified-session counts by public skill and safe resolution subject. | +These questions have specific limits: -For each `retention_subject`, the first observed `session_start` establishes D0. D1, D7, or D30 is present when at least one later session start is observed on that cohort day, from the same or a different agent; multiple sessions on one day count once. Q1 therefore measures later-session retention, not one long session or continued value: session start runs automatically once Symposium is installed. Q2 proves resolution, not activation. Q7 proves activation, not that the agent followed the skill or completed the task better; version 1 can answer Q7 only for Claude. Q3 records relationship edges, not a complete dependency set. Q4 counts only completed observations, so host termination can be invisible. +- For Q1, the first observed `session_start` for a `retention_subject` establishes D0. D1, D7, or D30 is present when at least one later session is observed on that cohort day, from the same or a different agent. Multiple sessions on one day count once. This measures a later observed session, not one long session or continued value; session start runs automatically once Symposium is installed. +- Q2 proves resolution, not activation. +- Q3 records relationship edges, not a complete dependency set. +- Q4 counts completed observations, so host termination can be invisible. +- Q7 proves activation, not that the agent followed the skill or completed the task better. Version 1 can answer Q7 only for Claude. ### Scope and boundaries -This RFD specifies the complete scope of local recording: consent, provenance, resolution evidence, event families, identifiers, agent capability gaps, storage, controls, rollout, documentation, and tests. +This RFD defines the complete scope of local recording: consent, provenance, resolution evidence, event families, identifiers, agent capability gaps, storage, controls, rollout, documentation, and tests. -It covers [#242](https://github.com/symposium-dev/symposium/issues/242) and the activity-metric portion of [#243](https://github.com/symposium-dev/symposium/issues/243). It does not cover #243's experimental outcome signals, [#244 uploading](https://github.com/symposium-dev/symposium/issues/244), or [#245 feedback collection](https://github.com/symposium-dev/symposium/issues/245). +It covers [#242](https://github.com/symposium-dev/symposium/issues/242) and the activity-metric portion of [#243](https://github.com/symposium-dev/symposium/issues/243). -It also does not specify new host-hook timeouts, public-identity inference from arbitrary git URLs, private-source names, or a fix for concurrent full-sync mutation of installed skills. Telemetry locking protects telemetry only. +It does not cover #243's experimental outcome signals, [#244 uploading](https://github.com/symposium-dev/symposium/issues/244), or [#245 feedback collection](https://github.com/symposium-dev/symposium/issues/245). It also leaves out new host-hook timeouts, public-identity inference from arbitrary git URLs, private-source names, and concurrent full-sync mutation of installed skills. Telemetry locking protects telemetry only. ### Vocabulary @@ -97,28 +109,31 @@ enabled = true consent-version = 1 ``` - | Effective state | Condition | Recording | | ----------------- | ---------------------------------------- | --------------------------------------------- | | `Disabled` | `enabled` absent or false | None; create no telemetry directory or state. | | `ConsentRequired` | enabled but consent version absent/stale | None until current disclosure is accepted. | | `Enabled` | enabled with current consent version | Events defined by this contract. | - Existing users with an unversioned `enabled = true` must consent again. Interactive `init` and `telemetry enable` present the same team-approved disclosure and default to no. Non-interactive calls require an explicit acknowledgement; editing the boolean alone cannot upgrade consent. -The [version 1 disclosure requirements](./reference/telemetry-command.md#disclosure-requirements) are authoritative. The command reference also gives a complete example for review, but does not pin its wording as implementation text. The team approves the final wording before recording is activated; `init` and `telemetry enable` then use the same snapshot-tested string. Editorial changes that preserve the required coverage and meaning do not require renewed consent. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [telemetry configuration reference](./reference/configuration.md) is authoritative for effective-state semantics. +The [version 1 disclosure requirements](./reference/telemetry-command.md#disclosure-requirements) define what the disclosure must cover. The command reference includes a complete example for review, but its wording is not fixed implementation text. The team approves the final wording before recording is activated. `init` and `telemetry enable` then use the same snapshot-tested string. + +Editorial changes that preserve coverage and meaning do not require renewed consent. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [telemetry configuration reference](./reference/configuration.md) defines the effective-state semantics. + +Adding an agent enum value creates a new schema version for each affected event kind because existing typed readers cannot parse the new value. It does not by itself require renewed consent when the fields, categories, timestamp precision, and correlation boundaries remain inside the accepted disclosure. -Adding an agent enum value creates a new schema version for each affected event kind because existing typed readers cannot parse that value. It does not by itself require renewed consent when the fields, categories, timestamp precision, and correlation boundaries remain inside the accepted disclosure. A new field, hook surface, or linkage for that agent does require a consent-version increase. +A new field, hook surface, or linkage for that agent does require a consent-version increase. No production event is wired until the typed catalogue, privacy contract, controls, documentation, and current consent flow all land. Earlier implementation steps remain inert behind `ConsentRequired`. ### Event contract -The schema is closed and typed: fixed variants, enums, counters, and bounded structures. It rejects arbitrary metadata maps, raw PM/agent payloads, errors, debug strings, command arguments, and sanitization fallbacks. Counters use checked unsigned arithmetic; overflow drops the batch or observation. +The schema is closed and typed. It accepts fixed variants, enums, counters, and bounded structures. It rejects arbitrary metadata maps, raw PM or agent payloads, errors, debug strings, command arguments, and sanitization fallbacks. -Every row has a per-kind schema version, fixed kind, random row id, UTC day, and Symposium version. Only completed `session_start` and `command` rows carry a wall-clock timestamp, truncated to one second. Millisecond durations remain because Q4 needs mergeable distributions. +Counters use checked unsigned arithmetic. Overflow drops the batch or observation. +Every row has a per-kind schema version, fixed kind, random row id, UTC day, and Symposium version. Only completed `session_start` and `command` rows carry a wall-clock timestamp, truncated to one second. Millisecond durations remain because Q4 needs mergeable distributions. | Kind | Purpose and emission | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------- | @@ -133,15 +148,15 @@ Every row has a per-kind schema version, fixed kind, random row id, UTC day, and | `command` | One completed eligible top-level command without arguments. | | `storage_limit` | At most one daily marker naming the low-volume operation whose whole batch did not fit. | - The `session_start` event is authoritative for observed-session and return measurements (Q1 and Q5). A `hook_metrics` row whose hook is `session_start` measures only that hook surface's reliability and latency for Q4; its invocation count is not a session count. -Ordinary hook lookup emits no resolution summary. A structured agent skill-use observation may update `extension_invocation_metrics` without emitting a resolution event; version 1 obtains that observation from Claude's `Skill` signal. Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./contract/recorded-data.md) for exact fields and invariants. +Ordinary hook lookup emits no resolution summary. A structured agent skill-use observation may update `extension_invocation_metrics` without emitting a resolution event. Version 1 obtains that observation from Claude's `Skill` signal. -### Event producers +Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./contract/recorded-data.md) for exact fields and invariants. -Producers return typed observations or reports; they never serialize telemetry or write files. The `Recorder` checks consent, applies public-identity and correlation rules, derives identifiers, and sends an accepted event batch or aggregate update to storage. +### Event producers +Producers return typed observations or reports. They never serialize telemetry or write files. The `Recorder` checks consent, applies public-identity and correlation rules, derives identifiers, and sends accepted event batches or aggregate updates to storage. | Rows | Producer boundary | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | @@ -154,23 +169,25 @@ Producers return typed observations or reports; they never serialize telemetry o | `command` | The top-level CLI dispatcher, after an eligible built-in or public plugin command reaches an outcome. | | `storage_limit` | The JSONL sink itself, when a complete low-volume batch cannot fit. | - -Agent and package-manager payloads therefore provide input to existing Symposium operations; they do not emit arbitrary telemetry. If a producer cannot construct its complete typed observation, it records nothing for that observation. +Agent and package-manager payloads provide input to existing Symposium operations; they do not emit arbitrary telemetry. If a producer cannot construct a complete typed observation, it records nothing for that observation. ### Identifiers and correlation boundaries -On first enabled recording, telemetry atomically creates a random 32-byte identity key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`), separate from the inspectable `/telemetry/` data directory. Symposium creates and replaces this state with owner-only permissions where the platform supports them. It derives the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension: +When enabled recording first needs identity state, Symposium atomically creates a random 32-byte key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). This file is separate from the inspectable `/telemetry/` data directory. Symposium creates and replaces it with owner-only permissions where the platform supports them. + +Identifiers use the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension: ```text HMAC(key, "telemetry::v1\0" || window || "\0" || dimension) ``` -The dimension is included so each identifier represents one installation for one package, agent, or command dimension, never the installation globally. +The dimension limits what an identifier can link. It represents one installation for one package, agent, or command dimension, never the installation globally. -Private state persists the identity key together with the current identifier-window and return-cohort anchors. Every recorder reads that state under the telemetry lock, so the same domain, window, and dimension produce the same subject across processes and restarts. Normal 30-day rollover changes the window input rather than replacing the key. `disable` and `clear` preserve the key and anchors; renewed consent and `reset-identifiers` replace the key and start a new cohort. +Private state keeps the identity key and the current identifier-window and return-cohort anchors. Every recorder reads that state under the telemetry lock, so the same domain, window, and dimension produce the same subject across processes and restarts. -The key is a secret, not anonymized telemetry. Possession permits candidate identifiers to be recomputed, so telemetry commands never print it and it remains outside the inspectable telemetry data directory. +Normal 30-day rollover changes the window input rather than replacing the key. `disable` and `clear` preserve the key and anchors. Renewed consent and `reset-identifiers` replace the key and start a new cohort. +The key is private state, not anonymized telemetry. Someone who has it can recompute candidate identifiers. Telemetry commands therefore never print it, and it remains outside the inspectable telemetry data directory. | Identifier | Scope | | ------------------- | ------------------------------------------------------------------------------------------------ | @@ -184,16 +201,19 @@ The key is a secret, not anonymized telemetry. Possession permits candidate iden | `plugin_subject` | One safe public plugin + 30-day window. | | `command_subject` | One safe command coordinate + 30-day window. | - The return subject is the sole cross-agent exception: it deduplicates Q1 but cannot link to other event kinds. A cohort remains stable through D30; the next observed session starts a new cohort. Accepting new consent or resetting identifiers rotates the key and starts another cohort. `session_id` is absent when the agent supplies none, including Copilot. Raw vendor ids never enter events or an unkeyed hash. There is no global installation/workspace id, and future analysis or upload must not reconstruct one. Missing identity state is created only when enabled; malformed existing state stops recording until explicit identifier reset. ### Public package and extension identity -Each package manager must return typed provenance with the resolved coordinate: registry URL, git, path, workspace, or unknown. Core, rather than the PM, maps allowlisted public registries to stable ecosystem labels. Raw URLs and provenance never enter an event; non-allowlisted sources are private/unnamed by default. +Each package manager must return typed provenance with the resolved coordinate: registry URL, git, path, workspace, or unknown. Core, rather than the package manager, maps allowlisted public registries to stable ecosystem labels. + +Raw URLs and provenance never enter an event. Sources outside the allowlist are private and unnamed by default. + +A named package requires allowlisted provenance, a valid public name, and an exact resolved version. Other inputs increment `unnamed_packages` and exactly one reason: `private_registry`, `git`, `path`, `workspace`, `unknown_source`, or `invalid_coordinate`. -A named package requires allowlisted provenance plus a valid public name and exact resolved version. Other inputs increment `unnamed_packages` and exactly one of `private_registry`, `git`, `path`, `workspace`, `unknown_source`, or `invalid_coordinate`. Source reasons take precedence; `invalid_coordinate` applies only to an otherwise public source with an invalid coordinate. Exact versions let the team isolate version-specific recommendation gaps. +Source reasons take precedence. `invalid_coordinate` applies only to an otherwise public source with an invalid coordinate. Exact versions let the team isolate version-specific recommendation gaps. `package_resolution.extension_match` distinguishes: @@ -207,25 +227,36 @@ Plugin, skill, and command names follow the same public-by-allowlist rule. This `extension_resolution` records the actual successful package-to-plugin-to-skill path. Predicate evaluation produces evidence in its original pass; telemetry never re-evaluates a predicate. -Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason (`private_source`, `non_package_predicate`, or `limit`). Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path. +Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason: `private_source`, `non_package_predicate`, or `limit`. + +Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path. + +Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`. The complete path is also limited to 16 evidence leaves and 4 KiB. These limits do not count filesystem path components; filesystem paths are never recorded. + +Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled. Only an enabled recorder serializes that evidence as telemetry. -Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`; the complete path is also bounded to 16 evidence leaves and 4 KiB. This limit does not refer to filesystem path components; filesystem paths are never recorded. Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled; only an enabled recorder serializes that evidence as telemetry. Existing cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`, but caching an entire `PredicateSet` would lose successful branches and witnesses and requires revisiting this design. +Cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`. Caching an entire `PredicateSet` would discard the successful branches and witnesses. Supporting such a cache would require a different witness design. This preserves actionable resolution relationships without recording a complete dependency snapshot. It proves that an extension resolved. A matching `extension_invocation_metrics` row separately proves that an agent activated the installed skill; version 1 can produce that row only for Claude. ### Hook aggregation and version 1 agent coverage -Each completed hook merges into one daily `hook_metrics` row per agent, surface, and active identifier epoch. Per-plugin preparation/execution merges into `plugin_hook_metrics`. Resets can create another epoch row on the same day. Fixed histogram bounds are `[5, 10, 25, 50, 100, 250, 500, 1000]` milliseconds so rows remain mergeable. +Each completed hook merges into one daily `hook_metrics` row per agent, surface, and active identifier epoch. Per-plugin preparation and execution merge into `plugin_hook_metrics`. An identifier reset can create another epoch row on the same day. -Outcome counters and histograms have exact sum invariants defined in the data contract. Identified-session counts are all-or-nothing: every observation must supply an id, and each keyed set is limited to 256. On a missing id, overflow, or state/snapshot mismatch, sets are discarded and the row remains incomplete for that day rather than publishing a plausible partial count. +Histogram bounds are fixed at `[5, 10, 25, 50, 100, 250, 500, 1000]` milliseconds so rows remain mergeable. -Private plugins merge into unnamed buckets. At most 128 public-plugin rows are named per UTC day across agents, hooks, and identifier epochs; later names merge into overflow buckets. Extension invocation metrics have a separate 128-public-skill daily limit. Each named subset is first-observed, so earlier-in-day plugins or skills are overrepresented when its limit is reached. Analysis must report overflow and must not treat named rows as a random sample. The combined aggregate snapshot may occupy at most 512 KiB of the shared daily allowance. +Outcome counters and histograms have the exact sum invariants defined in the data contract. Identified-session counts are all-or-nothing: every observation must supply an id, and each keyed set is limited to 256. + +On a missing id, overflow, or state/snapshot mismatch, Symposium discards the sets. The row remains incomplete for that day rather than publishing a plausible partial count. + +Private plugins merge into unnamed buckets. At most 128 public-plugin rows are named per UTC day across agents, hooks, and identifier epochs; later names merge into overflow buckets. Extension invocation metrics have a separate daily limit of 128 public skills. + +Each named subset is first-observed. Earlier-in-day plugins or skills are therefore overrepresented when the limit is reached. Analysis must report overflow and must not treat named rows as a random sample. The combined aggregate snapshot may occupy at most 512 KiB of the shared daily allowance. Hook rows disclose exact daily counts. `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` therefore approximate daily tool/prompt activity even without individual records. Extension invocation rows disclose exact daily skill-attempt, completion, and failure counts. The disclosure states both explicitly. This matrix defines which agent signals version 1 records. It is part of the producer contract, not implementation status or telemetry priority. Unsupported absence means unknown, not zero. - | Agent | Configuration | `SessionStart` | Session id | Fresh/resume | `Stop` | Skill invocation | | -------------- | ------------- | -------------- | ---------- | ------------ | ------ | -------------------------- | | Claude Code | yes | yes | yes | yes | yes | attempted/completed/failed | @@ -236,26 +267,30 @@ This matrix defines which agent signals version 1 records. It is part of the pro | OpenCode | yes | no | n/a | n/a | no | unsupported | | Goose | yes | no | n/a | n/a | no | unsupported | - Configured reach covers all seven agents; observed-session measures cover only registered hook integrations. `Stop` and structured skill invocation remain Claude-only in version 1. An unsupported agent produces no invocation row, which means unknown rather than zero. `Stop` is not required by Q1-Q7. ### Skill invocation attribution Agent-specific parsing ends at a normalized extension-use observation. The Claude adapter accepts only a fixture-tested `Skill` tool shape and produces a skill identifier plus one phase: `attempted`, `completed`, or `failed`. Raw Claude input and ephemeral invocation ids do not enter the recorder. -Full sync writes a versioned installation index under the agent skills parent at `.symposium/index-v1.json`. This generated, gitignored installation state is separate from configuration and telemetry. It maps the actual agent-facing identifier to the installed directory, marker fingerprint, eligibility, and safe public coordinate/path when one exists. Private identifiers may remain in this local index because they already exist in installed skill content; they are never serialized as telemetry. +Full sync writes a versioned installation index under the agent skills parent at `.symposium/index-v1.json`. This generated, gitignored file is installation state, not configuration or telemetry. + +The index maps the actual agent-facing identifier to the installed directory, marker fingerprint, eligibility, and safe public coordinate or path when one exists. Private identifiers may remain in the local index because they already exist in installed skill content. They are never serialized as telemetry. The index is atomically replaced after sync determines which installations succeeded. Before naming a public invocation, lookup verifies the Symposium marker and fingerprint. A hook sees an old or new complete index; a missing, corrupt, or stale mapping never falls back to a path or content guess. -Public matches reuse the `extension_subject` derived for the selected safe resolution path. Other observations merge into `unnamed` rows with one fixed reason: `ineligible`, `not_indexed`, `attribution_unavailable`, `ambiguous`, or `invalid_signal`. `not_indexed` means a valid attribution index has no matching agent-facing identifier; `attribution_unavailable` means the index is missing, corrupt, or stale. After 128 named public-skill rows in one UTC day, further public matches merge into `overflow`. +Public matches reuse the `extension_subject` derived for the selected safe resolution path. Other observations merge into `unnamed` rows with one fixed reason: `ineligible`, `not_indexed`, `attribution_unavailable`, `ambiguous`, or `invalid_signal`. + +`not_indexed` means a valid attribution index has no matching agent-facing identifier. `attribution_unavailable` means the index is missing, corrupt, or stale. After 128 named public-skill rows in one UTC day, further public matches merge into `overflow`. -Claude's existing `PreToolUse` and `PostToolUse` hooks increment attempts and completions. A `PostToolUseFailure` registration matched only to `Skill` increments failures and does not create generic failure-surface metrics. Each phase update is an independent lower bound: loss of one update means completed plus failed need not equal attempted and may exceed it. +Claude's existing `PreToolUse` and `PostToolUse` hooks increment attempts and completions. A `PostToolUseFailure` registration matched only to `Skill` increments failures; it does not create generic failure-surface metrics. + +Each phase update is an independent lower bound. If one update is lost, completed plus failed need not equal attempted and may exceed it. The same all-or-nothing 256-id rule applies to attempted and completed distinct-session sets. A missing id, overflow, or state/snapshot mismatch makes both counts incomplete for that row and day. Completion proves activation only; it does not show whether the agent followed the instructions or improved the result. ### Recording architecture - | Unit | Responsibility | | ------------------------- | --------------------------------------------------------------------------------- | | `Recorder` | Enforce consent, buffer typed batches, and isolate recording failures. | @@ -267,50 +302,122 @@ The same all-or-nothing 256-id rule applies to attempted and completed distinct- | `MetricAggregator` | Merge hook, plugin-hook, and extension-invocation observations into bounded rows. | | `JsonlSink` | Lock, cap, retain, append/replace, inspect, and clear files. | +Only an enabled `Recorder` owns the telemetry sink and identity components. Call sites cannot write directly. The generated installation index is functional sync state and exists independently of telemetry consent. + +A sync returns a structured report with provenance and witnesses. That report drives skill installation, atomic index replacement, and, when recording is enabled, one sanitized relationship batch. -Only an enabled `Recorder` owns telemetry sink and identity components; call sites cannot write directly. The generated installation index is functional sync state and exists independently of telemetry consent. A sync returns a structured report with provenance and witnesses, which installs skills, atomically replaces that index, and lets the recorder sanitize one buffered relationship batch when enabled. +A hook prepares its agent response before converting timings and outcomes into aggregate updates. When telemetry is disabled and no other sink is active, the observation router does not load the index or construct an extension-use observation. -A hook prepares its agent response, then converts timings/outcomes into aggregate updates where possible. The observation router does not load the index or construct an extension-use observation when telemetry is disabled and no other sink is active. Commands are measured once at top-level dispatch. Raw errors never enter telemetry. +Commands are measured once at top-level dispatch. Raw errors never enter telemetry. ### Storage, concurrency, and retention -Low-volume rows append to `events-YYYY-MM-DD.jsonl`. Current daily hook, plugin-hook, and extension-invocation aggregates live in a bounded, atomically replaced `metrics-YYYY-MM-DD.jsonl` snapshot under the inspectable telemetry data directory. The sibling private `telemetry-state.toml` holds the identity key, cohort/cleanup metadata, marker state, and temporary keyed session-count sets; these sets are never emitted and expire at day rollover. The telemetry lock remains in the data directory and guards both data and private state mutations. +#### Files and state + +Low-volume rows append to `events-YYYY-MM-DD.jsonl`. Current daily hook, plugin-hook, and extension-invocation aggregates live in a bounded, atomically replaced `metrics-YYYY-MM-DD.jsonl` snapshot under the inspectable telemetry data directory. + +The sibling private `telemetry-state.toml` holds the identity key, cohort and cleanup metadata, marker state, and temporary keyed session-count sets. These sets are never emitted and expire at day rollover. The telemetry lock remains in the data directory and guards data and private state mutations. + +#### Concurrent writes and failure -Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement. Private-state replacement likewise uses a temporary file beside `telemetry-state.toml` in the config directory; abandoned state and snapshot temporaries are ignored and cleaned lazily under the telemetry lock. No `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state/snapshot divergence and permanently mark affected daily session counts incomplete. +Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement. -The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. This is a safety ceiling, not expected volume or preallocation: it bounds damage from a producer bug or unexpectedly large resolution batch, while normal recording should remain well below it. Together with D31 expiry, it bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events; an ordinary batch that cannot fit is replaced by the daily marker and ordinary recording stops for that day. Relationship batches are never split. +Private-state replacement uses a temporary file beside `telemetry-state.toml` in the config directory. Abandoned state and snapshot temporaries are ignored and cleaned lazily under the telemetry lock. No `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state and snapshot divergence and permanently mark affected daily session counts incomplete. -Aggregate counters are lower bounds. No durable counter can quantify observations lost to lock contention, process termination, or I/O failure because those conditions can also prevent writing the counter; a cap-only counter would not measure total loss. +Aggregate counters are lower bounds. No durable counter can quantify observations lost to lock contention, process termination, or I/O failure because those conditions can also prevent writing the counter. A cap-only counter would not measure total loss. -Files survive D30 and become eligible for lazy deletion when `current_day - file_day > 30`, first on D31. `clear` deletes event/metric files and pending count sets but preserves consent and identity/cohort state. `reset-identifiers` rotates future identifiers without rewriting old files. `disable` stops recording but keeps files by default; uninstall also leaves them. Exact command behavior belongs to the [telemetry CLI reference](./reference/telemetry-command.md). +#### Size and retention + +The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. This is a safety ceiling, not expected volume or preallocation. It bounds damage from a producer bug or unexpectedly large resolution batch; normal recording should remain well below it. + +Together with D31 expiry, the daily allowance bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events. An ordinary batch that cannot fit is replaced by the daily marker, and ordinary recording stops for that day. Relationship batches are never split. + +Files survive D30 and become eligible for lazy deletion when `current_day - file_day > 30`, first on D31. `clear` deletes event and metric files plus pending count sets, but preserves consent and identity/cohort state. `reset-identifiers` rotates future identifiers without rewriting old files. + +`disable` stops recording but keeps files by default. Uninstalling Symposium also leaves them. The [telemetry CLI reference](./reference/telemetry-command.md) defines the exact command behavior. + +#### Multiple agents Concurrent agents produce separate session and agent/hook rows. The same public package/path derives the same dimension subject for unique-install deduplication, but no project id links the agents. A simultaneous flush may be dropped; concurrent mutation of installed skills remains a separate problem. -### Schema evolution and limitations +### Schema evolution + +Schema versions are per event kind. Semantic or correlation changes create a new version. Privacy expansions also require a new consent version. -Schema versions are per event kind. Semantic or correlation changes create a new version; privacy expansions also require a new consent version. Readers retain malformed and unknown lines, count them separately, and exclude them from typed analysis; raw `show` preserves their bytes. +Readers retain malformed and unknown lines, count them separately, and exclude them from typed analysis. Raw `show` preserves their bytes. -The ["What is never recorded" section of the data contract](./contract/recorded-data.md#what-is-never-recorded) is a producer rule. In summary: no prompt/tool content or per-invocation rows, raw errors/payloads, paths or workspace identity, environment/machine/account values, private-source names, arbitrary URLs, global identifiers, or timestamps finer than one second. +The ["What is never recorded" section of the data contract](./contract/recorded-data.md#what-is-never-recorded) is a producer rule. In summary, telemetry excludes prompt and tool content, per-invocation rows, raw errors and payloads, paths and workspace identity, environment/machine/account values, private-source names, arbitrary URLs, global identifiers, and timestamps finer than one second. Here, the per-invocation exclusion includes individual skill activations and raw agent-facing skill identifiers. -Scoped pseudonyms do not make a local directory anonymous. File/day/order and one buffered sync can expose co-occurrence; unusual public versions, public skill-use counts, agent/platform combinations, and exact counts can fingerprint an installation. Missing data is also non-random: busy multi-agent sessions contend more, terminated hooks lose final observations, and unsupported agents have configuration but not session or skill-invocation observations. Reports must state these limitations. +### Drawbacks and limitations + +This design accepts the following costs and limits: + +- Opt-in measurements describe participating installations, not the complete user population. Reports must state this selection bias. +- A completed skill activation does not prove that the skill was followed or improved the task. Version 1 also obtains structured skill-use observations only from Claude. +- Scoped pseudonyms do not make a local directory anonymous. File/day/order and one buffered sync can expose co-occurrence; unusual public versions, public skill-use counts, agent/platform combinations, and exact counts can fingerprint an installation. +- Best-effort recording undercounts activity. Busy multi-agent sessions contend more, terminated hooks lose final observations, and unsupported agents have configuration but not session or skill-invocation observations. A failure may also prevent writing a durable dropped-update counter, so the missing data cannot be measured completely. +- Recording performs bounded in-process work and one non-waiting lock attempt. It does not promise zero latency, although failure and contention never change the user operation's result. +- The daily safety cap permits ordinary retained data near 248 MiB through D30. The implementation also adds cross-cutting provenance, witness, attribution, aggregation, consent, and schema-maintenance work. + +### Rationale and alternatives + +#### Record no production telemetry + +This avoids the privacy, storage, and implementation costs above, but leaves Q1-Q7 unanswered. Controlled evaluation can determine whether a skill helps in a test scenario; it cannot show production reach, resolution gaps, or hook reliability. + +#### Record every hook and skill invocation + +Hooks are high-volume; Q4 needs rates and distributions, not traces. Resolution edges are low-volume and Q2/Q3 need the actual safe package-to-plugin-to-skill relationship. Aggregating hook and skill activity while retaining safe resolution events preserves that product signal without creating per-invocation rows. -### Extensibility +#### Defer scoped identifiers, provenance, and witnesses until upload + +These mechanisms already serve local measurement and privacy. Q1-Q7 need narrow deduplication and relationship evidence. Provenance prevents private coordinates from reaching an inspectable or shareable telemetry directory. + +Recording plain coordinates now would create weaker local files and permanently lose evidence needed by the measurement questions. `event_id` also gives cumulative metric rows stable identity across snapshot replacement and reset epochs. + +#### Omit exact package versions + +A package name alone cannot distinguish a missing recommendation from a version-specific resolution gap or regression. Exact versions provide that diagnostic only after the PM proves an allowlisted public origin and validates the coordinate; private and local versions remain unnamed. + +#### Record a complete dependency snapshot + +Per-package frequency and safe extension paths answer Q3 without an explicit project fingerprint. Same-file/day/batch order can still reveal co-occurrence locally, but a dependency-set or resolution id would make that linkage direct and persistent. + +#### Use SQLite instead of JSONL + +Low-volume events append, while hook metrics form a small bounded daily snapshot. JSONL keeps both byte-inspectable, preserves unknown versions and malformed lines, and needs no database. The process lock plus atomic snapshot replacement supplies the required concurrency behavior. + +### Unresolved questions + +No telemetry-contract question is intentionally left open for acceptance. Two implementation inputs remain: Step 7 requires team approval of the exact disclosure wording, and Step 5 requires before-and-after hook measurements. + +If either requires changing collection, privacy, or failure semantics, the RFD must be amended before recording is activated. + +### Future possibilities The event catalogue is closed for consent version 1, but new measurements can be added as typed, versioned event families. Every family goes through `Recorder` and remains subject to consent, public-identity rules, storage caps, retention, failure behavior, and the never-record exclusions. +#### Additional agent adapters + Agent-native extension-use payloads remain behind the normalized observation boundary. Version 1 implements only the Claude `Skill` adapter. Supporting another agent requires a thin adapter, native-signal conformance fixtures, capability documentation, and an event schema update; it does not duplicate attribution, aggregation, or storage. -The same model can later support eligible public plugins. A plugin would declare bounded feature names in its manifest and report only core-defined aggregates. Symposium would own and validate the schema, record and use the resulting data, and optionally provide aggregate reports to plugin authors. Plugins could not add arbitrary fields, bypass consent, or write directly to storage. This RFD implements only the event families listed above; plugin reporting requires a separate implementation and disclosure before collection begins. +#### Plugin telemetry + +The same model can later support eligible public plugins. A plugin would declare bounded feature names in its manifest and report only core-defined aggregates. Symposium would own and validate the schema, record and use the data, and optionally provide aggregate reports to plugin authors. -### Boundary with controlled evaluation +Plugins could not add arbitrary fields, bypass consent, or write directly to storage. This RFD implements only the event families listed above. Plugin reporting requires a separate implementation and disclosure before collection begins. -Production telemetry and a future evaluation harness may consume the same normalized extension-use observation but must use separate sinks and contracts. The telemetry sink writes only the daily aggregates defined here. An explicitly started harness may retain detailed per-run evidence in its isolated evaluation workspace without reading telemetry JSONL or relying on rotating telemetry identifiers. +#### Controlled evaluation + +Production telemetry and a future evaluation harness may consume the same normalized extension-use observation, but they must use separate sinks and contracts. The telemetry sink writes only the daily aggregates defined here. + +An explicitly started harness may retain detailed per-run evidence in its isolated evaluation workspace. It does not read telemetry JSONL or rely on rotating telemetry identifiers. This RFD preserves that internal seam but does not define a harness, scenario format, trace format, outcome grader, or token accounting. Controlled with/without comparisons answer whether a skill improved a task; production telemetry does not. -### Boundary with future upload +#### Upload Accepting this RFD is not consent to upload. A future RFD must define transport/authentication, renewed consent and scheduling, retry/idempotency, server retention/access/deletion, reporting thresholds, and incident handling. @@ -326,43 +433,21 @@ These remain proposed pages until implementation lands; shipped design/reference ## Frequently asked questions - - ### What does pseudonymous mean here? -A scoped identifier is derived from a random local secret rather than machine identity, but it still links observations inside one stated purpose and window. Rotation and domain separation limit that linkage; they do not make the local files anonymous. The secret key is not anonymized or included in telemetry; it remains private because possessing it permits candidate identifiers to be recomputed. - -### Why retain scoped identifiers, provenance, and witnesses before upload? - -They already serve local measurement and privacy. Q1-Q7 need narrow deduplication and relationship evidence; provenance prevents private coordinates from reaching an inspectable/shareable telemetry directory. Recording plain coordinates now would create weaker local files and permanently lose the evidence needed by those questions. `event_id` also gives cumulative metric rows stable identity across snapshot replacement and reset epochs. - -### Why include exact package versions? - -A package name alone cannot distinguish a missing recommendation from a version-specific resolution gap or regression. Exact versions provide that diagnostic only after the PM proves an allowlisted public origin and validates the coordinate; private and local versions remain unnamed. +A scoped identifier comes from a random local secret rather than machine identity. It still links observations inside one stated purpose and window. Rotation and domain separation limit that linkage; they do not make the local files anonymous. -### Why aggregate hooks but keep resolution events? - -Hooks are high-volume; Q4 needs rates and distributions, not traces. Resolution edges are low-volume and Q2/Q3 need the actual safe package-to-plugin-to-skill relationship. Aggregating those edges would erase the product signal. +The secret key is private state, not anonymized telemetry. It is not included in telemetry because someone who has it can recompute candidate identifiers. ### Does a completed skill activation mean the skill helped? No. It means an agent successfully activated the installed skill; version 1 can observe this only for Claude. It does not show whether the agent followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill. -### Why not record a complete dependency snapshot? - -Per-package frequency and safe extension paths answer Q3 without an explicit project fingerprint. Same-file/day/batch order can still reveal co-occurrence locally, but a dependency-set or resolution id would make that linkage direct and persistent. - -### Does recording delay or disrupt hooks? - -It performs bounded in-process work and one non-waiting lock attempt, so it does not promise zero latency. It never waits for another recorder, changes the user operation's result, or `fsync`s. Disabled consent, contention, caps, I/O failure, or termination can omit data; absence never proves that an action did not occur. - -### Why JSONL instead of SQLite? - -Low-volume events append, while hook metrics form a small bounded daily snapshot. JSONL keeps both byte-inspectable, preserves unknown versions and malformed lines, and needs no database. The process lock plus atomic snapshot replacement supplies the required concurrency behavior. - ## Implementation plan and status -The seven steps below are PR-sized. Steps 1-6 keep production collection inert; Step 7 activates the current consent version only after every producer and control is present. Tests and benchmarks may construct an enabled recorder only through a test-only API bound to a temporary telemetry home. There is no runtime environment-variable or configuration bypass. +The seven steps below are PR-sized. Steps 1-6 keep production collection inert. Step 7 activates the current consent version only after every producer and control is present. + +Tests and benchmarks may construct an enabled recorder only through a test-only API bound to a temporary telemetry home. There is no runtime environment-variable or configuration bypass. ```text 1. Contract and identity @@ -378,47 +463,66 @@ The seven steps below are PR-sized. Steps 1-6 keep production collection inert; 7. Consent and activation ``` -After Step 2, Steps 3 and 6 may proceed in parallel. Step 4 follows Step 3, Step 5 follows Step 4, and Step 7 waits for Steps 5 and 6. Step 3 isolates the risky core PM type seam from telemetry instrumentation. Steps 4 and 5 remain end-to-end measurement slices, and the plan remains seven PRs. +After Step 2, Steps 3 and 6 may proceed in parallel. Step 4 follows Step 3, Step 5 follows Step 4, and Step 7 waits for Steps 5 and 6. + +Step 3 isolates the risky core PM type seam from telemetry instrumentation. Steps 4 and 5 remain end-to-end measurement slices. The plan remains seven PRs. ### Step 1: Telemetry contract and identity -Replace the dormant event types with the closed producer schema: common fields, bounded witnesses, fixed outcomes/histograms, extension-invocation counters and buckets, and per-kind version dispatch. Add private telemetry-owned `/telemetry-state.toml`, atomic owner-only key creation where supported, scoped HMAC derivation, local 30-day windows, D0-D30 cohorts, and reset primitives. Keep this state outside the inspectable telemetry data directory. Do not add storage or emission. +Replace the dormant event types with the closed producer schema: common fields, bounded witnesses, fixed outcomes and histograms, extension-invocation counters and buckets, and per-kind version dispatch. -Verify all contract examples round-trip; bounds and unknown versions behave as specified; arbitrary/private data cannot serialize; identities separate domains, dimensions, agents, and windows; the same active-window inputs remain stable across recorders and restarts; normal rollover changes subjects without replacing the key; `disable`/`clear` preserve identity state; renewed consent/reset rotate it; and disabled paths create nothing. +Add private telemetry-owned `/telemetry-state.toml`, atomic owner-only key creation where supported, scoped HMAC derivation, local 30-day windows, D0-D30 cohorts, and reset primitives. Keep this state outside the inspectable telemetry data directory. Do not add storage or emission. -- [ ] PR: telemetry contract and scoped identity +Verify: +- All contract examples round-trip, and bounds and unknown versions behave as specified. +- Arbitrary or private data cannot serialize. +- Identities separate domains, dimensions, agents, and windows. The same active-window inputs remain stable across recorders and restarts, while normal rollover changes subjects without replacing the key. +- `disable` and `clear` preserve identity state; renewed consent and reset rotate it. +- Disabled paths create nothing. +- [ ] PR: telemetry contract and scoped identity ### Step 2: Storage and local controls -Add whole-batch event appends, non-waiting process locking, atomically replaced aggregate snapshots, daily caps/reservations, `storage_limit`, and lazy D31 cleanup. Add typed `status`, byte-preserving `show`, `clear`, and `reset-identifiers`. Expose a test-only enabled recorder bound to a caller-supplied temporary telemetry home for integration tests and benchmarks. No production caller writes through the sink yet, and no runtime bypass is added. +Add whole-batch event appends, non-waiting process locking, atomically replaced aggregate snapshots, daily caps and reservations, `storage_limit`, and lazy D31 cleanup. Add typed `status`, byte-preserving `show`, `clear`, and `reset-identifiers` commands. -Verify concurrent complete lines, old-or-new snapshots, whole-operation drops, cap/marker accounting, D30/D31 cleanup, malformed/unknown inspection, private-state permissions and separation, abandoned state/snapshot temporary cleanup, clear/reset semantics, test-only recorder isolation, and that management commands never record themselves. +Expose a test-only enabled recorder bound to a caller-supplied temporary telemetry home for integration tests and benchmarks. No production caller writes through the sink yet, and no runtime bypass is added. -- [ ] PR: telemetry storage and local controls +Verify: +- Concurrent complete lines, old-or-new snapshots, whole-operation drops, and cap/marker accounting. +- D30/D31 cleanup, malformed and unknown inspection, and abandoned state/snapshot temporary cleanup. +- Private-state permissions and separation, clear/reset semantics, and test-only recorder isolation. +- Management commands never record themselves. +- [ ] PR: telemetry storage and local controls ### Step 3: PM provenance and public-identity policy -Extend every PM result with typed provenance and exact coordinates; add core-owned public allowlists and coordinate validation. If an out-of-process PM protocol lands first, carry provenance through it as part of this step. All supported PMs must provide provenance before events are wired. This step emits no telemetry. - -Verify every PM and source class, malformed and wildcard coordinates, public allowlist behavior, unnamed-reason precedence, and that raw URLs and private/local coordinates cannot become public identities. +Extend every package-manager result with typed provenance and exact coordinates. Add core-owned public allowlists and coordinate validation. If an out-of-process package-manager protocol lands first, carry provenance through it as part of this step. -- [ ] PR: package provenance and public identity policy +All supported package managers must provide provenance before events are wired. This step emits no telemetry. +Verify every package manager and source class, malformed and wildcard coordinates, public allowlist behavior, unnamed-reason precedence, and that raw URLs and private or local coordinates cannot become public identities. +- [ ] PR: package provenance and public identity policy ### Step 4: Resolution witnesses, attribution index, and recording -Return safe evidence from the original predicate evaluation, preserving short-circuit and accepted cache behavior; whole-`PredicateSet` caching remains disallowed. Extend evidence through plugin/skill selection, return one structured full-sync report, and construct coherent `resolution_summary`, `package_resolution`, and `extension_resolution` batches. After installation, atomically replace the generated agent-facing attribution index with entries for successful managed skills and marker fingerprints. Ordinary read-only plugin lookup remains silent. +Return safe evidence from the original predicate evaluation while preserving short-circuit and accepted cache behavior. Whole-`PredicateSet` caching remains disallowed. -Verify `all`/`any`/`not` and opaque/limit paths, cache hits without reevaluation, every `extension_match` case, complete package-to-plugin-to-skill paths, whole-batch failure, successful/failed installation indexing, atomic old-or-new index reads, stale/corrupt/fingerprint-mismatch handling, collisions, and exclusion of raw paths, private names, and dependency snapshots from telemetry. +Carry evidence through plugin and skill selection, return one structured full-sync report, and construct coherent `resolution_summary`, `package_resolution`, and `extension_resolution` batches. After installation, atomically replace the generated agent-facing attribution index with successful managed skills and their marker fingerprints. Ordinary read-only plugin lookup remains silent. -- [ ] PR: resolution witnesses, installed attribution, and recording +Verify: +- `all`, `any`, `not`, and opaque/limit paths, plus cache hits without reevaluation. +- Every `extension_match` case and complete package-to-plugin-to-skill paths. +- Whole-batch failure, successful and failed installation indexing, and atomic old-or-new index reads. +- Stale, corrupt, fingerprint-mismatch, and collision handling. +- Raw paths, private names, and dependency snapshots stay out of telemetry. +- [ ] PR: resolution witnesses, installed attribution, and recording ### Step 5: Hook and extension-invocation telemetry @@ -426,26 +530,43 @@ Measure completed Symposium and plugin-hook handling. Merge top-level observatio Add the sink-neutral extension-use observation and Claude `Skill` adapter. Existing pre/post-tool hooks and a targeted failure hook update `extension_invocation_metrics` through the installed attribution index, with independent phase counters, fixed unnamed reasons, a separate 128-public-skill limit, and all-or-nothing attempted/completed session counts. Never emit a per-invocation row. -Measure the hook path before and after. Verify sanitized automatic/manual activation and lifecycle fixtures, unknown schemas, public/private/missing/ambiguous/stale attribution, marker validation, raw-input exclusion, all counter/histogram/cross-row invariants, independently dropped phases, boundary buckets, process merging, rollover/reset, 500-lifecycle boundedness, private/overflow behavior, missing/mixed/over-256 session ids, crash recovery, a worst-case snapshot within 512 KiB, integer overflow drops, a fake second sink, disabled-path short-circuiting, and unchanged agent output on recording failure. +Measure the hook path before and after. Verify: + +- Sanitized automatic and manual activation, lifecycle fixtures, and unknown schemas. +- Public, private, missing, ambiguous, and stale attribution; marker validation; and raw-input exclusion. +- Counter, histogram, and cross-row invariants; independently dropped phases; boundary buckets; process merging; and rollover/reset. +- Bounded behavior across 500 lifecycles, private and overflow buckets, and missing, mixed, or over-256 session ids. +- Crash recovery, a worst-case snapshot within 512 KiB, and integer-overflow drops. +- A fake second sink, disabled-path short-circuiting, and unchanged agent output on recording failure. - [ ] PR: bounded hook and extension-invocation telemetry +### Step 6: Session, configuration, and command telemetry +Add capability-aware `session_start`, daily `agent_configuration`, and eligible top-level `command` rows over the shared recorder. -### Step 6: Session, configuration, and command telemetry +The first non-telemetry recording-capable invocation of each UTC day attempts one all-agent configuration batch from the per-user agent list. It does not inspect agent-owned files. Hook internals, telemetry controls, arguments, and unsafe plugin command names remain excluded. -Add capability-aware `session_start`, daily `agent_configuration`, and eligible top-level `command` rows over the shared recorder. The first non-telemetry recording-capable invocation of each UTC day attempts one all-agent configuration batch from the per-user agent list; it does not inspect agent-owned files. Exclude hook internals, telemetry controls, arguments, and unsafe plugin command names. +Verify: -Verify the capability matrix, Copilot without a session id, Claude-only `Stop`, configuration-list semantics, same-day configuration deduplication, retry after a dropped configuration batch, fixed command vocabulary/public eligibility, and failures before or after command dispatch. +- The capability matrix, Copilot without a session id, and Claude-only `Stop`. +- Configuration-list semantics, same-day configuration deduplication, and retry after a dropped configuration batch. +- Fixed command vocabulary and public eligibility, plus failures before or after command dispatch. - [ ] PR: session reach and command telemetry +### Step 7: Consent, activation, and documentation +Add `consent-version`, one shared team-approved disclosure for `init` and `telemetry enable`, re-consent, explicit non-interactive acknowledgement, `enable` and `disable`, and final production wiring for Steps 4-6. -### Step 7: Consent, activation, and documentation +Review the final wording against the version 1 coverage requirements. The proposed prompt is a complete example, not the implementation string. There is no event-file migration because the dormant recorder was never called. Publish the proposed pages and update the current design and flow chapters plus `md/SUMMARY.md`. -Add `consent-version`, one shared team-approved disclosure for `init` and `telemetry enable`, re-consent, explicit non-interactive acknowledgement, `enable`/`disable`, and final production wiring for Steps 4-6. Review the final wording against the version 1 coverage requirements; the proposed prompt is a complete example, not the implementation string. There is no event-file migration because the dormant recorder was never called. Publish the proposed pages and update current design/flow chapters and `md/SUMMARY.md`. +Verify: -Verify new/existing configuration states, documented review of the final disclosure against every required coverage point, a snapshot proving `init` and `telemetry enable` use the identical approved text, interactive/non-interactive flows, no partial or disabled collection, absence of a runtime consent bypass, raw inspection/expiry, the full CLI/integration suite, hook-path benchmark, formatting, clippy, workspace tests, mdBook, and orphan checks. +- New and existing configuration states, plus documented review of the final disclosure against every required coverage point. +- A snapshot proving that `init` and `telemetry enable` use identical approved text. +- Interactive and non-interactive flows, no partial or disabled collection, and no runtime consent bypass. +- Raw inspection and expiry, the full CLI and integration suite, and the hook-path benchmark. +- Formatting, clippy, workspace tests, mdBook, and orphan checks. - [ ] PR: telemetry consent and recording activation diff --git a/md/rfds/telemetry-recording/contract/recorded-data.md b/md/rfds/telemetry-recording/contract/recorded-data.md index 3f916e38..b23aae1c 100644 --- a/md/rfds/telemetry-recording/contract/recorded-data.md +++ b/md/rfds/telemetry-recording/contract/recorded-data.md @@ -1,14 +1,13 @@ # What Symposium records -Telemetry is off by default, per-user, and stored only on your machine. Nothing described on this page is uploaded. `cargo agents telemetry show` exposes the exact stored bytes, and `cargo agents telemetry clear` deletes event and aggregate-metric files. The [never-record list](#what-is-never-recorded) summarizes the exclusions before the field-by-field contract. +Symposium records telemetry only after you opt in. The records stay on your machine; nothing described on this page is uploaded. `cargo agents telemetry show` displays the stored bytes, and `cargo agents telemetry clear` deletes event and aggregate-metric files. The [never-record list](#what-is-never-recorded) summarizes the exclusions. -This page is the exhaustive producer contract. If a field is not listed here, Symposium does not write it as telemetry under consent version 1. +This page is the complete producer contract for consent version 1. If a field is not listed here, Symposium does not write it as telemetry. ## Common fields Every JSONL row has: - | Field | Example | Meaning | | ----------- | --------------- | --------------------------------------- | | `v` | `1` | Schema version for this row kind. | @@ -17,8 +16,9 @@ Every JSONL row has: | `day` | `2026-08-03` | UTC calendar day. | | `symposium` | `0.4.0` | Symposium version that wrote the event. | +Completed operational events (`session_start` and `command`) also have `at`, an RFC3339 UTC timestamp truncated to one second. Resolution, configuration, and aggregate metric rows have only `day`. -Completed operational events (`session_start` and `command`) also have `at`, an RFC3339 UTC timestamp truncated to one second. Resolution, configuration, and aggregate metric rows have only `day`. Counters and durations are non-negative JSON integers fitting an unsigned 64-bit value. Arithmetic is checked; an overflowing batch or observation is dropped rather than wrapped. +Counters and durations are non-negative JSON integers that fit an unsigned 64-bit value. Symposium checks arithmetic and drops an overflowing batch or observation instead of wrapping the value. `event_id` exists to deduplicate a future retry of the same event or identify one cumulative metric row. A metric row keeps the `event_id` minted when its dimension first appears that UTC day as the row is rewritten. It is not an installation, session, account, or project identifier. @@ -41,10 +41,17 @@ These are independent row-shape examples, not one coherent operation or batch. T ## Scoped identifiers -When enabled recording first needs identity state, Symposium stores a random secret key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). The same state persists the current identifier-window and return-cohort anchors. Every recorder reads this state under the telemetry lock, so identical domain, window, and dimension inputs produce the same subject across processes and restarts. Normal 30-day rollover changes the window input without replacing the key; renewed consent or `telemetry reset-identifiers` replaces it. `telemetry disable` and `telemetry clear` preserve the key and anchors. +### Key and rotation + +When an enabled recorder first needs identity state, Symposium stores a random secret key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). The same state holds the current identifier-window and return-cohort anchors. Every recorder reads it under the telemetry lock, so identical domain, window, and dimension inputs produce the same subject across processes and restarts. + +Normal 30-day rollover changes the window input without replacing the key. Renewed consent or `telemetry reset-identifiers` replaces it. `telemetry disable` and `telemetry clear` preserve the key and anchors. -This file is separate from the inspectable `/telemetry/` data directory and has owner-only permissions where the platform supports them. Symposium derives keyed identifiers for narrow purposes. The key is a secret rather than anonymized telemetry: it is not written into events, printed by telemetry commands, or derived from your machine, and possession permits candidate identifiers to be recomputed. +This file is separate from the inspectable `/telemetry/` data directory and has owner-only permissions where the platform supports them. The key is private state, not anonymized telemetry. It is not written into events, printed by telemetry commands, or derived from your machine. Someone who has the key can recompute candidate identifiers. +### What identifiers can link + +Symposium derives each identifier for one narrow purpose: | Identifier | What it can link | Rotation | | ------------------- | -------------------------------------------------------------- | --------------------------------- | @@ -57,8 +64,9 @@ This file is separate from the inspectable `/telemetry/` data direct | `plugin_subject` | Daily hook aggregates for one eligible public plugin | 30 days | | `command_subject` | Repeated use of one eligible command | 30 days | +These values are pseudonymous, not anonymous: they deliberately permit limited linking inside the stated scope. `retention_subject` can link observed sessions across agents for D0-D30; this is the single exception needed for return measurement. No identifier links that cohort, a package subject, and a command subject, and there is no workspace id. -These values are pseudonymous, not anonymous: they deliberately permit limited linking inside the stated scope. `retention_subject` can link observed sessions across agents for D0-D30; this is the single exception needed for return measurement. No identifier links that cohort, a package subject, and a command subject, and there is no workspace id. However, all lines in your local telemetry directory come from your Symposium home; file order and same-day events can still suggest which observations happened together. +All lines in your local telemetry directory still come from your Symposium home. File order and same-day events can therefore suggest which observations happened together. ## Version 1 public identity allowlists @@ -75,8 +83,7 @@ Only the following stable labels can make package, plugin, skill, or plugin-comm ### `session_start` -A registered Symposium session-start hook completed. - +This row records a completed registered Symposium session-start hook. | Field | Values | Meaning | | ------------------- | ---------------------------------------------- | ------------------------------------------------------- | @@ -89,15 +96,15 @@ A registered Symposium session-start hook completed. | `retention_subject` | scoped id | Deduplicates this D0-D30 observed-session cohort. | | `cohort_day` | integer `0` through `30` | UTC days since the cohort's first observed session. | - GitHub Copilot does not currently supply a session id. OpenCode and Goose do not currently call Symposium through a registered session-start hook, so they do not produce this event. -These rows, not `hook_metrics` rows whose `hook` is `session_start`, are authoritative for observed-session and return measurements. For each `retention_subject`, the first row establishes D0; D1, D7, or D30 is present when at least one later session-start row has that `cohort_day`, regardless of agent or vendor session id. Multiple rows on the same cohort day count once. The aggregate hook rows measure only session-start hook reliability and latency. +These rows, not `hook_metrics` rows whose `hook` is `session_start`, are authoritative for observed-session and return measurements. For each `retention_subject`, the first row establishes D0. D1, D7, or D30 is present when at least one later session-start row has that `cohort_day`, regardless of agent or vendor session id. Multiple rows on the same cohort day count once. -### `agent_configuration` +The aggregate hook rows measure only session-start hook reliability and latency. -A daily observation of whether a supported agent is configured for Symposium. +### `agent_configuration` +This row records whether a supported agent is configured for Symposium that day. | Field | Values | Meaning | | --------------- | ------------------------------------------------------------------- | -------------------------------------------------------- | @@ -107,15 +114,15 @@ A daily observation of whether a supported agent is configured for Symposium. | `arch` | `x86_64`, `aarch64`, `other` | Architecture class for the running build. | | `agent_subject` | scoped id | Deduplicates one installation for this agent and window. | +On the first non-telemetry recording-capable invocation each UTC day, Symposium attempts one whole batch containing all seven agents. `configured` means the agent is present in Symposium's per-user configuration at that moment; Symposium does not inspect agent-owned configuration files. -On the first non-telemetry recording-capable invocation each UTC day, Symposium attempts one whole batch containing all seven agents. `configured` means the agent is present in Symposium's per-user configuration at that moment; Symposium does not inspect agent-owned configuration files. A successful batch is the day's snapshot, so later configuration changes appear on the next UTC day's snapshot. If locking, storage, or I/O drops the batch, it remains outstanding and a later eligible invocation retries it. +A successful batch is the day's snapshot, so later configuration changes appear on the next UTC day's snapshot. If locking, storage, or I/O drops the batch, it remains outstanding and a later eligible invocation retries it. `configured: true` does not claim that an agent session ran or that the agent-owned integration files remain intact. ### `resolution_summary` -A full sync reached an observed result after session start, manual sync, `use`, or removal. Package/plugin/skill counts are distinct coordinates within the sync, not duplicate declarations. - +This row records the result of a full sync after session start, manual sync, `use`, or removal. Package, plugin, and skill counts are distinct coordinates within the sync, not duplicate declarations. | Field | Values | Meaning | | ------------------ | ----------------------------------------------- | ----------------------------------------------------------------- | @@ -134,13 +141,11 @@ A full sync reached an observed result after session start, manual sync, `use`, `unnamed_package_reasons` has exactly `private_registry`, `git`, `path`, `workspace`, `unknown_source`, and `invalid_coordinate` counters. Each unnamed package increments exactly one. Source provenance takes precedence; `invalid_coordinate` is used only for an allowlisted public source with a malformed name or a missing, wildcard, or invalid exact version. The counters sum to `unnamed_packages`. - Read-only extension lookup on ordinary hook calls does not produce this event. ### `package_resolution` -One eligible public resolution-input package observed during a full sync. - +This row records one eligible public package used as resolution input during a full sync. | Field | Values | Meaning | | ------------------- | ----------------------- | -------------------------------------------------------------------- | @@ -150,15 +155,19 @@ One eligible public resolution-input package observed during a full sync. | `extension_match` | `public`, `unnamed_only`, `none` | What kind of resolved extension, if any, the package contributed to. | | `package_subject` | scoped id | Deduplicates this exact coordinate for 30 days. | +A package is named only when its package manager reports provenance matching a reviewed public-registry allowlist. Registry URLs themselves are not recorded. + +`extension_match` describes what the package contributed: -A package is named only when its package manager reports provenance matching a reviewed public-registry allowlist. Registry URLs themselves are not recorded. `extension_match: public` means at least one eligible public extension matched, including when unnamed content also matched. `unnamed_only` means extension content matched but none was eligible to name. `none` means no resolved extension matched. +- `public`: at least one eligible public extension matched, including when unnamed content also matched. +- `unnamed_only`: extension content matched, but none was eligible to name. +- `none`: no resolved extension matched. Events measure packages independently. Symposium does not record a workspace/resolution id, but the summary and contiguous relationship batch can still make the public package set for one sync recoverable, especially when only one sync occurred that day. Review the whole file before sharing it. ### `extension_resolution` -One public plugin or skill and one safe path that selected it. - +This row records one public plugin or skill and one safe path that selected it. | Field | Values | Meaning | | ------------------- | --------------------------- | --------------------------------------------------- | @@ -168,10 +177,8 @@ One public plugin or skill and one safe path that selected it. | `path` | bounded typed nodes | Actual safe package/predicate/extension chain. | | `extension_subject` | scoped id | Deduplicates this safe target/path for 30 days. | - Path nodes are limited to: - | Node `type` | Recorded content | | ----------- | -------------------------------------------------------------------- | | `package` | Eligible public ecosystem, name, and exact version. | @@ -181,15 +188,15 @@ Path nodes are limited to: | `not` | Marker only; the child is not recorded. | | `opaque` | Fixed reason: `private_source`, `non_package_predicate`, or `limit`. | +Shell commands, paths, environment variables, custom predicate names or arguments, and private package or extension names never enter a path. An opaque marker can represent their position. -Shell commands, paths, environment variables, custom predicate names/arguments, and private package/extension names never enter a path. Their position can be represented by an opaque marker. Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`; the complete path is also limited to 16 evidence leaves and 4 KiB. This limit does not refer to filesystem path components; filesystem paths are never recorded. +Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`. The complete path is also limited to 16 evidence leaves and 4 KiB. Evidence depth does not count filesystem components; filesystem paths are never recorded. This event says an extension resolved. It does not say that an agent read or used the extension; a matching `extension_invocation_metrics` aggregate separately reports observed agent activation. Version 1 can produce that aggregate only for Claude. ### `hook_metrics` -The cumulative aggregate for one UTC day, agent, hook surface, and active identifier epoch. An epoch normally lasts 30 days but ends early on identifier reset or renewed consent. A reset can therefore leave two rows for the same agent/hook/day with different `hook_subject` values. A completed hook updates the current row in place; Symposium does not append one telemetry row per invocation. - +This cumulative row combines completed hook observations for one UTC day, agent, hook surface, and active identifier epoch. An epoch normally lasts 30 days but ends early after identifier reset or renewed consent. A reset can therefore leave two rows for the same agent, hook, and day with different `hook_subject` values. Symposium updates the current row instead of appending one telemetry row per invocation. | Field | Values | Meaning | | --------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | @@ -205,13 +212,13 @@ The cumulative aggregate for one UTC day, agent, hook surface, and active identi | `identified_sessions_non_ok` | integer, optional | Those sessions with at least one non-`ok` observation; present only when counts are complete. | | `hook_subject` | scoped id | Links this agent/hook dimension inside its 30-day identifier window. | +`outcomes` and the duration histogram each sum to `invocations`. Exactly one outcome counter advances for each completed observation. Precedence is `internal_error`, then `blocked`, then `plugin_error`, then `ok`. -`outcomes` and the duration histogram each sum to `invocations`. The top-level outcome precedence is `internal_error`, then `blocked`, then `plugin_error`, then `ok`; exactly one counter advances per completed observation. Session counts remain complete only if every contributing observation supplied a session id and neither of the two sets exceeded 256 distinct ids for this row. On the first missing id or overflow, Symposium discards both sets, writes `session_counts_complete: false`, and omits both counts for the rest of that day. The raw and keyed session ids are never written into the aggregate file. +Session counts remain complete only when every contributing observation supplies a session id and neither set exceeds 256 distinct ids for the row. On the first missing id or overflow, Symposium discards both sets, writes `session_counts_complete: false`, and omits both counts for the rest of that day. Raw and keyed session ids are never written into the aggregate file. ### `plugin_hook_metrics` -The cumulative aggregate for one UTC day, agent, hook surface, active identifier epoch, and bounded plugin bucket. A reset can leave multiple otherwise identical `unnamed` or `overflow` rows distinguished only by `event_id`; public rows also receive a new `plugin_subject`. - +This cumulative row combines plugin-hook observations for one UTC day, agent, hook surface, active identifier epoch, and bounded plugin bucket. A reset can leave otherwise identical `unnamed` or `overflow` rows distinguished only by `event_id`; public rows also receive a new `plugin_subject`. | Field | Values | Meaning | | ----------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | @@ -230,8 +237,21 @@ The cumulative aggregate for one UTC day, agent, hook surface, active identifier | `identified_sessions_non_ok` | integer, optional | Those sessions with at least one non-`ok` attempt; present only when counts are complete. | | `plugin_subject` | scoped id, conditional | Present only with an eligible public plugin. | +#### Counting rules + +`attempts` counts terminal results. `executions` counts the subset that started child execution. A preparation failure is terminal `error` with no execution. `outcomes` and `prepare_ms` each sum to `attempts`; `execute_ms` sums to `executions`. Across every epoch and bucket for one agent, hook, and day, plugin `attempts` sum to the corresponding top-level `plugins_completed`. + +`blocked` means the plugin requested a block. `error` covers a closed preparation or execution failure. `ok` is every other completed attempt. The same 256-id all-or-nothing session rule used by `hook_metrics` applies. + +#### Plugin identity and row limits + +Eligible public plugins use `{source, name}` coordinates from the reviewed allowlist. Private, local, invalid, or otherwise ineligible plugins merge into one `unnamed` row per agent, hook, epoch, and day and expose no identity. + +At most 128 named public-plugin rows may appear across all agents, hooks, and identifier epochs in one UTC day. Attempts for later rows merge into `overflow` rows per agent, hook, and epoch and expose no identity. Identifier reset does not reset this daily limit. + +Named rows are first-observed, not sampled, so earlier-in-day public plugins are overrepresented when the limit is reached. Analysis must report overflow and must not treat the named subset as random. -`attempts` counts only terminal results and `executions` counts the subset that started child execution. A preparation failure is terminal `error` with no execution. `outcomes` and `prepare_ms` each sum to `attempts`; `execute_ms` sums to `executions`. Across every epoch and bucket for one agent/hook/day, plugin `attempts` sum to the corresponding top-level `plugins_completed`. `blocked` means the plugin requested a block, `error` covers any closed preparation/execution failure, and `ok` is every other completed attempt. The same 256-id all-or-nothing session rule applies. Eligible public plugins use `{source, name}` coordinates from the reviewed allowlist. Private, local, invalid, or otherwise ineligible plugins merge into one `unnamed` row per agent/hook/epoch/day and expose no identity. At most 128 named public-plugin rows may appear across all agents, hooks, and identifier epochs in one UTC day; attempts for later rows merge into `overflow` rows per agent/hook/epoch and expose no identity. Identifier reset does not reset this daily limit. Named rows are first-observed, not sampled: earlier-in-day public plugins are overrepresented when the limit is reached. Analysis must report overflow and must not treat the named subset as random. +### Rules shared by hook aggregates A latency histogram is exactly: @@ -241,13 +261,15 @@ A latency histogram is exactly: The nine non-overlapping millisecond buckets mean `<=5`, `(5,10]`, `(10,25]`, `(25,50]`, `(50,100]`, `(100,250]`, `(250,500]`, `(500,1000]`, and `>1000`. Bounds and count-array length cannot vary. Fixed histograms can be merged correctly across installations; locally calculated percentiles cannot. -Neither aggregate kind has `at`, an invocation id, a raw or scoped session id, or an individual invocation's time or outcome. No hook input, output, stdout, stderr, error text, or numeric exit detail is recorded. Symposium adds no timeout as part of telemetry. A host termination, lock conflict, storage limit, or I/O failure can omit an observation. Aggregate counts are therefore lower bounds. No durable dropped-update counter can cover every loss mode because lock contention, termination, and I/O failure can also prevent writing that counter; counting only cap rejections would imply false completeness. +Neither aggregate kind has `at`, an invocation id, a raw or scoped session id, or an individual invocation's time or outcome. No hook input, output, stdout, stderr, error text, or numeric exit detail is recorded. Symposium adds no timeout as part of telemetry. + +A host termination, lock conflict, storage limit, or I/O failure can omit an observation, so aggregate counts are lower bounds. No durable dropped-update counter can cover every loss mode because lock contention, termination, and I/O failure can also prevent writing that counter. Counting only cap rejections would imply false completeness. These rows do reveal exact daily counts for each hook surface. In particular, `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` counts are proxies for daily tool and prompt activity even though no individual activity record exists. ### `extension_invocation_metrics` -The cumulative aggregate for one UTC day, supported agent, active identifier epoch, and bounded skill bucket. Version 1 supports only Claude Code and accepts only its fixture-tested `Skill` signal. Symposium does not append one row per invocation. +This cumulative row combines skill-invocation observations for one UTC day, supported agent, active identifier epoch, and bounded skill bucket. Version 1 supports only Claude Code and accepts only its fixture-tested `Skill` signal. Symposium does not append one row per invocation. | Field | Values | Meaning | | ------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | @@ -265,9 +287,25 @@ The cumulative aggregate for one UTC day, supported agent, active identifier epo | `identified_sessions_completed` | integer, optional | Distinct sessions with a completed observation; present only when session counts are complete. | | `extension_subject` | scoped id, conditional | Present only for a public bucket; matches the selected safe resolution path for 30 days. | -`attempted`, `completed`, and `failed` are independent lower bounds. Each hook phase updates the snapshot separately, so loss of one update means completed plus failed need not equal attempted and may exceed it. The two session counts have the same independence. `failed` advances only when the supported agent emits its targeted failure signal; for version 1, that is Claude's `PostToolUseFailure:Skill` event. Denial, termination, or a missing terminal observation is not inferred as failure. Completion means the agent successfully activated the skill; it does not show whether the agent followed its instructions or improved the task result. +#### Count semantics + +`attempted`, `completed`, and `failed` are independent lower bounds. Each hook phase updates the snapshot separately. If one update is lost, completed plus failed need not equal attempted and may exceed it. The two session counts have the same independence. + +`failed` advances only when the supported agent emits its targeted failure signal. For version 1, that is Claude's `PostToolUseFailure:Skill` event. Symposium does not infer failure from denial, termination, or a missing terminal observation. Completion means that the agent activated the skill; it does not show whether the agent followed its instructions or improved the task result. + +#### Attribution + +Public attribution comes only from the generated installation index and a matching Symposium marker fingerprint. Unnamed reasons have these meanings: + +- `ineligible`: the installed skill was private, local, invalid, or otherwise unsafe to name. +- `not_indexed`: the index was valid and readable but contained no matching agent-facing identifier. +- `attribution_unavailable`: the index was missing, corrupt, or stale. +- `ambiguous`: more than one entry matched. +- `invalid_signal`: the Claude payload did not match the validated schema. -Public attribution comes only from the generated installation index and a matching Symposium marker fingerprint. `ineligible` combines private, local, invalid, and otherwise unsafe-to-name installed skills. `not_indexed` means the index was valid and readable but contained no entry matching the agent-facing identifier. `attribution_unavailable` means the index was missing, corrupt, or stale. `ambiguous` means more than one entry matched. `invalid_signal` means the Claude payload did not match the validated schema. No reason exposes the raw identifier. +No reason exposes the raw identifier. + +#### Row and session limits At most 128 public-skill rows may be named across identifier epochs in one UTC day. Later public skills merge into one `overflow` row per agent and epoch. Each fixed unnamed reason produces at most one row per agent and epoch. The public subset is first-observed, not sampled; analysis must report overflow and include unnamed counts when interpreting the public denominator. @@ -277,8 +315,7 @@ This aggregate has no `at`, duration, raw or scoped session id, invocation id, t ### `command` -One eligible top-level user command completed. - +This row records one completed eligible top-level user command. | Field | Values | Meaning | | ----------------- | ---------------- | ------------------------------------------------------------- | @@ -288,7 +325,6 @@ One eligible top-level user command completed. | `outcome` | `ok`, `error` | Closed result. | | `command_subject` | scoped id | Deduplicates this command for 30 days. | - Arguments are never recorded. Internal `hook`, all `telemetry` commands, and ineligible external/plugin commands do not produce command events. Built-in names are `init`, `sync`, `search`, `use`, `status`, `plugin_sync`, `plugin_list`, `plugin_show`, `plugin_validate`, `self_update`, and `crate_info`: @@ -305,7 +341,9 @@ An eligible plugin command contains only its reviewed public-source label, publi ### `storage_limit` -The next complete low-volume event batch did not fit in the shared daily 8 MiB allowance. In addition to the common fields, `dropped_operation` is `session_start`, `manual_sync`, `use`, `remove`, `init`, `configuration`, or `command`, identifying the top-level operation whose batch was rejected. This event appears at most once per UTC day and the marker itself counts toward 8 MiB. An aggregate-metric update that would exceed its separate 512 KiB maximum or the remaining shared allowance is dropped without this marker and does not stop low-volume recording. The marker does not report lock-contention, aggregate-update, or crash losses. +This row means that the next complete low-volume event batch did not fit in the shared daily 8 MiB allowance. In addition to the common fields, `dropped_operation` is `session_start`, `manual_sync`, `use`, `remove`, `init`, `configuration`, or `command`. It identifies the top-level operation whose batch was rejected. + +The row appears at most once per UTC day, and the marker itself counts toward 8 MiB. An aggregate-metric update that would exceed its separate 512 KiB maximum or the remaining shared allowance is dropped without this marker and does not stop low-volume recording. The marker does not report lock-contention, aggregate-update, or crash losses. ## What is never recorded @@ -324,12 +362,36 @@ The next complete low-volume event batch did not fit in the shared daily 8 MiB a ## Storage and expiry -Low-volume events are appended as JSON lines in `events-YYYY-MM-DD.jsonl` under the inspectable `/telemetry/` data directory (default `~/.symposium/telemetry/`). Current cumulative hook, plugin-hook, and extension-invocation aggregates are JSON lines in `metrics-YYYY-MM-DD.jsonl`; the bounded snapshot is rewritten atomically after a merge. The lock in this directory also guards sibling private state. A process uses one non-waiting lock attempt and may drop its complete buffered event batch or aggregate observation rather than delay your hook or command. Recording failures never change the user operation's result. +### Data files and recording + +Low-volume events are appended as JSON lines in `events-YYYY-MM-DD.jsonl` under the inspectable `/telemetry/` data directory (default `~/.symposium/telemetry/`). Current cumulative hook, plugin-hook, and extension-invocation aggregates are JSON lines in `metrics-YYYY-MM-DD.jsonl`. Symposium rewrites this bounded snapshot atomically after a merge. -The event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB daily allowance. The allowance is a safety ceiling, not expected volume or preallocation: it bounds damage from a producer bug or unexpectedly large resolution batch. Together with D31 expiry, it bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics may use at most 512 KiB, so high-volume hook and skill activity cannot consume the allowance reserved for resolution and configuration events. A file is eligible for deletion only when `current_utc_day - file_utc_day > 30`: a D0 file remains throughout D30 and is first eligible on D31. Cleanup is lazy, so an old file remains until a recording-capable invocation or telemetry command runs. +The lock in the telemetry directory also guards sibling private state. A recorder makes one non-waiting lock attempt. It may drop a complete buffered event batch or aggregate observation rather than delay your hook or command. Recording failures never change the user operation's result. -The sibling private `/telemetry-state.toml` holds the identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, bounded keyed session sets, and snapshot contribution counts used to calculate complete distinct-session counts. It is atomically created and replaced with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. The sets are not printed or copied into metric rows, are discarded at UTC-day rollover, and are removed by `telemetry clear` or `telemetry reset-identifiers`. State is atomically replaced before the corresponding metric snapshot; if a later snapshot write fails, a contribution-count mismatch on the next update discards the sets and permanently marks the row's session counts incomplete for that day. `telemetry clear` deletes event and aggregate-metric files and atomically rewrites private state to remove pending sets, while preserving the identity key and current anchors. `telemetry reset-identifiers` rotates future identifiers and starts a new retention cohort. `telemetry disable` stops recording; existing files remain unless its interactive clear offer is accepted or `telemetry clear` removes them later. +### Daily limits and retention + +The event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB daily allowance. This allowance is a safety ceiling, not expected volume or preallocation. It bounds damage from a producer bug or unexpectedly large resolution batch. + +Aggregate metrics may use at most 512 KiB, so high-volume hook and skill activity cannot consume the allowance reserved for resolution and configuration events. + +A file is eligible for deletion only when `current_utc_day - file_utc_day > 30`. A D0 file remains throughout D30 and is first eligible on D31. Together with D31 expiry, the daily limit bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Cleanup is lazy, so an old file remains until a recording-capable invocation or telemetry command runs. + +### Private state + +The sibling private `/telemetry-state.toml` holds the identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, bounded keyed session sets, and snapshot contribution counts used to calculate complete distinct-session counts. + +Symposium creates and replaces it atomically with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. + +The session sets are not printed or copied into metric rows. Symposium discards them at UTC-day rollover and removes them when `telemetry clear` or `telemetry reset-identifiers` runs. State is replaced before the corresponding metric snapshot. If a later snapshot write fails, a contribution-count mismatch on the next update discards the sets and permanently marks the row's session counts incomplete for that day. + +`telemetry clear` deletes event and aggregate-metric files and rewrites private state to remove pending sets while preserving the identity key and current anchors. `telemetry reset-identifiers` rotates future identifiers and starts a new retention cohort. `telemetry disable` stops recording; existing files remain unless the user accepts its interactive clear offer or runs `telemetry clear` later. + +### Installation index The generated `/.symposium/index-v1.json` file is installation state, not telemetry. It may contain local agent-facing identifiers, installed-directory fingerprints, eligibility, and safe public attribution needed to identify the skill Symposium installed. It is gitignored, atomically replaced after sync, excluded from `telemetry show` and `telemetry clear`, and not uploaded by this RFD. -Public names/versions and exact counts can be identifying when they are unusual. Exact daily hook counts disclose approximate prompt/tool activity by surface, and extension-invocation rows disclose exact skill-attempt, completion, and failure counts. Lock contention, process termination, full files, and I/O failures can make the log incomplete. The records are therefore pseudonymous, locally associated, and best-effort; they must not be described as anonymous or as proof that an unrecorded action did not happen. +### Limits of interpretation + +Public names, versions, and exact counts can be identifying when they are unusual. Exact daily hook counts disclose approximate prompt/tool activity by surface. Extension-invocation rows disclose exact skill-attempt, completion, and failure counts. + +Lock contention, process termination, full files, and I/O failures can make the log incomplete. The records are therefore pseudonymous, locally associated, and best-effort. They must not be described as anonymous or as proof that an unrecorded action did not happen. diff --git a/md/rfds/telemetry-recording/reference/configuration.md b/md/rfds/telemetry-recording/reference/configuration.md index d0af558c..c8b0abc2 100644 --- a/md/rfds/telemetry-recording/reference/configuration.md +++ b/md/rfds/telemetry-recording/reference/configuration.md @@ -1,6 +1,8 @@ # Telemetry configuration -Telemetry consent is a per-user setting in `~/.symposium/config.toml`. Symposium reads it only from the user configuration file; project configuration cannot enable, disable, or grant consent for telemetry. +Telemetry consent belongs to the current user. Symposium reads it from the user configuration file, normally `~/.symposium/config.toml`. Project configuration cannot enable, disable, or grant consent for telemetry. + +## Consent states ```toml [telemetry] @@ -8,16 +10,12 @@ enabled = true consent-version = 1 ``` - | Key | Type | Default | Meaning | | ----------------- | ---------------- | ------- | --------------------------------------------------- | | `enabled` | boolean | `false` | The user's stored choice to permit local recording. | | `consent-version` | positive integer | absent | Disclosure version the user acknowledged. | - -Both values must permit collection. For a binary whose current disclosure is -version 1: - +Symposium records telemetry only when `enabled` is true and `consent-version` matches the disclosure used by the current binary. For consent version 1: | Configuration | Effective state | | -------------------------------------------- | -------------------------------- | @@ -25,17 +23,28 @@ version 1: | `enabled = true` with absent/non-`1` version | Consent required; record nothing | | `enabled = true` and `consent-version = 1` | Enabled | +The binary chooses the current consent version. The configuration value records which disclosure the user accepted; it cannot select an arbitrary version. The [version 1 disclosure requirements](./telemetry-command.md#disclosure-requirements) define what that disclosure covers. + +The complete prompt on that page is an example, not fixed implementation text. Interactive `init` and `telemetry enable` present the same final text approved by the team. -The current version is owned by the binary, not accepted as an arbitrary configuration value. The [version 1 disclosure requirements](./telemetry-command.md#disclosure-requirements) are authoritative; the full prompt on that page is a non-normative example. Interactive `init` and `telemetry enable` present the same team-approved final text. +## When consent must be renewed -A future release may require a higher version after a change to collected categories, linkability, timestamp precision, public-name eligibility, retention, or a normative exclusion. It records nothing under an older acknowledgement until you consent again. +A future release may require a higher consent version after changing the recorded categories, linkability, timestamp precision, public-name eligibility, retention, or a normative exclusion. Until the user accepts that version, the effective state is `Consent required` and Symposium records nothing. Adding an agent enum value creates a new schema version for each affected event kind so typed readers do not reinterpret old schemas. It does not by itself require renewed consent when the recorded fields, categories, timestamp precision, and correlation boundaries remain unchanged. New fields, hook surfaces, or linkage for that agent do require a higher consent version. Accepting a newer consent version rotates the telemetry identity key and starts a new D0-D30 return cohort. Existing event and aggregate-metric files are neither rewritten nor deleted, but their scoped identifiers cannot link to later rows. +## Changing telemetry state + `cargo agents telemetry enable` presents the team-approved disclosure for the current version before writing these values. Interactive `cargo agents init` presents the same text for new users and existing unversioned opt-ins, defaulting to no. Non-interactive `init` does not grant or upgrade consent. -`cargo agents telemetry disable` sets `enabled = false` without deleting existing telemetry data files. Consent configuration is separate from the random identity key and current identifier-window/cohort anchors in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). This state is stored outside the inspectable `/telemetry/` data directory and uses owner-only permissions where supported. It persists across `disable` and `clear` so identifiers remain consistent inside the active window; it is not configuration, and Symposium never reads it from project configuration. +`cargo agents telemetry disable` sets `enabled = false` without deleting existing telemetry data files. + +## Related private and installation state + +Consent configuration is separate from the random identity key and current identifier-window and cohort anchors in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). This state sits outside the inspectable `/telemetry/` data directory and uses owner-only permissions where supported. + +It persists across `disable` and `clear`, keeping identifiers consistent inside the active window. It is not configuration, and Symposium never reads it from project configuration. Symposium-managed project skills may also have a generated `/.symposium/index-v1.json` installation index. It associates the agent-facing skill identifier with the resolved installation that sync produced. This file is gitignored installation state, not project configuration or telemetry; changing it cannot grant consent, and telemetry commands do not inspect or delete it. diff --git a/md/rfds/telemetry-recording/reference/telemetry-command.md b/md/rfds/telemetry-recording/reference/telemetry-command.md index 47444719..8c92096d 100644 --- a/md/rfds/telemetry-recording/reference/telemetry-command.md +++ b/md/rfds/telemetry-recording/reference/telemetry-command.md @@ -1,6 +1,6 @@ # `cargo agents telemetry` -Manage opt-in, per-user local telemetry. See [What Symposium records](../contract/recorded-data.md) for the exhaustive field list and [Telemetry configuration](./configuration.md) for consent semantics. +Use this command to manage per-user local telemetry: check whether it is enabled, opt in, inspect what is stored, stop recording, delete recorded data, or rotate future identifiers. See [What Symposium records](../contract/recorded-data.md) for the exhaustive field list and [Telemetry configuration](./configuration.md) for consent semantics. Telemetry is off by default. Nothing is uploaded. @@ -15,11 +15,13 @@ cargo agents telemetry clear cargo agents telemetry reset-identifiers ``` +Start with `status`. If you choose to participate, run `enable` and review the disclosure. Use `show` whenever you want to inspect the stored records. `disable` stops future recording, while `clear` deletes recorded data. + Telemetry management commands are never recorded as command telemetry. ## `status` -`status` shows the effective consent state and a physical/typed summary of local event and aggregate-metric files: +Use `status` to see the effective consent state and a summary of the local event and aggregate-metric files: ```console $ cargo agents telemetry status @@ -39,18 +41,26 @@ Stored files/bytes and physical lines cover daily event and aggregate-metric fil ## `enable` +`enable` presents the current disclosure and asks whether to begin local recording. It does not enable telemetry unless the user explicitly accepts. + ### Disclosure requirements Before asking for consent, the team-approved version 1 disclosure must make these points clear: - Telemetry is off by default, requires an explicit opt-in, and defaults to no in both interactive entry points. -- Recorded categories include observed sessions and configured agents; Symposium version, agent labels, and platform classes; public package names and exact versions; public resolution relationships and aggregate sync results; exact daily hook, plugin-hook, and agent skill-activation metrics, with structured skill activation available only for Claude in version 1; completed eligible commands without arguments; and storage-limit markers. +- The recorded categories are: + - observed sessions and configured agents, including Symposium version, agent labels, and platform classes; + - public package names and exact versions, public resolution relationships, aggregate sync results, and reasons packages remain unnamed; + - exact daily hook, plugin-hook, and agent skill-activation metrics, with structured skill activation available only for Claude in version 1; and + - completed eligible commands without arguments, plus storage-limit markers. - Exact hook-surface counts can approximate daily prompt/tool activity. The disclosure distinguishes second-precision session/command timestamps from day-only aggregate and resolution rows. - Purpose-scoped pseudonymous identifiers permit the stated links for up to 30 days, including one D0-D30 observed-session cohort across agents; they are not anonymous or a global project/workspace identity. - The main exclusions are stated plainly, including prompt/tool content, paths, project/workspace identity, environment and machine identity, raw errors and agent/package-manager payloads, private package/plugin/skill names, and individual hook or skill-invocation rows. - Recorded data and private identifier/counting state remain local in their stated locations, nothing is uploaded, data remains through D30 and is eligible for deletion on D31, and the disclosure names the inspection and deletion commands. -The [exhaustive field list](../contract/recorded-data.md) and [never-record list](../contract/recorded-data.md#what-is-never-recorded) define the details behind these requirements. The team may revise structure, tone, and wording before implementation, provided the final disclosure preserves every required point and its meaning. `cargo agents init` and `cargo agents telemetry enable` must use the same approved text and both default to no. +The [exhaustive field list](../contract/recorded-data.md) and [never-record list](../contract/recorded-data.md#what-is-never-recorded) define the details behind these requirements. The team may revise structure, tone, and wording before implementation, provided the final disclosure preserves every required point and its meaning. + +`cargo agents init` and `cargo agents telemetry enable` must use the same approved text and both default to no. ### Example version 1 disclosure @@ -107,7 +117,13 @@ Enabling telemetry does not rewrite or assign new identifiers to old stored line ## `show` -`show` prints stored event and current aggregate-snapshot JSONL lines in deterministic storage order. UTC days are ascending; within a day, append-only event lines come first in physical order and aggregate rows follow in kind, agent, hook or target scope, public source/name or unnamed reason, and event-id order. The event id is a tie-breaker when an identifier reset creates two aggregate epochs in one day. `--count N` returns the last `N` lines of that ordering: +`show` prints stored event and current aggregate-snapshot JSONL lines. The ordering is deterministic: + +1. UTC days appear in ascending order. +2. Within a day, append-only event lines appear first in physical order. +3. Aggregate rows follow in kind, agent, hook or target scope, public source/name or unnamed reason, and event-id order. + +The event id breaks ties when an identifier reset creates two aggregate epochs in one day. `--count N` returns the last `N` lines in this ordering: ```console $ cargo agents telemetry show --count 2 @@ -115,7 +131,11 @@ $ cargo agents telemetry show --count 2 {"v":1,"kind":"hook_metrics","event_id":"b563dd02-0301-4e2c-aac4-2e0d5dfaa977","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","invocations":500,"outcomes":{"ok":496,"blocked":1,"plugin_error":3,"internal_error":0},"plugins_attempted":500,"plugins_completed":500,"duration_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[8,17,76,144,181,68,6,0,0]},"session_counts_complete":true,"identified_sessions":4,"identified_sessions_non_ok":2,"hook_subject":"hok_b5b707841de7695912bec9b8bca382e8"} ``` -The command copies stored line bytes; it does not parse, normalize, repair, or pretty-print them. Unknown-version and malformed lines are shown as stored. The current day's aggregate file is a cumulative snapshot, not a history of its replaced versions. Aggregate rows have no `at`, so this storage order must not be interpreted as chronology. Lines in the output came from the same Symposium home, and their order or day can expose co-occurrence even though the rows have no global installation or workspace id. Review the complete output before sharing it. The output can be redirected to create a local copy: +The command copies stored line bytes; it does not parse, normalize, repair, or pretty-print them. Unknown-version and malformed lines are shown as stored. The current day's aggregate file is a cumulative snapshot, not a history of its replaced versions. + +Aggregate rows have no `at`, so storage order is not chronology. All output comes from the same Symposium home, and line order or day can expose co-occurrence even though the rows have no global installation or workspace id. Review the complete output before sharing it. + +Redirect the output to create a local copy: ```bash cargo agents telemetry show --count 100000 > telemetry.jsonl @@ -125,7 +145,7 @@ No separate export command is part of this RFD. ## `clear` -`clear` acquires the telemetry lock, deletes every `events-YYYY-MM-DD.jsonl` and `metrics-YYYY-MM-DD.jsonl` file, and discards pending aggregate session-count sets: +`clear` deletes every `events-YYYY-MM-DD.jsonl` and `metrics-YYYY-MM-DD.jsonl` file. It acquires the telemetry lock and also discards pending aggregate session-count sets: ```console $ cargo agents telemetry clear @@ -136,7 +156,7 @@ Deleted 12 telemetry data file(s) from ~/.symposium/telemetry/. ## `reset-identifiers` -`reset-identifiers` acquires the telemetry lock, replaces the secret identity key, discards pending aggregate session-count sets, and starts a new retention cohort for future rows: +`reset-identifiers` severs identifier linkage between future and existing rows. It acquires the telemetry lock, replaces the secret identity key, discards pending aggregate session-count sets, and starts a new retention cohort: ```console $ cargo agents telemetry reset-identifiers @@ -147,7 +167,7 @@ The command neither deletes nor rewrites old event or aggregate-metric rows. Ide If no identity state exists, the command reports that there is nothing to reset instead of creating a key. If existing state is unreadable or malformed, recording remains stopped until this command explicitly replaces it. -## Files, concurrency, and expiry +## Files and private state ```text ~/.symposium/ @@ -160,12 +180,34 @@ If no identity state exists, the command reports that there is nothing to reset Each project skills parent may also contain a generated `.symposium/index-v1.json` installation index. It maps agent-facing skill identifiers to Symposium-managed installations so a later hook can attribute a skill activation. The index is gitignored installation state, not telemetry: `show`, `clear`, retention, and identifier reset do not read or delete it, and this RFD does not upload it. -`telemetry-state.toml` is private Symposium state outside the inspectable telemetry data directory. It contains the secret identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, and bounded keyed session sets plus contribution counts used for complete aggregate session counts. All recorders read this state under the telemetry lock. Normal 30-day rollover changes the window anchor without replacing the key; renewed consent or `reset-identifiers` replaces it, while `disable` and `clear` preserve it. +`telemetry-state.toml` is private Symposium state outside the inspectable telemetry data directory. It contains the secret identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, and bounded keyed session sets plus contribution counts used for complete aggregate session counts. All recorders read this state under the telemetry lock. + +Normal 30-day rollover changes the window anchor without replacing the key. Renewed consent or `reset-identifiers` replaces the key; `disable` and `clear` preserve it. + +Symposium atomically creates and replaces the file with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. -Symposium atomically creates and replaces the file with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. `show`, `status`, data retention, and `clear` do not expose or delete the state file. `clear` atomically rewrites it only to remove pending sets, preserving the key and current anchors. The sets and contribution counts are never copied into metric rows and are discarded at day rollover or by `clear`/`reset-identifiers`. `show` and `status` do not lock writers, so their multi-file view is not an atomic snapshot. +`show`, `status`, data retention, and `clear` do not expose or delete the state file. `clear` rewrites it only to remove pending sets, preserving the key and current anchors. -Recorders make one non-waiting attempt on the lock in the telemetry data directory; that lock guards both data and private state mutations. On contention they drop the entire event batch or aggregate observation rather than delay the agent or command. Event batches are appended. Hook, plugin-hook, and extension-invocation observations are merged into a bounded, canonically ordered snapshot by a same-directory temporary write and atomic replace; a crash leaves either the old or new complete snapshot, while abandoned temporary files are ignored and cleaned lazily. Session-count state is atomically replaced first and carries the snapshot contribution count; a mismatch after a failed snapshot write discards the sets and makes the row's session counts incomplete for that day. Management commands can wait for the lock. A crash can still lose the last batch or metric update, or leave a partial final event line; `status` reports that line as malformed and `show` preserves it. +Session sets and contribution counts are never copied into metric rows. Symposium discards them at day rollover or when `clear` or `reset-identifiers` runs. `show` and `status` do not lock writers, so a summary spanning several files is not an atomic snapshot. + +## Concurrent recording + +Recorders make one non-waiting attempt on the lock in the telemetry data directory. The lock guards data and private state mutations. On contention, the recorder drops the entire event batch or aggregate observation rather than delaying the agent or command. + +Event batches are appended. Hook, plugin-hook, and extension-invocation observations are merged into a bounded, canonically ordered snapshot using a same-directory temporary write and atomic replace. A crash leaves either the old or new complete snapshot; abandoned temporary files are ignored and cleaned lazily. + +Session-count state is atomically replaced first and carries the snapshot contribution count. After a failed snapshot write, a mismatch discards the sets and makes the row's session counts incomplete for that day. + +Management commands can wait for the lock. A crash can still lose the last batch or metric update, or leave a partial final event line; `status` reports that line as malformed and `show` preserves it. Hook and extension-invocation counts are lower bounds. There is no durable all-cause dropped-update counter because contention, termination, and I/O failure can also prevent writing that counter. -Each UTC day's event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB allowance. This is a safety ceiling, not expected volume or preallocation: it bounds damage from a producer bug or unexpectedly large resolution batch. Together with D31 expiry, it bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics may use at most 512 KiB; an aggregate update that would exceed that maximum or the remaining shared allowance is dropped without stopping low-volume event recording. Telemetry data files remain through D30 and become eligible for deletion on D31, when `current_utc_day - file_utc_day > 30`. Cleanup runs lazily, at most once per day, when a recording-capable or telemetry command next runs. Uninstalling Symposium does not delete these files. +## Size and expiry + +Each UTC day's event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB allowance. This is a safety ceiling, not expected volume or preallocation. It bounds damage from a producer bug or unexpectedly large resolution batch. + +Aggregate metrics may use at most 512 KiB. An update that would exceed that maximum or the remaining shared allowance is dropped without stopping low-volume event recording. + +Telemetry files remain through D30 and become eligible for deletion on D31, when `current_utc_day - file_utc_day > 30`. Together with D31 expiry, the daily allowance bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. + +Cleanup runs lazily, at most once per day, when a recording-capable or telemetry command next runs. Uninstalling Symposium does not delete these files.