diff --git a/md/SUMMARY.md b/md/SUMMARY.md index 0ed860b4..e42cd1cc 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -96,6 +96,10 @@ - [Discovery & sync](./rfds/registry-centric-plugins/discovery-sync/README.md) - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) + - [Telemetry: recording events](./rfds/telemetry-recording/README.md) + - [What Symposium records](./rfds/telemetry-recording/contract/recorded-data.md) + - [`cargo agents telemetry`](./rfds/telemetry-recording/reference/telemetry-command.md) + - [Telemetry configuration](./rfds/telemetry-recording/reference/configuration.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/telemetry-recording/README.md b/md/rfds/telemetry-recording/README.md new file mode 100644 index 00000000..6d596dfb --- /dev/null +++ b/md/rfds/telemetry-recording/README.md @@ -0,0 +1,572 @@ +# Telemetry: recording events + +## TL;DR + +- Replace the experimental event list with a question-driven, closed schema for local telemetry. +- Keep recording off by default, per-user, local only, and gated by versioned consent. Existing unversioned opt-ins must consent again. +- Record observed sessions, configured agents, public package/extension resolution, aggregate agent skill invocation, aggregate hook reliability, command use, and known storage gaps. +- Never record individual prompt/tool activity rows, prompt or tool details, paths, private names, dependency snapshots, or a global installation/workspace id. +- Use purpose-scoped pseudonyms and non-waiting, best-effort recording. Telemetry failure must not disrupt hooks, sync, or commands. +- Defer upload, server-side handling, and subjective feedback to separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). + +Supporting pages: [data contract and exclusions](./contract/recorded-data.md), [telemetry command reference and consent disclosure](./reference/telemetry-command.md), and [configuration and consent states](./reference/configuration.md). + +## Motivation + +Symposium has no production evidence about which integrations people reach, which public packages resolve to plugins and skills, whether agents activate those skills, or whether hooks are slow or failing. + +The existing experimental telemetry was never wired into production. Its proposed prompt and tool rows would be high-volume without answering those questions. + +The team instead needs a low-volume, inspectable record of reach, resolution, skill activation, and reliability. That evidence can guide agent support, recommendation work, and investigation of hook cost. The schema must be agreed before collection begins so every field has a stated use and privacy boundary. + +This telemetry can show that a public skill resolved and that an agent activated it. It cannot show that the agent followed the skill or that the skill improved the task outcome. Version 1 observes structured skill activation only for Claude, but the event model and attribution boundary are agent-neutral. + +Controlled evaluation and explicit feedback remain separate follow-up efforts tracked under [#246](https://github.com/symposium-dev/symposium/issues/246). + +## Change in a nutshell + +From the user's perspective, telemetry follows one inspectable lifecycle: + +1. Telemetry begins disabled. `cargo agents init` and `cargo agents telemetry enable` present the same team-approved disclosure and default to no. +2. After versioned consent, recording-capable Symposium operations write only the disclosed local events and aggregates. +3. `cargo agents telemetry status` explains the effective state, while `show` exposes the stored bytes. +4. `disable` stops future recording without deleting existing files; `clear` removes recorded data, and `reset-identifiers` rotates future identifiers. +5. Nothing is uploaded under this RFD. + +Recording uses a JSONL schema built around those questions. A full sync emits one summary plus safe package and extension relationships. Hook and agent skill observations update bounded daily snapshots: + +```text +observed session -> session_start +full sync -> resolution_summary + -> package_resolution* + -> extension_resolution* +completed hook -> hook_metrics snapshot + -> plugin_hook_metrics snapshot +agent skill use -> extension_invocation_metrics snapshot +command -> command +``` + +The [exhaustive data contract](./contract/recorded-data.md) defines the fields, enums, exclusions, and complete JSONL examples. The main design has these invariants: + +- Only `Enabled` recording can create telemetry state or files. +- The producer accepts closed typed data, never arbitrary metadata or raw errors. +- Public names require allowlisted provenance and validation; everything else is unnamed or opaque. +- Pseudonyms are scoped to one measurement and normally rotate every 30 days. +- Routine hook and skill invocations update aggregates; they never become activity rows. +- A non-waiting process lock may drop a whole batch or observation but never delays another recorder. +- Files remain through D30 and first become eligible for lazy deletion on D31. +- Nothing is uploaded by this RFD. + +## Detailed plans + +### Measurement questions + +All measures describe opted-in installations, not the whole user population. Reports must state that selection bias. This RFD proposes the questions below to make the activity goals in [#246](https://github.com/symposium-dev/symposium/issues/246) and [#243](https://github.com/symposium-dev/symposium/issues/243) measurable. + +| # | Question | Operational definition | +| --- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| Q1 | Is another session observed after an installation's first session? | Deduplicate `session_start` by `retention_subject` and measure observed cohort days D1, D7, and D30. | +| Q2 | Which plugins and skills resolve? | Count `extension_resolution` occurrences and scoped subjects by public extension and safe witnessed path. | +| Q3 | Which public packages and versions occur, and what do they resolve? | Count `package_resolution` by public coordinate and `extension_match`; use paths carried by `extension_resolution`. | +| Q4 | Are Symposium and plugin hooks failing or slow? | Use daily invocation/attempt counters, outcomes, fixed latency histograms, and complete identified-session impact counts. | +| Q5 | Which agents, versions, and platforms have reach? | Keep configured-agent observations separate from observed hook sessions. | +| Q6 | Which command surfaces are used? | Count completed built-ins and eligible public plugin commands without arguments. | +| Q7 | Which resolved public skills do agents actually activate? | Count completed `extension_invocation_metrics` and complete identified-session counts by public skill and safe resolution subject. | + +These questions have specific limits: + +- For Q1, the first observed `session_start` for a `retention_subject` establishes D0. D1, D7, or D30 is present when at least one later session is observed on that cohort day, from the same or a different agent. Multiple sessions on one day count once. This measures a later observed session, not one long session or continued value; session start runs automatically once Symposium is installed. +- Q2 proves resolution, not activation. +- Q3 records relationship edges, not a complete dependency set. +- Q4 counts completed observations, so host termination can be invisible. +- Q7 proves activation, not that the agent followed the skill or completed the task better. Version 1 can answer Q7 only for Claude. + +### Scope and boundaries + +This RFD defines the complete scope of local recording: consent, provenance, resolution evidence, event families, identifiers, agent capability gaps, storage, controls, rollout, documentation, and tests. + +It covers [#242](https://github.com/symposium-dev/symposium/issues/242) and the activity-metric portion of [#243](https://github.com/symposium-dev/symposium/issues/243). + +It does not cover #243's experimental outcome signals, [#244 uploading](https://github.com/symposium-dev/symposium/issues/244), or [#245 feedback collection](https://github.com/symposium-dev/symposium/issues/245). It also leaves out new host-hook timeouts, public-identity inference from arbitrary git URLs, private-source names, and concurrent full-sync mutation of installed skills. Telemetry locking protects telemetry only. + +### Vocabulary + +**Discovery** finds candidate packages, plugins, and skills before consent is known. It never records telemetry. + +**Resolution** selects the packages and extensions that actually apply. Resolution events describe the final set and the evidence that selected it. + +An **observed session** is one whose registered `SessionStart` hook ran. It is narrower than a configured agent. + +A **skill activation** is a completed, structured agent invocation of a particular installed skill. It does not mean the agent followed the skill or that the skill improved the result. + +### Consent and activation + +Telemetry remains a per-user setting: + +```toml +[telemetry] +enabled = true +consent-version = 1 +``` + +| Effective state | Condition | Recording | +| ----------------- | ---------------------------------------- | --------------------------------------------- | +| `Disabled` | `enabled` absent or false | None; create no telemetry directory or state. | +| `ConsentRequired` | enabled but consent version absent/stale | None until current disclosure is accepted. | +| `Enabled` | enabled with current consent version | Events defined by this contract. | + +Existing users with an unversioned `enabled = true` must consent again. Interactive `init` and `telemetry enable` present the same team-approved disclosure and default to no. Non-interactive calls require an explicit acknowledgement; editing the boolean alone cannot upgrade consent. + +The [version 1 disclosure requirements](./reference/telemetry-command.md#disclosure-requirements) define what the disclosure must cover. The command reference includes a complete example for review, but its wording is not fixed implementation text. The team approves the final wording before recording is activated. `init` and `telemetry enable` then use the same snapshot-tested string. + +Editorial changes that preserve coverage and meaning do not require renewed consent. Increase the consent version when collection expands categories, user-derived fields, linkability, timestamp precision, public-name eligibility, or retention, or weakens a normative exclusion. Narrowing collection does not require renewed consent. The [telemetry configuration reference](./reference/configuration.md) defines the effective-state semantics. + +Adding an agent enum value creates a new schema version for each affected event kind because existing typed readers cannot parse the new value. It does not by itself require renewed consent when the fields, categories, timestamp precision, and correlation boundaries remain inside the accepted disclosure. + +A new field, hook surface, or linkage for that agent does require a consent-version increase. + +No production event is wired until the typed catalogue, privacy contract, controls, documentation, and current consent flow all land. Earlier implementation steps remain inert behind `ConsentRequired`. + +### Event contract + +The schema is closed and typed. It accepts fixed variants, enums, counters, and bounded structures. It rejects arbitrary metadata maps, raw PM or agent payloads, errors, debug strings, command arguments, and sanitization fallbacks. + +Counters use checked unsigned arithmetic. Overflow drops the batch or observation. + +Every row has a per-kind schema version, fixed kind, random row id, UTC day, and Symposium version. Only completed `session_start` and `command` rows carry a wall-clock timestamp, truncated to one second. Millisecond durations remain because Q4 needs mergeable distributions. + +| Kind | Purpose and emission | +| ------------------------------ | -------------------------------------------------------------------------------------------------------------------- | +| `session_start` | A registered session-start hook completed; optional agent-scoped session id and D0-D30 return subject. | +| `agent_configuration` | Daily configured/not-configured observation for each supported agent. | +| `resolution_summary` | One completed full-sync result with public/unnamed inputs, reason counts, resolved artifacts, changes, and duration. | +| `package_resolution` | One eligible public input coordinate with `public`, `unnamed_only`, or `none` extension match. | +| `extension_resolution` | One public plugin/skill and one bounded safe path that selected it. | +| `hook_metrics` | Cumulative daily counters and latency histogram per agent, hook, and identifier epoch. | +| `plugin_hook_metrics` | Cumulative daily counters/histograms per agent, hook, epoch, and bounded plugin bucket. | +| `extension_invocation_metrics` | Cumulative daily agent skill-attempt, completion, and failure counters per bounded public or unnamed bucket. | +| `command` | One completed eligible top-level command without arguments. | +| `storage_limit` | At most one daily marker naming the low-volume operation whose whole batch did not fit. | + +The `session_start` event is authoritative for observed-session and return measurements (Q1 and Q5). A `hook_metrics` row whose hook is `session_start` measures only that hook surface's reliability and latency for Q4; its invocation count is not a session count. + +Ordinary hook lookup emits no resolution summary. A structured agent skill-use observation may update `extension_invocation_metrics` without emitting a resolution event. Version 1 obtains that observation from Claude's `Skill` signal. + +Internal hook dispatch, telemetry management commands, and ineligible external commands emit no command event. See [What Symposium records](./contract/recorded-data.md) for exact fields and invariants. + +### Event producers + +Producers return typed observations or reports. They never serialize telemetry or write files. The `Recorder` checks consent, applies public-identity and correlation rules, derives identifiers, and sends accepted event batches or aggregate updates to storage. + +| Rows | Producer boundary | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `session_start` | The outer hook wrapper, after a registered `SessionStart` completes successfully. | +| `agent_configuration` | The first non-telemetry recording-capable invocation each UTC day, as one all-agent snapshot derived from Symposium's per-user agent list. | +| `resolution_summary`, `package_resolution`, `extension_resolution` | One structured full-sync report assembled while resolving and installing; telemetry does not rerun predicates or rediscover relationships. | +| `hook_metrics` | The outer hook wrapper, after the hook's final outcome and duration are known. | +| `plugin_hook_metrics` | The plugin dispatcher, around each applicable plugin's preparation and execution. | +| `extension_invocation_metrics` | The Claude `Skill` adapter normalizes targeted pre/success/failure hooks, then the generated installation index supplies safe attribution. | +| `command` | The top-level CLI dispatcher, after an eligible built-in or public plugin command reaches an outcome. | +| `storage_limit` | The JSONL sink itself, when a complete low-volume batch cannot fit. | + +Agent and package-manager payloads provide input to existing Symposium operations; they do not emit arbitrary telemetry. If a producer cannot construct a complete typed observation, it records nothing for that observation. + +### Identifiers and correlation boundaries + +When enabled recording first needs identity state, Symposium atomically creates a random 32-byte key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). This file is separate from the inspectable `/telemetry/` data directory. Symposium creates and replaces it with owner-only permissions where the platform supports them. + +Identifiers use the first 128 bits of HMAC-SHA-256 over a domain, locally anchored 30-day window, and exact dimension: + +```text +HMAC(key, "telemetry::v1\0" || window || "\0" || dimension) +``` + +The dimension limits what an identifier can link. It represents one installation for one package, agent, or command dimension, never the installation globally. + +Private state keeps the identity key and the current identifier-window and return-cohort anchors. Every recorder reads that state under the telemetry lock, so the same domain, window, and dimension produce the same subject across processes and restarts. + +Normal 30-day rollover changes the window input rather than replacing the key. `disable` and `clear` preserve the key and anchors. Renewed consent and `reset-identifiers` replace the key and start a new cohort. + +The key is private state, not anonymized telemetry. Someone who has it can recompute candidate identifiers. Telemetry commands therefore never print it, and it remains outside the inspectable telemetry data directory. + +| Identifier | Scope | +| ------------------- | ------------------------------------------------------------------------------------------------ | +| `event_id` | One row; random rather than derived. | +| `session_id` | Agent + vendor session id + 30-day window; optional. | +| `retention_subject` | One observed-session D0-D30 cohort; `session_start` only. | +| `agent_subject` | One agent + 30-day window. | +| `package_subject` | One safe public package coordinate + 30-day window. | +| `extension_subject` | One safe target/path + 30-day window; shared by its resolution and public invocation aggregates. | +| `hook_subject` | One agent/hook surface + 30-day window. | +| `plugin_subject` | One safe public plugin + 30-day window. | +| `command_subject` | One safe command coordinate + 30-day window. | + +The return subject is the sole cross-agent exception: it deduplicates Q1 but cannot link to other event kinds. A cohort remains stable through D30; the next observed session starts a new cohort. Accepting new consent or resetting identifiers rotates the key and starts another cohort. + +`session_id` is absent when the agent supplies none, including Copilot. Raw vendor ids never enter events or an unkeyed hash. There is no global installation/workspace id, and future analysis or upload must not reconstruct one. Missing identity state is created only when enabled; malformed existing state stops recording until explicit identifier reset. + +### Public package and extension identity + +Each package manager must return typed provenance with the resolved coordinate: registry URL, git, path, workspace, or unknown. Core, rather than the package manager, maps allowlisted public registries to stable ecosystem labels. + +Raw URLs and provenance never enter an event. Sources outside the allowlist are private and unnamed by default. + +A named package requires allowlisted provenance, a valid public name, and an exact resolved version. Other inputs increment `unnamed_packages` and exactly one reason: `private_registry`, `git`, `path`, `workspace`, `unknown_source`, or `invalid_coordinate`. + +Source reasons take precedence. `invalid_coordinate` applies only to an otherwise public source with an invalid coordinate. Exact versions let the team isolate version-specific recommendation gaps. + +`package_resolution.extension_match` distinguishes: + +- `public`: at least one eligible public extension matched; +- `unnamed_only`: extension content matched, but none was safe to name; +- `none`: no resolved extension matched. + +Plugin, skill, and command names follow the same public-by-allowlist rule. This requires extending the current PM representation, which loses source provenance after loading metadata. + +### Complete safe resolution paths + +`extension_resolution` records the actual successful package-to-plugin-to-skill path. Predicate evaluation produces evidence in its original pass; telemetry never re-evaluates a predicate. + +Safe nodes are public `package` and `extension` coordinates, `all` contributors, the successful `any` branch, an opaque `not` marker, or `opaque` with a fixed reason: `private_source`, `non_package_predicate`, or `limit`. + +Shell commands, paths, environment values, custom predicate details, workspace members, wildcards, private names, and a negated child never enter the path. + +Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`. The complete path is also limited to 16 evidence leaves and 4 KiB. These limits do not count filesystem path components; filesystem paths are never recorded. + +Full sync builds safe evidence for successful installations because the generated attribution index needs it even when telemetry is disabled. Only an enabled recorder serializes that evidence as telemetry. + +Cached booleans for non-package predicates may synthesize `opaque: non_package_predicate`. Caching an entire `PredicateSet` would discard the successful branches and witnesses. Supporting such a cache would require a different witness design. + +This preserves actionable resolution relationships without recording a complete dependency snapshot. It proves that an extension resolved. A matching `extension_invocation_metrics` row separately proves that an agent activated the installed skill; version 1 can produce that row only for Claude. + +### Hook aggregation and version 1 agent coverage + +Each completed hook merges into one daily `hook_metrics` row per agent, surface, and active identifier epoch. Per-plugin preparation and execution merge into `plugin_hook_metrics`. An identifier reset can create another epoch row on the same day. + +Histogram bounds are fixed at `[5, 10, 25, 50, 100, 250, 500, 1000]` milliseconds so rows remain mergeable. + +Outcome counters and histograms have the exact sum invariants defined in the data contract. Identified-session counts are all-or-nothing: every observation must supply an id, and each keyed set is limited to 256. + +On a missing id, overflow, or state/snapshot mismatch, Symposium discards the sets. The row remains incomplete for that day rather than publishing a plausible partial count. + +Private plugins merge into unnamed buckets. At most 128 public-plugin rows are named per UTC day across agents, hooks, and identifier epochs; later names merge into overflow buckets. Extension invocation metrics have a separate daily limit of 128 public skills. + +Each named subset is first-observed. Earlier-in-day plugins or skills are therefore overrepresented when the limit is reached. Analysis must report overflow and must not treat named rows as a random sample. The combined aggregate snapshot may occupy at most 512 KiB of the shared daily allowance. + +Hook rows disclose exact daily counts. `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` therefore approximate daily tool/prompt activity even without individual records. Extension invocation rows disclose exact daily skill-attempt, completion, and failure counts. The disclosure states both explicitly. + +This matrix defines which agent signals version 1 records. It is part of the producer contract, not implementation status or telemetry priority. Unsupported absence means unknown, not zero. + +| Agent | Configuration | `SessionStart` | Session id | Fresh/resume | `Stop` | Skill invocation | +| -------------- | ------------- | -------------- | ---------- | ------------ | ------ | -------------------------- | +| Claude Code | yes | yes | yes | yes | yes | attempted/completed/failed | +| Codex CLI | yes | yes | yes | yes | no | unsupported | +| GitHub Copilot | yes | yes | no | no | no | unsupported | +| Gemini CLI | yes | yes | yes | no | no | unsupported | +| Kiro | yes | yes | yes | no | no | unsupported | +| OpenCode | yes | no | n/a | n/a | no | unsupported | +| Goose | yes | no | n/a | n/a | no | unsupported | + +Configured reach covers all seven agents; observed-session measures cover only registered hook integrations. `Stop` and structured skill invocation remain Claude-only in version 1. An unsupported agent produces no invocation row, which means unknown rather than zero. `Stop` is not required by Q1-Q7. + +### Skill invocation attribution + +Agent-specific parsing ends at a normalized extension-use observation. The Claude adapter accepts only a fixture-tested `Skill` tool shape and produces a skill identifier plus one phase: `attempted`, `completed`, or `failed`. Raw Claude input and ephemeral invocation ids do not enter the recorder. + +Full sync writes a versioned installation index under the agent skills parent at `.symposium/index-v1.json`. This generated, gitignored file is installation state, not configuration or telemetry. + +The index maps the actual agent-facing identifier to the installed directory, marker fingerprint, eligibility, and safe public coordinate or path when one exists. Private identifiers may remain in the local index because they already exist in installed skill content. They are never serialized as telemetry. + +The index is atomically replaced after sync determines which installations succeeded. Before naming a public invocation, lookup verifies the Symposium marker and fingerprint. A hook sees an old or new complete index; a missing, corrupt, or stale mapping never falls back to a path or content guess. + +Public matches reuse the `extension_subject` derived for the selected safe resolution path. Other observations merge into `unnamed` rows with one fixed reason: `ineligible`, `not_indexed`, `attribution_unavailable`, `ambiguous`, or `invalid_signal`. + +`not_indexed` means a valid attribution index has no matching agent-facing identifier. `attribution_unavailable` means the index is missing, corrupt, or stale. After 128 named public-skill rows in one UTC day, further public matches merge into `overflow`. + +Claude's existing `PreToolUse` and `PostToolUse` hooks increment attempts and completions. A `PostToolUseFailure` registration matched only to `Skill` increments failures; it does not create generic failure-surface metrics. + +Each phase update is an independent lower bound. If one update is lost, completed plus failed need not equal attempted and may exceed it. + +The same all-or-nothing 256-id rule applies to attempted and completed distinct-session sets. A missing id, overflow, or state/snapshot mismatch makes both counts incomplete for that row and day. Completion proves activation only; it does not show whether the agent followed the instructions or improved the result. + +### Recording architecture + +| Unit | Responsibility | +| ------------------------- | --------------------------------------------------------------------------------- | +| `Recorder` | Enforce consent, buffer typed batches, and isolate recording failures. | +| `IdentityDeriver` | Own identity state and derive scoped pseudonyms. | +| `PublicIdentityPolicy` | Convert PM provenance into safe coordinates or unnamed counts. | +| `ResolutionWitness` | Carry evidence through one-pass resolution. | +| `InstalledExtensionIndex` | Persist actual agent-facing skill attribution after successful installation. | +| `ObservationRouter` | Normalize agent-native extension-use signals and send them only to active sinks. | +| `MetricAggregator` | Merge hook, plugin-hook, and extension-invocation observations into bounded rows. | +| `JsonlSink` | Lock, cap, retain, append/replace, inspect, and clear files. | + +Only an enabled `Recorder` owns the telemetry sink and identity components. Call sites cannot write directly. The generated installation index is functional sync state and exists independently of telemetry consent. + +A sync returns a structured report with provenance and witnesses. That report drives skill installation, atomic index replacement, and, when recording is enabled, one sanitized relationship batch. + +A hook prepares its agent response before converting timings and outcomes into aggregate updates. When telemetry is disabled and no other sink is active, the observation router does not load the index or construct an extension-use observation. + +Commands are measured once at top-level dispatch. Raw errors never enter telemetry. + +### Storage, concurrency, and retention + +#### Files and state + +Low-volume rows append to `events-YYYY-MM-DD.jsonl`. Current daily hook, plugin-hook, and extension-invocation aggregates live in a bounded, atomically replaced `metrics-YYYY-MM-DD.jsonl` snapshot under the inspectable telemetry data directory. + +The sibling private `telemetry-state.toml` holds the identity key, cohort and cleanup metadata, marker state, and temporary keyed session-count sets. These sets are never emitted and expire at day rollover. The telemetry lock remains in the data directory and guards data and private state mutations. + +#### Concurrent writes and failure + +Recorders make one non-waiting exclusive-lock attempt. Contention drops the entire buffered batch or aggregate observation. Event batches serialize before one append so concurrent lines cannot interleave. Snapshot updates use same-directory temporary replacement. + +Private-state replacement uses a temporary file beside `telemetry-state.toml` in the config directory. Abandoned state and snapshot temporaries are ignored and cleaned lazily under the telemetry lock. No `fsync` is promised, so a crash can still lose the latest update. Contribution counts detect state and snapshot divergence and permanently mark affected daily session counts incomplete. + +Aggregate counters are lower bounds. No durable counter can quantify observations lost to lock contention, process termination, or I/O failure because those conditions can also prevent writing the counter. A cap-only counter would not measure total loss. + +#### Size and retention + +The event file, aggregate snapshot, and reserved maximum-size `storage_limit` row share 8 MiB per day. This is a safety ceiling, not expected volume or preallocation. It bounds damage from a producer bug or unexpectedly large resolution batch; normal recording should remain well below it. + +Together with D31 expiry, the daily allowance bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Aggregate metrics receive at most 512 KiB. An oversized metric update is dropped without stopping low-volume events. An ordinary batch that cannot fit is replaced by the daily marker, and ordinary recording stops for that day. Relationship batches are never split. + +Files survive D30 and become eligible for lazy deletion when `current_day - file_day > 30`, first on D31. `clear` deletes event and metric files plus pending count sets, but preserves consent and identity/cohort state. `reset-identifiers` rotates future identifiers without rewriting old files. + +`disable` stops recording but keeps files by default. Uninstalling Symposium also leaves them. The [telemetry CLI reference](./reference/telemetry-command.md) defines the exact command behavior. + +#### Multiple agents + +Concurrent agents produce separate session and agent/hook rows. The same public package/path derives the same dimension subject for unique-install deduplication, but no project id links the agents. A simultaneous flush may be dropped; concurrent mutation of installed skills remains a separate problem. + +### Schema evolution + +Schema versions are per event kind. Semantic or correlation changes create a new version. Privacy expansions also require a new consent version. + +Readers retain malformed and unknown lines, count them separately, and exclude them from typed analysis. Raw `show` preserves their bytes. + +The ["What is never recorded" section of the data contract](./contract/recorded-data.md#what-is-never-recorded) is a producer rule. In summary, telemetry excludes prompt and tool content, per-invocation rows, raw errors and payloads, paths and workspace identity, environment/machine/account values, private-source names, arbitrary URLs, global identifiers, and timestamps finer than one second. + +Here, the per-invocation exclusion includes individual skill activations and raw agent-facing skill identifiers. + +### Drawbacks and limitations + +This design accepts the following costs and limits: + +- Opt-in measurements describe participating installations, not the complete user population. Reports must state this selection bias. +- A completed skill activation does not prove that the skill was followed or improved the task. Version 1 also obtains structured skill-use observations only from Claude. +- Scoped pseudonyms do not make a local directory anonymous. File/day/order and one buffered sync can expose co-occurrence; unusual public versions, public skill-use counts, agent/platform combinations, and exact counts can fingerprint an installation. +- Best-effort recording undercounts activity. Busy multi-agent sessions contend more, terminated hooks lose final observations, and unsupported agents have configuration but not session or skill-invocation observations. A failure may also prevent writing a durable dropped-update counter, so the missing data cannot be measured completely. +- Recording performs bounded in-process work and one non-waiting lock attempt. It does not promise zero latency, although failure and contention never change the user operation's result. +- The daily safety cap permits ordinary retained data near 248 MiB through D30. The implementation also adds cross-cutting provenance, witness, attribution, aggregation, consent, and schema-maintenance work. + +### Rationale and alternatives + +#### Record no production telemetry + +This avoids the privacy, storage, and implementation costs above, but leaves Q1-Q7 unanswered. Controlled evaluation can determine whether a skill helps in a test scenario; it cannot show production reach, resolution gaps, or hook reliability. + +#### Record every hook and skill invocation + +Hooks are high-volume; Q4 needs rates and distributions, not traces. Resolution edges are low-volume and Q2/Q3 need the actual safe package-to-plugin-to-skill relationship. Aggregating hook and skill activity while retaining safe resolution events preserves that product signal without creating per-invocation rows. + +#### Defer scoped identifiers, provenance, and witnesses until upload + +These mechanisms already serve local measurement and privacy. Q1-Q7 need narrow deduplication and relationship evidence. Provenance prevents private coordinates from reaching an inspectable or shareable telemetry directory. + +Recording plain coordinates now would create weaker local files and permanently lose evidence needed by the measurement questions. `event_id` also gives cumulative metric rows stable identity across snapshot replacement and reset epochs. + +#### Omit exact package versions + +A package name alone cannot distinguish a missing recommendation from a version-specific resolution gap or regression. Exact versions provide that diagnostic only after the PM proves an allowlisted public origin and validates the coordinate; private and local versions remain unnamed. + +#### Record a complete dependency snapshot + +Per-package frequency and safe extension paths answer Q3 without an explicit project fingerprint. Same-file/day/batch order can still reveal co-occurrence locally, but a dependency-set or resolution id would make that linkage direct and persistent. + +#### Use SQLite instead of JSONL + +Low-volume events append, while hook metrics form a small bounded daily snapshot. JSONL keeps both byte-inspectable, preserves unknown versions and malformed lines, and needs no database. The process lock plus atomic snapshot replacement supplies the required concurrency behavior. + +### Unresolved questions + +No telemetry-contract question is intentionally left open for acceptance. Two implementation inputs remain: Step 7 requires team approval of the exact disclosure wording, and Step 5 requires before-and-after hook measurements. + +If either requires changing collection, privacy, or failure semantics, the RFD must be amended before recording is activated. + +### Future possibilities + +The event catalogue is closed for consent version 1, but new measurements can be added as typed, versioned event families. Every family goes through `Recorder` and remains subject to consent, public-identity rules, storage caps, retention, failure behavior, and the never-record exclusions. + +#### Additional agent adapters + +Agent-native extension-use payloads remain behind the normalized observation boundary. Version 1 implements only the Claude `Skill` adapter. Supporting another agent requires a thin adapter, native-signal conformance fixtures, capability documentation, and an event schema update; it does not duplicate attribution, aggregation, or storage. + +#### Plugin telemetry + +The same model can later support eligible public plugins. A plugin would declare bounded feature names in its manifest and report only core-defined aggregates. Symposium would own and validate the schema, record and use the data, and optionally provide aggregate reports to plugin authors. + +Plugins could not add arbitrary fields, bypass consent, or write directly to storage. This RFD implements only the event families listed above. Plugin reporting requires a separate implementation and disclosure before collection begins. + +#### Controlled evaluation + +Production telemetry and a future evaluation harness may consume the same normalized extension-use observation, but they must use separate sinks and contracts. The telemetry sink writes only the daily aggregates defined here. + +An explicitly started harness may retain detailed per-run evidence in its isolated evaluation workspace. It does not read telemetry JSONL or rely on rotating telemetry identifiers. + +This RFD preserves that internal seam but does not define a harness, scenario format, trace format, outcome grader, or token accounting. Controlled with/without comparisons answer whether a skill improved a task; production telemetry does not. + +#### Upload + +Accepting this RFD is not consent to upload. A future RFD must define transport/authentication, renewed consent and scheduling, retry/idempotency, server retention/access/deletion, reporting thresholds, and incident handling. + +Upload may use only accepted local fields and must preserve scoped-correlation boundaries. It cannot create a global subject from identifiers, file order, batch membership, request grouping, or transport metadata. `event_id` and per-kind versions enable retry and mixed-version handling; they pre-approve no transport. + +### Proposed documentation + +- [What Symposium records](./contract/recorded-data.md): normative fields, enums, examples, and exclusions. +- [`cargo agents telemetry`](./reference/telemetry-command.md): controls, files, inspection, and concurrency. +- [Telemetry configuration](./reference/configuration.md): consent and effective-state semantics. + +These remain proposed pages until implementation lands; shipped design/reference chapters continue to describe the current binary. + +## Frequently asked questions + +### What does pseudonymous mean here? + +A scoped identifier comes from a random local secret rather than machine identity. It still links observations inside one stated purpose and window. Rotation and domain separation limit that linkage; they do not make the local files anonymous. + +The secret key is private state, not anonymized telemetry. It is not included in telemetry because someone who has it can recompute candidate identifiers. + +### Does a completed skill activation mean the skill helped? + +No. It means an agent successfully activated the installed skill; version 1 can observe this only for Claude. It does not show whether the agent followed the instructions or whether the task result improved. That causal question requires a controlled evaluation comparing equivalent runs with and without the skill. + +## Implementation plan and status + +The seven steps below are PR-sized. Steps 1-6 keep production collection inert. Step 7 activates the current consent version only after every producer and control is present. + +Tests and benchmarks may construct an enabled recorder only through a test-only API bound to a temporary telemetry home. There is no runtime environment-variable or configuration bypass. + +```text +1. Contract and identity + | +2. Storage and controls + / \ +3. PM provenance 6. Reach/commands + | +4. Resolution witnesses/index/events + | +5. Hook and extension-use metrics + \ / + 7. Consent and activation +``` + +After Step 2, Steps 3 and 6 may proceed in parallel. Step 4 follows Step 3, Step 5 follows Step 4, and Step 7 waits for Steps 5 and 6. + +Step 3 isolates the risky core PM type seam from telemetry instrumentation. Steps 4 and 5 remain end-to-end measurement slices. The plan remains seven PRs. + +### Step 1: Telemetry contract and identity + +Replace the dormant event types with the closed producer schema: common fields, bounded witnesses, fixed outcomes and histograms, extension-invocation counters and buckets, and per-kind version dispatch. + +Add private telemetry-owned `/telemetry-state.toml`, atomic owner-only key creation where supported, scoped HMAC derivation, local 30-day windows, D0-D30 cohorts, and reset primitives. Keep this state outside the inspectable telemetry data directory. Do not add storage or emission. + +Verify: + +- All contract examples round-trip, and bounds and unknown versions behave as specified. +- Arbitrary or private data cannot serialize. +- Identities separate domains, dimensions, agents, and windows. The same active-window inputs remain stable across recorders and restarts, while normal rollover changes subjects without replacing the key. +- `disable` and `clear` preserve identity state; renewed consent and reset rotate it. +- Disabled paths create nothing. + +- [ ] PR: telemetry contract and scoped identity + +### Step 2: Storage and local controls + +Add whole-batch event appends, non-waiting process locking, atomically replaced aggregate snapshots, daily caps and reservations, `storage_limit`, and lazy D31 cleanup. Add typed `status`, byte-preserving `show`, `clear`, and `reset-identifiers` commands. + +Expose a test-only enabled recorder bound to a caller-supplied temporary telemetry home for integration tests and benchmarks. No production caller writes through the sink yet, and no runtime bypass is added. + +Verify: + +- Concurrent complete lines, old-or-new snapshots, whole-operation drops, and cap/marker accounting. +- D30/D31 cleanup, malformed and unknown inspection, and abandoned state/snapshot temporary cleanup. +- Private-state permissions and separation, clear/reset semantics, and test-only recorder isolation. +- Management commands never record themselves. + +- [ ] PR: telemetry storage and local controls + +### Step 3: PM provenance and public-identity policy + +Extend every package-manager result with typed provenance and exact coordinates. Add core-owned public allowlists and coordinate validation. If an out-of-process package-manager protocol lands first, carry provenance through it as part of this step. + +All supported package managers must provide provenance before events are wired. This step emits no telemetry. + +Verify every package manager and source class, malformed and wildcard coordinates, public allowlist behavior, unnamed-reason precedence, and that raw URLs and private or local coordinates cannot become public identities. + +- [ ] PR: package provenance and public identity policy + +### Step 4: Resolution witnesses, attribution index, and recording + +Return safe evidence from the original predicate evaluation while preserving short-circuit and accepted cache behavior. Whole-`PredicateSet` caching remains disallowed. + +Carry evidence through plugin and skill selection, return one structured full-sync report, and construct coherent `resolution_summary`, `package_resolution`, and `extension_resolution` batches. After installation, atomically replace the generated agent-facing attribution index with successful managed skills and their marker fingerprints. Ordinary read-only plugin lookup remains silent. + +Verify: + +- `all`, `any`, `not`, and opaque/limit paths, plus cache hits without reevaluation. +- Every `extension_match` case and complete package-to-plugin-to-skill paths. +- Whole-batch failure, successful and failed installation indexing, and atomic old-or-new index reads. +- Stale, corrupt, fingerprint-mismatch, and collision handling. +- Raw paths, private names, and dependency snapshots stay out of telemetry. + +- [ ] PR: resolution witnesses, installed attribution, and recording + +### Step 5: Hook and extension-invocation telemetry + +Measure completed Symposium and plugin-hook handling. Merge top-level observations into `hook_metrics` and plugin preparation/execution into `plugin_hook_metrics`, with fixed outcomes/histograms, scoped subjects, public/unnamed/overflow buckets, the 128-named-row limit, and all-or-nothing identified-session counts. + +Add the sink-neutral extension-use observation and Claude `Skill` adapter. Existing pre/post-tool hooks and a targeted failure hook update `extension_invocation_metrics` through the installed attribution index, with independent phase counters, fixed unnamed reasons, a separate 128-public-skill limit, and all-or-nothing attempted/completed session counts. Never emit a per-invocation row. + +Measure the hook path before and after. Verify: + +- Sanitized automatic and manual activation, lifecycle fixtures, and unknown schemas. +- Public, private, missing, ambiguous, and stale attribution; marker validation; and raw-input exclusion. +- Counter, histogram, and cross-row invariants; independently dropped phases; boundary buckets; process merging; and rollover/reset. +- Bounded behavior across 500 lifecycles, private and overflow buckets, and missing, mixed, or over-256 session ids. +- Crash recovery, a worst-case snapshot within 512 KiB, and integer-overflow drops. +- A fake second sink, disabled-path short-circuiting, and unchanged agent output on recording failure. + +- [ ] PR: bounded hook and extension-invocation telemetry + +### Step 6: Session, configuration, and command telemetry + +Add capability-aware `session_start`, daily `agent_configuration`, and eligible top-level `command` rows over the shared recorder. + +The first non-telemetry recording-capable invocation of each UTC day attempts one all-agent configuration batch from the per-user agent list. It does not inspect agent-owned files. Hook internals, telemetry controls, arguments, and unsafe plugin command names remain excluded. + +Verify: + +- The capability matrix, Copilot without a session id, and Claude-only `Stop`. +- Configuration-list semantics, same-day configuration deduplication, and retry after a dropped configuration batch. +- Fixed command vocabulary and public eligibility, plus failures before or after command dispatch. + +- [ ] PR: session reach and command telemetry + +### Step 7: Consent, activation, and documentation + +Add `consent-version`, one shared team-approved disclosure for `init` and `telemetry enable`, re-consent, explicit non-interactive acknowledgement, `enable` and `disable`, and final production wiring for Steps 4-6. + +Review the final wording against the version 1 coverage requirements. The proposed prompt is a complete example, not the implementation string. There is no event-file migration because the dormant recorder was never called. Publish the proposed pages and update the current design and flow chapters plus `md/SUMMARY.md`. + +Verify: + +- New and existing configuration states, plus documented review of the final disclosure against every required coverage point. +- A snapshot proving that `init` and `telemetry enable` use identical approved text. +- Interactive and non-interactive flows, no partial or disabled collection, and no runtime consent bypass. +- Raw inspection and expiry, the full CLI and integration suite, and the hook-path benchmark. +- Formatting, clippy, workspace tests, mdBook, and orphan checks. + +- [ ] PR: telemetry consent and recording activation diff --git a/md/rfds/telemetry-recording/contract/recorded-data.md b/md/rfds/telemetry-recording/contract/recorded-data.md new file mode 100644 index 00000000..b23aae1c --- /dev/null +++ b/md/rfds/telemetry-recording/contract/recorded-data.md @@ -0,0 +1,397 @@ +# What Symposium records + +Symposium records telemetry only after you opt in. The records stay on your machine; nothing described on this page is uploaded. `cargo agents telemetry show` displays the stored bytes, and `cargo agents telemetry clear` deletes event and aggregate-metric files. The [never-record list](#what-is-never-recorded) summarizes the exclusions. + +This page is the complete producer contract for consent version 1. If a field is not listed here, Symposium does not write it as telemetry. + +## Common fields + +Every JSONL row has: + +| Field | Example | Meaning | +| ----------- | --------------- | --------------------------------------- | +| `v` | `1` | Schema version for this row kind. | +| `kind` | `session_start` | Row kind from the list below. | +| `event_id` | `9f2c41b6-...` | Identifier for this row; random when minted. | +| `day` | `2026-08-03` | UTC calendar day. | +| `symposium` | `0.4.0` | Symposium version that wrote the event. | + +Completed operational events (`session_start` and `command`) also have `at`, an RFC3339 UTC timestamp truncated to one second. Resolution, configuration, and aggregate metric rows have only `day`. + +Counters and durations are non-negative JSON integers that fit an unsigned 64-bit value. Symposium checks arithmetic and drops an overflowing batch or observation instead of wrapping the value. + +`event_id` exists to deduplicate a future retry of the same event or identify one cumulative metric row. A metric row keeps the `event_id` minted when its dimension first appears that UTC day as the row is rewritten. It is not an installation, session, account, or project identifier. + +## Example JSONL for every row kind + +These are independent row-shape examples, not one coherent operation or batch. The identifiers and coordinates are illustrative. An actual day may contain repeated low-volume events and one cumulative row per aggregate-metric dimension. + +```jsonl +{"v":1,"kind":"session_start","event_id":"9f2c41b6-495e-4c88-a22b-c597f8102aed","day":"2026-08-03","at":"2026-08-03T09:14:02Z","symposium":"0.4.0","agent":"claude","os":"linux","arch":"x86_64","start":"fresh","session_id":"sess_31d8b1916028f65a0c0521dc1f4c86fb","retention_subject":"ret_74ddf26f80ad8b58de7f03e6c632e654","cohort_day":0} +{"v":1,"kind":"agent_configuration","event_id":"1a77f7c8-2c45-4de2-8cc6-b507ac3605f8","day":"2026-08-03","symposium":"0.4.0","agent":"claude","configured":true,"os":"linux","arch":"x86_64","agent_subject":"agt_9255770e1679cb789796a9f9e86325c5"} +{"v":1,"kind":"resolution_summary","event_id":"6c3ef7a1-f782-4b47-9a25-829dd64e1ba2","day":"2026-08-03","symposium":"0.4.0","trigger":"session_start","outcome":"ok","duration_ms":142,"public_packages":2,"unnamed_packages":1,"unnamed_package_reasons":{"private_registry":1,"git":0,"path":0,"workspace":0,"unknown_source":0,"invalid_coordinate":0},"plugins":1,"skills":1,"installed":2,"updated":0,"reaped":0,"session_id":"sess_31d8b1916028f65a0c0521dc1f4c86fb"} +{"v":1,"kind":"package_resolution","event_id":"c03e7390-52b1-4e11-b7e7-9573c84e555a","day":"2026-08-03","symposium":"0.4.0","package":{"ecosystem":"cargo","name":"example-runtime","version":"1.2.3"},"extension_match":"public","package_subject":"pkg_f6db813c87209816ae4896f3e60dd774"} +{"v":1,"kind":"extension_resolution","event_id":"fa21a1d7-64a7-4e3e-9f6a-4572cf8f1939","day":"2026-08-03","symposium":"0.4.0","target":{"type":"skill","source":"symposium-recommendations","name":"example-debugging"},"path":[{"type":"package","ecosystem":"cargo","name":"example-runtime","version":"1.2.3"},{"type":"extension","extension_type":"plugin","source":"symposium-recommendations","name":"example-tools"},{"type":"extension","extension_type":"skill","source":"symposium-recommendations","name":"example-debugging"}],"extension_subject":"ext_6e68a75b9701bbad86cf32cd876e994a"} +{"v":1,"kind":"hook_metrics","event_id":"b563dd02-0301-4e2c-aac4-2e0d5dfaa977","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","invocations":500,"outcomes":{"ok":498,"blocked":1,"plugin_error":1,"internal_error":0},"plugins_attempted":500,"plugins_completed":500,"duration_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[40,80,180,130,55,12,2,1,0]},"identified_sessions":1,"identified_sessions_non_ok":1,"session_counts_complete":true,"hook_subject":"hok_51f4f4143ce32704e96086899dc66a27"} +{"v":1,"kind":"plugin_hook_metrics","event_id":"70dcad2a-fd19-4b5d-97cb-a7c7b52a81f1","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","plugin_scope":"public","plugin":{"source":"symposium-recommendations","name":"example-tools"},"attempts":500,"executions":500,"outcomes":{"ok":499,"blocked":0,"error":1},"prepare_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[400,90,10,0,0,0,0,0,0]},"execute_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[10,50,200,180,50,8,2,0,0]},"identified_sessions":1,"identified_sessions_non_ok":1,"session_counts_complete":true,"plugin_subject":"plg_d42e3f1fbac5bb001e1258a1e70b0d77"} +{"v":1,"kind":"command","event_id":"5d18caa8-84f7-4aa3-846c-99ea810ccd85","day":"2026-08-03","at":"2026-08-03T10:02:11Z","symposium":"0.4.0","command":{"type":"builtin","name":"use"},"duration_ms":820,"outcome":"ok","command_subject":"cmd_adf0c14ddc35b97762b5daae6f4119ce"} +{"v":1,"kind":"storage_limit","event_id":"8430f7f3-7ec5-4ca5-9c65-8f6e83eaa3de","day":"2026-08-03","symposium":"0.4.0","dropped_operation":"manual_sync"} +{"v":1,"kind":"extension_invocation_metrics","event_id":"7b3bda33-1fd7-4657-9637-4057268370dc","day":"2026-08-03","symposium":"0.4.0","agent":"claude","target_scope":"public","target":{"type":"skill","source":"symposium-recommendations","name":"example-debugging"},"attempted":14,"completed":12,"failed":2,"session_counts_complete":true,"identified_sessions":4,"identified_sessions_completed":3,"extension_subject":"ext_6e68a75b9701bbad86cf32cd876e994a"} +``` + +## Scoped identifiers + +### Key and rotation + +When an enabled recorder first needs identity state, Symposium stores a random secret key in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). The same state holds the current identifier-window and return-cohort anchors. Every recorder reads it under the telemetry lock, so identical domain, window, and dimension inputs produce the same subject across processes and restarts. + +Normal 30-day rollover changes the window input without replacing the key. Renewed consent or `telemetry reset-identifiers` replaces it. `telemetry disable` and `telemetry clear` preserve the key and anchors. + +This file is separate from the inspectable `/telemetry/` data directory and has owner-only permissions where the platform supports them. The key is private state, not anonymized telemetry. It is not written into events, printed by telemetry commands, or derived from your machine. Someone who has the key can recompute candidate identifiers. + +### What identifiers can link + +Symposium derives each identifier for one narrow purpose: + +| Identifier | What it can link | Rotation | +| ------------------- | -------------------------------------------------------------- | --------------------------------- | +| `session_id` | Events carrying the same vendor session id for one agent | 30 days; omitted when unavailable | +| `retention_subject` | Observed session days in one D0-D30 cohort | After cohort day 30 | +| `agent_subject` | Repeated configuration observations for one agent | 30 days | +| `package_subject` | Repeated observations of one public package/version | 30 days | +| `extension_subject` | Resolution and public invocation aggregates for one safe path | 30 days | +| `hook_subject` | Daily hook aggregates for one agent and hook surface | 30 days | +| `plugin_subject` | Daily hook aggregates for one eligible public plugin | 30 days | +| `command_subject` | Repeated use of one eligible command | 30 days | + +These values are pseudonymous, not anonymous: they deliberately permit limited linking inside the stated scope. `retention_subject` can link observed sessions across agents for D0-D30; this is the single exception needed for return measurement. No identifier links that cohort, a package subject, and a command subject, and there is no workspace id. + +All lines in your local telemetry directory still come from your Symposium home. File order and same-day events can therefore suggest which observations happened together. + +## Version 1 public identity allowlists + +Only the following stable labels can make package, plugin, skill, or plugin-command names eligible for recording under consent version 1: + +| Enum | Version 1 values | Used by | +| --- | --- | --- | +| Package ecosystem | `cargo` | `package.ecosystem` and package path nodes. | +| Public extension source | `symposium-recommendations`, `crates-io` | Resolution/invocation `target.source`, extension path nodes, `plugin.source`, and plugin-command `source`. | + +`crates-io` applies only when core proves allowlisted crates.io provenance; `symposium-recommendations` identifies the built-in registry. `user-plugins`, configured registries, paths, workspaces, and arbitrary git sources remain unnamed. Raw registry names and URLs are never enum values. Adding an ecosystem or public-source label expands name eligibility and requires a new consent version. + +## Event kinds + +### `session_start` + +This row records a completed registered Symposium session-start hook. + +| Field | Values | Meaning | +| ------------------- | ---------------------------------------------- | ------------------------------------------------------- | +| `at` | UTC second | When Symposium completed the session-start handling. | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro` | Agent that invoked the registered hook. | +| `os` | `linux`, `macos`, `windows`, `other` | OS class for the running Symposium build. | +| `arch` | `x86_64`, `aarch64`, `other` | Architecture class for the running build. | +| `start` | `fresh`, `resumed`, `unknown` | Agent-supplied lifecycle classification when available. | +| `session_id` | scoped id, optional | Omitted when the agent supplies no session id. | +| `retention_subject` | scoped id | Deduplicates this D0-D30 observed-session cohort. | +| `cohort_day` | integer `0` through `30` | UTC days since the cohort's first observed session. | + +GitHub Copilot does not currently supply a session id. OpenCode and Goose do not currently call Symposium through a registered session-start hook, so they do not produce this event. + +These rows, not `hook_metrics` rows whose `hook` is `session_start`, are authoritative for observed-session and return measurements. For each `retention_subject`, the first row establishes D0. D1, D7, or D30 is present when at least one later session-start row has that `cohort_day`, regardless of agent or vendor session id. Multiple rows on the same cohort day count once. + +The aggregate hook rows measure only session-start hook reliability and latency. + +### `agent_configuration` + +This row records whether a supported agent is configured for Symposium that day. + +| Field | Values | Meaning | +| --------------- | ------------------------------------------------------------------- | -------------------------------------------------------- | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro`, `opencode`, `goose` | Agent being checked. | +| `configured` | boolean | Whether the agent is listed in per-user Symposium configuration. | +| `os` | `linux`, `macos`, `windows`, `other` | OS class for the running Symposium build. | +| `arch` | `x86_64`, `aarch64`, `other` | Architecture class for the running build. | +| `agent_subject` | scoped id | Deduplicates one installation for this agent and window. | + +On the first non-telemetry recording-capable invocation each UTC day, Symposium attempts one whole batch containing all seven agents. `configured` means the agent is present in Symposium's per-user configuration at that moment; Symposium does not inspect agent-owned configuration files. + +A successful batch is the day's snapshot, so later configuration changes appear on the next UTC day's snapshot. If locking, storage, or I/O drops the batch, it remains outstanding and a later eligible invocation retries it. + +`configured: true` does not claim that an agent session ran or that the agent-owned integration files remain intact. + +### `resolution_summary` + +This row records the result of a full sync after session start, manual sync, `use`, or removal. Package, plugin, and skill counts are distinct coordinates within the sync, not duplicate declarations. + +| Field | Values | Meaning | +| ------------------ | ----------------------------------------------- | ----------------------------------------------------------------- | +| `trigger` | `session_start`, `manual_sync`, `use`, `remove` | Why full sync ran. | +| `outcome` | `ok`, `partial`, `error` | Closed result classification. | +| `duration_ms` | integer | Full resolution/sync duration. | +| `public_packages` | integer | Eligible public resolution-input packages. | +| `unnamed_packages` | integer | Private, local, unknown, invalid, or otherwise ineligible inputs. | +| `unnamed_package_reasons` | fixed counters | Mutually exclusive reasons that sum to `unnamed_packages`. | +| `plugins` | integer | Plugins in the final resolved set. | +| `skills` | integer | Skills in the final resolved set. | +| `installed` | integer | Artifacts newly installed by this sync. | +| `updated` | integer | Existing artifacts changed by this sync. | +| `reaped` | integer | Obsolete artifacts removed by this sync. | +| `session_id` | scoped id, optional | Present only for a sync inside an identified agent session. | + +`unnamed_package_reasons` has exactly `private_registry`, `git`, `path`, `workspace`, `unknown_source`, and `invalid_coordinate` counters. Each unnamed package increments exactly one. Source provenance takes precedence; `invalid_coordinate` is used only for an allowlisted public source with a malformed name or a missing, wildcard, or invalid exact version. The counters sum to `unnamed_packages`. + +Read-only extension lookup on ordinary hook calls does not produce this event. + +### `package_resolution` + +This row records one eligible public package used as resolution input during a full sync. + +| Field | Values | Meaning | +| ------------------- | ----------------------- | -------------------------------------------------------------------- | +| `package.ecosystem` | `cargo` | Stable public ecosystem label. | +| `package.name` | validated string | Public package name. | +| `package.version` | validated exact version | Exact resolved public version, never a range or `*`. | +| `extension_match` | `public`, `unnamed_only`, `none` | What kind of resolved extension, if any, the package contributed to. | +| `package_subject` | scoped id | Deduplicates this exact coordinate for 30 days. | + +A package is named only when its package manager reports provenance matching a reviewed public-registry allowlist. Registry URLs themselves are not recorded. + +`extension_match` describes what the package contributed: + +- `public`: at least one eligible public extension matched, including when unnamed content also matched. +- `unnamed_only`: extension content matched, but none was eligible to name. +- `none`: no resolved extension matched. + +Events measure packages independently. Symposium does not record a workspace/resolution id, but the summary and contiguous relationship batch can still make the public package set for one sync recoverable, especially when only one sync occurred that day. Review the whole file before sharing it. + +### `extension_resolution` + +This row records one public plugin or skill and one safe path that selected it. + +| Field | Values | Meaning | +| ------------------- | --------------------------- | --------------------------------------------------- | +| `target.type` | `plugin`, `skill` | Resolved extension type. | +| `target.source` | `symposium-recommendations`, `crates-io` | Stable label, never a configured URL or local name. | +| `target.name` | validated string | Name defined by eligible public content. | +| `path` | bounded typed nodes | Actual safe package/predicate/extension chain. | +| `extension_subject` | scoped id | Deduplicates this safe target/path for 30 days. | + +Path nodes are limited to: + +| Node `type` | Recorded content | +| ----------- | -------------------------------------------------------------------- | +| `package` | Eligible public ecosystem, name, and exact version. | +| `extension` | Eligible public plugin/skill source and name. | +| `all` | All safe children that contributed to success. | +| `any` | The branch that actually made the expression succeed. | +| `not` | Marker only; the child is not recorded. | +| `opaque` | Fixed reason: `private_source`, `non_package_predicate`, or `limit`. | + +Shell commands, paths, environment variables, custom predicate names or arguments, and private package or extension names never enter a path. An opaque marker can represent their position. + +Witness depth counts nested evidence nodes from the root, which is level 1, to a terminal package, extension, `not`, or opaque node. A subtree that would exceed level 8 becomes `opaque: limit`. The complete path is also limited to 16 evidence leaves and 4 KiB. Evidence depth does not count filesystem components; filesystem paths are never recorded. + +This event says an extension resolved. It does not say that an agent read or used the extension; a matching `extension_invocation_metrics` aggregate separately reports observed agent activation. Version 1 can produce that aggregate only for Claude. + +### `hook_metrics` + +This cumulative row combines completed hook observations for one UTC day, agent, hook surface, and active identifier epoch. An epoch normally lasts 30 days but ends early after identifier reset or renewed consent. A reset can therefore leave two rows for the same agent, hook, and day with different `hook_subject` values. Symposium updates the current row instead of appending one telemetry row per invocation. + +| Field | Values | Meaning | +| --------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro` | Invoking agent. | +| `hook` | `pre_tool_use`, `post_tool_use`, `user_prompt_submit`, `session_start`, `stop` | Symposium hook surface. | +| `invocations` | integer | Completed hook observations merged into the row. | +| `outcomes` | hook outcome counters | Exact counters named `ok`, `blocked`, `plugin_error`, and `internal_error`. | +| `plugins_attempted` | integer | Plugin hooks whose preparation began. | +| `plugins_completed` | integer | Plugin hooks with an observed terminal result. | +| `duration_ms` | latency histogram | Parsed-input-to-response-ready latency; telemetry update time is excluded. | +| `session_counts_complete` | boolean | Whether the two distinct identified-session counts are complete for every observation. | +| `identified_sessions` | integer, optional | Distinct identified sessions represented; present only when `session_counts_complete=true`. | +| `identified_sessions_non_ok` | integer, optional | Those sessions with at least one non-`ok` observation; present only when counts are complete. | +| `hook_subject` | scoped id | Links this agent/hook dimension inside its 30-day identifier window. | + +`outcomes` and the duration histogram each sum to `invocations`. Exactly one outcome counter advances for each completed observation. Precedence is `internal_error`, then `blocked`, then `plugin_error`, then `ok`. + +Session counts remain complete only when every contributing observation supplies a session id and neither set exceeds 256 distinct ids for the row. On the first missing id or overflow, Symposium discards both sets, writes `session_counts_complete: false`, and omits both counts for the rest of that day. Raw and keyed session ids are never written into the aggregate file. + +### `plugin_hook_metrics` + +This cumulative row combines plugin-hook observations for one UTC day, agent, hook surface, active identifier epoch, and bounded plugin bucket. A reset can leave otherwise identical `unnamed` or `overflow` rows distinguished only by `event_id`; public rows also receive a new `plugin_subject`. + +| Field | Values | Meaning | +| ----------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| `agent` | `claude`, `codex`, `copilot`, `gemini`, `kiro` | Invoking agent. | +| `hook` | `pre_tool_use`, `post_tool_use`, `user_prompt_submit`, `session_start`, `stop` | Symposium hook surface. | +| `plugin_scope` | `public`, `unnamed`, `overflow` | Whether the bucket names an eligible public plugin. | +| `plugin.source` | `symposium-recommendations`, `crates-io`, conditional | Present only when `plugin_scope=public`. | +| `plugin.name` | validated string, conditional | Present only when `plugin_scope=public`. | +| `attempts` | integer | Plugin-hook attempts that reached an observed terminal result. | +| `executions` | integer | Those completed attempts that reached child execution. | +| `outcomes` | plugin outcome counters | Exact counters named `ok`, `blocked`, and `error`. | +| `prepare_ms` | latency histogram | Preparation time for every attempt. | +| `execute_ms` | latency histogram | Child execution time for attempts counted by `executions`. | +| `session_counts_complete` | boolean | Whether the two distinct identified-session counts are complete for every attempt. | +| `identified_sessions` | integer, optional | Distinct identified sessions represented; present only when `session_counts_complete=true`. | +| `identified_sessions_non_ok` | integer, optional | Those sessions with at least one non-`ok` attempt; present only when counts are complete. | +| `plugin_subject` | scoped id, conditional | Present only with an eligible public plugin. | + +#### Counting rules + +`attempts` counts terminal results. `executions` counts the subset that started child execution. A preparation failure is terminal `error` with no execution. `outcomes` and `prepare_ms` each sum to `attempts`; `execute_ms` sums to `executions`. Across every epoch and bucket for one agent, hook, and day, plugin `attempts` sum to the corresponding top-level `plugins_completed`. + +`blocked` means the plugin requested a block. `error` covers a closed preparation or execution failure. `ok` is every other completed attempt. The same 256-id all-or-nothing session rule used by `hook_metrics` applies. + +#### Plugin identity and row limits + +Eligible public plugins use `{source, name}` coordinates from the reviewed allowlist. Private, local, invalid, or otherwise ineligible plugins merge into one `unnamed` row per agent, hook, epoch, and day and expose no identity. + +At most 128 named public-plugin rows may appear across all agents, hooks, and identifier epochs in one UTC day. Attempts for later rows merge into `overflow` rows per agent, hook, and epoch and expose no identity. Identifier reset does not reset this daily limit. + +Named rows are first-observed, not sampled, so earlier-in-day public plugins are overrepresented when the limit is reached. Analysis must report overflow and must not treat the named subset as random. + +### Rules shared by hook aggregates + +A latency histogram is exactly: + +```json +{"bounds":[5,10,25,50,100,250,500,1000],"counts":[0,0,0,0,0,0,0,0,0]} +``` + +The nine non-overlapping millisecond buckets mean `<=5`, `(5,10]`, `(10,25]`, `(25,50]`, `(50,100]`, `(100,250]`, `(250,500]`, `(500,1000]`, and `>1000`. Bounds and count-array length cannot vary. Fixed histograms can be merged correctly across installations; locally calculated percentiles cannot. + +Neither aggregate kind has `at`, an invocation id, a raw or scoped session id, or an individual invocation's time or outcome. No hook input, output, stdout, stderr, error text, or numeric exit detail is recorded. Symposium adds no timeout as part of telemetry. + +A host termination, lock conflict, storage limit, or I/O failure can omit an observation, so aggregate counts are lower bounds. No durable dropped-update counter can cover every loss mode because lock contention, termination, and I/O failure can also prevent writing that counter. Counting only cap rejections would imply false completeness. + +These rows do reveal exact daily counts for each hook surface. In particular, `pre_tool_use`, `post_tool_use`, and `user_prompt_submit` counts are proxies for daily tool and prompt activity even though no individual activity record exists. + +### `extension_invocation_metrics` + +This cumulative row combines skill-invocation observations for one UTC day, supported agent, active identifier epoch, and bounded skill bucket. Version 1 supports only Claude Code and accepts only its fixture-tested `Skill` signal. Symposium does not append one row per invocation. + +| Field | Values | Meaning | +| ------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| `agent` | `claude` | The only version 1 agent with a supported structured skill-invocation signal. | +| `target_scope` | `public`, `unnamed`, `overflow` | Whether the bucket names an eligible public skill. | +| `target.type` | `skill`, conditional | Present only when `target_scope=public`. | +| `target.source` | `symposium-recommendations`, `crates-io`, conditional | Reviewed public source; present only when `target_scope=public`. | +| `target.name` | validated string, conditional | Public skill name; present only when `target_scope=public`. | +| `unnamed_reason` | `ineligible`, `not_indexed`, `attribution_unavailable`, `ambiguous`, `invalid_signal`, conditional | Present only when `target_scope=unnamed`. | +| `attempted` | integer | Valid Claude `PreToolUse:Skill` observations merged into the row. | +| `completed` | integer | Successful Claude `PostToolUse:Skill` observations merged into the row. | +| `failed` | integer | Terminal Claude `PostToolUseFailure:Skill` observations merged into the row. | +| `session_counts_complete` | boolean | Whether both distinct identified-session counts are complete for all contributing observations. | +| `identified_sessions` | integer, optional | Distinct sessions with an attempted observation; present only when session counts are complete. | +| `identified_sessions_completed` | integer, optional | Distinct sessions with a completed observation; present only when session counts are complete. | +| `extension_subject` | scoped id, conditional | Present only for a public bucket; matches the selected safe resolution path for 30 days. | + +#### Count semantics + +`attempted`, `completed`, and `failed` are independent lower bounds. Each hook phase updates the snapshot separately. If one update is lost, completed plus failed need not equal attempted and may exceed it. The two session counts have the same independence. + +`failed` advances only when the supported agent emits its targeted failure signal. For version 1, that is Claude's `PostToolUseFailure:Skill` event. Symposium does not infer failure from denial, termination, or a missing terminal observation. Completion means that the agent activated the skill; it does not show whether the agent followed its instructions or improved the task result. + +#### Attribution + +Public attribution comes only from the generated installation index and a matching Symposium marker fingerprint. Unnamed reasons have these meanings: + +- `ineligible`: the installed skill was private, local, invalid, or otherwise unsafe to name. +- `not_indexed`: the index was valid and readable but contained no matching agent-facing identifier. +- `attribution_unavailable`: the index was missing, corrupt, or stale. +- `ambiguous`: more than one entry matched. +- `invalid_signal`: the Claude payload did not match the validated schema. + +No reason exposes the raw identifier. + +#### Row and session limits + +At most 128 public-skill rows may be named across identifier epochs in one UTC day. Later public skills merge into one `overflow` row per agent and epoch. Each fixed unnamed reason produces at most one row per agent and epoch. The public subset is first-observed, not sampled; analysis must report overflow and include unnamed counts when interpreting the public denominator. + +The same 256-id all-or-nothing rule applies to the attempted and completed session sets. On the first missing id, overflow, or state/snapshot mismatch, Symposium discards both sets, writes `session_counts_complete: false`, and omits both counts for the rest of that row and day. + +This aggregate has no `at`, duration, raw or scoped session id, invocation id, tool name, tool input/output, prompt, transcript, or individual outcome row. Unsupported agents emit no row, which means unknown rather than zero. + +### `command` + +This row records one completed eligible top-level user command. + +| Field | Values | Meaning | +| ----------------- | ---------------- | ------------------------------------------------------------- | +| `at` | UTC second | Completion time. | +| `command` | typed coordinate | Fixed built-in, or eligible public plugin command coordinate. | +| `duration_ms` | integer | Top-level command duration. | +| `outcome` | `ok`, `error` | Closed result. | +| `command_subject` | scoped id | Deduplicates this command for 30 days. | + +Arguments are never recorded. Internal `hook`, all `telemetry` commands, and ineligible external/plugin commands do not produce command events. + +Built-in names are `init`, `sync`, `search`, `use`, `status`, `plugin_sync`, `plugin_list`, `plugin_show`, `plugin_validate`, `self_update`, and `crate_info`: + +```json +{"type":"builtin","name":"use"} +``` + +An eligible plugin command contains only its reviewed public-source label, public plugin name, and declared command name: + +```json +{"type":"plugin","source":"symposium-recommendations","plugin":"example-tools","name":"example-check"} +``` + +### `storage_limit` + +This row means that the next complete low-volume event batch did not fit in the shared daily 8 MiB allowance. In addition to the common fields, `dropped_operation` is `session_start`, `manual_sync`, `use`, `remove`, `init`, `configuration`, or `command`. It identifies the top-level operation whose batch was rejected. + +The row appears at most once per UTC day, and the marker itself counts toward 8 MiB. An aggregate-metric update that would exceed its separate 512 KiB maximum or the remaining shared allowance is dropped without this marker and does not stop low-volume recording. The marker does not report lock-contention, aggregate-update, or crash losses. + +## What is never recorded + +- Individual prompt/tool activity records, per-invocation hook records, or individual skill-invocation records. +- Prompt text or any substring of it. +- Tool names, arguments, responses, or results. +- Shell commands, file contents, patches, or URLs from user/agent data. +- Error messages, debug strings, hook stdout/stderr, numeric exit details, or raw agent/package-manager payloads. +- Raw agent-facing or private skill identifiers and ephemeral invocation identifiers. +- File paths, workspace roots, project names, repository ids, or dependency-set snapshots. +- Environment names/values, hostname, username, home directory, IP address, model name, or account/vendor identifiers. +- Names or versions from private registries, git/path/workspace sources, or sources whose public identity is not allowlisted. +- Raw configured registry names/URLs or arbitrary git URLs. +- Raw vendor session ids, a global installation id, a workspace id, or one identifier shared across measurement purposes. +- Wall-clock timestamps more precise than one second. + +## Storage and expiry + +### Data files and recording + +Low-volume events are appended as JSON lines in `events-YYYY-MM-DD.jsonl` under the inspectable `/telemetry/` data directory (default `~/.symposium/telemetry/`). Current cumulative hook, plugin-hook, and extension-invocation aggregates are JSON lines in `metrics-YYYY-MM-DD.jsonl`. Symposium rewrites this bounded snapshot atomically after a merge. + +The lock in the telemetry directory also guards sibling private state. A recorder makes one non-waiting lock attempt. It may drop a complete buffered event batch or aggregate observation rather than delay your hook or command. Recording failures never change the user operation's result. + +### Daily limits and retention + +The event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB daily allowance. This allowance is a safety ceiling, not expected volume or preallocation. It bounds damage from a producer bug or unexpectedly large resolution batch. + +Aggregate metrics may use at most 512 KiB, so high-volume hook and skill activity cannot consume the allowance reserved for resolution and configuration events. + +A file is eligible for deletion only when `current_utc_day - file_utc_day > 30`. A D0 file remains throughout D30 and is first eligible on D31. Together with D31 expiry, the daily limit bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. Cleanup is lazy, so an old file remains until a recording-capable invocation or telemetry command runs. + +### Private state + +The sibling private `/telemetry-state.toml` holds the identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, bounded keyed session sets, and snapshot contribution counts used to calculate complete distinct-session counts. + +Symposium creates and replaces it atomically with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. + +The session sets are not printed or copied into metric rows. Symposium discards them at UTC-day rollover and removes them when `telemetry clear` or `telemetry reset-identifiers` runs. State is replaced before the corresponding metric snapshot. If a later snapshot write fails, a contribution-count mismatch on the next update discards the sets and permanently marks the row's session counts incomplete for that day. + +`telemetry clear` deletes event and aggregate-metric files and rewrites private state to remove pending sets while preserving the identity key and current anchors. `telemetry reset-identifiers` rotates future identifiers and starts a new retention cohort. `telemetry disable` stops recording; existing files remain unless the user accepts its interactive clear offer or runs `telemetry clear` later. + +### Installation index + +The generated `/.symposium/index-v1.json` file is installation state, not telemetry. It may contain local agent-facing identifiers, installed-directory fingerprints, eligibility, and safe public attribution needed to identify the skill Symposium installed. It is gitignored, atomically replaced after sync, excluded from `telemetry show` and `telemetry clear`, and not uploaded by this RFD. + +### Limits of interpretation + +Public names, versions, and exact counts can be identifying when they are unusual. Exact daily hook counts disclose approximate prompt/tool activity by surface. Extension-invocation rows disclose exact skill-attempt, completion, and failure counts. + +Lock contention, process termination, full files, and I/O failures can make the log incomplete. The records are therefore pseudonymous, locally associated, and best-effort. They must not be described as anonymous or as proof that an unrecorded action did not happen. diff --git a/md/rfds/telemetry-recording/reference/configuration.md b/md/rfds/telemetry-recording/reference/configuration.md new file mode 100644 index 00000000..c8b0abc2 --- /dev/null +++ b/md/rfds/telemetry-recording/reference/configuration.md @@ -0,0 +1,50 @@ +# Telemetry configuration + +Telemetry consent belongs to the current user. Symposium reads it from the user configuration file, normally `~/.symposium/config.toml`. Project configuration cannot enable, disable, or grant consent for telemetry. + +## Consent states + +```toml +[telemetry] +enabled = true +consent-version = 1 +``` + +| Key | Type | Default | Meaning | +| ----------------- | ---------------- | ------- | --------------------------------------------------- | +| `enabled` | boolean | `false` | The user's stored choice to permit local recording. | +| `consent-version` | positive integer | absent | Disclosure version the user acknowledged. | + +Symposium records telemetry only when `enabled` is true and `consent-version` matches the disclosure used by the current binary. For consent version 1: + +| Configuration | Effective state | +| -------------------------------------------- | -------------------------------- | +| absent section or `enabled = false` | Disabled | +| `enabled = true` with absent/non-`1` version | Consent required; record nothing | +| `enabled = true` and `consent-version = 1` | Enabled | + +The binary chooses the current consent version. The configuration value records which disclosure the user accepted; it cannot select an arbitrary version. The [version 1 disclosure requirements](./telemetry-command.md#disclosure-requirements) define what that disclosure covers. + +The complete prompt on that page is an example, not fixed implementation text. Interactive `init` and `telemetry enable` present the same final text approved by the team. + +## When consent must be renewed + +A future release may require a higher consent version after changing the recorded categories, linkability, timestamp precision, public-name eligibility, retention, or a normative exclusion. Until the user accepts that version, the effective state is `Consent required` and Symposium records nothing. + +Adding an agent enum value creates a new schema version for each affected event kind so typed readers do not reinterpret old schemas. It does not by itself require renewed consent when the recorded fields, categories, timestamp precision, and correlation boundaries remain unchanged. New fields, hook surfaces, or linkage for that agent do require a higher consent version. + +Accepting a newer consent version rotates the telemetry identity key and starts a new D0-D30 return cohort. Existing event and aggregate-metric files are neither rewritten nor deleted, but their scoped identifiers cannot link to later rows. + +## Changing telemetry state + +`cargo agents telemetry enable` presents the team-approved disclosure for the current version before writing these values. Interactive `cargo agents init` presents the same text for new users and existing unversioned opt-ins, defaulting to no. Non-interactive `init` does not grant or upgrade consent. + +`cargo agents telemetry disable` sets `enabled = false` without deleting existing telemetry data files. + +## Related private and installation state + +Consent configuration is separate from the random identity key and current identifier-window and cohort anchors in private `/telemetry-state.toml` (default `~/.symposium/telemetry-state.toml`). This state sits outside the inspectable `/telemetry/` data directory and uses owner-only permissions where supported. + +It persists across `disable` and `clear`, keeping identifiers consistent inside the active window. It is not configuration, and Symposium never reads it from project configuration. + +Symposium-managed project skills may also have a generated `/.symposium/index-v1.json` installation index. It associates the agent-facing skill identifier with the resolved installation that sync produced. This file is gitignored installation state, not project configuration or telemetry; changing it cannot grant consent, and telemetry commands do not inspect or delete it. diff --git a/md/rfds/telemetry-recording/reference/telemetry-command.md b/md/rfds/telemetry-recording/reference/telemetry-command.md new file mode 100644 index 00000000..8c92096d --- /dev/null +++ b/md/rfds/telemetry-recording/reference/telemetry-command.md @@ -0,0 +1,213 @@ +# `cargo agents telemetry` + +Use this command to manage per-user local telemetry: check whether it is enabled, opt in, inspect what is stored, stop recording, delete recorded data, or rotate future identifiers. See [What Symposium records](../contract/recorded-data.md) for the exhaustive field list and [Telemetry configuration](./configuration.md) for consent semantics. + +Telemetry is off by default. Nothing is uploaded. + +## Usage + +```text +cargo agents telemetry [status] +cargo agents telemetry enable [--acknowledge] +cargo agents telemetry disable +cargo agents telemetry show [--count N] +cargo agents telemetry clear +cargo agents telemetry reset-identifiers +``` + +Start with `status`. If you choose to participate, run `enable` and review the disclosure. Use `show` whenever you want to inspect the stored records. `disable` stops future recording, while `clear` deletes recorded data. + +Telemetry management commands are never recorded as command telemetry. + +## `status` + +Use `status` to see the effective consent state and a summary of the local event and aggregate-metric files: + +```console +$ cargo agents telemetry status +Telemetry: enabled (consent version 1) + data directory: ~/.symposium/telemetry/ + stored: 3 file(s), 28.4 KiB + physical lines: 71 + supported rows: 68 + unknown schemas: 2 + malformed lines: 1 + range: 2026-08-01 through 2026-08-03 +``` + +Possible states are `disabled`, `consent required`, and `enabled`. `Consent required` means the config contains an earlier or unversioned opt-in; Symposium records nothing until you accept the current disclosure. + +Stored files/bytes and physical lines cover daily event and aggregate-metric files; they exclude `.lock`, temporary files, and sibling private `telemetry-state.toml`. Supported rows are lines the current binary recognizes by kind and schema version. Unknown and malformed lines stay on disk and remain visible through `show`. `status` never prints the secret identity key or pending keyed session sets. + +## `enable` + +`enable` presents the current disclosure and asks whether to begin local recording. It does not enable telemetry unless the user explicitly accepts. + +### Disclosure requirements + +Before asking for consent, the team-approved version 1 disclosure must make these points clear: + +- Telemetry is off by default, requires an explicit opt-in, and defaults to no in both interactive entry points. +- The recorded categories are: + - observed sessions and configured agents, including Symposium version, agent labels, and platform classes; + - public package names and exact versions, public resolution relationships, aggregate sync results, and reasons packages remain unnamed; + - exact daily hook, plugin-hook, and agent skill-activation metrics, with structured skill activation available only for Claude in version 1; and + - completed eligible commands without arguments, plus storage-limit markers. +- Exact hook-surface counts can approximate daily prompt/tool activity. The disclosure distinguishes second-precision session/command timestamps from day-only aggregate and resolution rows. +- Purpose-scoped pseudonymous identifiers permit the stated links for up to 30 days, including one D0-D30 observed-session cohort across agents; they are not anonymous or a global project/workspace identity. +- The main exclusions are stated plainly, including prompt/tool content, paths, project/workspace identity, environment and machine identity, raw errors and agent/package-manager payloads, private package/plugin/skill names, and individual hook or skill-invocation rows. +- Recorded data and private identifier/counting state remain local in their stated locations, nothing is uploaded, data remains through D30 and is eligible for deletion on D31, and the disclosure names the inspection and deletion commands. + +The [exhaustive field list](../contract/recorded-data.md) and [never-record list](../contract/recorded-data.md#what-is-never-recorded) define the details behind these requirements. The team may revise structure, tone, and wording before implementation, provided the final disclosure preserves every required point and its meaning. + +`cargo agents init` and `cargo agents telemetry enable` must use the same approved text and both default to no. + +### Example version 1 disclosure + +The following non-normative example demonstrates complete coverage. It is a review aid and starting point for the final product copy, not the string the implementation must reproduce. + +```text +Symposium telemetry is off by default. If enabled, Symposium records: +- observed session starts and configured agents, with fresh/resumed status when + available, Symposium version, operating-system class, and architecture; +- public package names and exact versions, public plugin/skill resolution paths, + aggregate sync results, reasons packages remain unnamed, and whether only + unnamed extensions matched; +- exact daily Claude skill-activation attempts, completions, and failures, + public skill names, distinct-session counts when complete, fixed unnamed + reasons, and overflow counts; +- exact daily hook and plugin-hook counts, outcomes, and latency histograms; +- completed built-in and eligible public plugin commands, with outcome and + duration but without arguments; +- storage-limit markers naming only the affected operation; and +- purpose-scoped pseudonymous identifiers that link repeated observations for + up to 30 days, plus one D0-D30 observed-session cohort across agents. + +Counts for pre-tool-use, post-tool-use, and user-prompt-submit approximate daily +tool and prompt activity. Session starts and commands include UTC timestamps +truncated to one second; other rows include only the UTC day. +Pseudonymous does not mean anonymous: the scoped identifiers permit only the +links described above, and no identifier links all telemetry or identifies a +project or workspace. + +Symposium never records prompt or tool content, tool names or arguments, file +paths, project or workspace identity, environment values, hostname, username, +model/account/vendor identifiers, raw errors or agent/package-manager payloads, +private package, plugin, or skill names, raw agent-facing skill identifiers, +individual hook-invocation rows, or individual skill-invocation rows. + +Recorded data stays on this machine in ~/.symposium/telemetry/. A private +identifier key and bounded counting state stay in +~/.symposium/telemetry-state.toml; telemetry commands never print them. +Nothing is uploaded. +Files remain through day 30 and become eligible for deletion on day 31. Inspect +them with cargo agents telemetry show and delete them with +cargo agents telemetry clear. + +Enable telemetry under consent version 1? [y/N] +``` + +In a non-interactive environment, `enable` does not change config unless `--acknowledge` is supplied explicitly. This prevents scripts or a manually retained unversioned boolean from upgrading consent silently. + +Enabling telemetry does not rewrite or assign new identifiers to old stored lines. Accepting a new consent version rotates the secret identity key and starts a new retention cohort, severing old and new scoped identifiers. + +## `disable` + +`disable` stops future recording by setting `enabled = false`. Existing event and aggregate-metric files remain on disk. In an interactive terminal, `disable` offers to clear them and defaults to keeping them. In a non-interactive environment, it prints the `clear` command instead of deleting data. + +## `show` + +`show` prints stored event and current aggregate-snapshot JSONL lines. The ordering is deterministic: + +1. UTC days appear in ascending order. +2. Within a day, append-only event lines appear first in physical order. +3. Aggregate rows follow in kind, agent, hook or target scope, public source/name or unnamed reason, and event-id order. + +The event id breaks ties when an identifier reset creates two aggregate epochs in one day. `--count N` returns the last `N` lines in this ordering: + +```console +$ cargo agents telemetry show --count 2 +{"v":1,"kind":"command","event_id":"5d18caa8-84f7-4aa3-846c-99ea810ccd85","day":"2026-08-03","at":"2026-08-03T10:02:11Z","symposium":"0.4.0","command":{"type":"builtin","name":"use"},"duration_ms":820,"outcome":"ok","command_subject":"cmd_adf0c14ddc35b97762b5daae6f4119ce"} +{"v":1,"kind":"hook_metrics","event_id":"b563dd02-0301-4e2c-aac4-2e0d5dfaa977","day":"2026-08-03","symposium":"0.4.0","agent":"claude","hook":"pre_tool_use","invocations":500,"outcomes":{"ok":496,"blocked":1,"plugin_error":3,"internal_error":0},"plugins_attempted":500,"plugins_completed":500,"duration_ms":{"bounds":[5,10,25,50,100,250,500,1000],"counts":[8,17,76,144,181,68,6,0,0]},"session_counts_complete":true,"identified_sessions":4,"identified_sessions_non_ok":2,"hook_subject":"hok_b5b707841de7695912bec9b8bca382e8"} +``` + +The command copies stored line bytes; it does not parse, normalize, repair, or pretty-print them. Unknown-version and malformed lines are shown as stored. The current day's aggregate file is a cumulative snapshot, not a history of its replaced versions. + +Aggregate rows have no `at`, so storage order is not chronology. All output comes from the same Symposium home, and line order or day can expose co-occurrence even though the rows have no global installation or workspace id. Review the complete output before sharing it. + +Redirect the output to create a local copy: + +```bash +cargo agents telemetry show --count 100000 > telemetry.jsonl +``` + +No separate export command is part of this RFD. + +## `clear` + +`clear` deletes every `events-YYYY-MM-DD.jsonl` and `metrics-YYYY-MM-DD.jsonl` file. It acquires the telemetry lock and also discards pending aggregate session-count sets: + +```console +$ cargo agents telemetry clear +Deleted 12 telemetry data file(s) from ~/.symposium/telemetry/. +``` + +`clear` preserves the telemetry directory, lock, identity/cohort state, identity key, and consent setting. New rows in the same identifier window can therefore carry the same scoped subjects as cleared rows. Severing that future linkage also requires `reset-identifiers`. + +## `reset-identifiers` + +`reset-identifiers` severs identifier linkage between future and existing rows. It acquires the telemetry lock, replaces the secret identity key, discards pending aggregate session-count sets, and starts a new retention cohort: + +```console +$ cargo agents telemetry reset-identifiers +Telemetry identifiers reset. Existing telemetry data files were not changed. +``` + +The command neither deletes nor rewrites old event or aggregate-metric rows. Identifiers before and after the reset cannot be derived into each other from the data files. + +If no identity state exists, the command reports that there is nothing to reset instead of creating a key. If existing state is unreadable or malformed, recording remains stopped until this command explicitly replaces it. + +## Files and private state + +```text +~/.symposium/ +|-- telemetry-state.toml +`-- telemetry/ + |-- .lock + |-- metrics-2026-08-03.jsonl + `-- events-2026-08-03.jsonl +``` + +Each project skills parent may also contain a generated `.symposium/index-v1.json` installation index. It maps agent-facing skill identifiers to Symposium-managed installations so a later hook can attribute a skill activation. The index is gitignored installation state, not telemetry: `show`, `clear`, retention, and identifier reset do not read or delete it, and this RFD does not upload it. + +`telemetry-state.toml` is private Symposium state outside the inspectable telemetry data directory. It contains the secret identity key, current identifier-window and return-cohort anchors, cleanup and marker metadata, and bounded keyed session sets plus contribution counts used for complete aggregate session counts. All recorders read this state under the telemetry lock. + +Normal 30-day rollover changes the window anchor without replacing the key. Renewed consent or `reset-identifiers` replaces the key; `disable` and `clear` preserve it. + +Symposium atomically creates and replaces the file with owner-only permissions where supported. Replacement uses a same-directory temporary file beside `config.toml`; abandoned state temporaries are ignored and cleaned lazily under the telemetry lock. + +`show`, `status`, data retention, and `clear` do not expose or delete the state file. `clear` rewrites it only to remove pending sets, preserving the key and current anchors. + +Session sets and contribution counts are never copied into metric rows. Symposium discards them at day rollover or when `clear` or `reset-identifiers` runs. `show` and `status` do not lock writers, so a summary spanning several files is not an atomic snapshot. + +## Concurrent recording + +Recorders make one non-waiting attempt on the lock in the telemetry data directory. The lock guards data and private state mutations. On contention, the recorder drops the entire event batch or aggregate observation rather than delaying the agent or command. + +Event batches are appended. Hook, plugin-hook, and extension-invocation observations are merged into a bounded, canonically ordered snapshot using a same-directory temporary write and atomic replace. A crash leaves either the old or new complete snapshot; abandoned temporary files are ignored and cleaned lazily. + +Session-count state is atomically replaced first and carries the snapshot contribution count. After a failed snapshot write, a mismatch discards the sets and makes the row's session counts incomplete for that day. + +Management commands can wait for the lock. A crash can still lose the last batch or metric update, or leave a partial final event line; `status` reports that line as malformed and `show` preserves it. + +Hook and extension-invocation counts are lower bounds. There is no durable all-cause dropped-update counter because contention, termination, and I/O failure can also prevent writing that counter. + +## Size and expiry + +Each UTC day's event file, aggregate-metric snapshot, and reserved maximum-size `storage_limit` line share an 8 MiB allowance. This is a safety ceiling, not expected volume or preallocation. It bounds damage from a producer bug or unexpectedly large resolution batch. + +Aggregate metrics may use at most 512 KiB. An update that would exceed that maximum or the remaining shared allowance is dropped without stopping low-volume event recording. + +Telemetry files remain through D30 and become eligible for deletion on D31, when `current_utc_day - file_utc_day > 30`. Together with D31 expiry, the daily allowance bounds ordinary retained telemetry near 248 MiB, excluding temporary files and private state. + +Cleanup runs lazily, at most once per day, when a recording-capable or telemetry command next runs. Uninstalling Symposium does not delete these files.