Sync agenteye-cli skill (agenteye@4e839a6)

skills/agenteye-cli/SKILL.md

@@ -5,7 +5,7 @@ description: |-
 
   Trigger when the user wants to:
   • inspect agent telemetry — did agents error/fail/go flaky; sessions, events, latency, token usage, slowest models; eval/quality scores and whether quality dropped;
-  • operate the deployment — ack/assign/resolve firing alerts and incidents with notes; see who has access and change roles (e.g. make someone read-only); create or scope API keys (e.g. a CI key that only pushes events); change settings; run saved or ad-hoc ClickHouse queries.
+  • operate the deployment — ack/assign/resolve/mute/dismiss issues (alerts, reports, and audit findings) with notes; run and triage audits; see who has access and change roles (e.g. read-only); create or scope API keys (e.g. a push-only CI key); change settings; run saved or ad-hoc ClickHouse queries.
 
   Served by the `agenteye` CLI against an AgentEye platform.
 
@@ -68,10 +68,11 @@ Before real work, run `agenteye --json whoami` and react to the exit code:
 - **exit 4** → not signed in. Tell the user to run `agenteye login` (it emails a
   one-time code and prompts interactively — you can't complete it for them, and
   don't fabricate a token).
-- **base-url error / exit 2 or 3 about base-url** → the deployment URL isn't
-  set. Ask the user for their dashboard URL and pass `--base-url <url>` (or they
-  set `AGENTEYE_DASHBOARD_URL`). For a local dev stack it's usually
-  `http://localhost:3000`.
+- **base-url** → the CLI defaults to the hosted product,
+  `https://app.befailproof.ai`, so a plain `agenteye login` works out of the box.
+  Only pass `--base-url <url>` (or set `AGENTEYE_DASHBOARD_URL`) for a self-hosted
+  or dev deployment — a local dev stack is usually `http://localhost:3000`. A
+  scheme-less URL is rejected as a usage error (exit 2).
 - **exit 0** → `whoami` returns the active org slug and your permissions; trust
   that for the org name and to know what you're allowed to do before attempting a
   gated command (don't assume a particular org slug — read it from `whoami`).
@@ -101,8 +102,10 @@ ask** — don't rename-and-retry or rotate/regenerate the existing one. A
 
 State-changing commands: `keys create/update/disable/regenerate`,
 `users create/update/disable/enable`, `settings set`,
-`alerts create/update/delete/test`, the writing `incidents` subcommands
+`alerts create/update/delete/test`, the writing `issues` subcommands
 (`ack/assign/resolve/comment-add/comment-delete/subscribe/unsubscribe/open`),
+`audits create/edit/delete/run` and the finding-triage verbs
+(`ack/mute/dismiss/resolve/reopen/assign`),
 `query create/update/delete`, `agent rename/delete`, and `orgs switch`.
 Read-only commands (§5 "Observe") never need this.
 
@@ -123,7 +126,8 @@ you need a flag you don't already know.
 - `users list|show|create|update|disable|enable` — referenced by **email**.
 - `settings list|schema|set` — fixed registry; `schema` shows what each key accepts.
 - `alerts list|show|create|update|delete|test` — referenced by **name**.
-- `incidents list|count|show|ack|assign|resolve|comment-add|comment-list|comment-delete|subscribe|subscribers|unsubscribe|open` — by id (short ids accepted).
+- `issues list|count|show|ack|assign|resolve|comment-add|comment-list|comment-delete|subscribe|subscribers|unsubscribe|open` — by id (short ids accepted). **One board for everything needing attention**: alert breaches, hand-raised issues, and audit findings, told apart by a `source` of `alert` / `manual` / `audit`. (This group was called `incidents` before; the old name is gone.)
+- `audits list|show|create|edit|delete|run|runs` — scheduled sweeps, referenced by **name**; `audits findings|finding` + the triage verbs `ack|mute|dismiss|resolve|reopen|assign` act on a finding **id**. `audits run <name>` only *queues* a run (poll `audits runs <name>` for completion). See §8.
 
 **Analytics & assistant:**
 - `query list|show|create|update|delete|run|schema` — saved ClickHouse SQL + ad-hoc runner (`query run <name>` or `query run --sql "…"`); `query schema [table]` for table layout.
@@ -144,8 +148,9 @@ command (`list <kind>`, `whoami`, a `list` subcommand) before committing.
 | "how are my agents doing?", "show recent runs" | `sessions --since 24h` (add `--status error` for just failures) |
 | "are the evals / quality scores ok?", "did quality drop?" | `evals --aggregate`; drill with `evals --score <key>:..0.5` |
 | "how many events / how much traffic last week?" | `query schema` then `query run --sql "SELECT count() FROM events WHERE ts >= now() - INTERVAL 7 DAY"` |
-| "is anything on fire?", "any alerts firing / open incidents?" | `alerts list` + `incidents list` (and `incidents count`) |
-| "ack / look at / resolve that incident" | `incidents list` → `incidents show <id>` → **confirm** → `incidents ack`/`resolve <id>` |
+| "is anything on fire?", "any alerts firing / open issues?" | `alerts list` + `issues list` (and `issues count`) |
+| "ack / look at / resolve that issue" | `issues list` → `issues show <id>` → **confirm** → `issues ack`/`resolve <id>` |
+| "run an audit", "what did the audit find?", "any findings to triage?" | `audits list` → `audits run <name>` (queues) → `audits runs <name>` (wait for `succeeded`) → `audits findings --audit <name>`; triage with `audits resolve/mute/dismiss <id>` — **confirm first** |
 | "give CI / this service an API key" | `keys create <name> --add events:add` (scope to what they describe) — **state it, then create**; capture the one-time secret |
 | "who has access?", "add / remove a teammate", "make them read-only" | `users list` / `users show <email>` / `users create`/`update`/`disable` |
 | "change a setting", "what can I configure?" | `settings schema` (what's tunable) then `settings set <key> --value …` — **confirm first** |
@@ -185,13 +190,38 @@ agenteye --json events --full --session-id run-001 --all | jq '.events[].payload
   limit: **`--all --limit 1000`** (or higher). When you only need the totals, use
   `--aggregate` — it covers the whole window regardless of row caps, so it's the
   reliable cross-check that you pulled everything.
-- **Triage flow:** `incidents list` → `incidents show <id>` (read the activity
-  log) → confirm with the user → `incidents ack <id>` or `resolve <id>`.
+- **Triage flow:** `issues list` → `issues show <id>` (read the activity
+  log) → confirm with the user → `issues ack <id>` or `resolve <id>`.
 - **Investigate a regression:** `evals --aggregate` to see which score dropped →
   `evals --score helpfulness:..0.5` to list the bad runs → `events --session-id <id>`
   to see what happened inside one.
 
 When you've pulled what you need, answer the user in prose or a small table —
 don't paste raw JSON back unless they asked for it.
 
-<!-- ci: no-op touch to exercise the skill-sync trigger (safe to remove) -->
+## 8. Audits — the async sweep, and how findings become issues
+
+An **audit** is a scheduled sweep that analyses recent agent behaviour (errors,
+runaway tool loops, leaked secrets, low eval scores, …) and emits **findings**.
+Two things about the flow matter when driving it from the CLI:
+
+- **`audits run <name>` is asynchronous — it only *queues*.** A `{"queued": true}`
+  does NOT mean the run finished (the analysis can take minutes). Poll
+  `audits runs <name>` until the newest row reads `succeeded` (or `failed`) before
+  reading findings — don't assume results are ready on the call that queued them.
+  A disabled audit, or one already mid-run, refuses to queue (exit 1).
+- **Findings ARE issues — it's one bucket.** Every finding graduates to an issue
+  (`source = audit`) and carries its full content there, so the same problem shows
+  up under both `audits findings` and `issues list`. Triage is **globally
+  consistent in both directions**: `audits resolve <finding-id>` closes the linked
+  issue, and `issues resolve <issue-id>` on an audit issue resolves the finding —
+  either surface works, they never disagree. Triage a finding with
+  `audits ack|mute|dismiss|resolve|reopen <id>` (durable **mute/dismiss** suppress
+  the pattern org-wide by fingerprint; **resolve** leaves no suppression, so a true
+  recurrence reopens as new). Reads need `audits:read`, every mutation
+  `audits:write` (note: triaging a finding needs `audits:write`, not an `issues:*`
+  permission — the audit is the system of record and the issue follows it).
+
+Typical end-to-end: `audits list` → `audits run <name>` → poll `audits runs <name>`
+→ `audits findings --audit <name>` (highest priority first) → `audits finding <id>`
+for the full write-up → **confirm with the user** → `audits resolve <id>`.

skills/agenteye-cli/references/commands.md

@@ -13,7 +13,8 @@ command**, **`--json` to parse**, **branch on exit codes**.
 - [users](#users)
 - [settings](#settings)
 - [alerts](#alerts)
-- [incidents](#incidents)
+- [audits](#audits)
+- [issues](#issues)
 - [query](#query)
 - [agent](#agent)
 
@@ -22,7 +23,7 @@ Set on the CLI, **before** the subcommand. Precedence: flag > env var > config f
 
 | Flag | Env var | Meaning |
 |---|---|---|
-| `--base-url <url>` | `AGENTEYE_DASHBOARD_URL` | Dashboard URL. **Required** (no default); must start with `http://`/`https://`. |
+| `--base-url <url>` | `AGENTEYE_DASHBOARD_URL` | Dashboard URL. Defaults to `https://app.befailproof.ai` (the hosted product); override for self-hosted/dev. Must start with `http://`/`https://`. |
 | `--org <slug>` | `AGENTEYE_ORG` | Active org for this command (multi-tenant override). |
 | `--token <t>` | `AGENTEYE_CLI_TOKEN` | Session token (normally from config after `login`). |
 | `--json` | `AGENTEYE_CLI_JSON` | Machine-readable JSON to stdout, nothing else. |
@@ -118,15 +119,41 @@ A fixed registry — you read/inspect/change existing keys, you cannot create ne
 - `settings list` — `key · value · type · updated` (secrets masked).
 - `settings schema` — `key · type · accepts · description` (what each key accepts).
 - `settings set <key> (--value V | --json-value JSON | --file f)` — exactly one value source. Unknown key → exit 6. No-op if unchanged. Server validation errors surface as `✗ <message>` (e.g. range bounds). Some keys are sensitive (signing secrets, sign-in allowlist) — confirm carefully.
-
-## incidents
-Alert incidents; referenced by id (short ids accepted, `--show-id` shows them).
-- `incidents list [--state firing|acknowledged|resolved] [--severity ...] [--alert-id <id>] [--limit N]`
-- `incidents count`
-- `incidents show <id>` — identity + comments + subscribers + **activity log** (read this before acting).
-- `incidents ack <id>` · `incidents assign <id> <member>` (member must be an operator) · `incidents resolve <id>` (calm confirm) · `incidents open --alert-id <id> [--severity ...]`
-- `incidents comment-add <id> <text>` · `comment-list <id>` · `comment-delete <id> <comment-id>`
-- `incidents subscribe <id>` · `unsubscribe <id>` · `subscribers <id>`
+  - `allowed_sign_ins` restricts which of the organization's members may sign in; it does not grant access to anyone else. An **empty list means no restriction** (every member can sign in), so clearing it widens access rather than removing it. A non-empty list admits only matching addresses and locks out every other member. Entries are exact addresses or `*@domain.tld`; a bare `*` is rejected — use an empty list. Saving a list that does not include your own address is refused, because you could not sign in again.
+
+## audits
+Scheduled sweeps over agent activity, and the **findings** they produce. Audits are referenced by **name** (UUID id also accepted); findings by **id** (short ids shown in the table, `--show-id` for the full ones).
+- `audits list [--enabled-only] [--show-id]` — `created · name · every · findings · status · last run`; disabled audits are dimmed and the footer carries the on/off split plus the open-finding total.
+- `audits show <name>` — identity + `schedule` / `scope` / `analysis` / `channels` cards. The creator and the raw `scope`/`channels` blobs live here and in `--json`, not in the list.
+- `audits create <name> [--file f] [--description ...] [--enabled|--disabled] [--schedule-interval-secs N] [--schedule-anchor ISO8601] [--window-mode fixed|since_last] [--lookback-window-secs N] [--scope JSON] [--ignore-error-type <csv>] [--llm|--no-llm] [--top-k N] [--sensitivity low|medium|high] [--channels JSON]` — everything but the name has a server default. Name collision is pre-checked (exit 2). No confirm (creating isn't destructive).
+- `audits edit <name> [--name ...] [same flags as create] [--yes]` — the server replaces the whole definition, so a flag-only edit re-sends the current audit with your change applied (needs read **and** write). Rename onto an existing name → exit 2. Confirms first.
+- `audits delete <name> [--yes]` — amber preview (naming the findings and run history that go with it) + confirm.
+- `audits run <name>` — queue a run **now**, ahead of schedule. Success means queued, not finished; a disabled audit or one already mid-run is refused with the server's explanation (exit 1). JSON `{"queued": true}`.
+- `audits runs <name> [--limit N] [--show-id]` — run history, newest first: `started · status · trigger · findings · new · took`. A failed run's `error` and each run's `stats`/`report` are in `--json` only.
+- `audits findings [--audit <name>] [--run-id <id>] [--status <csv>] [--limit N] [--offset N] [--show-id]` — the triage queue, highest priority first: `id · title · severity · status · kind · seen · last`. With no `--status` you get the live set (open + recurring); valid statuses are `open recurring resolved dismissed muted`. `--audit` takes an audit **name**.
+- `audits finding <id>` — one finding in full: identity + `analysis` (what + likely cause) + `recommendation` (do / impact / effort) + `scope` + `evidence`. Empty sections are omitted.
+- `audits ack <id> [--reason ...]` — seen, stays visible, ranked lower. No confirm.
+- `audits mute <id> [--reason ...] [--yes]` — stop surfacing this pattern in future runs (durable). Confirms first.
+- `audits dismiss <id> [--reason ...] [--yes]` — judged not worth acting on; suppressed like mute. Confirms first.
+- `audits resolve <id> [--yes]` — you fixed it. Leaves **no** suppression, so a genuine recurrence is raised as new. Confirms first.
+- `audits reopen <id>` — back to `open` **and** clears any mute/dismiss suppression. The undo for the three above.
+- `audits assign <id> --to <email>` — set the owner; the status is untouched. `--to` is required (exit 2 without it).
+
+The title column truncates to whatever width is left so the fixed columns always survive — read the full text with `audits finding <id>` or `--json`. A bad `--status`, `--window-mode`, `--sensitivity`, a non-ISO-8601 `--schedule-anchor`, or an out-of-range `--schedule-interval-secs`/`--lookback-window-secs` is rejected before any request (exit 2). `--schedule-anchor` pins the fixed UTC slot runs land on (`anchor + N * interval`), so a slow run or `audits run` can't drift the cadence; omit it on create and the server uses the next 09:00 UTC. An unknown audit name → exit 6; an unknown or malformed finding id → calm `✗ no finding …`, exit 6. Reads need `audits:read`, every mutation `audits:write`.
+
+**`audits run` is async — it only queues.** Success is `{"queued": true}`, not a finished run; the analysis can take minutes. Poll `audits runs <name>` until the newest row is `succeeded`/`failed` before reading `audits findings`, rather than assuming results exist on the call that queued them.
+
+**Findings and issues are one bucket.** Every finding graduates to an issue (`source: audit`) that stores the finding's full content, so the same problem appears under both `audits findings` and `issues list`. Triage is consistent in **both** directions and needs `audits:write` either way: `audits resolve|mute|dismiss|ack|reopen <finding-id>` mirrors onto the linked issue, and `issues resolve <issue-id>` on an audit issue mirrors back onto the finding — the two never disagree. **resolve** leaves no suppression (a genuine recurrence reopens as new); **mute/dismiss** suppress the pattern org-wide by fingerprint.
+
+## issues
+The single board for everything needing human attention — alert breaches (`source: alert`), hand-raised issues (`manual`), and audit findings (`audit`). Referenced by id (short ids accepted, `--show-id` shows them). This group was **renamed from `incidents`**; the old name no longer exists. Reads and ack/comment need `issues:read`; opening, assigning, and subscribing others need `issues:create`; resolving needs `issues:close`.
+- `issues list [--state firing|acknowledged|resolved] [--alert-id <id>] [--limit N] [--show-id]` — there is **no** `--severity` filter on this command.
+- `issues count`
+- `issues show <id>` — identity + comments + subscribers + **activity log** (read this before acting). An audit-born issue (`source: audit`) carries the full finding it graduated from and back-links to the audit/run.
+- `issues ack <id>` · `issues assign <id> --assignee <member>` (repeatable; omit to clear all assignees; each must be an operator) · `issues resolve <id>` (calm confirm). **On an audit issue these stay in sync with the finding** — resolving the issue resolves the underlying audit finding, so it can't reappear on the next run (equivalently, triage it with `audits resolve <finding-id>`; both surfaces agree).
+- `issues open --summary <text> (--title <text> | --alert-id <id>) [--title ...] [--severity ...]` — `--title` is **required** for a standalone issue (nothing to borrow a name from); with `--alert-id` it is optional and defaults to the alert's name. Missing `--title` on the standalone path → exit 2.
+- `issues comment-add <id> (--body <text> | --file <path>)` — `--file -` reads stdin; exactly one of the two · `comment-list <id>` · `comment-delete <id> <comment-id>`
+- `issues subscribe <id> [--email <addr>]` · `unsubscribe <id> [--email <addr>]` · `subscribers <id>` — `--email` defaults to you; naming someone else needs `issues:create`
 
 Malformed (non-UUID) id → calm `✗ no incident …` exit 6. Assign to a non-operator → clean 422 message.