From be4cbc6d4fa9eb3b63bc9351128ca909b594c93e Mon Sep 17 00:00:00 2001 From: agenteye-skill-sync Date: Fri, 24 Jul 2026 17:08:48 +0000 Subject: [PATCH] Sync agenteye-cli skill from agenteye@31423de MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated mirror of cli/skill/. Source of truth: FailproofAI/agenteye — do not hand-edit. --- skills/agenteye-cli/SKILL.md | 54 +++++++++++++++++----- skills/agenteye-cli/references/commands.md | 49 +++++++++++++++----- 2 files changed, 80 insertions(+), 23 deletions(-) diff --git a/skills/agenteye-cli/SKILL.md b/skills/agenteye-cli/SKILL.md index 3284a5c..026af5d 100644 --- a/skills/agenteye-cli/SKILL.md +++ b/skills/agenteye-cli/SKILL.md @@ -5,7 +5,7 @@ description: |- Trigger when the user wants to: • inspect agent telemetry — did agents error/fail/go flaky; sessions, events, latency, token usage, slowest models; eval/quality scores and whether quality dropped; - • operate the deployment — ack/assign/resolve firing alerts and incidents with notes; see who has access and change roles (e.g. make someone read-only); create or scope API keys (e.g. a CI key that only pushes events); change settings; run saved or ad-hoc ClickHouse queries. + • operate the deployment — ack/assign/resolve/mute/dismiss issues (alerts, reports, and audit findings) with notes; run and triage audits; see who has access and change roles (e.g. read-only); create or scope API keys (e.g. a push-only CI key); change settings; run saved or ad-hoc ClickHouse queries. Served by the `agenteye` CLI against an AgentEye platform. @@ -68,10 +68,11 @@ Before real work, run `agenteye --json whoami` and react to the exit code: - **exit 4** → not signed in. Tell the user to run `agenteye login` (it emails a one-time code and prompts interactively — you can't complete it for them, and don't fabricate a token). -- **base-url error / exit 2 or 3 about base-url** → the deployment URL isn't - set. Ask the user for their dashboard URL and pass `--base-url ` (or they - set `AGENTEYE_DASHBOARD_URL`). For a local dev stack it's usually - `http://localhost:3000`. +- **base-url** → the CLI defaults to the hosted product, + `https://app.befailproof.ai`, so a plain `agenteye login` works out of the box. + Only pass `--base-url ` (or set `AGENTEYE_DASHBOARD_URL`) for a self-hosted + or dev deployment — a local dev stack is usually `http://localhost:3000`. A + scheme-less URL is rejected as a usage error (exit 2). - **exit 0** → `whoami` returns the active org slug and your permissions; trust that for the org name and to know what you're allowed to do before attempting a gated command (don't assume a particular org slug — read it from `whoami`). @@ -101,8 +102,10 @@ ask** — don't rename-and-retry or rotate/regenerate the existing one. A State-changing commands: `keys create/update/disable/regenerate`, `users create/update/disable/enable`, `settings set`, -`alerts create/update/delete/test`, the writing `incidents` subcommands +`alerts create/update/delete/test`, the writing `issues` subcommands (`ack/assign/resolve/comment-add/comment-delete/subscribe/unsubscribe/open`), +`audits create/edit/delete/run` and the finding-triage verbs +(`ack/mute/dismiss/resolve/reopen/assign`), `query create/update/delete`, `agent rename/delete`, and `orgs switch`. Read-only commands (§5 "Observe") never need this. @@ -123,7 +126,8 @@ you need a flag you don't already know. - `users list|show|create|update|disable|enable` — referenced by **email**. - `settings list|schema|set` — fixed registry; `schema` shows what each key accepts. - `alerts list|show|create|update|delete|test` — referenced by **name**. -- `incidents list|count|show|ack|assign|resolve|comment-add|comment-list|comment-delete|subscribe|subscribers|unsubscribe|open` — by id (short ids accepted). +- `issues list|count|show|ack|assign|resolve|comment-add|comment-list|comment-delete|subscribe|subscribers|unsubscribe|open` — by id (short ids accepted). **One board for everything needing attention**: alert breaches, hand-raised issues, and audit findings, told apart by a `source` of `alert` / `manual` / `audit`. (This group was called `incidents` before; the old name is gone.) +- `audits list|show|create|edit|delete|run|runs` — scheduled sweeps, referenced by **name**; `audits findings|finding` + the triage verbs `ack|mute|dismiss|resolve|reopen|assign` act on a finding **id**. `audits run ` only *queues* a run (poll `audits runs ` for completion). See §8. **Analytics & assistant:** - `query list|show|create|update|delete|run|schema` — saved ClickHouse SQL + ad-hoc runner (`query run ` or `query run --sql "…"`); `query schema [table]` for table layout. @@ -144,8 +148,9 @@ command (`list `, `whoami`, a `list` subcommand) before committing. | "how are my agents doing?", "show recent runs" | `sessions --since 24h` (add `--status error` for just failures) | | "are the evals / quality scores ok?", "did quality drop?" | `evals --aggregate`; drill with `evals --score :..0.5` | | "how many events / how much traffic last week?" | `query schema` then `query run --sql "SELECT count() FROM events WHERE ts >= now() - INTERVAL 7 DAY"` | -| "is anything on fire?", "any alerts firing / open incidents?" | `alerts list` + `incidents list` (and `incidents count`) | -| "ack / look at / resolve that incident" | `incidents list` → `incidents show ` → **confirm** → `incidents ack`/`resolve ` | +| "is anything on fire?", "any alerts firing / open issues?" | `alerts list` + `issues list` (and `issues count`) | +| "ack / look at / resolve that issue" | `issues list` → `issues show ` → **confirm** → `issues ack`/`resolve ` | +| "run an audit", "what did the audit find?", "any findings to triage?" | `audits list` → `audits run ` (queues) → `audits runs ` (wait for `succeeded`) → `audits findings --audit `; triage with `audits resolve/mute/dismiss ` — **confirm first** | | "give CI / this service an API key" | `keys create --add events:add` (scope to what they describe) — **state it, then create**; capture the one-time secret | | "who has access?", "add / remove a teammate", "make them read-only" | `users list` / `users show ` / `users create`/`update`/`disable` | | "change a setting", "what can I configure?" | `settings schema` (what's tunable) then `settings set --value …` — **confirm first** | @@ -185,8 +190,8 @@ agenteye --json events --full --session-id run-001 --all | jq '.events[].payload limit: **`--all --limit 1000`** (or higher). When you only need the totals, use `--aggregate` — it covers the whole window regardless of row caps, so it's the reliable cross-check that you pulled everything. -- **Triage flow:** `incidents list` → `incidents show ` (read the activity - log) → confirm with the user → `incidents ack ` or `resolve `. +- **Triage flow:** `issues list` → `issues show ` (read the activity + log) → confirm with the user → `issues ack ` or `resolve `. - **Investigate a regression:** `evals --aggregate` to see which score dropped → `evals --score helpfulness:..0.5` to list the bad runs → `events --session-id ` to see what happened inside one. @@ -194,4 +199,29 @@ agenteye --json events --full --session-id run-001 --all | jq '.events[].payload When you've pulled what you need, answer the user in prose or a small table — don't paste raw JSON back unless they asked for it. - +## 8. Audits — the async sweep, and how findings become issues + +An **audit** is a scheduled sweep that analyses recent agent behaviour (errors, +runaway tool loops, leaked secrets, low eval scores, …) and emits **findings**. +Two things about the flow matter when driving it from the CLI: + +- **`audits run ` is asynchronous — it only *queues*.** A `{"queued": true}` + does NOT mean the run finished (the analysis can take minutes). Poll + `audits runs ` until the newest row reads `succeeded` (or `failed`) before + reading findings — don't assume results are ready on the call that queued them. + A disabled audit, or one already mid-run, refuses to queue (exit 1). +- **Findings ARE issues — it's one bucket.** Every finding graduates to an issue + (`source = audit`) and carries its full content there, so the same problem shows + up under both `audits findings` and `issues list`. Triage is **globally + consistent in both directions**: `audits resolve ` closes the linked + issue, and `issues resolve ` on an audit issue resolves the finding — + either surface works, they never disagree. Triage a finding with + `audits ack|mute|dismiss|resolve|reopen ` (durable **mute/dismiss** suppress + the pattern org-wide by fingerprint; **resolve** leaves no suppression, so a true + recurrence reopens as new). Reads need `audits:read`, every mutation + `audits:write` (note: triaging a finding needs `audits:write`, not an `issues:*` + permission — the audit is the system of record and the issue follows it). + +Typical end-to-end: `audits list` → `audits run ` → poll `audits runs ` +→ `audits findings --audit ` (highest priority first) → `audits finding ` +for the full write-up → **confirm with the user** → `audits resolve `. diff --git a/skills/agenteye-cli/references/commands.md b/skills/agenteye-cli/references/commands.md index 7304cb4..0aebc59 100644 --- a/skills/agenteye-cli/references/commands.md +++ b/skills/agenteye-cli/references/commands.md @@ -13,7 +13,8 @@ command**, **`--json` to parse**, **branch on exit codes**. - [users](#users) - [settings](#settings) - [alerts](#alerts) -- [incidents](#incidents) +- [audits](#audits) +- [issues](#issues) - [query](#query) - [agent](#agent) @@ -22,7 +23,7 @@ Set on the CLI, **before** the subcommand. Precedence: flag > env var > config f | Flag | Env var | Meaning | |---|---|---| -| `--base-url ` | `AGENTEYE_DASHBOARD_URL` | Dashboard URL. **Required** (no default); must start with `http://`/`https://`. | +| `--base-url ` | `AGENTEYE_DASHBOARD_URL` | Dashboard URL. Defaults to `https://app.befailproof.ai` (the hosted product); override for self-hosted/dev. Must start with `http://`/`https://`. | | `--org ` | `AGENTEYE_ORG` | Active org for this command (multi-tenant override). | | `--token ` | `AGENTEYE_CLI_TOKEN` | Session token (normally from config after `login`). | | `--json` | `AGENTEYE_CLI_JSON` | Machine-readable JSON to stdout, nothing else. | @@ -118,15 +119,41 @@ A fixed registry — you read/inspect/change existing keys, you cannot create ne - `settings list` — `key · value · type · updated` (secrets masked). - `settings schema` — `key · type · accepts · description` (what each key accepts). - `settings set (--value V | --json-value JSON | --file f)` — exactly one value source. Unknown key → exit 6. No-op if unchanged. Server validation errors surface as `✗ ` (e.g. range bounds). Some keys are sensitive (signing secrets, sign-in allowlist) — confirm carefully. - -## incidents -Alert incidents; referenced by id (short ids accepted, `--show-id` shows them). -- `incidents list [--state firing|acknowledged|resolved] [--severity ...] [--alert-id ] [--limit N]` -- `incidents count` -- `incidents show ` — identity + comments + subscribers + **activity log** (read this before acting). -- `incidents ack ` · `incidents assign ` (member must be an operator) · `incidents resolve ` (calm confirm) · `incidents open --alert-id [--severity ...]` -- `incidents comment-add ` · `comment-list ` · `comment-delete ` -- `incidents subscribe ` · `unsubscribe ` · `subscribers ` + - `allowed_sign_ins` restricts which of the organization's members may sign in; it does not grant access to anyone else. An **empty list means no restriction** (every member can sign in), so clearing it widens access rather than removing it. A non-empty list admits only matching addresses and locks out every other member. Entries are exact addresses or `*@domain.tld`; a bare `*` is rejected — use an empty list. Saving a list that does not include your own address is refused, because you could not sign in again. + +## audits +Scheduled sweeps over agent activity, and the **findings** they produce. Audits are referenced by **name** (UUID id also accepted); findings by **id** (short ids shown in the table, `--show-id` for the full ones). +- `audits list [--enabled-only] [--show-id]` — `created · name · every · findings · status · last run`; disabled audits are dimmed and the footer carries the on/off split plus the open-finding total. +- `audits show ` — identity + `schedule` / `scope` / `analysis` / `channels` cards. The creator and the raw `scope`/`channels` blobs live here and in `--json`, not in the list. +- `audits create [--file f] [--description ...] [--enabled|--disabled] [--schedule-interval-secs N] [--schedule-anchor ISO8601] [--window-mode fixed|since_last] [--lookback-window-secs N] [--scope JSON] [--ignore-error-type ] [--llm|--no-llm] [--top-k N] [--sensitivity low|medium|high] [--channels JSON]` — everything but the name has a server default. Name collision is pre-checked (exit 2). No confirm (creating isn't destructive). +- `audits edit [--name ...] [same flags as create] [--yes]` — the server replaces the whole definition, so a flag-only edit re-sends the current audit with your change applied (needs read **and** write). Rename onto an existing name → exit 2. Confirms first. +- `audits delete [--yes]` — amber preview (naming the findings and run history that go with it) + confirm. +- `audits run ` — queue a run **now**, ahead of schedule. Success means queued, not finished; a disabled audit or one already mid-run is refused with the server's explanation (exit 1). JSON `{"queued": true}`. +- `audits runs [--limit N] [--show-id]` — run history, newest first: `started · status · trigger · findings · new · took`. A failed run's `error` and each run's `stats`/`report` are in `--json` only. +- `audits findings [--audit ] [--run-id ] [--status ] [--limit N] [--offset N] [--show-id]` — the triage queue, highest priority first: `id · title · severity · status · kind · seen · last`. With no `--status` you get the live set (open + recurring); valid statuses are `open recurring resolved dismissed muted`. `--audit` takes an audit **name**. +- `audits finding ` — one finding in full: identity + `analysis` (what + likely cause) + `recommendation` (do / impact / effort) + `scope` + `evidence`. Empty sections are omitted. +- `audits ack [--reason ...]` — seen, stays visible, ranked lower. No confirm. +- `audits mute [--reason ...] [--yes]` — stop surfacing this pattern in future runs (durable). Confirms first. +- `audits dismiss [--reason ...] [--yes]` — judged not worth acting on; suppressed like mute. Confirms first. +- `audits resolve [--yes]` — you fixed it. Leaves **no** suppression, so a genuine recurrence is raised as new. Confirms first. +- `audits reopen ` — back to `open` **and** clears any mute/dismiss suppression. The undo for the three above. +- `audits assign --to ` — set the owner; the status is untouched. `--to` is required (exit 2 without it). + +The title column truncates to whatever width is left so the fixed columns always survive — read the full text with `audits finding ` or `--json`. A bad `--status`, `--window-mode`, `--sensitivity`, a non-ISO-8601 `--schedule-anchor`, or an out-of-range `--schedule-interval-secs`/`--lookback-window-secs` is rejected before any request (exit 2). `--schedule-anchor` pins the fixed UTC slot runs land on (`anchor + N * interval`), so a slow run or `audits run` can't drift the cadence; omit it on create and the server uses the next 09:00 UTC. An unknown audit name → exit 6; an unknown or malformed finding id → calm `✗ no finding …`, exit 6. Reads need `audits:read`, every mutation `audits:write`. + +**`audits run` is async — it only queues.** Success is `{"queued": true}`, not a finished run; the analysis can take minutes. Poll `audits runs ` until the newest row is `succeeded`/`failed` before reading `audits findings`, rather than assuming results exist on the call that queued them. + +**Findings and issues are one bucket.** Every finding graduates to an issue (`source: audit`) that stores the finding's full content, so the same problem appears under both `audits findings` and `issues list`. Triage is consistent in **both** directions and needs `audits:write` either way: `audits resolve|mute|dismiss|ack|reopen ` mirrors onto the linked issue, and `issues resolve ` on an audit issue mirrors back onto the finding — the two never disagree. **resolve** leaves no suppression (a genuine recurrence reopens as new); **mute/dismiss** suppress the pattern org-wide by fingerprint. + +## issues +The single board for everything needing human attention — alert breaches (`source: alert`), hand-raised issues (`manual`), and audit findings (`audit`). Referenced by id (short ids accepted, `--show-id` shows them). This group was **renamed from `incidents`**; the old name no longer exists. Reads and ack/comment need `issues:read`; opening, assigning, and subscribing others need `issues:create`; resolving needs `issues:close`. +- `issues list [--state firing|acknowledged|resolved] [--alert-id ] [--limit N] [--show-id]` — there is **no** `--severity` filter on this command. +- `issues count` +- `issues show ` — identity + comments + subscribers + **activity log** (read this before acting). An audit-born issue (`source: audit`) carries the full finding it graduated from and back-links to the audit/run. +- `issues ack ` · `issues assign --assignee ` (repeatable; omit to clear all assignees; each must be an operator) · `issues resolve ` (calm confirm). **On an audit issue these stay in sync with the finding** — resolving the issue resolves the underlying audit finding, so it can't reappear on the next run (equivalently, triage it with `audits resolve `; both surfaces agree). +- `issues open --summary (--title | --alert-id ) [--title ...] [--severity ...]` — `--title` is **required** for a standalone issue (nothing to borrow a name from); with `--alert-id` it is optional and defaults to the alert's name. Missing `--title` on the standalone path → exit 2. +- `issues comment-add (--body | --file )` — `--file -` reads stdin; exactly one of the two · `comment-list ` · `comment-delete ` +- `issues subscribe [--email ]` · `unsubscribe [--email ]` · `subscribers ` — `--email` defaults to you; naming someone else needs `issues:create` Malformed (non-UUID) id → calm `✗ no incident …` exit 6. Assign to a non-operator → clean 422 message.