diff --git a/docs/ADRs/0058-agent-registration.md b/docs/ADRs/0058-agent-registration.md index 5a25379af9..f67fc62170 100644 --- a/docs/ADRs/0058-agent-registration.md +++ b/docs/ADRs/0058-agent-registration.md @@ -94,3 +94,11 @@ in the sections below. - [ADR 0045](0045-forge-portable-harness-schema.md) -- harness composition via `base:` URLs - [ADR 0057](0057-repos-management.md) -- repos management for per-repo installations - [Bring Your Own Agent](../guides/user/bring-your-own-agent.md) -- user-facing guide for agent registration + +## Notes + +- `fullsend agent new` (added for + [#6966](https://github.com/fullsend-ai/fullsend/issues/6966)) generates a + complete custom agent and then registers it through the `agent add` path + described here. It adds a generator in front of this decision; the + registration model itself is unchanged. diff --git a/docs/ADRs/0102-generate-custom-agents-from-the-cli.md b/docs/ADRs/0102-generate-custom-agents-from-the-cli.md new file mode 100644 index 0000000000..be4b6d13ba --- /dev/null +++ b/docs/ADRs/0102-generate-custom-agents-from-the-cli.md @@ -0,0 +1,127 @@ +--- +title: "102. Generate custom agents from the CLI" +status: Accepted +relates_to: + - agent-architecture +topics: + - cli + - harness + - onboarding +--- + +# 102. Generate custom agents from the CLI + +Date: 2026-09-03 + +## Status + +Accepted + +## Context + +Building a custom agent from scratch is the most common piece of negative user +feedback on fullsend. The +[Bring Your Own Agent guide](../guides/user/bring-your-own-agent.md) asks the +author to hand-write a harness with roughly thirty fields, an agent definition +whose `tools:` must stay consistent with its body, a result schema, a +post-script that safely turns model output into a forge mutation, a CEL +`trigger`, and a `policies/base.yaml` that a per-repo install does not vendor. + +Most of those mistakes are invisible until the first dispatch after merge +(#6830). Three of them are invisible even then: + +- A harness with no `trigger:` registers, validates, and appears in + `fullsend agent list`, and is then skipped by dispatch with a bare `continue` + and no annotation — while resolve and load failures both emit `::error::`. + None of the seven fleet harnesses has a `trigger:` to copy from, because the + fleet is dispatched by stage workflows instead. +- A `role:` the hosted mint does not serve is only regex-checked locally and + surfaces as an opaque `403` at run time (#6563). +- A `policy:` or provider path naming a file that is not there fails at run + time, or worse degrades to a warning and a sandbox that cannot reach Vertex + (#6834). + +[#4839](https://github.com/fullsend-ai/fullsend/issues/4839) records a July +decision to prefer guides over a CLI for this. That decision was taken in Slack +and was never written up as an ADR, so this is a new decision rather than a +superseding one. Two things have changed since: user feedback that the guides +are not enough, and the precedent of +[GitHub Agentic Workflows](https://github.com/github/gh-aw), which solved the +same onboarding problem with `gh aw new` — a generator that writes a minimal +valid source, leaving the user to edit prose rather than plumbing. + +## Options + +**Improve the guides further.** Cheapest, and the status quo. But the failure +modes above are not comprehension failures — a reader who understands the guide +perfectly still cannot tell that an omitted `trigger:` means the agent will +never run, because nothing reports it. + +**Ship the stub templates from `fullsend-ai/agents` and have the CLI fetch and +pin them like `agent add` does.** Keeps template changes on the agents repo's +release cadence. Rejected for the first cut: it makes the generator's happy +path depend on network and `allowed_remote_resources` state, which reintroduces +the "fails only at first dispatch" class this change exists to remove, and it +creates a merge-order dependency between the two repositories. Revisitable +later as an opt-in `--template-ref`. + +**Generate from templates embedded in the CLI.** Chosen. + +## Decision + +Add `fullsend agent new `, which writes a complete, valid, registered +agent from a minimum of parameters and validates the result before returning. + +Four properties make it worth having rather than being a scaffolding +convenience: + +1. **A trigger is mandatory.** The command refuses to write a trigger-less + harness, and `--on` presets emit expressions taken verbatim from the CEL + reference — a test asserts the generator and the documentation stay the same + text. The `command:` and `pr-opened` presets both refuse events from forks, + which matters because the default trigger is attached to every generated + agent, including agents with role `coder`. +2. **The role table is closed and checked.** `--role` accepts the five fleet + roles the hosted mint serves. It is hardcoded rather than derived from + `mintcore.BuiltInRoles()`, because derivation would re-admit `scribe` — + which `config.ValidRoles()` deliberately excludes as a mint-only dogfood + role — and would fail open for any future canonical role with no provider + pairing. A unit test asserts each row still matches + `mintcore.RolePermissionsFor`. +3. **The generator writes what a per-repo install does not vendor.** + `policies/base.yaml`, and the providers and profiles the chosen role needs, + are written when absent and never overwritten. Providers are referenced by + path rather than bare name, because the embedded provider fallback fills in + only the OpenAI provider. +4. **The result is validated in process.** Everything is rendered into a + scratch directory and loaded through the same loader dispatch uses, so a + harness that would fail validation never leaves a partially written + `.fullsend` behind. + +Templates are embedded with `go:embed`, matching how the repository already +ships scaffold content. + +## Consequences + +- Creating a working custom agent becomes one command, and the three + silently-fatal mistakes above become generation-time errors with actionable + text. The Bring Your Own Agent guide keeps its hand-written path as the + explanation of what was generated. +- Harness shape is now encoded in a second place. Mitigated but not + eliminated: the harness is built as a `harness.Harness` value and marshalled + rather than formatted as text, so the generator cannot emit a field the + validator does not know about; golden tests pin the generated bytes; and the + role table has a drift test against the mint. A new *required* harness field + would still need a matching change here, and the golden tests are what would + catch it. +- `ValidateRunnerEnvWith` is not run at generation time — it requires every + `${VAR}` in the harness `env` blocks to be set in the calling process, which + is true in CI and false on a developer's machine — so an unset variable + still surfaces at `fullsend run`. +- The default sandbox image digests are compiled in, so they are repinned by + hand on the same cadence as the fleet repin PRs. A golden test makes a repin + visible in review. +- `fullsend lock` and `fullsend run` are unchanged and do not share the + generator's check helper: they interleave minting, runner-env validation and + `${VAR}` expansion between the same steps in different orders, so sharing one + helper would change their behaviour. diff --git a/docs/architecture.md b/docs/architecture.md index c7128f52dc..68b08f7cac 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -396,6 +396,7 @@ the inheritance model: fullsend defaults, then repo baseline (`config.base.yaml` - Runtime resolution: `fullsend run ` resolves agents in two tiers: (1) config entries from `OrgConfig.Agents` (highest priority), (2) runtime fallback to the `fullsend-ai/agents` repository for known first-party agents not in config. The agents-repo fallback is a transitional mechanism for the agent extraction; it will be removed once all users have migrated to config-driven registration (ADR 0058 Phase 5). - Config lookup: config entries are looked up directly via `findConfigAgentEntry`; the agents-repo fallback operates independently when the agent is not found in config. Builds on [ADR 0045](ADRs/0045-forge-portable-harness-schema.md) harness identity model. - CLI management: `fullsend agent add|list|set|update|remove` manages config entries and auto-pins URLs to a commit SHA with an integrity hash. +- Agent generation: `fullsend agent new ` writes a complete custom agent — harness, agent definition, result schema, post-script, and the policy, providers and profiles a per-repo install does not vendor — validates it with the loader dispatch uses, and registers it through the `agent add` path above. A `trigger:` is mandatory, because a trigger-less harness registers and validates and is then silently never dispatched; `--role` is a closed table of the roles the hosted mint serves, so an unservable role fails locally rather than as a `403` at first dispatch ([ADR 0102](ADRs/0102-generate-custom-agents-from-the-cli.md)). **Open questions:** diff --git a/docs/cli/agent.md b/docs/cli/agent.md index 63a2b05822..47c6049d88 100644 --- a/docs/cli/agent.md +++ b/docs/cli/agent.md @@ -4,7 +4,7 @@ sidebar_label: fullsend agent # fullsend agent -Manage agent registrations in fullsend config. Add, list, set (runtime, model, effort), update, and remove agents. +Manage agents in fullsend config. Generate a new agent, add, list, set (runtime, model, effort), update, and remove agents. `agent add` and `agent update` fetch remote content and resolve GitHub URLs. Authentication is via `gh` CLI or `GH_TOKEN` environment variable. @@ -12,12 +12,365 @@ Manage agent registrations in fullsend config. Add, list, set (runtime, model, e | Command | Description | |---------|-------------| +| `fullsend agent new ` | Generate a complete custom agent and register it | | `fullsend agent add ` | Register an agent in config | | `fullsend agent list` | List registered agents | | `fullsend agent update [sha]` | Update a URL agent to a new commit SHA | | `fullsend agent set ` | Set an agent's runtime, model or effort | | `fullsend agent remove ` | Remove an agent from config | +## `agent new` + +Generate a complete, valid, runnable custom agent and register it. Every file +an agent needs is written for you; the only one you have to edit is the +instructions the agent follows. + +```bash +fullsend agent new lint-docs --fullsend-dir .fullsend \ + --role triage --description "Check docs changes for broken links" +``` + +``` + ✓ Created agent "lint-docs" in .fullsend + harness/lint-docs.yaml + agents/lint-docs.md + schemas/lint-docs-result.schema.json + scripts/post-lint-docs.sh + policies/base.yaml + providers/vertex-ai.yaml + providers/github-ro.yaml + profiles/fullsend-vertex-ai.yaml + profiles/fullsend-github-ro.yaml + ✓ Added agent "lint-docs" + +Next: + 1. Fill in the marked sections of agents/lint-docs.md — that file is the agent's prompt. + 2. Test locally: + fullsend run lint-docs --fullsend-dir .fullsend \ + --target-repo . --env-file .env.local + .env.local needs GITHUB_ISSUE_URL, ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION + and GH_TOKEN. See docs/guides/user/running-agents-locally.md. + 3. Commit .fullsend, then comment `/fs-lint-docs` on an issue or pull request to run it in CI. +``` + +The generated tree: + +```bash +find .fullsend -type f | sort +``` + +``` +.fullsend/agents/lint-docs.md +.fullsend/config.yaml +.fullsend/harness/lint-docs.yaml +.fullsend/policies/base.yaml +.fullsend/profiles/fullsend-github-ro.yaml +.fullsend/profiles/fullsend-vertex-ai.yaml +.fullsend/providers/github-ro.yaml +.fullsend/providers/vertex-ai.yaml +.fullsend/schemas/lint-docs-result.schema.json +.fullsend/scripts/post-lint-docs.sh +``` + +Only `agents/lint-docs.md` needs your attention — it is the agent's prompt and +it ships with marked sections to fill in. Everything else is complete. + +### What gets written + +| File | Written | Overwritten by `--force` | +|------|---------|--------------------------| +| `harness/.yaml` | always | yes | +| `agents/.md` | always | yes | +| `schemas/-result.schema.json` | always | yes | +| `scripts/post-.sh` (mode 0755) | always | yes | +| `policies/base.yaml` | when absent | **no** | +| `providers/*.yaml` (per role) | when absent | **no** | +| `profiles/*.yaml` (per role) | when absent | **no** | +| `scripts/validate-output-schema.sh` | with `--validation-loop`, when absent | **no** | +| `config.yaml` `agents:` entry | unless `--no-register` | n/a | + +The policy, provider and profile files are shared by every agent in the +directory, so they are never overwritten — including with `--force`. +`fullsend github setup` does not copy these into your repository, which is why +`agent new` writes them when they are absent. + +### Flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--fullsend-dir` | | Path to the `.fullsend` configuration directory (required) | +| `-f`, `--file` | | Read the agent definition from a spec YAML file | +| `--role` | `triage` | Mint role the agent runs as (see the table below) | +| `--description` | `Custom agent.` | One-line description; written to both the harness and the agent definition | +| `--on` | `command:/fs-` | Trigger preset; mutually exclusive with `--trigger` | +| `--trigger` | | A trigger written by hand, in CEL (the expression language dispatch evaluates); mutually exclusive with `--on` | +| `--model` | `opus` | Model for the agent | +| `--effort` | `high` | Effort level (`low`, `medium`, `high`, `xhigh`, `max`) | +| `--runtime` | | Agent runtime recorded in `config.yaml` (`claude`, `pi` or `codex`) | +| `--slug` | `-` | Names the GitHub App to look for when the agent is installed; `` comes from the `origin` remote | +| `--image` | per-role pin | Container image the agent runs inside | +| `--timeout-minutes` | `15` | Agent timeout in minutes | +| `--validation-loop` | `false` | Add a `validation_loop` checking output against the schema | +| `--no-register` | `false` | Write the files but do not touch `config.yaml` | +| `--force` | `false` | Overwrite generated files (never shared assets) | +| `--dry-run` | `false` | Validate and print what would be written, writing nothing | + +### Roles + +`--role` is not the agent's name. It decides which GitHub identity the agent +acts as and what that identity may do. + +Agents do not carry long-lived credentials. At run time they ask a service +called the **mint** for a short-lived GitHub token, and `role:` is what they +ask for. The mint only issues tokens for roles it knows, so a role it does not +serve fails at the first run rather than at generation time — which is why this +command refuses an unknown one up front. The hosted mint serves these: + +| `--role` | Permissions | Providers | +|----------|-------------|-----------| +| `triage` (default) | `contents:read`, `issues:write`, `metadata:read` | vertex-ai, github-ro | +| `review` | `contents:read`, `pull_requests:write`, `issues:write`, `checks:read`, `metadata:read` | vertex-ai, github-ro | +| `coder` | `contents:write`, `packages:read`, `pull_requests:write`, `issues:write`, `checks:read`, `metadata:read` | vertex-ai, github | +| `retro` | `actions:read`, `contents:read`, `pull_requests:write`, `issues:write`, `metadata:read` | vertex-ai, github-ro, github-artifacts | +| `prioritize` | `contents:read`, `issues:write`, `organization_projects:write`, `metadata:read` | vertex-ai, github-ro | + +Pick the role whose permissions fit what the agent does. An unknown role fails +immediately with this table, rather than returning `403` from the mint the +first time the agent runs. To use a role the hosted mint does not serve, you +need to run your own — see +[Custom Agent Identity](../guides/user/custom-agent-identity.md). + +### Triggers + +A trigger is the rule that decides which GitHub events start the agent — +a comment, a label, a new issue, a pull request. Every generated agent gets +one, because an agent without a trigger is accepted everywhere and then simply +never runs, with nothing reported anywhere to tell you why. `agent new` +therefore refuses to write one without a trigger. `--on` takes a preset: + +| `--on` | Fires when | +|--------|-----------| +| `command:/` (default `/fs-`) | Someone comments the slash command on an issue, or on a pull request that is not from a fork | +| `label: