diff --git a/docs/plans/2026-02-27-node-resource-api-restructuring.md b/docs/plans/2026-02-27-node-resource-api-restructuring.md deleted file mode 100644 index 7ff596ec3..000000000 --- a/docs/plans/2026-02-27-node-resource-api-restructuring.md +++ /dev/null @@ -1,719 +0,0 @@ -# Node-Centric API Restructuring Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Restructure the REST API so that `/node/{hostname}` is the top-level -resource for all machine operations, moving target from query params to path -params and nesting network/command under node. - -**Architecture:** Replace flat domain paths (`/node/status`, `/network/ping`, -`/command/exec`) with resource-oriented paths (`/node/{hostname}`, -`/node/{hostname}/network/ping`, `/node/{hostname}/command/exec`). The -`{hostname}` path segment replaces the `target_hostname` query parameter, -accepting the same values: `_any`, `_all`, literal hostnames, or `key:value` -label selectors. Individual sub-resource endpoints (disk, memory, load, os, -uptime) are added alongside the composite status. - -**Tech Stack:** Go 1.25, oapi-codegen (strict-server), Echo v4, NATS JetStream, -testify/suite - ---- - -## New API Layout - -``` -# Node (resource-oriented, target in path) -GET /node/{hostname} composite status -GET /node/{hostname}/disk disk usage -GET /node/{hostname}/memory memory stats -GET /node/{hostname}/load load averages -GET /node/{hostname}/os OS info -GET /node/{hostname}/uptime uptime -GET /node/{hostname}/hostname hostname + labels - -# Network (nested under node) -GET /node/{hostname}/network/dns/{interfaceName} DNS config -PUT /node/{hostname}/network/dns update DNS -POST /node/{hostname}/network/ping ping - -# Command (nested under node) -POST /node/{hostname}/command/exec execute command -POST /node/{hostname}/command/shell shell command - -# Unchanged domains -GET /agent list agents -GET /agent/{hostname} agent details -POST /job create job -GET /job list jobs -GET /job/{id} get job -DELETE /job/{id} delete job -POST /job/{id}/retry retry job -GET /job/status queue stats -GET /health liveness -GET /health/ready readiness -GET /health/status detailed status -GET /audit list entries -GET /audit/{id} get entry -GET /audit/export export all -``` - -### Query Param Policy - -After this change, query params are ONLY used for **filtering and pagination** -on collection endpoints (`/job`, `/audit`). Resource identification always uses -path parameters. Complex input data uses request bodies. - -### Reserved Hostname Values - -The `{hostname}` path segment accepts: - -| Value | Meaning | -| --------------- | ------------------------------- | -| `_any` | Load-balanced to any agent | -| `_all` | Broadcast to all agents | -| `web-01` | Direct routing to specific host | -| `group:web.dev` | Label-based routing | - ---- - -## Phasing - -### Phase 1 — API Guidelines & Task File - -Update `api-guidelines.md` to document the node-centric resource model, path -parameter conventions, and query param policy. Create in-progress task file. - -### Phase 2 — OpenAPI Specs & Code Generation - -Rewrite the three OpenAPI specs (node, network, command) with the new path -structure. Merge network and command specs into the node spec since they now -share the `/node/{hostname}` prefix. Regenerate `*.gen.go`. - -### Phase 3 — Handlers - -Update handlers to extract `{hostname}` from path instead of query params. Add -new handlers for the individual sub-resources (disk, memory, load, os, uptime). -Update handler wiring. - -### Phase 4 — JobClient Methods - -Add new `QueryNodeDisk`, `QueryNodeMemory`, `QueryNodeLoad`, `QueryNodeOS`, -`QueryNodeUptime` methods (plus broadcast variants) to the job client. The agent -already handles these operations. - -### Phase 5 — CLI & SDK - -Update CLI commands to build paths with target instead of query params. -Restructure CLI command tree (`client node disk --target web-01` → builds -`GET /node/web-01/disk`). Update SDK. - -### Phase 6 — Tests & Documentation - -Update all integration tests, feature docs, CLI docs, architecture docs. Remove -stale `target_hostname` references. - ---- - -## Task 1: Update API Guidelines - -**Files:** - -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` - -**Step 1: Update the API guidelines document** - -Add two new sections after the existing four guidelines: - -```markdown -5. **Node as Top-Level Resource** - -All operations that target a managed machine are nested under -`/node/{hostname}`. The `{hostname}` path segment identifies the target and -accepts literal hostnames, reserved routing values (`_any`, `_all`), or label -selectors (`key:value`). - -Sub-resources represent distinct capabilities of the node: - -| Path Pattern | Domain | -| ---------------------------------------------- | ------- | -| `/node/{hostname}` | Status | -| `/node/{hostname}/disk` | Node | -| `/node/{hostname}/memory` | Node | -| `/node/{hostname}/network/dns/{interfaceName}` | Network | -| `/node/{hostname}/command/exec` | Command | - -6. **Path Parameters Over Query Parameters** - -Use path parameters for **resource identification and targeting**. Use query -parameters only for **filtering and pagination** on collection endpoints (e.g., -`/job?status=completed&limit=20`). - -Never use query parameters to identify which resource to act on. Complex input -data belongs in request bodies. -``` - -**Step 2: Commit** - -``` -docs: update API guidelines with node-centric resource model -``` - ---- - -## Task 2: Create In-Progress Task File - -**Files:** - -- Create: - `docs/docs/sidebar/development/tasks/in-progress/2026-02-27-node-resource-api-restructuring.md` - -**Step 1: Create the task file** - -```markdown ---- -title: Node-centric API restructuring -status: in-progress -created: 2026-02-27 -updated: 2026-02-27 ---- - -## Objective - -Restructure the REST API so `/node/{hostname}` is the top-level resource for all -machine operations. Move target from query params to path params. Nest network -and command under node. Add individual sub-resource endpoints (disk, memory, -load, os, uptime). - -## Notes - -- Plan: `docs/plans/2026-02-27-node-resource-api-restructuring.md` -- Agent already handles individual operations (disk, memory, load, etc.) -- Job routing logic unchanged — only HTTP layer changes -- Reserved hostnames: `_any`, `_all`, label selectors (`key:value`) - -## Outcome - -_To be filled in when complete._ -``` - -**Step 2: Commit** - -``` -docs: add node-centric API restructuring task -``` - ---- - -## Task 3: Rewrite Node OpenAPI Spec - -This is the core spec change. The node spec absorbs network and command paths -since they all share the `/node/{hostname}` prefix. - -**Files:** - -- Modify: `internal/api/node/gen/api.yaml` -- Modify: `internal/api/node/gen/cfg.yaml` (add import mappings for - network/command shared schemas if needed) -- Delete: `internal/api/network/gen/api.yaml` (paths move to node spec) -- Delete: `internal/api/command/gen/api.yaml` (paths move to node spec) - -**Step 1: Rewrite `internal/api/node/gen/api.yaml`** - -The new spec must include: - -**Paths:** - -- `GET /node/{hostname}` — composite status (was `/node/status`) -- `GET /node/{hostname}/hostname` — hostname + labels -- `GET /node/{hostname}/disk` — disk usage (NEW) -- `GET /node/{hostname}/memory` — memory stats (NEW) -- `GET /node/{hostname}/load` — load averages (NEW) -- `GET /node/{hostname}/os` — OS info (NEW) -- `GET /node/{hostname}/uptime` — uptime (NEW) -- `GET /node/{hostname}/network/dns/{interfaceName}` — DNS config -- `PUT /node/{hostname}/network/dns` — update DNS -- `POST /node/{hostname}/network/ping` — ping -- `POST /node/{hostname}/command/exec` — exec command -- `POST /node/{hostname}/command/shell` — shell command - -**Path parameter `hostname`:** - -```yaml -parameters: - - name: hostname - in: path - required: true - description: >- - Target agent hostname, reserved routing value (_any, _all), or label - selector (key:value). - schema: - type: string - minLength: 1 - x-oapi-codegen-extra-tags: - validate: required,min=1,valid_target -``` - -**Key design decisions:** - -- `target_hostname` query param removed from ALL endpoints -- `hostname` path param added to ALL endpoints -- Network and command schemas (request/response types) copied into node spec or - imported via `$ref` from `common/gen/api.yaml` -- Existing response schemas preserved (no API response breaking changes) -- Security scopes remain: `node:read`, `network:read`, `network:write`, - `command:execute` - -**New response schemas for individual sub-resources:** - -```yaml -DiskUsageCollectionResponse: - type: object - properties: - job_id: - type: string - format: uuid - results: - type: array - items: - $ref: '#/components/schemas/DiskUsageResponse' - -DiskUsageResponse: - type: object - properties: - hostname: - type: string - disks: - type: array - items: - $ref: './common/DiskUsageItem' - error: - type: string -``` - -Follow the same pattern for Memory, Load, OS, Uptime responses. - -**Step 2: Regenerate code** - -```bash -go generate ./internal/api/node/gen/... -``` - -**Step 3: Verify generated code compiles** - -```bash -go build ./internal/api/node/... -``` - -**Step 4: Commit** - -``` -feat: rewrite node OpenAPI spec with resource-oriented paths -``` - ---- - -## Task 4: Consolidate Handler Packages - -Since network and command paths are now under `/node/{hostname}/...`, the -handlers can either stay in their packages (with the node spec importing their -logic) or be consolidated. The cleanest approach: keep separate handler structs -but register them all through the node spec's generated interface. - -**Files:** - -- Modify: `internal/api/node/types.go` — add network/command dependencies -- Modify: `internal/api/node/node.go` — expand factory to accept all - dependencies -- Create: `internal/api/node/node_disk_get.go` — new handler -- Create: `internal/api/node/node_memory_get.go` — new handler -- Create: `internal/api/node/node_load_get.go` — new handler -- Create: `internal/api/node/node_os_get.go` — new handler -- Create: `internal/api/node/node_uptime_get.go` — new handler -- Move: network handler logic into `internal/api/node/network_*.go` -- Move: command handler logic into `internal/api/node/command_*.go` -- Modify: `internal/api/handler_node.go` — register all handlers -- Remove: `internal/api/handler_network.go` (absorbed by node) -- Remove: `internal/api/handler_command.go` (absorbed by node) -- Modify: `internal/api/types.go` — remove network/command handler fields -- Modify: `internal/api/handler.go` — remove network/command from - `CreateHandlers` -- Modify: `cmd/api_helpers.go` — remove `GetNetworkHandler`, `GetCommandHandler` - from `ServerManager` interface - -**Step 1: Update `internal/api/node/types.go`** - -The Node struct needs the JobClient (already has it). The JobClient interface -already has all Query/Modify methods for network and command. No new -dependencies needed — just new handler methods. - -**Step 2: Write failing tests for new sub-resource handlers** - -For each new handler (disk, memory, load, os, uptime), write a public test -following the existing `node_status_get_public_test.go` pattern: - -```go -// node_disk_get_public_test.go -func (suite *NodeDiskGetPublicTestSuite) TestGetNodeDisk() { - tests := []struct { - name string - hostname string - // ... - }{ - // success, validation error, job client error cases - } -} -``` - -**Step 3: Implement handlers** - -Each new handler follows the same pattern as `GetNodeStatus`: - -1. Extract `hostname` from `request.Hostname` (path param) -2. Validate with `validation.Struct` -3. Check `job.IsBroadcastTarget(hostname)` for broadcast fork -4. Call appropriate `JobClient.QueryNode*` method -5. Build and return response - -**Step 4: Update existing handlers** - -Modify `node_status_get.go` and `node_hostname_get.go` to extract hostname from -`request.Hostname` (path param) instead of `request.Params.TargetHostname` -(query param). - -Move network handlers (`network_ping_post.go`, `network_dns_*.go`) and command -handlers (`command_exec_post.go`, `command_shell_post.go`) into the node -package. Update them to extract hostname from path param. - -**Step 5: Update handler wiring** - -In `internal/api/handler_node.go`, the `GetNodeHandler` method now registers ALL -handlers (node + network + command) since they all come from the same generated -spec. - -Remove `GetNetworkHandler` and `GetCommandHandler` from the `ServerManager` -interface and handler.go. - -**Step 6: Run tests** - -```bash -go test ./internal/api/node/... -``` - -**Step 7: Commit** - -``` -feat: consolidate node/network/command handlers under node package -``` - ---- - -## Task 5: Add JobClient Methods for New Sub-Resources - -The agent already handles `node.disk.get`, `node.memory.get`, etc. We need -JobClient methods to create and wait for these jobs. - -**Files:** - -- Modify: `internal/job/client/query.go` — add new Query methods -- Modify: `internal/job/client/types.go` (or interface file) — add to interface -- Create: `internal/job/client/query_node_test.go` — tests for new methods - -**Step 1: Write failing tests** - -Follow the existing `TestQueryNodeStatus` pattern in `query_public_test.go`. - -**Step 2: Add methods** - -New methods needed: - -- `QueryNodeDisk(ctx, hostname)` + `QueryNodeDiskBroadcast` -- `QueryNodeMemory(ctx, hostname)` + `QueryNodeMemoryBroadcast` -- `QueryNodeLoad(ctx, hostname)` + `QueryNodeLoadBroadcast` -- `QueryNodeOS(ctx, hostname)` + `QueryNodeOSBroadcast` -- `QueryNodeUptime(ctx, hostname)` + `QueryNodeUptimeBroadcast` - -Each follows the exact same pattern as `QueryNodeStatus` but uses different -operation constants (`OperationNodeDiskGet`, etc.). - -**Step 3: Run tests** - -```bash -go test ./internal/job/client/... -``` - -**Step 4: Commit** - -``` -feat: add job client methods for individual node sub-resources -``` - ---- - -## Task 6: Update CLI Commands - -**Files:** - -- Modify: `cmd/client_node.go` — add subcommands for disk, memory, load, os, - uptime -- Create: `cmd/client_node_disk_get.go` -- Create: `cmd/client_node_memory_get.go` -- Create: `cmd/client_node_load_get.go` -- Create: `cmd/client_node_os_get.go` -- Create: `cmd/client_node_uptime_get.go` -- Modify: `cmd/client_node_status_get.go` — update for path-based target -- Modify: `cmd/client_node_hostname_get.go` — same -- Move: `cmd/client_network_*.go` → `cmd/client_node_network_*.go` -- Move: `cmd/client_command_*.go` → `cmd/client_node_command_*.go` -- Modify: `cmd/client.go` — update command tree - -**Step 1: Update CLI command tree** - -New structure: - -``` -osapi client node status --target web-01 -osapi client node hostname --target web-01 -osapi client node disk --target web-01 -osapi client node memory --target web-01 -osapi client node load --target web-01 -osapi client node os --target web-01 -osapi client node uptime --target web-01 -osapi client node network dns get --interface eth0 --target web-01 -osapi client node network dns update ... --target web-01 -osapi client node network ping --address 1.1.1.1 --target web-01 -osapi client node command exec --command ls --target web-01 -osapi client node command shell --command "ls -la" --target web-01 -``` - -The `--target` flag is still a CLI flag — it gets placed into the URL path by -the SDK, not as a query param. - -**Step 2: Commit** - -``` -feat: restructure CLI commands under node with sub-resources -``` - ---- - -## Task 7: Update SDK - -**Files (in osapi-sdk repo):** - -- Sync new `api.yaml` specs -- Regenerate client code -- Update service wrappers - -**Step 1: Copy updated api.yaml to SDK** - -The SDK pulls specs via gilt. During development, manually copy -`internal/api/node/gen/api.yaml` to the SDK. - -**Step 2: Regenerate SDK** - -```bash -cd ../osapi-sdk && just generate -``` - -**Step 3: Update SDK service wrappers** - -The `Node` service now includes network and command methods. Path construction -changes from query params to path segments. - -**Step 4: Commit (in SDK repo)** - -``` -feat: update SDK for node-centric API paths -``` - ---- - -## Task 8: Integration Tests - -**Files:** - -- Modify: all `*_integration_test.go` in `internal/api/node/` -- Create: new integration tests for disk, memory, load, os, uptime -- Remove: `internal/api/network/*_integration_test.go` (moved to node) -- Remove: `internal/api/command/*_integration_test.go` (moved to node) - -Every integration test must verify: - -- Valid input returns correct response -- Invalid `{hostname}` returns 400 -- Missing token returns 401 -- Wrong permissions return 403 -- Valid token with correct scope returns 200/202 - -**Step 1: Write integration tests for new endpoints** - -**Step 2: Update existing integration tests for path change** - -**Step 3: Run full test suite** - -```bash -just test -``` - -**Step 4: Commit** - -``` -test: update integration tests for node-centric API paths -``` - ---- - -## Task 9: Update Documentation - -**Files:** - -- Modify: `docs/docs/sidebar/features/node-management.md` -- Modify: `docs/docs/sidebar/features/network-management.md` -- Modify: `docs/docs/sidebar/features/command-execution.md` -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` (permissions table) -- Modify: `docs/docs/sidebar/usage/cli/client/node/` docs -- Create: CLI docs for new subcommands (disk, memory, load, os, uptime) -- Move: network/command CLI docs under node -- Modify: `CLAUDE.md` — update architecture quick reference - -**Step 1: Update feature docs** - -**Step 2: Update CLI docs** - -**Step 3: Update architecture docs** - -**Step 4: Run docs build** - -```bash -just docs::build -``` - -**Step 5: Commit** - -``` -docs: update documentation for node-centric API restructuring -``` - ---- - -## Task 10: Cleanup - -**Files:** - -- Remove: `internal/api/network/` package (if fully absorbed) -- Remove: `internal/api/command/` package (if fully absorbed) -- Remove: stale CLI command files -- Verify: no remaining `target_hostname` query param references - -**Step 1: Search for stale references** - -```bash -grep -r "target_hostname" --include="*.go" --include="*.yaml" -grep -r "TargetHostname" --include="*.go" -``` - -**Step 2: Remove dead code** - -**Step 3: Final verification** - -```bash -just generate -go build ./... -just test -just go::vet -``` - -**Step 4: Commit** - -``` -refactor: remove stale network/command packages and target_hostname refs -``` - ---- - -## Resolved Decisions - -- **Colon in path segments** — keep `:` as the label selector delimiter. It is a - valid URL path character and does not conflict with Echo routing (oapi-codegen - uses `{hostname}` style, not `:hostname`). -- **Path param validation** — confirmed: oapi-codegen strict-server mode does - NOT generate validate tags on path param request object structs. Each handler - must manually validate `request.Hostname` using a shared helper: - `validateHostname(request.Hostname)` → calls `validation.Var()` with - `required,min=1,valid_target`. Add YAML comments in OpenAPI specs wherever - path param `x-oapi-codegen-extra-tags` appears, noting the tags are not - generated in strict-server mode and validation is handled manually in - handlers. -- **RBAC permissions** — keep existing granular permissions unchanged - (`node:read`, `network:read`, `network:write`, `command:execute`). Permissions - map to capabilities, not URL structure. Existing JWT tokens continue to work. -- **Spec merging** — merge network and command specs into the node spec since - all paths share `/node/{hostname}` prefix. - -## Task 11: File oapi-codegen Feature Request - -**Step 1: Open an issue on oapi-codegen** - -File a feature request on `github.com/oapi-codegen/oapi-codegen` asking for -`x-oapi-codegen-extra-tags` on path parameters to be propagated to -`RequestObject` struct fields in strict-server mode. Currently the tags only -work for request body properties and query parameter `*Params` structs. Path -params in strict mode become plain struct fields with no extra tags. - -Include a minimal reproducer (OpenAPI spec + cfg.yaml with -`strict-server: true`) showing the expected vs actual generated code. - ---- - -## Outcome - -All tasks complete. The restructuring was implemented across three Claude Code -sessions (context exhausted twice). - -### What was done - -- **OpenAPI spec**: Merged network and command specs into a single node spec - with 12 endpoints under `/node/{hostname}/...`. Regenerated `*.gen.go`. -- **Handlers**: Consolidated `internal/api/network/` and `internal/api/command/` - into `internal/api/node/`. Added 5 new sub-resource handlers (disk, memory, - load, os, uptime). Removed stale handler wiring (`GetNetworkHandler`, - `GetCommandHandler`). -- **JobClient**: Added `QueryNodeDisk`, `QueryNodeMemory`, `QueryNodeLoad`, - `QueryNodeOS`, `QueryNodeUptime` methods (plus broadcast variants). -- **CLI**: Restructured command tree — `client network *` → - `client node network *`, `client command *` → `client node command *`. All - commands use `sdkClient.Node.*`. -- **SDK**: Consolidated `NetworkService` and `CommandService` into - `NodeService`. Merged and pushed as - [osapi-sdk#9](https://github.com/osapi-io/osapi-sdk/pull/9). Fixed README - example link in [osapi-sdk#10](https://github.com/osapi-io/osapi-sdk/pull/10). -- **Tests**: All unit tests (testify/suite, table-driven), integration tests - (RBAC + validation), and bats CLI tests updated. 26 packages pass. -- **Documentation**: CLI docs moved under `node/` directory. Feature docs, - architecture docs, API guidelines, CLAUDE.md, and audit example paths updated. - No stale references remain. -- **Cleanup**: Deleted `internal/api/network/`, `internal/api/command/`, old CLI - files, old SDK services. Removed `target_hostname` query param references. - -### oapi-codegen feature request - -Filed -[oapi-codegen#2261](https://github.com/oapi-codegen/oapi-codegen/issues/2261): -`x-oapi-codegen-extra-tags` on path parameters are not propagated to -`RequestObject` struct fields in strict-server mode. Workaround: manual -`validation.Var()` calls in each handler. - -### Verification - -``` -go build ./... # passes -just go::unit # 26 packages pass -just go::vet # lint clean -``` - -## Risk Notes - -- **Breaking API change** — all clients must update. Document migration. -- **SDK sync** — SDK must be updated before CLI works with new paths. diff --git a/docs/plans/2026-02-28-orchestrator-design.md b/docs/plans/2026-02-28-orchestrator-design.md deleted file mode 100644 index c226bc52b..000000000 --- a/docs/plans/2026-02-28-orchestrator-design.md +++ /dev/null @@ -1,217 +0,0 @@ -# Orchestrator Design - -## Goal - -A Go library that gives operators orchestration primitives on top of the -osapi-sdk. Operators write Go programs that define tasks with dependencies, and -the library handles DAG resolution, parallel execution, idempotency reporting, -and error handling. - -## Motivation - -OSAPI provides primitives: submit a job, target a host, get a result. But -there's no way to express "run A then B" or "only run C if A changed something" -or "run these three things in parallel, then converge." Today the only -sequencing option is polling `job get` in a loop (what `job run` does for a -single job). - -Operators managing fleets need orchestration: install a package, then configure -DNS, then start a service — across multiple hosts, with dependencies between -steps, and accurate reporting of what changed. - -## Approach - -**Client-side orchestration library.** The orchestrator runs on the operator's -machine (or a control node), calls the SDK, and tracks execution. No new server -components. The API server stays stateless. Agents stay dumb. - -This follows the Ansible model (push from client) rather than the Chef/Puppet -model (pull from agent) or the Kubernetes model (server-side controllers). The -escape hatch is clean: if server-side durability is needed later, a -`POST /orchestrate` endpoint can accept the same task definitions and run them -internally. - -**Operation-level idempotency.** The orchestration library doesn't implement -idempotency — it trusts the platform. Each write operation checks current state -before mutating and returns an accurate `changed` field. The DNS operation -already does this. All future write operations follow the same pattern. The -orchestrator consumes `changed` for conditionals and reporting. - -## Core Concepts - -Four primitives: - -| Concept | What it is | -| ---------- | ------------------------------------------------------------- | -| **Task** | A unit of work — wraps an SDK call or custom function | -| **Plan** | A DAG of tasks with dependency edges | -| **Runner** | Resolves the DAG, executes in topological order, parallelizes | -| **Report** | Per-task status and aggregate convergence summary | - -## Task Definition - -Two styles, both returning the same `*Result`: - -### Declarative — standard SDK operations - -```go -installPkg := plan.Task("install-nginx", &orchestrator.Op{ - Operation: "command.exec", - Target: "_all", - Params: map[string]any{ - "command": "apt", - "args": []string{"install", "-y", "nginx"}, - }, -}) -``` - -### Functional — custom logic - -```go -verify := plan.TaskFunc("verify-nginx", func( - ctx context.Context, - client *osapi.Client, -) (*orchestrator.Result, error) { - resp, err := client.Command.Exec(ctx, "nginx", []string{"-t"}, "_all") - if err != nil { - return nil, err - } - return &orchestrator.Result{Changed: false}, nil -}) -``` - -## Dependencies and Execution Order - -```go -createUser := plan.Task("create-user", &orchestrator.Op{...}) -installNginx := plan.Task("install-nginx", &orchestrator.Op{...}) -configureDNS := plan.Task("configure-dns", &orchestrator.Op{...}) -startNginx := plan.Task("start-nginx", &orchestrator.Op{...}) - -installNginx.DependsOn(createUser) -startNginx.DependsOn(installNginx, configureDNS) -``` - -DAG: - -``` -createUser ──→ installNginx ──→ startNginx -configureDNS ─────────────────↗ -``` - -The runner executes `createUser` and `configureDNS` in parallel, then -`installNginx` after `createUser` completes, then `startNginx` after both -`installNginx` and `configureDNS` complete. - -## Conditional Execution - -```go -// Only run if dependency actually changed something -startNginx.DependsOn(installNginx).OnlyIfChanged() - -// Custom guard -startNginx.When(func(results orchestrator.Results) bool { - return results.Get("install-nginx").Changed -}) -``` - -## Error Handling - -```go -plan := orchestrator.NewPlan( - client, - orchestrator.OnError(orchestrator.StopAll), // default -) -``` - -Three strategies: - -- `StopAll` — fail fast, cancel everything (default) -- `Continue` — skip dependents, keep running independent tasks -- `Retry(n)` — retry n times before failing - -## Running and Reporting - -```go -report, err := plan.Run(ctx) - -fmt.Println(report.Summary()) -// 4 tasks: 2 changed, 1 unchanged, 1 skipped -// Total duration: 12.3s - -for _, r := range report.Tasks { - fmt.Printf("%s: %s (changed=%v, duration=%s)\n", - r.Name, r.Status, r.Changed, r.Duration) -} -``` - -## How Idempotency Fits - -The orchestration library delegates to the SDK, which delegates to the API, -which delegates to agents, which run operations. Operations own idempotency: - -``` -Plan.Run() - → Task calls SDK - → SDK calls API - → API creates job → Agent runs operation - → Operation checks state, mutates only if needed - → Returns changed: true/false - → Result flows back - → Task gets Result{Changed: bool} - → Runner uses Changed for conditionals and reporting -``` - -The DNS operation is the reference implementation: read current state, compare -to desired, skip if equal, mutate if different, report accurately. All future -write operations follow this pattern. - -For command exec/shell: always `changed: true`. These are inherently -non-idempotent. The orchestration layer handles command idempotency via guards -(`When`, `OnlyIfChanged`) — not the operation. - -## Operation Idempotency Standard - -Every write operation MUST: - -1. Read current state before mutating -2. Compare desired vs current — return `{Changed: false}` if equal -3. Mutate only if different — return `{Changed: true}` -4. Preserve unspecified fields (partial updates keep existing values) - -Integration tests for write operations MUST verify: - -- First call returns `changed: true` -- Second identical call returns `changed: false` - -## Project Structure - -The orchestrator lives inside `osapi-sdk` as a sibling package to the core SDK -client: - -``` -osapi-sdk/ -├── pkg/ -│ ├── osapi/ # Core SDK client -│ └── orchestrator/ # DAG-based task orchestration -│ ├── options.go # ErrorStrategy, PlanOption, OnError -│ ├── plan.go # Plan, NewPlan, Validate, Run, Explain -│ ├── task.go # Task, Op, TaskFn, DependsOn, When -│ ├── runner.go # DAG resolution, parallel execution, job polling -│ └── result.go # Result, TaskResult, Report, Summary -└── examples/ - ├── basic/ # Simple SDK client usage - ├── discovery/ # Runnable DAG against live OSAPI - └── orchestrator/ # Declarative deployment DAG -``` - -## What Comes Later (Not This Design) - -- **Hooks** — callback system for consumer-controlled logging and progress -- **Levels()** — structured access to the levelized DAG -- **Per-task error strategy** — `task.OnError(orchestrator.Continue)` -- **YAML DSL** — parse YAML into Plan/Task structs at runtime -- **Dry-run mode** — `plan.DryRun(ctx)` shows what would execute -- **Checkpointing** — save progress to disk, resume after crash -- **Event triggers** — run a plan when agent comes online -- **Server-side execution** — `POST /orchestrate` for durable runs diff --git a/docs/plans/2026-03-02-demo-recording-design.md b/docs/plans/2026-03-02-demo-recording-design.md deleted file mode 100644 index 460092416..000000000 --- a/docs/plans/2026-03-02-demo-recording-design.md +++ /dev/null @@ -1,88 +0,0 @@ -# Demo Recording Design - -## Overview - -Create a VHS-scripted terminal recording (GIF) for the README that sells OSAPI's -value in ~30-60 seconds. Narrative: "zero to managed system in 30 seconds." - -## Tool - -[VHS](https://github.com/charmbracelet/vhs) — scriptable `.tape` files that -render to GIF. Reproducible, version-controllable. - -## Output - -- Format: GIF -- Location: `asset/demo.gif` (embedded in README) -- Duration: ~36 seconds - -## Demo Flow - -### Scene 1: Start (~5s) - -``` -$ osapi start -``` - -One command boots NATS, API server, and agent. Brief pause on startup output. - -### Scene 2: Health check (~5s) - -``` -$ osapi client health status -``` - -Full system health — component status, agent metrics, job stats. "Everything's -green." - -### Scene 3: Node discovery (~8s) - -``` -$ osapi client agent list -$ osapi client node status -``` - -Agent registered with OS info, load, memory. Rich node status with uptime, disk, -memory, load averages. - -### Scene 4: Run a command (~8s) - -``` -$ osapi client node command exec --command "uname" --args "-a" -``` - -Async job submission + result. Demonstrates the job system implicitly. - -### Scene 5: Audit trail (~5s) - -``` -$ osapi client audit list -``` - -Audit log showing all API calls we just made. "Everything is tracked." - -### Scene 6: JSON output (~5s) - -``` -$ osapi client node status --json -``` - -Structured JSON output — shows automation-friendliness. - -## Out of Scope (future multi-host recording) - -- `--target _all` / `--target hostname` broadcasting -- Agent labels and label-based routing -- Job lifecycle (add/list/get/retry/delete) -- DNS get/update, ping -- Token generation / RBAC - -## Implementation Notes - -- VHS `.tape` file lives at project root (`demo.tape`) -- Need VHS installed (`brew install vhs`) -- Uses tmux with a top/bottom split: top pane runs `osapi start` (server logs - visible), bottom pane runs client commands -- VHS drives tmux via keystrokes — starts tmux, splits pane, types commands into - each pane -- GIF stored in `asset/demo.gif`, referenced from README diff --git a/docs/plans/2026-03-02-stdout-stderr-flags-design.md b/docs/plans/2026-03-02-stdout-stderr-flags-design.md deleted file mode 100644 index 8456bca4b..000000000 --- a/docs/plans/2026-03-02-stdout-stderr-flags-design.md +++ /dev/null @@ -1,89 +0,0 @@ -# Design: --stdout and --stderr flags for command exec/shell - -## Problem - -Running a remote command with OSAPI requires piping through `jq` to see the -actual output: - -```bash -osapi client node command exec --command ls --json | jq -r '.results[0].stdout' -``` - -The default table view truncates stdout to 50 characters and flattens multi-line -output. There's no way to get raw command output directly. - -## Solution - -Add `--stdout` and `--stderr` flags to `node command exec` and -`node command shell` CLI commands. These print the remote command's raw output -directly to the terminal. - -## Behavior - -### Flags - -- `--stdout` — print remote stdout to terminal stdout -- `--stderr` — print remote stderr to terminal stderr (fd 2) -- Both together — print stdout to fd 1, stderr to fd 2 -- Mutually exclusive with `--json` -- Neither flag — current behavior (table display) - -### Single-host output - -Raw output, no decoration: - -``` -$ osapi client node command exec --command ls --args "-la" --stdout -total 48 -drwxr-xr-x 12 john staff 384 Mar 2 10:00 . --rw-r--r-- 1 john staff 1234 Mar 2 09:30 main.go -``` - -### Multi-host output - -Hostname-prefixed per line, hostname dimmed with lipgloss: - -``` -$ osapi client node command exec --target _all --command hostname --stdout - web-01 web-01.example.com - web-02 web-02.example.com - db-01 db-01.example.com -``` - -Multi-line stdout from multiple hosts: - -``` -$ osapi client node command exec --target _all --command ls --stdout - web-01 file1 - web-01 file2 - web-02 file1 - web-02 file3 -``` - -### Exit code propagation - -The CLI process exits with the remote command's exit code. For multi-host, exits -non-zero if any host returned non-zero. - -## Scope - -- CLI-only change — no API or protocol changes -- Applies to `node command exec` and `node command shell` -- No streaming — still synchronous request/response via NATS -- No architecture changes - -## Files to change - -- `cmd/client_node_command_exec.go` — add flags + output logic -- `cmd/client_node_command_shell.go` — add flags + output logic -- `docs/docs/sidebar/usage/cli/client/node/command-exec.md` — document flags - with examples -- `docs/docs/sidebar/usage/cli/client/node/command-shell.md` — document flags - with examples -- Tests for the new output paths - -## Non-goals - -- True streaming (would require WebSocket/SSE + architecture changes) -- Interactive commands (stdin passthrough) -- Changes to API response format diff --git a/docs/plans/2026-03-02-stdout-stderr-flags.md b/docs/plans/2026-03-02-stdout-stderr-flags.md deleted file mode 100644 index 1cdb1181a..000000000 --- a/docs/plans/2026-03-02-stdout-stderr-flags.md +++ /dev/null @@ -1,510 +0,0 @@ -# --stdout/--stderr Flags Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Add `--stdout` and `--stderr` flags to `node command exec` and -`node command shell` CLI commands for raw command output without `jq`. - -**Architecture:** CLI-only change. Add two boolean flags that bypass the -table/JSON display and print raw remote stdout/stderr directly. Multi-host -output prefixes each line with a dimmed hostname. Exit code propagates from the -remote command. - -**Tech Stack:** Go, Cobra, lipgloss (existing `cli.DimStyle`) - ---- - -### Task 1: Add PrintRawOutput helper to internal/cli - -**Files:** - -- Create: `internal/cli/raw_output.go` -- Create: `internal/cli/raw_output_test.go` - -**Step 1: Write the failing test** - -Create `internal/cli/raw_output_test.go`: - -```go -package cli_test - -import ( - "bytes" - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/cli" -) - -type RawOutputPublicTestSuite struct { - suite.Suite -} - -func TestRawOutputPublicTestSuite(t *testing.T) { - suite.Run(t, new(RawOutputPublicTestSuite)) -} - -func (suite *RawOutputPublicTestSuite) TestPrintRawOutput_SingleHost() { - tests := []struct { - name string - results []cli.RawResult - wantOut string - wantErr string - }{ - { - name: "single host stdout only", - results: []cli.RawResult{ - {Hostname: "server1", Stdout: "file1\nfile2\n", Stderr: ""}, - }, - wantOut: "file1\nfile2\n", - wantErr: "", - }, - { - name: "single host stderr only", - results: []cli.RawResult{ - {Hostname: "server1", Stdout: "", Stderr: "permission denied\n"}, - }, - wantOut: "", - wantErr: "permission denied\n", - }, - { - name: "single host both", - results: []cli.RawResult{ - {Hostname: "server1", Stdout: "output\n", Stderr: "warning\n"}, - }, - wantOut: "output\n", - wantErr: "warning\n", - }, - { - name: "single host empty output", - results: []cli.RawResult{{Hostname: "server1"}}, - wantOut: "", - wantErr: "", - }, - } - - for _, tc := range tests { - suite.Run(tc.name, func() { - var stdout, stderr bytes.Buffer - cli.PrintRawOutput(&stdout, &stderr, tc.results, true, true) - assert.Equal(suite.T(), tc.wantOut, stdout.String()) - assert.Equal(suite.T(), tc.wantErr, stderr.String()) - }) - } -} - -func (suite *RawOutputPublicTestSuite) TestPrintRawOutput_MultiHost() { - tests := []struct { - name string - results []cli.RawResult - wantOut string - }{ - { - name: "multi host stdout prefixed", - results: []cli.RawResult{ - {Hostname: "web-01", Stdout: "file1\nfile2\n"}, - {Hostname: "web-02", Stdout: "file3\n"}, - }, - // Hostname prefix is added (without lipgloss color in test) - wantOut: "web-01 file1\nweb-01 file2\nweb-02 file3\n", - }, - } - - for _, tc := range tests { - suite.Run(tc.name, func() { - var stdout, stderr bytes.Buffer - cli.PrintRawOutputPlain(&stdout, &stderr, tc.results, true, false) - assert.Equal(suite.T(), tc.wantOut, stdout.String()) - }) - } -} -``` - -**Step 2: Run test to verify it fails** - -Run: `go test -run TestRawOutputPublicTestSuite -v ./internal/cli/...` Expected: -FAIL — `PrintRawOutput`, `PrintRawOutputPlain`, `RawResult` not defined - -**Step 3: Write minimal implementation** - -Create `internal/cli/raw_output.go`: - -```go -package cli - -import ( - "fmt" - "io" - "strings" -) - -// RawResult holds raw command output for a single host. -type RawResult struct { - Hostname string - Stdout string - Stderr string - ExitCode int -} - -// PrintRawOutput writes raw command output to the given writers. -// For single results, output is printed without hostname prefix. -// For multiple results, each line is prefixed with a dimmed hostname. -// showStdout/showStderr control which streams are printed. -func PrintRawOutput( - stdout io.Writer, - stderr io.Writer, - results []RawResult, - showStdout bool, - showStderr bool, -) { - multiHost := len(results) > 1 - - for _, r := range results { - if showStdout && r.Stdout != "" { - writeLines(stdout, r.Hostname, r.Stdout, multiHost, true) - } - if showStderr && r.Stderr != "" { - writeLines(stderr, r.Hostname, r.Stderr, multiHost, true) - } - } -} - -// PrintRawOutputPlain writes raw output without lipgloss styling. -// Used for testing and non-TTY output. -func PrintRawOutputPlain( - stdout io.Writer, - stderr io.Writer, - results []RawResult, - showStdout bool, - showStderr bool, -) { - multiHost := len(results) > 1 - - for _, r := range results { - if showStdout && r.Stdout != "" { - writeLines(stdout, r.Hostname, r.Stdout, multiHost, false) - } - if showStderr && r.Stderr != "" { - writeLines(stderr, r.Hostname, r.Stderr, multiHost, false) - } - } -} - -func writeLines( - w io.Writer, - hostname string, - content string, - multiHost bool, - styled bool, -) { - lines := strings.Split(content, "\n") - for _, line := range lines { - if line == "" && content[len(content)-1] == '\n' { - continue - } - if multiHost { - prefix := hostname - if styled { - prefix = DimStyle.Render(hostname) - } - fmt.Fprintf(w, "%s %s\n", prefix, line) - } else { - fmt.Fprintln(w, line) - } - } -} -``` - -**Step 4: Run test to verify it passes** - -Run: `go test -run TestRawOutputPublicTestSuite -v ./internal/cli/...` Expected: -PASS - -**Step 5: Commit** - -``` -git add internal/cli/raw_output.go internal/cli/raw_output_test.go -git commit -m "feat(cli): add PrintRawOutput helper for --stdout/--stderr flags" -``` - ---- - -### Task 2: Add --stdout/--stderr flags to command exec - -**Files:** - -- Modify: `cmd/client_node_command_exec.go` - -**Step 1: Add flag definitions in init()** - -Add after line 132 (`Int("timeout", ...)`): - -```go - clientNodeCommandExecCmd.PersistentFlags(). - Bool("stdout", false, "Print only remote stdout") - clientNodeCommandExecCmd.PersistentFlags(). - Bool("stderr", false, "Print only remote stderr") -``` - -**Step 2: Add raw output handling in the Run function** - -Read the new flags after `timeout` (after line 43): - -```go - showStdout, _ := cmd.Flags().GetBool("stdout") - showStderr, _ := cmd.Flags().GetBool("stderr") -``` - -Replace the `case http.StatusAccepted:` block (lines 66-99) with: - -```go - case http.StatusAccepted: - if jsonOutput { - fmt.Println(string(resp.Body)) - return - } - - if (showStdout || showStderr) && resp.JSON202 != nil { - results := make([]cli.RawResult, 0, len(resp.JSON202.Results)) - maxExitCode := 0 - for _, r := range resp.JSON202.Results { - exitCode := 0 - if r.ExitCode != nil { - exitCode = *r.ExitCode - } - if exitCode > maxExitCode { - maxExitCode = exitCode - } - results = append(results, cli.RawResult{ - Hostname: r.Hostname, - Stdout: cli.SafeString(r.Stdout), - Stderr: cli.SafeString(r.Stderr), - ExitCode: exitCode, - }) - } - cli.PrintRawOutput(os.Stdout, os.Stderr, results, showStdout, showStderr) - if maxExitCode != 0 { - os.Exit(maxExitCode) - } - return - } - - if resp.JSON202 != nil && resp.JSON202.JobId != nil { - fmt.Println() - cli.PrintKV("Job ID", resp.JSON202.JobId.String()) - } - - if resp.JSON202 != nil && len(resp.JSON202.Results) > 0 { - results := make([]cli.ResultRow, 0, len(resp.JSON202.Results)) - for _, r := range resp.JSON202.Results { - results = append(results, cli.ResultRow{ - Hostname: r.Hostname, - Changed: r.Changed, - Error: r.Error, - Fields: []string{ - cli.SafeString(r.Stdout), - cli.SafeString(r.Stderr), - cli.IntToSafeString(r.ExitCode), - formatDurationMs(r.DurationMs), - }, - }) - } - headers, rows := cli.BuildBroadcastTable(results, []string{ - "STDOUT", - "STDERR", - "EXIT CODE", - "DURATION", - }) - cli.PrintCompactTable([]cli.Section{{Headers: headers, Rows: rows}}) - } -``` - -Add `"os"` to the imports. - -**Step 3: Run tests and build** - -Run: `go build ./... && just go::unit` Expected: PASS - -**Step 4: Commit** - -``` -git add cmd/client_node_command_exec.go -git commit -m "feat(cli): add --stdout/--stderr flags to node command exec" -``` - ---- - -### Task 3: Add --stdout/--stderr flags to command shell - -**Files:** - -- Modify: `cmd/client_node_command_shell.go` - -**Step 1: Add flag definitions in init()** - -Add after line 118 (`Int("timeout", ...)`): - -```go - clientNodeCommandShellCmd.PersistentFlags(). - Bool("stdout", false, "Print only remote stdout") - clientNodeCommandShellCmd.PersistentFlags(). - Bool("stderr", false, "Print only remote stderr") -``` - -**Step 2: Add raw output handling in the Run function** - -Read the new flags after `timeout` (after line 41): - -```go - showStdout, _ := cmd.Flags().GetBool("stdout") - showStderr, _ := cmd.Flags().GetBool("stderr") -``` - -Replace the `case http.StatusAccepted:` block (lines 62-96) with the same -pattern as Task 2 (identical logic). - -Add `"os"` to the imports. - -**Step 3: Run tests and build** - -Run: `go build ./... && just go::unit` Expected: PASS - -**Step 4: Commit** - -``` -git add cmd/client_node_command_shell.go -git commit -m "feat(cli): add --stdout/--stderr flags to node command shell" -``` - ---- - -### Task 4: Update CLI documentation - -**Files:** - -- Modify: `docs/docs/sidebar/usage/cli/client/node/command/exec.md` -- Modify: `docs/docs/sidebar/usage/cli/client/node/command/shell.md` - -**Step 1: Update exec.md** - -Add a new section after "## JSON Output" and before "## Flags": - -````markdown -## Raw Output - -Use `--stdout` to print only the remote command's stdout, without the table -wrapper: - -```bash -$ osapi client node command exec --command ls --args "-la" --stdout -total 48 -drwxr-xr-x 12 john staff 384 Mar 2 10:00 . --rw-r--r-- 1 john staff 1234 Mar 2 09:30 main.go -``` -```` - -Use `--stderr` to print only stderr: - -```bash -$ osapi client node command exec --command ls --args "/nonexistent" --stderr -ls: cannot access '/nonexistent': No such file or directory -``` - -Both flags can be combined. When targeting multiple hosts, each line is prefixed -with the hostname: - -```bash -$ osapi client node command exec --command hostname --target _all --stdout - web-01 web-01.example.com - web-02 web-02.example.com -``` - -The CLI exit code matches the remote command's exit code, making it scriptable: - -```bash -$ osapi client node command exec --command "test" --args "-f,/etc/hosts" --stdout && echo exists -exists -``` - -```` - -Add the new flags to the Flags table: - -```markdown -| `--stdout` | Print only remote stdout | | -| `--stderr` | Print only remote stderr | | -```` - -**Step 2: Update shell.md** - -Add the same "## Raw Output" section with shell-appropriate examples: - -````markdown -## Raw Output - -Use `--stdout` to print only the remote command's stdout: - -```bash -$ osapi client node command shell --command "df -h / | tail -1" --stdout -/dev/sda1 50G 12G 35G 26% / -``` -```` - -Use `--stderr` to print only stderr: - -```bash -$ osapi client node command shell --command "cat /nonexistent" --stderr -cat: /nonexistent: No such file or directory -``` - -Both flags can be combined. When targeting multiple hosts, each line is prefixed -with the hostname: - -```bash -$ osapi client node command shell --command "uname -r" --target _all --stdout - web-01 5.15.0-91-generic - web-02 5.15.0-91-generic -``` - -The CLI exit code matches the remote command's exit code. - -```` - -Add the new flags to the Flags table: - -```markdown -| `--stdout` | Print only remote stdout | | -| `--stderr` | Print only remote stderr | | -```` - -**Step 3: Verify docs build** - -Run: `just docs::fmt-check` (or `just docs::build` if available) - -**Step 4: Commit** - -``` -git add docs/docs/sidebar/usage/cli/client/node/command/exec.md -git add docs/docs/sidebar/usage/cli/client/node/command/shell.md -git commit -m "docs: add --stdout/--stderr flag documentation for exec and shell" -``` - ---- - -### Task 5: Final verification - -**Step 1: Run full test suite** - -Run: `just test` Expected: All lint + unit tests pass - -**Step 2: Build and smoke test** - -Run: `go build -o osapi . && ./osapi client node command exec --help` Expected: -`--stdout` and `--stderr` flags visible in help output - -**Step 3: Commit any fixes** - -If anything needed fixing, commit with appropriate message. diff --git a/docs/plans/2026-03-03-agent-facts-design.md b/docs/plans/2026-03-03-agent-facts-design.md deleted file mode 100644 index 93ba6b34c..000000000 --- a/docs/plans/2026-03-03-agent-facts-design.md +++ /dev/null @@ -1,235 +0,0 @@ -# Design: Agent Fact Collection System - -## Problem - -OSAPI agents register with basic metadata via heartbeat (OS info, uptime, load, -memory), but there's no extensible fact collection system. The -osapi-orchestrator needs host-level facts to enable Ansible-style conditional -execution — "only run on Ubuntu hosts", "skip hosts with < 4GB RAM", "group -hosts by OS distribution". - -Today the orchestrator can only target by hostname or label. It can't make -decisions based on what a host _is_ (architecture, kernel, network interfaces, -cloud region). - -## Design - -### Fact Categories - -**Phase 1 — Built-in facts (cheap, always collected):** - -| Category | Facts | Source | -| -------- | -------------------------------------------------------- | -------------------------- | -| System | architecture, kernel_version, fqdn, service_mgr, pkg_mgr | `host.Provider` extensions | -| Hardware | cpu_count | `host.Provider` extension | -| Network | interfaces (name, ipv4, ipv6, mac) | New `netinfo.Provider` | - -**Phase 2 — Additional providers (opt-in):** - -| Provider | Facts | Source | -| -------- | --------------------------------------------- | ---------------------------------------- | -| Cloud | instance_id, region, instance_type, public_ip | Cloud metadata endpoints (AWS/GCP/Azure) | -| Local | arbitrary key-value data | JSON/YAML files in `/etc/osapi/facts.d/` | - -All Phase 1 facts are sub-millisecond calls. Phase 2 providers may involve -network I/O (cloud metadata) or file I/O (local facts). - -### Storage: Same API, Separate KV - -The heartbeat serves two purposes today: liveness ("I'm alive") and state ("what -I look like"). Splitting these lets each optimize independently. - -**Registry KV (existing)** — lean heartbeat, frequent refresh: - -- Hostname, labels, timestamps -- 10s refresh, 30s TTL -- ~200 bytes per agent - -**Facts KV (new `agent-facts` bucket)** — richer data, less frequent: - -- OS, architecture, kernel, CPU, memory, interfaces, load, uptime -- Extended facts from future providers -- 60s refresh, 5min TTL -- 1-10KB per agent (grows with future providers) - -The API merges both KVs into a single `AgentInfo` response. Consumers never know -about the split. - -### Provider Pattern (Not a Plugin System) - -Facts are gathered through the existing provider layer — the same pattern used -for `hostProvider.GetOSInfo()`, `loadProvider.GetAverageStats()`, etc. There is -no plugin system and no `Collector` interface. - -**Extend `host.Provider`** with new methods: - -- `GetArchitecture() (string, error)` -- `GetKernelVersion() (string, error)` -- `GetFQDN() (string, error)` -- `GetCPUCount() (int, error)` -- `GetServiceManager() (string, error)` -- `GetPackageManager() (string, error)` - -**New `netinfo.Provider`** for network interface facts: - -- `GetInterfaces() ([]NetworkInterface, error)` - -The facts writer calls these providers exactly like the heartbeat calls its -providers — errors are non-fatal, the agent writes whatever data it gathered. - -Future cloud metadata and local facts would be additional providers added to the -agent when needed, following the same pattern. - -### Data Structure - -```go -type FactsRegistration struct { - Architecture string `json:"architecture,omitempty"` - KernelVersion string `json:"kernel_version,omitempty"` - CPUCount int `json:"cpu_count,omitempty"` - FQDN string `json:"fqdn,omitempty"` - ServiceMgr string `json:"service_mgr,omitempty"` - PackageMgr string `json:"package_mgr,omitempty"` - Interfaces []NetworkInterface `json:"interfaces,omitempty"` - Facts map[string]any `json:"facts,omitempty"` -} -``` - -The `Facts map[string]any` field is reserved for future providers that produce -unstructured data (cloud metadata, local facts). - -### API Exposure - -No new endpoints. Existing `GET /agent` and `GET /agent/{hostname}` return -`AgentInfo` which includes all facts. The API server reads both the registry and -facts KV buckets and merges them. - -The orchestrator calls `Agent.List()` once and gets everything needed for host -filtering — no second API call. - -### Orchestrator Integration - -Facts enable four key patterns in the orchestrator DSL: - -**1. Pre-routing host discovery (filter by facts):** - -```go -hosts, _ := o.Discover(ctx, "_all", - orchestrator.OS("Ubuntu"), - orchestrator.Arch("amd64"), - orchestrator.MinMemory(8 * GB), -) -``` - -**2. Fact-aware When guards:** - -```go -o.CommandShell("_all", "apt upgrade -y"). - WhenFact(func(f orchestrator.Facts) bool { - return f.OS.Distribution == "Ubuntu" - }) -``` - -**3. Group-by-fact (multi-distro playbooks):** - -```go -groups, _ := o.GroupByFact(ctx, "os.distribution") -for distro, hosts := range groups { - o.CommandShell(hosts[0], installCmd(distro)) -} -``` - -**4. Facts in TaskFunc (custom logic):** - -```go -o.TaskFunc("decide", func(ctx context.Context, r orchestrator.Results) (*sdk.Result, error) { - agents, _ := r.ListAgents(ctx) - // Use agent facts for decisions -}) -``` - -### Configuration - -```yaml -nats: - facts: - bucket: 'agent-facts' - ttl: '5m' - storage: 'file' - replicas: 1 - -agent: - facts: - interval: '60s' -``` - -## What Changes Where - -### OSAPI (this repo) - -1. `internal/job/types.go` — add `NetworkInterface`, `FactsRegistration`, and - new typed fields on `AgentInfo` -2. `internal/provider/node/host/types.go` — extend `Provider` interface with - `GetArchitecture`, `GetKernelVersion`, `GetFQDN`, `GetCPUCount`, - `GetServiceManager`, `GetPackageManager` -3. `internal/provider/node/host/ubuntu.go` (+ other platforms) — implement new - methods -4. `internal/provider/network/netinfo/` — new provider for `GetInterfaces()` -5. `internal/agent/types.go` — add `factsKV` and `netinfoProvider` fields -6. `internal/agent/agent.go` — accept new provider, start facts loop -7. `internal/agent/facts.go` — facts writer (calls providers, writes KV) -8. `internal/agent/factory.go` — create netinfo provider -9. `internal/config/types.go` — add `NATSFacts` and `AgentFacts` config -10. `cmd/nats_helpers.go` — create facts KV bucket -11. `cmd/api_helpers.go` — wire factsKV into natsBundle and job client -12. `internal/job/client/client.go` — add `FactsKV` option -13. `internal/job/client/query.go` — merge facts into ListAgents/GetAgent -14. `internal/api/agent/gen/api.yaml` — extend AgentInfo schema -15. `internal/api/agent/agent_list.go` — update buildAgentInfo mapping -16. `osapi.yaml` — default config values -17. Documentation (see below) - -### Documentation Updates - -18. `docs/docs/sidebar/features/node-management.md` — update "Agent vs. Node" - section to explain facts, add facts to "What It Manages" table -19. `docs/docs/sidebar/architecture/system-architecture.md` — add `agent-facts` - KV bucket to component map, update NATS layers -20. `docs/docs/sidebar/architecture/job-architecture.md` — add section on facts - collection, describe 60s interval and KV storage -21. `docs/docs/sidebar/usage/configuration.md` — add `nats.facts` and - `agent.facts` config sections, env var table, section reference -22. `docs/docs/sidebar/usage/cli/client/agent/list.md` — update example output - and column table with facts data -23. `docs/docs/sidebar/usage/cli/client/agent/get.md` — add facts fields to - output example and field table -24. `docs/docs/sidebar/usage/cli/client/health/status.md` — add agent-facts - bucket to KV buckets section - -### SDK (osapi-sdk) - -25. Sync api.yaml, regenerate — `AgentInfo` gets new fields automatically - -### Orchestrator (osapi-orchestrator) - -26. `Discover()` method — query `Agent.List()`, apply fact predicates -27. Fact predicates — `OS()`, `Arch()`, `MinMemory()`, `FactEquals()`, etc. -28. `WhenFact()` step method -29. `GroupByFact()` method - -## What This Does NOT Change - -- NATS routing unchanged — `_all`, `_any`, labels work as before -- No agent-side filtering — facts filter at publisher (orchestrator) side -- No new API endpoints — facts are richer `AgentInfo` data -- Labels remain the primary routing mechanism; facts are for conditional logic - and discovery -- Existing heartbeat liveness behavior unchanged -- No plugin system — facts are gathered through the provider layer - -## Phases - -- **Phase 1**: Typed facts via providers, separate KV, API exposure, docs -- **Phase 2**: Cloud metadata provider, local facts provider -- **Phase 3**: Orchestrator DSL extensions (`Discover`, `WhenFact`, - `GroupByFact`) diff --git a/docs/plans/2026-03-03-agent-facts.md b/docs/plans/2026-03-03-agent-facts.md deleted file mode 100644 index febd7fd56..000000000 --- a/docs/plans/2026-03-03-agent-facts.md +++ /dev/null @@ -1,916 +0,0 @@ -# Agent Facts Collection System — Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Add extensible fact collection to agents via the provider layer, -stored in a separate KV bucket, merged into existing API responses, enabling -orchestrator-side host filtering. - -**Architecture:** Extend `host.Provider` with new fact methods (architecture, -kernel, FQDN, CPU count, service manager, package manager). Create a new -`netinfo.Provider` for network interfaces. The agent gathers facts on a 60s -interval and writes them to a dedicated `agent-facts` KV bucket. The job client -merges facts into `AgentInfo` when serving `ListAgents`/`GetAgent`. No plugin -system — everything goes through providers. - -**Tech Stack:** Go 1.25, NATS JetStream KV, gopsutil, oapi-codegen, -testify/suite, gomock - -**Design doc:** `docs/plans/2026-03-03-agent-facts-design.md` - ---- - -### Task 1: Add Types — NetworkInterface, FactsRegistration, AgentInfo fields - -**Files:** - -- Modify: `internal/job/types.go` -- Test: `internal/job/types_public_test.go` (or appropriate existing test file) - -**Step 1: Write the failing test** - -Add a test for JSON round-trip of `FactsRegistration` and `NetworkInterface`. -Use testify/suite table-driven pattern. Verify all fields serialize and -deserialize correctly, including the `Facts map[string]any` field. - -**Step 2: Run test to verify it fails** - -```bash -go test -run TestFactsRegistration -v ./internal/job/... -``` - -Expected: FAIL — types undefined. - -**Step 3: Write minimal implementation** - -Add to `internal/job/types.go`: - -```go -// NetworkInterface represents a network interface with its address. -type NetworkInterface struct { - Name string `json:"name"` - IPv4 string `json:"ipv4,omitempty"` - MAC string `json:"mac,omitempty"` -} - -// FactsRegistration represents an agent's facts entry in the facts KV bucket. -type FactsRegistration struct { - Architecture string `json:"architecture,omitempty"` - KernelVersion string `json:"kernel_version,omitempty"` - CPUCount int `json:"cpu_count,omitempty"` - FQDN string `json:"fqdn,omitempty"` - ServiceMgr string `json:"service_mgr,omitempty"` - PackageMgr string `json:"package_mgr,omitempty"` - Interfaces []NetworkInterface `json:"interfaces,omitempty"` - Facts map[string]any `json:"facts,omitempty"` -} -``` - -Add the same typed fields to the existing `AgentInfo` struct (after -`AgentVersion`): `Architecture`, `KernelVersion`, `CPUCount`, `FQDN`, -`ServiceMgr`, `PackageMgr`, `Interfaces`, `Facts`. - -**Step 4: Run test to verify it passes** - -```bash -go test -run TestFactsRegistration -v ./internal/job/... -``` - -**Step 5: Commit** - -``` -feat(job): add NetworkInterface and FactsRegistration types -``` - ---- - -### Task 2: Add Config Types — NATSFacts and AgentFacts - -**Files:** - -- Modify: `internal/config/types.go` - -**Step 1: Add config structs** - -Add `NATSFacts` after `NATSRegistry`: - -```go -type NATSFacts struct { - Bucket string `mapstructure:"bucket"` - TTL string `mapstructure:"ttl"` - Storage string `mapstructure:"storage"` - Replicas int `mapstructure:"replicas"` -} -``` - -Add `Facts NATSFacts` field to the `NATS` struct. - -Add `AgentFacts` after `AgentConsumer`: - -```go -type AgentFacts struct { - Interval string `mapstructure:"interval"` -} -``` - -Add `Facts AgentFacts` field to `AgentConfig`. - -**Step 2: Verify build** - -```bash -go build ./... -``` - -**Step 3: Commit** - -``` -feat(config): add NATSFacts and AgentFacts config types -``` - ---- - -### Task 3: Extend host.Provider with Fact Methods - -**Files:** - -- Modify: `internal/provider/node/host/types.go` — add methods to interface -- Modify: `internal/provider/node/host/ubuntu.go` — implement for Ubuntu -- Modify: `internal/provider/node/host/mocks/types.gen.go` — update mock - defaults -- Test: `internal/provider/node/host/ubuntu_public_test.go` or similar - -**Step 1: Write failing tests** - -Add table-driven tests for each new method: `GetArchitecture`, -`GetKernelVersion`, `GetFQDN`, `GetCPUCount`, `GetServiceManager`, -`GetPackageManager`. Test success cases and that errors don't panic. - -**Step 2: Run tests to verify they fail** - -```bash -go test -run TestGetArchitecture -v ./internal/provider/node/host/... -``` - -**Step 3: Add methods to Provider interface** - -In `internal/provider/node/host/types.go`: - -```go -type Provider interface { - GetUptime() (time.Duration, error) - GetHostname() (string, error) - GetOSInfo() (*OSInfo, error) - GetArchitecture() (string, error) - GetKernelVersion() (string, error) - GetFQDN() (string, error) - GetCPUCount() (int, error) - GetServiceManager() (string, error) - GetPackageManager() (string, error) -} -``` - -**Step 4: Implement in Ubuntu provider** - -In `internal/provider/node/host/ubuntu.go`: - -- `GetArchitecture()` → `runtime.GOARCH` -- `GetKernelVersion()` → `host.KernelVersion()` from gopsutil -- `GetFQDN()` → `os.Hostname()` (FQDN lookup optional) -- `GetCPUCount()` → `runtime.NumCPU()` -- `GetServiceManager()` → check `/run/systemd/system` existence → `"systemd"` -- `GetPackageManager()` → check executable existence (`apt`, `yum`, `dnf`) - -Wrap gopsutil/stdlib calls in package-level function variables for testability, -following the existing pattern (e.g., `hostInfoFn`). - -**Step 5: Regenerate mocks** - -```bash -go generate ./internal/provider/node/host/... -``` - -Update `mocks/types.gen.go` to add defaults for new methods in -`NewDefaultMockProvider`: - -```go -mock.EXPECT().GetArchitecture().Return("amd64", nil).AnyTimes() -mock.EXPECT().GetKernelVersion().Return("5.15.0-91-generic", nil).AnyTimes() -mock.EXPECT().GetFQDN().Return("default-hostname.local", nil).AnyTimes() -mock.EXPECT().GetCPUCount().Return(4, nil).AnyTimes() -mock.EXPECT().GetServiceManager().Return("systemd", nil).AnyTimes() -mock.EXPECT().GetPackageManager().Return("apt", nil).AnyTimes() -``` - -**Step 6: Run all tests** - -```bash -go test -v ./internal/provider/node/host/... -go build ./... -``` - -**Step 7: Commit** - -``` -feat(provider): extend host.Provider with fact methods -``` - ---- - -### Task 4: Create netinfo.Provider for Network Interfaces - -**Files:** - -- Create: `internal/provider/network/netinfo/types.go` -- Create: `internal/provider/network/netinfo/netinfo.go` -- Create: `internal/provider/network/netinfo/mocks/` (generate) -- Test: `internal/provider/network/netinfo/netinfo_public_test.go` - -**Step 1: Write failing test** - -Test `GetInterfaces()` returns non-loopback, up interfaces with name, IPv4, and -MAC. Use table-driven pattern. Mock `net.Interfaces` via a package-level -function variable. - -**Step 2: Define the interface** - -In `types.go`: - -```go -package netinfo - -import "github.com/osapi-io/osapi/internal/job" - -type Provider interface { - GetInterfaces() ([]job.NetworkInterface, error) -} -``` - -**Step 3: Implement** - -In `netinfo.go`: - -```go -package netinfo - -import ( - "net" - - "github.com/osapi-io/osapi/internal/job" -) - -type Netinfo struct{} - -func New() *Netinfo { return &Netinfo{} } - -var netInterfacesFn = net.Interfaces - -func (n *Netinfo) GetInterfaces() ([]job.NetworkInterface, error) { - ifaces, err := netInterfacesFn() - if err != nil { - return nil, err - } - - var result []job.NetworkInterface - for _, iface := range ifaces { - if iface.Flags&net.FlagLoopback != 0 || iface.Flags&net.FlagUp == 0 { - continue - } - - ni := job.NetworkInterface{ - Name: iface.Name, - MAC: iface.HardwareAddr.String(), - } - - addrs, err := iface.Addrs() - if err == nil { - for _, addr := range addrs { - if ipNet, ok := addr.(*net.IPNet); ok && ipNet.IP.To4() != nil { - ni.IPv4 = ipNet.IP.String() - break - } - } - } - - result = append(result, ni) - } - - return result, nil -} -``` - -**Step 4: Generate mocks and add defaults** - -```bash -# Add generate.go with //go:generate directive -go generate ./internal/provider/network/netinfo/... -``` - -Create `mocks/types.gen.go` with `NewDefaultMockProvider` returning a stub -interface list. - -**Step 5: Run tests** - -```bash -go test -v ./internal/provider/network/netinfo/... -``` - -**Step 6: Commit** - -``` -feat(provider): add netinfo.Provider for network interface facts -``` - ---- - -### Task 5: Facts KV Bucket Infrastructure - -**Files:** - -- Modify: `internal/cli/nats.go` — add `BuildFactsKVConfig` -- Modify: `cmd/nats_helpers.go` — create facts KV in `setupJetStream` -- Modify: `cmd/api_helpers.go` — add `factsKV` to `natsBundle`, pass to job - client and metrics provider -- Modify: `internal/job/client/client.go` — add `FactsKV` to `Options` and - `factsKV` to `Client` - -**Step 1: Add BuildFactsKVConfig** - -In `internal/cli/nats.go`, add after `BuildRegistryKVConfig` (follow the exact -same pattern): - -```go -func BuildFactsKVConfig( - namespace string, - factsCfg config.NATSFacts, -) jetstream.KeyValueConfig { - factsBucket := job.ApplyNamespaceToInfraName(namespace, factsCfg.Bucket) - factsTTL, _ := time.ParseDuration(factsCfg.TTL) - - return jetstream.KeyValueConfig{ - Bucket: factsBucket, - TTL: factsTTL, - Storage: ParseJetstreamStorageType(factsCfg.Storage), - Replicas: factsCfg.Replicas, - } -} -``` - -**Step 2: Create facts KV in setupJetStream** - -In `cmd/nats_helpers.go`, add after the registry KV block (line ~165): - -```go -if appConfig.NATS.Facts.Bucket != "" { - factsKVConfig := cli.BuildFactsKVConfig(namespace, appConfig.NATS.Facts) - if _, err := nc.CreateOrUpdateKVBucketWithConfig(ctx, factsKVConfig); err != nil { - return fmt.Errorf("create facts KV bucket %s: %w", factsKVConfig.Bucket, err) - } -} -``` - -**Step 3: Wire into natsBundle and job client** - -Add `factsKV jetstream.KeyValue` to `natsBundle` struct. - -In `connectNATSBundle`, create the facts KV bucket (only if configured) and pass -it as `FactsKV` in `jobclient.Options`. - -Add `factsKV` to the returned `natsBundle`. - -In `newMetricsProvider`, add `b.factsKV` to the `KVInfoFn` buckets slice. - -**Step 4: Add to job client** - -In `internal/job/client/client.go`: - -- Add `FactsKV jetstream.KeyValue` to `Options` -- Add `factsKV jetstream.KeyValue` to `Client` struct -- Assign in `New()`: `factsKV: opts.FactsKV,` - -**Step 5: Verify build** - -```bash -go build ./... -``` - -**Step 6: Commit** - -``` -feat(nats): add facts KV bucket infrastructure -``` - ---- - -### Task 6: Facts Writer in Agent - -**Files:** - -- Create: `internal/agent/facts.go` -- Create: `internal/agent/facts_test.go` (internal tests) -- Modify: `internal/agent/types.go` — add `factsKV` and `netinfoProvider` -- Modify: `internal/agent/agent.go` — add params to `New()`, call `startFacts()` - in `Start()` -- Modify: `internal/agent/factory.go` — create netinfo provider -- Modify: `cmd/agent_helpers.go` — pass `factsKV` and netinfo provider - -**Step 1: Add fields to Agent struct** - -In `internal/agent/types.go`, add: - -```go -factsKV jetstream.KeyValue -netinfoProvider netinfo.Provider -``` - -**Step 2: Update New() and factory** - -In `internal/agent/agent.go`, add `netinfoProvider netinfo.Provider` and -`factsKV jetstream.KeyValue` parameters. Assign them. - -In `internal/agent/factory.go`, add `netinfo.New()` to the provider factory -return values. Update `CreateProviders()` signature. - -**Step 3: Write failing test for writeFacts** - -Create `internal/agent/facts_test.go` (internal, `package agent`). Use -`FactsTestSuite` with gomock. Mock the `factsKV.Put()` call. Verify the written -data contains architecture, cpu_count, interfaces. Follow the existing -`heartbeat_test.go` pattern exactly. - -Test cases: - -- `"when Put succeeds writes facts"` — verify JSON contains expected fields -- `"when Put fails logs warning"` — verify no panic -- `"when marshal fails logs warning"` — override `marshalJSON` variable - -**Step 4: Run test to verify it fails** - -```bash -go test -run TestWriteFacts -v ./internal/agent/... -``` - -**Step 5: Implement facts.go** - -Create `internal/agent/facts.go`: - -```go -package agent - -// factsInterval controls the fact refresh period. -var factsInterval = 60 * time.Second - -func (a *Agent) startFacts(ctx context.Context, hostname string) { - if a.factsKV == nil { - return - } - a.writeFacts(ctx, hostname) - a.wg.Add(1) - go func() { - defer a.wg.Done() - ticker := time.NewTicker(factsInterval) - defer ticker.Stop() - for { - select { - case <-ctx.Done(): - return - case <-ticker.C: - a.writeFacts(ctx, hostname) - } - } - }() -} - -func (a *Agent) writeFacts(ctx context.Context, hostname string) { - reg := job.FactsRegistration{} - - // Call providers — errors are non-fatal - if arch, err := a.hostProvider.GetArchitecture(); err == nil { - reg.Architecture = arch - } - if kv, err := a.hostProvider.GetKernelVersion(); err == nil { - reg.KernelVersion = kv - } - if fqdn, err := a.hostProvider.GetFQDN(); err == nil { - reg.FQDN = fqdn - } - if count, err := a.hostProvider.GetCPUCount(); err == nil { - reg.CPUCount = count - } - if mgr, err := a.hostProvider.GetServiceManager(); err == nil { - reg.ServiceMgr = mgr - } - if mgr, err := a.hostProvider.GetPackageManager(); err == nil { - reg.PackageMgr = mgr - } - if ifaces, err := a.netinfoProvider.GetInterfaces(); err == nil { - reg.Interfaces = ifaces - } - - data, err := marshalJSON(reg) - if err != nil { - a.logger.Warn("failed to marshal facts", ...) - return - } - - key := factsKey(hostname) - if _, err := a.factsKV.Put(ctx, key, data); err != nil { - a.logger.Warn("failed to write facts", ...) - } -} - -func factsKey(hostname string) string { - return "facts." + job.SanitizeHostname(hostname) -} -``` - -**Step 6: Wire into Start()** - -In `internal/agent/server.go`, after `a.startHeartbeat(a.ctx, hostname)`: - -```go -a.startFacts(a.ctx, hostname) -``` - -**Step 7: Update cmd/agent_helpers.go** - -Pass `b.factsKV` and the netinfo provider to `agent.New()`. - -**Step 8: Run tests** - -```bash -go test -v ./internal/agent/... -go build ./... -``` - -**Step 9: Commit** - -``` -feat(agent): add facts writer with provider-based collection -``` - ---- - -### Task 7: Merge Facts into ListAgents and GetAgent - -**Files:** - -- Modify: `internal/job/client/query.go` — add `mergeFacts` helper -- Test: `internal/job/client/query_public_test.go` - -**Step 1: Write failing test** - -Add test cases for facts merging. Test: - -- Facts KV has data → fields appear in AgentInfo -- Facts KV is nil → graceful degradation (fields empty) -- Facts KV Get returns error → graceful degradation - -Follow existing test patterns in `query_public_test.go`. - -**Step 2: Run test to verify it fails** - -```bash -go test -run TestListAgentsWithFacts -v ./internal/job/client/... -``` - -**Step 3: Implement mergeFacts** - -Add to `internal/job/client/query.go`: - -```go -func (c *Client) mergeFacts(ctx context.Context, info *job.AgentInfo) { - if c.factsKV == nil { - return - } - - key := "facts." + job.SanitizeHostname(info.Hostname) - entry, err := c.factsKV.Get(ctx, key) - if err != nil { - return - } - - var facts job.FactsRegistration - if err := json.Unmarshal(entry.Value(), &facts); err != nil { - return - } - - info.Architecture = facts.Architecture - info.KernelVersion = facts.KernelVersion - info.CPUCount = facts.CPUCount - info.FQDN = facts.FQDN - info.ServiceMgr = facts.ServiceMgr - info.PackageMgr = facts.PackageMgr - info.Interfaces = facts.Interfaces - info.Facts = facts.Facts -} -``` - -Call `c.mergeFacts(ctx, &info)` in both `ListAgents` (after -`agentInfoFromRegistration`) and `GetAgent` (after building info). - -**Step 4: Run tests** - -```bash -go test -v ./internal/job/client/... -``` - -**Step 5: Commit** - -``` -feat(job): merge facts KV data into ListAgents and GetAgent -``` - ---- - -### Task 8: OpenAPI Spec and API Handler - -**Files:** - -- Modify: `internal/api/agent/gen/api.yaml` -- Run: `go generate ./internal/api/agent/gen/...` -- Modify: `internal/api/agent/agent_list.go` — update `buildAgentInfo` -- Test: `internal/api/agent/agent_list_public_test.go` (or existing test) - -**Step 1: Extend OpenAPI spec** - -Add to `AgentInfo` properties in `api.yaml`: - -```yaml -architecture: - type: string - description: CPU architecture. - example: 'amd64' -kernel_version: - type: string - description: OS kernel version. - example: '5.15.0-91-generic' -cpu_count: - type: integer - description: Number of logical CPUs. - example: 4 -fqdn: - type: string - description: Fully qualified domain name. - example: 'web-01.example.com' -service_mgr: - type: string - description: Init system. - example: 'systemd' -package_mgr: - type: string - description: Package manager. - example: 'apt' -interfaces: - type: array - items: - $ref: '#/components/schemas/NetworkInterfaceResponse' -facts: - type: object - additionalProperties: true - description: Extended facts from additional providers. -``` - -Add `NetworkInterfaceResponse` schema: - -```yaml -NetworkInterfaceResponse: - type: object - properties: - name: - type: string - example: 'eth0' - ipv4: - type: string - example: '192.168.1.10' - mac: - type: string - example: '00:11:22:33:44:55' - required: - - name -``` - -**Step 2: Regenerate** - -```bash -go generate ./internal/api/agent/gen/... -``` - -**Step 3: Update buildAgentInfo** - -In `internal/api/agent/agent_list.go`, add mappings for new fields after the -existing memory block. Map each non-zero/non-empty field. Map `Interfaces` as -`[]gen.NetworkInterfaceResponse`. - -Check the generated field names in `agent.gen.go` and match them exactly. - -**Step 4: Run tests** - -```bash -go test -v ./internal/api/agent/... -go build ./... -``` - -**Step 5: Commit** - -``` -feat(api): expose agent facts in AgentInfo responses -``` - ---- - -### Task 9: Default Config - -**Files:** - -- Modify: `osapi.yaml` - -**Step 1: Add defaults** - -Add `nats.facts` section after `nats.registry`: - -```yaml -facts: - bucket: 'agent-facts' - ttl: '5m' - storage: 'file' - replicas: 1 -``` - -Add `agent.facts` section after `agent.labels`: - -```yaml -facts: - interval: '60s' -``` - -**Step 2: Verify config loads** - -```bash -go build ./... -``` - -**Step 3: Commit** - -``` -chore: add default facts config to osapi.yaml -``` - ---- - -### Task 10: Update Documentation — Configuration Reference - -**Files:** - -- Modify: `docs/docs/sidebar/usage/configuration.md` - -**Step 1: Add environment variable mappings** - -Add to the env var table: - -| `nats.facts.bucket` | `OSAPI_NATS_FACTS_BUCKET` | | `nats.facts.ttl` | -`OSAPI_NATS_FACTS_TTL` | | `nats.facts.storage` | `OSAPI_NATS_FACTS_STORAGE` | | -`nats.facts.replicas` | `OSAPI_NATS_FACTS_REPLICAS` | | `agent.facts.interval` | -`OSAPI_AGENT_FACTS_INTERVAL` | - -**Step 2: Add section references** - -Add `nats.facts` section reference table (Bucket, TTL, Storage, Replicas). Add -`agent.facts` section reference table (Interval). - -**Step 3: Update full YAML reference** - -Add the `nats.facts` and `agent.facts` blocks to the full reference YAML with -inline comments. - -**Step 4: Commit** - -``` -docs: add facts configuration reference -``` - ---- - -### Task 11: Update Documentation — Feature and Architecture Pages - -**Files:** - -- Modify: `docs/docs/sidebar/features/node-management.md` -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` -- Modify: `docs/docs/sidebar/architecture/job-architecture.md` - -**Step 1: Update node-management.md** - -- In "Agent vs. Node" section, add that agents now expose typed system facts - (architecture, kernel, FQDN, CPU count, network interfaces) in addition to the - basic heartbeat metrics. -- Clarify: facts are gathered every 60s via providers, stored in a separate - `agent-facts` KV bucket with a 5-minute TTL. -- Add a "System Facts" row to the "What It Manages" table. - -**Step 2: Update system-architecture.md** - -- Add `agent-facts` KV bucket to the NATS JetStream section alongside - `agent-registry`. -- Update the component map table to mention facts in the Agent/Provider layer - description. - -**Step 3: Update job-architecture.md** - -- Add a brief section on facts collection: - - Facts are collected independently from the job system. - - 60-second interval, separate KV bucket. - - Providers gather system facts (architecture, kernel, network interfaces, - etc.). - - API merges registry + facts KV into a single AgentInfo response. - -**Step 4: Commit** - -``` -docs: update feature and architecture pages with facts -``` - ---- - -### Task 12: Update Documentation — CLI Pages - -**Files:** - -- Modify: `docs/docs/sidebar/usage/cli/client/agent/list.md` -- Modify: `docs/docs/sidebar/usage/cli/client/agent/get.md` -- Modify: `docs/docs/sidebar/usage/cli/client/health/status.md` - -**Step 1: Update agent list.md** - -Update the example output to show any new facts-derived columns if the CLI is -updated to display them (e.g., ARCH column). If no CLI column changes are -planned for Phase 1, add a note that `--json` output includes full facts data. - -**Step 2: Update agent get.md** - -Add facts fields to the example output and field description table: - -| Architecture | CPU architecture (e.g., amd64) | | Kernel | OS kernel version | -| FQDN | Fully qualified domain name | | CPUs | Number of logical CPUs | | -Service Mgr | Init system (e.g., systemd) | | Package Mgr | Package manager -(e.g., apt) | | Interfaces | Network interfaces with IPv4 and MAC | - -Update the example output block to show these new fields. - -**Step 3: Update health status.md** - -Add `agent-facts` to the KV buckets section in the example output (e.g., -`Bucket: agent-facts (2 keys, 1.5 KB)`). - -**Step 4: Commit** - -``` -docs: update CLI docs with agent facts output -``` - ---- - -### Task 13: Final Verification - -**Step 1: Build** - -```bash -go build ./... -``` - -**Step 2: Unit tests** - -```bash -just go::unit -``` - -**Step 3: Lint** - -```bash -just go::vet -``` - -**Step 4: Format** - -```bash -just go::fmt -``` - -**Step 5: Docs format** - -```bash -just docs::fmt-check -``` - -All must pass. Fix any issues found. - ---- - -## Out of Scope (Phase 2+) - -- Cloud metadata provider (AWS/GCP/Azure metadata endpoints) -- Local facts provider (`/etc/osapi/facts.d/` JSON/YAML files) -- CLI column changes for `agent list` (facts available via `--json`) -- Orchestrator DSL extensions (`Discover`, `WhenFact`, `GroupByFact`) -- SDK sync and regeneration -- `Facts map[string]any` population (reserved for Phase 2 providers) diff --git a/docs/plans/2026-03-05-agent-facts-routes-factref.md b/docs/plans/2026-03-05-agent-facts-routes-factref.md deleted file mode 100644 index 06c40a87d..000000000 --- a/docs/plans/2026-03-05-agent-facts-routes-factref.md +++ /dev/null @@ -1,239 +0,0 @@ -# Agent Facts, Routes, Fact References, and Timeline Fix - -## Context - -Agents collect system facts (OS, memory, load, interfaces, etc.) but lack two -useful capabilities: (1) knowing the primary network interface and full routing -table, and (2) allowing CLI/API parameters to reference agent facts dynamically. -For example, a user should be able to run: - -``` -osapi client network dns get --interface-name @fact.interface.primary --target _all -``` - -...and have each agent resolve `@fact.interface.primary` to its own primary -interface before executing the operation. - -Additionally, the `agent get` CLI output is missing timeline events -(cordon/uncordon history) — the data path exists but timeline should always be -displayed. - -This is a multi-phase effort. All phases stay on a single branch before pushing -upstream. - -## Repo - -All changes in `osapi` at `/Users/john/git/osapi-io/osapi/`. - ---- - -## Phase 1: Fix Timeline Display and Configs - -### Step 1.1: Sync local/nerd configs with osapi.yaml - -`configs/osapi.local.yaml` and `configs/osapi.nerd.yaml` are missing sections -that exist in `osapi.yaml`: - -- **`nats.state`** — missing in both. This is why timeline isn't showing: - `stateKV` is nil so `GetAgentTimeline()` returns early. -- **`nats.facts`** — missing in `osapi.nerd.yaml` -- **`telemetry.metrics`** — missing in both -- **`agent.facts`** — missing in `osapi.nerd.yaml` -- **`agent.conditions`** — missing in both - -Add these sections to both configs to match `osapi.yaml`. - -### Step 1.2: Always show Timeline section in agent get CLI - -File: `cmd/client_agent_get.go` - -Line 169: change `if len(data.Timeline) > 0` to always display the Timeline -section. Show empty table or "No events" when empty. - -### Step 1.3: Always show Timeline section in job get CLI - -File: `internal/cli/ui.go` - -Line 601: same fix — change `if len(resp.Timeline) > 0` to always display the -Timeline section for job details. - ---- - -## Phase 2: Route Collection and Primary Interface - -### Step 2.1: Add Route type to job types - -File: `internal/job/types.go` - -Add a `Route` struct: - -```go -type Route struct { - Destination string `json:"destination"` - Gateway string `json:"gateway"` - Interface string `json:"interface"` - Mask string `json:"mask,omitempty"` - Metric int `json:"metric,omitempty"` - Flags string `json:"flags,omitempty"` -} -``` - -Add fields to `FactsRegistration`: - -```go -PrimaryInterface string `json:"primary_interface,omitempty"` -Routes []Route `json:"routes,omitempty"` -``` - -Add same fields to `AgentInfo`. - -### Step 2.2: Add route provider to netinfo - -File: `internal/provider/network/netinfo/types.go` - -Extend `Provider` interface: - -```go -type Provider interface { - GetInterfaces() ([]job.NetworkInterface, error) - GetRoutes() ([]job.Route, error) - GetPrimaryInterface() (string, error) -} -``` - -### Step 2.3: Linux route implementation - -File: `internal/provider/network/netinfo/linux_get_routes.go` (build tag: -`//go:build linux`) - -Parse `/proc/net/route` using Go (no exec). Use injectable `RouteReaderFn` -(defaults to `os.Open("/proc/net/route")`) for testing. The default route -(destination `00000000`) determines the primary interface. - -Return all routes as `[]job.Route` and identify the primary interface from the -default route entry. - -### Step 2.4: Darwin route stub - -File: `internal/provider/network/netinfo/darwin_get_routes.go` (build tag: -`//go:build darwin`) - -Stub that returns empty routes and empty primary interface (or uses a heuristic -like first interface with a default gateway). Darwin route detection can be -improved later. - -### Step 2.5: Collect routes in agent facts - -File: `internal/agent/facts.go` - -In `writeFacts()`, call `a.netinfoProvider.GetRoutes()` and -`a.netinfoProvider.GetPrimaryInterface()`. Add results to `FactsRegistration`. - -Cache `FactsRegistration` on the Agent struct as `cachedFacts` for use by fact -reference resolution (Phase 3). - -### Step 2.6: Expose via API and CLI - -- `internal/job/client/query.go` `mergeFacts()`: map new fields -- `internal/api/agent/gen/api.yaml`: add `primary_interface` and `routes` to - AgentInfo schema -- `internal/api/agent/agent_list.go` `buildAgentInfo()`: map fields -- `cmd/client_agent_get.go`: display primary interface and routes -- SDK: update `Agent` type and agent spec - -### Step 2.7: Tests - -- `internal/provider/network/netinfo/linux_get_routes_public_test.go`: - table-driven tests for `/proc/net/route` parsing (mock file content via - `RouteReaderFn`) -- Update existing facts test to verify new fields - ---- - -## Phase 3: `@fact.X` Resolution - -### Step 3.1: Fact reference resolver - -New file: `internal/agent/factref.go` - -```go -func ResolveFacts( - params map[string]any, - facts *job.FactsRegistration, -) (map[string]any, error) -``` - -Walk all string values in the params map. For each string containing `@fact.X`, -resolve against the facts struct: - -- `@fact.interface.primary` → `facts.PrimaryInterface` -- `@fact.hostname` → agent hostname -- `@fact.arch` → `facts.Architecture` -- `@fact.os` → `facts.OSInfo` distribution -- `@fact.kernel` → `facts.KernelVersion` -- Extensible: `@fact.custom.X` → `facts.Facts["X"]` - -If a reference cannot be resolved, return an error (fail the job). Multiple -references in one string are supported: -`"@fact.interface.primary on @fact.hostname"` → `"eth0 on web-01"`. - -### Step 3.2: Inject resolution in handler - -File: `internal/agent/handler.go` - -In `handleJobMessage()`, after unmarshalling the `jobRequest` (line ~163) and -before `processJobOperation()` (line ~225), call `ResolveFacts()` on the job -request parameters using the agent's cached facts. Replace the request params -with resolved values. - -If resolution fails (unresolvable reference), fail the job with an error message -indicating which fact reference could not be resolved. - -### Step 3.3: Tests - -File: `internal/agent/factref_public_test.go` - -Table-driven tests: - -- Simple substitution (`@fact.interface.primary` → `eth0`) -- Multiple references in one string -- Nested map values -- Unknown fact reference → error -- No `@fact.` references → params unchanged -- Nil facts → error for any reference -- Custom facts via `@fact.custom.X` - ---- - -## Phase 4 (Future): File Upload and Templates - -Deferred — will be planned separately after Phases 1-3 are complete. Will use -NATS Object Store for blob storage and Go `text/template` for file content -rendering with fact data. - ---- - -## Verification - -After each phase: - -```bash -go build ./... -just go::unit -just go::vet -``` - -Integration test after Phase 2: - -```bash -# Start osapi, then: -go run main.go client agent get --hostname --json | jq .primary_interface -go run main.go client agent get --hostname --json | jq .routes -``` - -Integration test after Phase 3: - -```bash -go run main.go client network dns get \ - --interface-name @fact.interface.primary --target _all -``` diff --git a/docs/plans/2026-03-05-node-conditions-drain-design.md b/docs/plans/2026-03-05-node-conditions-drain-design.md deleted file mode 100644 index 5d784b22e..000000000 --- a/docs/plans/2026-03-05-node-conditions-drain-design.md +++ /dev/null @@ -1,336 +0,0 @@ -# Node Conditions and Agent Drain - -## Context - -OSAPI agents collect rich system metrics (memory, load, disk, CPU count) via -heartbeat and facts, but operators must manually interpret raw numbers to detect -problems. Kubernetes solves this with node conditions — threshold-based booleans -that surface "is anything wrong?" at a glance. - -Additionally, there's no way to gracefully remove an agent from the job routing -pool for maintenance without stopping the process entirely. When an agent stops, -it vanishes from the registry and looks identical to a crash. Kubernetes handles -this with cordon/drain. - -This design adds both features to OSAPI. - -## Node Conditions - -### Condition Types - -Three conditions derived from existing heartbeat and facts data, evaluated -agent-side on each heartbeat tick (10s): - -| Condition | Default Threshold | Data Source | -| ---------------- | -------------------- | ----------------------------------------------- | -| `MemoryPressure` | memory used > 90% | `MemoryStats` (heartbeat) | -| `HighLoad` | load1 > 2× CPU count | `LoadAverages` (heartbeat) + `CPUCount` (facts) | -| `DiskPressure` | any disk > 90% used | `DiskStats` (new in heartbeat) | - -### Condition Structure - -Each condition has: - -```go -type Condition struct { - Type string `json:"type"` - Status bool `json:"status"` - Reason string `json:"reason,omitempty"` - LastTransitionTime time.Time `json:"last_transition_time"` -} -``` - -- `Status`: `true` = condition is active (pressure/overload detected) -- `Reason`: human-readable explanation (e.g., "memory 94% used (15.1/16.0 GB)") -- `LastTransitionTime`: when the condition last changed from true→false or - false→true - -### Configuration - -Thresholds configurable in `osapi.yaml` with sensible defaults: - -```yaml -agent: - conditions: - memory_pressure_threshold: 90 # percent used - high_load_multiplier: 2.0 # load1 / cpu_count - disk_pressure_threshold: 90 # percent used -``` - -### Evaluation - -Conditions are evaluated in the agent during `writeRegistration()`. The agent -maintains previous condition state in memory to track `LastTransitionTime` — -only updated when the boolean flips. - -DiskPressure requires adding disk stats to the heartbeat. The existing -`disk.Provider` already implements `GetUsage()` so the data is available. Disk -collection joins the existing non-fatal provider pattern: if it fails, the -DiskPressure condition is simply not evaluated. - -### Storage - -Conditions are stored as part of `AgentRegistration` in the registry KV bucket. -No new KV bucket needed. - -```go -type AgentRegistration struct { - // ... existing fields ... - Conditions []Condition `json:"conditions,omitempty"` -} -``` - -### CLI Display - -`agent list` gains a CONDITIONS column showing active conditions: - -``` -HOSTNAME STATUS CONDITIONS LOAD OS -web-01 Ready HighLoad,MemoryPressure 4.12 Ubuntu 24.04 -web-02 Ready - 0.31 Ubuntu 24.04 -db-01 Ready DiskPressure 1.22 Ubuntu 24.04 -``` - -`agent get` shows full condition details and state timeline: - -``` -Conditions: - MemoryPressure: true (memory 94% used, 15.1/16.0 GB) since 2m ago - HighLoad: true (load 4.12, threshold 4.00 for 2 CPUs) since 5m ago - DiskPressure: false - -Timeline: - TIMESTAMP EVENT HOSTNAME MESSAGE - 2026-03-05 10:00:00 drain web-01 Drain initiated - 2026-03-05 10:05:23 cordoned web-01 All jobs completed - 2026-03-05 12:00:00 undrain web-01 Resumed accepting jobs -``` - -## Agent Drain - -### State Machine - -Agents gain an explicit state field with three values: - -``` -Ready ──(drain)──> Draining ──(jobs done)──> Cordoned - ^ │ - └──────────────(undrain)───────────────────────┘ -``` - -| State | Meaning | -| ---------- | ------------------------------------------------ | -| `Ready` | Accepting and processing jobs (default) | -| `Draining` | Finishing in-flight jobs, not accepting new ones | -| `Cordoned` | Fully drained, idle, not accepting jobs | - -### Mechanism - -1. Operator calls `POST /agent/{hostname}/drain` -2. API writes a `drain.{hostname}` key to the state KV bucket -3. Agent checks for drain key on each heartbeat tick (10s) -4. When drain flag detected: - - Agent transitions state to `Draining` - - Agent unsubscribes from NATS consumer (stops receiving new jobs) - - In-flight jobs continue to completion -5. Once WaitGroup drains (no in-flight jobs), state becomes `Cordoned` -6. `POST /agent/{hostname}/undrain` deletes the drain key -7. Agent detects drain key removal on next heartbeat: - - Transitions state to `Ready` - - Re-subscribes to NATS consumer - -### API Endpoints - -``` -POST /agent/{hostname}/drain # Start draining -POST /agent/{hostname}/undrain # Resume accepting jobs -``` - -Both return 200 on success, 404 if agent not found, 409 if already in the -requested state. - -### Permission - -New `agent:write` permission. Added to the `admin` role by default. - -### Storage - -Agent state transitions are recorded as **append-only events** in the state KV -bucket (`agent-state`, no TTL), following the same pattern used for job status -events (see `WriteStatusEvent` in `internal/job/client/agent.go`). - -Events reuse the existing `TimelineEvent` type (`internal/job/types.go`) — the -same type used for job lifecycle events. This type is generic (Timestamp, Event, -Hostname, Message, Error) and not job-specific: - -``` -Key format: timeline.{sanitized_hostname}.{event}.{unix_nano} -Value: TimelineEvent JSON -``` - -Events: `ready`, `drain`, `cordoned`, `undrain` - -On the SDK side, `TimelineEvent` is promoted from `job_types.go` to a shared -top-level type in `pkg/osapi/types.go`. Both `JobDetail.Timeline` and -`Agent.Timeline` reference the same type. - -Current state is **computed from the latest event**, just like job status is -computed via `computeStatusFromEvents`. This preserves the full transition -history (Ready → Draining → Cordoned → Ready → Draining → ...) and eliminates -race conditions by never updating existing keys. - -The drain intent uses a separate key: `drain.{sanitized_hostname}`. The API -writes this key to signal drain; the agent reads it on heartbeat and writes the -state transition event. The API deletes the key on undrain. - -The `AgentRegistration` also carries the current state for quick reads without -scanning events: - -```go -type AgentRegistration struct { - // ... existing fields ... - State string `json:"state,omitempty"` // Ready, Draining, Cordoned -} -``` - -### CLI Commands - -```bash -osapi client agent drain --hostname web-01 -osapi client agent undrain --hostname web-01 -``` - -`agent list` and `agent get` show the state in the STATUS column. - -## OpenAPI Changes - -### AgentInfo Schema - -Add to existing `AgentInfo`: - -```yaml -state: - type: string - enum: [Ready, Draining, Cordoned] - description: Agent scheduling state. -conditions: - type: array - items: - $ref: '#/components/schemas/NodeCondition' -``` - -New schema: - -```yaml -NodeCondition: - type: object - properties: - type: - type: string - enum: [MemoryPressure, HighLoad, DiskPressure] - status: - type: boolean - reason: - type: string - last_transition_time: - type: string - format: date-time - required: [type, status, last_transition_time] -``` - -### New Endpoints - -```yaml -/agent/{hostname}/drain: - post: - summary: Drain an agent - description: Stop the agent from accepting new jobs. - security: - - BearerAuth: [] - responses: - 200: ... - 404: ... - 409: ... - -/agent/{hostname}/undrain: - post: - summary: Undrain an agent - description: Resume accepting jobs on a drained agent. - security: - - BearerAuth: [] - responses: - 200: ... - 404: ... - 409: ... -``` - -### Permission Updates - -```yaml -# New permission -agent:write - -# Updated admin role -admin: - permissions: - - agent:read - - agent:write # new - - node:read - - ... -``` - -## Implementation Scope - -### Provider Changes - -- Extend heartbeat to collect disk stats (reuse existing `disk.Provider`) -- Add condition evaluation logic to agent heartbeat - -### Agent Changes - -- Add `Condition` type and evaluation functions -- Add state field to `AgentRegistration` -- Add drain flag detection on heartbeat tick -- Add consumer subscribe/unsubscribe for drain/undrain transitions -- Add condition threshold config support - -### API Changes - -- New drain/undrain endpoints in the agent API domain -- Extend `AgentInfo` schema with `state` and `conditions` -- Add `agent:write` permission and wire into scope middleware - -### CLI Changes - -- `agent drain` and `agent undrain` commands -- CONDITIONS column in `agent list` -- Condition details and state timeline in `agent get` -- State shown in STATUS column - -### SDK Changes - -- Promote `TimelineEvent` from `job_types.go` to shared `types.go` -- Both `JobDetail.Timeline` and `Agent.Timeline` use the same type -- Add `Agent.Drain()` and `Agent.Undrain()` methods -- Add conditions, state, and timeline to `Agent` type - -### Config Changes - -- `agent.conditions` section with threshold defaults - -## Testing - -- **Unit**: condition evaluation logic (threshold math, transition tracking), - state machine transitions, drain flag detection -- **HTTP wiring**: drain/undrain endpoints with RBAC (401, 403, 200, 404, 409) -- **Integration**: drain agent → submit job → verify not routed to drained agent - → undrain → verify jobs resume - -## Verification - -```bash -just generate # regenerate specs + code -go build ./... # compiles -just go::unit # tests pass -just go::vet # lint passes -``` diff --git a/docs/plans/2026-03-05-node-conditions-drain.md b/docs/plans/2026-03-05-node-conditions-drain.md deleted file mode 100644 index 209ebcdb9..000000000 --- a/docs/plans/2026-03-05-node-conditions-drain.md +++ /dev/null @@ -1,1387 +0,0 @@ -# Node Conditions & Agent Drain Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Add Kubernetes-inspired node conditions (MemoryPressure, HighLoad, -DiskPressure) and agent drain/cordon lifecycle to OSAPI. - -**Architecture:** Conditions are evaluated agent-side on each heartbeat tick -using existing provider data, stored in AgentRegistration. Drain uses -append-only timeline events in the registry KV bucket (reusing the existing -`TimelineEvent` type from job lifecycle), with a separate drain intent key the -API writes and the agent reads on heartbeat. State transitions trigger NATS -consumer subscribe/unsubscribe. - -**Tech Stack:** Go 1.25, NATS JetStream KV, Echo REST API, OpenAPI codegen, -testify/suite - -**Design Doc:** `docs/plans/2026-03-05-node-conditions-drain-design.md` - ---- - -## Task 1: Add Condition type and evaluation functions - -**Files:** - -- Create: `internal/agent/condition.go` -- Create: `internal/agent/condition_test.go` - -**Step 1: Write the failing tests** - -```go -// internal/agent/condition_test.go -package agent - -import ( - "testing" - "time" - - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/job" - "github.com/osapi-io/osapi/internal/provider/node/disk" - "github.com/osapi-io/osapi/internal/provider/node/load" - "github.com/osapi-io/osapi/internal/provider/node/mem" -) - -type ConditionTestSuite struct { - suite.Suite -} - -func TestConditionTestSuite(t *testing.T) { - suite.Run(t, new(ConditionTestSuite)) -} - -func (s *ConditionTestSuite) TestEvaluateMemoryPressure() { - tests := []struct { - name string - stats *mem.Stats - threshold int - wantStatus bool - wantReason string - }{ - { - name: "above threshold", - stats: &mem.Stats{Total: 16000000000, Used: 15000000000, Free: 1000000000}, - threshold: 90, - wantStatus: true, - }, - { - name: "below threshold", - stats: &mem.Stats{Total: 16000000000, Used: 8000000000, Free: 8000000000}, - threshold: 90, - wantStatus: false, - }, - { - name: "nil stats", - stats: nil, - threshold: 90, - wantStatus: false, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - c := evaluateMemoryPressure(tt.stats, tt.threshold, nil) - s.Equal(tt.wantStatus, c.Status) - s.Equal(job.ConditionMemoryPressure, c.Type) - }) - } -} - -func (s *ConditionTestSuite) TestEvaluateHighLoad() { - tests := []struct { - name string - loadAvg *load.AverageStats - cpuCount int - multiplier float64 - wantStatus bool - }{ - { - name: "above threshold", - loadAvg: &load.AverageStats{OneMin: 5.0}, - cpuCount: 2, - multiplier: 2.0, - wantStatus: true, - }, - { - name: "below threshold", - loadAvg: &load.AverageStats{OneMin: 1.0}, - cpuCount: 2, - multiplier: 2.0, - wantStatus: false, - }, - { - name: "nil load", - loadAvg: nil, - cpuCount: 2, - multiplier: 2.0, - wantStatus: false, - }, - { - name: "zero cpus", - loadAvg: &load.AverageStats{OneMin: 5.0}, - cpuCount: 0, - multiplier: 2.0, - wantStatus: false, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - c := evaluateHighLoad(tt.loadAvg, tt.cpuCount, tt.multiplier, nil) - s.Equal(tt.wantStatus, c.Status) - s.Equal(job.ConditionHighLoad, c.Type) - }) - } -} - -func (s *ConditionTestSuite) TestEvaluateDiskPressure() { - tests := []struct { - name string - disks []disk.UsageStats - threshold int - wantStatus bool - }{ - { - name: "one disk above threshold", - disks: []disk.UsageStats{ - {Name: "/dev/sda1", Total: 100000, Used: 95000, Free: 5000}, - }, - threshold: 90, - wantStatus: true, - }, - { - name: "all disks below threshold", - disks: []disk.UsageStats{ - {Name: "/dev/sda1", Total: 100000, Used: 50000, Free: 50000}, - }, - threshold: 90, - wantStatus: false, - }, - { - name: "nil disks", - disks: nil, - threshold: 90, - wantStatus: false, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - c := evaluateDiskPressure(tt.disks, tt.threshold, nil) - s.Equal(tt.wantStatus, c.Status) - s.Equal(job.ConditionDiskPressure, c.Type) - }) - } -} - -func (s *ConditionTestSuite) TestLastTransitionTimeTracking() { - prev := []job.Condition{{ - Type: job.ConditionMemoryPressure, Status: false, - LastTransitionTime: time.Now().Add(-5 * time.Minute), - }} - // Flip from false -> true: should update LastTransitionTime - c := evaluateMemoryPressure( - &mem.Stats{Total: 100, Used: 95, Free: 5}, 90, prev, - ) - s.True(c.Status) - s.True(c.LastTransitionTime.After(time.Now().Add(-1 * time.Second))) - - // Same status (true -> true): should keep old LastTransitionTime - prev2 := []job.Condition{c} - c2 := evaluateMemoryPressure( - &mem.Stats{Total: 100, Used: 95, Free: 5}, 90, prev2, - ) - s.True(c2.Status) - s.Equal(c.LastTransitionTime, c2.LastTransitionTime) -} -``` - -**Step 2: Run tests to verify they fail** - -Run: `go test -run TestConditionTestSuite -v ./internal/agent/` Expected: FAIL — -`evaluateMemoryPressure` not defined - -**Step 3: Write minimal implementation** - -```go -// internal/agent/condition.go -package agent - -import ( - "fmt" - "time" - - "github.com/osapi-io/osapi/internal/job" - "github.com/osapi-io/osapi/internal/provider/node/disk" - "github.com/osapi-io/osapi/internal/provider/node/load" - "github.com/osapi-io/osapi/internal/provider/node/mem" -) - -// findPrevCondition returns the previous condition of the given type, -// or nil if not found. -func findPrevCondition( - condType string, - prev []job.Condition, -) *job.Condition { - for i := range prev { - if prev[i].Type == condType { - return &prev[i] - } - } - return nil -} - -// transitionTime returns the previous LastTransitionTime if status -// hasn't changed, otherwise returns now. -func transitionTime( - condType string, - newStatus bool, - prev []job.Condition, -) time.Time { - if p := findPrevCondition(condType, prev); p != nil { - if p.Status == newStatus { - return p.LastTransitionTime - } - } - return time.Now() -} - -func evaluateMemoryPressure( - stats *mem.Stats, - threshold int, - prev []job.Condition, -) job.Condition { - c := job.Condition{Type: job.ConditionMemoryPressure} - if stats == nil || stats.Total == 0 { - c.LastTransitionTime = transitionTime(c.Type, false, prev) - return c - } - pct := float64(stats.Used) / float64(stats.Total) * 100 - c.Status = pct > float64(threshold) - if c.Status { - c.Reason = fmt.Sprintf( - "memory %.0f%% used (%.1f/%.1f GB)", - pct, - float64(stats.Used)/1024/1024/1024, - float64(stats.Total)/1024/1024/1024, - ) - } - c.LastTransitionTime = transitionTime(c.Type, c.Status, prev) - return c -} - -func evaluateHighLoad( - loadAvg *load.AverageStats, - cpuCount int, - multiplier float64, - prev []job.Condition, -) job.Condition { - c := job.Condition{Type: job.ConditionHighLoad} - if loadAvg == nil || cpuCount == 0 { - c.LastTransitionTime = transitionTime(c.Type, false, prev) - return c - } - threshold := float64(cpuCount) * multiplier - c.Status = loadAvg.OneMin > threshold - if c.Status { - c.Reason = fmt.Sprintf( - "load %.2f, threshold %.2f for %d CPUs", - loadAvg.OneMin, threshold, cpuCount, - ) - } - c.LastTransitionTime = transitionTime(c.Type, c.Status, prev) - return c -} - -func evaluateDiskPressure( - disks []disk.UsageStats, - threshold int, - prev []job.Condition, -) job.Condition { - c := job.Condition{Type: job.ConditionDiskPressure} - if len(disks) == 0 { - c.LastTransitionTime = transitionTime(c.Type, false, prev) - return c - } - for _, d := range disks { - if d.Total == 0 { - continue - } - pct := float64(d.Used) / float64(d.Total) * 100 - if pct > float64(threshold) { - c.Status = true - c.Reason = fmt.Sprintf( - "%s %.0f%% used (%.1f/%.1f GB)", - d.Name, pct, - float64(d.Used)/1024/1024/1024, - float64(d.Total)/1024/1024/1024, - ) - break - } - } - c.LastTransitionTime = transitionTime(c.Type, c.Status, prev) - return c -} -``` - -**Step 4: Run tests to verify they pass** - -Run: `go test -run TestConditionTestSuite -v ./internal/agent/` Expected: PASS - -**Step 5: Commit** - -```bash -git add internal/agent/condition.go internal/agent/condition_test.go -git commit -m "feat(agent): add condition evaluation functions" -``` - ---- - -## Task 2: Add Condition and State types to job domain - -**Files:** - -- Modify: `internal/job/types.go:273-331` (AgentRegistration, AgentInfo) - -**Step 1: Write the types** - -Add to `internal/job/types.go` after existing types: - -```go -// Condition type constants. -const ( - ConditionMemoryPressure = "MemoryPressure" - ConditionHighLoad = "HighLoad" - ConditionDiskPressure = "DiskPressure" -) - -// Agent state constants. -const ( - AgentStateReady = "Ready" - AgentStateDraining = "Draining" - AgentStateCordoned = "Cordoned" -) - -// Condition represents a node condition evaluated agent-side. -type Condition struct { - Type string `json:"type"` - Status bool `json:"status"` - Reason string `json:"reason,omitempty"` - LastTransitionTime time.Time `json:"last_transition_time"` -} - -``` - -The existing `TimelineEvent` type (line 177) is already generic and will be -reused for agent state transitions — no new event type needed. - -Add fields to `AgentRegistration`: - -```go -Conditions []Condition `json:"conditions,omitempty"` -State string `json:"state,omitempty"` -``` - -Add fields to `AgentInfo`: - -```go -Conditions []Condition `json:"conditions,omitempty"` -State string `json:"state,omitempty"` -Timeline []TimelineEvent `json:"timeline,omitempty"` -``` - -**Step 2: Run existing tests** - -Run: `go test ./internal/job/... -count=1` Expected: PASS (additive change) - -**Step 3: Commit** - -```bash -git add internal/job/types.go -git commit -m "feat(job): add Condition type and agent state constants" -``` - ---- - -## Task 3: Add conditions config to AgentConfig - -**Files:** - -- Modify: `internal/config/types.go:262-277` -- Modify: `configs/osapi.yaml` -- Modify: `configs/osapi.local.yaml` - -**Step 1: Add config struct** - -Add to `internal/config/types.go`: - -```go -// AgentConditions holds threshold configuration for node conditions. -type AgentConditions struct { - MemoryPressureThreshold int `mapstructure:"memory_pressure_threshold"` - HighLoadMultiplier float64 `mapstructure:"high_load_multiplier"` - DiskPressureThreshold int `mapstructure:"disk_pressure_threshold"` -} -``` - -Add field to `AgentConfig`: - -```go -Conditions AgentConditions `mapstructure:"conditions,omitempty"` -``` - -**Step 2: Set defaults in osapi.yaml and osapi.local.yaml** - -```yaml -agent: - conditions: - memory_pressure_threshold: 90 - high_load_multiplier: 2.0 - disk_pressure_threshold: 90 -``` - -**Step 3: Verify compilation** - -Run: `go build ./...` Expected: compiles - -**Step 4: Commit** - -```bash -git add internal/config/types.go configs/osapi.yaml configs/osapi.local.yaml -git commit -m "feat(config): add agent conditions threshold configuration" -``` - ---- - -## Task 4: Add disk stats to heartbeat and evaluate conditions - -**Files:** - -- Modify: `internal/agent/heartbeat.go:88-134` (writeRegistration) -- Modify: `internal/agent/types.go:45-81` (add prevConditions, cpuCount) - -**Step 1: Add fields to Agent struct** - -In `internal/agent/types.go`, add to Agent struct: - -```go -// prevConditions tracks condition state between heartbeats. -prevConditions []job.Condition - -// cpuCount cached from facts for HighLoad evaluation. -cpuCount int -``` - -**Step 2: Extend writeRegistration** - -In `internal/agent/heartbeat.go`, after memory stats collection (~line 111), -add: - -```go -// Collect disk stats (non-fatal). -var diskStats []disk.UsageStats -if stats, err := a.diskProvider.GetLocalUsageStats(); err == nil { - diskStats = stats -} - -// Evaluate conditions. -conditions := []job.Condition{ - evaluateMemoryPressure( - memStats, - a.appConfig.Agent.Conditions.MemoryPressureThreshold, - a.prevConditions, - ), - evaluateHighLoad( - loadAvg, - a.cpuCount, - a.appConfig.Agent.Conditions.HighLoadMultiplier, - a.prevConditions, - ), - evaluateDiskPressure( - diskStats, - a.appConfig.Agent.Conditions.DiskPressureThreshold, - a.prevConditions, - ), -} -a.prevConditions = conditions -``` - -Add `Conditions: conditions` to the `AgentRegistration` literal. - -**Step 3: Set cpuCount from facts** - -In `internal/agent/facts.go` (the `writeFacts` function), after collecting -`CPUCount`, add: - -```go -a.cpuCount = cpuCount -``` - -**Step 4: Run tests** - -Run: `go test ./internal/agent/... -count=1` Expected: PASS - -**Step 5: Commit** - -```bash -git add internal/agent/heartbeat.go internal/agent/types.go internal/agent/facts.go -git commit -m "feat(agent): evaluate node conditions on heartbeat tick" -``` - ---- - -## Task 5: Add drain timeline event storage functions - -**Files:** - -- Modify: `internal/job/client/agent.go:39-85` -- Create: `internal/job/client/agent_timeline_test.go` - -**Step 1: Write failing tests** - -```go -// internal/job/client/agent_timeline_test.go -package client_test - -// Test WriteAgentTimelineEvent writes append-only key to registryKV. -// Test ComputeAgentState returns latest state from timeline events. -// Test GetAgentTimeline returns sorted timeline events. -``` - -Table-driven tests: - -- `WriteAgentTimelineEvent` writes key like - `timeline.{hostname}.{event}.{unix_nano}` -- `ComputeAgentState` with no events returns "Ready" -- `ComputeAgentState` with drain event returns "Draining" -- `ComputeAgentState` with cordoned event returns "Cordoned" -- `ComputeAgentState` with undrain event returns "Ready" - -**Step 2: Run tests to verify they fail** - -Run: `go test -run TestAgentTimeline -v ./internal/job/client/` Expected: FAIL - -**Step 3: Implement** - -Add to `internal/job/client/agent.go`: - -```go -// WriteAgentTimelineEvent writes an append-only timeline event -// for an agent state transition. Reuses the same TimelineEvent -// type used by job lifecycle events. -func (c *Client) WriteAgentTimelineEvent( - _ context.Context, - hostname, event, message string, -) error { - now := time.Now() - key := fmt.Sprintf( - "timeline.%s.%s.%d", - job.SanitizeHostname(hostname), - event, - now.UnixNano(), - ) - data, _ := json.Marshal(job.TimelineEvent{ - Timestamp: now, - Event: event, - Hostname: hostname, - Message: message, - }) - _, err := c.registryKV.Put(key, data) - return err -} - -// GetAgentTimeline returns sorted timeline events for a hostname. -func (c *Client) GetAgentTimeline( - ctx context.Context, - hostname string, -) ([]job.TimelineEvent, error) { - prefix := "timeline." + job.SanitizeHostname(hostname) + "." - // List keys with prefix, unmarshal, sort by Timestamp - // Return sorted events -} - -// ComputeAgentState returns the current state from timeline events. -func ComputeAgentState( - events []job.TimelineEvent, -) string { - if len(events) == 0 { - return job.AgentStateReady - } - latest := events[len(events)-1] - switch latest.Event { - case "drain": - return job.AgentStateDraining - case "cordoned": - return job.AgentStateCordoned - case "undrain", "ready": - return job.AgentStateReady - default: - return job.AgentStateReady - } -} -``` - -Add `WriteAgentTimelineEvent`, `GetAgentTimeline` to the `JobClient` interface -in `internal/job/client/types.go`. Regenerate mocks. - -**Step 4: Run tests** - -Run: `go test -run TestAgentTimeline -v ./internal/job/client/` Expected: PASS - -**Step 5: Commit** - -```bash -git add internal/job/client/agent.go internal/job/client/agent_timeline_test.go \ - internal/job/client/types.go internal/job/client/mock_*.go -git commit -m "feat(job): add append-only timeline events for agent drain" -``` - ---- - -## Task 6: Add drain/undrain API endpoints - -**Files:** - -- Modify: `internal/api/agent/gen/api.yaml` -- Create: `internal/api/agent/agent_drain.go` -- Create: `internal/api/agent/agent_drain_public_test.go` - -**Step 1: Add to OpenAPI spec** - -Add to `internal/api/agent/gen/api.yaml`: - -```yaml -/agent/{hostname}/drain: - post: - operationId: drainAgent - summary: Drain an agent - description: Stop the agent from accepting new jobs. - security: - - BearerAuth: - - 'agent:write' - parameters: - - name: hostname - in: path - required: true - schema: - type: string - responses: - '200': - description: Agent drain initiated. - content: - application/json: - schema: - type: object - properties: - message: - type: string - '404': - description: Agent not found. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' - '409': - description: Agent already in requested state. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' - -/agent/{hostname}/undrain: - post: - operationId: undrainAgent - summary: Undrain an agent - description: Resume accepting jobs on a drained agent. - security: - - BearerAuth: - - 'agent:write' - parameters: - - name: hostname - in: path - required: true - schema: - type: string - responses: - '200': - description: Agent undrain initiated. - content: - application/json: - schema: - type: object - properties: - message: - type: string - '404': - description: Agent not found. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' - '409': - description: Agent not in draining/cordoned state. - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorResponse' -``` - -Add `agent:write` to BearerAuth scopes. Add `state` and `conditions` fields to -`AgentInfo` schema. Add `NodeCondition` schema. - -Run: `just generate` to regenerate `*.gen.go`. - -**Step 2: Write failing tests** - -```go -// internal/api/agent/agent_drain_public_test.go -// Table-driven tests for DrainAgent and UndrainAgent: -// - 200: agent found and drain initiated -// - 404: agent not found -// - 409: already draining/cordoned -// - HTTP wiring: RBAC (401, 403 without agent:write, 200 with agent:write) -``` - -**Step 3: Implement handlers** - -```go -// internal/api/agent/agent_drain.go -package agent - -func (a *Agent) DrainAgent( - ctx context.Context, - request gen.DrainAgentRequestObject, -) (gen.DrainAgentResponseObject, error) { - hostname := request.Hostname - - // 1. Verify agent exists - agentInfo, err := a.JobClient.GetAgent(ctx, hostname) - if err != nil { - return gen.DrainAgent404JSONResponse{...}, nil - } - - // 2. Check not already draining - if agentInfo.State == job.AgentStateDraining || - agentInfo.State == job.AgentStateCordoned { - return gen.DrainAgent409JSONResponse{...}, nil - } - - // 3. Write drain intent key - // 4. Write state event - return gen.DrainAgent200JSONResponse{...}, nil -} - -func (a *Agent) UndrainAgent( - ctx context.Context, - request gen.UndrainAgentRequestObject, -) (gen.UndrainAgentResponseObject, error) { - // Similar: verify exists, check state, delete drain key, write event -} -``` - -**Step 4: Run tests** - -Run: `go test ./internal/api/agent/... -count=1` Expected: PASS - -**Step 5: Commit** - -```bash -git add internal/api/agent/gen/api.yaml internal/api/agent/gen/*.gen.go \ - internal/api/agent/agent_drain.go internal/api/agent/agent_drain_public_test.go -git commit -m "feat(api): add drain/undrain endpoints with RBAC" -``` - ---- - -## Task 7: Add agent:write permission - -**Files:** - -- Modify: `internal/authtoken/permissions.go:27-37` (add constant) -- Modify: `internal/authtoken/permissions.go:53-81` (add to admin role) - -**Step 1: Add permission constant** - -```go -PermAgentWrite Permission = "agent:write" -``` - -**Step 2: Add to admin role** - -In `DefaultRolePermissions`, add `PermAgentWrite` to the `admin` slice. - -**Step 3: Run tests** - -Run: `go test ./internal/authtoken/... -count=1` Expected: PASS - -**Step 4: Commit** - -```bash -git add internal/authtoken/permissions.go -git commit -m "feat(auth): add agent:write permission for drain operations" -``` - ---- - -## Task 8: Wire drain endpoints into server - -**Files:** - -- Modify: `internal/api/handler_agent.go:34-61` -- Modify: `internal/api/handler_agent_public_test.go` - -**Step 1: Update handler registration** - -The `GetAgentHandler` already wires all agent gen handlers through -`scopeMiddleware`. After regenerating the OpenAPI code (Task 6), the new -`DrainAgent` and `UndrainAgent` methods on the strict server interface will be -picked up automatically by `RegisterHandlers`. - -No code change needed in `handler_agent.go` unless `unauthenticatedOperations` -needs updating (it doesn't — drain requires auth). - -**Step 2: Verify compilation** - -Run: `go build ./...` Expected: compiles - -**Step 3: Add handler test cases** - -Add test cases to `handler_agent_public_test.go` for drain/undrain handler -registration. - -**Step 4: Commit** - -```bash -git add internal/api/handler_agent.go internal/api/handler_agent_public_test.go -git commit -m "feat(api): wire drain/undrain handlers into server" -``` - ---- - -## Task 9: Add drain detection to agent heartbeat - -**Files:** - -- Modify: `internal/agent/heartbeat.go:88-134` -- Modify: `internal/agent/server.go:32-61` -- Create: `internal/agent/drain.go` -- Create: `internal/agent/drain_test.go` - -**Step 1: Write failing tests** - -```go -// internal/agent/drain_test.go -// Test checkDrainFlag: returns true when drain key exists -// Test checkDrainFlag: returns false when drain key absent -// Test handleDrainTransition: unsubscribes consumers when draining -// Test handleUndrainTransition: resubscribes consumers when undrained -``` - -**Step 2: Implement drain detection** - -```go -// internal/agent/drain.go -package agent - -// checkDrainFlag reads drain.{hostname} from registryKV. -func (a *Agent) checkDrainFlag( - ctx context.Context, - hostname string, -) bool { - key := "drain." + job.SanitizeHostname(hostname) - _, err := a.registryKV.Get(ctx, key) - return err == nil -} - -// handleDrainDetection checks drain flag on each heartbeat. -func (a *Agent) handleDrainDetection( - ctx context.Context, - hostname string, -) { - drainRequested := a.checkDrainFlag(ctx, hostname) - - switch { - case drainRequested && a.state == job.AgentStateReady: - a.state = job.AgentStateDraining - a.unsubscribeConsumers() - // Write timeline event: "drain", "Drain initiated" - // When WaitGroup drains, transition to Cordoned - - case !drainRequested && a.state == job.AgentStateCordoned: - a.state = job.AgentStateReady - a.resubscribeConsumers(ctx, hostname) - // Write timeline event: "undrain", "Resumed accepting jobs" - } -} -``` - -**Step 3: Add state field to Agent struct** - -In `internal/agent/types.go`: - -```go -state string // Ready, Draining, Cordoned -``` - -Initialize to `job.AgentStateReady` in `Start()`. - -**Step 4: Call from heartbeat** - -In `writeRegistration()`, add `a.handleDrainDetection(ctx, hostname)` and -include `State: a.state` in the registration. - -**Step 5: Run tests** - -Run: `go test ./internal/agent/... -count=1` Expected: PASS - -**Step 6: Commit** - -```bash -git add internal/agent/drain.go internal/agent/drain_test.go \ - internal/agent/heartbeat.go internal/agent/types.go internal/agent/server.go -git commit -m "feat(agent): detect drain flag and manage consumer lifecycle" -``` - ---- - -## Task 10: Extend buildAgentInfo with conditions and state - -**Files:** - -- Modify: `internal/api/agent/agent_list.go:59-171` (buildAgentInfo) -- Modify: `internal/api/agent/agent_list_public_test.go` -- Modify: `internal/job/client/query.go:479-493` (agentInfoFromRegistration) - -**Step 1: Update agentInfoFromRegistration** - -Add to the returned `AgentInfo`: - -```go -Conditions: reg.Conditions, -State: reg.State, -``` - -**Step 2: Update buildAgentInfo** - -Map conditions and state from `job.AgentInfo` to `gen.AgentInfo`: - -```go -if len(a.Conditions) > 0 { - conditions := make([]gen.NodeCondition, 0, len(a.Conditions)) - for _, c := range a.Conditions { - nc := gen.NodeCondition{ - Type: gen.NodeConditionType(c.Type), - Status: c.Status, - LastTransitionTime: c.LastTransitionTime, - } - if c.Reason != "" { - nc.Reason = &c.Reason - } - conditions = append(conditions, nc) - } - info.Conditions = &conditions -} - -if a.State != "" { - state := gen.AgentInfoState(a.State) - info.State = &state -} -``` - -**Step 3: Update status derivation** - -Change status logic: if `a.State` is set, use it; otherwise default to `Ready` -(existing behavior). - -**Step 4: Add test cases** - -Add table-driven test case for agent with conditions and Draining/Cordoned -states. - -**Step 5: Run tests** - -Run: `go test ./internal/api/agent/... -count=1` Expected: PASS - -**Step 6: Commit** - -```bash -git add internal/api/agent/agent_list.go internal/api/agent/agent_list_public_test.go \ - internal/job/client/query.go -git commit -m "feat(api): expose conditions and state in agent responses" -``` - ---- - -## Task 11: Add timeline to GetAgent response - -**Files:** - -- Modify: `internal/job/client/query.go:423-445` (GetAgent) -- Modify: `internal/job/client/query_public_test.go` - -**Step 1: Extend GetAgent to fetch timeline events** - -After building `AgentInfo`, fetch timeline events: - -```go -timeline, err := c.GetAgentTimeline(ctx, hostname) -if err == nil { - info.Timeline = timeline -} -``` - -**Step 2: Add test cases** - -Test GetAgent returns timeline events when present. - -**Step 3: Run tests** - -Run: `go test ./internal/job/client/... -count=1` Expected: PASS - -**Step 4: Commit** - -```bash -git add internal/job/client/query.go internal/job/client/query_public_test.go -git commit -m "feat(job): include timeline events in GetAgent response" -``` - ---- - -## Task 12: Update SDK with conditions, state, drain/undrain - -**Files:** - -- Modify: `osapi-sdk/pkg/osapi/gen/agent/api.yaml` (copy from osapi) -- Modify: `osapi-sdk/pkg/osapi/agent.go` (add Drain, Undrain methods) -- Modify: `osapi-sdk/pkg/osapi/agent_types.go` (add conditions, state, timeline - to Agent type) -- Create: `osapi-sdk/pkg/osapi/types.go` (promote TimelineEvent to shared type) -- Modify: `osapi-sdk/pkg/osapi/job_types.go` (remove TimelineEvent, import from - types.go) - -**Step 1: Promote TimelineEvent to shared type** - -Move `TimelineEvent` from `job_types.go` to a new `types.go`: - -```go -// pkg/osapi/types.go - -// TimelineEvent represents a lifecycle event. Used by both job -// timelines and agent state transition history. -type TimelineEvent struct { - Timestamp string - Event string - Hostname string - Message string - Error string -} -``` - -Update `job_types.go` to remove the `TimelineEvent` definition — -`JobDetail.Timeline` now references the shared type. - -**Step 2: Sync OpenAPI spec** - -Copy `internal/api/agent/gen/api.yaml` to -`osapi-sdk/pkg/osapi/gen/agent/api.yaml`. - -Run `redocly join` + `go generate` in the SDK. - -**Step 3: Add domain types** - -```go -// In agent_types.go -type Agent struct { - // ... existing fields ... - State string - Conditions []Condition - Timeline []TimelineEvent // shared type from types.go -} - -type Condition struct { - Type string - Status bool - Reason string - LastTransitionTime time.Time -} -``` - -**Step 4: Add Drain/Undrain methods** - -```go -func (s *AgentService) Drain( - ctx context.Context, - hostname string, -) (*Response[any], error) { - // POST /agent/{hostname}/drain -} - -func (s *AgentService) Undrain( - ctx context.Context, - hostname string, -) (*Response[any], error) { - // POST /agent/{hostname}/undrain -} -``` - -**Step 4: Run SDK tests** - -Run: `go test ./pkg/osapi/... -count=1` Expected: PASS - -**Step 5: Commit (in osapi-sdk repo)** - -```bash -git add pkg/osapi/ -git commit -m "feat(agent): add conditions, state, drain/undrain support" -``` - ---- - -## Task 13: Add CONDITIONS column to agent list CLI - -**Files:** - -- Modify: `cmd/client_agent_list.go` - -**Step 1: Add CONDITIONS column** - -In the table builder for `agent list`, add a column that joins active condition -type names: - -```go -conditions := "-" -if len(agent.Conditions) > 0 { - active := make([]string, 0) - for _, c := range agent.Conditions { - if c.Status { - active = append(active, c.Type) - } - } - if len(active) > 0 { - conditions = strings.Join(active, ",") - } -} -``` - -Headers: `HOSTNAME`, `STATUS`, `CONDITIONS`, `LABELS`, `AGE`, `LOAD`, `OS` - -**Step 2: Use State for STATUS column** - -Replace hardcoded "Ready" with `agent.State` (defaulting to "Ready" if empty). - -**Step 3: Run `go build ./cmd/...`** - -Expected: compiles - -**Step 4: Commit** - -```bash -git add cmd/client_agent_list.go -git commit -m "feat(cli): add CONDITIONS column and state to agent list" -``` - ---- - -## Task 14: Add conditions and timeline to agent get CLI - -**Files:** - -- Modify: `cmd/client_agent_get.go:58-141` - -**Step 1: Add state to agent get output** - -After the Status KV line, display the State: - -```go -if data.State != "" && data.State != "Ready" { - cli.PrintKV("State", data.State) -} -``` - -**Step 2: Add conditions section** - -```go -if len(data.Conditions) > 0 { - condRows := make([][]string, 0, len(data.Conditions)) - for _, c := range data.Conditions { - status := "false" - if c.Status { - status = "true" - } - reason := "" - if c.Reason != "" { - reason = c.Reason - } - since := cli.FormatAge(time.Since(c.LastTransitionTime)) + " ago" - condRows = append(condRows, []string{c.Type, status, reason, since}) - } - sections = append(sections, cli.Section{ - Title: "Conditions", - Headers: []string{"TYPE", "STATUS", "REASON", "SINCE"}, - Rows: condRows, - }) -} -``` - -**Step 3: Add timeline section** - -Same pattern as `DisplayJobDetail` in `internal/cli/ui.go:600-615`: - -```go -if len(data.Timeline) > 0 { - timelineRows := make([][]string, 0, len(data.Timeline)) - for _, te := range data.Timeline { - timelineRows = append(timelineRows, []string{ - te.Timestamp, te.Event, te.Hostname, te.Message, te.Error, - }) - } - sections = append(sections, cli.Section{ - Title: "Timeline", - Headers: []string{"TIMESTAMP", "EVENT", "HOSTNAME", "MESSAGE", "ERROR"}, - Rows: timelineRows, - }) -} -``` - -**Step 4: Run `go build ./cmd/...`** - -Expected: compiles - -**Step 5: Commit** - -```bash -git add cmd/client_agent_get.go -git commit -m "feat(cli): display conditions and timeline in agent get" -``` - ---- - -## Task 15: Add agent drain/undrain CLI commands - -**Files:** - -- Create: `cmd/client_agent_drain.go` -- Create: `cmd/client_agent_undrain.go` - -**Step 1: Create drain command** - -```go -// cmd/client_agent_drain.go -var clientAgentDrainCmd = &cobra.Command{ - Use: "drain", - Short: "Drain an agent", - Long: `Stop an agent from accepting new jobs. In-flight jobs complete.`, - Run: func(cmd *cobra.Command, _ []string) { - ctx := cmd.Context() - hostname, _ := cmd.Flags().GetString("hostname") - - resp, err := sdkClient.Agent.Drain(ctx, hostname) - if err != nil { - cli.HandleError(err, logger) - return - } - - if jsonOutput { - fmt.Println(string(resp.RawJSON())) - return - } - - fmt.Printf("Agent %s drain initiated\n", hostname) - }, -} -``` - -**Step 2: Create undrain command** - -Similar pattern for `undrain`. - -**Step 3: Register commands** - -```go -func init() { - clientAgentCmd.AddCommand(clientAgentDrainCmd) - clientAgentDrainCmd.Flags().String("hostname", "", "Hostname of the agent to drain") - _ = clientAgentDrainCmd.MarkFlagRequired("hostname") -} -``` - -**Step 4: Run `go build ./cmd/...`** - -Expected: compiles - -**Step 5: Commit** - -```bash -git add cmd/client_agent_drain.go cmd/client_agent_undrain.go -git commit -m "feat(cli): add agent drain and undrain commands" -``` - ---- - -## Task 16: Update documentation - -**Files:** - -- Modify: `docs/docs/sidebar/features/agent-management.md` (or create) -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/usage/cli/client/agent/` - -**Step 1: Add conditions and drain docs** - -Document: - -- Condition types and thresholds -- Drain lifecycle (Ready → Draining → Cordoned) -- CLI commands (`agent drain`, `agent undrain`) -- Configuration section for `agent.conditions` - -**Step 2: Update permission table** - -Add `agent:write` to the permissions table in configuration.md. - -**Step 3: Commit** - -```bash -git add docs/ -git commit -m "docs: add node conditions and agent drain documentation" -``` - ---- - -## Task 17: Final verification - -**Step 1: Regenerate** - -Run: `just generate` Expected: no diff - -**Step 2: Build** - -Run: `go build ./...` Expected: compiles - -**Step 3: Unit tests** - -Run: `just go::unit` Expected: PASS - -**Step 4: Lint** - -Run: `just go::vet` Expected: clean - -**Step 5: Coverage check** - -Run: -`go test -coverprofile=coverage.out ./internal/agent/... ./internal/job/client/... ./internal/api/agent/...` -Expected: condition.go, drain.go, agent_drain.go at 100% - ---- - -## Verification - -```bash -just generate # regenerate specs + code -go build ./... # compiles -just go::unit # tests pass -just go::vet # lint passes -``` diff --git a/docs/plans/2026-03-06-file-deploy-template-design.md b/docs/plans/2026-03-06-file-deploy-template-design.md deleted file mode 100644 index e9cf012af..000000000 --- a/docs/plans/2026-03-06-file-deploy-template-design.md +++ /dev/null @@ -1,275 +0,0 @@ -# File Deploy & Template Rendering Design - -## Context - -OSAPI manages system configuration through async jobs. Current operations (DNS, -disk, memory, commands) send small JSON payloads through NATS KV. File -management — deploying config files, rendering templates with per-host facts — -requires transferring larger blobs and tracking deployed state for idempotency. - -Ansible's approach transfers the full file every run to verify whether it -changed. We want SHA-based idempotency: compute the hash of what should be on -disk, compare against what was last deployed, and skip the transfer when nothing -changed. - -## Goals - -- Upload files to a central store (NATS Object Store) via the REST API -- Deploy files to agent hosts with mode, owner, and group control -- Render Go `text/template` files agent-side using live facts + user vars -- SHA-based idempotency — skip transfer when content hasn't changed -- Report `changed: true/false` so orchestrator guards (`OnlyIfChanged`) work -- **Shared primitive** — the Object Store layer is reusable by future providers - (firmware, packages, certs, scripts), not tied to the file provider - -## Design Decisions - -- **Approach B: single operation with `content_type` flag.** One `file.deploy` - operation. A `content_type` field (`raw` or `template`) controls whether the - agent renders content before writing. SHA is computed on the **rendered - output** for templates, so fact changes trigger redeployment. -- **NATS Object Store** for blob storage. It handles chunking automatically (KV - has a ~1MB value limit). Files are uploaded once and pulled by agents on - demand. -- **Dedicated `file-state` KV bucket** for SHA tracking. Keyed by - `.`. No TTL — deployed state persists until - explicitly removed. Separate from `agent-state` to keep concerns clean. - Visible to the API server for fleet-wide deployment status. -- **Agent-side template rendering.** Raw Go template stored in Object Store. - Agent renders locally using its cached facts + user-supplied vars. Consistent - with how `@fact.*` resolution works today — each host gets its own output. -- **Mode + owner/group in job params.** Agent sets permissions after writing. - Defaults to umask/current user when not specified. - -## Architecture - -### Shared Object Store Primitive - -The Object Store client is a **shared agent dependency** — injected at startup -like `execManager`, `hostProvider`, or `factsKV`. Any provider can use it to -pull blobs. - -``` -┌─────────────────────────────────┐ -│ Object Store │ ← shared NATS resource -│ (file-objects bucket) │ -└──────────┬──────────────────────┘ - │ - ┌─────┴──────┐ - │ Agent │ - │ .objStore │ ← injected handle - └─────┬──────┘ - │ - ┌────────┼────────────┬───────────────┐ - │ │ │ │ -file firmware package cert -provider provider provider provider -(now) (future) (future) (future) -``` - -Future providers that would consume the Object Store: - -| Provider | Operation | Usage | -| ----------------- | ---------------------------------------------- | ----- | -| `firmware.update` | Pull binary, run flash tool | -| `package.install` | Pull `.deb`/`.rpm`, install via `dpkg`/`rpm` | -| `cert.deploy` | Pull TLS cert/key, write with restricted perms | -| `script.run` | Pull script file, execute with args | - -Each provider owns its domain logic but shares: Object Store download, SHA -comparison, and state tracking from the `file-state` KV bucket. - -### Data Flow - -**Upload phase** (new REST endpoint): - -1. Client sends file content via `POST /file` with metadata (name) -2. API server stores content in NATS Object Store (`file-objects`) -3. Returns object reference: `{name, sha256, size}` - -**Deploy phase** (job system — `file.deploy` operation): - -1. Client creates job with `file.deploy` targeting host(s) -2. Job data: object name, destination path, mode, owner, group, content_type, - optional template vars -3. Agent pulls object from Object Store -4. If `content_type: "template"` — renders with Go `text/template` -5. Computes SHA of final content (rendered or raw) -6. Checks `file-state` KV — if SHA matches, returns `changed: false` -7. If different — writes file, sets perms, updates state KV, returns - `changed: true` - -**Status check** (read-only — `file.status` operation): - -1. Agent reads local file SHA, compares against `file-state` KV -2. Reports: in-sync, drifted, or missing - -## Data Structures - -### NATS Configuration - -```yaml -nats: - objects: - bucket: 'file-objects' - max_bytes: 524288000 # 500 MiB - storage: 'file' - replicas: 1 - - file_state: - bucket: 'file-state' - storage: 'file' - replicas: 1 - # No TTL — deployed file state persists -``` - -### File State KV Entry - -Keyed by `.`: - -```json -{ - "object_name": "nginx.conf", - "path": "/etc/nginx/nginx.conf", - "sha256": "abc123...", - "mode": "0644", - "owner": "root", - "group": "root", - "deployed_at": "2026-03-06T...", - "content_type": "raw" -} -``` - -### Job Request Data (`file.deploy`) - -```json -{ - "object_name": "nginx.conf", - "path": "/etc/nginx/nginx.conf", - "mode": "0644", - "owner": "root", - "group": "root", - "content_type": "template", - "vars": { - "worker_count": 4, - "upstream": "10.0.0.5" - } -} -``` - -### Template Rendering Context - -```go -type TemplateContext struct { - Facts *job.FactsRegistration - Vars map[string]any - Hostname string -} -``` - -Example template: - -``` -worker_processes {{ .Vars.worker_count }}; -# Running on {{ .Hostname }} ({{ .Facts.Architecture }}) -server {{ .Vars.upstream }}:{{ if eq .Facts.Architecture "arm64" }}8081{{ else }}8080{{ end }}; -``` - -## API Endpoints - -| Method | Path | Permission | Description | -| --------------------- | ------------ | --------------------------- | ----------- | -| `POST /file` | `file:write` | Upload file to Object Store | -| `GET /file` | `file:read` | List stored objects | -| `GET /file/{name}` | `file:read` | Get object metadata | -| `DELETE /file/{name}` | `file:write` | Remove stored object | - -Deploy and status go through the existing job system as `file.deploy` and -`file.status` operations. No new job endpoints needed. - -### Permissions - -New permissions: `file:read`, `file:write`. Added to `admin` and `write` -built-in roles. - -## Agent-Side Architecture - -The agent gets two new dependencies: - -- **`objectStore`** — NATS Object Store handle. Any provider can use it. -- **`fileStateKV`** — dedicated KV for tracking deployed file SHAs. - -The file provider implements: - -- `Deploy(req) → (Result, error)` — pull from Object Store, optionally render - template, SHA compare, write file, set perms, update state -- `Status(req) → (Result, error)` — read-only: compare local file SHA against - state KV - -The processor dispatch adds a `file` category alongside `node`, `network`, -`command`. - -## SDK & Orchestrator Integration - -### SDK (`osapi-sdk`) - -New `FileService`: - -- `Upload(ctx, name, content)` — upload to Object Store -- `List(ctx)` — list stored objects -- `Get(ctx, name)` — get object metadata -- `Delete(ctx, name)` — remove object - -Deploy uses existing `Job.Create()` with operation `file.deploy`. - -### Orchestrator (`osapi-orchestrator`) - -```go -o := orchestrator.New(client) - -upload := o.FileUpload("nginx.conf", "./local/nginx.conf.tmpl") -deploy := o.FileTemplate("_all", "nginx.conf", "/etc/nginx/nginx.conf", - map[string]any{"worker_count": 4}, - orchestrator.WithMode("0644"), - orchestrator.WithOwner("root", "root"), -).After(upload) - -reload := o.CommandExec("_all", "nginx", []string{"-s", "reload"}). - After(deploy). - OnlyIfChanged() -``` - -- `FileDeploy()` — raw file deploy step -- `FileTemplate()` — deploy with `content_type: "template"` -- `OnlyIfChanged` works naturally via `changed` response field -- Template vars support `@fact.*` references (resolved agent-side) - -## Verification - -After implementation: - -```bash -# Upload a file -osapi client file upload --name nginx.conf --file ./nginx.conf - -# Deploy raw file -osapi client node file deploy \ - --object nginx.conf \ - --path /etc/nginx/nginx.conf \ - --mode 0644 --owner root --group root \ - --target _all - -# Deploy template -osapi client node file deploy \ - --object nginx.conf.tmpl \ - --path /etc/nginx/nginx.conf \ - --content-type template \ - --var worker_count=4 \ - --mode 0644 --owner root --group root \ - --target _all - -# Check status (idempotent re-run should show changed: false) -osapi client node file status \ - --path /etc/nginx/nginx.conf \ - --target _all -``` diff --git a/docs/plans/2026-03-06-file-deploy-template.md b/docs/plans/2026-03-06-file-deploy-template.md deleted file mode 100644 index cd473b648..000000000 --- a/docs/plans/2026-03-06-file-deploy-template.md +++ /dev/null @@ -1,1983 +0,0 @@ -# File Deploy & Template Rendering Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Add file management (upload/list/get/delete via Object Store), file -deployment with SHA-based idempotency, and Go template rendering with per-host -facts. - -**Architecture:** NATS Object Store as shared blob storage, dedicated -`file-state` KV for SHA tracking, single `file.deploy` job operation with -`content_type` flag (raw/template), agent-side `text/template` rendering. Object -Store is a shared primitive — future providers (firmware, packages, certs) reuse -the same infrastructure. - -**Tech Stack:** Go 1.25, NATS JetStream Object Store, `text/template`, -oapi-codegen, testify/suite, gomock. - -**Design doc:** `docs/plans/2026-03-06-file-deploy-template-design.md` - ---- - -## Prerequisites - -### nats-client Object Store Support - -The `github.com/osapi-io/nats-client` package needs Object Store methods before -this plan can start. Add to the nats-client repo: - -```go -// In pkg/client/types.go or new objectstore.go -func (c *Client) CreateOrUpdateObjectStore( - ctx context.Context, - cfg jetstream.ObjectStoreConfig, -) (jetstream.ObjectStore, error) - -func (c *Client) ObjectStore( - ctx context.Context, - name string, -) (jetstream.ObjectStore, error) -``` - -Then update `internal/messaging/types.go` in osapi to add: - -```go -CreateOrUpdateObjectStore( - ctx context.Context, - cfg jetstream.ObjectStoreConfig, -) (jetstream.ObjectStore, error) - -ObjectStore( - ctx context.Context, - name string, -) (jetstream.ObjectStore, error) -``` - -This is a separate PR on the nats-client repo. Once merged, `go get` the new -version before starting Task 1. - ---- - -## Task 1: NATS Configuration for Object Store + File-State KV - -Add config structs, builder functions, and startup creation for the two new NATS -resources. - -**Files:** - -- Modify: `internal/config/types.go` -- Modify: `internal/cli/nats.go` -- Modify: `internal/cli/nats_public_test.go` -- Modify: `cmd/nats_helpers.go` -- Modify: `internal/messaging/types.go` -- Modify: `docs/docs/sidebar/usage/configuration.md` - -### Step 1: Add config structs - -In `internal/config/types.go`, add two new types and fields to `NATS`: - -```go -// NATSObjects configuration for the NATS Object Store bucket. -type NATSObjects struct { - // Bucket is the Object Store bucket name for file content. - Bucket string `mapstructure:"bucket"` - MaxBytes int64 `mapstructure:"max_bytes"` - Storage string `mapstructure:"storage"` // "file" or "memory" - Replicas int `mapstructure:"replicas"` -} - -// NATSFileState configuration for the file deployment state KV bucket. -// No TTL — deployed file state persists until explicitly removed. -type NATSFileState struct { - // Bucket is the KV bucket name for file deployment SHA tracking. - Bucket string `mapstructure:"bucket"` - Storage string `mapstructure:"storage"` // "file" or "memory" - Replicas int `mapstructure:"replicas"` -} -``` - -Add to `NATS` struct: - -```go -type NATS struct { - // ... existing fields ... - Objects NATSObjects `mapstructure:"objects,omitempty"` - FileState NATSFileState `mapstructure:"file_state,omitempty"` -} -``` - -### Step 2: Add NATSClient Object Store methods - -In `internal/messaging/types.go`, add to the `NATSClient` interface: - -```go -// Object Store operations -CreateOrUpdateObjectStore( - ctx context.Context, - cfg jetstream.ObjectStoreConfig, -) (jetstream.ObjectStore, error) -ObjectStore( - ctx context.Context, - name string, -) (jetstream.ObjectStore, error) -``` - -### Step 3: Add builder functions - -In `internal/cli/nats.go`, add: - -```go -// BuildObjectStoreConfig builds a jetstream.ObjectStoreConfig from -// objects config values. -func BuildObjectStoreConfig( - namespace string, - objectsCfg config.NATSObjects, -) jetstream.ObjectStoreConfig { - bucket := job.ApplyNamespaceToInfraName(namespace, objectsCfg.Bucket) - - return jetstream.ObjectStoreConfig{ - Bucket: bucket, - MaxBytes: objectsCfg.MaxBytes, - Storage: ParseJetstreamStorageType(objectsCfg.Storage), - Replicas: objectsCfg.Replicas, - } -} - -// BuildFileStateKVConfig builds a jetstream.KeyValueConfig from -// file state config values. No TTL — deployed state persists. -func BuildFileStateKVConfig( - namespace string, - fileStateCfg config.NATSFileState, -) jetstream.KeyValueConfig { - bucket := job.ApplyNamespaceToInfraName(namespace, fileStateCfg.Bucket) - - return jetstream.KeyValueConfig{ - Bucket: bucket, - Storage: ParseJetstreamStorageType(fileStateCfg.Storage), - Replicas: fileStateCfg.Replicas, - } -} -``` - -### Step 4: Add startup creation - -In `cmd/nats_helpers.go` `setupJetStream()`, add after the state KV block: - -```go -// Create Object Store bucket for file content -if appConfig.NATS.Objects.Bucket != "" { - objStoreConfig := cli.BuildObjectStoreConfig(namespace, appConfig.NATS.Objects) - if _, err := nc.CreateOrUpdateObjectStore(ctx, objStoreConfig); err != nil { - return fmt.Errorf("create Object Store bucket %s: %w", objStoreConfig.Bucket, err) - } -} - -// Create file-state KV bucket for deployment SHA tracking -if appConfig.NATS.FileState.Bucket != "" { - fileStateKVConfig := cli.BuildFileStateKVConfig(namespace, appConfig.NATS.FileState) - if _, err := nc.CreateOrUpdateKVBucketWithConfig(ctx, fileStateKVConfig); err != nil { - return fmt.Errorf("create file-state KV bucket %s: %w", fileStateKVConfig.Bucket, err) - } -} -``` - -### Step 5: Add default config values - -Add to `osapi.yaml` and the configuration docs the new sections: - -```yaml -nats: - objects: - bucket: 'file-objects' - max_bytes: 524288000 # 500 MiB - storage: 'file' - replicas: 1 - - file_state: - bucket: 'file-state' - storage: 'file' - replicas: 1 -``` - -### Step 6: Run tests and verify - -```bash -go build ./... -just go::unit -``` - -### Step 7: Commit - -```bash -git add internal/config/types.go internal/cli/nats.go \ - internal/messaging/types.go cmd/nats_helpers.go -git commit -m "feat(config): add Object Store and file-state KV config" -``` - ---- - -## Task 2: Permissions — Add file:read and file:write - -Add file permissions to the auth system before creating API endpoints. - -**Files:** - -- Modify: `internal/authtoken/permissions.go` -- Modify: `internal/authtoken/permissions_public_test.go` - -### Step 1: Add permission constants - -In `internal/authtoken/permissions.go`, add: - -```go -const ( - // ... existing ... - PermFileRead Permission = "file:read" - PermFileWrite Permission = "file:write" -) -``` - -Add to `AllPermissions`: - -```go -var AllPermissions = []Permission{ - // ... existing ... - PermFileRead, - PermFileWrite, -} -``` - -Add to `DefaultRolePermissions`: - -```go -"admin": { - // ... existing ... - PermFileRead, - PermFileWrite, -}, -"write": { - // ... existing ... - PermFileRead, - PermFileWrite, -}, -"read": { - // ... existing ... - PermFileRead, -}, -``` - -### Step 2: Update permission tests - -Add test cases to the existing permissions test suite to verify the new -permissions resolve correctly for admin, write, and read roles. - -### Step 3: Run tests - -```bash -go test ./internal/authtoken/... -count=1 -v -``` - -### Step 4: Commit - -```bash -git add internal/authtoken/permissions.go \ - internal/authtoken/permissions_public_test.go -git commit -m "feat(auth): add file:read and file:write permissions" -``` - ---- - -## Task 3: File API Domain — OpenAPI Spec + Code Generation - -Create the `/file` REST API domain for Object Store management. - -**Files:** - -- Create: `internal/api/file/gen/api.yaml` -- Create: `internal/api/file/gen/cfg.yaml` -- Create: `internal/api/file/gen/generate.go` -- Generated: `internal/api/file/gen/file.gen.go` - -### Step 1: Write OpenAPI spec - -Create `internal/api/file/gen/api.yaml`: - -```yaml -openapi: '3.0.0' -info: - title: File Management API - version: 1.0.0 - -tags: - - name: file - x-displayName: File - description: Manage files in the Object Store. - -paths: - /file: - post: - operationId: PostFile - summary: Upload a file to Object Store - description: > - Stores file content in NATS Object Store. Returns the object reference - with SHA256 and size. - tags: [file] - security: - - BearerAuth: - - 'file:write' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/FileUploadRequest' - responses: - '201': - description: File uploaded successfully. - content: - application/json: - schema: - $ref: '#/components/schemas/FileUploadResponse' - '400': - description: Invalid input. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '401': - description: Unauthorized. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Internal server error. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - - get: - operationId: GetFiles - summary: List stored files - description: Returns metadata for all files in the Object Store. - tags: [file] - security: - - BearerAuth: - - 'file:read' - responses: - '200': - description: List of stored files. - content: - application/json: - schema: - $ref: '#/components/schemas/FileListResponse' - '401': - description: Unauthorized. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Internal server error. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - - /file/{name}: - get: - operationId: GetFileByName - summary: Get file metadata - description: Returns metadata for a specific file in the Object Store. - tags: [file] - security: - - BearerAuth: - - 'file:read' - parameters: - - $ref: '#/components/parameters/FileName' - responses: - '200': - description: File metadata. - content: - application/json: - schema: - $ref: '#/components/schemas/FileInfoResponse' - '401': - description: Unauthorized. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '404': - description: File not found. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Internal server error. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - - delete: - operationId: DeleteFile - summary: Delete a file from Object Store - description: Removes a file from the Object Store. - tags: [file] - security: - - BearerAuth: - - 'file:write' - parameters: - - $ref: '#/components/parameters/FileName' - responses: - '200': - description: File deleted. - content: - application/json: - schema: - $ref: '#/components/schemas/FileDeleteResponse' - '401': - description: Unauthorized. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '404': - description: File not found. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Internal server error. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - -components: - securitySchemes: - BearerAuth: - type: http - scheme: bearer - bearerFormat: JWT - - parameters: - FileName: - name: name - in: path - required: true - schema: - type: string - description: The name of the file in the Object Store. - # NOTE: path param x-oapi-codegen-extra-tags does not generate - # tags on RequestObject structs in strict-server mode. - # Validated manually in handler. - x-oapi-codegen-extra-tags: - validate: required,min=1,max=255 - - schemas: - FileUploadRequest: - type: object - properties: - name: - type: string - description: > - Name to store the file under in the Object Store. - x-oapi-codegen-extra-tags: - validate: required,min=1,max=255 - content: - type: string - format: byte - description: > - Base64-encoded file content. - x-oapi-codegen-extra-tags: - validate: required - required: [name, content] - - FileUploadResponse: - type: object - properties: - name: - type: string - sha256: - type: string - size: - type: integer - format: int64 - required: [name, sha256, size] - - FileListResponse: - type: object - properties: - files: - type: array - items: - $ref: '#/components/schemas/FileInfo' - required: [files] - - FileInfo: - type: object - properties: - name: - type: string - sha256: - type: string - size: - type: integer - format: int64 - required: [name, size] - - FileInfoResponse: - type: object - properties: - name: - type: string - sha256: - type: string - size: - type: integer - format: int64 - required: [name, sha256, size] - - FileDeleteResponse: - type: object - properties: - name: - type: string - deleted: - type: boolean - required: [name, deleted] -``` - -### Step 2: Write codegen config - -Create `internal/api/file/gen/cfg.yaml`: - -```yaml -package: gen -generate: - strict-server: true - echo-server: true - models: true -import-mapping: - ../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/api/common/gen -output: file.gen.go -``` - -### Step 3: Write generate directive - -Create `internal/api/file/gen/generate.go`: - -```go -package gen - -//go:generate go run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen --config cfg.yaml api.yaml -``` - -### Step 4: Generate code - -```bash -go generate ./internal/api/file/gen/... -``` - -### Step 5: Commit - -```bash -git add internal/api/file/gen/ -git commit -m "feat(api): add file domain OpenAPI spec and codegen" -``` - ---- - -## Task 4: File API Handler — Upload, List, Get, Delete - -Implement the file API handler with all four endpoints. - -**Files:** - -- Create: `internal/api/file/types.go` -- Create: `internal/api/file/file.go` -- Create: `internal/api/file/file_upload.go` -- Create: `internal/api/file/file_list.go` -- Create: `internal/api/file/file_get.go` -- Create: `internal/api/file/file_delete.go` -- Create: `internal/api/file/file_upload_public_test.go` -- Create: `internal/api/file/file_list_public_test.go` -- Create: `internal/api/file/file_get_public_test.go` -- Create: `internal/api/file/file_delete_public_test.go` - -### Step 1: Write types.go - -```go -package file - -import ( - "context" - "log/slog" - - "github.com/nats-io/nats.go/jetstream" -) - -// ObjectStoreManager abstracts NATS Object Store operations for testing. -type ObjectStoreManager interface { - PutBytes( - ctx context.Context, - name string, - data []byte, - ) (*jetstream.ObjectInfo, error) - GetBytes( - ctx context.Context, - name string, - ) ([]byte, error) - GetInfo( - ctx context.Context, - name string, - ) (*jetstream.ObjectInfo, error) - Delete( - ctx context.Context, - name string, - ) error - List( - ctx context.Context, - ) ([]*jetstream.ObjectInfo, error) -} - -// File handles file management REST API endpoints. -type File struct { - objStore ObjectStoreManager - logger *slog.Logger -} -``` - -**Note:** The `ObjectStoreManager` interface wraps `jetstream.ObjectStore` so -handlers can be tested with mocks. The actual `jetstream.ObjectStore` satisfies -this interface. Verify that the `jetstream.ObjectStore` interface matches — the -`List` method may return a lister instead of a slice; adapt accordingly. - -### Step 2: Write file.go factory - -```go -package file - -import ( - "log/slog" - - gen "github.com/osapi-io/osapi/internal/api/file/gen" -) - -var _ gen.StrictServerInterface = (*File)(nil) - -// New creates a new File handler. -func New( - logger *slog.Logger, - objStore ObjectStoreManager, -) *File { - return &File{ - objStore: objStore, - logger: logger, - } -} -``` - -### Step 3: Write upload handler (file_upload.go) - -Decode base64 content from request body, store in Object Store, return -reference. Use `validation.Struct(request.Body)` for input validation. - -### Step 4: Write failing tests for upload - -Create `file_upload_public_test.go` with table-driven suite: - -- when valid upload succeeds (201) -- when name is empty (400, validation error) -- when content is empty (400, validation error) -- when Object Store put fails (500) - -Include `TestPostFileHTTP` and `TestPostFileRBACHTTP` methods. - -### Step 5: Implement remaining handlers - -Follow the same test-first pattern for list, get, delete: - -- `file_list.go` — iterate Object Store, return file info array -- `file_get.go` — get info by name, return 404 if not found -- `file_delete.go` — delete by name, return 404 if not found - -### Step 6: Run tests - -```bash -go test ./internal/api/file/... -count=1 -v -``` - -### Step 7: Commit - -```bash -git add internal/api/file/ -git commit -m "feat(api): implement file upload, list, get, delete handlers" -``` - ---- - -## Task 5: File API Server Wiring - -Wire the file handler into the API server. - -**Files:** - -- Create: `internal/api/handler_file.go` -- Create: `internal/api/handler_file_public_test.go` -- Modify: `internal/api/types.go` -- Modify: `internal/api/handler.go` -- Modify: `cmd/api_helpers.go` - -### Step 1: Create handler_file.go - -Follow the pattern from `handler_node.go`. All file endpoints require -authentication. The handler factory takes an `ObjectStoreManager`: - -```go -func (s *Server) GetFileHandler( - objStore file.ObjectStoreManager, -) []func(e *echo.Echo) { - var tokenManager TokenValidator = authtoken.New(s.logger) - - fileHandler := file.New(s.logger, objStore) - - strictHandler := fileGen.NewStrictHandler( - fileHandler, - []fileGen.StrictMiddlewareFunc{ - func(handler strictecho.StrictEchoHandlerFunc, _ string) strictecho.StrictEchoHandlerFunc { - return scopeMiddleware( - handler, - tokenManager, - s.appConfig.API.Server.Security.SigningKey, - fileGen.BearerAuthScopes, - s.customRoles, - ) - }, - }, - ) - - return []func(e *echo.Echo){ - func(e *echo.Echo) { - fileGen.RegisterHandlers(e, strictHandler) - }, - } -} -``` - -### Step 2: Update types.go - -No new fields needed on `Server` — the Object Store is passed directly to -`GetFileHandler()`. - -### Step 3: Update handler.go - -In `registerAPIHandlers()` (or equivalent), add: - -```go -handlers = append(handlers, sm.GetFileHandler(objStore)...) -``` - -### Step 4: Update startup wiring - -In `cmd/api_helpers.go`, create the Object Store handle at startup and pass it -to the file handler: - -```go -// Create Object Store handle for file management API -var objStore jetstream.ObjectStore -if appConfig.NATS.Objects.Bucket != "" { - objStoreName := job.ApplyNamespaceToInfraName(namespace, appConfig.NATS.Objects.Bucket) - objStore, err = nc.ObjectStore(ctx, objStoreName) - // handle error -} -``` - -### Step 5: Add handler test - -Create `handler_file_public_test.go` following the pattern of -`handler_node_public_test.go`. - -### Step 6: Update combined OpenAPI spec - -Add the file spec to `internal/api/gen/api.yaml` merged spec. - -### Step 7: Run tests and verify - -```bash -go build ./... -go test ./internal/api/... -count=1 -v -``` - -### Step 8: Commit - -```bash -git add internal/api/handler_file.go internal/api/handler_file_public_test.go \ - internal/api/types.go internal/api/handler.go \ - cmd/api_helpers.go internal/api/gen/api.yaml -git commit -m "feat(api): wire file handler into API server" -``` - ---- - -## Task 6: Job Types + File Provider Interface - -Define operation constants, request/response types, and the file provider -interface. - -**Files:** - -- Modify: `internal/job/types.go` -- Create: `internal/provider/file/types.go` -- Create: `internal/provider/file/mocks/types.gen.go` -- Create: `internal/provider/file/mocks/mocks.go` - -### Step 1: Add operation constants - -In `internal/job/types.go`: - -```go -// File operations -const ( - OperationFileDeployExecute = "file.deploy.execute" - OperationFileStatusGet = "file.status.get" -) -``` - -### Step 2: Define file state type - -In `internal/job/types.go`, add the file state KV entry structure: - -```go -// FileState represents a deployed file's state in the file-state KV. -// Keyed by .. -type FileState struct { - ObjectName string `json:"object_name"` - Path string `json:"path"` - SHA256 string `json:"sha256"` - Mode string `json:"mode,omitempty"` - Owner string `json:"owner,omitempty"` - Group string `json:"group,omitempty"` - DeployedAt string `json:"deployed_at"` - ContentType string `json:"content_type"` -} -``` - -### Step 3: Define provider interface - -Create `internal/provider/file/types.go`: - -```go -package file - -import "context" - -// DeployRequest contains parameters for deploying a file to disk. -type DeployRequest struct { - ObjectName string `json:"object_name"` - Path string `json:"path"` - Mode string `json:"mode,omitempty"` - Owner string `json:"owner,omitempty"` - Group string `json:"group,omitempty"` - ContentType string `json:"content_type"` // "raw" or "template" - Vars map[string]any `json:"vars,omitempty"` -} - -// DeployResult contains the result of a file deploy operation. -type DeployResult struct { - Changed bool `json:"changed"` - SHA256 string `json:"sha256"` - Path string `json:"path"` -} - -// StatusRequest contains parameters for checking file status. -type StatusRequest struct { - Path string `json:"path"` -} - -// StatusResult contains the result of a file status check. -type StatusResult struct { - Path string `json:"path"` - Status string `json:"status"` // "in-sync", "drifted", "missing" - SHA256 string `json:"sha256,omitempty"` -} - -// Provider defines the interface for file operations. -type Provider interface { - Deploy( - ctx context.Context, - req DeployRequest, - ) (*DeployResult, error) - Status( - ctx context.Context, - req StatusRequest, - ) (*StatusResult, error) -} -``` - -### Step 4: Generate mocks - -Create `internal/provider/file/mocks/mocks.go`: - -```go -package mocks - -//go:generate mockgen -source=../types.go -destination=types.gen.go -package=mocks -``` - -Run: - -```bash -go generate ./internal/provider/file/mocks/... -``` - -### Step 5: Commit - -```bash -git add internal/job/types.go internal/provider/file/ -git commit -m "feat(file): add job operation constants and provider interface" -``` - ---- - -## Task 7: File Provider Implementation — Deploy with SHA Idempotency - -Implement the core deploy logic: pull from Object Store, SHA compare, write -file, set permissions, update state KV. - -**Files:** - -- Create: `internal/provider/file/provider.go` -- Create: `internal/provider/file/deploy.go` -- Create: `internal/provider/file/deploy_public_test.go` -- Create: `internal/provider/file/status.go` -- Create: `internal/provider/file/status_public_test.go` - -### Step 1: Write provider constructor - -Create `internal/provider/file/provider.go`: - -```go -package file - -import ( - "context" - "log/slog" - - "github.com/nats-io/nats.go/jetstream" - "github.com/spf13/afero" - - "github.com/osapi-io/osapi/internal/job" -) - -// FileProvider implements file deploy and status operations. -type FileProvider struct { - logger *slog.Logger - fs afero.Fs - objStore jetstream.ObjectStore - stateKV jetstream.KeyValue - hostname string - cachedFacts *job.FactsRegistration -} - -// New creates a new FileProvider. -func New( - logger *slog.Logger, - fs afero.Fs, - objStore jetstream.ObjectStore, - stateKV jetstream.KeyValue, - hostname string, - cachedFacts *job.FactsRegistration, -) *FileProvider { - return &FileProvider{ - logger: logger, - fs: fs, - objStore: objStore, - stateKV: stateKV, - hostname: hostname, - cachedFacts: cachedFacts, - } -} -``` - -**Note:** The provider uses `afero.Fs` for filesystem abstraction (testable -without writing real files). The `objStore` and `stateKV` are NATS JetStream -interfaces — mock them in tests. - -### Step 2: Write failing deploy tests - -Create `deploy_public_test.go` with table-driven cases: - -| Case | Setup | Expected | -| ------------------------------- | --------------------------------------------------------- | ------------------------------ | -| when deploy succeeds (new file) | Mock: objStore returns content, stateKV has no entry | changed: true, file written | -| when deploy succeeds (changed) | Mock: objStore returns content, stateKV has different SHA | changed: true, file written | -| when deploy skips (unchanged) | Mock: objStore returns content, stateKV has same SHA | changed: false, no write | -| when Object Store get fails | Mock: objStore returns error | error | -| when file write fails | Mock: fs write fails | error | -| when state KV put fails | Mock: stateKV put fails | error | -| when mode is set | Mock: success | file written with correct mode | - -### Step 3: Implement deploy - -Create `deploy.go`. Core logic: - -1. Pull content from Object Store: `objStore.GetBytes(ctx, req.ObjectName)` -2. If `content_type == "template"`, render (delegate to Task 8) -3. Compute SHA256 of final content -4. Build state key: `hostname + "." + sha256(req.Path)` -5. Check `stateKV.Get(ctx, stateKey)` — if SHA matches, return - `{changed: false}` -6. Write file using `afero.WriteFile(fs, req.Path, content, mode)` -7. If owner/group set, `fs.Chown` (skip if not root or on macOS) -8. Update stateKV with new `FileState` -9. Return `{changed: true, sha256: sha}` - -```go -func (p *FileProvider) Deploy( - ctx context.Context, - req DeployRequest, -) (*DeployResult, error) { - // 1. Pull content from Object Store - content, err := p.objStore.GetBytes(ctx, req.ObjectName) - if err != nil { - return nil, fmt.Errorf("failed to get object %q: %w", req.ObjectName, err) - } - - // 2. Template rendering (if applicable) - if req.ContentType == "template" { - content, err = p.renderTemplate(content, req.Vars) - if err != nil { - return nil, fmt.Errorf("failed to render template: %w", err) - } - } - - // 3. Compute SHA of final content - sha := computeSHA256(content) - - // 4. Check state for idempotency - stateKey := buildStateKey(p.hostname, req.Path) - existing, _ := p.stateKV.Get(ctx, stateKey) - if existing != nil { - var state job.FileState - if json.Unmarshal(existing.Value(), &state) == nil && state.SHA256 == sha { - return &DeployResult{Changed: false, SHA256: sha, Path: req.Path}, nil - } - } - - // 5. Write file - mode := parseFileMode(req.Mode) - if err := afero.WriteFile(p.fs, req.Path, content, mode); err != nil { - return nil, fmt.Errorf("failed to write file %q: %w", req.Path, err) - } - - // 6. Update state KV - state := job.FileState{ - ObjectName: req.ObjectName, - Path: req.Path, - SHA256: sha, - Mode: req.Mode, - Owner: req.Owner, - Group: req.Group, - DeployedAt: time.Now().UTC().Format(time.RFC3339), - ContentType: req.ContentType, - } - stateBytes, _ := json.Marshal(state) - if _, err := p.stateKV.Put(ctx, stateKey, stateBytes); err != nil { - return nil, fmt.Errorf("failed to update file state: %w", err) - } - - return &DeployResult{Changed: true, SHA256: sha, Path: req.Path}, nil -} -``` - -### Step 4: Implement helper functions - -```go -func computeSHA256(data []byte) string { - h := sha256.Sum256(data) - return hex.EncodeToString(h[:]) -} - -func buildStateKey(hostname, path string) string { - pathHash := computeSHA256([]byte(path)) - return hostname + "." + pathHash -} - -func parseFileMode(mode string) os.FileMode { - if mode == "" { - return 0o644 - } - m, err := strconv.ParseUint(mode, 8, 32) - if err != nil { - return 0o644 - } - return os.FileMode(m) -} -``` - -### Step 5: Write failing status tests - -Create `status_public_test.go`: - -| Case | Setup | Expected | -| ------------------- | ----------------------------------- | ----------------- | -| when file in sync | Local SHA matches state KV SHA | status: "in-sync" | -| when file drifted | Local SHA differs from state KV SHA | status: "drifted" | -| when file missing | File doesn't exist on disk | status: "missing" | -| when no state entry | stateKV has no entry for path | status: "missing" | - -### Step 6: Implement status - -```go -func (p *FileProvider) Status( - ctx context.Context, - req StatusRequest, -) (*StatusResult, error) { - stateKey := buildStateKey(p.hostname, req.Path) - - entry, err := p.stateKV.Get(ctx, stateKey) - if err != nil { - return &StatusResult{Path: req.Path, Status: "missing"}, nil - } - - var state job.FileState - if err := json.Unmarshal(entry.Value(), &state); err != nil { - return nil, fmt.Errorf("failed to parse file state: %w", err) - } - - // Check if file exists on disk - data, err := afero.ReadFile(p.fs, req.Path) - if err != nil { - return &StatusResult{Path: req.Path, Status: "missing"}, nil - } - - localSHA := computeSHA256(data) - if localSHA == state.SHA256 { - return &StatusResult{Path: req.Path, Status: "in-sync", SHA256: localSHA}, nil - } - - return &StatusResult{Path: req.Path, Status: "drifted", SHA256: localSHA}, nil -} -``` - -### Step 7: Run tests - -```bash -go test ./internal/provider/file/... -count=1 -v -``` - -### Step 8: Commit - -```bash -git add internal/provider/file/ -git commit -m "feat(file): implement deploy with SHA idempotency and status check" -``` - ---- - -## Task 8: Template Rendering - -Add Go `text/template` rendering support to the file provider. - -**Files:** - -- Create: `internal/provider/file/template.go` -- Create: `internal/provider/file/template_public_test.go` - -### Step 1: Define template context - -In `template.go`: - -```go -// TemplateContext is the data available to Go templates during rendering. -type TemplateContext struct { - Facts *job.FactsRegistration - Vars map[string]any - Hostname string -} -``` - -### Step 2: Write failing template tests - -Create `template_public_test.go`: - -| Case | Template | Vars/Facts | Expected | -| ---------------------------- | ------------------------------------------------------------------ | ------------------------------- | ------------------ | -| when simple var substitution | `server {{ .Vars.host }}` | `{"host":"10.0.0.1"}` | `server 10.0.0.1` | -| when fact reference | `arch: {{ .Facts.Architecture }}` | Facts with Architecture="amd64" | `arch: amd64` | -| when conditional | `{{ if eq .Facts.Architecture "arm64" }}arm{{ else }}x86{{ end }}` | Architecture="amd64" | `x86` | -| when hostname | `# {{ .Hostname }}` | hostname="web-01" | `# web-01` | -| when invalid template syntax | `{{ .Invalid` | — | error | -| when nil facts | `{{ .Hostname }}` | nil facts | uses hostname only | - -### Step 3: Implement renderTemplate - -```go -func (p *FileProvider) renderTemplate( - rawTemplate []byte, - vars map[string]any, -) ([]byte, error) { - tmpl, err := template.New("file").Parse(string(rawTemplate)) - if err != nil { - return nil, fmt.Errorf("failed to parse template: %w", err) - } - - ctx := TemplateContext{ - Facts: p.cachedFacts, - Vars: vars, - Hostname: p.hostname, - } - - var buf bytes.Buffer - if err := tmpl.Execute(&buf, ctx); err != nil { - return nil, fmt.Errorf("failed to execute template: %w", err) - } - - return buf.Bytes(), nil -} -``` - -### Step 4: Run tests - -```bash -go test ./internal/provider/file/... -count=1 -v -``` - -### Step 5: Commit - -```bash -git add internal/provider/file/template.go \ - internal/provider/file/template_public_test.go -git commit -m "feat(file): add Go text/template rendering with facts and vars" -``` - ---- - -## Task 9: Agent Wiring + Processor Dispatch - -Add Object Store, file-state KV, and file provider to the agent. Add `file` -category to the processor dispatcher. - -**Files:** - -- Modify: `internal/agent/types.go` -- Modify: `internal/agent/agent.go` (New constructor) -- Create: `internal/agent/processor_file.go` -- Create: `internal/agent/processor_file_test.go` -- Modify: `internal/agent/processor.go` -- Modify: `internal/agent/processor_test.go` -- Modify: `cmd/agent_helpers.go` -- Modify: `cmd/api_helpers.go` - -### Step 1: Update Agent struct - -In `internal/agent/types.go`, add: - -```go -import ( - // ... existing ... - fileProv "github.com/osapi-io/osapi/internal/provider/file" -) - -type Agent struct { - // ... existing fields ... - - // File provider for file deploy/status operations - fileProvider fileProv.Provider - - // Object Store handle (shared primitive for future providers) - objStore jetstream.ObjectStore - - // File-state KV for SHA tracking - fileStateKV jetstream.KeyValue -} -``` - -### Step 2: Update constructor - -In `internal/agent/agent.go`, add parameters to `New()`: - -```go -func New( - // ... existing params ... - fileProvider fileProv.Provider, - objStore jetstream.ObjectStore, - fileStateKV jetstream.KeyValue, -) *Agent { -``` - -### Step 3: Create processor_file.go - -```go -func (a *Agent) processFileOperation( - jobRequest job.Request, -) (json.RawMessage, error) { - baseOperation := strings.Split(jobRequest.Operation, ".")[0] - - switch baseOperation { - case "deploy": - return a.processFileDeploy(jobRequest) - case "status": - return a.processFileStatus(jobRequest) - default: - return nil, fmt.Errorf("unsupported file operation: %s", jobRequest.Operation) - } -} - -func (a *Agent) processFileDeploy( - jobRequest job.Request, -) (json.RawMessage, error) { - var req fileProv.DeployRequest - if err := json.Unmarshal(jobRequest.Data, &req); err != nil { - return nil, fmt.Errorf("failed to parse file deploy data: %w", err) - } - - result, err := a.fileProvider.Deploy(context.Background(), req) - if err != nil { - return nil, fmt.Errorf("file deploy failed: %w", err) - } - - return json.Marshal(result) -} - -func (a *Agent) processFileStatus( - jobRequest job.Request, -) (json.RawMessage, error) { - var req fileProv.StatusRequest - if err := json.Unmarshal(jobRequest.Data, &req); err != nil { - return nil, fmt.Errorf("failed to parse file status data: %w", err) - } - - result, err := a.fileProvider.Status(context.Background(), req) - if err != nil { - return nil, fmt.Errorf("file status failed: %w", err) - } - - return json.Marshal(result) -} -``` - -### Step 4: Update processor.go dispatch - -Add to `processJobOperation()`: - -```go -case "file": - return a.processFileOperation(jobRequest) -``` - -### Step 5: Write processor tests - -Add test cases to `processor_test.go` for the file category, and create -`processor_file_test.go` for the file sub-dispatch. - -### Step 6: Update startup wiring - -In `cmd/agent_helpers.go`: - -```go -// Create Object Store handle -var objStore jetstream.ObjectStore -if appConfig.NATS.Objects.Bucket != "" { - objStoreName := job.ApplyNamespaceToInfraName(namespace, appConfig.NATS.Objects.Bucket) - objStore, _ = nc.ObjectStore(ctx, objStoreName) -} - -// Create file-state KV -var fileStateKV jetstream.KeyValue -if appConfig.NATS.FileState.Bucket != "" { - fileStateKVConfig := cli.BuildFileStateKVConfig(namespace, appConfig.NATS.FileState) - fileStateKV, _ = nc.CreateOrUpdateKVBucketWithConfig(ctx, fileStateKVConfig) -} - -// Create file provider (after agent hostname is resolved) -fileProvider := fileProv.New(log, appFs, objStore, fileStateKV, hostname, nil) - -a := agent.New( - // ... existing args ... - fileProvider, - objStore, - fileStateKV, -) -``` - -**Note:** The file provider's `cachedFacts` is initially nil and gets updated -when facts are collected. Add a method or field update in the facts collection -loop to keep the file provider's facts current. - -### Step 7: Update all existing tests that call agent.New() - -Every test that constructs an `Agent` needs the new parameters. Pass `nil` for -file provider, objStore, and fileStateKV in tests that don't exercise file -operations. - -### Step 8: Run tests - -```bash -go build ./... -go test ./internal/agent/... -count=1 -v -``` - -### Step 9: Commit - -```bash -git add internal/agent/ cmd/agent_helpers.go cmd/api_helpers.go -git commit -m "feat(agent): wire file provider and Object Store into agent" -``` - ---- - -## Task 10: Job Client Methods for File Deploy/Status - -Add convenience methods to the job client for triggering file operations. - -**Files:** - -- Modify: `internal/job/client/types.go` (JobClient interface) -- Create: `internal/job/client/file.go` -- Create: `internal/job/client/file_public_test.go` -- Modify: `internal/job/mocks/job_client.gen.go` (regenerate) - -### Step 1: Add interface methods - -In `internal/job/client/types.go`, add to `JobClient`: - -```go -// File operations -ModifyFileDeploy( - ctx context.Context, - hostname string, - objectName string, - path string, - contentType string, - mode string, - owner string, - group string, - vars map[string]any, -) (string, string, bool, error) - -QueryFileStatus( - ctx context.Context, - hostname string, - path string, -) (string, *file.StatusResult, error) -``` - -### Step 2: Write failing tests - -Test the job creation, subject routing, and response parsing. - -### Step 3: Implement methods - -Follow the pattern of `ModifyNetworkDNS` and `QueryNodeStatus`: - -```go -func (c *Client) ModifyFileDeploy( - ctx context.Context, - hostname string, - objectName string, - path string, - contentType string, - mode string, - owner string, - group string, - vars map[string]any, -) (string, string, bool, error) { - data, _ := json.Marshal(file.DeployRequest{ - ObjectName: objectName, - Path: path, - Mode: mode, - Owner: owner, - Group: group, - ContentType: contentType, - Vars: vars, - }) - - req := &job.Request{ - Type: job.TypeModify, - Category: "file", - Operation: job.OperationFileDeployExecute, - Data: json.RawMessage(data), - } - - subject := job.BuildSubjectFromTarget(job.JobsModifyPrefix, hostname) - jobID, resp, err := c.publishAndWait(ctx, subject, req) - if err != nil { - return "", "", false, err - } - - changed := resp.Changed != nil && *resp.Changed - return jobID, resp.Hostname, changed, nil -} -``` - -### Step 4: Regenerate mocks - -```bash -go generate ./internal/job/mocks/... -``` - -### Step 5: Run tests - -```bash -go test ./internal/job/client/... -count=1 -v -``` - -### Step 6: Commit - -```bash -git add internal/job/client/ internal/job/mocks/ -git commit -m "feat(job): add file deploy and status job client methods" -``` - ---- - -## Task 11: Node API Endpoints for File Deploy/Status - -Add REST endpoints for triggering file deploy and status through the node -domain. - -**Files:** - -- Modify: `internal/api/node/gen/api.yaml` -- Regenerate: `internal/api/node/gen/node.gen.go` -- Create: `internal/api/node/file_deploy_post.go` -- Create: `internal/api/node/file_deploy_post_public_test.go` -- Create: `internal/api/node/file_status_post.go` -- Create: `internal/api/node/file_status_post_public_test.go` - -### Step 1: Add to node OpenAPI spec - -Add paths and schemas to `internal/api/node/gen/api.yaml`: - -```yaml -/node/{hostname}/file/deploy: - post: - operationId: PostNodeFileDeploy - summary: Deploy a file from Object Store to the host - security: - - BearerAuth: - - 'file:write' - parameters: - - $ref: '#/components/parameters/Hostname' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/FileDeployRequest' - responses: - '202': - description: File deploy job accepted. - content: - application/json: - schema: - $ref: '#/components/schemas/FileDeployResponse' - '400': - description: Invalid input. - '500': - description: Internal error. - -/node/{hostname}/file/status: - post: - operationId: PostNodeFileStatus - summary: Check deployment status of a file on the host - security: - - BearerAuth: - - 'file:read' - parameters: - - $ref: '#/components/parameters/Hostname' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/FileStatusRequest' - responses: - '200': - description: File status. - content: - application/json: - schema: - $ref: '#/components/schemas/FileStatusResponse' - '400': - description: Invalid input. - '500': - description: Internal error. -``` - -Add schemas: - -```yaml -FileDeployRequest: - type: object - properties: - object_name: - type: string - x-oapi-codegen-extra-tags: - validate: required,min=1,max=255 - path: - type: string - x-oapi-codegen-extra-tags: - validate: required,min=1 - mode: - type: string - owner: - type: string - group: - type: string - content_type: - type: string - enum: [raw, template] - x-oapi-codegen-extra-tags: - validate: required,oneof=raw template - vars: - type: object - additionalProperties: true - required: [object_name, path, content_type] - -FileStatusRequest: - type: object - properties: - path: - type: string - x-oapi-codegen-extra-tags: - validate: required,min=1 - required: [path] -``` - -### Step 2: Regenerate - -```bash -go generate ./internal/api/node/gen/... -``` - -### Step 3: Implement handlers - -Follow the pattern of `network_dns_put_by_interface.go`. Each handler: - -1. Validates hostname -2. Validates request body -3. Calls the job client method -4. Returns the response - -### Step 4: Write tests - -Table-driven tests with HTTP wiring and RBAC tests for each endpoint. - -### Step 5: Run tests - -```bash -go test ./internal/api/node/... -count=1 -v -``` - -### Step 6: Commit - -```bash -git add internal/api/node/ -git commit -m "feat(api): add node file deploy and status endpoints" -``` - ---- - -## Task 12: CLI Commands - -Add CLI commands for file management and file deployment. - -**Files:** - -- Create: `cmd/client_file.go` — parent command -- Create: `cmd/client_file_upload.go` -- Create: `cmd/client_file_list.go` -- Create: `cmd/client_file_get.go` -- Create: `cmd/client_file_delete.go` -- Create: `cmd/client_node_file.go` — parent under node -- Create: `cmd/client_node_file_deploy.go` -- Create: `cmd/client_node_file_status.go` - -### Step 1: File management commands - -`osapi client file upload`: - -``` ---name Name for the file in Object Store (required) ---file Path to local file to upload (required) -``` - -`osapi client file list` — no extra flags - -`osapi client file get --name ` — show metadata - -`osapi client file delete --name ` — remove from Object Store - -### Step 2: Node file commands - -`osapi client node file deploy`: - -``` ---object Object name in Object Store (required) ---path Destination path on host (required) ---content-type "raw" or "template" (default: "raw") ---mode File mode (e.g., "0644") ---owner File owner ---group File group ---var Template var (key=value, repeatable) --T, --target Target host (default: _any) --j, --json Raw JSON output -``` - -`osapi client node file status`: - -``` ---path File path to check (required) --T, --target Target host (default: _any) --j, --json Raw JSON output -``` - -### Step 3: Implement commands - -Follow the pattern of `cmd/client_node_command_exec.go`. Read local file, base64 -encode, call SDK upload. For deploy, call SDK deploy. Handle all response codes -in switch block. - -### Step 4: Test manually - -```bash -go build ./... && ./osapi client file upload --help -./osapi client node file deploy --help -``` - -### Step 5: Commit - -```bash -git add cmd/client_file*.go cmd/client_node_file*.go -git commit -m "feat(cli): add file upload/list/get/delete and deploy/status commands" -``` - ---- - -## Task 13: SDK Integration - -Update the `osapi-sdk` to support the new file endpoints. - -**Files (in osapi-sdk repo):** - -- Copy: `pkg/osapi/gen/file/api.yaml` (from osapi) -- Create: `pkg/osapi/file.go` — FileService -- Modify: `.gilt.yml` — add file spec overlay -- Regenerate client code - -### Step 1: Add file API spec to SDK - -Copy `internal/api/file/gen/api.yaml` → `pkg/osapi/gen/file/api.yaml`. - -### Step 2: Update gilt overlay - -Add file domain to `.gilt.yml` so `just generate` pulls the spec. - -### Step 3: Create FileService - -```go -type FileService struct { - client *Client -} - -func (s *FileService) Upload(ctx context.Context, name string, content []byte) (*FileInfo, error) -func (s *FileService) List(ctx context.Context) ([]FileInfo, error) -func (s *FileService) Get(ctx context.Context, name string) (*FileInfo, error) -func (s *FileService) Delete(ctx context.Context, name string) error -``` - -Deploy/status use the existing job system through `NodeService` or as separate -methods. - -### Step 4: Regenerate and test - -```bash -just generate -go test ./... -``` - -### Step 5: Commit and push SDK - -Separate PR on osapi-sdk repo. - ---- - -## Task 14: Orchestrator Integration - -Add file operations to `osapi-orchestrator`. - -**Files (in osapi-orchestrator repo):** - -- Create: `pkg/orchestrator/file.go` -- Create: example `examples/file-deploy/main.go` - -### Step 1: Add orchestrator steps - -```go -func (o *Orchestrator) FileUpload(name, localPath string) *Step -func (o *Orchestrator) FileDeploy(target, objectName, destPath string, opts ...FileOption) *Step -func (o *Orchestrator) FileTemplate(target, objectName, destPath string, vars map[string]any, opts ...FileOption) *Step -``` - -`FileOption` funcs: - -```go -func WithMode(mode string) FileOption -func WithOwner(owner, group string) FileOption -``` - -### Step 2: OnlyIfChanged integration - -`FileDeploy` and `FileTemplate` return `changed: true/false` in the result, so -`OnlyIfChanged()` guards work naturally: - -```go -upload := o.FileUpload("nginx.conf", "./local/nginx.conf.tmpl") -deploy := o.FileTemplate("_all", "nginx.conf", "/etc/nginx/nginx.conf", - map[string]any{"worker_count": 4}, - WithMode("0644"), - WithOwner("root", "root"), -).After(upload) - -reload := o.CommandExec("_all", "nginx", []string{"-s", "reload"}). - After(deploy). - OnlyIfChanged() -``` - -### Step 3: Commit - -Separate PR on osapi-orchestrator repo. - ---- - -## Task 15: Documentation - -Update docs for the new feature. - -**Files:** - -- Create: `docs/docs/sidebar/features/file-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/file/file.md` -- Create: `docs/docs/sidebar/usage/cli/client/file/upload.md` -- Create: `docs/docs/sidebar/usage/cli/client/file/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/file/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/file/delete.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/file/file.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/file/deploy.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/file/status.md` -- Modify: `docs/docusaurus.config.ts` — add to Features dropdown -- Modify: `docs/docs/sidebar/usage/configuration.md` — add new config -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` — add - endpoints - -### Step 1: Feature page - -Create `file-management.md` covering: - -- What it manages (file deployment with SHA idempotency) -- How it works (Object Store + file-state KV) -- Template rendering with facts -- Permissions (`file:read`, `file:write`) -- Links to CLI and API docs - -### Step 2: CLI docs - -One page per command with usage examples, flags table, and `--json` output. - -### Step 3: Config docs - -Add `nats.objects` and `nats.file_state` sections with env vars: - -| Config Key | Env Var | -| -------------------------- | -------------------------------- | -| `nats.objects.bucket` | `OSAPI_NATS_OBJECTS_BUCKET` | -| `nats.objects.max_bytes` | `OSAPI_NATS_OBJECTS_MAX_BYTES` | -| `nats.objects.storage` | `OSAPI_NATS_OBJECTS_STORAGE` | -| `nats.objects.replicas` | `OSAPI_NATS_OBJECTS_REPLICAS` | -| `nats.file_state.bucket` | `OSAPI_NATS_FILE_STATE_BUCKET` | -| `nats.file_state.storage` | `OSAPI_NATS_FILE_STATE_STORAGE` | -| `nats.file_state.replicas` | `OSAPI_NATS_FILE_STATE_REPLICAS` | - -### Step 4: Commit - -```bash -git add docs/ -git commit -m "docs: add file management feature documentation" -``` - ---- - -## Shared Primitive: Object Store for Future Providers - -The Object Store and file-state KV infrastructure built in this plan is designed -as a **shared primitive**. The agent's `objStore` handle is injected at startup -and available to any provider. Future providers that would consume this -infrastructure: - -| Provider | Operation | Usage | -| ----------------- | --------------------------- | ------------------------------- | -| `firmware.update` | Pull binary, run flash tool | Object Store for firmware blobs | -| `package.install` | Pull `.deb`/`.rpm`, install | Object Store for packages | -| `cert.deploy` | Pull TLS cert/key | Object Store + restricted perms | -| `script.run` | Pull script, execute | Object Store for scripts | - -Each provider reuses: Object Store download, SHA comparison, and state tracking -from the `file-state` KV bucket. No new infrastructure needed. - ---- - -## Verification - -After all tasks complete: - -```bash -# Full test suite -just test - -# Manual verification -osapi client file upload --name nginx.conf --file ./nginx.conf -osapi client file list -osapi client file get --name nginx.conf -osapi client node file deploy \ - --object nginx.conf --path /etc/nginx/nginx.conf \ - --mode 0644 --owner root --group root --target _all -osapi client node file status --path /etc/nginx/nginx.conf --target _all - -# Idempotency check (second run should show changed: false) -osapi client node file deploy \ - --object nginx.conf --path /etc/nginx/nginx.conf \ - --mode 0644 --target _all -``` diff --git a/docs/plans/2026-03-06-multipart-file-upload.md b/docs/plans/2026-03-06-multipart-file-upload.md deleted file mode 100644 index a8b9418ce..000000000 --- a/docs/plans/2026-03-06-multipart-file-upload.md +++ /dev/null @@ -1,179 +0,0 @@ -# Multipart File Upload with Streaming - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Migrate the file upload endpoint from JSON with base64-encoded content -to `multipart/form-data` with streaming to NATS Object Store. Add a -`content_type` metadata field to track file purpose (raw vs template). Increase -Object Store bucket size for large file support. - -**Architecture:** Two-pass from temp file — Go's `ParseMultipartForm(32 MiB)` -spools large files to disk. First pass computes SHA-256 for idempotency check; -second pass streams to NATS via `Put(io.Reader)`. Memory is bounded at ~32 MiB -regardless of file size. Content type stored as NATS object header on upload; -deploy reads it from stored metadata. - -**Tech Stack:** Go 1.25, NATS JetStream Object Store, oapi-codegen -`multipart/form-data`, testify/suite, gomock. - -**Design doc:** N/A — plan originated from conversation. - ---- - -## Step 1: Add `Put` to ObjectStoreManager Interface - -**File:** `internal/api/file/types.go` - -Add streaming `Put` method alongside existing `PutBytes`: - -```go -Put( - ctx context.Context, - meta *jetstream.ObjectMeta, - reader io.Reader, - opts ...jetstream.ObjectOpt, -) (*jetstream.ObjectInfo, error) -``` - -Add `io` to imports. Keep `PutBytes` — it's still used elsewhere. - -Regenerate mock: - -```bash -go generate ./internal/api/file/mocks/... -``` - ---- - -## Step 2: Update OpenAPI Spec - -**File:** `internal/api/file/gen/api.yaml` - -### 2a: Change POST /file request body to multipart/form-data - -Replace `application/json` + `FileUploadRequest` with: - -```yaml -requestBody: - description: The file to upload. - required: true - content: - multipart/form-data: - schema: - type: object - properties: - name: - type: string - description: The name of the file in the Object Store. - example: 'nginx.conf' - content_type: - type: string - description: > - How the file should be treated during deploy. "raw" writes bytes - as-is; "template" renders with Go text/template and agent facts. - default: raw - enum: - - raw - - template - file: - type: string - format: binary - description: The file content. - required: - - name - - file -``` - -### 2b: Add `content_type` to response schemas - -Add `content_type` string field to `FileUploadResponse`, `FileInfo`, and -`FileInfoResponse`. Add to their `required` arrays. - -### 2c: Regenerate - -```bash -go generate ./internal/api/file/gen/... -``` - ---- - -## Step 3: Rewrite Upload Handler - -**File:** `internal/api/file/file_upload.go` - -Replace JSON-based handler with multipart streaming: - -1. Extract form fields (name, content_type) from multipart body -2. Validate name manually (multipart fields don't use struct tags) -3. Open multipart file as `io.ReadSeeker` -4. First pass: compute SHA-256 via `io.Copy(hash, file)` -5. Idempotency check against existing digest -6. Second pass: `file.Seek(0, 0)` then stream to NATS via `Put(meta, file)` -7. Store content_type as `Osapi-Content-Type` NATS header on the object - -If oapi-codegen strict-server doesn't parse multipart correctly, fall back to -custom Echo handler registered in `handler_file.go`. - ---- - -## Step 4: Add `content_type` to Get/List Handlers - -**Files:** `internal/api/file/file_get.go`, `internal/api/file/file_list.go` - -Read `Osapi-Content-Type` from NATS object headers and include in responses. - ---- - -## Step 5: Increase Object Store Bucket Size - -**Files:** `configs/osapi.yaml`, `configs/osapi.local.yaml` - -Change `max_bytes` from `104857600` (100 MiB) to `10737418240` (10 GiB). - ---- - -## Step 6: Update Tests - -**File:** `internal/api/file/file_upload_public_test.go` - -- Rewrite `TestPostFile` for multipart request objects -- Rewrite `TestPostFileHTTP` to send `multipart/form-data` -- Rewrite `TestPostFileRBACHTTP` similarly -- Update file_get and file_list tests to assert `content_type` -- Add mock expectations for `Put` (streaming) instead of `PutBytes` - ---- - -## Step 7: Update CLI - -**File:** `cmd/client_file_upload.go` - -- Add `--content-type` flag (default `raw`) -- Stream file from disk via `os.Open` instead of `os.ReadFile` -- Pass content_type to SDK `Upload` call -- Show `Content-Type` in output - ---- - -## Step 8: Update SDK - -**Files in `osapi-sdk`:** - -- Copy updated `api.yaml` to SDK, regenerate with `redocly join` + `go generate` -- Add `ContentType` field to `FileUpload`, `FileItem`, `FileMetadata` types -- Change `Upload` method to accept `io.Reader` and `contentType` parameter -- Build multipart request body in SDK - ---- - -## Verification - -```bash -go generate ./internal/api/file/gen/... -go generate ./internal/api/file/mocks/... -go build ./... -go test ./internal/api/file/... -count=1 -v -just go::unit -just go::vet -``` diff --git a/docs/plans/2026-03-07-sdk-monorepo-migration-design.md b/docs/plans/2026-03-07-sdk-monorepo-migration-design.md deleted file mode 100644 index 978115db5..000000000 --- a/docs/plans/2026-03-07-sdk-monorepo-migration-design.md +++ /dev/null @@ -1,181 +0,0 @@ -# SDK Monorepo Migration - -**Date:** 2026-03-07 **Status:** Design **Author:** @retr0h - -## Problem - -The SDK living in a separate repo (`osapi-io/osapi-sdk`) creates friction: - -- OpenAPI specs must be synced via gilt overlay from osapi's `main` branch -- Every API change requires a two-repo dance: merge osapi, run `just generate` - in the SDK, merge SDK, update `go.mod` in osapi -- Per-example directories each with their own `go.mod` are a maintenance burden -- The SDK has no external consumers — it's only used by the osapi CLI - -## Solution - -Move the SDK into the osapi repo as `pkg/sdk/`. Two incremental PRs. - -## Package Layout - -``` -pkg/sdk/ -├── osapi/ ← PR 1 -│ ├── gen/ -│ │ ├── cfg.yaml ← points to ../../../internal/api/gen/api.yaml -│ │ ├── generate.go ← just oapi-codegen, no gilt -│ │ └── client.gen.go -│ ├── osapi.go -│ ├── transport.go -│ ├── errors.go -│ ├── response.go -│ ├── types.go -│ ├── agent.go -│ ├── agent_types.go -│ ├── audit.go -│ ├── audit_types.go -│ ├── file.go -│ ├── file_types.go -│ ├── health.go -│ ├── health_types.go -│ ├── job.go -│ ├── job_types.go -│ ├── metrics.go -│ ├── node.go -│ ├── node_types.go -│ └── *_test.go -└── orchestrator/ ← PR 2 - ├── plan.go - ├── task.go - ├── options.go - ├── result.go - ├── runner.go - └── *_test.go -``` - -Import paths change to: - -- `github.com/osapi-io/osapi/pkg/sdk/osapi` -- `github.com/osapi-io/osapi/pkg/sdk/orchestrator` - -## Spec Generation - -No more gilt. The `cfg.yaml` in `pkg/sdk/osapi/gen/` references the server's -combined spec directly: - -```yaml -# cfg.yaml -input: ../../../internal/api/gen/api.yaml -``` - -Single source of truth. Specs can never drift. Regenerate with -`go generate ./pkg/sdk/...`. - -## Examples - -Flatten from per-directory modules to individual files in two directories: - -``` -examples/sdk/ -├── osapi/ -│ ├── go.mod ← replace ../../../pkg/sdk -│ ├── go.sum -│ ├── health.go ← go run health.go -│ ├── node.go -│ ├── agent.go -│ ├── audit.go -│ ├── command.go -│ ├── file.go -│ ├── job.go -│ ├── metrics.go -│ └── network.go -└── orchestrator/ - ├── go.mod ← replace ../../../pkg/sdk - ├── go.sum - ├── basic.go - ├── parallel.go - ├── guards.go - ├── hooks.go - ├── retry.go - ├── broadcast.go - ├── error_strategy.go - ├── file_deploy.go - ├── only_if_changed.go - ├── only_if_failed.go - ├── result_decode.go - ├── task_func.go - └── task_func_results.go -``` - -All files are `package main`. Run with `go run health.go`. - -## Documentation - -### Docusaurus SDK Sidebar - -New top-level sidebar section: - -``` -docs/docs/sidebar/sdk/ -├── sdk.md ← Overview with DocCardList -├── client/ -│ ├── client.md ← Client overview, New(), options, transport -│ ├── agent.md -│ ├── audit.md -│ ├── file.md -│ ├── health.md -│ ├── job.md -│ ├── metrics.md -│ └── node.md -└── orchestrator/ - ├── orchestrator.md ← Overview, Plan/Task/Run - ├── operations.md ← Built-in operations reference - ├── hooks.md ← Hooks and error strategies - └── examples.md ← Example walkthroughs -``` - -Content migrated from the osapi-sdk `docs/osapi/` and `docs/orchestration/` -directories. Landing page uses `` cards. - -### README and CLAUDE.md Updates - -- **README.md**: Add SDK link in the docs/features section. Remove sibling repo - references. -- **CLAUDE.md**: Update SDK references to reflect `pkg/sdk/` location. Simplify - "Adding a New API Domain" Step 5 — no gilt, just `go generate ./pkg/sdk/...`. - Remove sibling repo references but keep SDK documentation (now pointing to - in-repo paths). -- **docusaurus.config.ts**: Add "SDK" to the navbar Features dropdown. - -## Cleanup - -### PR 1 (SDK client) - -- Copy `osapi-sdk/pkg/osapi/` → `pkg/sdk/osapi/` -- Update `pkg/sdk/osapi/gen/cfg.yaml` to reference `internal/api/gen/api.yaml` -- Remove `generate.go` gilt step (oapi-codegen only) -- Flatten `osapi-sdk/examples/osapi/` → `examples/sdk/osapi/` -- Update all `cmd/*.go` imports: `github.com/osapi-io/osapi-sdk/pkg/osapi` → - `github.com/osapi-io/osapi/pkg/sdk/osapi` -- Remove `github.com/osapi-io/osapi-sdk` from `go.mod` -- Create Docusaurus client pages -- Update README.md, CLAUDE.md - -### PR 2 (Orchestrator) - -- Copy `osapi-sdk/pkg/orchestrator/` → `pkg/sdk/orchestrator/` -- Update orchestrator imports to use new SDK client path -- Flatten `osapi-sdk/examples/orchestration/` → `examples/sdk/orchestrator/` -- Create Docusaurus orchestrator pages - -### Post-merge - -- User archives `osapi-io/osapi-sdk` repo on GitHub - -## Scalability Note: `kv.Keys()` - -Not related to this migration but documented here for context — the SDK's -`QueueStats()` and `List()` methods rely on the server's `kv.Keys()` call. See -the [Job Architecture](../docs/sidebar/architecture/job-architecture.md) -performance section for the known scalability constraint and mitigation -approaches. diff --git a/docs/plans/2026-03-07-sdk-monorepo-migration.md b/docs/plans/2026-03-07-sdk-monorepo-migration.md deleted file mode 100644 index c557f3f47..000000000 --- a/docs/plans/2026-03-07-sdk-monorepo-migration.md +++ /dev/null @@ -1,522 +0,0 @@ -# SDK Monorepo Migration (PR 1: Client) Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Move the SDK client library from `osapi-io/osapi-sdk` into this repo -as `pkg/sdk/osapi/`, flatten examples, add Docusaurus SDK docs, and update all -references. - -**Architecture:** Copy `osapi-sdk/pkg/osapi/` into `pkg/sdk/osapi/`, rewrite the -codegen to read the server's combined spec directly (no gilt), update all 18 Go -import paths, flatten 9 example directories into individual files, create -Docusaurus SDK sidebar pages, and clean up CLAUDE.md/README.md references. - -**Tech Stack:** Go, oapi-codegen, Docusaurus, Cobra CLI - -**Design doc:** `docs/plans/2026-03-07-sdk-monorepo-migration-design.md` - ---- - -### Task 1: Copy SDK client package - -**Files:** - -- Create: `pkg/sdk/osapi/` (all `.go` files from `osapi-sdk/pkg/osapi/`) -- Create: `pkg/sdk/osapi/gen/` (cfg.yaml, generate.go, client.gen.go) - -**Step 1: Copy source files** - -```bash -mkdir -p pkg/sdk/osapi/gen -cp ../osapi-sdk/pkg/osapi/*.go pkg/sdk/osapi/ -cp ../osapi-sdk/pkg/osapi/gen/client.gen.go pkg/sdk/osapi/gen/ -``` - -**Step 2: Create new cfg.yaml** - -Create `pkg/sdk/osapi/gen/cfg.yaml`: - -```yaml ---- -package: gen -output: client.gen.go -generate: - models: true - client: true -output-options: - skip-prune: true -``` - -**Step 3: Create new generate.go** - -Create `pkg/sdk/osapi/gen/generate.go`: - -```go -// Package gen contains generated code for the OSAPI REST API client. -package gen - -//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen -config cfg.yaml ../../../internal/api/gen/api.yaml -``` - -No gilt — oapi-codegen reads the server's combined spec directly. - -**Step 4: Update package import paths in all copied files** - -In every `.go` file under `pkg/sdk/osapi/` (non-test, non-gen), replace: - -``` -"github.com/osapi-io/osapi-sdk/pkg/osapi/gen" -``` - -with: - -``` -"github.com/osapi-io/osapi/pkg/sdk/osapi/gen" -``` - -In every `_test.go` file under `pkg/sdk/osapi/`, replace: - -``` -"github.com/osapi-io/osapi-sdk/pkg/osapi/gen" -``` - -with: - -``` -"github.com/osapi-io/osapi/pkg/sdk/osapi/gen" -``` - -And for public test files, replace: - -``` -"github.com/osapi-io/osapi-sdk/pkg/osapi" -``` - -with: - -``` -"github.com/osapi-io/osapi/pkg/sdk/osapi" -``` - -**Step 5: Regenerate client to verify** - -```bash -cd pkg/sdk/osapi/gen && go generate ./... -``` - -Verify `client.gen.go` is regenerated without errors. - -**Step 6: Commit** - -```bash -git add pkg/sdk/ -git commit -m "feat(sdk): copy client library into pkg/sdk/osapi" -``` - ---- - -### Task 2: Update Go imports and remove external SDK dependency - -**Files:** - -- Modify: `go.mod` (remove `github.com/osapi-io/osapi-sdk` require) -- Modify: 18 Go files (update import paths) - -**Step 1: Update all Go imports** - -In every file listed below, replace `"github.com/osapi-io/osapi-sdk/pkg/osapi"` -with `"github.com/osapi-io/osapi/pkg/sdk/osapi"`: - -- `cmd/client.go` -- `cmd/client_agent_get.go` -- `cmd/client_audit_export.go` -- `cmd/client_file_upload.go` (aliased import: `osapi "..."`) -- `cmd/client_health_status.go` -- `cmd/client_job_list.go` -- `cmd/client_job_run.go` -- `cmd/client_node_command_exec.go` -- `cmd/client_node_command_shell.go` -- `cmd/client_node_file_deploy.go` -- `cmd/client_node_status_get.go` -- `internal/audit/export/types.go` -- `internal/audit/export/file.go` -- `internal/audit/export/file_test.go` -- `internal/audit/export/export_public_test.go` -- `internal/audit/export/file_public_test.go` -- `internal/cli/ui.go` -- `internal/cli/ui_public_test.go` - -**Step 2: Remove external SDK from go.mod** - -Remove the `github.com/osapi-io/osapi-sdk` line from the `require` block in -`go.mod`. Then run: - -```bash -go mod tidy -``` - -This will remove the SDK from `go.sum` as well. - -**Step 3: Build and test** - -```bash -go build ./... -go test ./... -count=1 -timeout 120s -``` - -**Step 4: Commit** - -```bash -git add -A -git commit -m "refactor(sdk): update imports to pkg/sdk/osapi" -``` - ---- - -### Task 3: Flatten SDK client examples - -**Files:** - -- Create: `examples/sdk/osapi/go.mod` -- Create: `examples/sdk/osapi/health.go` (and 8 more) - -**Step 1: Create examples directory and go.mod** - -```bash -mkdir -p examples/sdk/osapi -``` - -Create `examples/sdk/osapi/go.mod`: - -``` -module github.com/osapi-io/osapi/examples/sdk/osapi - -go 1.25.0 - -replace github.com/osapi-io/osapi => ../../../ - -require github.com/osapi-io/osapi v0.0.0 -``` - -Then run: - -```bash -cd examples/sdk/osapi && go mod tidy -``` - -**Step 2: Create flattened example files** - -Copy each example's `main.go` into a single file, updating the import path. Each -file is `package main` and self-contained. - -From `../osapi-sdk/examples/osapi/`: - -| Source directory | Target file | -| ----------------- | ------------------------------- | -| `health/main.go` | `examples/sdk/osapi/health.go` | -| `node/main.go` | `examples/sdk/osapi/node.go` | -| `agent/main.go` | `examples/sdk/osapi/agent.go` | -| `audit/main.go` | `examples/sdk/osapi/audit.go` | -| `command/main.go` | `examples/sdk/osapi/command.go` | -| `file/main.go` | `examples/sdk/osapi/file.go` | -| `job/main.go` | `examples/sdk/osapi/job.go` | -| `metrics/main.go` | `examples/sdk/osapi/metrics.go` | -| `network/main.go` | `examples/sdk/osapi/network.go` | - -In each file, replace: - -``` -"github.com/osapi-io/osapi-sdk/pkg/osapi" -``` - -with: - -``` -"github.com/osapi-io/osapi/pkg/sdk/osapi" -``` - -**Step 3: Verify examples compile** - -```bash -cd examples/sdk/osapi && go build ./... -``` - -Note: `go build ./...` on `package main` files in the same directory will verify -they all compile. They won't run without a live server, but compilation proves -imports are correct. - -**Step 4: Commit** - -```bash -git add examples/sdk/ -git commit -m "feat(sdk): add flattened client examples" -``` - ---- - -### Task 4: Create Docusaurus SDK client pages - -**Files:** - -- Create: `docs/docs/sidebar/sdk/sdk.md` -- Create: `docs/docs/sidebar/sdk/client/client.md` -- Create: `docs/docs/sidebar/sdk/client/agent.md` -- Create: `docs/docs/sidebar/sdk/client/audit.md` -- Create: `docs/docs/sidebar/sdk/client/file.md` -- Create: `docs/docs/sidebar/sdk/client/health.md` -- Create: `docs/docs/sidebar/sdk/client/job.md` -- Create: `docs/docs/sidebar/sdk/client/metrics.md` -- Create: `docs/docs/sidebar/sdk/client/node.md` -- Modify: `docs/docusaurus.config.ts` - -**Step 1: Create SDK landing page** - -Create `docs/docs/sidebar/sdk/sdk.md`: - -```markdown ---- -sidebar_position: 6 ---- - -# SDK - -OSAPI provides a Go SDK for programmatic access to the REST API. The SDK -includes a typed client library and a DAG-based orchestrator for composing -multi-step operations. - - -``` - -**Step 2: Create client overview page** - -Create `docs/docs/sidebar/sdk/client/client.md`. Migrate content from -`osapi-sdk/docs/osapi/README.md`: services table, client options, targeting -table. Adapt to Docusaurus format with `` for per-service pages. - -**Step 3: Create per-service pages** - -Create one page per service (`agent.md`, `audit.md`, `file.md`, `health.md`, -`job.md`, `metrics.md`, `node.md`). Migrate content from -`osapi-sdk/docs/osapi/{service}.md`. Each page covers the service methods, -parameters, return types, and a usage example. - -**Step 4: Update docusaurus.config.ts** - -Add "SDK" to the Features navbar dropdown: - -```typescript -{ - label: 'SDK', - to: 'sidebar/sdk/sdk', -}, -``` - -Update the `specPath` for the API docs plugin from -`../../osapi-sdk/pkg/osapi/gen/api.yaml` to `../internal/api/gen/api.yaml`. - -**Step 5: Update the API docs specPath** - -In `docs/docusaurus.config.ts`, change: - -```typescript -specPath: '../../osapi-sdk/pkg/osapi/gen/api.yaml', -``` - -to: - -```typescript -specPath: '../internal/api/gen/api.yaml', -``` - -And remove the GitHub download URL reference to the SDK repo. - -**Step 6: Verify docs build** - -```bash -cd docs && bun run build -``` - -**Step 7: Commit** - -```bash -git add docs/ -git commit -m "docs(sdk): add client library pages to Docusaurus" -``` - ---- - -### Task 5: Update CLAUDE.md - -**Files:** - -- Modify: `CLAUDE.md` - -**Step 1: Update architecture section** - -Change line ~41 from: - -``` -- **`osapi-sdk`** - External SDK for programmatic REST API access (sibling repo, linked via `replace` in `go.mod`) -``` - -to: - -``` -- **`pkg/sdk/`** - Go SDK for programmatic REST API access (`osapi/` client library, `orchestrator/` DAG runner) -``` - -**Step 2: Rewrite "Update SDK" Step 5** - -Replace the entire Step 5 section (lines ~174-196) with: - -```markdown -### Step 5: Update SDK - -The SDK client library lives in `pkg/sdk/osapi/`. Its generated HTTP client uses -the same combined OpenAPI spec as the server (`internal/api/gen/api.yaml`). - -**When modifying existing API specs:** - -1. Make changes to `internal/api/{domain}/gen/api.yaml` in this repo -2. Run `just generate` to regenerate server code (this also regenerates the - combined spec via `redocly join`) -3. Run `go generate ./pkg/sdk/osapi/gen/...` to regenerate the SDK client -4. Update the SDK service wrappers in `pkg/sdk/osapi/{domain}.go` if new - response codes were added -5. Update CLI switch blocks in `cmd/` if new response codes were added - -**When adding a new API domain:** - -1. Add a service wrapper in `pkg/sdk/osapi/{domain}.go` -2. Run `go generate ./pkg/sdk/osapi/gen/...` to pick up the new domain's spec - from the combined `api.yaml` -``` - -**Step 3: Remove sibling repo references** - -Remove any remaining references to `osapi-sdk` as a "sibling repo" or "external" -dependency. Keep documentation about the SDK but update paths to `pkg/sdk/`. - -**Step 4: Commit** - -```bash -git add CLAUDE.md -git commit -m "docs: update CLAUDE.md for in-repo SDK" -``` - ---- - -### Task 6: Update README.md and system-architecture.md - -**Files:** - -- Modify: `README.md` -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` - -**Step 1: Update README.md** - -Replace the "Sister Projects" section. Remove `osapi-sdk` from the sister -projects table (it's now in-repo). Add an SDK link in the Documentation section: - -```markdown -## 📖 Documentation - -- [Getting Started](https://osapi-io.github.io/osapi/) -- [Features](https://osapi-io.github.io/osapi/sidebar/features/) -- [SDK](https://osapi-io.github.io/osapi/sidebar/sdk/sdk) -- [CLI Reference](https://osapi-io.github.io/osapi/sidebar/usage/) -- [Architecture](https://osapi-io.github.io/osapi/sidebar/architecture/) -``` - -If `osapi-orchestrator` is still a separate repo, keep it in sister projects but -remove `osapi-sdk`. - -**Step 2: Update system-architecture.md** - -Change line ~19 from: - -``` -| **SDK Client** | `osapi-sdk` (external) | OpenAPI-generated client used by CLI | -``` - -to: - -``` -| **SDK Client** | `pkg/sdk/osapi` | OpenAPI-generated client used by CLI | -``` - -Update the mermaid diagram reference from `SDK["SDK Client (osapi-sdk)"]` to -`SDK["SDK Client (pkg/sdk/osapi)"]`. - -**Step 3: Commit** - -```bash -git add README.md docs/docs/sidebar/architecture/system-architecture.md -git commit -m "docs: update README and architecture for in-repo SDK" -``` - ---- - -### Task 7: Final verification - -**Step 1: Full build** - -```bash -go build ./... -``` - -**Step 2: Full test suite** - -```bash -go test ./... -count=1 -timeout 120s -``` - -**Step 3: Lint** - -```bash -just go::vet -``` - -**Step 4: Regenerate to verify codegen pipeline** - -```bash -go generate ./pkg/sdk/osapi/gen/... -go build ./... -``` - -**Step 5: Verify examples compile** - -```bash -cd examples/sdk/osapi && go build ./... -``` - -**Step 6: Verify docs build** - -```bash -cd docs && bun run build -``` - ---- - -## Files Summary - -| Action | Path | -| ------ | ------------------------------------------------------- | -| Create | `pkg/sdk/osapi/*.go` (all source + test files) | -| Create | `pkg/sdk/osapi/gen/cfg.yaml` | -| Create | `pkg/sdk/osapi/gen/generate.go` | -| Create | `pkg/sdk/osapi/gen/client.gen.go` | -| Create | `examples/sdk/osapi/go.mod` | -| Create | `examples/sdk/osapi/*.go` (9 example files) | -| Create | `docs/docs/sidebar/sdk/sdk.md` | -| Create | `docs/docs/sidebar/sdk/client/client.md` | -| Create | `docs/docs/sidebar/sdk/client/{service}.md` (7 files) | -| Modify | `cmd/*.go` (11 files — import path) | -| Modify | `internal/audit/export/*.go` (5 files — import path) | -| Modify | `internal/cli/ui.go`, `ui_public_test.go` (import path) | -| Modify | `go.mod` (remove external SDK) | -| Modify | `CLAUDE.md` | -| Modify | `README.md` | -| Modify | `docs/docs/sidebar/architecture/system-architecture.md` | -| Modify | `docs/docusaurus.config.ts` | diff --git a/docs/plans/2026-03-09-unified-domain-endpoint-architecture-design.md b/docs/plans/2026-03-09-unified-domain-endpoint-architecture-design.md deleted file mode 100644 index 42ae3a47a..000000000 --- a/docs/plans/2026-03-09-unified-domain-endpoint-architecture-design.md +++ /dev/null @@ -1,124 +0,0 @@ -# Unified Domain Endpoint Architecture - -## Problem - -The system has two parallel paths for executing operations: - -1. **Domain endpoints** (`GET /node/{hostname}`, `PUT /network/...`, etc.) — - synchronous, typed, used by CLI and external consumers. Internally create a - job via `publishAndWait()` and return the full result in one HTTP response. - -2. **Generic job endpoint** (`POST /job`) — asynchronous, untyped. The - orchestrator creates raw jobs with operation strings and `map[string]any` - params, then polls `GET /job/{id}` for results. - -Both paths create the exact same NATS job under the hood. The orchestrator -bypasses the domain endpoints entirely, duplicating job creation, polling, -broadcast handling, and result extraction. Every new operation must be wired in -both paths. The generic job endpoint also bypasses the typed validation that -domain endpoints provide. - -## Decision - -Remove `POST /job`. All job creation goes through domain endpoints. The -orchestrator calls typed SDK client methods directly instead of creating raw -jobs. - -## Architecture - -### Before - -``` -DSL method → Op{Operation: "node.hostname.get"} → POST /job → poll GET /job/{id} -``` - -### After - -``` -DSL method → client.Node.Hostname(ctx, target) → GET /node/{target} → result -``` - -The orchestrator becomes a pure DAG runner. It does not know about jobs, NATS, -or polling. Domain endpoints handle job creation, agent communication, broadcast -collection, and waiting internally. - -## What Gets Removed - -### From the API - -- `POST /job` endpoint and OpenAPI spec -- `PostJob` handler in `internal/api/job/` -- `CreateJob` in `internal/job/client/jobs.go` -- SDK `JobService.Create()` method - -### From the orchestrator - -- `Op` struct (operation string + params map) -- `executeOp()` — generic job creation + polling -- `pollJob()` — polling loop with exponential backoff -- `countExpectedAgents()` — broadcast agent counting -- `hostResultsFromResponses()` — response parsing -- `extractHostResults()` — fallback host result extraction -- `isCommandOp()` — command exit code checking -- `parseAgentDurations()` — agent timing extraction - -## What Stays - -### Job endpoints (observability and management) - -- `GET /job` — list/filter jobs -- `GET /job/{id}` — get job details (debug via job ID from domain responses) -- `DELETE /job/{id}` — delete a job -- `POST /job/{id}/retry` — retry a failed job -- `GET /job/stats` — job statistics - -### Domain endpoints - -All existing domain endpoints remain. They are the single path for job creation. -Every domain endpoint already returns a job ID in its response for -debugging/audit correlation. - -### Orchestrator DSL - -User-facing DSL methods are unchanged: - -- `o.NodeHostnameGet("web-01")` -- `o.NetworkDNSUpdate("web-01", params)` -- `o.TaskFunc("name", fn)` -- Guards, retry, error strategies, hooks — all unchanged - -## What Changes - -### Domain endpoint responses — enrichment - -Domain endpoint responses need to include agent timing/duration data. This is -currently only available through the job polling path. Per-host results for -broadcast operations need the same metadata the orchestrator currently extracts. - -### SDK client — typed responses - -SDK service method responses carry job ID, changed, duration, and host results -uniformly. Each response has a consistent shape the orchestrator can work with. - -### Orchestrator results — typed - -`Result.Data` (`map[string]any`) is replaced with typed results per operation. -Guards and `When()` predicates work with typed accessors. This is a breaking -change. - -### Broadcast handling — delegated - -The orchestrator no longer manages broadcast polling or expected agent counts. -Domain endpoints handle `_all` and label selector targets internally via -`publishAndWait()` and return collected per-host results. - -## Key Design Decisions - -| Decision | Choice | Rationale | -| ---------------------------- | ---------------------------- | ---------------------------------------------------------------- | -| Remove `POST /job` | Yes | Eliminates duplicate path, forces typed validation | -| Synchronous domain endpoints | Keep `publishAndWait` | Simple consumer DX, orchestrator uses goroutines for parallelism | -| Broadcast handling | Delegate to domain endpoints | One implementation, orchestrator stays simple | -| Result types | Typed per operation | Type safety over generic maps | -| Versioning | Breaking change, no v2 | Project is early enough | -| Job ID in responses | Keep | Essential for debugging when things break | diff --git a/docs/plans/2026-03-09-unified-domain-endpoint-architecture.md b/docs/plans/2026-03-09-unified-domain-endpoint-architecture.md deleted file mode 100644 index ebebcd986..000000000 --- a/docs/plans/2026-03-09-unified-domain-endpoint-architecture.md +++ /dev/null @@ -1,328 +0,0 @@ -# Unified Domain Endpoint Architecture Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Remove the generic `POST /job` endpoint and refactor the orchestrator -to call typed SDK client methods through domain endpoints instead of creating -raw jobs. - -**Architecture:** The orchestrator's 13 `Op`-based DSL methods are converted to -`TaskFunc` calls that invoke SDK client methods directly (matching the pattern -already used by HealthCheck, FileUpload, FileChanged, AgentList, AgentGet). The -`Op` struct, `executeOp`, `pollJob`, and all broadcast polling logic are removed -from the SDK orchestrator engine. Domain endpoints handle job creation, waiting, -and broadcast collection internally via `publishAndWait`. - -**Tech Stack:** Go 1.25, osapi (monorepo), osapi-orchestrator (DSL layer) - ---- - -## Repo Layout - -| Repo | Path | Role | -| ------------------ | ----------------------- | ---------------------------------- | -| osapi | `pkg/sdk/orchestrator/` | SDK orchestrator engine | -| osapi | `pkg/sdk/client/` | SDK client (typed service methods) | -| osapi | `internal/api/job/` | Job API endpoints | -| osapi-orchestrator | `pkg/orchestrator/` | User-facing DSL | - -Changes span both repos. osapi changes land first (SDK engine + API), then -osapi-orchestrator (DSL layer). - ---- - -### Task 1: Remove `POST /job` endpoint from API - -Remove the job creation endpoint. List, get, delete, retry, and stats remain. - -**Files:** - -- Modify: `osapi/internal/api/job/gen/api.yaml` — remove `post` under `/job` -- Modify: `osapi/internal/api/job/job_create.go` — delete file -- Modify: `osapi/internal/api/job/job_create_public_test.go` — delete file -- Modify: `osapi/internal/api/handler_job.go` — remove unauthenticated - operations entry for PostJob if present -- Modify: `osapi/internal/job/client/jobs.go` — remove `CreateJob` method -- Modify: `osapi/internal/job/client/jobs_public_test.go` — remove CreateJob - test cases -- Modify: `osapi/pkg/sdk/client/job.go` — remove `Create` method -- Modify: `osapi/pkg/sdk/client/job_types.go` — remove `JobCreated` type if only - used by Create - -**Step 1:** Remove `post` operation from `/job` path in the OpenAPI spec. - -**Step 2:** Run `just generate` to regenerate server code. - -**Step 3:** Delete `job_create.go` and `job_create_public_test.go`. - -**Step 4:** Remove `CreateJob` from `internal/job/client/jobs.go` and its tests. - -**Step 5:** Remove `Create` from `pkg/sdk/client/job.go`. Remove `JobCreated` -type if unused after this change (check if `Retry` also uses it — if so, keep -it). - -**Step 6:** Run `go build ./...` and fix any compile errors. - -**Step 7:** Run `just go::unit` and fix any test failures. - -**Step 8:** Commit. - -```bash -git commit -m "feat(api)!: remove POST /job endpoint - -All job creation now goes through typed domain endpoints. -Job list, get, delete, retry, and stats endpoints remain." -``` - ---- - -### Task 2: Remove `Op`, `executeOp`, `pollJob` from SDK orchestrator engine - -Strip the generic job creation and polling machinery from the SDK orchestrator. -After this, the SDK engine only supports `TaskFunc` (and `TaskFuncWithResults`). - -**Files:** - -- Modify: `osapi/pkg/sdk/orchestrator/plan.go` — remove `Task(name, op)` method - that accepts `*Op` -- Modify: `osapi/pkg/sdk/orchestrator/runner.go` — remove `executeOp`, - `pollJob`, `countExpectedAgents`, `hostResultsFromResponses`, - `extractHostResults`, `isCommandOp`, `parseAgentDurations`, `isTransient`, - `IsBroadcastTarget` -- Modify: `osapi/pkg/sdk/orchestrator/result.go` — remove `Op` struct, remove - `agentDurations` internal field from `Result` -- Modify: `osapi/pkg/sdk/orchestrator/runner_test.go` — remove tests for removed - functions (`TestBackoffDelay`, `TestIsTransient`, - `TestRunTaskStoresResultForAllPaths` if it uses Op) -- Modify: `osapi/pkg/sdk/orchestrator/runner_broadcast_test.go` — remove tests - for removed functions (`TestIsBroadcastTarget`, `TestExtractHostResults`, - `TestHostResultsFromResponses`, `TestParseAgentDurations`, - `TestCountExpectedAgents`, `TestIsCommandOp`, `TestExecuteOp*`, - `TestPollJob*`) -- Modify: `osapi/pkg/sdk/orchestrator/plan_public_test.go` — update - `TestRunOpTask` and `TestRunOpTaskErrors` to use TaskFunc instead of Op -- Modify: `osapi/pkg/sdk/orchestrator/options.go` — check if backoff imports are - still needed (they won't be if pollJob is removed) - -**Step 1:** Remove the `Op` struct from `result.go`. - -**Step 2:** Remove `Task(name string, op *Op)` from `plan.go`. Keep `TaskFunc` -and `TaskFuncWithResults`. - -**Step 3:** Remove all polling and broadcast functions from `runner.go`: -`executeOp`, `pollJob`, `countExpectedAgents`, `hostResultsFromResponses`, -`extractHostResults`, `isCommandOp`, `parseAgentDurations`, `isTransient`, -`IsBroadcastTarget`. - -**Step 4:** Remove `agentDurations` field from `Result`. Keep `JobID`, -`Changed`, `Data`, `Status`, `JobDuration`, `HostResults`. - -**Step 5:** Clean up imports — remove `backoff`, `client`, `strings`, `errors` -if no longer needed in `runner.go`. - -**Step 6:** Delete or update tests. Remove all test methods that exercise -removed code. Update `TestRunOpTask`/`TestRunOpTaskErrors` in -`plan_public_test.go` — these should be converted to use `TaskFunc` calls that -hit a test HTTP server through SDK client methods. - -**Step 7:** Run `go build ./...` and -`go test ./pkg/sdk/orchestrator/... -count=1`. - -**Step 8:** Commit. - -```bash -git commit -m "feat(sdk)!: remove Op struct and job polling from orchestrator - -The SDK orchestrator engine now only supports TaskFunc and -TaskFuncWithResults. Operation execution goes through typed -SDK client methods called from TaskFunc closures." -``` - ---- - -### Task 3: Convert orchestrator DSL operation methods to TaskFunc - -Convert the 13 `Op`-based methods in osapi-orchestrator to use `TaskFunc` with -SDK client calls. This follows the existing pattern used by HealthCheck, -FileUpload, FileChanged, AgentList, and AgentGet. - -**Files:** - -- Modify: `osapi-orchestrator/pkg/orchestrator/ops.go` — rewrite 13 methods -- Modify: `osapi-orchestrator/pkg/orchestrator/ops_test.go` — update tests -- Modify: `osapi-orchestrator/pkg/orchestrator/ops_public_test.go` — update - tests -- Modify: `osapi-orchestrator/pkg/orchestrator/orchestrator.go` — remove - `newStep` if no longer used - -**Before (example):** - -```go -func (o *Orchestrator) NodeHostnameGet(target string) *Step { - return o.newStep(&sdk.Op{ - Operation: opNodeHostnameGet, - Target: target, - }) -} -``` - -**After (example):** - -```go -func (o *Orchestrator) NodeHostnameGet(target string) *Step { - name := o.nextOpName("get-hostname") - task := o.plan.TaskFunc( - name, - func(ctx context.Context, c *osapi.Client) (*sdk.Result, error) { - resp, err := c.Node.Hostname(ctx, target) - if err != nil { - return nil, fmt.Errorf("get hostname: %w", err) - } - return &sdk.Result{ - JobID: resp.Data.JobID, - Changed: anyChanged(resp.Data.Results), - Data: mustRawToMap(resp.RawJSON()), - }, nil - }, - ) - return &Step{task: task} -} -``` - -**Step 1:** Add a helper `nextOpName(prefix string) string` to replace the -operation-constant-based name generation (or reuse existing name logic). - -**Step 2:** Convert each method one at a time. The 13 methods and their SDK -client calls: - -| DSL Method | SDK Call | Notes | -| -------------------------------------------------- | ------------------------------------------------------- | ------------------ | -| `NodeHostnameGet(target)` | `c.Node.Hostname(ctx, target)` | | -| `NodeStatusGet(target)` | `c.Node.Status(ctx, target)` | | -| `NodeUptimeGet(target)` | `c.Node.Uptime(ctx, target)` | | -| `NodeDiskGet(target)` | `c.Node.Disk(ctx, target)` | | -| `NodeMemoryGet(target)` | `c.Node.Memory(ctx, target)` | | -| `NodeLoadGet(target)` | `c.Node.Load(ctx, target)` | | -| `NetworkDNSGet(target, iface)` | `c.Node.GetDNS(ctx, target, iface)` | | -| `NetworkDNSUpdate(target, iface, servers, search)` | `c.Node.UpdateDNS(ctx, target, iface, servers, search)` | | -| `NetworkPingDo(target, addr)` | `c.Node.Ping(ctx, target, addr)` | | -| `CommandExec(target, cmd, args)` | `c.Node.Exec(ctx, ExecRequest{...})` | Build ExecRequest | -| `CommandShell(target, cmd)` | `c.Node.Shell(ctx, ShellRequest{...})` | Build ShellRequest | -| `FileDeploy(target, opts)` | `c.Node.FileDeploy(ctx, FileDeployOpts{...})` | Map opts | -| `FileStatusGet(target, path)` | `c.Node.FileStatus(ctx, target, path)` | | - -**Step 3:** Remove the operation constants (`opNodeHostnameGet`, etc.) and -`newStep` method — they are no longer needed. - -**Step 4:** Remove `mustRawToMap` if no longer used (check FileUpload, -AgentList, AgentGet — they still use it, so it probably stays). - -**Step 5:** Update tests. The existing tests create test HTTP servers and verify -the orchestrator calls the right endpoints. Update them to expect domain -endpoint paths instead of `POST /job`. - -**Step 6:** Run `go build ./...` and `go test ./pkg/orchestrator/... -count=1`. - -**Step 7:** Commit. - -```bash -git commit -m "feat!: use typed SDK client methods instead of generic Op - -All 13 operation methods now call domain endpoints through the -SDK client. The generic Op struct and job creation path are gone." -``` - ---- - -### Task 4: Update examples - -All examples in osapi-orchestrator need to work with the new API. The DSL method -signatures are unchanged, so most examples should compile without modification. -Verify each one. - -**Files:** - -- Check: `osapi-orchestrator/examples/features/*.go` -- Check: `osapi-orchestrator/examples/operations/*.go` - -**Step 1:** Run `go build -o /dev/null` for each example file to verify -compilation. - -**Step 2:** Fix any import or type changes. - -**Step 3:** Commit if changes were needed. - ---- - -### Task 5: Update documentation - -**Files:** - -- Modify: `osapi-orchestrator/docs/features/README.md` — update if it references - Op or job creation -- Modify: `osapi-orchestrator/docs/gen/orchestrator.md` — regenerate -- Modify: `osapi/docs/docs/sidebar/architecture/job-architecture.md` — note that - POST /job was removed -- Modify: `osapi/docs/docs/sidebar/architecture/system-architecture.md` — update - endpoint tables - -**Step 1:** Regenerate API docs: `just generate` in osapi, `just docs::generate` -in osapi-orchestrator (if applicable). - -**Step 2:** Update architecture docs to reflect that domain endpoints are the -sole job creation path. - -**Step 3:** Commit. - ---- - -### Task 6: Clean up SDK client - -After removing `JobService.Create`, verify the SDK client's job module is clean. - -**Files:** - -- Modify: `osapi/pkg/sdk/client/job.go` — verify Create is gone -- Modify: `osapi/pkg/sdk/client/gen/` — regenerate from updated OpenAPI spec - (POST /job removed) - -**Step 1:** Run `go generate ./pkg/sdk/client/gen/...` to regenerate the SDK -client from the updated combined spec. - -**Step 2:** Verify `JobService` only has `Get`, `Delete`, `List`, `Retry`, -`Stats`. - -**Step 3:** Run full test suite. - -**Step 4:** Commit. - ---- - -### Task 7: Final verification - -**Step 1:** In osapi: `go build ./... && just go::unit` - -**Step 2:** In osapi-orchestrator: `go build ./... && just go::unit` - -**Step 3:** Run integration tests if available: `just go::unit-int` - -**Step 4:** Verify all examples compile. - ---- - -## Ordering - -Tasks 1 and 2 are in osapi and can be done together on one branch. Task 3 is in -osapi-orchestrator and depends on Tasks 1-2 being published (or linked via -`replace`). Tasks 4-6 are follow-ups. Task 7 is final verification. - -## What Does NOT Change - -- Domain endpoint handlers (`internal/api/node/`, `internal/api/network/`, etc.) -- `publishAndWait` and broadcast handling in `internal/job/client/query.go` -- Job observability endpoints (GET, DELETE, list, retry, stats) -- CLI commands (they call domain endpoints, not POST /job) -- DSL method signatures (users' DAG code is unchanged) -- Guards, retry, error strategies, hooks -- TaskFunc and TaskFuncWithResults diff --git a/docs/plans/2026-03-11-container-runtime-design.md b/docs/plans/2026-03-11-container-runtime-design.md deleted file mode 100644 index df4156510..000000000 --- a/docs/plans/2026-03-11-container-runtime-design.md +++ /dev/null @@ -1,294 +0,0 @@ -# Container Runtime Management - -## Problem - -OSAPI manages Linux system configuration but has no way to manage containers -running on a host. As containerized workloads become standard, operators need to -create, start, stop, inspect, and execute commands in containers through the -same API and CLI they use for everything else. - -We need: - -1. Container lifecycle management (Docker first, LXD/Podman later) as a new API - domain - -## Decision - -Add a `container` API domain with a pluggable runtime driver interface. Docker -is the first implementation using the Go SDK. - -## Architecture - -### Runtime Driver Interface - -A `runtime.Driver` interface in `internal/provider/container/runtime/` abstracts -container runtime operations. The Docker implementation uses -`github.com/docker/docker/client` to talk to the Docker socket. - -```go -type Driver interface { - Create(ctx context.Context, params CreateParams) (*Container, error) - Start(ctx context.Context, id string) error - Stop(ctx context.Context, id string, timeout *time.Duration) error - Remove(ctx context.Context, id string, force bool) error - List(ctx context.Context, params ListParams) ([]Container, error) - Inspect(ctx context.Context, id string) (*ContainerDetail, error) - Exec(ctx context.Context, id string, params ExecParams) (*ExecResult, error) - Pull(ctx context.Context, image string) (*PullResult, error) -} -``` - -**Types:** - -- `CreateParams` — image (required), name, env vars, port mappings, volumes, - command override, auto-start flag -- `ListParams` — optional filters: state (running/stopped/all), name prefix, - image, limit -- `Container` — ID, name, image, state (running/stopped/created), created - timestamp -- `ContainerDetail` — everything in `Container` plus network settings, port - mappings, mounts, resource limits, health status -- `ExecParams` — command (string slice), env vars, working directory -- `ExecResult` — stdout, stderr, exit code (mirrors `command.Result` pattern) -- `PullResult` — image ID, tag, size - -The Docker implementation lives in -`internal/provider/container/runtime/docker/`. Future drivers (LXD, Podman) -implement the same interface. - -### API Domain - -The `container` domain nests under `/node/{hostname}`, consistent with how disk, -memory, and DNS are scoped to a node. - -| Method | Path | Operation | Permission | -| -------- | --------------------------------------- | --------- | ------------------- | -| `POST` | `/node/{hostname}/container` | Create | `container:write` | -| `GET` | `/node/{hostname}/container` | List | `container:read` | -| `GET` | `/node/{hostname}/container/{id}` | Inspect | `container:read` | -| `POST` | `/node/{hostname}/container/{id}/start` | Start | `container:write` | -| `POST` | `/node/{hostname}/container/{id}/stop` | Stop | `container:write` | -| `DELETE` | `/node/{hostname}/container/{id}` | Remove | `container:write` | -| `POST` | `/node/{hostname}/container/{id}/exec` | Exec | `container:execute` | -| `POST` | `/node/{hostname}/container/pull` | Pull | `container:write` | - -**Permissions:** `container:read`, `container:write`, and `container:execute`. -The `execute` permission is separate from lifecycle management, matching the -precedent set by `command:execute`. - -**Role updates:** - -- `admin` gains `container:read`, `container:write`, `container:execute` -- `write` gains `container:read`, `container:write` -- `read` gains `container:read` - -**Path parameter `{id}`:** The `{id}` parameter accepts a Docker container ID -(hex string or short prefix) or container name. Unlike job and audit IDs which -use `format: uuid`, this parameter uses `type: string` with a `pattern` regex in -the OpenAPI spec to validate the allowed character set -(`[a-zA-Z0-9][a-zA-Z0-9_.-]*`). A custom validator tag is not needed — the -Docker SDK resolves both formats and returns a typed error if the container is -not found. - -**Error responses:** - -- `400` — validation failures on Create (missing image), Exec (missing command), - Pull (missing image), and List (invalid filter values) -- `404` — Inspect, Start, Stop, Remove, Exec when the container ID/name does not - resolve to an existing container -- `409` — Start on an already-running container, Stop on an already-stopped - container -- `500` — Docker daemon errors, socket unreachable - -**Request bodies:** - -Create: - -```json -{ - "image": "ubuntu:24.04", - "name": "my-container", - "command": ["/bin/bash"], - "env": { "FOO": "bar" }, - "ports": [{ "host": 8080, "container": 80 }], - "volumes": [{ "host": "/data", "container": "/mnt/data" }], - "auto_start": true -} -``` - -Exec: - -```json -{ - "command": ["useradd", "testuser"], - "env": { "HOME": "/home/testuser" }, - "working_dir": "/root" -} -``` - -Stop (optional body): - -```json -{ - "timeout": 10 -} -``` - -Remove uses a query parameter: -`DELETE /node/{hostname}/container/{id}?force=true`. No request body. This is -consistent with the existing `DELETE` endpoints in the codebase which carry no -body. - -**Pull is asynchronous.** `POST /node/{hostname}/container/pull` creates a job -and returns a job ID immediately. The pull proceeds in the background on the -agent. Clients poll `GET /job/{id}` for completion, consistent with how all -other state-changing operations work through the job system. Large image pulls -can take minutes; blocking the HTTP response would be unreliable. - -**List query parameters:** - -| Parameter | Type | Description | -| --------- | ------ | --------------------------------------------------- | -| `state` | string | Filter by state: `running`, `stopped`, `all` | -| `limit` | int | Maximum number of containers to return (default 50) | - -### Agent Wiring - -Container operations route through the existing job system. The job category is -`container`, and the operation field matches the endpoint (create, start, stop, -remove, list, inspect, exec, pull). - -- `internal/agent/types.go` — add `containerProvider` field -- `internal/agent/factory.go` — create Docker driver and container service. - Conditional on Docker socket availability: if the socket is not reachable, the - provider is `nil` and container jobs return a descriptive error ("container - runtime not available"). No startup failure. -- `internal/agent/processor.go` — add `container` case to category switch -- `internal/agent/processor_container.go` — dispatch by operation - -### Server Wiring - -Following the existing handler pattern: - -- `internal/api/handler_container.go` — add `GetContainerHandler()` method on - `Server`. Wraps the handler with `NewStrictHandler` + `scopeMiddleware`. No - unauthenticated operations — all container endpoints require auth. -- `internal/api/handler.go` — call `GetContainerHandler()` in - `RegisterHandlers()` and append results -- `internal/api/handler_public_test.go` — add `TestGetContainerHandler` -- `cmd/api_helpers.go` — add `GetContainerHandler()` to the `ServerManager` - interface and call it in `registerAPIHandlers()` -- `cmd/api_server_start.go` — initialize the container handler with the Docker - driver and pass it to `api.New()` - -The `Server` struct does not store handler references as fields. Handlers are -constructed via `GetXxxHandler()` methods and returned as closures, consistent -with all existing domains. - -### Configuration - -No new configuration sections are needed in `osapi.yaml`. The Docker driver -connects to the Docker socket at its default path (`/var/run/docker.sock` on -Linux, the default Docker Desktop socket on macOS). If Docker is not available, -the provider is nil and container operations fail gracefully. - -Future configuration (if needed) could add a `container` section for socket path -overrides, but this is out of scope for the initial implementation. - -### Package Layout - -``` -internal/provider/container/ -├── runtime/ -│ ├── driver.go # Driver interface + types -│ └── docker/ -│ ├── docker.go # Docker SDK implementation -│ └── docker_test.go -├── provider.go # Service struct wrapping Driver -└── types.go # Domain types - -internal/api/container/ -├── gen/ -│ ├── api.yaml # OpenAPI spec -│ ├── cfg.yaml # oapi-codegen config -│ └── generate.go # go:generate directive -├── types.go # Domain struct, interfaces -├── container.go # New(), interface check -├── container_create.go # Create handler -├── container_list.go # List handler -├── container_inspect.go # Inspect handler -├── container_start.go # Start handler -├── container_stop.go # Stop handler -├── container_remove.go # Remove handler -├── container_exec.go # Exec handler -├── container_pull.go # Pull handler -└── *_public_test.go # Tests (unit + HTTP wiring + RBAC) - -internal/api/ -├── handler_container.go # GetContainerHandler() method -├── handler.go # +RegisterHandlers() wiring -└── handler_public_test.go # +TestGetContainerHandler - -cmd/ -├── client_container.go # parent command -├── client_container_create.go # CLI per endpoint -├── client_container_list.go -├── client_container_inspect.go -├── client_container_start.go -├── client_container_stop.go -├── client_container_remove.go -├── client_container_exec.go -└── client_container_pull.go - -pkg/sdk/client/container.go # SDK service wrapper -``` - -### Documentation - -- `docs/docs/sidebar/features/container-management.md` — feature page -- `docs/docs/sidebar/usage/cli/client/container/container.md` — parent CLI page - with `` -- `docs/docs/sidebar/usage/cli/client/container/{operation}.md` — one page per - CLI subcommand -- `docs/docusaurus.config.ts` — add to Features navbar dropdown -- `docs/docs/sidebar/usage/configuration.md` — note that no new config sections - are needed (Docker socket auto-detected) -- `docs/docs/sidebar/architecture/system-architecture.md` — add container - endpoints to the endpoint tables - -### Verification - -```bash -just generate # regenerate specs + code -go build ./... # compiles -just go::unit # tests pass -just go::vet # lint passes -``` - -## Key Design Decisions - -| Decision | Choice | Rationale | -| ----------------------------- | ------------------------------- | ------------------------------------------------------------------------- | -| Runtime driver interface | `runtime.Driver` | Pluggable for Docker now, LXD/Podman later | -| Docker interaction | Go SDK, not CLI | Typed responses, proper error handling, no output parsing | -| API nesting | Under `/node/{hostname}` | Containers run on a node, consistent with existing API conventions | -| Separate `execute` permission | `container:execute` | Running commands in containers is a distinct privilege from lifecycle ops | -| Graceful absence | Nil provider, descriptive error | Agents without Docker still work for all other providers | -| `{id}` parameter | String with pattern, not UUID | Docker IDs are hex strings/names, not UUIDs | -| Remove force flag | Query parameter, no body | Consistent with existing DELETE endpoints | -| Pull behavior | Async via job system | Large pulls can take minutes; blocking HTTP is unreliable | - -## What Was Removed - -The original design included two additional layers that have been dropped: - -- **`provider run` CLI subcommand** — A hidden command to run OSAPI providers - inside containers via `docker exec`. Removed because we are not running OSAPI - inside containers. -- **Orchestrator DSL `In(target)` / `Docker()`** — Scoped plan context that - intercepted SDK client calls and routed them through `docker exec` + - `provider run`. Removed because it depended on `provider run`. - -Container operations are managed through the standard API/CLI/SDK path, the same -as every other OSAPI domain. The orchestrator can compose container operations -with host operations using `TaskFunc` — no special DSL extensions needed. diff --git a/docs/plans/2026-03-11-container-runtime.md b/docs/plans/2026-03-11-container-runtime.md deleted file mode 100644 index 05fe47cb2..000000000 --- a/docs/plans/2026-03-11-container-runtime.md +++ /dev/null @@ -1,1508 +0,0 @@ -# Container Runtime Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> (if subagents available) or superpowers:executing-plans to implement this -> plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add container lifecycle management (Docker) as a new API domain with -create, start, stop, remove, list, inspect, exec, and pull operations. - -**Architecture:** A `runtime.Driver` interface abstracts container runtimes with -Docker as the first implementation via the Go SDK. Container lifecycle is -exposed as a new API domain under `/node/{hostname}/container`. - -**Tech Stack:** Go, Docker Go SDK (`github.com/docker/docker/client`), -oapi-codegen, Echo, testify/suite, NATS JetStream - -**Spec:** `docs/plans/2026-03-11-container-runtime-design.md` - ---- - -## Chunk 1: Runtime Driver Interface and Docker Implementation - -### Task 1: Runtime Driver Interface Types - -**Files:** - -- Create: `internal/provider/container/runtime/driver.go` - -- [ ] **Step 1: Write the driver interface and types** - -Create the `runtime` package with the `Driver` interface and all supporting -types. Reference `internal/provider/command/types.go` for the param/result -struct pattern. - -```go -// Package runtime defines the container runtime driver interface. -package runtime - -import ( - "context" - "time" -) - -// Driver defines container runtime operations. -// Implementations: Docker (now), LXD/Podman (later). -type Driver interface { - Ping(ctx context.Context) error - Create(ctx context.Context, params CreateParams) (*Container, error) - Start(ctx context.Context, id string) error - Stop(ctx context.Context, id string, timeout *time.Duration) error - Remove(ctx context.Context, id string, force bool) error - List(ctx context.Context, params ListParams) ([]Container, error) - Inspect(ctx context.Context, id string) (*ContainerDetail, error) - Exec(ctx context.Context, id string, params ExecParams) (*ExecResult, error) - Pull(ctx context.Context, image string) (*PullResult, error) -} - -// CreateParams contains parameters for container creation. -type CreateParams struct { - // Image is the container image (required). - Image string `json:"image"` - // Name is an optional container name. - Name string `json:"name,omitempty"` - // Command overrides the image's default command. - Command []string `json:"command,omitempty"` - // Env sets environment variables. - Env map[string]string `json:"env,omitempty"` - // Ports maps host ports to container ports. - Ports []PortMapping `json:"ports,omitempty"` - // Volumes maps host paths to container paths. - Volumes []VolumeMapping `json:"volumes,omitempty"` - // AutoStart starts the container after creation. - AutoStart bool `json:"auto_start,omitempty"` -} - -// PortMapping maps a host port to a container port. -type PortMapping struct { - Host int `json:"host"` - Container int `json:"container"` -} - -// VolumeMapping maps a host path to a container path. -type VolumeMapping struct { - Host string `json:"host"` - Container string `json:"container"` -} - -// ListParams contains parameters for listing containers. -type ListParams struct { - // State filters by container state: "running", "stopped", "all". - State string `json:"state,omitempty"` - // Limit caps the number of results. - Limit int `json:"limit,omitempty"` -} - -// Container holds summary info for a container. -type Container struct { - ID string `json:"id"` - Name string `json:"name"` - Image string `json:"image"` - State string `json:"state"` - Created time.Time `json:"created"` -} - -// ContainerDetail holds detailed info for a container. -type ContainerDetail struct { - Container - NetworkSettings *NetworkSettings `json:"network_settings,omitempty"` - Ports []PortMapping `json:"ports,omitempty"` - Mounts []VolumeMapping `json:"mounts,omitempty"` - Health string `json:"health,omitempty"` -} - -// NetworkSettings holds container network configuration. -type NetworkSettings struct { - IPAddress string `json:"ip_address,omitempty"` - Gateway string `json:"gateway,omitempty"` -} - -// ExecParams contains parameters for executing a command in a container. -type ExecParams struct { - // Command is the command and arguments. - Command []string `json:"command"` - // Env sets environment variables. - Env map[string]string `json:"env,omitempty"` - // WorkingDir sets the working directory. - WorkingDir string `json:"working_dir,omitempty"` -} - -// ExecResult contains the output of a command execution in a container. -type ExecResult struct { - Stdout string `json:"stdout"` - Stderr string `json:"stderr"` - ExitCode int `json:"exit_code"` -} - -// PullResult contains the result of an image pull. -type PullResult struct { - ImageID string `json:"image_id"` - Tag string `json:"tag"` - Size int64 `json:"size"` -} -``` - -- [ ] **Step 2: Verify it compiles** - -Run: `go build ./internal/provider/container/runtime/...` Expected: compiles -with no errors - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/container/runtime/driver.go -git commit -m "feat(container): add runtime driver interface and types" -``` - ---- - -### Task 2: Docker Driver Implementation - -**Files:** - -- Create: `internal/provider/container/runtime/docker/docker.go` -- Create: `internal/provider/container/runtime/docker/docker_public_test.go` - -- [ ] **Step 1: Add Docker SDK dependency** - -Run: `go get github.com/docker/docker@latest` - -- [ ] **Step 2: Write failing tests for the Docker driver** - -Create the test suite with table-driven tests. Since the Docker driver talks to -a real Docker socket, tests should use an interface mock or be structured so -they can run against a real Docker daemon in integration. For unit tests, mock -the Docker client interface. - -```go -package docker_test - -import ( - "context" - "testing" - - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/provider/container/runtime" - "github.com/osapi-io/osapi/internal/provider/container/runtime/docker" -) - -type DockerDriverPublicTestSuite struct { - suite.Suite - ctx context.Context - driver runtime.Driver -} - -func (s *DockerDriverPublicTestSuite) SetupTest() { - s.ctx = context.Background() - d, err := docker.New() - s.Require().NoError(err) - s.driver = d -} - -func (s *DockerDriverPublicTestSuite) TestNew() { - tests := []struct { - name string - validateFunc func(d runtime.Driver) - }{ - { - name: "returns non-nil driver", - validateFunc: func(d runtime.Driver) { - s.NotNil(d) - }, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - d, err := docker.New() - s.Require().NoError(err) - tt.validateFunc(d) - }) - } -} - -func TestDockerDriverPublicTestSuite(t *testing.T) { - suite.Run(t, new(DockerDriverPublicTestSuite)) -} -``` - -- [ ] **Step 3: Run test to verify it fails** - -Run: -`go test -run TestDockerDriverPublicTestSuite -v ./internal/provider/container/runtime/docker/...` -Expected: FAIL — `docker` package does not exist - -- [ ] **Step 4: Write the Docker driver implementation** - -Implement the `Driver` interface using `github.com/docker/docker/client`. Each -method maps to the corresponding Docker Engine API call. The constructor accepts -no arguments and creates a client from environment defaults (DOCKER_HOST or -default socket). - -```go -// Package docker implements the runtime.Driver interface using the Docker Engine API. -package docker - -import ( - "bytes" - "context" - "fmt" - "io" - "strings" - "time" - - "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/filters" - "github.com/docker/docker/api/types/image" - "github.com/docker/docker/api/types/mount" - "github.com/docker/docker/api/types/network" - dockerclient "github.com/docker/docker/client" - "github.com/docker/go-connections/nat" - - "github.com/osapi-io/osapi/internal/provider/container/runtime" -) - -// Driver implements runtime.Driver using the Docker Engine API. -type Driver struct { - client dockerclient.APIClient -} - -// New creates a new Docker driver using default client options. -func New() (*Driver, error) { - cli, err := dockerclient.NewClientWithOpts( - dockerclient.FromEnv, - dockerclient.WithAPIVersionNegotiation(), - ) - if err != nil { - return nil, fmt.Errorf("create docker client: %w", err) - } - - return &Driver{client: cli}, nil -} - -// NewWithClient creates a Docker driver with an injected client (for testing). -func NewWithClient( - client dockerclient.APIClient, -) *Driver { - return &Driver{client: client} -} -``` - -Implement each method: `Create`, `Start`, `Stop`, `Remove`, `List`, `Inspect`, -`Exec`, `Pull`. Each translates runtime types to Docker API types and back. -Follow the multi-line function signature convention from CLAUDE.md. - -- [ ] **Step 5: Run tests to verify they pass** - -Run: -`go test -run TestDockerDriverPublicTestSuite -v ./internal/provider/container/runtime/docker/...` -Expected: PASS - -- [ ] **Step 6: Verify compilation** - -Run: `go build ./internal/provider/container/runtime/...` Expected: compiles -with no errors - -- [ ] **Step 7: Commit** - -```bash -git add internal/provider/container/runtime/docker/ -git commit -m "feat(container): add Docker runtime driver implementation" -``` - ---- - -### Task 3: Container Provider Service - -**Files:** - -- Create: `internal/provider/container/provider.go` -- Create: `internal/provider/container/types.go` -- Create: `internal/provider/container/provider_public_test.go` - -- [ ] **Step 1: Write the provider types** - -The provider wraps a `runtime.Driver` and presents a domain interface that the -agent and API handler can call. Reference `internal/api/node/types.go` for the -struct pattern. - -```go -// Package container provides the container management provider. -package container - -import ( - "github.com/osapi-io/osapi/internal/provider/container/runtime" -) - -// Provider defines the container management interface. -// All methods accept context.Context for cancellation and timeout propagation, -// which is important since the Docker daemon is a remote service. -type Provider interface { - Create(ctx context.Context, params runtime.CreateParams) (*runtime.Container, error) - Start(ctx context.Context, id string) error - Stop(ctx context.Context, id string, timeout *time.Duration) error - Remove(ctx context.Context, id string, force bool) error - List(ctx context.Context, params runtime.ListParams) ([]runtime.Container, error) - Inspect(ctx context.Context, id string) (*runtime.ContainerDetail, error) - Exec(ctx context.Context, id string, params runtime.ExecParams) (*runtime.ExecResult, error) - Pull(ctx context.Context, image string) (*runtime.PullResult, error) -} -``` - -- [ ] **Step 2: Write failing tests** - -```go -package container_test - -import ( - "testing" - - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/provider/container" - "github.com/osapi-io/osapi/internal/provider/container/runtime" -) - -type ProviderPublicTestSuite struct { - suite.Suite -} - -func (s *ProviderPublicTestSuite) TestNew() { - tests := []struct { - name string - validateFunc func(p container.Provider) - }{ - { - name: "returns non-nil provider", - validateFunc: func(p container.Provider) { - s.NotNil(p) - }, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - var driver runtime.Driver // nil driver for unit test - p := container.New(driver) - tt.validateFunc(p) - }) - } -} - -func TestProviderPublicTestSuite(t *testing.T) { - suite.Run(t, new(ProviderPublicTestSuite)) -} -``` - -- [ ] **Step 3: Run test to verify it fails** - -Run: -`go test -run TestProviderPublicTestSuite -v ./internal/provider/container/...` -Expected: FAIL — `container.New` not defined - -- [ ] **Step 4: Write the provider implementation** - -```go -package container - -import ( - "context" - "time" - - "github.com/osapi-io/osapi/internal/provider/container/runtime" -) - -// Service implements Provider by delegating to a runtime.Driver. -type Service struct { - driver runtime.Driver -} - -// New creates a new container provider service. -func New( - driver runtime.Driver, -) *Service { - return &Service{driver: driver} -} -``` - -Implement each method, delegating to `s.driver` with a background context. Each -method is a thin pass-through that creates a context and calls the driver. - -- [ ] **Step 5: Run tests to verify they pass** - -Run: -`go test -run TestProviderPublicTestSuite -v ./internal/provider/container/...` -Expected: PASS - -- [ ] **Step 6: Commit** - -```bash -git add internal/provider/container/ -git commit -m "feat(container): add container provider service" -``` - ---- - -## Chunk 2: Job System Integration - -### Task 4: Job Types and Operation Constants - -**Files:** - -- Modify: `internal/job/types.go` -- Modify: `internal/job/subjects.go` - -- [ ] **Step 1: Add container data types to job/types.go** - -Add data structs for container operations, following the `CommandExecData` -pattern at `internal/job/types.go:229-239`. - -```go -// ContainerCreateData represents data for container creation. -type ContainerCreateData struct { - Image string `json:"image"` - Name string `json:"name,omitempty"` - Command []string `json:"command,omitempty"` - Env map[string]string `json:"env,omitempty"` - Ports []PortMapping `json:"ports,omitempty"` - Volumes []VolumeMapping `json:"volumes,omitempty"` - AutoStart bool `json:"auto_start,omitempty"` -} - -// PortMapping maps a host port to a container port (job layer). -// Intentionally duplicated from runtime.PortMapping to keep the job -// layer decoupled from the provider layer. Both have the same shape. -type PortMapping struct { - Host int `json:"host"` - Container int `json:"container"` -} - -// VolumeMapping maps a host path to a container path (job layer). -// Intentionally duplicated from runtime.VolumeMapping for the same reason. -type VolumeMapping struct { - Host string `json:"host"` - Container string `json:"container"` -} - -// ContainerStopData represents data for stopping a container. -type ContainerStopData struct { - Timeout *int `json:"timeout,omitempty"` -} - -// ContainerRemoveData represents data for removing a container. -type ContainerRemoveData struct { - Force bool `json:"force,omitempty"` -} - -// ContainerListData represents data for listing containers. -type ContainerListData struct { - State string `json:"state,omitempty"` - Limit int `json:"limit,omitempty"` -} - -// ContainerExecData represents data for executing a command in a container. -type ContainerExecData struct { - Command []string `json:"command"` - Env map[string]string `json:"env,omitempty"` - WorkingDir string `json:"working_dir,omitempty"` -} - -// ContainerPullData represents data for pulling an image. -type ContainerPullData struct { - Image string `json:"image"` -} -``` - -- [ ] **Step 2: Add container operation constants** - -Find the existing operation constants (e.g., `OperationCommandExecExecute`) and -add container equivalents: - -```go -// Container operation types -const ( - OperationContainerCreate = "container.create" - OperationContainerStart = "container.start" - OperationContainerStop = "container.stop" - OperationContainerRemove = "container.remove" - OperationContainerList = "container.list" - OperationContainerInspect = "container.inspect" - OperationContainerExec = "container.exec" - OperationContainerPull = "container.pull" -) -``` - -- [ ] **Step 3: Add `SubjectCategoryContainer` constant to subjects.go** - -```go -const ( - SubjectCategoryContainer = "container" -) -``` - -- [ ] **Step 4: Verify compilation** - -Run: `go build ./internal/job/...` Expected: compiles - -- [ ] **Step 5: Commit** - -```bash -git add internal/job/types.go internal/job/subjects.go -git commit -m "feat(container): add container job types and operation constants" -``` - ---- - -### Task 5: Job Client Container Methods - -**Files:** - -- Create: `internal/job/client/modify_container.go` -- Create: `internal/job/client/modify_container_public_test.go` -- Modify: `internal/job/client/types.go` (add methods to `JobClient` interface) - -- [ ] **Step 1: Add container methods to the JobClient interface** - -Open `internal/job/client/types.go` and add methods for each container -operation. Follow the existing pattern from `ModifyCommandExec`. - -- [ ] **Step 2: Write failing tests** - -Create `internal/job/client/modify_container_public_test.go` following the same -table-driven suite pattern as -`internal/job/client/modify_command_public_test.go`. Test the `Create` method -first as the representative case — success, job failure, and publish error. - -- [ ] **Step 3: Run test to verify it fails** - -Run: -`go test -run TestModifyContainerPublicTestSuite -v ./internal/job/client/...` -Expected: FAIL — methods not implemented - -- [ ] **Step 4: Implement the job client container methods** - -Create `internal/job/client/modify_container.go`. Each method marshals the -appropriate data struct, builds a `job.Request` with category `"container"` and -the correct operation constant, then calls `publishAndWait`. Follow the pattern -from `internal/job/client/modify_command.go:33-62`. - -Implement: `ModifyContainerCreate`, `ModifyContainerStart`, -`ModifyContainerStop`, `ModifyContainerRemove`, `QueryContainerList`, -`QueryContainerInspect`, `ModifyContainerExec`, `ModifyContainerPull`. - -Note: `List` and `Inspect` are query operations (`job.TypeQuery`), while all -others are modify operations (`job.TypeModify`). - -- [ ] **Step 5: Run tests to verify they pass** - -Run: -`go test -run TestModifyContainerPublicTestSuite -v ./internal/job/client/...` -Expected: PASS - -- [ ] **Step 6: Commit** - -```bash -git add internal/job/client/modify_container.go \ - internal/job/client/modify_container_public_test.go \ - internal/job/client/types.go -git commit -m "feat(container): add container job client methods" -``` - ---- - -### Task 6: Agent Processor Container Dispatch - -**Files:** - -- Modify: `internal/agent/types.go` -- Modify: `internal/agent/factory.go` -- Modify: `internal/agent/processor.go` -- Create: `internal/agent/processor_container.go` -- Create: `internal/agent/processor_container_test.go` - -- [ ] **Step 1: Add `containerProvider` field to Agent struct** - -In `internal/agent/types.go`, add: - -```go -// Container provider -containerProvider containerProv.Provider -``` - -Add the import for the container provider package. - -- [ ] **Step 2: Update the factory to create container provider** - -In `internal/agent/factory.go`, add Docker driver creation. The factory should -check Docker socket availability and return `nil` if unavailable: - -```go -// Create container provider (conditional on Docker availability) -var containerProvider containerProv.Provider -dockerDriver, err := docker.New() -if err == nil { - if pingErr := dockerDriver.Ping(context.Background()); pingErr == nil { - containerProvider = containerProv.New(dockerDriver) - } else { - f.logger.Info("Docker not available, container operations disabled", - slog.String("error", pingErr.Error())) - } -} else { - f.logger.Info("Docker client creation failed, container operations disabled", - slog.String("error", err.Error())) -} -``` - -Update `CreateProviders` return signature to include the container provider as -the 9th return value. - -**IMPORTANT: This changes the return signature.** The following callers must -also be updated in this step: - -- `internal/agent/agent.go` — the `New()` constructor must accept a - `containerProv.Provider` parameter and assign it to the struct field -- `cmd/agent_helpers.go` or wherever `CreateProviders()` is called — update the - call site to capture the 9th return value and pass it to `agent.New()` -- All existing test files that call `agent.New()` — pass `nil` as the container - provider parameter. Affected files include: - - `internal/agent/processor_command_test.go` - - `internal/agent/processor_file_test.go` - - `internal/agent/processor_test.go` - - Any other files in `internal/agent/` that construct an `Agent` - -Search for all usages with: `grep -r "agent.New(" internal/ cmd/` - -- [ ] **Step 3: Add container case to processor dispatch** - -In `internal/agent/processor.go`, add to the category switch in -`processJobOperation`: - -```go -case "container": - return a.processContainerOperation(jobRequest) -``` - -- [ ] **Step 4: Write failing test for container processor** - -Create `internal/agent/processor_container_test.go` with `package agent` -(internal test — matches existing `processor_command_test.go` pattern since -`processContainerOperation` is unexported). Use a table-driven test for -`processContainerOperation`. Test the nil-provider case (returns "container -runtime not available" error) and a successful dispatch case with a mock -provider. - -- [ ] **Step 5: Run test to verify it fails** - -Run: `go test -run TestProcessContainerOperation -v ./internal/agent/...` -Expected: FAIL — `processContainerOperation` not defined - -- [ ] **Step 6: Implement container processor** - -Create `internal/agent/processor_container.go`: - -Note: The existing processor dispatch (`processJobOperation`) does not pass -context to sub-processors. Since the container provider is the first to require -`context.Context`, the processor chain needs context propagation. Either thread -`context.Context` through from `processJobOperation` or use -`context.Background()` as a starting point (matching the existing pattern where -processors don't receive context). The preferred approach is to add context to -`processContainerOperation` and update the dispatch call: - -In `processor.go`, change the container case to: - -```go -case "container": - return a.processContainerOperation(ctx, jobRequest) -``` - -Where `ctx` is derived from the handler's context (check how `handleJobMessage` -creates/receives its context). - -```go -package agent - -import ( - "context" - "encoding/json" - "fmt" - - "github.com/osapi-io/osapi/internal/job" - "github.com/osapi-io/osapi/internal/provider/container/runtime" -) - -func (a *Agent) processContainerOperation( - ctx context.Context, - jobRequest job.Request, -) (json.RawMessage, error) { - if a.containerProvider == nil { - return nil, fmt.Errorf("container runtime not available") - } - - switch jobRequest.Operation { - case job.OperationContainerCreate: - var data job.ContainerCreateData - if err := json.Unmarshal(jobRequest.Data, &data); err != nil { - return nil, fmt.Errorf("unmarshal create data: %w", err) - } - result, err := a.containerProvider.Create(ctx, runtime.CreateParams{ - Image: data.Image, - Name: data.Name, - Command: data.Command, - Env: data.Env, - AutoStart: data.AutoStart, - // Map ports and volumes from job types to runtime types - }) - if err != nil { - return nil, err - } - return json.Marshal(result) - - case job.OperationContainerStart: - // jobRequest.Data contains the container ID as a string - var id string - if err := json.Unmarshal(jobRequest.Data, &id); err != nil { - return nil, fmt.Errorf("unmarshal start data: %w", err) - } - return nil, a.containerProvider.Start(ctx, id) - - // ... remaining operations follow the same pattern, always passing ctx - - default: - return nil, fmt.Errorf("unsupported container operation: %s", jobRequest.Operation) - } -} -``` - -Implement all eight operation cases. - -- [ ] **Step 7: Run tests to verify they pass** - -Run: `go test -run TestProcessContainerOperation -v ./internal/agent/...` -Expected: PASS - -- [ ] **Step 8: Run full test suite** - -Run: `just go::unit` Expected: all tests pass (no regressions from agent -changes) - -- [ ] **Step 9: Commit** - -```bash -git add internal/agent/types.go internal/agent/factory.go \ - internal/agent/processor.go internal/agent/processor_container.go \ - internal/agent/processor_container_test.go -git commit -m "feat(container): add agent container processor dispatch" -``` - ---- - -## Chunk 3: OpenAPI Spec and API Handlers - -### Task 7: OpenAPI Specification - -**Files:** - -- Create: `internal/api/container/gen/api.yaml` -- Create: `internal/api/container/gen/cfg.yaml` -- Create: `internal/api/container/gen/generate.go` - -- [ ] **Step 1: Write the OpenAPI spec** - -Create `internal/api/container/gen/api.yaml`. Model it after -`internal/api/node/gen/api.yaml`. Key differences: - -- All paths under `/node/{hostname}/container` -- `{id}` parameter uses `type: string` with - `pattern: ^[a-zA-Z0-9][a-zA-Z0-9_.-]*$` (not `format: uuid`) -- Security scopes: `container:read`, `container:write`, `container:execute` -- Request body validation via `x-oapi-codegen-extra-tags` -- Error responses: 400, 401, 403, 404, 409, 500 per the spec - -Define these paths: - -- `POST /node/{hostname}/container` — CreateContainer -- `GET /node/{hostname}/container` — ListContainers -- `GET /node/{hostname}/container/{id}` — InspectContainer -- `POST /node/{hostname}/container/{id}/start` — StartContainer -- `POST /node/{hostname}/container/{id}/stop` — StopContainer -- `DELETE /node/{hostname}/container/{id}` — RemoveContainer (force as query - param) -- `POST /node/{hostname}/container/{id}/exec` — ExecContainer -- `POST /node/{hostname}/container/pull` — PullImage - -Define schemas: `ContainerCreateRequest`, `ContainerExecRequest`, -`ContainerStopRequest`, `ContainerPullRequest`, `ContainerResponse`, -`ContainerDetailResponse`, `ContainerListResponse`, `ContainerExecResponse`, -`ContainerResultCollectionResponse`. - -Use `x-oapi-codegen-extra-tags` for validation: - -```yaml -properties: - image: - type: string - x-oapi-codegen-extra-tags: - validate: required,min=1 - command: - type: array - items: - type: string - x-oapi-codegen-extra-tags: - validate: required,min=1 -``` - -- [ ] **Step 2: Write the codegen config** - -Create `internal/api/container/gen/cfg.yaml`: - -```yaml ---- -package: gen -output: container.gen.go -generate: - models: true - echo-server: true - strict-server: true -import-mapping: - ../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/api/common/gen -output-options: - skip-prune: true -``` - -- [ ] **Step 3: Write the generate directive** - -Create `internal/api/container/gen/generate.go`: - -```go -// Package gen contains generated code for the container API. -package gen - -//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen -config cfg.yaml api.yaml -``` - -- [ ] **Step 4: Run code generation** - -Run: `go generate ./internal/api/container/gen/...` Expected: `container.gen.go` -is generated with no errors - -- [ ] **Step 5: Verify compilation** - -Run: `go build ./internal/api/container/...` Expected: compiles - -- [ ] **Step 6: Commit** - -```bash -git add internal/api/container/gen/ -git commit -m "feat(container): add OpenAPI spec and code generation" -``` - ---- - -### Task 8: Container API Handler - -**Files:** - -- Create: `internal/api/container/types.go` -- Create: `internal/api/container/container.go` -- Create: `internal/api/container/container_create.go` -- Create: `internal/api/container/container_create_public_test.go` -- Create: `internal/api/container/container_list.go` -- Create: `internal/api/container/container_list_public_test.go` -- Create: `internal/api/container/container_inspect.go` -- Create: `internal/api/container/container_inspect_public_test.go` -- Create: `internal/api/container/container_start.go` -- Create: `internal/api/container/container_start_public_test.go` -- Create: `internal/api/container/container_stop.go` -- Create: `internal/api/container/container_stop_public_test.go` -- Create: `internal/api/container/container_remove.go` -- Create: `internal/api/container/container_remove_public_test.go` -- Create: `internal/api/container/container_exec.go` -- Create: `internal/api/container/container_exec_public_test.go` -- Create: `internal/api/container/container_pull.go` -- Create: `internal/api/container/container_pull_public_test.go` - -This task implements one handler at a time, TDD-style. The `create` handler is -shown in detail as the template; remaining handlers follow the same pattern. - -- [ ] **Step 1: Write types.go** - -```go -package container - -import ( - "log/slog" - - "github.com/osapi-io/osapi/internal/job/client" -) - -// Container implementation of the Container APIs operations. -type Container struct { - JobClient client.JobClient - logger *slog.Logger -} -``` - -- [ ] **Step 2: Write container.go** - -```go -// Package container provides container management API handlers. -package container - -import ( - "log/slog" - - "github.com/osapi-io/osapi/internal/api/container/gen" - "github.com/osapi-io/osapi/internal/job/client" -) - -var _ gen.StrictServerInterface = (*Container)(nil) - -// New factory to create a new instance. -func New( - logger *slog.Logger, - jobClient client.JobClient, -) *Container { - return &Container{ - JobClient: jobClient, - logger: logger, - } -} -``` - -- [ ] **Step 3: Write failing test for CreateContainer** - -Create `container_create_public_test.go`. Follow the pattern from -`internal/api/node/command_exec_post_public_test.go`: - -- Test validation failure (missing image → 400) -- Test success (202 with job ID and container info) -- Test error (500 from job client) -- TestCreateContainerHTTP (raw HTTP through Echo stack) -- TestCreateContainerRBACHTTP (401/403/200) - -- [ ] **Step 4: Run test to verify it fails** - -Run: -`go test -run TestContainerCreatePublicTestSuite -v ./internal/api/container/...` -Expected: FAIL - -- [ ] **Step 5: Implement CreateContainer handler** - -Follow `internal/api/node/command_exec_post.go` as the template: - -- Validate hostname with `validateHostname` -- Validate request body with `validation.Struct` -- Call `s.JobClient.ModifyContainerCreate` -- Return 202 with job ID and result - -- [ ] **Step 6: Run test to verify it passes** - -Run: -`go test -run TestContainerCreatePublicTestSuite -v ./internal/api/container/...` -Expected: PASS - -- [ ] **Step 7: Commit create handler** - -```bash -git add internal/api/container/types.go internal/api/container/container.go \ - internal/api/container/container_create.go \ - internal/api/container/container_create_public_test.go -git commit -m "feat(container): add create container handler" -``` - -- [ ] **Step 8: Implement remaining handlers (TDD cycle each)** - -For each of: `list`, `inspect`, `start`, `stop`, `remove`, `exec`, `pull`: - -1. Write the failing test file -2. Run test → FAIL -3. Write the handler -4. Run test → PASS -5. Verify coverage: - `go test -coverprofile=cover.out ./internal/api/container/ && go tool cover -func=cover.out | grep -v '100.0%'` - Expected: no uncovered lines in the new handler file -6. Commit - -Commit message pattern: `feat(container): add {operation} container handler` - -- [ ] **Step 9: Run full test suite** - -Run: `just go::unit` Expected: all tests pass - -- [ ] **Step 10: Commit if any remaining files** - ---- - -## Chunk 4: Server Wiring and Permissions - -### Task 9: Permissions - -**Files:** - -- Modify: `internal/authtoken/permissions.go` - -- [ ] **Step 1: Add container permission constants** - -Add to `internal/authtoken/permissions.go`: - -```go -PermContainerRead Permission = "container:read" -PermContainerWrite Permission = "container:write" -PermContainerExecute Permission = "container:execute" -``` - -- [ ] **Step 2: Add to AllPermissions slice** - -Append the three new permissions. - -- [ ] **Step 3: Update DefaultRolePermissions** - -- `admin`: add `PermContainerRead`, `PermContainerWrite`, `PermContainerExecute` -- `write`: add `PermContainerRead`, `PermContainerWrite` -- `read`: add `PermContainerRead` - -- [ ] **Step 4: Run existing auth tests** - -Run: `go test -v ./internal/authtoken/...` Expected: PASS (or update any tests -that assert on the full permission set) - -- [ ] **Step 5: Commit** - -```bash -git add internal/authtoken/permissions.go -git commit -m "feat(container): add container permissions and role mappings" -``` - ---- - -### Task 10: Server Handler Wiring - -**Files:** - -- Create: `internal/api/handler_container.go` -- Modify: `internal/api/handler_public_test.go` -- Modify: `cmd/api_helpers.go` -- Modify: `cmd/api_server_start.go` - -Note: `internal/api/handler.go` does NOT need modification. It only contains the -`RegisterHandlers()` pass-through method. Handler wiring happens in -`cmd/api_helpers.go:registerAPIHandlers()`. - -- [ ] **Step 1: Write failing test for GetContainerHandler** - -Add to `internal/api/handler_public_test.go`, following the -`TestGetHealthHandler` pattern: - -```go -func (s *HandlerPublicTestSuite) TestGetContainerHandler() { - tests := []struct { - name string - validate func([]func(e *echo.Echo)) - }{ - { - name: "returns container handler functions", - validate: func(handlers []func(e *echo.Echo)) { - s.NotEmpty(handlers) - }, - }, - { - name: "closure registers routes and middleware executes", - validate: func(handlers []func(e *echo.Echo)) { - e := echo.New() - for _, h := range handlers { - h(e) - } - s.NotEmpty(e.Routes()) - }, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - handlers := s.server.GetContainerHandler(s.mockJobClient) - tt.validate(handlers) - }) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test -run TestGetContainerHandler -v ./internal/api/...` Expected: FAIL -— `GetContainerHandler` not defined - -- [ ] **Step 3: Implement GetContainerHandler** - -Create `internal/api/handler_container.go`. Follow -`internal/api/handler_node.go` exactly — no unauthenticated operations: - -```go -package api - -import ( - "github.com/labstack/echo/v4" - strictecho "github.com/oapi-codegen/runtime/strictmiddleware/echo" - - "github.com/osapi-io/osapi/internal/api/container" - containerGen "github.com/osapi-io/osapi/internal/api/container/gen" - "github.com/osapi-io/osapi/internal/authtoken" - "github.com/osapi-io/osapi/internal/job/client" -) - -// GetContainerHandler returns container handler for registration. -func (s *Server) GetContainerHandler( - jobClient client.JobClient, -) []func(e *echo.Echo) { - var tokenManager TokenValidator = authtoken.New(s.logger) - - containerHandler := container.New(s.logger, jobClient) - - strictHandler := containerGen.NewStrictHandler( - containerHandler, - []containerGen.StrictMiddlewareFunc{ - func(handler strictecho.StrictEchoHandlerFunc, _ string) strictecho.StrictEchoHandlerFunc { - return scopeMiddleware( - handler, - tokenManager, - s.appConfig.API.Server.Security.SigningKey, - containerGen.BearerAuthScopes, - s.customRoles, - ) - }, - }, - ) - - return []func(e *echo.Echo){ - func(e *echo.Echo) { - containerGen.RegisterHandlers(e, strictHandler) - }, - } -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `go test -run TestGetContainerHandler -v ./internal/api/...` Expected: PASS - -- [ ] **Step 5: Add to ServerManager interface** - -In `cmd/api_helpers.go`, add to the `ServerManager` interface: - -```go -// GetContainerHandler returns container handler for registration. -GetContainerHandler(jobClient jobclient.JobClient) []func(e *echo.Echo) -``` - -- [ ] **Step 6: Wire into registerAPIHandlers** - -In `cmd/api_helpers.go`, add to `registerAPIHandlers`: - -```go -handlers = append(handlers, sm.GetContainerHandler(jc)...) -``` - -- [ ] **Step 7: Wire startup dependencies** - -In `cmd/api_server_start.go`, ensure the container handler is passed. Since -`GetContainerHandler` takes only a `jobClient` (same as `GetNodeHandler`), no -new dependencies are needed — it's already wired through the existing -`registerAPIHandlers` call after step 6. - -- [ ] **Step 8: Regenerate combined spec** - -Run: `just generate` Expected: combined spec at `internal/api/gen/api.yaml` -includes container paths - -- [ ] **Step 9: Verify full build** - -Run: `go build ./...` Expected: compiles - -- [ ] **Step 10: Run full test suite** - -Run: `just go::unit` Expected: all tests pass - -- [ ] **Step 11: Commit** - -```bash -git add internal/api/handler_container.go internal/api/handler_public_test.go \ - cmd/api_helpers.go cmd/api_server_start.go -git commit -m "feat(container): wire container handler into API server" -``` - ---- - -## Chunk 5: SDK Client and CLI Commands - -### Task 11: SDK Client Container Service - -**Files:** - -- Create: `pkg/sdk/client/container.go` -- Modify: `pkg/sdk/client/client.go` (add Container field) - -- [ ] **Step 1: Regenerate SDK client from combined spec** - -Run: `go generate ./pkg/sdk/client/gen/...` Expected: SDK client picks up new -container endpoints - -- [ ] **Step 2: Write the ContainerService** - -Create `pkg/sdk/client/container.go`. Follow `pkg/sdk/client/health.go` for the -service wrapper pattern. Each method calls the generated client method, handles -response codes (200/202, 400, 401, 403, 404, 409, 500), and returns typed -results. - -Methods: `Create`, `List`, `Inspect`, `Start`, `Stop`, `Remove`, `Exec`, `Pull`. - -- [ ] **Step 3: Add Container field to Client** - -In `pkg/sdk/client/client.go`, add: - -```go -Container *ContainerService -``` - -Initialize it in the constructor. - -- [ ] **Step 4: Verify compilation** - -Run: `go build ./pkg/sdk/...` Expected: compiles - -- [ ] **Step 5: Commit** - -```bash -git add pkg/sdk/client/container.go pkg/sdk/client/client.go \ - pkg/sdk/client/gen/ -git commit -m "feat(container): add SDK container service" -``` - ---- - -### Task 12: CLI Commands - -**Files:** - -- Create: `cmd/client_container.go` -- Create: `cmd/client_container_create.go` -- Create: `cmd/client_container_list.go` -- Create: `cmd/client_container_inspect.go` -- Create: `cmd/client_container_start.go` -- Create: `cmd/client_container_stop.go` -- Create: `cmd/client_container_remove.go` -- Create: `cmd/client_container_exec.go` -- Create: `cmd/client_container_pull.go` - -- [ ] **Step 1: Write parent command** - -Create `cmd/client_container.go`. Follow `cmd/client_health.go`: - -```go -package cmd - -import ( - "github.com/spf13/cobra" -) - -var clientContainerCmd = &cobra.Command{ - Use: "container", - Short: "Container management operations", - Long: `Manage containers on target nodes.`, -} - -func init() { - clientCmd.AddCommand(clientContainerCmd) -} -``` - -- [ ] **Step 2: Write create subcommand** - -Create `cmd/client_container_create.go`. Follow `cmd/client_health_status.go` -for the pattern. Use flags: `--target` (required), `--image` (required), -`--name`, `--env`, `--port`, `--volume`, `--auto-start`, `--json`. - -Handle all response codes in the switch block: 202, 400 (`handleUnknownError`), -401/403 (`handleAuthError`), 500 (`handleUnknownError`). - -- [ ] **Step 3: Write remaining subcommands** - -For each operation: `list`, `inspect`, `start`, `stop`, `remove`, `exec`, -`pull`. Each subcommand: - -- Registers under `clientContainerCmd` -- Uses flags (e.g., `--id` for operations on a specific container, `--target` - for node targeting) -- Supports `--json` for raw output -- Handles all API response codes - -Commit message pattern per batch: - -```bash -git commit -m "feat(container): add container CLI commands" -``` - -- [ ] **Step 4: Verify build** - -Run: `go build ./...` Expected: compiles - -- [ ] **Step 5: Verify help output** - -Run: `go run main.go client container --help` Expected: shows subcommands -(create, list, inspect, start, stop, remove, exec, pull) - -- [ ] **Step 6: Commit** - -```bash -git add cmd/client_container*.go -git commit -m "feat(container): add container CLI commands" -``` - ---- - -## Chunk 6: Documentation and Verification - -### Task 13: Documentation - -**Files:** - -- Create: `docs/docs/sidebar/features/container-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/container.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/create.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/inspect.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/start.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/stop.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/remove.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/exec.md` -- Create: `docs/docs/sidebar/usage/cli/client/container/pull.md` -- Modify: `docs/docusaurus.config.ts` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` - -- [ ] **Step 1: Write feature page** - -Create `docs/docs/sidebar/features/container-management.md`. Follow the template -from existing feature pages in the `features/` directory. - -- [ ] **Step 2: Write CLI documentation pages** - -Create the parent page with `` and one page per CLI subcommand -with usage examples and `--json` output. - -- [ ] **Step 3: Update docusaurus.config.ts** - -Add "Container Management" to the Features navbar dropdown. - -- [ ] **Step 4: Update configuration.md** - -Add a note that no new configuration sections are needed for container -management (Docker socket is auto-detected). Also update the Permissions table -to include `container:read`, `container:write`, `container:execute` and update -the role mappings table to show which roles get which container permissions. - -- [ ] **Step 5: Update system-architecture.md** - -Add container endpoints to the endpoint tables. - -- [ ] **Step 6: Check docs formatting** - -Run: `just docs::fmt-check` Expected: passes (or fix formatting) - -- [ ] **Step 7: Commit** - -```bash -git add docs/ -git commit -m "docs: add container management documentation" -``` - ---- - -### Task 14: Integration Test Smoke Suite - -**Files:** - -- Create: `test/integration/container_test.go` - -Note: Integration tests require a running Docker daemon. They are guarded by -`//go:build integration` and run with `just go::unit-int`. This task creates a -minimal smoke test. Write tests (mutations) must be guarded by -`skipWrite(s.T())`. - -- [ ] **Step 1: Create integration test file** - -Follow existing patterns in `test/integration/`. The test should: - -- Create a container (`POST /node/{hostname}/container`) -- List containers and verify it appears -- Inspect the container -- Exec a command inside it -- Stop and remove the container -- Pull a known small image (e.g., `alpine:latest`) - -- [ ] **Step 2: Commit** - -```bash -git add test/integration/container_test.go -git commit -m "test(container): add integration test smoke suite" -``` - ---- - -### Task 15: Final Verification - -- [ ] **Step 1: Regenerate all specs and code** - -Run: `just generate` Expected: no errors, all generated files up to date - -- [ ] **Step 2: Build** - -Run: `go build ./...` Expected: compiles - -- [ ] **Step 3: Run all unit tests** - -Run: `just go::unit` Expected: all tests pass - -- [ ] **Step 4: Run linter** - -Run: `just go::vet` Expected: no lint errors - -- [ ] **Step 5: Run full test suite with coverage** - -Run: `just test` Expected: all checks pass (lint + unit + coverage) - -- [ ] **Step 6: Verify new packages have 100% coverage** - -Run coverage and check that every new package has 100% line coverage: - -```bash -go test -race -coverprofile=.coverage/cover.out -v ./... -grep -v -f .coverignore .coverage/cover.out > .coverage/cover.tmp && mv .coverage/cover.tmp .coverage/cover.out -go tool cover -func=.coverage/cover.out | grep 'container' | grep -v '100.0%' -``` - -Expected: **no output** (all container packages at 100%). - -If any lines are uncovered, add tests before proceeding. The `.coverignore` -already excludes `/cmd/`, `/gen/`, `main.go`, and `/mocks/`, so handler tests in -`internal/api/container/` and provider tests in `internal/provider/container/` -are what matter. - -Also verify that overall project coverage did not decrease by comparing with the -Codecov baseline. Run: - -```bash -go tool cover -func=.coverage/cover.out | tail -1 -``` - -This shows the total coverage percentage. It should be at or above the -pre-existing level. - -- [ ] **Step 7: Commit any formatting fixes** - -If `just go::fmt` produces changes: - -```bash -just go::fmt -git add -u -git commit -m "style: format container code" -``` diff --git a/docs/plans/2026-03-13-container-runtime-rename-design.md b/docs/plans/2026-03-13-container-runtime-rename-design.md deleted file mode 100644 index 35b44359e..000000000 --- a/docs/plans/2026-03-13-container-runtime-rename-design.md +++ /dev/null @@ -1,190 +0,0 @@ -# Container Runtime: Docker-Specific Domain Design - -## Problem - -The current container domain uses a generic `container` name with an -auto-detecting runtime driver. This is wrong — the user decides which runtime to -use, not the agent. Docker, LXD, and Podman are fundamentally different systems -with different concepts, options, and behaviors. A shared abstraction would be -lowest-common-denominator or leak everywhere. - -## Decision - -Rename the `container` domain to `docker`. Each future runtime (LXD, Podman) -becomes its own independent domain — no shared interface, no shared types, no -abstraction tax. - -The CLI groups runtimes under a `container` parent command for discoverability. -API paths mirror this with `/container/docker/`. - -## Architecture - -### Naming Convention - -| Layer | Current | New | -| ------------ | ------------------------------ | ----------------------------------- | -| API paths | `/node/{hostname}/container` | `/node/{hostname}/container/docker` | -| Permissions | `container:read/write/execute` | `docker:read/write/execute` | -| CLI | `client container list` | `client container docker list` | -| SDK | `client.Container.Pull()` | `client.Docker.Pull()` | -| Job category | `container` | `docker` | -| Provider pkg | `internal/provider/container/` | `internal/provider/docker/` | -| API pkg | `internal/api/container/` | `internal/api/docker/` | - -### API Endpoints - -| Method | Path | Operation | Permission | -| -------- | ---------------------------------------------- | --------- | ---------------- | -| `POST` | `/node/{hostname}/container/docker` | Create | `docker:write` | -| `GET` | `/node/{hostname}/container/docker` | List | `docker:read` | -| `GET` | `/node/{hostname}/container/docker/{id}` | Inspect | `docker:read` | -| `POST` | `/node/{hostname}/container/docker/{id}/start` | Start | `docker:write` | -| `POST` | `/node/{hostname}/container/docker/{id}/stop` | Stop | `docker:write` | -| `DELETE` | `/node/{hostname}/container/docker/{id}` | Remove | `docker:write` | -| `POST` | `/node/{hostname}/container/docker/{id}/exec` | Exec | `docker:execute` | -| `POST` | `/node/{hostname}/container/docker/pull` | Pull | `docker:write` | - -### CLI - -``` -osapi client container docker list [--target HOST] [--state STATE] [--limit N] -osapi client container docker create --target HOST --image IMAGE [--name NAME] ... -osapi client container docker inspect --target HOST --id ID -osapi client container docker start --target HOST --id ID -osapi client container docker stop --target HOST --id ID [--timeout SECONDS] -osapi client container docker remove --target HOST --id ID [--force] -osapi client container docker exec --target HOST --id ID --command CMD... -osapi client container docker pull --target HOST --image IMAGE -``` - -The `container` command is a parent with `` for grouping. Each -runtime is a subcommand. Future runtimes add `client container lxd`, -`client container podman`, etc. - -### Role Updates - -| Role | Permissions | -| ------- | ----------------------------------------------- | -| `admin` | `docker:read`, `docker:write`, `docker:execute` | -| `write` | `docker:read`, `docker:write` | -| `read` | `docker:read` | - -### Package Layout - -``` -internal/provider/docker/ -├── docker.go # Provider struct, New() -├── types.go # CreateParams, Container, etc. -├── docker_test.go # Tests -└── (no runtime/ subdirectory) - -internal/api/docker/ -├── gen/ -│ ├── api.yaml # OpenAPI spec -│ ├── cfg.yaml # oapi-codegen config -│ └── generate.go # go:generate directive -├── types.go # Domain struct, interfaces -├── docker.go # New(), interface check -├── docker_create.go # Create handler -├── docker_list.go # List handler -├── docker_inspect.go # Inspect handler -├── docker_start.go # Start handler -├── docker_stop.go # Stop handler -├── docker_remove.go # Remove handler -├── docker_exec.go # Exec handler -├── docker_pull.go # Pull handler -└── *_public_test.go # Tests - -internal/api/ -├── handler_docker.go # GetDockerHandler() method -└── handler.go # +RegisterHandlers() wiring - -cmd/ -├── client_container.go # parent: `container` subcommand -├── client_container_docker.go # parent: `docker` subcommand -├── client_container_docker_create.go -├── client_container_docker_list.go -├── client_container_docker_inspect.go -├── client_container_docker_start.go -├── client_container_docker_stop.go -├── client_container_docker_remove.go -├── client_container_docker_exec.go -└── client_container_docker_pull.go - -pkg/sdk/client/ -├── docker.go # DockerService -└── docker_types.go # DockerResult, etc. - -internal/agent/ -├── processor_docker.go # docker case + dispatch -└── types.go # dockerProvider field -``` - -### No Shared Runtime Interface - -The `runtime.Driver` interface in -`internal/provider/container/runtime/driver.go` is removed. The Docker provider -defines its own types directly. When LXD is added, it gets its own provider -package (`internal/provider/lxd/`) with its own types — LXD concepts (instances, -profiles, projects) don't map to Docker concepts (images, containers, layers). - -Each runtime is fully independent: - -- Own API domain, paths, and OpenAPI schemas -- Own CLI subcommands under `client container ` -- Own SDK service (`client.Docker`, `client.Lxd`) -- Own permissions (`docker:read`, `lxd:read`) -- Own provider package with own types -- Own orchestrator helpers - -### Orchestrator DSL - -Convenience methods on `*Plan` in `pkg/sdk/orchestrator/` wrap `client.Docker.*` -calls so users don't write boilerplate TaskFunc bodies: - -```go -plan.DockerPull("pull-image", target, "ubuntu:24.04") -plan.DockerCreate("create-app", target, gen.DockerCreateRequest{...}) -plan.DockerExec("run-cmd", target, "my-app", gen.DockerExecRequest{...}) -plan.DockerInspect("check", target, "my-app") -plan.DockerStart("start", target, "my-app") -plan.DockerStop("stop", target, "my-app", gen.DockerStopRequest{...}) -plan.DockerRemove("cleanup", target, "my-app", &gen.DeleteNodeDockerByIDParams{...}) -``` - -Each returns `*Task` for chaining (`DependsOn`, `OnlyIfChanged`, etc.). Future -runtimes add `plan.LxdLaunch(...)`, `plan.LxdExec(...)` — no shared interface. - -### Documentation - -- `docs/docs/sidebar/features/container-management.md` — update to describe - Docker as the first runtime, explain the per-runtime model -- CLI docs: restructure under `container/docker/` -- SDK orchestrator docs: update container-targeting to use `plan.DockerPull` - etc. - -## What This Changes - -This is a mechanical rename + restructure of the existing fully-built container -domain. No behavior changes. The scope is: - -1. Rename ~40+ files across all layers (API, CLI, SDK, agent, provider, job - types, tests, docs) -2. Remove `internal/provider/container/runtime/driver.go` shared interface -3. Flatten `internal/provider/container/runtime/docker/` → - `internal/provider/docker/` -4. Add `container` parent CLI command -5. Add orchestrator DSL helpers -6. Update all docs - -## Key Design Decisions - -| Decision | Choice | Rationale | -| ---------------------------- | ------------------- | ------------------------------------------------------ | -| User chooses runtime | Yes | Agent shouldn't guess; Docker/LXD/Podman are different | -| Separate domains per runtime | Yes | No useful shared abstraction across runtimes | -| CLI nesting | `container docker` | Groups runtimes for discoverability | -| API path nesting | `/container/docker` | Mirrors CLI structure | -| No shared interface | Yes | LXD concepts don't map to Docker concepts | -| Flat provider packages | Yes | No shared parent code to justify nesting | -| Orchestrator helpers | Methods on Plan | Eliminates TaskFunc boilerplate | diff --git a/docs/plans/2026-03-13-container-runtime-rename.md b/docs/plans/2026-03-13-container-runtime-rename.md deleted file mode 100644 index e73ac9418..000000000 --- a/docs/plans/2026-03-13-container-runtime-rename.md +++ /dev/null @@ -1,976 +0,0 @@ -# Container → Docker Domain Rename Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Rename the generic `container` domain to `docker`, nest under a -`container` parent in CLI/API paths, remove the shared `runtime.Driver` -interface, and add orchestrator DSL helpers. - -**Architecture:** Mechanical rename across all layers (API, provider, agent, -job, CLI, SDK, permissions, docs, tests) from `container` to `docker`. API paths -change from `/node/{hostname}/container` to `/node/{hostname}/container/docker`. -CLI changes from `client container list` to `client container docker list`. The -shared `runtime.Driver` interface is removed — Docker provider owns its types -directly. New orchestrator DSL methods (`plan.DockerPull`, etc.) wrap SDK client -calls. - -**Tech Stack:** Go, Echo, oapi-codegen, Cobra, testify/suite, Docker Go SDK - -**Spec:** `docs/plans/2026-03-13-container-runtime-rename-design.md` - ---- - -## Chunk 1: OpenAPI Spec + Permissions - -### Task 1: Rename OpenAPI Spec - -**Files:** - -- Modify: `internal/api/container/gen/api.yaml` - -**Step 1:** Rename the directory: - -```bash -git mv internal/api/container internal/api/docker -``` - -**Step 2:** Edit `internal/api/docker/gen/api.yaml`: - -- Change all paths from `/node/{hostname}/container` to - `/node/{hostname}/container/docker` -- Change all security scopes from `container:read/write/execute` to - `docker:read/write/execute` -- Rename all schema names from `Container*` to `Docker*` (e.g., - `ContainerCreateRequest` → `DockerCreateRequest`, `ContainerResponse` → - `DockerResponse`) -- Rename all operation IDs from `*Container*` to `*Docker*` -- Update tag names and descriptions from "Container" to "Docker" -- Update the `ContainerId` parameter to `DockerId` - -**Step 3:** Update `internal/api/docker/gen/cfg.yaml`: - -- Change output filename from `container.gen.go` to `docker.gen.go` -- Update package name if needed - -**Step 4:** Update `internal/api/docker/gen/generate.go`: - -- Update the `go:generate` directive path if needed - -**Step 5:** Regenerate: - -```bash -go generate ./internal/api/docker/gen/... -``` - -**Step 6:** Verify generation succeeded — `docker.gen.go` should exist: - -```bash -ls internal/api/docker/gen/docker.gen.go -``` - -**Step 7:** Delete old generated file if it still exists: - -```bash -rm -f internal/api/docker/gen/container.gen.go -``` - -**Step 8:** Commit: - -```bash -git add internal/api/docker/ internal/api/container/ -git commit -m "refactor: rename container OpenAPI spec to docker" -``` - ---- - -### Task 2: Rename Permissions - -**Files:** - -- Modify: `internal/authtoken/permissions.go` - -**Step 1:** Rename the permission constants: - -- `PermContainerRead` → `PermDockerRead` with value `"docker:read"` -- `PermContainerWrite` → `PermDockerWrite` with value `"docker:write"` -- `PermContainerExecute` → `PermDockerExecute` with value `"docker:execute"` - -**Step 2:** Update the `AllPermissions` slice to use the new names. - -**Step 3:** Update the default role mappings (admin, write, read) to use the new -permission constants. - -**Step 4:** Search for any other files referencing the old permission constants: - -```bash -grep -rn 'PermContainer\|container:read\|container:write\|container:execute' \ - --include='*.go' . | grep -v '.gen.go' | grep -v '_test.go' -``` - -Fix all references found (likely in `internal/api/docker/` handlers and -`cmd/api_helpers.go`). - -**Step 5:** Verify it compiles: - -```bash -go build ./internal/authtoken/... ./internal/api/docker/... -``` - -**Step 6:** Run permission tests: - -```bash -go test ./internal/authtoken/... -count=1 -``` - -**Step 7:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container permissions to docker" -``` - ---- - -## Chunk 2: Provider Layer - -### Task 3: Flatten and Rename Provider - -**Files:** - -- Rename: `internal/provider/container/` → `internal/provider/docker/` -- Remove: `internal/provider/container/runtime/driver.go` (shared interface) - -**Step 1:** Move the Docker driver implementation up and rename: - -```bash -git mv internal/provider/container internal/provider/docker -``` - -**Step 2:** The directory structure should become: - -``` -internal/provider/docker/ -├── provider.go -├── types.go -├── provider_public_test.go -├── mocks/ -├── runtime/ -│ ├── driver.go ← DELETE this (shared interface) -│ ├── docker/ -│ │ ├── docker.go -│ │ └── docker_public_test.go -│ └── mocks/ -``` - -Move `runtime/docker/docker.go` types and implementation into the parent -package, or keep `runtime/docker/` as the actual Docker SDK driver. The provider -in `provider.go` already wraps the driver, so the structure can stay — just -update the package import paths from `internal/provider/container/...` to -`internal/provider/docker/...`. - -**Step 3:** Delete the shared `runtime.Driver` interface: - -```bash -rm internal/provider/docker/runtime/driver.go -``` - -**Step 4:** Update the Docker driver (`runtime/docker/docker.go`) to define its -own interface or use concrete types instead of the removed `runtime.Driver`. The -provider in `provider.go` should type-assert or use the concrete Docker driver -type. - -**Step 5:** Update all import paths in the provider package from -`internal/provider/container` to `internal/provider/docker`. - -**Step 6:** Update package declarations — `package container` → `package docker` -in `provider.go`, `types.go`, etc. - -**Step 7:** Rename the `Provider` interface methods and types if they use -`Container` prefix (check `types.go`). - -**Step 8:** Update mock generation directives in `mocks/generate.go`. - -**Step 9:** Regenerate mocks: - -```bash -go generate ./internal/provider/docker/mocks/... -go generate ./internal/provider/docker/runtime/mocks/... -``` - -**Step 10:** Verify it compiles: - -```bash -go build ./internal/provider/docker/... -``` - -**Step 11:** Run tests: - -```bash -go test ./internal/provider/docker/... -count=1 -``` - -**Step 12:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container provider to docker" -``` - ---- - -## Chunk 3: Job Types - -### Task 4: Rename Job Types and Operations - -**Files:** - -- Modify: `internal/job/types.go` -- Modify: `internal/job/client/modify_container.go` (rename to - `modify_docker.go`) -- Modify: `internal/job/client/modify_container_public_test.go` (rename to - `modify_docker_public_test.go`) - -**Step 1:** In `internal/job/types.go`, rename all container operation -constants: - -- `OperationContainerCreate` → `OperationDockerCreate` with value - `"docker.create.execute"` -- `OperationContainerStart` → `OperationDockerStart` with value - `"docker.start.execute"` -- `OperationContainerStop` → `OperationDockerStop` with value - `"docker.stop.execute"` -- `OperationContainerRemove` → `OperationDockerRemove` with value - `"docker.remove.execute"` -- `OperationContainerList` → `OperationDockerList` with value - `"docker.list.get"` -- `OperationContainerInspect` → `OperationDockerInspect` with value - `"docker.inspect.get"` -- `OperationContainerExec` → `OperationDockerExec` with value - `"docker.exec.execute"` -- `OperationContainerPull` → `OperationDockerPull` with value - `"docker.pull.execute"` - -**Step 2:** Rename all data types: - -- `ContainerCreateData` → `DockerCreateData` -- `ContainerStopData` → `DockerStopData` -- `ContainerRemoveData` → `DockerRemoveData` -- `ContainerListData` → `DockerListData` -- `ContainerExecData` → `DockerExecData` -- `ContainerPullData` → `DockerPullData` - -**Step 3:** Rename the job client file: - -```bash -git mv internal/job/client/modify_container.go \ - internal/job/client/modify_docker.go -git mv internal/job/client/modify_container_public_test.go \ - internal/job/client/modify_docker_public_test.go -``` - -**Step 4:** Update function names in the renamed files (e.g., `ModifyContainer*` -→ `ModifyDocker*` or whatever the current pattern is). - -**Step 5:** Search for all remaining references to old constant/type names: - -```bash -grep -rn 'OperationContainer\|ContainerCreateData\|ContainerStopData\|ContainerRemoveData\|ContainerListData\|ContainerExecData\|ContainerPullData' \ - --include='*.go' . | grep -v '.gen.go' -``` - -Fix all references found. - -**Step 6:** Verify it compiles: - -```bash -go build ./internal/job/... -``` - -**Step 7:** Run tests: - -```bash -go test ./internal/job/... -count=1 -``` - -**Step 8:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container job types to docker" -``` - ---- - -## Chunk 4: Agent Layer - -### Task 5: Rename Agent Processor and Wiring - -**Files:** - -- Rename: `internal/agent/processor_container.go` → - `internal/agent/processor_docker.go` -- Rename: `internal/agent/processor_container_test.go` → - `internal/agent/processor_docker_test.go` -- Modify: `internal/agent/types.go` -- Modify: `internal/agent/agent.go` -- Modify: `internal/agent/factory.go` -- Modify: `internal/agent/processor.go` -- Modify: `internal/agent/factory_test.go` -- Modify: `internal/agent/factory_public_test.go` - -**Step 1:** Rename processor files: - -```bash -git mv internal/agent/processor_container.go \ - internal/agent/processor_docker.go -git mv internal/agent/processor_container_test.go \ - internal/agent/processor_docker_test.go -``` - -**Step 2:** In `processor_docker.go`: - -- Rename `processContainerOperation` → `processDockerOperation` -- Rename `processContainerCreate` → `processDockerCreate` (and all 8 process - methods) -- Change `a.containerProvider` → `a.dockerProvider` -- Update import from `internal/provider/container` to `internal/provider/docker` - -**Step 3:** In `processor.go`: - -- Change `case "container":` → `case "docker":` -- Change `a.processContainerOperation` → `a.processDockerOperation` - -**Step 4:** In `types.go`: - -- Change `containerProvider containerProv.Provider` → - `dockerProvider dockerProv.Provider` -- Update the import alias from `containerProv` to `dockerProv` - -**Step 5:** In `agent.go`: - -- Update parameter name from `containerProvider` to `dockerProvider` -- Update field assignment - -**Step 6:** In `factory.go`: - -- Rename `containerProvider` variable to `dockerProvider` -- Update import from `internal/provider/container` to `internal/provider/docker` -- Update the return value - -**Step 7:** In `factory_test.go` and `factory_public_test.go`: - -- Update variable names and comments - -**Step 8:** In `processor_docker_test.go`: - -- Update all function names and references - -**Step 9:** Verify it compiles: - -```bash -go build ./internal/agent/... -``` - -**Step 10:** Run tests: - -```bash -go test ./internal/agent/... -count=1 -``` - -**Step 11:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container agent wiring to docker" -``` - ---- - -## Chunk 5: API Handlers - -### Task 6: Rename API Handler Files and Wiring - -**Files:** - -- Modify: all files in `internal/api/docker/` (already moved in Task 1) -- Rename: `internal/api/handler_container.go` → `internal/api/handler_docker.go` -- Modify: `internal/api/handler.go` -- Modify: `internal/api/types.go` -- Modify: `internal/api/handler_public_test.go` -- Modify: `cmd/api_helpers.go` - -**Step 1:** In `internal/api/docker/`: - -- Rename all files: `container_create.go` → `docker_create.go`, etc. (8 handler - files + 8 test files) -- Update package declaration from `package container` to `package docker` -- Rename the `Container` struct to `Docker` -- Update `New()` to return `*Docker` -- Rename handler methods (e.g., `PostNodeContainer` → `PostNodeContainerDocker`) -- Update all gen import aliases from `containerGen` to `dockerGen` -- Update compile-time interface check -- Update references to old job operation constants -- Update references to old data types - -**Step 2:** Rename and update `internal/api/docker/types.go`: - -- Rename the struct and any interfaces - -**Step 3:** Rename and update `internal/api/docker/convert.go`: - -- Update function names and types - -**Step 4:** Rename and update `internal/api/docker/validate.go`: - -- Update function names - -**Step 5:** Rename server wiring: - -```bash -git mv internal/api/handler_container.go internal/api/handler_docker.go -``` - -**Step 6:** In `handler_docker.go`: - -- Rename `GetContainerHandler` → `GetDockerHandler` -- Update imports from `internal/api/container` to `internal/api/docker` -- Update scope references from `container:*` to `docker:*` - -**Step 7:** In `types.go`, rename the handler field and option function. - -**Step 8:** In `handler.go`, update the `RegisterHandlers` call. - -**Step 9:** In `handler_public_test.go`, rename `TestGetContainerHandler` → -`TestGetDockerHandler`. - -**Step 10:** In `cmd/api_helpers.go`, update `GetContainerHandler` → -`GetDockerHandler`. - -**Step 11:** Regenerate the combined spec and SDK client: - -```bash -just generate -go generate ./pkg/sdk/client/gen/... -``` - -**Step 12:** Verify it compiles: - -```bash -go build ./... -``` - -**Step 13:** Run tests: - -```bash -go test ./internal/api/docker/... ./internal/api/... -count=1 -``` - -**Step 14:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container API handlers to docker" -``` - ---- - -## Chunk 6: CLI - -### Task 7: Restructure CLI Commands - -**Files:** - -- Modify: `cmd/client_container.go` (becomes parent with just `` - grouping) -- Create: `cmd/client_container_docker.go` (new `docker` subcommand) -- Rename: all `cmd/client_container_*.go` → `cmd/client_container_docker_*.go` - -**Step 1:** Update `cmd/client_container.go` to be a thin parent command: - -```go -var clientContainerCmd = &cobra.Command{ - Use: "container", - Short: "Container runtime management", - Long: `Manage containers using runtime-specific subcommands.`, -} - -func init() { - clientCmd.AddCommand(clientContainerCmd) -} -``` - -**Step 2:** Create `cmd/client_container_docker.go`: - -```go -var clientContainerDockerCmd = &cobra.Command{ - Use: "docker", - Short: "Docker container operations", - Long: `Manage Docker containers on target nodes.`, -} - -func init() { - clientContainerCmd.AddCommand(clientContainerDockerCmd) -} -``` - -**Step 3:** Rename all subcommand files: - -```bash -git mv cmd/client_container_create.go cmd/client_container_docker_create.go -git mv cmd/client_container_list.go cmd/client_container_docker_list.go -git mv cmd/client_container_inspect.go cmd/client_container_docker_inspect.go -git mv cmd/client_container_start.go cmd/client_container_docker_start.go -git mv cmd/client_container_stop.go cmd/client_container_docker_stop.go -git mv cmd/client_container_remove.go cmd/client_container_docker_remove.go -git mv cmd/client_container_exec.go cmd/client_container_docker_exec.go -git mv cmd/client_container_pull.go cmd/client_container_docker_pull.go -``` - -**Step 4:** In each renamed file: - -- Change parent command registration from `clientContainerCmd.AddCommand(...)` - to `clientContainerDockerCmd.AddCommand(...)` -- Rename cobra command variables from `clientContainer*Cmd` to - `clientContainerDocker*Cmd` -- Update SDK client calls from `c.Container.*` to `c.Docker.*` -- Update generated type references from `gen.Container*` to `gen.Docker*` -- Update `gen.GetNodeContainerParams*` to `gen.GetNodeDockerParams*` (or - whatever the regenerated names are) - -**Step 5:** Verify the CLI compiles: - -```bash -go build ./cmd/... -``` - -**Step 6:** Verify the command tree looks right: - -```bash -go run main.go client container --help -go run main.go client container docker --help -``` - -**Step 7:** Commit: - -```bash -git add -A && git commit -m "refactor: nest docker CLI under client container docker" -``` - ---- - -## Chunk 7: SDK Client - -### Task 8: Rename SDK Client Service - -**Files:** - -- Rename: `pkg/sdk/client/container.go` → `pkg/sdk/client/docker.go` -- Rename: `pkg/sdk/client/container_types.go` → `pkg/sdk/client/docker_types.go` -- Rename: `pkg/sdk/client/container_public_test.go` → - `pkg/sdk/client/docker_public_test.go` -- Rename: `pkg/sdk/client/container_types_test.go` → - `pkg/sdk/client/docker_types_test.go` -- Modify: `pkg/sdk/client/osapi.go` - -**Step 1:** Rename files: - -```bash -git mv pkg/sdk/client/container.go pkg/sdk/client/docker.go -git mv pkg/sdk/client/container_types.go pkg/sdk/client/docker_types.go -git mv pkg/sdk/client/container_public_test.go \ - pkg/sdk/client/docker_public_test.go -git mv pkg/sdk/client/container_types_test.go \ - pkg/sdk/client/docker_types_test.go -``` - -**Step 2:** In `docker.go`: - -- Rename `ContainerService` → `DockerService` -- Update all method bodies to use the regenerated `gen.Docker*` type names -- Update error message prefixes - -**Step 3:** In `docker_types.go`: - -- Rename all types: `ContainerResult` → `DockerResult`, `ContainerListResult` → - `DockerListResult`, etc. -- Rename all converter functions: `containerResultCollectionFromGen` → - `dockerResultCollectionFromGen`, etc. - -**Step 4:** In `osapi.go`: - -- Change field `Container *ContainerService` → `Docker *DockerService` -- Update initialization: `c.Container = &ContainerService{...}` → - `c.Docker = &DockerService{...}` -- Update comment - -**Step 5:** In test files, update all type and method references. - -**Step 6:** Search for remaining `Container` references in the SDK: - -```bash -grep -rn 'Container' pkg/sdk/client/ --include='*.go' | grep -v '.gen.go' -``` - -Fix all remaining references. - -**Step 7:** Verify it compiles: - -```bash -go build ./pkg/sdk/client/... -``` - -**Step 8:** Run tests: - -```bash -go test ./pkg/sdk/client/... -count=1 -``` - -**Step 9:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container SDK client to docker" -``` - ---- - -## Chunk 8: Orchestrator DSL Helpers - -### Task 9: Add Orchestrator Docker Methods - -**Files:** - -- Create: `pkg/sdk/orchestrator/docker.go` -- Create: `pkg/sdk/orchestrator/docker_public_test.go` - -**Step 1:** Write the test file `docker_public_test.go` with a test suite -covering each helper method. Use table-driven tests. Mock the client responses -or test that the correct TaskFunc is created: - -```go -func (s *DockerPublicTestSuite) TestDockerPull() { - tests := []struct { - name string - target string - image string - validateFunc func(task *orchestrator.Task) - }{ - { - name: "creates task with correct name", - target: "_any", - image: "ubuntu:24.04", - validateFunc: func(task *orchestrator.Task) { - s.Equal("pull-image", task.Name()) - }, - }, - } - // ... -} -``` - -Test each method: `DockerPull`, `DockerCreate`, `DockerExec`, `DockerInspect`, -`DockerStart`, `DockerStop`, `DockerRemove`, `DockerList`. - -**Step 2:** Run tests to verify they fail: - -```bash -go test ./pkg/sdk/orchestrator/... -count=1 -run TestDocker -``` - -**Step 3:** Implement `docker.go` with methods on `*Plan`: - -```go -// DockerPull creates a task that pulls a Docker image on the target host. -func (p *Plan) DockerPull( - name string, - target string, - image string, -) *Task { - return p.TaskFunc(name, func( - ctx context.Context, - c *osapiclient.Client, - ) (*Result, error) { - resp, err := c.Docker.Pull(ctx, target, gen.DockerPullRequest{ - Image: image, - }) - if err != nil { - return nil, err - } - r := resp.Data.Results[0] - return &Result{ - Changed: true, - Data: map[string]any{ - "image_id": r.ImageID, - "tag": r.Tag, - "size": r.Size, - }, - }, nil - }) -} -``` - -Follow the same pattern for all 8 operations. Each method: - -- Takes the task name, target, and operation-specific params -- Returns `*Task` for chaining -- Wraps the SDK client call in a `TaskFunc` -- Sets `Changed: true` for mutations, `Changed: false` for reads (inspect, list) -- Populates `Data` with relevant result fields - -Methods to implement: - -- `DockerPull(name, target, image string) *Task` -- `DockerCreate(name, target string, body gen.DockerCreateRequest) *Task` -- `DockerStart(name, target, id string) *Task` -- `DockerStop(name, target, id string, body gen.DockerStopRequest) *Task` -- `DockerRemove(name, target, id string, params *gen.DeleteNodeContainerDockerByIDParams) *Task` -- `DockerExec(name, target, id string, body gen.DockerExecRequest) *Task` -- `DockerInspect(name, target, id string) *Task` -- `DockerList(name, target string, params *gen.GetNodeContainerDockerParams) *Task` - -**Step 4:** Run tests to verify they pass: - -```bash -go test ./pkg/sdk/orchestrator/... -count=1 -``` - -**Step 5:** Commit: - -```bash -git add -A && git commit -m "feat: add orchestrator Docker DSL helpers" -``` - ---- - -### Task 10: Rewrite Container Targeting Example - -**Files:** - -- Modify: `examples/sdk/orchestrator/features/container-targeting.go` - -**Step 1:** Rewrite the example to use the new DSL helpers: - -```go -plan := orchestrator.NewPlan(apiClient, - orchestrator.WithHooks(hooks), - orchestrator.OnError(orchestrator.Continue), -) - -pull := plan.DockerPull("pull-image", target, containerImage) - -create := plan.DockerCreate("create-container", target, - gen.DockerCreateRequest{ - Image: containerImage, - Name: ptr(containerName), - AutoStart: &autoStart, - Command: &[]string{"sleep", "600"}, - }, -) -create.DependsOn(pull) - -plan.DockerExec("exec-hostname", target, containerName, - gen.DockerExecRequest{Command: []string{"hostname"}}, -).DependsOn(create) - -plan.DockerInspect("inspect", target, containerName).DependsOn(create) - -plan.DockerRemove("cleanup", target, containerName, - &gen.DeleteNodeContainerDockerByIDParams{Force: &force}, -).DependsOn(create) -``` - -**Step 2:** Update SDK client example too: `examples/sdk/client/container.go` — -update to use `c.Docker.*` and `gen.Docker*` types. - -**Step 3:** Verify examples compile: - -```bash -go build ./examples/... -``` - -**Step 4:** Commit: - -```bash -git add -A && git commit -m "refactor: update examples to use docker DSL" -``` - ---- - -## Chunk 9: Integration Tests - -### Task 11: Rename Integration Tests - -**Files:** - -- Rename: `test/integration/container_test.go` → - `test/integration/docker_test.go` - -**Step 1:** Rename the file: - -```bash -git mv test/integration/container_test.go test/integration/docker_test.go -``` - -**Step 2:** Update the test to: - -- Use `c.Docker.*` instead of `c.Container.*` -- Use `gen.Docker*` types instead of `gen.Container*` -- Update CLI commands from `container list` to `container docker list` -- Rename suite/test names from `Container*` to `Docker*` - -**Step 3:** Verify it compiles: - -```bash -go build ./test/integration/... -``` - -**Step 4:** Commit: - -```bash -git add -A && git commit -m "refactor: rename container integration tests to docker" -``` - ---- - -## Chunk 10: Documentation - -### Task 12: Update Documentation - -**Files:** - -- Modify: `docs/docs/sidebar/features/container-management.md` — update to - describe Docker as first runtime, update all paths/permissions/CLI examples -- Rename: CLI docs from `docs/.../container/` to restructure under - `docs/.../container/docker/` -- Modify: SDK orchestrator docs to reference Docker methods -- Modify: `docs/docs/sidebar/usage/configuration.md` — update permission tables -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` — update - endpoint tables -- Modify: `docs/docusaurus.config.ts` — update navbar links -- Modify: `CLAUDE.md` — update permission tables in role descriptions - -**Step 1:** Update `container-management.md`: - -- Update title, description -- Update all path examples from `/container` to `/container/docker` -- Update permissions table from `container:*` to `docker:*` -- Update CLI examples from `client container list` to - `client container docker list` -- Update role descriptions - -**Step 2:** Restructure CLI docs: - -- Current: `docs/.../cli/client/container/container.mdx` (parent) -- New: `docs/.../cli/client/container/container.mdx` stays as parent grouping -- Create: `docs/.../cli/client/container/docker/` directory -- Move per-operation docs into the docker subdirectory -- Update all command examples - -**Step 3:** Update SDK orchestrator operation docs: - -- Rename `container-create.md` → `docker-create.md`, etc. -- Update code examples to use `plan.DockerCreate(...)` etc. - -**Step 4:** Update configuration.md permission tables. - -**Step 5:** Update system-architecture.md endpoint tables. - -**Step 6:** Regenerate API docs: - -```bash -just docs::generate-api -``` - -**Step 7:** Commit: - -```bash -git add -A && git commit -m "docs: update documentation for docker domain rename" -``` - ---- - -## Chunk 11: Verify - -### Task 13: Full Verification - -**Step 1:** Regenerate everything: - -```bash -just generate -``` - -**Step 2:** Build: - -```bash -go build ./... -``` - -**Step 3:** Run all unit tests: - -```bash -just go::unit -``` - -**Step 4:** Run lint: - -```bash -just go::vet -``` - -**Step 5:** Search for any remaining `container` references that should be -`docker` (excluding the `container` parent command which is intentional): - -```bash -grep -rn 'container:read\|container:write\|container:execute' \ - --include='*.go' . | grep -v '.gen.go' -grep -rn 'ContainerService\|ContainerResult\|ContainerCreateData' \ - --include='*.go' . | grep -v '.gen.go' -grep -rn 'OperationContainer' --include='*.go' . -grep -rn 'containerProvider' --include='*.go' . -grep -rn 'processContainer' --include='*.go' . -``` - -All should return empty. - -**Step 6:** Verify CLI works: - -```bash -go run main.go client container --help -go run main.go client container docker --help -``` - -**Step 7:** Commit any final fixes: - -```bash -git add -A && git commit -m "chore: final verification cleanup" -``` - ---- - -## Files Modified - -| Repo | File | Change | -| ----- | --------------------------------------------------------------- | -------------------------------------- | -| osapi | `internal/api/container/` → `internal/api/docker/` | Full directory rename + content update | -| osapi | `internal/api/handler_container.go` → `handler_docker.go` | Rename + update | -| osapi | `internal/api/handler.go` | Update registration | -| osapi | `internal/api/types.go` | Rename handler field | -| osapi | `internal/api/handler_public_test.go` | Rename test | -| osapi | `internal/provider/container/` → `internal/provider/docker/` | Full directory rename | -| osapi | `internal/provider/container/runtime/driver.go` | DELETE | -| osapi | `internal/agent/processor_container.go` → `processor_docker.go` | Rename + update | -| osapi | `internal/agent/types.go` | Rename field | -| osapi | `internal/agent/agent.go` | Rename parameter | -| osapi | `internal/agent/factory.go` | Rename variable | -| osapi | `internal/agent/processor.go` | Update case | -| osapi | `internal/job/types.go` | Rename 8 constants + 6 types | -| osapi | `internal/job/client/modify_container.go` → `modify_docker.go` | Rename + update | -| osapi | `internal/authtoken/permissions.go` | Rename 3 constants | -| osapi | `cmd/client_container.go` | Simplify to parent | -| osapi | `cmd/client_container_docker.go` | NEW parent for docker | -| osapi | `cmd/client_container_*.go` → `client_container_docker_*.go` | Rename 8 files | -| osapi | `cmd/api_helpers.go` | Update handler call | -| osapi | `pkg/sdk/client/container.go` → `docker.go` | Rename + update | -| osapi | `pkg/sdk/client/container_types.go` → `docker_types.go` | Rename + update | -| osapi | `pkg/sdk/client/osapi.go` | Rename field | -| osapi | `pkg/sdk/orchestrator/docker.go` | NEW — DSL helpers | -| osapi | `pkg/sdk/orchestrator/docker_public_test.go` | NEW — tests | -| osapi | `examples/sdk/orchestrator/features/container-targeting.go` | Rewrite with DSL | -| osapi | `examples/sdk/client/container.go` | Update SDK calls | -| osapi | `test/integration/container_test.go` → `docker_test.go` | Rename + update | -| osapi | `docs/` (multiple) | Update paths, permissions, examples | diff --git a/docs/plans/2026-03-13-orchestrator-op-layer.md b/docs/plans/2026-03-13-orchestrator-op-layer.md deleted file mode 100644 index fdb6894e8..000000000 --- a/docs/plans/2026-03-13-orchestrator-op-layer.md +++ /dev/null @@ -1,361 +0,0 @@ -# Orchestrator SDK Bridge Helpers Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to -> implement this plan task-by-task. - -**Goal:** Add bridge helpers to the SDK orchestrator package, achieve 100% -coverage, fix all examples so they compile and are complete, update docs, then -remove the misplaced docker DSL. - -**Architecture:** The SDK orchestrator package provides the DAG engine (plan, -task, runner) plus bridge utilities (`CollectionResult`, `StructToMap`). -Domain-specific operation methods belong in consumer packages like -`osapi-orchestrator`, not in the SDK. Examples demonstrate the `TaskFunc` -pattern. - -**Tech Stack:** Go 1.25, generics, testify/suite, httptest - ---- - -### Task 1: Add CollectionResult and StructToMap bridge helpers - -**Files:** - -- Create: `pkg/sdk/orchestrator/bridge.go` -- Test: `pkg/sdk/orchestrator/bridge_public_test.go` - -**Step 1: Write tests for `StructToMap` and `CollectionResult`** - -Create `bridge_public_test.go` with `BridgePublicTestSuite`: - -Test `StructToMap`: - -- Converts struct with json tags to map -- Returns nil for nil input -- Handles nested structs -- Omits zero-value fields with `omitempty` - -Test `CollectionResult`: - -- Single result extracts JobID, Changed, HostResults with Data -- Multiple results — Changed is true when any host changed -- Empty results — returns result with empty HostResults -- HostResult.Data auto-populated via StructToMap when mapper leaves it nil -- HostResult.Data preserved when mapper sets it explicitly - -Use `client.HostnameResult`, `client.CommandResult` etc. as test inputs since -those are the real SDK types consumers will pass. - -**Step 2: Run tests to verify they fail** - -Run: `go test ./pkg/sdk/orchestrator/... -run TestBridgePublicTestSuite -v` -Expected: FAIL — `CollectionResult` and `StructToMap` not defined - -**Step 3: Implement `bridge.go`** - -```go -package orchestrator - -import ( - "encoding/json" - - osapiclient "github.com/osapi-io/osapi/pkg/sdk/client" -) - -// StructToMap converts a struct to map[string]any using its JSON -// tags. Returns nil if v is nil or cannot be marshaled. -func StructToMap(v any) map[string]any - -// CollectionResult builds a Result from a Collection response. -// It iterates all results, applies the toHostResult mapper to -// build per-host details, and auto-populates HostResult.Data -// via StructToMap when the mapper leaves it nil. Changed is true -// if any host reported a change. -func CollectionResult[T any]( - col osapiclient.Collection[T], - toHostResult func(T) HostResult, -) *Result -``` - -Mirrors `osapi-orchestrator`'s `buildResult` and `toMap` — but exported and in -the SDK where it belongs. - -**Step 4: Run tests to verify they pass** - -Run: `go test ./pkg/sdk/orchestrator/... -run TestBridgePublicTestSuite -v` -Expected: PASS - -**Step 5: Check coverage** - -Run: -`go test ./pkg/sdk/orchestrator/... -coverprofile=/tmp/bridge.out && go tool cover -func=/tmp/bridge.out | grep bridge` -Expected: 100% on bridge.go - -**Step 6: Commit** - -``` -feat(orchestrator): add CollectionResult and StructToMap helpers -``` - ---- - -### Task 2: Delete docker.go DSL and its tests - -**Files:** - -- Delete: `pkg/sdk/orchestrator/docker.go` -- Delete: `pkg/sdk/orchestrator/docker_public_test.go` - -**Step 1: Delete the files** - -```bash -rm pkg/sdk/orchestrator/docker.go -rm pkg/sdk/orchestrator/docker_public_test.go -``` - -**Step 2: Run SDK tests** - -Run: `go test ./pkg/sdk/orchestrator/... -count=1` Expected: PASS (engine + -bridge tests pass, docker tests gone) - -**Step 3: Check what breaks** - -Run: `go build ./... 2>&1` Expected: Compilation failures in -`container-targeting.go` (references `plan.DockerPull` etc.). Note the failures -— fixed in Task 3. - -**Step 4: Commit** - -``` -refactor(orchestrator): remove docker DSL methods - -Domain-specific operation methods belong in consumer packages -like osapi-orchestrator, not in the SDK engine. The SDK provides -CollectionResult and StructToMap as bridge helpers instead. -``` - ---- - -### Task 3: Fix container-targeting example - -**Files:** - -- Modify: `examples/sdk/orchestrator/features/container-targeting.go` - -**Step 1: Rewrite to use `TaskFunc` with `CollectionResult`** - -Replace all `plan.DockerPull()`, `plan.DockerCreate()`, etc. with -`plan.TaskFunc()` calls that use the SDK client directly and -`orchestrator.CollectionResult()` to build results. - -Keep the same DAG structure: pre-cleanup → pull → create → exec x3 + inspect + -deliberately-fails → cleanup. - -Pre-cleanup remains a `TaskFunc` that swallows errors. - -Each docker operation becomes: - -```go -pull := plan.TaskFunc("pull-image", func( - ctx context.Context, - c *client.Client, -) (*orchestrator.Result, error) { - resp, err := c.Docker.Pull(ctx, target, gen.DockerPullRequest{ - Image: containerImage, - }) - if err != nil { - return nil, err - } - - return orchestrator.CollectionResult(resp.Data, - func(r client.DockerPullResult) orchestrator.HostResult { - return orchestrator.HostResult{ - Hostname: r.Hostname, - Changed: r.Changed, - Error: r.Error, - } - }, - ), nil -}) -``` - -**Step 2: Build** - -Run: `go build examples/sdk/orchestrator/features/container-targeting.go` -Expected: Compiles successfully - -**Step 3: Commit** - -``` -refactor: update container-targeting to use TaskFunc with bridge helpers -``` - ---- - -### Task 4: Fix all broken operation examples and add docker examples - -**Files:** - -- Modify: 13 files in `examples/sdk/orchestrator/operations/` that use - `plan.Task(&Op{...})` (all except `file-upload.go` which already uses - `TaskFunc`) -- Modify: feature examples that use `plan.Task(&Op{...})`: `basic.go`, - `broadcast.go`, `error-strategy.go`, `file-deploy-workflow.go`, `guards.go`, - `hooks.go`, `only-if-changed.go`, `parallel.go`, `result-decode.go`, - `task-func-results.go`, `task-func.go` -- Create: 8 docker operation examples: `docker-pull.go`, `docker-create.go`, - `docker-list.go`, `docker-inspect.go`, `docker-start.go`, `docker-stop.go`, - `docker-remove.go`, `docker-exec.go` - -**Step 1: Rewrite operation examples to use `TaskFunc`** - -Each currently does: - -```go -plan.Task("get-hostname", &orchestrator.Op{ - Operation: "node.hostname.get", - Target: "_any", -}) -``` - -Replace with: - -```go -plan.TaskFunc("get-hostname", func( - ctx context.Context, - c *client.Client, -) (*orchestrator.Result, error) { - resp, err := c.Node.Hostname(ctx, "_any") - if err != nil { - return nil, err - } - - return orchestrator.CollectionResult(resp.Data, - func(r client.HostnameResult) orchestrator.HostResult { - return orchestrator.HostResult{ - Hostname: r.Hostname, - Changed: r.Changed, - Error: r.Error, - } - }, - ), nil -}) -``` - -Apply this pattern to all 13 operation files and all feature files. - -For operations with params (command exec, DNS update, file deploy, etc.), unpack -from the example's local variables into the SDK request types directly — no -`Params map[string]any` needed. - -**Step 2: Create 8 docker operation examples** - -Follow the exact same pattern as node/command examples. One file per docker -operation in `examples/sdk/orchestrator/operations/`. - -**Step 3: Build every example individually** - -```bash -for f in examples/sdk/orchestrator/operations/*.go; do - go build "$f" 2>&1 || echo "FAIL: $f" -done -for f in examples/sdk/orchestrator/features/*.go; do - go build "$f" 2>&1 || echo "FAIL: $f" -done -``` - -Expected: ALL files compile. Zero failures. This is a hard gate — do not proceed -until every example compiles. - -**Step 4: Commit** - -``` -fix: rewrite all orchestrator examples to use TaskFunc -``` - ---- - -### Task 5: Update orchestrator docs - -**Files:** - -- Modify: `docs/docs/sidebar/sdk/orchestrator/orchestrator.md` -- Modify: all operation doc pages in - `docs/docs/sidebar/sdk/orchestrator/operations/` -- Modify: `docs/docs/sidebar/sdk/orchestrator/features/container-targeting.md` - -**Step 1: Update orchestrator overview** - -Add documentation for `CollectionResult` and `StructToMap` as SDK-provided -bridge helpers. Update the usage examples to show `TaskFunc` pattern instead of -`plan.Task(&Op{...})`. - -**Step 2: Update operation doc pages** - -Each page currently shows the `plan.Task(&Op{...})` pattern. Update to show -`plan.TaskFunc()` with `CollectionResult`. Match the code in the corresponding -example file exactly. - -**Step 3: Update container-targeting feature doc** - -Update code examples to match the rewritten `container-targeting.go`. - -**Step 4: Build docs** - -Run: `cd docs && bun run build` Expected: Build succeeds, no broken links - -**Step 5: Commit** - -``` -docs: update orchestrator docs for TaskFunc pattern -``` - ---- - -### Task 6: Final verification - -**Step 1: Full test suite** - -Run: `go test ./... -count=1` Expected: All packages pass - -**Step 2: SDK coverage check** - -Run: -`go test ./pkg/sdk/... -coverprofile=/tmp/sdk.out && go tool cover -func=/tmp/sdk.out | grep -v gen | grep -v '100.0%'` -Expected: All SDK packages at 100% (excluding gen) - -**Step 3: Lint and format** - -```bash -find . -type f -name '*.go' -not -name '*.gen.go' -not -name '*.pb.go' \ - -not -path './.worktrees/*' -not -path './.claude/*' \ - | xargs go tool github.com/segmentio/golines \ - --base-formatter="go tool mvdan.cc/gofumpt" -w -go tool github.com/golangci/golangci-lint/v2/cmd/golangci-lint run \ - --config .golangci.yml -``` - -Expected: 0 issues - -**Step 4: Verify all examples compile** - -```bash -for f in examples/sdk/orchestrator/operations/*.go; do - go build "$f" 2>&1 || echo "FAIL: $f" -done -for f in examples/sdk/orchestrator/features/*.go; do - go build "$f" 2>&1 || echo "FAIL: $f" -done -``` - -Expected: Zero failures - -**Step 5: Build docs** - -Run: `cd docs && bun run build` Expected: No broken links - -**Step 6: Commit any remaining fixes** - -``` -chore: final verification cleanup -``` diff --git a/docs/plans/2026-03-13-sdk-quality-fixes-design.md b/docs/plans/2026-03-13-sdk-quality-fixes-design.md deleted file mode 100644 index 7d63a16f0..000000000 --- a/docs/plans/2026-03-13-sdk-quality-fixes-design.md +++ /dev/null @@ -1,137 +0,0 @@ -# SDK Quality Fixes Design - -## Problem - -A code review identified 9 issues across the OSAPI SDK and its primary consumer -(`osapi-orchestrator`). The central gap is that the SDK's bridge helpers are -incomplete, forcing `osapi-orchestrator` to reimplement result conversion and -duplicate ~200 lines of type definitions. - -## Fixes — osapi SDK (this repo) - -### #1: CollectionResult populates Result.Data - -`CollectionResult` currently only populates `HostResult.Data` per-host. It -leaves `Result.Data` nil, so `osapi-orchestrator` must call -`mustRawToMap(resp.RawJSON())` separately for every operation. - -**Fix:** Add a `rawJSON []byte` parameter. When non-nil, unmarshal it into -`Result.Data`. Callers pass `resp.RawJSON()` or `nil`. - -```go -func CollectionResult[T any]( - col Collection[T], - rawJSON []byte, - toHostResult func(T) HostResult, -) *Result -``` - -Update all callers in examples and tests. - -### #2: Docker SDK request types - -`DockerService.Create`, `List`, `Stop`, and `Remove` expose `gen.*` request -types. Every other service wraps gen types into SDK-defined types. - -**Fix:** Define in `docker_types.go`: - -- `DockerCreateOpts` — Image, Name, Command, Env, Ports, Volumes, AutoStart -- `DockerStopOpts` — Timeout -- `DockerListParams` — State -- `DockerRemoveParams` — Force - -Map to gen types inside the service methods. Consumers no longer import `gen`. - -### #5: Collection[T].First() - -Every consumer blindly indexes `Results[0]` with no bounds check. - -**Fix:** Add to `Collection[T]` in `response.go`: - -```go -func (c Collection[T]) First() (T, bool) { - if len(c.Results) == 0 { - var zero T - return zero, false - } - return c.Results[0], true -} -``` - -### #7: JSON tags on SDK result types - -SDK result types (`HostnameResult`, `DiskResult`, `CommandResult`, etc.) lack -`json:"..."` tags. `StructToMap` cannot produce correct keys without them, -forcing consumers to use `RawJSON()` as a workaround. - -**Fix:** Add `json` tags to all result types in `node_types.go`, -`docker_types.go`, `file_types.go`, `audit_types.go`, `health_types.go`, -`job_types.go`, `agent_types.go`. - -### #8: AuditService.Get UUID error wrapping - -`AuditService.Get` returns raw UUID parse error without context. -`JobService.Get` wraps it correctly. - -**Fix:** Wrap with `fmt.Errorf("invalid audit ID: %w", err)`. - -## Fixes — osapi-orchestrator (separate repo) - -### #4: mustRawToMap panic → error return - -`mustRawToMap` panics on invalid JSON. A proxy 502 or truncated response would -crash the process. - -**Fix:** Change return to `(map[string]any, error)`. Propagate error through all -callers. - -After SDK fix #1 lands, `mustRawToMap` can be deleted entirely since -`CollectionResult` handles raw JSON internally. - -### #6: Delete duplicated types - -`result_types.go` (~200 lines) redefines SDK types: `HostnameResult`, -`DiskResult`, `MemoryResult`, `LoadResult`, `CommandResult`, `PingResult`, -`DNSConfigResult`, `DNSUpdateResult`, `FileDeployOpts`, `FileDeployResult`, -`FileStatusResult`, `FileUploadResult`, `FileChangedResult`, `AgentResult`, -`AgentListResult`, plus sub-types. - -**Fix:** Delete `result_types.go`. Use `client.*` types directly throughout -`ops.go` and any other files that reference these types. This eliminates the -duplicate definitions and the field-by-field copies in `ops.go`. - -### #9: HealthCheck target parameter - -`HealthCheck` accepts a `target` parameter but ignores it. Liveness checks hit -the API server directly — target routing doesn't apply. - -**Fix:** Remove the unused parameter. - -### #10: Report.Summary() duplication - -The orchestrator reimplements `Summary()` instead of delegating to the SDK's -version. - -**Fix:** Delegate to `sdk.Report.Summary()`. - -### Additional: Replace buildResult/toMap with SDK helpers - -Once SDK fixes #1 and #7 land: - -- Replace local `buildResult` with `orchestrator.CollectionResult` -- Replace local `toMap` with `orchestrator.StructToMap` -- Delete `mustRawToMap` (no longer needed) - -## What Does NOT Change - -- The orchestrator DAG engine (plan, task, runner) — already solid -- `Response[T]` and `Collection[T]` pattern — correct design -- Error hierarchy (`checkError`, `AuthError`, etc.) — clean -- `MetricsService` using `http.DefaultClient` — intentional, `/metrics` is a - Prometheus endpoint outside the auth middleware - -## Order of Operations - -1. Fix osapi SDK first (#1, #2, #5, #7, #8) — with tests, 100% coverage -2. Fix osapi-orchestrator (#4, #6, #9, #10, plus adopt SDK helpers) — depends on - SDK changes being published diff --git a/docs/plans/2026-03-13-sdk-quality-fixes.md b/docs/plans/2026-03-13-sdk-quality-fixes.md deleted file mode 100644 index de0fcc425..000000000 --- a/docs/plans/2026-03-13-sdk-quality-fixes.md +++ /dev/null @@ -1,507 +0,0 @@ -# SDK Quality Fixes Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Fix 9 SDK quality issues identified by code review — add JSON tags to -result types, wrap Docker gen types, add Collection.First(), fix -CollectionResult to populate Result.Data, fix error wrapping, then update -osapi-orchestrator to use SDK types directly. - -**Architecture:** Fix the SDK client types first (JSON tags, Docker wrappers, -Collection.First), then fix the bridge helper (CollectionResult), then update -osapi-orchestrator to consume the improvements (delete duplicated types, use SDK -helpers). - -**Tech Stack:** Go 1.25, generics, testify/suite - ---- - -## Chunk 1: osapi SDK fixes - -### Task 1: Add JSON tags to all SDK client result types - -**Files:** - -- Modify: `pkg/sdk/client/node_types.go` -- Modify: `pkg/sdk/client/docker_types.go` -- Modify: `pkg/sdk/client/file_types.go` -- Modify: `pkg/sdk/client/audit_types.go` -- Modify: `pkg/sdk/client/health_types.go` -- Modify: `pkg/sdk/client/job_types.go` -- Modify: `pkg/sdk/client/agent_types.go` - -- [ ] **Step 1: Add JSON tags to node_types.go** - -Add `json:"..."` tags to all exported struct fields in these types: -`Collection`, `Disk`, `HostnameResult`, `NodeStatus`, `DiskResult`, -`MemoryResult`, `LoadResult`, `OSInfoResult`, `UptimeResult`, `DNSConfig`, -`DNSUpdateResult`, `PingResult`, `CommandResult`, `LoadAverage`, `Memory`, -`OSInfo`. - -Use snake_case keys matching the API response format. For example: - -```go -type Collection[T any] struct { - Results []T `json:"results"` - JobID string `json:"job_id"` -} - -type HostnameResult struct { - Hostname string `json:"hostname"` - Error string `json:"error,omitempty"` - Changed bool `json:"changed"` - Labels map[string]string `json:"labels,omitempty"` -} -``` - -Apply to all types — every exported field gets a `json` tag. - -- [ ] **Step 2: Add JSON tags to docker_types.go** - -Same pattern for: `DockerResult`, `DockerListResult`, `DockerSummaryItem`, -`DockerDetailResult`, `DockerActionResult`, `DockerExecResult`, -`DockerPullResult`. - -- [ ] **Step 3: Add JSON tags to remaining types files** - -Add tags to all result/model types in `file_types.go`, `audit_types.go`, -`health_types.go`, `job_types.go`, `agent_types.go`. - -- [ ] **Step 4: Run tests** - -Run: `go test ./pkg/sdk/client/... -count=1` Expected: PASS — JSON tags don't -break existing behavior. - -- [ ] **Step 5: Commit** - -``` -feat(sdk): add JSON tags to all client result types -``` - ---- - -### Task 2: Add Collection[T].First() method - -**Files:** - -- Modify: `pkg/sdk/client/node_types.go` -- Test: `pkg/sdk/client/node_types_test.go` (or appropriate test file) - -- [ ] **Step 1: Write the failing test** - -Add to the existing client test suite: - -```go -func (s *SuiteType) TestCollectionFirst() { - tests := []struct { - name string - col client.Collection[client.HostnameResult] - validateFunc func(client.HostnameResult, bool) - }{ - { - name: "returns first result and true", - col: client.Collection[client.HostnameResult]{ - Results: []client.HostnameResult{ - {Hostname: "web-01"}, - {Hostname: "web-02"}, - }, - JobID: "job-1", - }, - validateFunc: func(r client.HostnameResult, ok bool) { - s.True(ok) - s.Equal("web-01", r.Hostname) - }, - }, - { - name: "returns zero value and false when empty", - col: client.Collection[client.HostnameResult]{ - Results: []client.HostnameResult{}, - }, - validateFunc: func(r client.HostnameResult, ok bool) { - s.False(ok) - s.Equal("", r.Hostname) - }, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - r, ok := tt.col.First() - tt.validateFunc(r, ok) - }) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./pkg/sdk/client/... -run TestCollectionFirst -v` Expected: FAIL — -`First` not defined - -- [ ] **Step 3: Implement First()** - -Add to `node_types.go` after `Collection` definition: - -```go -// First returns the first result and true, or the zero value -// and false if the collection is empty. -func (c Collection[T]) First() (T, bool) { - if len(c.Results) == 0 { - var zero T - return zero, false - } - - return c.Results[0], true -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `go test ./pkg/sdk/client/... -run TestCollectionFirst -v` Expected: PASS - -- [ ] **Step 5: Commit** - -``` -feat(sdk): add Collection[T].First() method -``` - ---- - -### Task 3: Wrap Docker gen types with SDK-defined request types - -**Files:** - -- Modify: `pkg/sdk/client/docker_types.go` -- Modify: `pkg/sdk/client/docker.go` -- Modify: `pkg/sdk/client/docker_public_test.go` - -- [ ] **Step 1: Define SDK Docker request types in docker_types.go** - -Add after the result types: - -```go -// DockerCreateOpts contains options for creating a container. -type DockerCreateOpts struct { - // Image is the container image reference (required). - Image string - // Name is an optional container name. - Name string - // Command overrides the image's default command. - Command []string - // Env is environment variables in KEY=VALUE format. - Env []string - // Ports is port mappings in host_port:container_port format. - Ports []string - // Volumes is volume mounts in host_path:container_path format. - Volumes []string - // AutoStart starts the container after creation (default true). - AutoStart *bool -} - -// DockerStopOpts contains options for stopping a container. -type DockerStopOpts struct { - // Timeout is seconds to wait before killing. Zero uses default. - Timeout int -} - -// DockerListParams contains parameters for listing containers. -type DockerListParams struct { - // State filters by state: "running", "stopped", "all". - State string - // Limit caps the number of results. - Limit int -} - -// DockerRemoveParams contains parameters for removing a container. -type DockerRemoveParams struct { - // Force forces removal of a running container. - Force bool -} -``` - -- [ ] **Step 2: Update Docker service methods to use SDK types** - -Change method signatures in `docker.go`: - -`Create`: Change `body gen.DockerCreateRequest` to `opts DockerCreateOpts`. -Inside, build `gen.DockerCreateRequest` from the opts fields, converting zero -values to nil pointers. - -`Stop`: Change `body gen.DockerStopRequest` to `opts DockerStopOpts`. Build -`gen.DockerStopRequest` from opts. - -`List`: Change `params *gen.GetNodeContainerDockerParams` to -`params *DockerListParams`. Build gen params from SDK params. - -`Remove`: Change `params *gen.DeleteNodeContainerDockerByIDParams` to -`params *DockerRemoveParams`. Build gen params from SDK params. - -- [ ] **Step 3: Update tests** - -Update `docker_public_test.go` to use the new SDK types instead of gen types. -Also update any examples that reference the old signatures. - -- [ ] **Step 4: Update all callers** - -Search for `gen.DockerCreateRequest`, `gen.DockerStopRequest`, -`gen.GetNodeContainerDockerParams`, `gen.DeleteNodeContainerDockerByIDParams` -in: - -- `examples/sdk/client/container.go` -- `examples/sdk/orchestrator/features/container-targeting.go` -- `examples/sdk/orchestrator/operations/docker-*.go` - -Replace with the new SDK types. - -- [ ] **Step 5: Build and test** - -Run: `go build ./...` Run: `go test ./pkg/sdk/client/... -count=1` Run: Build -each docker example: -`go build examples/sdk/orchestrator/operations/docker-pull.go` etc. Expected: -All compile, all tests pass - -- [ ] **Step 6: Commit** - -``` -refactor(sdk): wrap Docker gen types with SDK-defined request types -``` - ---- - -### Task 4: Fix CollectionResult to populate Result.Data - -**Files:** - -- Modify: `pkg/sdk/orchestrator/bridge.go` -- Modify: `pkg/sdk/orchestrator/bridge_public_test.go` -- Modify: `pkg/sdk/orchestrator/bridge_test.go` - -- [ ] **Step 1: Update CollectionResult signature** - -Add `rawJSON []byte` parameter: - -```go -func CollectionResult[T any]( - col client.Collection[T], - rawJSON []byte, - toHostResult func(T) HostResult, -) *Result -``` - -When `rawJSON` is non-nil, unmarshal into `Result.Data`. Use `jsonUnmarshalFn` -(already injectable for testing). - -- [ ] **Step 2: Update tests** - -Update all test cases in `bridge_public_test.go` to pass `nil` for `rawJSON` -(existing behavior preserved). Add new test cases: - -- rawJSON populated: pass valid JSON, verify Result.Data is set -- rawJSON nil: verify Result.Data is nil (existing behavior) -- rawJSON invalid: verify Result.Data is nil (graceful degradation) - -- [ ] **Step 3: Update all callers** - -Update all example files that call `CollectionResult` to pass `resp.RawJSON()` -as the second argument. - -- [ ] **Step 4: Run tests** - -Run: `go test ./pkg/sdk/orchestrator/... -count=1` Run: Build all examples -Expected: PASS, all compile - -- [ ] **Step 5: Commit** - -``` -feat(orchestrator): populate Result.Data from raw JSON in CollectionResult -``` - ---- - -### Task 5: Fix AuditService.Get UUID error wrapping - -**Files:** - -- Modify: `pkg/sdk/client/audit.go:78-81` - -- [ ] **Step 1: Fix the error wrapping** - -Change: - -```go -parsedID, err := uuid.Parse(id) -if err != nil { - return nil, err -} -``` - -To: - -```go -parsedID, err := uuid.Parse(id) -if err != nil { - return nil, fmt.Errorf("invalid audit ID: %w", err) -} -``` - -- [ ] **Step 2: Update test if one exists** - -Check if there's a test for invalid audit ID. If so, update the expected error -message. - -- [ ] **Step 3: Run tests** - -Run: `go test ./pkg/sdk/client/... -count=1` Expected: PASS - -- [ ] **Step 4: Commit** - -``` -fix(sdk): wrap audit UUID parse error with context -``` - ---- - -### Task 6: Final SDK verification - -- [ ] **Step 1: Full test suite** - -Run: `go test ./... -count=1` Expected: All pass - -- [ ] **Step 2: Lint and format** - -```bash -find . -type f -name '*.go' -not -name '*.gen.go' -not -name '*.pb.go' \ - -not -path './.worktrees/*' -not -path './.claude/*' \ - | xargs go tool github.com/segmentio/golines \ - --base-formatter="go tool mvdan.cc/gofumpt" -w -go tool github.com/golangci/golangci-lint/v2/cmd/golangci-lint run \ - --config .golangci.yml -``` - -Expected: 0 issues - -- [ ] **Step 3: Coverage** - -Run: -`go test ./pkg/sdk/... -coverprofile=/tmp/sdk.out && go tool cover -func=/tmp/sdk.out | grep -v gen | grep -v '100.0%'` -Expected: All SDK packages at 100% (excluding gen) - -- [ ] **Step 4: Build all examples** - -```bash -for f in examples/sdk/orchestrator/operations/*.go; do - go build "$f" 2>&1 || echo "FAIL: $f" -done -for f in examples/sdk/orchestrator/features/*.go; do - go build "$f" 2>&1 || echo "FAIL: $f" -done -``` - -Expected: Zero failures - -- [ ] **Step 5: Commit any fixes** - -``` -chore: SDK quality fixes verification -``` - ---- - -## Chunk 2: osapi-orchestrator fixes - -These changes are in the separate repo at `~/git/osapi-io/osapi-orchestrator/`. - -### Task 7: Update osapi-orchestrator to use SDK types directly - -**Files:** - -- Delete: `pkg/orchestrator/result_types.go` -- Modify: `pkg/orchestrator/ops.go` -- Modify: `pkg/orchestrator/result.go` -- Modify: any test files that reference deleted types - -**Prerequisites:** Tasks 1-6 must be complete and the updated osapi SDK must be -available (update `go.mod` to point at the new version or use a `replace` -directive). - -- [ ] **Step 1: Update go.mod to use latest SDK** - -Either `go get github.com/osapi-io/osapi@latest` or add a `replace` directive -pointing to the local checkout. - -- [ ] **Step 2: Delete result_types.go** - -Remove the file entirely. All types it defines have equivalents in the SDK -`client` package. - -- [ ] **Step 3: Update ops.go imports and types** - -Replace all local type references with SDK `client.*` types: - -- `HostnameResult` → `client.HostnameResult` -- `CommandResult` → `client.CommandResult` -- `FileDeployOpts` → `client.FileDeployOpts` -- etc. - -Replace `buildResult` calls with `orchestrator.CollectionResult`: - -```go -return orchestrator.CollectionResult(resp.Data, resp.RawJSON(), - func(r client.HostnameResult) orchestrator.HostResult { - return orchestrator.HostResult{ - Hostname: r.Hostname, - Changed: r.Changed, - Error: r.Error, - } - }, -), nil -``` - -Delete `buildResult`, `toMap`, `mustRawToMap` helper functions. - -- [ ] **Step 4: Fix mustRawToMap callers that aren't Collection** - -For non-collection operations (FileDeploy, FileStatus, FileUpload, FileChanged, -AgentList, AgentGet), replace: - -```go -Data: mustRawToMap(resp.RawJSON()), -``` - -With: - -```go -Data: orchestrator.StructToMap(resp.Data), -``` - -This works now because SDK types have JSON tags (Task 1). - -- [ ] **Step 5: Update result.go** - -Delete duplicated `Summary()` — delegate to `sdk.Report.Summary()`. - -- [ ] **Step 6: Fix HealthCheck target parameter** - -Remove the unused `target string` parameter from `HealthCheck()`. Update all -callers. - -- [ ] **Step 7: Update tests** - -Fix all test files that reference deleted types or changed signatures. Run full -test suite. - -- [ ] **Step 8: Build and test** - -Run: `go test ./... -count=1` Run: `go build ./...` Expected: All pass, all -compile - -- [ ] **Step 9: Commit** - -``` -refactor: use SDK types directly, remove duplicated types - -Delete result_types.go (~200 lines of duplicated SDK types). -Replace buildResult/toMap/mustRawToMap with SDK bridge helpers. -Use client.* types directly throughout ops.go. -``` diff --git a/docs/plans/2026-03-14-component-health-design.md b/docs/plans/2026-03-14-component-health-design.md deleted file mode 100644 index 7f76224f1..000000000 --- a/docs/plans/2026-03-14-component-health-design.md +++ /dev/null @@ -1,313 +0,0 @@ -# Component Health and Notifications Design - -## Problem - -Only agents heartbeat. The API server and NATS server have no presence in the -registry — if they're degraded, the only signal is a failed HTTP call or a NATS -timeout. There's no single view showing all component health, process resource -usage, or condition state. Conditions exist on agents but nothing reacts to -them. - -## Goals - -1. All three component types (agent, API server, NATS server) heartbeat with - process metrics and conditions. -2. Health status (`/health/status`) shows a unified component table with TYPE, - HOSTNAME, STATUS, CONDITIONS, AGE, CPU, MEM. -3. Condition transitions trigger a pluggable notification interface (logging - stub for now, extensible to Slack/email/webhook later). -4. Remove the `osapi client metrics` CLI command (Prometheus endpoint stays for - scraping). - -## Component Heartbeat - -### What gets written - -Every component writes a heartbeat to the registry KV bucket on a configurable -interval. The payload includes: - -```go -type ComponentRegistration struct { - // Type is "agent", "api", or "nats". - Type string - Hostname string - StartedAt time.Time - RegisteredAt time.Time - - // Process metrics — CPU and RSS for the running process. - Process *ProcessMetrics - - // Conditions — evaluated against thresholds. - Conditions []Condition - - // Agent-specific fields (nil for api/nats). - // Labels, OSInfo, LoadAverages, MemoryStats, etc. - // These remain on AgentRegistration which embeds - // ComponentRegistration. -} - -type ProcessMetrics struct { - CPUPercent float64 - RSSBytes int64 - Goroutines int -} -``` - -Agent heartbeat already collects host-level data (OS, load, memory, disk). That -stays. The new `ProcessMetrics` is added alongside it — CPU/memory for the osapi -process itself, not the host. - -### KV key structure - -Keep the existing `agent-registry` bucket. Add a type prefix to keys: - -``` -agent.web-01 → AgentRegistration (embeds ComponentRegistration) -agent.web-02 → AgentRegistration -api.api-server-01 → ComponentRegistration -nats.nats-server-01 → ComponentRegistration -``` - -The existing key format is `agents.{hostname}`. Changing to `agent.{hostname}` -(no trailing s) is a breaking change. Options: - -**Option A**: Keep `agents.` prefix for backward compatibility. Use `api.` and -`nats.` for new component types. `ListAgents` filters by `agents.` prefix. - -**Option B**: Migrate to `agent.` prefix. One-time breaking change. Cleaner -going forward. - -**Recommendation**: Option A. No migration needed. The prefix inconsistency -(`agents.` vs `agent`) is cosmetic and not worth a breaking change. - -### TTL - -Same TTL as agent registry (configurable, default 30s). If a component's -heartbeat expires, it disappears from health status — the same liveness -mechanism agents use. - -### Collection - -Process metrics are collected using Go's `runtime` package and `os.Process()`: - -- `runtime.NumGoroutine()` — goroutine count -- `process.MemoryInfo().RSS` — resident set size (via gopsutil or - /proc/self/status) -- `process.CPUPercent()` — CPU usage since last sample (via gopsutil) - -These are cheap calls — safe to run every heartbeat interval. - -### Where heartbeat runs - -- **Agent**: already has `startHeartbeat()`. Add `ProcessMetrics` to the - existing `AgentRegistration`. -- **API server**: add `startHeartbeat()` to the API server lifecycle. Writes a - `ComponentRegistration` with type `"api"`. -- **NATS server**: add `startHeartbeat()` to the NATS server lifecycle. Writes a - `ComponentRegistration` with type `"nats"`. If the NATS server is external - (not embedded), this heartbeat doesn't run — and that's fine, the component - just won't appear in the table. - -### Conditions - -Agent conditions already exist: `MemoryPressure`, `HighLoad`, `DiskPressure`. -These are host-level. - -Add process-level conditions for all components: - -- `ProcessMemoryPressure` — process RSS exceeds threshold -- `ProcessHighCPU` — process CPU exceeds threshold - -Thresholds are configurable in `osapi.yaml` under each component's config -section. Conditions are evaluated on the component side and written to the -heartbeat — same pattern as agent host conditions. - -## Health Status Enrichment - -### Component table - -`GET /health/status` reads all keys from the registry KV bucket, groups by type -prefix, and returns a component list: - -```json -{ - "status": "ok", - "components": { - "api": {"status": "ok"}, - "nats": {"status": "ok"}, - "kv": {"status": "ok"} - }, - "registry": [ - { - "type": "api", - "hostname": "api-server-01", - "status": "Ready", - "conditions": [], - "age": "7h 6m", - "cpu_percent": 2.1, - "mem_bytes": 134217728 - }, - { - "type": "nats", - "hostname": "nats-server-01", - "status": "Ready", - "conditions": [], - "age": "7h 6m", - "cpu_percent": 0.3, - "mem_bytes": 67108864 - }, - { - "type": "agent", - "hostname": "web-01", - "status": "Ready", - "conditions": ["DiskPressure"], - "age": "7h 6m", - "cpu_percent": 1.2, - "mem_bytes": 100663296 - } - ], - "jobs": { ... }, - "nats": { ... }, - "streams": [ ... ], - "kv_buckets": [ ... ] -} -``` - -The `registry` array replaces the current `agents` field which only shows agent -count/ready. The existing infrastructure sections (jobs, NATS, streams, KV -buckets, object stores, consumers) stay as-is. - -### CLI output - -`osapi client health status` renders: - -``` -=== Components === - -TYPE HOSTNAME STATUS CONDITIONS AGE CPU MEM -api api-server-01 Ready - 7h 6m 2.1% 128MB -nats nats-server-01 Ready - 7h 6m 0.3% 64MB -agent Johns-MacBook-Pro-2.local Ready DiskPressure 7h 6m 1.2% 96MB -agent web-02 Ready - 3h 2m 0.8% 82MB - -=== Jobs === - - Pending: 2 - Completed: 147 - Failed: 3 - -=== NATS === - - URL: nats://localhost:4222 - Version: 2.10.x - Streams: 1 (1,234 msgs, 5.2 MB) - -=== KV Buckets === - - job-queue: 42 keys, 1.1 MB - agent-registry: 2 keys, 4.2 KB -``` - -Components table at the top — answers "is everything healthy?" at a glance. - -## Condition Notifications - -### Architecture - -The API server watches the registry KV bucket for condition transitions. When a -condition appears or disappears, it dispatches a notification through a -pluggable interface. - -```go -// Notifier sends notifications when component conditions change. -type Notifier interface { - Notify(ctx context.Context, event ConditionEvent) error -} - -type ConditionEvent struct { - ComponentType string // "agent", "api", "nats" - Hostname string - Condition string // "MemoryPressure", "DiskPressure", etc. - Status bool // true = condition active, false = resolved - Reason string - Timestamp time.Time -} -``` - -### Watcher - -The API server starts a KV watcher on the registry bucket. On each update, it -compares the previous condition set to the current one and emits -`ConditionEvent`s for transitions. - -The watcher runs as a background goroutine in the API server lifecycle. It's -designed to be extractable into a separate process later — the only dependency -is NATS KV access and a `Notifier` implementation. - -### Implementations - -**Phase 1 (this design):** - -- `LogNotifier` — logs condition events at INFO level. Default. - -**Future phases:** - -- `SlackNotifier` — posts to a Slack webhook -- `EmailNotifier` — sends email via SMTP -- `WebhookNotifier` — POSTs to a configurable URL - -### Configuration - -```yaml -notifications: - enabled: true - notifier: log - # Future: - # notifier: slack - # slack: - # webhook_url: https://hooks.slack.com/... - # notifier: webhook - # webhook: - # url: https://example.com/alerts -``` - -Top-level `notifications` key in `osapi.yaml`. The `notifier` field selects the -implementation. For now only `log` is available. - -## Remove `osapi client metrics` CLI - -Delete `cmd/client_metrics.go`. The `/metrics` HTTP endpoint stays (Prometheus -scrapes it directly). The CLI command that fetches and prints raw Prometheus -text is not useful for humans. - -Also delete: - -- `pkg/sdk/client/metrics.go` — SDK MetricsService -- `docs/docs/sidebar/sdk/client/metrics.md` — SDK metrics docs -- CLI docs for the metrics command - -The `MetricsService` in the SDK is the only service that bypasses the auth -transport (`http.DefaultClient`). Removing it eliminates that inconsistency. - -## What Does NOT Change - -- Agent heartbeat data (OS, load, memory, disk, labels, facts) — stays -- Agent conditions (MemoryPressure, HighLoad, DiskPressure) — stays -- `osapi client agent list` output — stays (shows agent-specific detail) -- `osapi client agent get` output — stays -- `/metrics` Prometheus HTTP endpoint — stays -- KV bucket configuration — no new buckets needed -- NATS namespace — no changes - -## Order of Implementation - -1. Process metrics collection (shared between all components) -2. API server heartbeat -3. NATS server heartbeat (embedded only) -4. Agent heartbeat enrichment (add ProcessMetrics) -5. Health status API changes (registry array, component table) -6. Health status CLI changes (component table output) -7. Condition notification watcher + LogNotifier -8. Configuration (`notifications` section in osapi.yaml) -9. Remove `osapi client metrics` CLI + SDK MetricsService -10. Documentation updates diff --git a/docs/plans/2026-03-14-component-health.md b/docs/plans/2026-03-14-component-health.md deleted file mode 100644 index a99fdca17..000000000 --- a/docs/plans/2026-03-14-component-health.md +++ /dev/null @@ -1,670 +0,0 @@ -# Component Health Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make all three components (agent, API server, NATS server) heartbeat -with process metrics and conditions, enrich health status with a unified -component table, add a pluggable condition notification system, and remove the -metrics CLI command. - -**Architecture:** Add shared process metrics collection used by all components. -API server and NATS server get heartbeat writers alongside the existing agent -heartbeat. Health status reads all component registrations from KV and renders a -unified table. A KV watcher on the API server dispatches condition transitions -through a pluggable `Notifier` interface. - -**Tech Stack:** Go 1.25, gopsutil (process metrics), NATS JetStream KV, -testify/suite - ---- - -## Chunk 1: Process Metrics and Component Registration Types - -### Task 1: Add process metrics collector - -**Files:** - -- Create: `internal/provider/process/process.go` -- Create: `internal/provider/process/types.go` -- Test: `internal/provider/process/process_public_test.go` - -- [ ] **Step 1: Define types** - -Create `types.go` with: - -```go -package process - -// Metrics holds process-level resource usage. -type Metrics struct { - CPUPercent float64 `json:"cpu_percent"` - RSSBytes int64 `json:"rss_bytes"` - Goroutines int `json:"goroutines"` -} - -// Provider collects process metrics. -type Provider interface { - GetMetrics() (*Metrics, error) -} -``` - -- [ ] **Step 2: Implement provider** - -Create `process.go` using `runtime` and `os` packages. Use -`github.com/shirou/gopsutil/v4/process` (already a dependency via -host/disk/mem/load providers) for CPU% and RSS: - -```go -func New() Provider { return &provider{pid: int32(os.Getpid())} } - -func (p *provider) GetMetrics() (*Metrics, error) { - proc, err := gopsutil.NewProcess(p.pid) - // cpu%, rss, runtime.NumGoroutine() -} -``` - -- [ ] **Step 3: Write tests** - -Test that `GetMetrics` returns non-nil with positive goroutine count and -non-negative CPU/RSS. Use real process — no mocking needed for a self-inspection -provider. - -- [ ] **Step 4: Add mockgen** - -Create `internal/provider/process/mocks/generate.go`: - -```go -//go:generate go tool github.com/golang/mock/mockgen -source=../types.go -destination=types.gen.go -package=mocks -``` - -Run `go generate ./internal/provider/process/mocks/...` - -- [ ] **Step 5: Build and test** - -Run: `go build ./...` and `go test ./internal/provider/process/...` - -- [ ] **Step 6: Commit** - -``` -feat: add process metrics provider (CPU, RSS, goroutines) -``` - ---- - -### Task 2: Add ComponentRegistration type - -**Files:** - -- Modify: `internal/job/types.go` - -- [ ] **Step 1: Add ComponentRegistration** - -Add after `AgentRegistration`: - -```go -// ComponentRegistration represents a component's heartbeat entry -// in the KV registry. Used by API server and NATS server. -type ComponentRegistration struct { - Type string `json:"type"` - Hostname string `json:"hostname"` - StartedAt time.Time `json:"started_at"` - RegisteredAt time.Time `json:"registered_at"` - Process *ProcessMetrics `json:"process,omitempty"` - Conditions []Condition `json:"conditions,omitempty"` - Version string `json:"version,omitempty"` -} - -// ProcessMetrics holds process-level resource usage. -type ProcessMetrics struct { - CPUPercent float64 `json:"cpu_percent"` - RSSBytes int64 `json:"rss_bytes"` - Goroutines int `json:"goroutines"` -} -``` - -- [ ] **Step 2: Add ProcessMetrics to AgentRegistration** - -Add field to `AgentRegistration`: - -```go -Process *ProcessMetrics `json:"process,omitempty"` -``` - -This keeps agent-specific fields (OS, load, memory, labels) on -`AgentRegistration` while sharing `ProcessMetrics` with all component types. - -- [ ] **Step 3: Build** - -Run: `go build ./...` - -- [ ] **Step 4: Commit** - -``` -feat: add ComponentRegistration and ProcessMetrics types -``` - ---- - -## Chunk 2: API Server and NATS Server Heartbeats - -### Task 3: Add API server heartbeat - -**Files:** - -- Create: `internal/api/heartbeat.go` -- Create: `internal/api/heartbeat_test.go` -- Modify: `cmd/api_server_setup.go` - -- [ ] **Step 1: Implement API server heartbeat writer** - -Create `internal/api/heartbeat.go`: - -```go -// StartHeartbeat writes a ComponentRegistration to the registry KV -// on a configurable interval. Call Stop() or cancel the context to -// shut down. -func StartHeartbeat( - ctx context.Context, - logger *slog.Logger, - registryKV jetstream.KeyValue, - hostname string, - version string, - processProvider process.Provider, - interval time.Duration, -) -``` - -Key format: `api.{hostname}`. Writes `ComponentRegistration` with `Type: "api"`. -Collects process metrics each tick. Evaluates process-level conditions -(ProcessMemoryPressure, ProcessHighCPU) using configurable thresholds. - -Follow the same pattern as `internal/agent/heartbeat.go`: - -- Ticker loop with context cancellation -- Deregister on shutdown (delete KV key) -- Log warnings on errors, don't fail - -- [ ] **Step 2: Write tests** - -Test in `heartbeat_test.go` (internal test, package `api`): - -- Writes registration to mock KV on tick -- Deletes key on context cancel -- Process metrics populated - -- [ ] **Step 3: Wire into API server startup** - -In `cmd/api_server_setup.go`, after connecting to NATS and getting the registry -KV: - -- Create process provider -- Start heartbeat goroutine -- Stop on shutdown - -- [ ] **Step 4: Build and test** - -Run: `go build ./...` and `go test ./internal/api/...` - -- [ ] **Step 5: Commit** - -``` -feat: add API server heartbeat to component registry -``` - ---- - -### Task 4: Add NATS server heartbeat - -**Files:** - -- Create: `cmd/nats_heartbeat.go` -- Modify: `cmd/nats_setup.go` -- Modify: `cmd/start.go` - -- [ ] **Step 1: Implement NATS server heartbeat** - -Create `cmd/nats_heartbeat.go` with a heartbeat function similar to the API -server's, but writes with key `nats.{hostname}` and `Type: "nats"`. - -The NATS server heartbeat needs its own NATS client connection (separate from -the server itself) to write to KV. This is created during `setupJetStream`. - -- [ ] **Step 2: Wire into NATS server startup** - -In `cmd/nats_setup.go` or `cmd/start.go`, start the NATS heartbeat after -JetStream is set up. Only when running the embedded server (not when connecting -to an external NATS cluster). - -- [ ] **Step 3: Build and test** - -Run: `go build ./...` - -- [ ] **Step 4: Commit** - -``` -feat: add NATS server heartbeat to component registry -``` - ---- - -### Task 5: Add process metrics to agent heartbeat - -**Files:** - -- Modify: `internal/agent/heartbeat.go` -- Modify: `internal/agent/types.go` -- Modify: `internal/agent/agent.go` -- Modify: `internal/agent/factory.go` - -- [ ] **Step 1: Add process provider to agent** - -Add `processProvider process.Provider` to `Agent` struct. Initialize in -`factory.go` with `process.New()`. Pass through `New()`. - -- [ ] **Step 2: Collect process metrics in heartbeat** - -In `writeRegistration`, add: - -```go -if pm, err := a.processProvider.GetMetrics(); err == nil { - reg.Process = &job.ProcessMetrics{ - CPUPercent: pm.CPUPercent, - RSSBytes: pm.RSSBytes, - Goroutines: pm.Goroutines, - } -} -``` - -- [ ] **Step 3: Update tests** - -Update heartbeat tests to provide a mock process provider. - -- [ ] **Step 4: Build and test** - -Run: `go build ./...` and `go test ./internal/agent/...` - -- [ ] **Step 5: Commit** - -``` -feat: add process metrics to agent heartbeat -``` - ---- - -## Chunk 3: Health Status Enrichment - -### Task 6: Update health OpenAPI spec and MetricsProvider - -**Files:** - -- Modify: `internal/api/health/gen/api.yaml` -- Modify: `internal/api/health/types.go` - -- [ ] **Step 1: Add ComponentEntry schema to OpenAPI spec** - -Add to the health spec's schemas section: - -```yaml -ComponentEntry: - type: object - properties: - type: - type: string - description: Component type (agent, api, nats). - hostname: - type: string - status: - type: string - conditions: - type: array - items: - type: string - age: - type: string - cpu_percent: - type: number - mem_bytes: - type: integer - format: int64 -``` - -Add `registry` field to `StatusResponse`: - -```yaml -registry: - type: array - items: - $ref: '#/components/schemas/ComponentEntry' - description: All registered components with health details. -``` - -- [ ] **Step 2: Regenerate** - -Run: `just generate` - -- [ ] **Step 3: Add GetComponentRegistry to MetricsProvider** - -Add method to `MetricsProvider` interface: - -```go -GetComponentRegistry(ctx context.Context) ([]ComponentEntry, error) -``` - -Add `ComponentEntry` type to `types.go`: - -```go -type ComponentEntry struct { - Type string - Hostname string - Status string - Conditions []string - Age string - CPUPercent float64 - MemBytes int64 -} -``` - -Update `ClosureMetricsProvider` with `ComponentRegistryFn`. - -- [ ] **Step 4: Build** - -Run: `go build ./...` - -- [ ] **Step 5: Commit** - -``` -feat: add ComponentEntry to health spec and MetricsProvider -``` - ---- - -### Task 7: Implement component registry collection - -**Files:** - -- Modify: `cmd/api_server_setup.go` -- Modify: `internal/api/health/health_status_get.go` - -- [ ] **Step 1: Add ComponentRegistryFn to metrics provider setup** - -In `cmd/api_server_setup.go`, add the `ComponentRegistryFn` closure that reads -all keys from the registry KV bucket, parses each as either `AgentRegistration` -or `ComponentRegistration` (based on key prefix), and returns -`[]ComponentEntry`. - -Key prefix routing: - -- `agents.*` → parse as `AgentRegistration`, type = "agent" -- `api.*` → parse as `ComponentRegistration`, type = "api" -- `nats.*` → parse as `ComponentRegistration`, type = "nats" - -For agents, map conditions from the `Conditions` field. For all types, calculate -age from `StartedAt`. Extract CPU/MEM from `ProcessMetrics`. - -- [ ] **Step 2: Add registry to populateMetrics** - -In `health_status_get.go`, add a `collect("registry", ...)` call that runs -`GetComponentRegistry` and maps to the response schema. - -- [ ] **Step 3: Update tests** - -Add test cases for the registry collection — agents + API + NATS components. - -- [ ] **Step 4: Build and test** - -Run: `go build ./...` and `go test ./internal/api/health/...` - -- [ ] **Step 5: Commit** - -``` -feat: collect component registry in health status -``` - ---- - -### Task 8: Update health status CLI output - -**Files:** - -- Modify: `cmd/client_health_status.go` - -- [ ] **Step 1: Add component table to CLI output** - -Render the component registry as a table at the top of the health status output. -Format: - -``` -=== Components === - -TYPE HOSTNAME STATUS CONDITIONS AGE CPU MEM -api api-server-01 Ready - 7h 6m 2.1% 128MB -nats nats-server-01 Ready - 7h 6m 0.3% 64MB -agent web-01 Ready DiskPressure 7h 6m 1.2% 96MB -``` - -Use `cli.PrintCompactTable` with the component data from the response. Format -CPU as `X.X%`, MEM with `FormatBytes()`. - -Keep existing infrastructure sections (Jobs, NATS, Streams, etc.) below the -component table. - -- [ ] **Step 2: Build and test manually** - -Run: `go build ./...` and test with `go run main.go client health status` - -- [ ] **Step 3: Commit** - -``` -feat: add component table to health status CLI output -``` - ---- - -## Chunk 4: Condition Notifications - -### Task 9: Add Notifier interface and LogNotifier - -**Files:** - -- Create: `internal/notify/types.go` -- Create: `internal/notify/log.go` -- Test: `internal/notify/log_public_test.go` - -- [ ] **Step 1: Define Notifier interface and ConditionEvent** - -```go -package notify - -type ConditionEvent struct { - ComponentType string - Hostname string - Condition string - Status bool // true = active, false = resolved - Reason string - Timestamp time.Time -} - -type Notifier interface { - Notify(ctx context.Context, event ConditionEvent) error -} -``` - -- [ ] **Step 2: Implement LogNotifier** - -```go -type LogNotifier struct { - logger *slog.Logger -} - -func NewLogNotifier(logger *slog.Logger) *LogNotifier - -func (n *LogNotifier) Notify(ctx context.Context, event ConditionEvent) error -``` - -Logs at INFO level: `"condition transition"` with structured fields for -component type, hostname, condition, status, reason. - -- [ ] **Step 3: Write tests** - -Test `LogNotifier.Notify` produces no error. Verify the interface is satisfied. - -- [ ] **Step 4: Commit** - -``` -feat: add Notifier interface and LogNotifier -``` - ---- - -### Task 10: Add condition watcher - -**Files:** - -- Create: `internal/notify/watcher.go` -- Test: `internal/notify/watcher_test.go` -- Modify: `cmd/api_server_setup.go` - -- [ ] **Step 1: Implement KV watcher** - -```go -type Watcher struct { - kv jetstream.KeyValue - notifier Notifier - logger *slog.Logger - prev map[string][]string // hostname → active conditions -} - -func NewWatcher( - kv jetstream.KeyValue, - notifier Notifier, - logger *slog.Logger, -) *Watcher - -func (w *Watcher) Start(ctx context.Context) error -``` - -Watches the registry KV bucket. On each update, parses the registration, -compares conditions to `prev`, and emits `ConditionEvent`s for transitions (new -condition → active, removed condition → resolved). - -- [ ] **Step 2: Write tests** - -Test condition transition detection: - -- New condition appears → Notify called with Status=true -- Condition disappears → Notify called with Status=false -- No change → no notification - -- [ ] **Step 3: Wire into API server** - -In `cmd/api_server_setup.go`, create the watcher with LogNotifier and start it -as a background goroutine. Stop on shutdown. - -- [ ] **Step 4: Add config** - -Add to `internal/config/types.go`: - -```go -type NotificationsConfig struct { - Enabled bool `mapstructure:"enabled"` - Notifier string `mapstructure:"notifier"` -} -``` - -Add `Notifications NotificationsConfig` to `Config` struct. - -Update `docs/docs/sidebar/usage/configuration.md` with the new `notifications` -section. - -- [ ] **Step 5: Build and test** - -Run: `go build ./...` and `go test ./internal/notify/...` - -- [ ] **Step 6: Commit** - -``` -feat: add condition watcher with LogNotifier -``` - ---- - -## Chunk 5: Cleanup and Documentation - -### Task 11: Remove metrics CLI command - -**Files:** - -- Delete: `cmd/client_metrics.go` -- Delete: `pkg/sdk/client/metrics.go` -- Delete: `pkg/sdk/client/metrics_public_test.go` -- Delete: `docs/docs/sidebar/sdk/client/metrics.md` -- Modify: `pkg/sdk/client/osapi.go` (remove Metrics field) -- Modify: `docs/docs/sidebar/sdk/client/client.md` (remove Metrics row) - -- [ ] **Step 1: Delete files** - -Remove the CLI command, SDK service, tests, and docs. - -- [ ] **Step 2: Update SDK client** - -Remove `Metrics *MetricsService` from `Client` struct and the initialization in -`New()`. - -- [ ] **Step 3: Update docs** - -Remove Metrics from the SDK client services table. - -- [ ] **Step 4: Build and test** - -Run: `go build ./...` and `go test ./...` - -- [ ] **Step 5: Commit** - -``` -refactor: remove metrics CLI command and SDK MetricsService - -The /metrics Prometheus HTTP endpoint stays for scraping. -The CLI command that prints raw Prometheus text is removed. -``` - ---- - -### Task 12: Documentation and verification - -**Files:** - -- Modify: `docs/docs/sidebar/features/health-checks.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` -- Modify: `CLAUDE.md` - -- [ ] **Step 1: Update health checks feature doc** - -Document the component registry, process metrics, and condition notifications. - -- [ ] **Step 2: Update configuration reference** - -Add `notifications` section. Document process condition thresholds. Add new env -vars to the table. - -- [ ] **Step 3: Update architecture docs** - -Mention component heartbeat in the architecture overview. - -- [ ] **Step 4: Full verification** - -```bash -go build ./... -go test ./... -count=1 -just go::vet -cd docs && bun run build -``` - -All must pass. - -- [ ] **Step 5: Commit** - -``` -docs: update docs for component health and notifications -``` diff --git a/docs/plans/2026-03-19-rename-api-to-controller-design.md b/docs/plans/2026-03-19-rename-api-to-controller-design.md deleted file mode 100644 index 920374c9f..000000000 --- a/docs/plans/2026-03-19-rename-api-to-controller-design.md +++ /dev/null @@ -1,210 +0,0 @@ -# Rename API Server to Controller — Design Spec - -## Goal - -Rename the "API server" to "controller" to reflect its role as the control plane -process that owns multiple sub-components (REST API, notification watcher, -heartbeat, and future metrics server). - -## Motivation - -The API server today does more than serve HTTP: - -- Runs the REST API (Echo) -- Runs the notification watcher (condition monitoring) -- Runs the component heartbeat -- Will run a metrics/ops server in the future - -"Controller" captures what it actually is — the control plane process. The API -is just one thing it exposes. - -## Config Changes - -### Before - -```yaml -api: - client: - url: 'http://localhost:8080' - security: - bearer_token: '' - server: - port: 8080 - nats: - host: localhost - port: 4222 - client_name: osapi-api - namespace: osapi - auth: - type: none - security: - signing_key: '' - cors: - allow_origins: [...] - roles: {} -``` - -### After - -```yaml -controller: - client: - url: 'http://localhost:8080' - security: - bearer_token: '' - api: - port: 8080 - security: - signing_key: '' - cors: - allow_origins: [...] - roles: {} - nats: - host: localhost - port: 4222 - client_name: osapi-api - namespace: osapi - auth: - type: none -``` - -Key changes: - -- `api` → `controller` (top-level) -- `api.server` → `controller.api` (the HTTP server config) -- `api.server.nats` → `controller.nats` (moved up, not nested under api) -- `api.client` → `controller.client` (unchanged structure) - -### Go config types - -```go -// Controller replaces the old API struct. -type Controller struct { - Client Client `mapstructure:"client"` - API APIServer `mapstructure:"api" mask:"struct"` - NATS NATSConnection `mapstructure:"nats"` -} - -// APIServer holds the HTTP server config (port + security). -// Replaces the old Server struct minus the NATS connection -// (which moves to Controller.NATS). -type APIServer struct { - Port int `mapstructure:"port"` - Security ServerSecurity `mapstructure:"security" mask:"struct"` -} -``` - -The `validate:"required"` tags on `signing_key` and `bearer_token` remain on -their existing structs (`ServerSecurity`, `ClientSecurity`). Validation works -the same way — Viper unmarshals into the new structure and the validator walks -the nested structs. - -### Environment variable mapping - -| Config Key | Environment Variable | -| -------------------------------------------- | -------------------------------------------------- | -| `controller.client.url` | `OSAPI_CONTROLLER_CLIENT_URL` | -| `controller.client.security.bearer_token` | `OSAPI_CONTROLLER_CLIENT_SECURITY_BEARER_TOKEN` | -| `controller.api.port` | `OSAPI_CONTROLLER_API_PORT` | -| `controller.api.security.signing_key` | `OSAPI_CONTROLLER_API_SECURITY_SIGNING_KEY` | -| `controller.api.security.cors.allow_origins` | `OSAPI_CONTROLLER_API_SECURITY_CORS_ALLOW_ORIGINS` | -| `controller.nats.host` | `OSAPI_CONTROLLER_NATS_HOST` | -| `controller.nats.port` | `OSAPI_CONTROLLER_NATS_PORT` | -| `controller.nats.client_name` | `OSAPI_CONTROLLER_NATS_CLIENT_NAME` | -| `controller.nats.namespace` | `OSAPI_CONTROLLER_NATS_NAMESPACE` | -| `controller.nats.auth.type` | `OSAPI_CONTROLLER_NATS_AUTH_TYPE` | - -## CLI Changes - -| Before | After | -| ------------------------ | ------------------------ | -| `osapi api server start` | `osapi controller start` | - -Unchanged: - -- `osapi client *` — all client commands stay the same -- `osapi agent start` -- `osapi nats server start` -- `osapi start` (all-in-one) — calls controller instead of API server - -## Code Changes - -### Directory moves - -| From | To | -| ------------------ | ----------------------------- | -| `internal/api/` | `internal/controller/api/` | -| `internal/notify/` | `internal/controller/notify/` | - -### New files - -| File | Purpose | -| ----------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -| `internal/controller/controller.go` | Controller struct with Start/Stop. Owns the API server, heartbeat, and condition watcher. Implements `cli.Lifecycle`. | -| `cmd/controller.go` | `controllerCmd` parent command | -| `cmd/controller_start.go` | `controller start` subcommand | -| `cmd/controller_setup.go` | Setup logic (moved from `api_server_setup.go`), config paths updated | - -### Removed files - -| File | Reason | -| ------------------------- | ------------------------------------- | -| `cmd/api_server.go` | Replaced by `cmd/controller.go` | -| `cmd/api_server_start.go` | Replaced by `cmd/controller_start.go` | -| `cmd/api_server_setup.go` | Replaced by `cmd/controller_setup.go` | - -### Modified files - -| File | Change | -| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -| `internal/config/types.go` | Replace `API` struct with `Controller` containing `APIServer`, `Client`, `NATSConnection`. Update `Config` struct field. | -| `cmd/start.go` | Call controller instead of API server | -| `cmd/client.go` | `appConfig.API` → `appConfig.Controller` | -| `cmd/client_*.go` | Update all config references | -| `test/integration/integration_test.go` | Update `api server start` → `controller start` in test harness | - -### Heartbeat - -`internal/api/heartbeat.go` moves to `internal/controller/heartbeat.go`. The -heartbeat registers the controller as a component in the registry KV. It is a -controller lifecycle concern — the API server doesn't need to know about it. The -agent has its own heartbeat in `internal/agent/` which is unrelated. - -### Notify - -`internal/notify/` moves to `internal/controller/notify/`. The condition watcher -monitors the registry KV and dispatches notifications. It runs as a goroutine -owned by the controller — it has no reason to exist outside the controller -process. - -## What doesn't change - -- All REST API paths (`/node/`, `/job/`, `/health/`, etc.) -- `osapi client *` CLI commands -- SDK client (`pkg/sdk/client/`) — no config references, only HTTP -- Agent code (`internal/agent/`) -- NATS server code -- OpenAPI specs and generated code (stays under `internal/controller/api/`) - -## Docs updates - -- `docs/docs/sidebar/usage/configuration.md` — full config reference with new - `controller.*` keys and env var table -- `docs/docs/sidebar/architecture/architecture.md` — rename "API Server" to - "Controller" in process descriptions -- `docs/docs/sidebar/architecture/system-architecture.md` — update package - layout, handler structure references -- `docs/docs/sidebar/usage/cli/` — update command docs for `controller start` -- `docs/docs/sidebar/development/development.md` — quick reference commands -- `docs/docs/sidebar/features/health-checks.md` — update references -- `docs/docs/sidebar/features/notifications.md` — update references -- `CLAUDE.md` — update architecture section (`internal/api/` → - `internal/controller/api/`), cmd references, config references - -## Breaking changes - -- Config: `api.*` → `controller.*` -- Env vars: `OSAPI_API_*` → `OSAPI_CONTROLLER_*` -- CLI: `osapi api server start` → `osapi controller start` -- Integration tests: `api server start` → `controller start` -- `osapi.yaml`: must be updated before upgrading diff --git a/docs/plans/2026-03-19-rename-api-to-controller.md b/docs/plans/2026-03-19-rename-api-to-controller.md deleted file mode 100644 index f9954211f..000000000 --- a/docs/plans/2026-03-19-rename-api-to-controller.md +++ /dev/null @@ -1,620 +0,0 @@ -# Rename API Server to Controller Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> (if subagents available) or superpowers:executing-plans to implement this -> plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Rename the "API server" to "controller" across config, CLI, code -structure, and docs to reflect its role as the control plane process. - -**Architecture:** Move `internal/api/` to `internal/controller/api/` and -`internal/notify/` to `internal/controller/notify/`. Create a new -`internal/controller/controller.go` that owns the API server, heartbeat, and -condition watcher. Rename `osapi api server start` to `osapi controller start`. -Update all config from `api.*` to `controller.*`. - -**Tech Stack:** Go 1.25, Cobra CLI, Viper config, Echo HTTP - ---- - -## Chunk 1: Config types and YAML - -### Task 1: Update config types - -**Files:** - -- Modify: `internal/config/types.go` - -- [ ] **Step 1: Replace API struct with Controller** - -Replace the `API` struct and `Server` struct with `Controller` and `APIServer`: - -```go -// Controller holds the control plane configuration. -type Controller struct { - Client Client `mapstructure:"client"` - API APIServer `mapstructure:"api" mask:"struct"` - NATS NATSConnection `mapstructure:"nats"` -} - -// APIServer holds the HTTP server config (port + security). -type APIServer struct { - Port int `mapstructure:"port"` - Security ServerSecurity `mapstructure:"security" mask:"struct"` -} -``` - -Update the `Config` struct field: - -```go -type Config struct { - Controller Controller `mapstructure:"controller" mask:"struct"` - Agent AgentConfig `mapstructure:"agent,omitempty"` - // ... rest unchanged -} -``` - -Remove the old `API` and `Server` structs. Keep `Client`, `ClientSecurity`, -`ServerSecurity`, `CORS`, `CustomRole`, `NATSConnection` unchanged. - -- [ ] **Step 2: Verify it compiles (it won't — many references to fix)** - -Run: `go build ./internal/config/...` Expected: PASS (the config package itself -should compile) - -- [ ] **Step 3: Commit** - -``` -refactor(config): rename API struct to Controller -``` - ---- - -### Task 2: Update config YAML files - -**Files:** - -- Modify: `osapi.yaml` -- Modify: `test/integration/osapi.yaml` - -- [ ] **Step 1: Update osapi.yaml** - -Replace the `api:` section: - -```yaml -controller: - client: - url: 'http://0.0.0.0:8080' - security: - bearer_token: '' - api: - port: 8080 - security: - signing_key: '' - cors: - allow_origins: - - 'http://localhost:3001' - - 'https://osapi-io.github.io' - nats: - host: 'localhost' - port: 4222 - client_name: 'osapi-api' - namespace: 'osapi' - auth: - type: 'none' -``` - -- [ ] **Step 2: Update test/integration/osapi.yaml** - -Same structure change: - -```yaml -controller: - client: - url: http://127.0.0.1:8080 - security: - bearer_token: placeholder - api: - port: 8080 - security: - signing_key: 111fdb0cfd9788fa6af8815f856a0374bf7a0174ad62fa8b98ec07a55f68d8d8 - cors: - allow_origins: [] - nats: - host: localhost - port: 4222 - client_name: osapi-api-integration - namespace: '' - auth: - type: none -``` - -- [ ] **Step 3: Commit** - -``` -refactor(config): rename api to controller in YAML files -``` - ---- - -## Chunk 2: Directory moves - -### Task 3: Move internal/api/ to internal/controller/api/ - -**Files:** - -- Move: `internal/api/` → `internal/controller/api/` -- Move: `internal/api/heartbeat.go` → `internal/controller/heartbeat.go` -- Move: `internal/api/heartbeat_test.go` → - `internal/controller/heartbeat_test.go` - -- [ ] **Step 1: Create directory and move files** - -```bash -mkdir -p internal/controller -git mv internal/api internal/controller/api -git mv internal/controller/api/heartbeat.go internal/controller/heartbeat.go -git mv internal/controller/api/heartbeat_test.go internal/controller/heartbeat_test.go -``` - -- [ ] **Step 2: Update package declaration in heartbeat files** - -Change `package api` to `package controller` in: - -- `internal/controller/heartbeat.go` -- `internal/controller/heartbeat_test.go` - -Update imports in heartbeat.go to reference `internal/controller/api` where -needed. - -- [ ] **Step 3: Update all import paths project-wide** - -Find and replace all occurrences: - -- `"github.com/osapi-io/osapi/internal/api"` → - `"github.com/osapi-io/osapi/internal/controller/api"` -- `"github.com/osapi-io/osapi/internal/api/` → - `"github.com/osapi-io/osapi/internal/controller/api/` - -Files that import `internal/api`: - -- `cmd/api_server_setup.go` (will become `cmd/controller_setup.go` in Task 5) -- `cmd/nats_heartbeat.go` -- `cmd/start.go` -- All `internal/api/handler_*.go` files (now under `internal/controller/api/`) -- All domain packages (`internal/controller/api/health/`, etc.) — these use - relative imports within the api package so they may not need changes - -- [ ] **Step 4: Verify compilation** - -Run: `go build ./...` - -- [ ] **Step 5: Commit** - -``` -refactor: move internal/api to internal/controller/api -``` - ---- - -### Task 4: Move internal/notify/ to internal/controller/notify/ - -**Files:** - -- Move: `internal/notify/` → `internal/controller/notify/` - -- [ ] **Step 1: Move directory** - -```bash -git mv internal/notify internal/controller/notify -``` - -- [ ] **Step 2: Update all import paths** - -Find and replace: - -- `"github.com/osapi-io/osapi/internal/notify"` → - `"github.com/osapi-io/osapi/internal/controller/notify"` - -Files that import `internal/notify`: - -- `cmd/api_server_setup.go` (will become `cmd/controller_setup.go`) - -- [ ] **Step 3: Verify compilation** - -Run: `go build ./...` - -- [ ] **Step 4: Commit** - -``` -refactor: move internal/notify to internal/controller/notify -``` - ---- - -## Chunk 3: Controller struct and CMD files - -### Task 5: Create controller.go - -**Files:** - -- Create: `internal/controller/controller.go` - -- [ ] **Step 1: Create the Controller struct** - -```go -package controller - -// Controller is the control plane process. It owns the API server, -// component heartbeat, and condition watcher. -type Controller struct { - apiServer *api.Server - // Additional fields will be added as heartbeat and watcher - // are refactored into this struct in future work. -} -``` - -For now this is a thin wrapper. The existing setup logic in -`cmd/api_server_setup.go` already manages the lifecycle. The controller struct -provides a home for future sub-component ownership. - -- [ ] **Step 2: Commit** - -``` -feat: add internal/controller/controller.go -``` - ---- - -### Task 6: Rename CMD files - -**Files:** - -- Remove: `cmd/api_server.go` -- Remove: `cmd/api_server_start.go` -- Remove: `cmd/api_server_setup.go` -- Create: `cmd/controller.go` -- Create: `cmd/controller_start.go` -- Create: `cmd/controller_setup.go` - -- [ ] **Step 1: Move and rename files** - -```bash -git mv cmd/api_server.go cmd/controller.go -git mv cmd/api_server_start.go cmd/controller_start.go -git mv cmd/api_server_setup.go cmd/controller_setup.go -``` - -- [ ] **Step 2: Update cmd/controller.go** - -Rename `apiServerCmd` to `controllerCmd`. Change: - -- `Use: "server"` → `Use: "start"` -- Parent command: registered under `rootCmd` not `apiCmd` -- Remove the `apiCmd` parent entirely -- Update all `appConfig.API` references to `appConfig.Controller` -- Update log messages from "api server" to "controller" - -The command becomes `osapi controller start` (two levels: controller → start). -Actually per the spec it's just `osapi controller start` where `controller` is -the parent and `start` is the subcommand. Keep the parent for future subcommands -(e.g., `controller status`). - -- [ ] **Step 3: Update cmd/controller_start.go** - -- Rename `apiServerStartCmd` to `controllerStartCmd` -- Update `Use` and `Short` descriptions -- Change `appConfig.API.NATS` to `appConfig.Controller.NATS` -- Update import from `internal/api` to `internal/controller/api` - -- [ ] **Step 4: Update cmd/controller_setup.go** - -- Rename `setupAPIServer` to `setupController` -- Rename `registerAPIHandlers` to `registerControllerHandlers` -- Rename `startAPIHeartbeat` to `startControllerHeartbeat` -- Update all `appConfig.API` to `appConfig.Controller`: - - `appConfig.API.Port` → `appConfig.Controller.API.Port` - - `appConfig.API.NATS` → `appConfig.Controller.NATS` - - `appConfig.API.Server.Security.SigningKey` → - `appConfig.Controller.API.Security.SigningKey` - - `appConfig.API.Server.Security.CORS.AllowOrigins` → - `appConfig.Controller.API.Security.CORS.AllowOrigins` - - `appConfig.API.Server.Security.Roles` → - `appConfig.Controller.API.Security.Roles` -- Update import paths: - - - `internal/api` → `internal/controller/api` - - `internal/notify` → `internal/controller/notify` - -- [ ] **Step 5: Verify compilation** - -Run: `go build ./...` - -- [ ] **Step 6: Commit** - -``` -refactor: rename api server cmd to controller -``` - ---- - -### Task 7: Update cmd/start.go - -**Files:** - -- Modify: `cmd/start.go` - -- [ ] **Step 1: Update references** - -- `setupAPIServer` → `setupController` -- `appConfig.API.NATS` → `appConfig.Controller.NATS` -- `apiBundle` → `controllerBundle` -- Update `Short` and `Long` descriptions: "API server" → "controller" -- Update log component label: `"component", "api"` → `"component", "controller"` - -- [ ] **Step 2: Verify compilation** - -Run: `go build ./...` - -- [ ] **Step 3: Commit** - -``` -refactor: update start.go for controller rename -``` - ---- - -### Task 8: Update client and token commands - -**Files:** - -- Modify: `cmd/client.go` -- Modify: `cmd/token_generate.go` -- Modify: `cmd/token_validate.go` - -- [ ] **Step 1: Update cmd/client.go** - -- `appConfig.API.URL` → `appConfig.Controller.Client.URL` -- `appConfig.API.Client.Security.BearerToken` → - `appConfig.Controller.Client.Security.BearerToken` -- Viper binding: `"api.client.url"` → `"controller.client.url"` -- Log message: `"api.client.url"` → `"controller.client.url"` - -- [ ] **Step 2: Update cmd/token_generate.go** - -- All `appConfig.API` references → `appConfig.Controller` - -- [ ] **Step 3: Update cmd/token_validate.go** - -- All `appConfig.API` references → `appConfig.Controller` - -- [ ] **Step 4: Update cmd/nats_heartbeat.go** - -- Update import from `internal/api` to `internal/controller/api` - -- [ ] **Step 5: Verify compilation and run tests** - -Run: `go build ./... && go test ./cmd/... -count=1` - -- [ ] **Step 6: Commit** - -``` -refactor: update client and token commands for controller config -``` - ---- - -## Chunk 4: Integration tests and verification - -### Task 9: Update integration tests - -**Files:** - -- Modify: `test/integration/integration_test.go` -- Modify: `test/integration/osapi.yaml` (already done in Task 2) - -- [ ] **Step 1: Update serverEnv()** - -```go -func serverEnv() []string { - return append(os.Environ(), - fmt.Sprintf("OSAPI_NATS_SERVER_PORT=%d", natsPort), - fmt.Sprintf("OSAPI_NATS_SERVER_STORE_DIR=%s", storeDir), - fmt.Sprintf("OSAPI_CONTROLLER_API_PORT=%d", apiPort), - fmt.Sprintf("OSAPI_CONTROLLER_NATS_PORT=%d", natsPort), - fmt.Sprintf("OSAPI_AGENT_NATS_PORT=%d", natsPort), - fmt.Sprintf("OSAPI_CONTROLLER_CLIENT_SECURITY_BEARER_TOKEN=%s", token), - ) -} -``` - -- [ ] **Step 2: Update clientEnv()** - -```go -func clientEnv() []string { - return append(os.Environ(), - fmt.Sprintf("OSAPI_CONTROLLER_CLIENT_URL=http://127.0.0.1:%d", apiPort), - fmt.Sprintf("OSAPI_CONTROLLER_CLIENT_SECURITY_BEARER_TOKEN=%s", token), - ) -} -``` - -- [ ] **Step 3: Verify full build and unit tests** - -```bash -go build ./... -go test ./... -count=1 -``` - -- [ ] **Step 4: Run integration tests** - -```bash -just go::unit-int -``` - -- [ ] **Step 5: Commit** - -``` -refactor: update integration tests for controller config -``` - ---- - -## Chunk 5: Documentation - -### Task 10: Update CLAUDE.md - -**Files:** - -- Modify: `CLAUDE.md` - -- [ ] **Step 1: Update architecture section** - -- `cmd/` description: replace "api server" with "controller" -- `internal/api/` → `internal/controller/api/` -- Add `internal/controller/` description -- Add `internal/controller/notify/` description -- Update "Adding a New API Domain" section paths -- Update config references throughout - -- [ ] **Step 2: Commit** - -``` -docs: update CLAUDE.md for controller rename -``` - ---- - -### Task 11: Update Docusaurus docs - -**Files:** - -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `docs/docs/sidebar/architecture/system-architecture.md` -- Modify: `docs/docs/sidebar/development/development.md` -- Modify: `docs/docs/sidebar/features/health-checks.md` -- Modify: `docs/docs/sidebar/features/notifications.md` -- Modify: `docs/docs/sidebar/intro.md` - -- [ ] **Step 1: Update configuration.md** - -Replace all `api.*` config keys with `controller.*` in: - -- YAML examples -- Environment variable table -- Section reference tables - -- [ ] **Step 2: Update architecture.md** - -- "API Server" → "Controller" in process descriptions -- `osapi api server start` → `osapi controller start` - -- [ ] **Step 3: Update system-architecture.md** - -- Package layout: `internal/api/` → `internal/controller/api/` -- Handler structure references - -- [ ] **Step 4: Update development.md** - -- Quick reference: `osapi api server start` → `osapi controller start` - -- [ ] **Step 5: Update feature docs** - -- health-checks.md: update any "API server" references -- notifications.md: update any "API server" references - -- [ ] **Step 6: Update intro.md** - -- Quickstart section: update startup commands if they reference - `api server start` - -- [ ] **Step 7: Verify docs build** - -```bash -just docs::build -``` - -- [ ] **Step 8: Commit** - -``` -docs: update all docs for controller rename -``` - ---- - -## Chunk 6: Final verification - -### Task 12: Full verification - -- [ ] **Step 1: Build** - -```bash -go build ./... -``` - -- [ ] **Step 2: Unit tests** - -```bash -go test ./... -count=1 -``` - -- [ ] **Step 3: Lint** - -```bash -just go::vet -``` - -- [ ] **Step 4: Integration tests** - -```bash -just go::unit-int -``` - -- [ ] **Step 5: Verify CLI** - -```bash -go run main.go controller start --help -go run main.go start --help -go run main.go client --help -``` - -- [ ] **Step 6: Docs build** - -```bash -just docs::build -``` - -- [ ] **Step 7: Final commit if any fixups needed** - ---- - -## Files Modified Summary - -| File | Change | -| ------------------------------------------------------- | -------------------------------------------- | -| `internal/config/types.go` | `API` → `Controller`, `Server` → `APIServer` | -| `osapi.yaml` | `api:` → `controller:` | -| `test/integration/osapi.yaml` | `api:` → `controller:` | -| `internal/api/` → `internal/controller/api/` | Directory move | -| `internal/notify/` → `internal/controller/notify/` | Directory move | -| `internal/controller/heartbeat.go` | Moved from `internal/api/`, package rename | -| `internal/controller/heartbeat_test.go` | Moved from `internal/api/`, package rename | -| `internal/controller/controller.go` | New file | -| `cmd/api_server.go` → `cmd/controller.go` | Rename + update | -| `cmd/api_server_start.go` → `cmd/controller_start.go` | Rename + update | -| `cmd/api_server_setup.go` → `cmd/controller_setup.go` | Rename + update | -| `cmd/start.go` | Config path updates | -| `cmd/client.go` | Config path + viper binding updates | -| `cmd/token_generate.go` | Config path updates | -| `cmd/token_validate.go` | Config path updates | -| `cmd/nats_heartbeat.go` | Import path update | -| `test/integration/integration_test.go` | Env var updates | -| `CLAUDE.md` | Architecture references | -| `docs/docs/sidebar/usage/configuration.md` | Full config reference | -| `docs/docs/sidebar/architecture/architecture.md` | Process descriptions | -| `docs/docs/sidebar/architecture/system-architecture.md` | Package layout | -| `docs/docs/sidebar/development/development.md` | Quick reference | -| `docs/docs/sidebar/features/health-checks.md` | References | -| `docs/docs/sidebar/features/notifications.md` | References | -| `docs/docs/sidebar/intro.md` | Startup commands | diff --git a/docs/plans/2026-03-21-per-component-metrics-design.md b/docs/plans/2026-03-21-per-component-metrics-design.md deleted file mode 100644 index 139043939..000000000 --- a/docs/plans/2026-03-21-per-component-metrics-design.md +++ /dev/null @@ -1,197 +0,0 @@ -# Per-Component Metrics and Sub-Component Health — Design Spec - -## Goal - -Add per-component `/metrics` endpoints on dedicated ports for the controller, -agent, and NATS server. Each component gets its own Prometheus registry and OTEL -MeterProvider. Add sub-component health reporting to the controller's -`/health/status` endpoint. - -## Motivation - -Today `/metrics` is served on the controller's API port (8080) using a global -OTEL meter provider. This mixes API traffic with metrics scraping, doesn't -support per-component isolation, and provides no metrics for the agent or NATS -server. Operators need independent metrics endpoints for each component, and -visibility into whether internal services (notifier, heartbeat, consumers) are -running. - -## Config - -### Shared type - -```go -// OpsServer configures the per-component metrics HTTP server. -type OpsServer struct { - Enabled bool `mapstructure:"enabled"` - Port int `mapstructure:"port"` -} -``` - -### YAML - -```yaml -controller: - metrics: - enabled: true # default: true - port: 9090 # default: 9090 - -agent: - metrics: - enabled: true # default: true - port: 9091 # default: 9091 - -nats: - server: - metrics: - enabled: true # default: true - port: 9092 # default: 9092 -``` - -### Environment variables - -| Config Key | Environment Variable | -| ----------------------------- | ----------------------------------- | -| `controller.metrics.enabled` | `OSAPI_CONTROLLER_METRICS_ENABLED` | -| `controller.metrics.port` | `OSAPI_CONTROLLER_METRICS_PORT` | -| `agent.metrics.enabled` | `OSAPI_AGENT_METRICS_ENABLED` | -| `agent.metrics.port` | `OSAPI_AGENT_METRICS_PORT` | -| `nats.server.metrics.enabled` | `OSAPI_NATS_SERVER_METRICS_ENABLED` | -| `nats.server.metrics.port` | `OSAPI_NATS_SERVER_METRICS_PORT` | - -## Architecture - -### New package: `internal/ops/` - -A lightweight HTTP server that serves `/metrics` on a dedicated port. Each -component creates its own instance with isolated Prometheus registry and OTEL -MeterProvider. - -```go -package ops - -type Server struct { ... } - -func New(port int, logger *slog.Logger) *Server -func (s *Server) MeterProvider() *sdkmetric.MeterProvider -func (s *Server) Start() -func (s *Server) Stop(ctx context.Context) -``` - -- Implements `cli.Lifecycle` -- Creates its own `prometheus.Registry` (not the global default) -- Registers Go runtime and process collectors on that registry -- Creates an OTEL `MeterProvider` backed by a `prometheus.Exporter` tied to that - registry -- Does NOT call `otel.SetMeterProvider()` — no global state -- Serves `promhttp.HandlerFor(registry)` on `/metrics` - -### Per-component wiring - -Each component: - -1. Checks `metrics.enabled` in config -2. If enabled, creates `ops.New(port, logger)` -3. Starts/stops alongside the main process -4. Uses `server.MeterProvider()` to create component-specific OTEL instruments - -In single-process mode (`osapi start`), three ops servers run on three ports, -each with isolated metrics. - -### Removal from API port - -The controller's `/metrics` endpoint is removed from port 8080. The -`handler_metrics.go` file and metrics domain package -(`internal/controller/api/metrics/`) are deleted. Metrics are served exclusively -on the ops port. - -`/health`, `/health/ready`, and `/health/status` remain on port 8080. - -## Sub-component health - -The controller's `/health/status` endpoint adds internal service status to the -existing `components` map: - -```json -{ - "status": "ok", - "components": { - "nats": "ok", - "kv": "ok", - "notifier": "ok", - "heartbeat": "ok" - } -} -``` - -### Component status values - -| Component | When `ok` | When `disabled` | When `error` | -| ----------- | -------------------------------- | ------------------------------ | ----------------- | -| `nats` | Connected | — | Connection failed | -| `kv` | Accessible | — | Access failed | -| `notifier` | Watcher running | `notifications.enabled: false` | — | -| `heartbeat` | Always (started unconditionally) | — | — | - -The `disabled` status is a new value. Today only `ok` and error strings exist. A -disabled component is not unhealthy — it was intentionally turned off. - -### Agent and NATS sub-components - -No `/health` endpoint on agent or NATS metrics ports. Their status is visible -through the controller's `/health/status` via the registry (heartbeat data). - -Future work could add `/health` to the ops server if needed for k8s probes. - -## Code changes - -### New files - -| File | Purpose | -| ----------------------------- | ----------------------------------------------- | -| `internal/ops/server.go` | Ops server: Start/Stop, registry, MeterProvider | -| `internal/ops/server_test.go` | Unit tests | -| `internal/ops/types.go` | Interface definitions | - -### Modified files - -| File | Change | -| ----------------------------- | --------------------------------------------------------------------- | -| `internal/config/types.go` | Add `OpsServer` to `Controller`, `AgentConfig`, `NATSServer` | -| `cmd/controller_start.go` | Create and start ops server | -| `cmd/controller_setup.go` | Remove metrics handler from API, add notifier/heartbeat to components | -| `cmd/agent_start.go` | Create and start ops server | -| `cmd/nats_server_start.go` | Create and start ops server | -| `cmd/start.go` | Wire all three ops servers, stop them on shutdown | -| `configs/osapi.yaml` | Add metrics sections | -| `test/integration/osapi.yaml` | Add metrics sections (disabled or test ports) | - -### Removed files - -| File | Reason | -| -------------------------------------------- | ----------------------------- | -| `internal/controller/api/handler_metrics.go` | Metrics moved to ops server | -| `internal/controller/api/metrics/` | Entire metrics domain package | - -### Docs - -| File | Change | -| ------------------------------------------------ | ------------------------------------- | -| `docs/docs/sidebar/usage/configuration.md` | Add metrics config for all components | -| `docs/docs/sidebar/features/metrics.md` | Update for per-component metrics | -| `docs/docs/sidebar/architecture/architecture.md` | Mention ops servers | -| `CLAUDE.md` | Add `internal/ops/` to architecture | - -## What doesn't change - -- `/health`, `/health/ready`, `/health/status` stay on port 8080 -- Agent and NATS heartbeat mechanism unchanged -- SDK client unchanged -- All REST API endpoints unchanged -- Existing OTEL tracing unchanged - -## Breaking changes - -- `/metrics` removed from port 8080 — scrapers must update to port 9090 -- `telemetry.metrics.path` config key becomes unused (path is always `/metrics` - on the ops port) diff --git a/docs/plans/2026-03-21-per-component-metrics.md b/docs/plans/2026-03-21-per-component-metrics.md deleted file mode 100644 index b30493529..000000000 --- a/docs/plans/2026-03-21-per-component-metrics.md +++ /dev/null @@ -1,879 +0,0 @@ -# Per-Component Metrics and Sub-Component Health Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> (if subagents available) or superpowers:executing-plans to implement this -> plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add per-component `/metrics` endpoints on dedicated ports for the -controller, agent, and NATS server with isolated Prometheus registries and OTEL -MeterProviders. Add sub-component health to the controller's `/health/status`. - -**Architecture:** New `internal/ops/` package provides a lightweight HTTP server -with its own Prometheus registry. Each component creates one if -`metrics.enabled` is true. Remove `/metrics` from the controller's API port. Add -`notifier` and `heartbeat` to the `/health/status` components map. - -**Tech Stack:** Go 1.25, OTEL SDK, Prometheus client, Echo HTTP - ---- - -## Chunk 1: Config and ops server - -### Task 1: Add OpsServer config type - -**Files:** - -- Modify: `internal/config/types.go` - -- [ ] **Step 1: Add OpsServer struct** - -Add after the existing `MetricsConfig` struct: - -```go -// OpsServer configures the per-component metrics HTTP server. -type OpsServer struct { - // Enabled activates the metrics server (default: true). - Enabled *bool `mapstructure:"enabled"` - // Port the metrics server listens on. - Port int `mapstructure:"port"` -} - -// IsEnabled returns true if the ops server is enabled. -// Defaults to true when Enabled is nil. -func (o OpsServer) IsEnabled() bool { - if o.Enabled == nil { - return true - } - return *o.Enabled -} -``` - -Use `*bool` so we can distinguish "not set" (default true) from "explicitly set -to false". - -- [ ] **Step 2: Add to Controller, AgentConfig, NATSServer** - -Add `Metrics OpsServer` field to each: - -```go -type Controller struct { - Client Client `mapstructure:"client"` - API APIServer `mapstructure:"api" mask:"struct"` - NATS NATSConnection `mapstructure:"nats"` - Metrics OpsServer `mapstructure:"metrics"` -} -``` - -```go -type AgentConfig struct { - // ... existing fields ... - Metrics OpsServer `mapstructure:"metrics"` -} -``` - -```go -type NATSServer struct { - // ... existing fields ... - Metrics OpsServer `mapstructure:"metrics"` -} -``` - -- [ ] **Step 3: Verify config package compiles** - -Run: `go build ./internal/config/...` - -- [ ] **Step 4: Commit** - -``` -feat(config): add OpsServer metrics config to all components -``` - ---- - -### Task 2: Update YAML config files - -**Files:** - -- Modify: `configs/osapi.yaml` -- Modify: `test/integration/osapi.yaml` - -- [ ] **Step 1: Add metrics sections to configs/osapi.yaml** - -Under `controller:`: - -```yaml -controller: - metrics: - enabled: true - port: 9090 -``` - -Under `agent:`: - -```yaml -agent: - metrics: - enabled: true - port: 9091 -``` - -Under `nats.server:`: - -```yaml -nats: - server: - metrics: - enabled: true - port: 9092 -``` - -- [ ] **Step 2: Add metrics to test/integration/osapi.yaml** - -Use `enabled: false` for integration tests (avoid port conflicts): - -```yaml -controller: - metrics: - enabled: false - -agent: - metrics: - enabled: false -``` - -No NATS metrics in integration config (it's already minimal). - -- [ ] **Step 3: Commit** - -``` -feat(config): add metrics sections to YAML configs -``` - ---- - -### Task 3: Create internal/ops package - -**Files:** - -- Create: `internal/ops/server.go` -- Create: `internal/ops/server_test.go` - -- [ ] **Step 1: Write the test** - -```go -package ops_test - -import ( - "fmt" - "io" - "log/slog" - "net/http" - "testing" - "time" - - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/ops" -) - -type ServerPublicTestSuite struct { - suite.Suite -} - -func (s *ServerPublicTestSuite) TestStartAndStop() { - tests := []struct { - name string - port int - validateFunc func() - }{ - { - name: "serves metrics endpoint", - port: 19090, - validateFunc: func() { - resp, err := http.Get("http://127.0.0.1:19090/metrics") - s.Require().NoError(err) - defer resp.Body.Close() - s.Equal(200, resp.StatusCode) - - body, err := io.ReadAll(resp.Body) - s.Require().NoError(err) - s.Contains(string(body), "go_goroutines") - }, - }, - } - - for _, tc := range tests { - s.Run(tc.name, func() { - logger := slog.Default() - srv := ops.New(tc.port, logger) - srv.Start() - - // Give server time to bind. - time.Sleep(100 * time.Millisecond) - - tc.validateFunc() - - ctx, cancel := context.WithTimeout( - context.Background(), - 5*time.Second, - ) - defer cancel() - srv.Stop(ctx) - }) - } -} - -func (s *ServerPublicTestSuite) TestMeterProvider() { - logger := slog.Default() - srv := ops.New(19091, logger) - s.NotNil(srv.MeterProvider()) -} - -func TestServerPublicTestSuite(t *testing.T) { - suite.Run(t, new(ServerPublicTestSuite)) -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./internal/ops/... -count=1 -v` Expected: FAIL (package doesn't -exist) - -- [ ] **Step 3: Write the implementation** - -```go -// Package ops provides a lightweight HTTP server for per-component -// Prometheus metrics. -package ops - -import ( - "context" - "fmt" - "log/slog" - "net/http" - "time" - - "github.com/prometheus/client_golang/prometheus" - "github.com/prometheus/client_golang/prometheus/collectors" - "github.com/prometheus/client_golang/prometheus/promhttp" - prometheusExporter "go.opentelemetry.io/otel/exporters/prometheus" - sdkmetric "go.opentelemetry.io/otel/sdk/metric" -) - -// Server is a lightweight HTTP server that serves /metrics with an -// isolated Prometheus registry and OTEL MeterProvider. -type Server struct { - httpServer *http.Server - logger *slog.Logger - registry *prometheus.Registry - meterProvider *sdkmetric.MeterProvider -} - -// New creates a new ops server on the given port. -func New( - port int, - logger *slog.Logger, -) *Server { - reg := prometheus.NewRegistry() - reg.MustRegister(collectors.NewGoCollector()) - reg.MustRegister(collectors.NewProcessCollector( - collectors.ProcessCollectorOpts{}, - )) - - exporter, err := prometheusExporter.New( - prometheusExporter.WithRegisterer(reg), - ) - if err != nil { - logger.Error("failed to create prometheus exporter", "error", err) - return nil - } - - mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(exporter)) - - mux := http.NewServeMux() - mux.Handle("/metrics", promhttp.HandlerFor( - reg, - promhttp.HandlerOpts{Registry: reg}, - )) - - return &Server{ - httpServer: &http.Server{ - Addr: fmt.Sprintf(":%d", port), - Handler: mux, - ReadHeaderTimeout: 10 * time.Second, - }, - logger: logger, - registry: reg, - meterProvider: mp, - } -} - -// MeterProvider returns the isolated OTEL MeterProvider for this server. -// Components use this to create instruments that appear on this server's -// /metrics endpoint. -func (s *Server) MeterProvider() *sdkmetric.MeterProvider { - return s.meterProvider -} - -// Registry returns the isolated Prometheus registry for this server. -func (s *Server) Registry() *prometheus.Registry { - return s.registry -} - -// Start starts the HTTP server in a background goroutine. -func (s *Server) Start() { - go func() { - s.logger.Info("ops server started", "addr", s.httpServer.Addr) - if err := s.httpServer.ListenAndServe(); err != nil && - err != http.ErrServerClosed { - s.logger.Error("ops server error", "error", err) - } - }() -} - -// Stop gracefully shuts down the HTTP server and meter provider. -func (s *Server) Stop(ctx context.Context) { - if err := s.meterProvider.Shutdown(ctx); err != nil { - s.logger.Error("meter provider shutdown error", "error", err) - } - - if err := s.httpServer.Shutdown(ctx); err != nil { - s.logger.Error("ops server shutdown error", "error", err) - } - - s.logger.Info("ops server stopped") -} -``` - -- [ ] **Step 4: Run tests** - -Run: `go test ./internal/ops/... -count=1 -v` Expected: PASS - -- [ ] **Step 5: Commit** - -``` -feat: add internal/ops package for per-component metrics -``` - ---- - -## Chunk 2: Wire ops server into components - -### Task 4: Wire into controller - -**Files:** - -- Modify: `cmd/controller_start.go` -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Update controller_start.go** - -After `telemetry.InitMeter()`, add ops server creation: - -```go -var opsServer *ops.Server -if appConfig.Controller.Metrics.IsEnabled() { - opsServer = ops.New( - appConfig.Controller.Metrics.Port, - log.With("component", "controller-ops"), - ) -} -``` - -Start it before `sm.Start()`: - -```go -if opsServer != nil { - opsServer.Start() -} -``` - -Add to shutdown: - -```go -cli.RunServer(ctx, sm, func() { - if opsServer != nil { - opsServer.Stop(context.Background()) - } - _ = shutdownMeter(context.Background()) - _ = shutdownTracer(context.Background()) - cli.CloseNATSClient(b.nc) -}) -``` - -Add import: `"github.com/osapi-io/osapi/internal/ops"` - -- [ ] **Step 2: Remove metrics from API port** - -In `cmd/controller_setup.go`, remove the `metricsHandler` and `metricsPath` -parameters from `setupController()`. Remove `sm.GetMetricsHandler()` call from -`registerControllerHandlers()`. - -Update `setupController` signature: - -```go -func setupController( - ctx context.Context, - log *slog.Logger, - natsConfig config.NATSConnection, -) (*api.Server, *natsBundle) { -``` - -Remove `metricsHandler` and `metricsPath` from `controller_start.go` call. - -- [ ] **Step 3: Delete metrics handler and domain package** - -```bash -rm internal/controller/api/handler_metrics.go -rm -rf internal/controller/api/metrics/ -``` - -Remove `GetMetricsHandler` method from `internal/controller/api/types.go` if it -exists, and remove it from `registerControllerHandlers()`. - -- [ ] **Step 4: Verify build and tests** - -```bash -go build ./... -go test ./... -count=1 -``` - -- [ ] **Step 5: Commit** - -``` -feat: wire ops server into controller, remove /metrics from API port -``` - ---- - -### Task 5: Wire into agent - -**Files:** - -- Modify: `cmd/agent_start.go` - -- [ ] **Step 1: Add ops server to agent startup** - -After agent setup, create and start the ops server: - -```go -var opsServer *ops.Server -if appConfig.Agent.Metrics.IsEnabled() { - opsServer = ops.New( - appConfig.Agent.Metrics.Port, - logger.With("component", "agent-ops"), - ) - opsServer.Start() -} -``` - -Add to shutdown: - -```go -cli.RunServer(ctx, agentServer, func() { - if opsServer != nil { - opsServer.Stop(context.Background()) - } - _ = shutdownTracer(context.Background()) - cli.CloseNATSClient(b.nc) -}) -``` - -Add import: `"github.com/osapi-io/osapi/internal/ops"` - -- [ ] **Step 2: Verify build** - -```bash -go build ./... -``` - -- [ ] **Step 3: Commit** - -``` -feat: wire ops server into agent -``` - ---- - -### Task 6: Wire into NATS server - -**Files:** - -- Modify: `cmd/nats_server_start.go` - -- [ ] **Step 1: Add ops server to NATS server startup** - -After NATS server setup: - -```go -var opsServer *ops.Server -if appConfig.NATS.Server.Metrics.IsEnabled() { - opsServer = ops.New( - appConfig.NATS.Server.Metrics.Port, - logger.With("component", "nats-ops"), - ) - opsServer.Start() -} -``` - -Add to shutdown/cleanup. - -- [ ] **Step 2: Verify build** - -```bash -go build ./... -``` - -- [ ] **Step 3: Commit** - -``` -feat: wire ops server into NATS server -``` - ---- - -### Task 7: Wire into combined start - -**Files:** - -- Modify: `cmd/start.go` - -- [ ] **Step 1: Create ops servers for all three components** - -After component setup, before `composite.Start()`: - -```go -var controllerOps, agentOps, natsOps *ops.Server - -if appConfig.Controller.Metrics.IsEnabled() { - controllerOps = ops.New( - appConfig.Controller.Metrics.Port, - logger.With("component", "controller-ops"), - ) -} -if appConfig.Agent.Metrics.IsEnabled() { - agentOps = ops.New( - appConfig.Agent.Metrics.Port, - logger.With("component", "agent-ops"), - ) -} -if appConfig.NATS.Server.Metrics.IsEnabled() { - natsOps = ops.New( - appConfig.NATS.Server.Metrics.Port, - logger.With("component", "nats-ops"), - ) -} -``` - -Start them before `composite.Start()`: - -```go -for _, o := range []*ops.Server{controllerOps, agentOps, natsOps} { - if o != nil { - o.Start() - } -} -``` - -Stop them in the shutdown closure: - -```go -cli.RunServer(ctx, composite, func() { - for _, o := range []*ops.Server{controllerOps, agentOps, natsOps} { - if o != nil { - o.Stop(context.Background()) - } - } - _ = shutdownMeter(context.Background()) - _ = shutdownTracer(context.Background()) - cli.CloseNATSClient(agentBundle.nc) - cli.CloseNATSClient(controllerBundle.nc) -}) -``` - -Also remove `metricsHandler`/`metricsPath` from `setupController()` call. - -- [ ] **Step 2: Verify build and tests** - -```bash -go build ./... -go test ./... -count=1 -``` - -- [ ] **Step 3: Commit** - -``` -feat: wire ops servers into combined start -``` - ---- - -## Chunk 3: Sub-component health - -### Task 8: Add notifier and heartbeat to /health/status components - -**Files:** - -- Modify: `cmd/controller_setup.go` -- Modify: `internal/controller/api/health/health_status_get.go` -- Modify: `internal/controller/api/health/types.go` - -- [ ] **Step 1: Add component status fields to health handler** - -In `internal/controller/api/health/types.go`, add to the `Health` struct or -`MetricsProvider` a way to report sub-component status. The simplest approach is -to pass them as static values when creating the health handler. - -Add to the `Health` struct in `internal/controller/api/health/health.go`: - -```go -type Health struct { - // ... existing fields ... - SubComponents map[string]string -} -``` - -- [ ] **Step 2: Populate in health_status_get.go** - -In `GetHealthStatus`, add sub-components to the response `Components` map: - -```go -for k, v := range h.SubComponents { - components[k] = gen.ComponentHealth{ - Status: v, - } -} -``` - -- [ ] **Step 3: Wire in controller_setup.go** - -When creating the health handler, pass sub-component status: - -```go -subComponents := map[string]string{ - "heartbeat": "ok", -} -if appConfig.Notifications.Enabled { - subComponents["notifier"] = "ok" -} else { - subComponents["notifier"] = "disabled" -} -``` - -Pass to health handler constructor. - -- [ ] **Step 4: Update tests** - -Add test case for sub-components in -`internal/controller/api/health/health_status_get_public_test.go`. - -- [ ] **Step 5: Verify tests pass** - -```bash -go test ./internal/controller/api/health/... -count=1 -v -``` - -- [ ] **Step 6: Commit** - -``` -feat: add notifier and heartbeat to /health/status components -``` - ---- - -## Chunk 4: Cleanup and docs - -### Task 9: Remove old telemetry.metrics.path config - -**Files:** - -- Modify: `internal/config/types.go` - -- [ ] **Step 1: Remove Path from MetricsConfig** - -The `MetricsConfig.Path` field is no longer used — the ops server always serves -on `/metrics`. Remove the field or leave it for backwards compat. - -Since nothing reads it anymore, remove it: - -```go -// MetricsConfig is retained for future telemetry configuration. -type MetricsConfig struct{} -``` - -Or remove `MetricsConfig` entirely and simplify `Telemetry`: - -```go -type Telemetry struct { - Tracing TracingConfig `mapstructure:"tracing,omitempty"` -} -``` - -- [ ] **Step 2: Remove metricsPath references from controller_start.go** - -Remove the `metricsHandler, metricsPath, shutdownMeter` variables if `InitMeter` -is no longer called (since ops server handles it). - -Check if `InitMeter` is still needed for OTEL initialization. If not, remove the -call. - -- [ ] **Step 3: Verify build and tests** - -```bash -go build ./... -go test ./... -count=1 -``` - -- [ ] **Step 4: Commit** - -``` -refactor: remove unused telemetry.metrics.path config -``` - ---- - -### Task 10: Update handler_public_test.go - -**Files:** - -- Modify: `internal/controller/api/handler_public_test.go` - -- [ ] **Step 1: Remove GetMetricsHandler test** - -The `TestGetMetricsHandler` test case tests the removed handler. Delete it. - -- [ ] **Step 2: Verify tests pass** - -```bash -go test ./internal/controller/api/... -count=1 -``` - -- [ ] **Step 3: Commit** - -``` -test: remove GetMetricsHandler test -``` - ---- - -### Task 11: Update docs - -**Files:** - -- Modify: `CLAUDE.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/features/metrics.md` - -- [ ] **Step 1: Update CLAUDE.md** - -Add `internal/ops/` to architecture section. - -- [ ] **Step 2: Update configuration.md** - -Add `controller.metrics`, `agent.metrics`, `nats.server.metrics` sections with -the `enabled` and `port` fields. Add env var mappings. Remove -`telemetry.metrics.path` if removed. - -- [ ] **Step 3: Update metrics.md** - -Update to describe per-component metrics: - -- Controller on port 9090 -- Agent on port 9091 -- NATS on port 9092 -- Each has isolated registry -- Configurable via `metrics.enabled` and `metrics.port` - -- [ ] **Step 4: Commit** - -``` -docs: update docs for per-component metrics -``` - ---- - -## Chunk 5: Verification - -### Task 12: Full verification - -- [ ] **Step 1: Build** - -```bash -go build ./... -``` - -- [ ] **Step 2: Unit tests** - -```bash -go test ./... -count=1 -``` - -- [ ] **Step 3: Lint** - -```bash -just go::vet -``` - -- [ ] **Step 4: Manual verification** - -Start osapi and verify: - -```bash -go run main.go start -f configs/osapi.yaml -``` - -In another terminal: - -```bash -# Controller metrics -curl http://localhost:9090/metrics | head -5 - -# Agent metrics -curl http://localhost:9091/metrics | head -5 - -# NATS metrics -curl http://localhost:9092/metrics | head -5 - -# Health status shows sub-components -go run main.go client health status --json | jq .components -``` - -Verify `/metrics` is NOT served on port 8080: - -```bash -curl http://localhost:8080/metrics # should 404 -``` - -- [ ] **Step 5: Integration tests** - -```bash -just go::unit-int -``` - -- [ ] **Step 6: Final commit if fixups needed** - ---- - -## Files Summary - -| File | Change | -| ----------------------------------------------------------------- | ------------------------------------------------------------- | -| `internal/config/types.go` | Add `OpsServer`, add `Metrics` to Controller/Agent/NATSServer | -| `internal/ops/server.go` | New: ops server with isolated registry | -| `internal/ops/server_test.go` | New: tests | -| `cmd/controller_start.go` | Create/start/stop ops server | -| `cmd/controller_setup.go` | Remove metricsHandler params, add sub-components | -| `cmd/agent_start.go` | Create/start/stop ops server | -| `cmd/nats_server_start.go` | Create/start/stop ops server | -| `cmd/start.go` | Wire all three ops servers | -| `configs/osapi.yaml` | Add metrics sections | -| `test/integration/osapi.yaml` | Add metrics (disabled) | -| `internal/controller/api/handler_metrics.go` | Delete | -| `internal/controller/api/metrics/` | Delete entire package | -| `internal/controller/api/handler_public_test.go` | Remove metrics test | -| `internal/controller/api/health/health.go` | Add SubComponents field | -| `internal/controller/api/health/health_status_get.go` | Emit sub-components | -| `internal/controller/api/health/health_status_get_public_test.go` | Add test | -| `CLAUDE.md` | Add `internal/ops/` | -| `docs/docs/sidebar/usage/configuration.md` | Add metrics config | -| `docs/docs/sidebar/features/metrics.md` | Rewrite for per-component | diff --git a/docs/plans/2026-03-22-cron-management-design.md b/docs/plans/2026-03-22-cron-management-design.md deleted file mode 100644 index a6625f4aa..000000000 --- a/docs/plans/2026-03-22-cron-management-design.md +++ /dev/null @@ -1,136 +0,0 @@ -# Cron Drop-in Management Design - -## Goal - -Add cron drop-in file management (`/etc/cron.d/`) to OSAPI. This is the first -provider under the `scheduled/` domain. Crontab (user crontabs) and systemd -timer management will follow as separate providers. - -## Provider - -**Location:** `internal/provider/scheduled/cron/` - -**Interface:** - -```go -type Provider interface { - List() ([]CronEntry, error) - Get(name string) (*CronEntry, error) - Create(entry CronEntry) (*CreateResult, error) - Update(entry CronEntry) (*UpdateResult, error) - Delete(name string) (*DeleteResult, error) -} -``` - -**CronEntry:** - -```go -type CronEntry struct { - Name string `json:"name"` - Schedule string `json:"schedule"` - User string `json:"user"` - Command string `json:"command"` -} -``` - -**Result types** include `Changed bool` and `Error string` fields per -convention. - -**Debian provider:** Reads and writes `/etc/cron.d/{name}` files using -`afero.Fs`. Each file contains: - -``` -# Managed by osapi -SCHEDULE USER COMMAND -``` - -File permissions: 0644 (standard for `/etc/cron.d/` files). The name is -sanitized to prevent path traversal. Names must be alphanumeric with hyphens and -underscores only. - -**Darwin provider:** Returns `provider.ErrUnsupported` for all operations. Jobs -targeting macOS agents get `StatusSkipped`. - -**Linux stub:** Returns `provider.ErrUnsupported`. - -## API - -**Path:** `/node/{hostname}/schedule/cron` and -`/node/{hostname}/schedule/cron/{name}` - -**Endpoints:** - -| Method | Path | Operation | Permission | -| ------ | --------------------------------------- | ------------- | ------------ | -| GET | `/node/{hostname}/schedule/cron` | `cron.list` | `cron:read` | -| GET | `/node/{hostname}/schedule/cron/{name}` | `cron.get` | `cron:read` | -| POST | `/node/{hostname}/schedule/cron` | `cron.create` | `cron:write` | -| PUT | `/node/{hostname}/schedule/cron/{name}` | `cron.update` | `cron:write` | -| DELETE | `/node/{hostname}/schedule/cron/{name}` | `cron.delete` | `cron:write` | - -**OpenAPI spec:** `internal/controller/api/schedule/gen/api.yaml` - -**Validation (via `x-oapi-codegen-extra-tags`):** - -- `name` — required, alphanum with hyphens/underscores -- `schedule` — required (cron expression, validated by provider) -- `command` — required -- `user` — optional, defaults to `root` - -**Response format:** Same collection/result pattern as other domains. List -returns `CronCollectionResponse` with `results` array. - -## Job Routing - -- Query: `cron.list`, `cron.get` -- Modify: `cron.create`, `cron.update`, `cron.delete` - -**Job category:** `schedule` **Job operations:** `cron.list`, `cron.get`, -`cron.create`, `cron.update`, `cron.delete` - -## SDK - -**New operations in `pkg/sdk/client/operations.go`:** - -```go -OpCronList JobOperation = "cron.list" -OpCronGet JobOperation = "cron.get" -OpCronCreate JobOperation = "cron.create" -OpCronUpdate JobOperation = "cron.update" -OpCronDelete JobOperation = "cron.delete" -``` - -**New permissions in `pkg/sdk/client/permissions.go`:** - -```go -PermCronRead Permission = "cron:read" -PermCronWrite Permission = "cron:write" -``` - -Add `cron:read` and `cron:write` to the `admin` and `write` default roles. Add -`cron:read` to the `read` role. - -**New `CronService`** on the SDK client with typed result types in -`pkg/sdk/client/schedule.go` and `pkg/sdk/client/schedule_types.go`. - -## CLI - -``` -osapi client node schedule cron list --hostname web-01 -osapi client node schedule cron get --hostname web-01 --name backup -osapi client node schedule cron create --hostname web-01 --name backup \ - --schedule "0 2 * * *" --command "/usr/local/bin/backup.sh" --user root -osapi client node schedule cron update --hostname web-01 --name backup \ - --schedule "0 3 * * *" -osapi client node schedule cron delete --hostname web-01 --name backup -``` - -All commands support `--json` for raw output. - -## Additional Changes - -- Move `internal/provider/process/` to `internal/provider/node/process/` -- Add cron provider to agent factory with platform switch -- Wire schedule handler in controller setup -- Update CLAUDE.md with new domain -- Documentation: feature page, CLI reference, API reference, config reference diff --git a/docs/plans/2026-03-22-file-backed-meta-providers-design.md b/docs/plans/2026-03-22-file-backed-meta-providers-design.md deleted file mode 100644 index 84320de1f..000000000 --- a/docs/plans/2026-03-22-file-backed-meta-providers-design.md +++ /dev/null @@ -1,341 +0,0 @@ -# File-Backed Meta Providers Design - -## Goal - -Refactor the cron provider (and establish the pattern for future providers like -systemd, sysctl, apt sources) to delegate file writes to the file provider -instead of using raw `afero.WriteFile`. This gives all file-writing providers -SHA tracking, idempotency, drift detection, and template rendering for free. - -Additionally, add `Undeploy` to the file provider (remove file from disk while -keeping the object in the store), and add `protected` object support so -system-managed templates cannot be deleted by users. - -## Architecture - -### Meta Provider Pattern - -A meta provider is a domain-specific provider that writes files to well-known -paths. It does not write to the filesystem directly. Instead, it: - -1. Determines the destination path and permissions based on domain rules -2. Delegates to the file provider's `Deploy()` method -3. Gets SHA tracking, idempotency, and template rendering for free - -``` -User Meta Provider File Provider - │ │ │ - ├─ file upload ──────────────────────────────────────►│ (object store) - │ │ │ - ├─ cron create ──────────►│ │ - │ (--object, --schedule)│ │ - │ ├─ Deploy(object, path, ──►│ - │ │ mode, content_type) │ - │ │ ├─ fetch from obj store - │ │ ├─ render template (if applicable) - │ │ ├─ SHA check (idempotent) - │ │ ├─ write to disk - │ │ ├─ update file-state KV - │ │◄─ DeployResult ──────────┤ - │◄─ CronCreateResponse ──┤ │ -``` - -### Examples Across Domains - -| Meta Provider | Object Content | Deploy Path | Mode | -| ------------- | --------------------- | -------------------------------- | ---- | -| cron (sched) | cron.d formatted line | `/etc/cron.d/{name}` | 0644 | -| cron (intv) | shell script | `/etc/cron.{interval}/{name}` | 0755 | -| systemd | unit file | `/etc/systemd/osapi/{name}` | 0644 | -| sysctl | sysctl conf | `/etc/sysctl.d/{name}.conf` | 0644 | -| apt sources | repo entry | `/etc/apt/sources.list.d/{name}` | 0644 | - -All meta providers follow the same flow: user uploads content → meta provider -determines path + permissions + validation → `fileProvider.Deploy()` → SHA -tracked, idempotent, no magic headers. - -### FileDeployer Interface - -Meta providers depend on a narrow interface, not the full `file.Provider`: - -```go -// FileDeployer is the narrow interface for providers that deploy -// files to well-known paths. Cron, systemd, sysctl, etc. -type FileDeployer interface { - Deploy(ctx context.Context, req DeployRequest) (*DeployResult, error) - Undeploy(ctx context.Context, req UndeployRequest) (*UndeployResult, error) -} -``` - -The existing `file.Service` satisfies `FileDeployer` automatically since it -already has `Deploy`. We add `Undeploy` as part of this work. - -### Template Rendering - -The file provider already supports Go `text/template` rendering when -`ContentType: "template"`. Templates have access to: - -```go -type TemplateContext struct { - Facts map[string]any // agent facts (arch, kernel, OS, etc.) - Vars map[string]any // user-supplied variables - Hostname string // agent hostname -} -``` - -Meta providers pass `ContentType` and `Vars` through to `Deploy()`. A cron -script template can reference `{{ .Hostname }}`, `{{ .Facts.os_family }}`, or -user-supplied `{{ .Vars.region }}`. The same uploaded template renders -differently per host. - -## File Provider Changes - -### Undeploy Method - -Removes a deployed file from disk. The object stays in the object store. The -file-state KV entry is updated to record the undeploy (not deleted — it serves -as an audit trail). - -```go -type UndeployRequest struct { - Path string `json:"path"` -} - -type UndeployResult struct { - Changed bool `json:"changed"` - Path string `json:"path"` -} -``` - -Behavior: - -- If file exists on disk: remove it, update file-state KV, `Changed: true` -- If file does not exist: no-op, `Changed: false` -- Object store entry is untouched -- `client file list` still shows the object - -### Undeploy API Endpoint - -``` -DELETE /node/{hostname}/file/deploy/{name} -``` - -Removes the deployed file from disk on the target node. The object stays in the -store for redeployment or audit purposes. This is the inverse of -`POST /node/{hostname}/file/deploy`. - -### Protected Objects - -System-managed templates ship with osapi and cannot be deleted by users. These -are templates that meta providers reference (e.g., a standard systemd unit -template). - -**Storage:** Objects in the NATS object store with a `osapi/` name prefix are -protected. Convention-based, no metadata changes needed. - -``` -osapi/systemd-unit.tmpl → protected (cannot delete) -osapi/sysctl-conf.tmpl → protected (cannot delete) -backup.sh → user-managed (deletable) -my-nginx.conf → user-managed (deletable) -``` - -**Enforcement:** The `file delete` handler checks if the object name starts with -`osapi/` and returns 403 if so. - -**Seeding:** The agent seeds system templates into the object store on startup -(idempotent — skip if already present). Templates are embedded in the binary via -`go:embed`. - -**Listing:** `client file list` shows both system and user objects. A `source` -column indicates `system` vs `user`. - -## Cron Provider Refactor - -### API Changes - -The `command` field is removed. A new `object` field references an uploaded file -in the object store. The cron provider deploys the object to the correct path -with the correct permissions. - -**CronCreateRequest:** - -```yaml -CronCreateRequest: - type: object - required: - - name - - object - properties: - name: - type: string - description: > - Name for the cron entry. Used as the filename under /etc/cron.d/ or - /etc/cron.{interval}/. - object: - type: string - description: > - Name of the uploaded file in the object store to deploy as the cron - entry content. - schedule: - type: string - description: > - Cron schedule expression (e.g., "*/5 * * * *"). Mutually exclusive with - interval. - interval: - type: string - description: > - Periodic interval (hourly, daily, weekly, monthly). Mutually exclusive - with schedule. - enum: [hourly, daily, weekly, monthly] - user: - type: string - description: > - User to run the command as. Only applies to cron.d entries. - content_type: - type: string - description: > - "raw" or "template". When "template", the file content is rendered - through Go's text/template engine with facts and vars. - enum: [raw, template] - default: raw - vars: - type: object - description: > - Template variables. Only used when content_type is "template". -``` - -**CronUpdateRequest:** - -```yaml -CronUpdateRequest: - type: object - properties: - object: - type: string - description: > - New object to deploy (redeploy with updated content). - schedule: - type: string - user: - type: string - content_type: - type: string - enum: [raw, template] - vars: - type: object -``` - -### Provider Changes - -The Debian cron provider takes a `FileDeployer` dependency: - -```go -type Debian struct { - logger *slog.Logger - fs afero.Fs - fileDeployer file.FileDeployer -} -``` - -**Create:** - -1. Validate name and schedule/interval -2. Check uniqueness across all cron directories -3. Determine path and mode: - - Schedule → `/etc/cron.d/{name}`, 0644 - - Interval → `/etc/cron.{interval}/{name}`, 0755 -4. Call - `fileDeployer.Deploy(ctx, file.DeployRequest{ ObjectName: entry.Object, Path: path, Mode: mode, ContentType: entry.ContentType, Vars: entry.Vars, })` -5. Return `CreateResult{Changed: result.Changed}` - -**Update:** - -1. Validate name, find existing entry path -2. Call `fileDeployer.Deploy()` with new object/vars — idempotent, skips if SHA - unchanged -3. Return `UpdateResult{Changed: result.Changed}` - -**Delete:** - -1. Validate name, find existing entry path -2. Call `fileDeployer.Undeploy(ctx, file.UndeployRequest{Path: path})` -3. Return `DeleteResult{Changed: result.Changed}` - -**List:** - -1. Scan `/etc/cron.d/` and `/etc/cron.{interval}/` directories -2. For each file, compute the file-state KV key and check if it has a state - entry — if yes, it is managed by osapi -3. Return entries with metadata from the file-state KV (object name, SHA, etc.) -4. No `# Managed by osapi` header — the file-state KV is the source of truth - -**Get:** - -1. Compute file-state KV key for the expected path -2. Look up state entry -3. Read file from disk for current content -4. Return entry with metadata - -### Removed - -- `buildFileContent()` — content comes from the uploaded object -- `# Managed by osapi` header — file-state KV is the source of truth -- `command` field — replaced by `object` reference -- Direct `afero.WriteFile` calls — replaced by `fileDeployer.Deploy()` - -## Agent Wiring Changes - -The cron provider factory needs the file provider: - -```go -// factory.go -func (f *ProviderFactory) CreateProviders() (...) { - // ... - var cronProvider cronProv.Provider - switch plat { - case "debian": - cronProvider = cronProv.NewDebianProvider( - f.logger, f.appFs, fileProvider) - // ... - } -} -``` - -The cron provider must be created after the file provider. The factory already -creates the file provider first. - -## SDK Changes - -### Client - -- `CronCreateOpts`: remove `Command`, add `Object`, `ContentType`, `Vars` -- `CronUpdateOpts`: remove `Command`, add `Object`, `ContentType`, `Vars` -- Add `FileUndeploy(ctx, hostname, name)` method to `FileService` -- Add `source` field to file list results (system vs user) - -### CLI - -- `client node schedule cron create`: remove `--command`, add `--object`, - `--content-type`, `--vars` -- `client node schedule cron update`: remove `--command`, add `--object`, - `--content-type`, `--vars` -- `client node file undeploy`: new command to remove deployed file from disk -- `client node file list`: add SOURCE column (osapi/user) - -## Scope Summary - -| Area | Change | -| ---------------- | --------------------------------------------------------- | -| File provider | Add `Undeploy` method, `FileDeployer` interface | -| File API | Add undeploy endpoint | -| File handler | Protected object check on delete, undeploy handler | -| File SDK/CLI | `undeploy` command, `source` column in list | -| Cron provider | Refactor to use `FileDeployer`, remove direct file writes | -| Cron API | `command` → `object`, add `content_type`/`vars` | -| Cron handler | Pass through new fields | -| Cron SDK/CLI | Update opts and flags | -| Agent wiring | Pass file provider to cron provider | -| System templates | `go:embed` + seeding on startup | -| Tests | All of the above | -| Docs | Feature pages, CLI docs, API docs | diff --git a/docs/plans/2026-03-22-file-backed-meta-providers.md b/docs/plans/2026-03-22-file-backed-meta-providers.md deleted file mode 100644 index 4d0b6d940..000000000 --- a/docs/plans/2026-03-22-file-backed-meta-providers.md +++ /dev/null @@ -1,1315 +0,0 @@ -# File-Backed Meta Providers Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> (if subagents available) or superpowers:executing-plans to implement this -> plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Refactor the cron provider to delegate file writes to the file -provider for SHA tracking and idempotency, add file undeploy, and add protected -object support. - -**Architecture:** Meta providers (cron, future systemd/sysctl) depend on a -narrow `FileDeployer` interface instead of writing files directly. The file -provider gains an `Undeploy` method. Objects with a `system/` prefix are -protected from deletion. - -**Tech Stack:** Go 1.25, NATS JetStream (Object Store + KV), Echo, Cobra, -oapi-codegen (strict-server), testify/suite - ---- - -## Chunk 1: File Provider — Undeploy + FileDeployer Interface - -### Task 1: Add FileDeployer interface and Undeploy types - -**Files:** - -- Modify: `internal/provider/file/types.go` - -- [ ] **Step 1: Add UndeployRequest, UndeployResult, and FileDeployer - interface** - -Add after the existing `Provider` interface: - -```go -// UndeployRequest contains parameters for removing a deployed file from disk. -type UndeployRequest struct { - // Path is the filesystem path to undeploy. - Path string `json:"path"` -} - -// UndeployResult contains the result of a file undeploy operation. -type UndeployResult struct { - // Changed indicates whether the file was removed. - Changed bool `json:"changed"` - // Path is the filesystem path that was undeployed. - Path string `json:"path"` -} - -// FileDeployer is the narrow interface for providers that deploy files -// to well-known paths. Meta providers (cron, systemd, sysctl) depend -// on this instead of the full Provider interface. -type FileDeployer interface { - // Deploy writes file content from the object store to the target - // path with SHA tracking and idempotency. - Deploy( - ctx context.Context, - req DeployRequest, - ) (*DeployResult, error) - // Undeploy removes a deployed file from disk. The object store - // entry and file-state KV record are preserved. - Undeploy( - ctx context.Context, - req UndeployRequest, - ) (*UndeployResult, error) -} -``` - -- [ ] **Step 2: Verify it compiles** - -Run: `go build ./internal/provider/file/...` - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/file/types.go -git commit -m "feat: add FileDeployer interface and Undeploy types to file provider" -``` - -### Task 2: Implement Undeploy method - -**Files:** - -- Create: `internal/provider/file/undeploy.go` -- Create: `internal/provider/file/undeploy_public_test.go` - -- [ ] **Step 1: Write the failing test** - -Create `internal/provider/file/undeploy_public_test.go` with a testify/suite. -Follow the pattern from `deploy_public_test.go`. Test cases: - -1. "when file exists on disk" — file exists, state exists → removes file, - updates state with `Undeployed: true`, returns `Changed: true` -2. "when file does not exist" — no file on disk → returns `Changed: false` -3. "when file exists but no state entry" — file exists, no KV entry → removes - file, returns `Changed: true` (no state to update) -4. "when fs remove fails" — `fs.Remove` returns error → returns error - -Each test should: - -- Set up afero.MemMapFs with files as needed -- Set up mock stateKV (or real in-memory KV) -- Call `provider.Undeploy(ctx, req)` -- Assert Changed, Path, and file absence on disk - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test ./internal/provider/file/... -run TestUndeploy -v` Expected: FAIL -(method not implemented) - -- [ ] **Step 3: Implement Undeploy** - -Create `internal/provider/file/undeploy.go`: - -```go -package file - -import ( - "context" - "encoding/json" - "fmt" - "log/slog" - "time" - - "github.com/osapi-io/osapi/internal/job" -) - -// Undeploy removes a deployed file from disk. The object store entry -// is preserved. The file-state KV is updated to record the undeploy. -func (p *Service) Undeploy( - ctx context.Context, - req UndeployRequest, -) (*UndeployResult, error) { - // Check if file exists on disk. - _, err := p.fs.Stat(req.Path) - if err != nil { - p.logger.Debug( - "file not on disk, nothing to undeploy", - slog.String("path", req.Path), - ) - - return &UndeployResult{ - Changed: false, - Path: req.Path, - }, nil - } - - if err := p.fs.Remove(req.Path); err != nil { - return nil, fmt.Errorf("failed to remove file %q: %w", req.Path, err) - } - - // Update file-state KV if entry exists. - stateKey := buildStateKey(p.hostname, req.Path) - entry, err := p.stateKV.Get(ctx, stateKey) - if err == nil { - var state job.FileState - if unmarshalErr := json.Unmarshal(entry.Value(), &state); unmarshalErr == nil { - state.UndeployedAt = time.Now().UTC().Format(time.RFC3339) - - stateBytes, marshalErr := marshalJSON(state) - if marshalErr == nil { - _, _ = p.stateKV.Put(ctx, stateKey, stateBytes) - } - } - } - - p.logger.Info( - "file undeployed", - slog.String("path", req.Path), - slog.Bool("changed", true), - ) - - return &UndeployResult{ - Changed: true, - Path: req.Path, - }, nil -} -``` - -- [ ] **Step 4: Add UndeployedAt to FileState** - -In `internal/job/types.go`, add `UndeployedAt` field to `FileState`: - -```go -type FileState struct { - ObjectName string `json:"object_name"` - Path string `json:"path"` - SHA256 string `json:"sha256"` - Mode string `json:"mode,omitempty"` - Owner string `json:"owner,omitempty"` - Group string `json:"group,omitempty"` - DeployedAt string `json:"deployed_at"` - ContentType string `json:"content_type"` - UndeployedAt string `json:"undeployed_at,omitempty"` -} -``` - -- [ ] **Step 5: Run tests** - -Run: `go test ./internal/provider/file/... -v` Expected: All pass - -- [ ] **Step 6: Commit** - -```bash -git add internal/provider/file/undeploy.go \ - internal/provider/file/undeploy_public_test.go \ - internal/job/types.go -git commit -m "feat: implement Undeploy method on file provider" -``` - -### Task 3: Add file undeploy to agent processor - -**Files:** - -- Modify: `internal/agent/processor_file.go` -- Modify: `pkg/sdk/client/operations.go` -- Modify: `internal/job/types.go` (operation constant alias) - -- [ ] **Step 1: Add operation constant** - -In `pkg/sdk/client/operations.go`, add: - -```go -OpFileUndeploy JobOperation = "file.undeploy.execute" -``` - -In `internal/job/types.go`, add alias: - -```go -OperationFileUndeployExecute = client.OpFileUndeploy -``` - -- [ ] **Step 2: Add undeploy case to processor** - -In `internal/agent/processor_file.go`, add `case "undeploy"` to the switch: - -```go -case "undeploy": - return a.processFileUndeploy(jobRequest) -``` - -Add the handler method: - -```go -func (a *Agent) processFileUndeploy( - jobRequest job.Request, -) (json.RawMessage, error) { - var req fileProv.UndeployRequest - if err := json.Unmarshal(jobRequest.Data, &req); err != nil { - return nil, fmt.Errorf("failed to parse file undeploy data: %w", err) - } - - result, err := a.fileProvider.Undeploy(context.Background(), req) - if err != nil { - return nil, fmt.Errorf("file undeploy failed: %w", err) - } - - return json.Marshal(result) -} -``` - -- [ ] **Step 3: Update file Provider interface** - -In `internal/provider/file/types.go`, add `Undeploy` to the existing `Provider` -interface: - -```go -type Provider interface { - Deploy(ctx context.Context, req DeployRequest) (*DeployResult, error) - Undeploy(ctx context.Context, req UndeployRequest) (*UndeployResult, error) - Status(ctx context.Context, req StatusRequest) (*StatusResult, error) -} -``` - -- [ ] **Step 4: Regenerate mocks if needed, verify compile** - -Run: `go build ./internal/agent/...` - -- [ ] **Step 5: Commit** - -```bash -git add internal/agent/processor_file.go \ - pkg/sdk/client/operations.go \ - internal/job/types.go \ - internal/provider/file/types.go -git commit -m "feat: add file undeploy operation to agent processor" -``` - -### Task 4: Add file undeploy API endpoint - -**Files:** - -- Modify: `internal/controller/api/node/gen/api.yaml` -- Create: `internal/controller/api/node/file_undeploy_post.go` -- Create: `internal/controller/api/node/file_undeploy_post_public_test.go` -- Modify: `internal/job/client/file.go` (add `ModifyFileUndeploy`) - -- [ ] **Step 1: Add endpoint to OpenAPI spec** - -In `internal/controller/api/node/gen/api.yaml`, add a new path: - -```yaml -/node/{hostname}/file/undeploy: - post: - summary: Undeploy a file - description: > - Remove a deployed file from disk on the target node. The object stays in - the store for redeployment or audit. - tags: - - file_operations - operationId: PostNodeFileUndeploy - security: - - BearerAuth: - - file:write - parameters: - - $ref: '#/components/parameters/Hostname' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/FileUndeployRequest' - responses: - '200': - description: File undeployed. - content: - application/json: - schema: - $ref: '#/components/schemas/FileUndeployResponse' - '400': - description: Invalid request. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '401': ... - '403': ... - '500': ... -``` - -Add schemas: - -```yaml -FileUndeployRequest: - type: object - required: - - path - properties: - path: - type: string - description: Filesystem path to undeploy. - x-oapi-codegen-extra-tags: - validate: 'required,min=1' - -FileUndeployResponse: - type: object - properties: - job_id: - type: string - hostname: - type: string - changed: - type: boolean -``` - -- [ ] **Step 2: Regenerate code** - -Run: `just generate` - -- [ ] **Step 3: Add ModifyFileUndeploy to job client** - -In `internal/job/client/file.go`, add: - -```go -func (c *Client) ModifyFileUndeploy( - ctx context.Context, - hostname string, - path string, -) (string, string, bool, error) { - req := fileProv.UndeployRequest{Path: path} - // ... same pattern as ModifyFileDeploy but with OperationFileUndeployExecute -} -``` - -- [ ] **Step 4: Implement handler** - -Create `internal/controller/api/node/file_undeploy_post.go` following the -pattern from `file_deploy_post.go`. - -- [ ] **Step 5: Write tests** - -Create `internal/controller/api/node/file_undeploy_post_public_test.go`. - -- [ ] **Step 6: Verify** - -Run: `go test ./internal/controller/api/node/... -v` - -- [ ] **Step 7: Commit** - -```bash -git add internal/controller/api/node/gen/api.yaml \ - internal/controller/api/node/file_undeploy_post.go \ - internal/controller/api/node/file_undeploy_post_public_test.go \ - internal/job/client/file.go -git commit -m "feat: add file undeploy API endpoint" -``` - -### Task 5: Add file undeploy to SDK and CLI - -**Files:** - -- Modify: `pkg/sdk/client/node.go` (or `file.go` if separate) -- Modify: `pkg/sdk/client/file_types.go` -- Create: `cmd/client_node_file_undeploy.go` - -- [ ] **Step 1: Add SDK method** - -Add `FileUndeploy` method to the appropriate service following existing -`FileDeploy` pattern. Add `FileUndeployOpts` and `FileUndeployResult` types. - -- [ ] **Step 2: Add CLI command** - -Create `cmd/client_node_file_undeploy.go` with `--path` flag following the -pattern from `client_node_file_deploy.go`. - -- [ ] **Step 3: Regenerate SDK client** - -Run: `go generate ./pkg/sdk/client/gen/...` - -- [ ] **Step 4: Write tests and verify** - -Run: `go test ./pkg/sdk/client/... -v && go build ./cmd/...` - -- [ ] **Step 5: Commit** - -```bash -git add pkg/sdk/client/ cmd/client_node_file_undeploy.go -git commit -m "feat: add file undeploy to SDK and CLI" -``` - ---- - -## Chunk 2: Protected Objects - -### Task 6: Add protected object check to file delete handler - -**Files:** - -- Modify: `internal/controller/api/file/file_delete.go` -- Modify: `internal/controller/api/file/file_delete_public_test.go` -- Modify: `internal/controller/api/file/file_list.go` (add source column) - -- [ ] **Step 1: Add protected check to DeleteFileByName** - -In `file_delete.go`, add check before deletion: - -```go -if strings.HasPrefix(request.Name, "system/") { - errMsg := fmt.Sprintf("cannot delete system file: %s", request.Name) - return gen.DeleteFileByName403JSONResponse{Error: &errMsg}, nil -} -``` - -- [ ] **Step 2: Add source field to file list response** - -In `file_list.go`, when building the response, set source based on prefix: - -```go -source := "user" -if strings.HasPrefix(info.Name, "system/") { - source = "system" -} -``` - -Update the OpenAPI spec to include `source` field in the file list response. - -- [ ] **Step 3: Write tests** - -Add test cases: - -- "when deleting system file returns 403" -- "when deleting user file succeeds" -- "when listing shows source column" - -- [ ] **Step 4: Verify** - -Run: `go test ./internal/controller/api/file/... -v` - -- [ ] **Step 5: Update SDK file list types** - -Add `Source` field to `FileItem` in `pkg/sdk/client/file_types.go`. - -- [ ] **Step 6: Update CLI file list** - -Add SOURCE column to `cmd/client_file_list.go` table output. - -- [ ] **Step 7: Commit** - -```bash -git add internal/controller/api/file/ pkg/sdk/client/file_types.go \ - cmd/client_file_list.go -git commit -m "feat: add protected object support and source column to file list" -``` - ---- - -## Chunk 3: Cron Provider Refactor - -### Task 7: Update cron provider interface and Entry type - -**Files:** - -- Modify: `internal/provider/scheduled/cron/types.go` - -- [ ] **Step 1: Update Entry struct** - -Replace `Command string` with `Object string`, add `ContentType` and `Vars`: - -```go -type Entry struct { - Name string `json:"name"` - Object string `json:"object,omitempty"` - Schedule string `json:"schedule,omitempty"` - Interval string `json:"interval,omitempty"` - Source string `json:"source,omitempty"` - User string `json:"user,omitempty"` - ContentType string `json:"content_type,omitempty"` - Vars map[string]any `json:"vars,omitempty"` -} -``` - -- [ ] **Step 2: Update Provider interface** - -Add `context.Context` to all methods that will now call the file provider: - -```go -type Provider interface { - List(ctx context.Context) ([]Entry, error) - Get(ctx context.Context, name string) (*Entry, error) - Create(ctx context.Context, entry Entry) (*CreateResult, error) - Update(ctx context.Context, entry Entry) (*UpdateResult, error) - Delete(ctx context.Context, name string) (*DeleteResult, error) -} -``` - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/scheduled/cron/types.go -git commit -m "refactor: update cron Entry type and Provider interface for file-backed pattern" -``` - -### Task 8: Refactor Debian cron provider to use FileDeployer - -**Files:** - -- Modify: `internal/provider/scheduled/cron/debian.go` -- Modify: `internal/provider/scheduled/cron/debian_public_test.go` - -- [ ] **Step 1: Update Debian struct and constructor** - -```go -type Debian struct { - logger *slog.Logger - fs afero.Fs - fileDeployer file.FileDeployer - stateKV jetstream.KeyValue - hostname string -} - -func NewDebianProvider( - logger *slog.Logger, - fs afero.Fs, - fileDeployer file.FileDeployer, - stateKV jetstream.KeyValue, - hostname string, -) *Debian { - return &Debian{ - logger: logger, - fs: fs, - fileDeployer: fileDeployer, - stateKV: stateKV, - hostname: hostname, - } -} -``` - -Note: `stateKV` and `hostname` are needed for `List`/`Get` to check which files -are managed by osapi (replacing `# Managed by osapi` header). - -- [ ] **Step 2: Refactor Create** - -Replace direct `afero.WriteFile` with `fileDeployer.Deploy()`: - -```go -func (d *Debian) Create( - ctx context.Context, - entry Entry, -) (*CreateResult, error) { - if err := validateName(entry.Name); err != nil { - return nil, err - } - - if existingPath, _ := d.findEntryPath(entry.Name); existingPath != "" { - return nil, fmt.Errorf("cron entry %q already exists", entry.Name) - } - - filePath, perm := d.entryFilePath(entry) - - result, err := d.fileDeployer.Deploy(ctx, file.DeployRequest{ - ObjectName: entry.Object, - Path: filePath, - Mode: fmt.Sprintf("%04o", perm), - ContentType: entry.ContentType, - Vars: entry.Vars, - }) - if err != nil { - return nil, fmt.Errorf("create cron entry: %w", err) - } - - return &CreateResult{ - Name: entry.Name, - Changed: result.Changed, - }, nil -} -``` - -- [ ] **Step 3: Refactor Update** - -Same pattern — call `fileDeployer.Deploy()` (idempotent, SHA check handles -unchanged content): - -```go -func (d *Debian) Update( - ctx context.Context, - entry Entry, -) (*UpdateResult, error) { - if err := validateName(entry.Name); err != nil { - return nil, err - } - - filePath, perm := d.findEntryPath(entry.Name) - if filePath == "" { - return nil, fmt.Errorf("cron entry %q does not exist", entry.Name) - } - - result, err := d.fileDeployer.Deploy(ctx, file.DeployRequest{ - ObjectName: entry.Object, - Path: filePath, - Mode: fmt.Sprintf("%04o", perm), - ContentType: entry.ContentType, - Vars: entry.Vars, - }) - if err != nil { - return nil, fmt.Errorf("update cron entry: %w", err) - } - - return &UpdateResult{ - Name: entry.Name, - Changed: result.Changed, - }, nil -} -``` - -- [ ] **Step 4: Refactor Delete** - -Use `fileDeployer.Undeploy()`: - -```go -func (d *Debian) Delete( - ctx context.Context, - name string, -) (*DeleteResult, error) { - if err := validateName(name); err != nil { - return nil, err - } - - filePath, _ := d.findEntryPath(name) - if filePath == "" { - return &DeleteResult{ - Name: name, - Changed: false, - }, nil - } - - result, err := d.fileDeployer.Undeploy(ctx, file.UndeployRequest{ - Path: filePath, - }) - if err != nil { - return nil, fmt.Errorf("delete cron entry: %w", err) - } - - return &DeleteResult{ - Name: name, - Changed: result.Changed, - }, nil -} -``` - -- [ ] **Step 5: Refactor List** - -Replace `# Managed by osapi` header check with file-state KV lookup: - -```go -func (d *Debian) List( - ctx context.Context, -) ([]Entry, error) { - var result []Entry - - cronDirEntries, err := afero.ReadDir(d.fs, cronDir) - if err != nil { - return nil, fmt.Errorf("list cron entries: %w", err) - } - - for _, entry := range cronDirEntries { - if entry.IsDir() { - continue - } - - if !d.isManagedFile(ctx, cronDir+"/"+entry.Name()) { - continue - } - - cronEntry := d.buildEntryFromState(ctx, entry.Name(), cronDir, "cron.d") - if cronEntry != nil { - result = append(result, *cronEntry) - } - } - - for _, interval := range periodicIntervals { - dir := periodicDirs[interval] - dirEntries, err := afero.ReadDir(d.fs, dir) - if err != nil { - continue - } - - for _, entry := range dirEntries { - if entry.IsDir() { - continue - } - - if !d.isManagedFile(ctx, dir+"/"+entry.Name()) { - continue - } - - cronEntry := d.buildEntryFromState(ctx, entry.Name(), dir, interval) - if cronEntry != nil { - result = append(result, *cronEntry) - } - } - } - - return result, nil -} -``` - -Add helper methods: - -```go -// isManagedFile checks if the file at path has a file-state KV entry. -func (d *Debian) isManagedFile( - ctx context.Context, - path string, -) bool { - stateKey := file.BuildStateKey(d.hostname, path) - _, err := d.stateKV.Get(ctx, stateKey) - return err == nil -} - -// buildEntryFromState creates an Entry from file-state KV metadata. -func (d *Debian) buildEntryFromState( - ctx context.Context, - name string, - dir string, - source string, -) *Entry { - // Build entry from state metadata and filesystem - // ... -} -``` - -Note: `file.BuildStateKey` needs to be exported (currently `buildStateKey` is -unexported). Export it in `internal/provider/file/deploy.go`. - -- [ ] **Step 6: Refactor Get** - -Replace header-based parsing with state-based lookup: - -```go -func (d *Debian) Get( - ctx context.Context, - name string, -) (*Entry, error) { - if err := validateName(name); err != nil { - return nil, err - } - - filePath, _ := d.findEntryPath(name) - if filePath == "" { - return nil, fmt.Errorf("cron entry %q: not found", name) - } - - if !d.isManagedFile(ctx, filePath) { - return nil, fmt.Errorf("cron entry %q is not managed by osapi", name) - } - - return d.buildEntryFromPath(ctx, name, filePath) -} -``` - -- [ ] **Step 7: Remove dead code** - -Delete `buildFileContent()`, `readCronFile()`, `readPeriodicFile()`, and the -`managedHeader` constant. - -- [ ] **Step 8: Update tests** - -Rewrite `debian_public_test.go` to use mock `FileDeployer` and `KeyValue` -interfaces instead of checking file content for `# Managed by osapi`. - -- [ ] **Step 9: Verify** - -Run: `go test ./internal/provider/scheduled/cron/... -v` - -- [ ] **Step 10: Commit** - -```bash -git add internal/provider/scheduled/cron/ internal/provider/file/deploy.go -git commit -m "refactor: cron provider uses FileDeployer instead of direct file writes" -``` - -### Task 9: Update Darwin and Linux stubs - -**Files:** - -- Modify: `internal/provider/scheduled/cron/darwin.go` -- Modify: `internal/provider/scheduled/cron/linux.go` - -- [ ] **Step 1: Add context.Context to stub methods** - -Both stubs return `ErrUnsupported` but need updated signatures: - -```go -func (d *Darwin) List(ctx context.Context) ([]Entry, error) { - return nil, provider.ErrUnsupported -} -// ... same for Get, Create, Update, Delete -``` - -- [ ] **Step 2: Verify** - -Run: `go build ./internal/provider/scheduled/cron/...` - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/scheduled/cron/darwin.go \ - internal/provider/scheduled/cron/linux.go -git commit -m "refactor: update cron provider stubs for context parameter" -``` - -### Task 10: Update agent processor and wiring - -**Files:** - -- Modify: `internal/agent/processor_schedule.go` -- Modify: `internal/agent/factory.go` -- Modify: `internal/agent/types.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Pass context to cron provider calls** - -In `processor_schedule.go`, pass `context.Background()` (or job context) to all -cron provider calls. - -- [ ] **Step 2: Update factory to accept file provider** - -In `factory.go`, `CreateProviders` needs to accept the file provider and pass it -to `NewDebianProvider`. Alternatively, create the cron provider in -`cmd/agent_setup.go` after both providers are initialized. - -The cleanest approach: create the cron provider in `agent_setup.go` after the -file provider, since the cron provider depends on the file provider + stateKV + -hostname which are all available there: - -```go -var cronProvider cronProv.Provider -switch platform.Detect() { -case "debian": - cronProvider = cronProv.NewDebianProvider( - logger, appFs, fileProvider, fileStateKV, hostname) -// ... -} -``` - -- [ ] **Step 3: Verify** - -Run: `go build ./... && go test ./internal/agent/... -v` - -- [ ] **Step 4: Commit** - -```bash -git add internal/agent/ cmd/agent_setup.go -git commit -m "refactor: wire cron provider with file deployer dependency" -``` - ---- - -## Chunk 4: Cron API, Handler, Job Client Changes - -### Task 11: Update cron OpenAPI spec - -**Files:** - -- Modify: `internal/controller/api/schedule/gen/api.yaml` - -- [ ] **Step 1: Update CronCreateRequest** - -Replace `command` with `object`, add `content_type` and `vars`: - -```yaml -CronCreateRequest: - type: object - required: - - name - - object - properties: - name: - type: string - x-oapi-codegen-extra-tags: - validate: 'required,min=1,max=64' - object: - type: string - description: > - Name of the uploaded file in the object store. - x-oapi-codegen-extra-tags: - validate: 'required,min=1' - schedule: - type: string - x-oapi-codegen-extra-tags: - validate: 'required_without=Interval,excluded_with=Interval,omitempty,cron_schedule' - interval: - type: string - enum: [hourly, daily, weekly, monthly] - x-oapi-codegen-extra-tags: - validate: - 'required_without=Schedule,excluded_with=Schedule,omitempty,oneof=hourly - daily weekly monthly' - user: - type: string - content_type: - type: string - enum: [raw, template] - x-oapi-codegen-extra-tags: - validate: 'omitempty,oneof=raw template' - vars: - type: object - additionalProperties: true -``` - -- [ ] **Step 2: Update CronUpdateRequest** - -Replace `command` with `object`, add `content_type` and `vars`. - -- [ ] **Step 3: Update CronEntry response schema** - -Add `object` field, remove `command`. Add `content_type`. - -- [ ] **Step 4: Regenerate** - -Run: `just generate` - -- [ ] **Step 5: Commit** - -```bash -git add internal/controller/api/schedule/gen/ -git commit -m "feat: update cron OpenAPI spec for file-backed pattern" -``` - -### Task 12: Update cron handler - -**Files:** - -- Modify: `internal/controller/api/schedule/cron_create.go` -- Modify: `internal/controller/api/schedule/cron_update.go` -- Modify: `internal/controller/api/schedule/cron_get.go` -- Modify: `internal/controller/api/schedule/cron_list_get.go` -- Modify: all corresponding `*_public_test.go` files - -- [ ] **Step 1: Update PostNodeScheduleCron** - -Map `Object`, `ContentType`, `Vars` from request body to `cronProv.Entry` -instead of `Command`. - -- [ ] **Step 2: Update PutNodeScheduleCron** - -Same mapping changes for update. - -- [ ] **Step 3: Update response mapping in list/get** - -Map `Object`, `ContentType` from response instead of `Command`. - -- [ ] **Step 4: Update tests** - -Fix all handler test fixtures to use `object` instead of `command`. - -- [ ] **Step 5: Verify** - -Run: `go test ./internal/controller/api/schedule/... -v` - -- [ ] **Step 6: Commit** - -```bash -git add internal/controller/api/schedule/ -git commit -m "refactor: update cron handlers for file-backed pattern" -``` - -### Task 13: Update job client for cron - -**Files:** - -- Modify: `internal/job/client/schedule_cron.go` - -- [ ] **Step 1: Update cron job client methods** - -The `Entry` type has changed (no `Command`, has `Object`, `ContentType`, -`Vars`). The job client marshals this into the job request data. The methods -themselves don't need logic changes since they pass the full `Entry` — but the -`Entry` struct has changed, so this should just compile. - -- [ ] **Step 2: Verify** - -Run: `go build ./internal/job/...` - -- [ ] **Step 3: Commit** - -```bash -git add internal/job/client/schedule_cron.go -git commit -m "refactor: update cron job client for updated Entry type" -``` - ---- - -## Chunk 5: SDK, CLI, and Docs - -### Task 14: Update SDK cron types and methods - -**Files:** - -- Modify: `pkg/sdk/client/schedule_types.go` -- Modify: `pkg/sdk/client/schedule.go` - -- [ ] **Step 1: Update SDK types** - -```go -type CronCreateOpts struct { - Name string - Object string - Schedule string - Interval string - User string - ContentType string - Vars map[string]any -} - -type CronUpdateOpts struct { - Object string - Schedule string - User string - ContentType string - Vars map[string]any -} - -type CronEntryResult struct { - Name string `json:"name"` - Object string `json:"object,omitempty"` - Schedule string `json:"schedule,omitempty"` - Interval string `json:"interval,omitempty"` - Source string `json:"source,omitempty"` - User string `json:"user,omitempty"` - ContentType string `json:"content_type,omitempty"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Update SDK methods** - -In `schedule.go`, update `CronCreate` and `CronUpdate` to map `Object`, -`ContentType`, `Vars` to the gen request types. Update conversion functions to -handle new fields. - -- [ ] **Step 3: Regenerate SDK client** - -Run: `go generate ./pkg/sdk/client/gen/...` - -- [ ] **Step 4: Write tests and verify** - -Run: `go test ./pkg/sdk/client/... -v` - -- [ ] **Step 5: Commit** - -```bash -git add pkg/sdk/client/ -git commit -m "refactor: update SDK cron types for file-backed pattern" -``` - -### Task 15: Update CLI cron commands - -**Files:** - -- Modify: `cmd/client_node_schedule_cron_create.go` -- Modify: `cmd/client_node_schedule_cron_update.go` -- Modify: `cmd/client_node_schedule_cron_get.go` -- Modify: `cmd/client_node_schedule_cron_list.go` - -- [ ] **Step 1: Update create command** - -Remove `--command` flag, add `--object` (required), `--content-type` (optional, -default "raw"), `--var` (repeatable key=value). - -- [ ] **Step 2: Update update command** - -Remove `--command`, add `--object`, `--content-type`, `--var`. - -- [ ] **Step 3: Update list output** - -Replace COMMAND column with OBJECT column. - -- [ ] **Step 4: Update get output** - -Replace Command field with Object field. - -- [ ] **Step 5: Verify** - -Run: `go build ./cmd/...` - -- [ ] **Step 6: Commit** - -```bash -git add cmd/client_node_schedule_cron_*.go -git commit -m "refactor: update CLI cron commands for file-backed pattern" -``` - -### Task 16: Update SDK example and docs - -**Files:** - -- Modify: `examples/sdk/client/cron.go` -- Modify: `docs/docs/sidebar/features/cron-management.md` -- Modify: CLI docs for cron commands -- Modify: `docs/docs/sidebar/features/file-management.md` (undeploy section) - -- [ ] **Step 1: Update SDK example** - -Update `examples/sdk/client/cron.go` to use `Object` instead of `Command`: - -```go -createResp, err := c.Schedule.CronCreate(ctx, target, client.CronCreateOpts{ - Name: "backup-daily", - Schedule: "0 2 * * *", - Object: "backup-script", - User: "root", -}) -``` - -- [ ] **Step 2: Update cron feature docs** - -Update `docs/docs/sidebar/features/cron-management.md` to document: - -- Object-based workflow (upload → create) -- Template support with content_type and vars -- No `# Managed by osapi` header -- File-state KV tracking - -- [ ] **Step 3: Update file management docs** - -Add undeploy section to `docs/docs/sidebar/features/file-management.md`. -Document protected objects with `system/` prefix. - -- [ ] **Step 4: Update CLI docs** - -Update CLI reference docs for cron create, update, get, list with new flags. Add -CLI docs for `client node file undeploy`. - -- [ ] **Step 5: Commit** - -```bash -git add examples/sdk/client/cron.go docs/ -git commit -m "docs: update cron and file docs for file-backed meta provider pattern" -``` - ---- - -## Chunk 6: System Template Seeding - -### Task 17: Add system template seeding on agent startup - -**Files:** - -- Create: `internal/agent/templates/` (embedded templates) -- Modify: `cmd/agent_setup.go` (seed on startup) - -- [ ] **Step 1: Create embedded templates directory** - -Create `internal/agent/templates/` with initial system templates. For now, just -a placeholder `system/cron-wrapper.tmpl` that can be used as a reference: - -``` -#!/bin/sh -# {{ .Vars.description }} -{{ .Vars.command }} -``` - -- [ ] **Step 2: Add seeding function** - -Create `internal/agent/seed.go` with: - -```go -//go:embed templates/* -var systemTemplates embed.FS - -func SeedSystemTemplates( - ctx context.Context, - objStore jetstream.ObjectStore, -) error { - // Walk embedded templates, upload each with "system/" prefix - // Skip if already present (idempotent) -} -``` - -- [ ] **Step 3: Call from agent startup** - -In `cmd/agent_setup.go`, call `agent.SeedSystemTemplates()` after Object Store -is initialized. - -- [ ] **Step 4: Write tests** - -- [ ] **Step 5: Commit** - -```bash -git add internal/agent/seed.go internal/agent/templates/ cmd/agent_setup.go -git commit -m "feat: seed system templates on agent startup" -``` - ---- - -## Chunk 7: Verification - -### Task 18: Full verification - -- [ ] **Step 1: Build** - -Run: `go build ./...` - -- [ ] **Step 2: Unit tests** - -Run: `just go::unit` - -- [ ] **Step 3: Lint** - -Run: `just go::vet` - -- [ ] **Step 4: Format** - -Run: `just go::fmt` - -- [ ] **Step 5: Generate** - -Run: `just generate` — verify no diff - -- [ ] **Step 6: Commit any fixes** - ---- - -## Files Modified Summary - -| File | Change | -| ---------------------------------------------------------------- | --------------------------------------------------------- | -| `internal/provider/file/types.go` | Add FileDeployer, UndeployRequest/Result, update Provider | -| `internal/provider/file/undeploy.go` | New: Undeploy method | -| `internal/provider/file/undeploy_public_test.go` | New: Undeploy tests | -| `internal/provider/file/deploy.go` | Export BuildStateKey | -| `internal/job/types.go` | Add UndeployedAt to FileState, operation alias | -| `internal/agent/processor_file.go` | Add undeploy case | -| `internal/agent/processor_schedule.go` | Pass context to cron calls | -| `internal/agent/factory.go` | Update cron provider creation | -| `internal/agent/types.go` | No change if factory handles wiring | -| `internal/agent/seed.go` | New: system template seeding | -| `internal/agent/templates/` | New: embedded templates | -| `internal/controller/api/node/gen/api.yaml` | Add undeploy endpoint | -| `internal/controller/api/node/file_undeploy_post.go` | New: undeploy handler | -| `internal/controller/api/node/file_undeploy_post_public_test.go` | New: tests | -| `internal/controller/api/file/file_delete.go` | Protected object check | -| `internal/controller/api/file/file_list.go` | Source column | -| `internal/controller/api/schedule/gen/api.yaml` | command→object, add content_type/vars | -| `internal/controller/api/schedule/cron_create.go` | Map new fields | -| `internal/controller/api/schedule/cron_update.go` | Map new fields | -| `internal/controller/api/schedule/cron_get.go` | Map new fields | -| `internal/controller/api/schedule/cron_list_get.go` | Map new fields | -| `internal/provider/scheduled/cron/types.go` | Update Entry, add ctx to interface | -| `internal/provider/scheduled/cron/debian.go` | Full refactor to FileDeployer | -| `internal/provider/scheduled/cron/darwin.go` | Add ctx to stubs | -| `internal/provider/scheduled/cron/linux.go` | Add ctx to stubs | -| `internal/job/client/file.go` | Add ModifyFileUndeploy | -| `pkg/sdk/client/operations.go` | Add OpFileUndeploy | -| `pkg/sdk/client/schedule_types.go` | command→object, add content_type/vars | -| `pkg/sdk/client/schedule.go` | Map new fields | -| `pkg/sdk/client/file_types.go` | Add FileUndeployResult, Source to FileItem | -| `cmd/agent_setup.go` | Cron provider wiring, template seeding | -| `cmd/client_node_file_undeploy.go` | New: undeploy CLI command | -| `cmd/client_node_schedule_cron_create.go` | --command→--object, add flags | -| `cmd/client_node_schedule_cron_update.go` | Same flag changes | -| `cmd/client_node_schedule_cron_list.go` | OBJECT column | -| `cmd/client_node_schedule_cron_get.go` | Object field | -| `examples/sdk/client/cron.go` | Use Object instead of Command | -| `docs/` | Feature pages, CLI docs | diff --git a/docs/plans/2026-03-22-health-probes-app-metrics-design.md b/docs/plans/2026-03-22-health-probes-app-metrics-design.md deleted file mode 100644 index 1ce96d9dd..000000000 --- a/docs/plans/2026-03-22-health-probes-app-metrics-design.md +++ /dev/null @@ -1,177 +0,0 @@ -# Health Probes and Application Metrics Design - -## Goal - -Add `/health` (liveness) and `/health/ready` (readiness) probes to the -per-component metrics server, and register application-specific Prometheus -metrics for each component. Bridge health into metrics with an -`osapi_component_up` gauge. - -## Background - -Each OSAPI component (controller, agent, NATS server) runs a per-component -metrics server (`internal/telemetry/metrics/server.go`) with an isolated -Prometheus registry and OTEL MeterProvider. Today the metrics server only -exposes Go runtime and process collector metrics at `/metrics`. There are no -health probes on the metrics server — the controller has complex `/health`, -`/health/ready`, `/health/status` endpoints on its API server (with auth, -OpenAPI types, metrics aggregation), but agent and NATS have nothing. - -Operators need: - -- Liveness and readiness probes on every component for container orchestrators -- Application metrics (request counts, job throughput, latency) for dashboards -- A Prometheus gauge that reflects component readiness for alerting - -## Design - -### Health probes on the metrics server - -Add `/health` and `/health/ready` routes to the existing metrics server HTTP -mux. Implementation lives in a new file `internal/telemetry/metrics/health.go`. - -**`/health` (liveness):** - -Always returns `{"status":"ok"}` with HTTP 200 if the process is running. No -dependencies, no checks. Used by container orchestrators to detect hung -processes. - -**`/health/ready` (readiness):** - -Calls an injected readiness function. Returns `{"status":"ready"}` with HTTP 200 -when the component can do its job, or `{"status":"not_ready","error":"..."}` -with HTTP 503 when it cannot. - -Readiness semantics per component: - -| Component | Ready when | -| ---------- | --------------------------------- | -| Controller | NATS connected, KV accessible | -| Agent | NATS connected, consumers started | -| NATS | JetStream available | - -Both routes are registered unconditionally in `New()`. The readiness function is -injected post-construction via `SetReadinessFunc(fn func() error)`, following -the same pattern as `agent.SetSubComponents()`. If no readiness function is set, -`/health/ready` returns 503 with `"readiness check not configured"`. Each -component wires its own check in `cmd/`. - -Both health endpoints return `Content-Type: application/json`. - -The readiness function is called on every `/health/ready` request and on every -Prometheus scrape (via `GaugeFunc`). Implementations should be fast and -non-blocking. If a readiness check needs to contact an external service (e.g., -NATS ping), it should use a short timeout (5 seconds) to avoid blocking the HTTP -response. - -The controller's existing API health endpoints (`/health`, `/health/ready`, -`/health/status`) remain unchanged — they serve a different purpose (API -clients, auth-gated status). The metrics server health probes are for -infrastructure tooling (Kubernetes, load balancers) and don't require auth. - -### `osapi_component_up` gauge - -A Prometheus gauge registered on each component's metrics server registry. Value -is `1` when ready, `0` when not. Evaluated lazily on each Prometheus scrape -using a `prometheus.GaugeFunc` that calls the same readiness function as -`/health/ready`. `GaugeFunc` is used because OTEL does not support -lazy-evaluated gauges — this is the one metric that uses the native Prometheus -client directly rather than OTEL. - -Each component's gauge is on its own isolated registry (different port), so -there is no label collision. When federating into a single Prometheus instance, -operators distinguish components using the `job` or `instance` label from their -scrape config. - -This bridges health into metrics — operators can alert on -`osapi_component_up == 0` without polling the health endpoint separately. - -### Application metrics - -Each component registers application-specific metrics via the metrics server's -`MeterProvider()` or `Registry()` at startup. - -**Controller metrics:** - -| Metric | Type | Labels | Description | -| ------------------------------------ | --------- | -------------------- | ------------------ | -| `osapi_api_requests_total` | counter | method, path, status | HTTP request count | -| `osapi_api_request_duration_seconds` | histogram | method, path | Request latency | -| `osapi_jobs_created_total` | counter | | Jobs submitted | -| `osapi_component_up` | gauge | | 1 = ready, 0 = not | - -**Agent metrics:** - -| Metric | Type | Labels | Description | -| ----------------------------- | --------- | ------ | ------------------------ | -| `osapi_jobs_processed_total` | counter | status | Jobs completed/failed | -| `osapi_jobs_active` | gauge | | Currently executing jobs | -| `osapi_job_duration_seconds` | histogram | | Job execution time | -| `osapi_heartbeat_age_seconds` | gauge | | Time since last write | -| `osapi_component_up` | gauge | | 1 = ready, 0 = not | - -**NATS server metrics:** - -| Metric | Type | Description | -| -------------------- | ----- | ------------------ | -| `osapi_component_up` | gauge | 1 = ready, 0 = not | - -The embedded NATS server exposes its own monitoring metrics natively. -`osapi_component_up` is the only custom metric needed. - -### Metrics registration approach - -Controller HTTP metrics (`osapi_api_requests_total`, -`osapi_api_request_duration_seconds`) are collected via the `otelecho` -middleware on the Echo server, using the metrics server's `MeterProvider()`. -This keeps the API server unaware of Prometheus — it instruments via OTEL, and -the Prometheus exporter on the metrics server handles the translation. The -`path` label uses Echo's route template (e.g., `/api/v1/node/:hostname`) not the -literal request path, to avoid unbounded cardinality. - -Agent job metrics (`osapi_jobs_processed_total`, `osapi_jobs_active`, -`osapi_job_duration_seconds`) are instrumented in the agent's handler/ processor -layer using the metrics server's `MeterProvider()`. The `MeterProvider` must be -passed to the agent (or set post-construction) so the agent package doesn't -import the metrics package directly. - -`osapi_heartbeat_age_seconds` is a `GaugeFunc` registered on the Prometheus -registry. The agent exposes a `LastHeartbeatTime() time.Time` method, and the -`cmd/` wiring layer creates a closure over it for the `GaugeFunc`. - -### File structure - -``` -internal/telemetry/metrics/ - server.go # Existing — add SetReadinessFunc, wire health routes - health.go # New — /health and /health/ready handlers - types.go # Existing — add ReadinessFunc field - server_test.go # Update for new functionality - server_public_test.go # Update for new functionality - health_test.go # New — health handler tests - health_public_test.go # New — health handler public tests -``` - -### Configuration - -No config changes. Health probes are always available when the metrics server is -enabled. There's no reason to want metrics without health or vice versa. - -### Documentation - -Update `docs/docs/sidebar/features/metrics.md`: - -- Add "Health Probes" section documenting `/health` and `/health/ready` -- Add "Application Metrics Reference" section with tables of all custom metrics, - their types, labels, and which component reports them -- Update the "What It Exposes" section - -Update `docs/docs/sidebar/features/health-checks.md`: - -- Cross-reference the metrics server health probes -- Note that `/health` and `/health/ready` are available on each component's - metrics port without authentication - -Update `docs/docs/sidebar/usage/configuration.md`: - -- Note the health probe endpoints in the metrics server sections diff --git a/docs/plans/2026-03-22-health-probes-app-metrics.md b/docs/plans/2026-03-22-health-probes-app-metrics.md deleted file mode 100644 index 8e3399bb0..000000000 --- a/docs/plans/2026-03-22-health-probes-app-metrics.md +++ /dev/null @@ -1,766 +0,0 @@ -# Health Probes and Application Metrics Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> (if subagents available) or superpowers:executing-plans to implement this -> plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add `/health` and `/health/ready` probes to the per-component metrics -server, register application-specific Prometheus metrics for each component, and -bridge health into metrics with an `osapi_component_up` gauge. - -**Architecture:** Health handlers live in a new `health.go` file under -`internal/telemetry/metrics/`. The existing metrics `Server` gains a -`SetReadinessFunc` method and registers `/health` + `/health/ready` routes in -`New()`. Each component wires its readiness check and application metrics in -`cmd/`. The `otelecho` middleware (already imported) gets the metrics server's -`MeterProvider` for controller HTTP metrics. Agent job metrics are instrumented -via OTEL in the handler layer. - -**Tech Stack:** Go, Prometheus client_golang, OpenTelemetry (otel/metric), -otelecho middleware, testify/suite - ---- - -## Chunk 1: Health probes on the metrics server - -### Task 1: Add health handlers - -**Files:** - -- Create: `internal/telemetry/metrics/health.go` -- Create: `internal/telemetry/metrics/health_public_test.go` -- Modify: `internal/telemetry/metrics/types.go` -- Modify: `internal/telemetry/metrics/server.go` - -- [ ] **Step 1: Add `readinessFunc` field to `Server` struct** - -In `internal/telemetry/metrics/types.go`, add the field: - -```go -type Server struct { - httpServer *http.Server - logger *slog.Logger - registry *prometheus.Registry - meterProvider *sdkmetric.MeterProvider - readinessFunc func() error -} -``` - -- [ ] **Step 2: Create `health.go` with liveness and readiness handlers** - -Create `internal/telemetry/metrics/health.go`: - -```go -package metrics - -import ( - "encoding/json" - "net/http" -) - -// handleHealth returns a liveness probe response. -// Always returns 200 OK if the process is running. -func (s *Server) handleHealth( - w http.ResponseWriter, - _ *http.Request, -) { - w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusOK) - _ = json.NewEncoder(w).Encode(map[string]string{ - "status": "ok", - }) -} - -// handleReady returns a readiness probe response. -// Returns 200 when the component is ready, 503 when not. -func (s *Server) handleReady( - w http.ResponseWriter, - _ *http.Request, -) { - w.Header().Set("Content-Type", "application/json") - - if s.readinessFunc == nil { - w.WriteHeader(http.StatusServiceUnavailable) - _ = json.NewEncoder(w).Encode(map[string]string{ - "status": "not_ready", - "error": "readiness check not configured", - }) - return - } - - if err := s.readinessFunc(); err != nil { - w.WriteHeader(http.StatusServiceUnavailable) - _ = json.NewEncoder(w).Encode(map[string]string{ - "status": "not_ready", - "error": err.Error(), - }) - return - } - - w.WriteHeader(http.StatusOK) - _ = json.NewEncoder(w).Encode(map[string]string{ - "status": "ready", - }) -} -``` - -- [ ] **Step 3: Refactor `New()` to support health routes and add - `SetReadinessFunc`** - -Refactor `internal/telemetry/metrics/server.go` so the `Server` is created -before the mux routes are registered (health handlers are methods on `*Server` -and need a receiver). The full refactored `New()`: - -```go -func New( - host string, - port int, - logger *slog.Logger, -) *Server { - reg := prometheus.NewRegistry() - reg.MustRegister(collectors.NewGoCollector()) - reg.MustRegister(collectors.NewProcessCollector( - collectors.ProcessCollectorOpts{}, - )) - - exporter, err := prometheusNewFn( - prometheusExporter.WithRegisterer(reg), - ) - if err != nil { - logger.Error("failed to create prometheus exporter", "error", err) - return nil - } - - mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(exporter)) - - srv := &Server{ - logger: logger, - registry: reg, - meterProvider: mp, - } - - mux := http.NewServeMux() - mux.Handle("/metrics", promhttp.HandlerFor( - reg, - promhttp.HandlerOpts{Registry: reg}, - )) - mux.HandleFunc("/health", srv.handleHealth) - mux.HandleFunc("/health/ready", srv.handleReady) - - srv.httpServer = &http.Server{ - Addr: fmt.Sprintf("%s:%d", host, port), - Handler: mux, - ReadHeaderTimeout: 10 * time.Second, - } - - return srv -} -``` - -Add the setter method: - -```go -// SetReadinessFunc sets the function called by /health/ready and the -// osapi_component_up gauge. The function should return nil when the -// component is ready, or an error describing why it is not. -func (s *Server) SetReadinessFunc( - fn func() error, -) { - s.readinessFunc = fn -} -``` - -- [ ] **Step 4: Write tests for health handlers** - -Create `internal/telemetry/metrics/health_public_test.go` with a -`HealthPublicTestSuite`. Test cases: - -1. `/health` returns 200 with `{"status":"ok"}` -2. `/health/ready` returns 503 when no readiness func set -3. `/health/ready` returns 503 when readiness func returns error -4. `/health/ready` returns 200 when readiness func returns nil -5. Both endpoints return `Content-Type: application/json` - -Use the same `getFreePort()` + `Start()`/`Stop()` pattern as -`server_public_test.go`. - -- [ ] **Step 5: Run tests** - -```bash -go test -count=1 ./internal/telemetry/metrics/... -v -``` - -- [ ] **Step 6: Verify 100% coverage** - -```bash -go test -coverprofile=/tmp/cover.out ./internal/telemetry/metrics/ && \ - go tool cover -func=/tmp/cover.out | grep metrics -``` - -- [ ] **Step 7: Commit** - -```bash -git add internal/telemetry/metrics/ -git commit -m "feat: add /health and /health/ready probes to metrics server" -``` - ---- - -### Task 2: Add `osapi_component_up` gauge - -**Files:** - -- Modify: `internal/telemetry/metrics/server.go` -- Modify: `internal/telemetry/metrics/server_public_test.go` - -- [ ] **Step 1: Register `osapi_component_up` GaugeFunc in `New()`** - -After creating the registry and before creating the mux, register the gauge: - -```go -reg.MustRegister(prometheus.NewGaugeFunc( - prometheus.GaugeOpts{ - Name: "osapi_component_up", - Help: "Whether the component is ready (1) or not (0).", - }, - func() float64 { - if srv.readinessFunc == nil { - return 0 - } - if srv.readinessFunc() != nil { - return 0 - } - return 1 - }, -)) -``` - -Note: the GaugeFunc closure captures `srv` (the `*Server` pointer). Since -`readinessFunc` is set post-construction via `SetReadinessFunc`, the closure -sees the updated value at scrape time. - -- [ ] **Step 2: Add test for `osapi_component_up` in metrics output** - -Add test cases to `TestStartAndStop` in `server_public_test.go`: - -1. When no readiness func set, `/metrics` contains `osapi_component_up 0` -2. After `SetReadinessFunc(func() error { return nil })`, `/metrics` contains - `osapi_component_up 1` -3. After `SetReadinessFunc(func() error { return errors.New("...") })`, - `/metrics` contains `osapi_component_up 0` - -- [ ] **Step 3: Run tests and verify coverage** - -```bash -go test -count=1 -coverprofile=/tmp/cover.out \ - ./internal/telemetry/metrics/... && \ - go tool cover -func=/tmp/cover.out | grep metrics -``` - -- [ ] **Step 4: Commit** - -```bash -git add internal/telemetry/metrics/ -git commit -m "feat: add osapi_component_up gauge to metrics server" -``` - ---- - -### Task 3: Wire readiness checks for all three components - -**Files:** - -- Modify: `cmd/controller_start.go` -- Modify: `cmd/agent_start.go` -- Modify: `cmd/agent_setup.go` -- Modify: `cmd/nats_server_start.go` -- Modify: `cmd/start.go` -- Modify: `internal/agent/agent.go` (add `IsReady()` method) -- Modify: `internal/agent/agent_public_test.go` (test `IsReady()`) - -- [ ] **Step 1: Add `IsReady()` method to agent** - -In `internal/agent/agent.go`: - -```go -// IsReady returns nil when the agent is ready to process jobs, -// or an error describing why it is not. -func (a *Agent) IsReady() error { - if a.ctx == nil || a.ctx.Err() != nil { - return fmt.Errorf("agent not started") - } - return nil -} -``` - -Test in `agent_public_test.go`: call before `Start()` → error, after `Start()` → -nil. - -- [ ] **Step 2: Wire controller readiness** - -The `NATSChecker` is created inside `setupController` as a local variable and -not stored on the bundle. Add a `checker` field to `natsBundle` (or the -equivalent controller bundle struct in `cmd/controller_setup.go`) and store the -`NATSChecker` there so it can be accessed from the startup command. - -Then in `cmd/controller_start.go`, after creating the metrics server: - -```go -if metricsServer != nil { - metricsServer.SetReadinessFunc(func() error { - return b.checker.CheckHealth(context.Background()) - }) -} -``` - -- [ ] **Step 3: Wire agent readiness** - -Note: `setupAgent` returns `(cli.Lifecycle, *natsBundle)`, not `*agent.Agent`. -The `cli.Lifecycle` interface only has `Start()` and `Stop(ctx)`. To call -`IsReady()`, `SetMeterProvider()`, and `LastHeartbeatTime()`, change -`setupAgent` to return `*agent.Agent` directly (it satisfies `cli.Lifecycle` -since it has `Start()` and `Stop(ctx)`). Update the call sites in -`agent_start.go` and `start.go`. - -Then in `cmd/agent_start.go`, after creating the metrics server: - -```go -if metricsServer != nil { - metricsServer.SetReadinessFunc(func() error { - return a.IsReady() - }) -} -``` - -`IsReady()` will return error until `a.Start()` is called. - -- [ ] **Step 4: Wire NATS readiness** - -The `natsembedded.Server` does not expose `JetStreamEnabled()` on its public -interface. Since `setupNATSServer` only returns successfully after JetStream -infrastructure is fully configured, the NATS server is always ready if the -process is running. Use a simple always-ready check: - -```go -if metricsServer != nil { - metricsServer.SetReadinessFunc(func() error { - return nil - }) -} -``` - -If a more meaningful readiness check is needed later (e.g., checking that the -NATS server is still accepting connections), the `natsembedded` package can be -extended with a health method. For now, the `osapi_component_up` gauge will -reflect 1 as long as the process is running, and the heartbeat TTL expiry -handles the "server is down" case in the registry. - -- [ ] **Step 5: Wire readiness in combined start mode** - -In `cmd/start.go`, wire each metrics server's readiness after creation, using -the same patterns as the standalone commands. - -- [ ] **Step 6: Run full test suite** - -```bash -go build ./... && go test -count=1 ./... -``` - -- [ ] **Step 7: Commit** - -```bash -git add cmd/ internal/agent/ -git commit -m "feat: wire readiness checks for all components" -``` - ---- - -## Chunk 2: Application metrics - -### Task 4: Controller HTTP metrics via otelecho - -**Files:** - -- Modify: `internal/controller/api/server.go` -- Modify: `internal/controller/api/types.go` -- Modify: `cmd/controller_start.go` -- Modify: `cmd/controller_setup.go` -- Modify: `cmd/start.go` - -The `otelecho` middleware is already wired in `server.go:54`: - -```go -e.Use(otelecho.Middleware("osapi-api")) -``` - -Currently it uses the global OTEL provider (from tracing). To route metrics to -the metrics server's isolated `MeterProvider`, we need to pass the -`MeterProvider` to the middleware via options. - -- [ ] **Step 1: Add `MeterProvider` option to `Server`** - -In `internal/controller/api/types.go`, add a field and option: - -```go -import sdkmetric "go.opentelemetry.io/otel/sdk/metric" - -// In Server struct: -meterProvider *sdkmetric.MeterProvider - -// Option: -func WithMeterProvider(mp *sdkmetric.MeterProvider) Option { - return func(s *Server) { - s.meterProvider = mp - } -} -``` - -- [ ] **Step 2: Move otelecho middleware after option application** - -In `server.go`, the current `e.Use(otelecho.Middleware("osapi-api"))` at line 54 -runs before the options loop at lines 77-79, so `s.meterProvider` is always nil -when the middleware is registered. - -Fix: move all `e.Use(...)` calls to after the options loop. The order should be: - -1. Create echo instance -2. Apply options (which sets `s.meterProvider`, `s.auditStore`, etc.) -3. Register middleware (otelecho with optional MeterProvider, slogecho, recover, - requestID, CORS, audit) - -```go -// After opts loop: -otelOpts := []otelecho.Option{ - otelecho.WithTracerProvider(otel.GetTracerProvider()), -} -if s.meterProvider != nil { - otelOpts = append(otelOpts, - otelecho.WithMeterProvider(s.meterProvider)) -} -e.Use(otelecho.Middleware("osapi-api", otelOpts...)) -e.Use(slogecho.New(logger)) -e.Use(middleware.Recover()) -e.Use(middleware.RequestID()) -e.Use(middleware.CORSWithConfig(corsConfig)) -``` - -Remove the duplicate `e.Use(middleware.Recover())` that currently exists at -line 59. - -- [ ] **Step 3: Pass `MeterProvider` from cmd** - -In `cmd/controller_start.go` (and the controller setup in `start.go`), pass the -metrics server's `MeterProvider` to `api.New()`: - -```go -if metricsServer != nil { - opts = append(opts, api.WithMeterProvider(metricsServer.MeterProvider())) -} -``` - -Check how `setupController` creates the API server — the options are built in -`controller_setup.go`. Pass the `MeterProvider` through. - -- [ ] **Step 4: Add `osapi_jobs_created_total` counter** - -Job creation happens in `internal/controller/api/job/` handlers that call -`jc.PublishJob()`. The `jobclient.JobClient` is passed into the job handler via -`sm.GetJobHandler(jc)`. To instrument job creation: - -1. Add a `jobsCreated metric.Int64Counter` field to the job handler struct in - `internal/controller/api/job/types.go` -2. Create the counter in `cmd/controller_setup.go` using the metrics server's - `MeterProvider` and pass it to the job handler via a new option (e.g., - `job.WithJobsCreatedCounter(counter)`) -3. In each job handler that calls `PublishJob`, increment the counter after a - successful publish - -If the metrics server is nil (disabled), skip counter creation and the handler -nil-checks the counter before incrementing (same pattern as agent metrics). - -- [ ] **Step 5: Test controller metrics appear in `/metrics` output** - -Write an integration-style test or verify manually that after wiring, hitting -the API server and then scraping `/metrics` on the metrics port shows -`osapi_api_request_duration_seconds` and `osapi_api_requests_total` (these come -from otelecho automatically). - -- [ ] **Step 6: Commit** - -```bash -git add internal/controller/api/ cmd/ -git commit -m "feat: add controller HTTP and job creation metrics" -``` - ---- - -### Task 5: Agent job metrics - -**Files:** - -- Modify: `internal/agent/types.go` (add meter fields) -- Modify: `internal/agent/agent.go` (add `SetMeterProvider()`) -- Modify: `internal/agent/handler.go` (instrument job processing) -- Modify: `internal/agent/heartbeat.go` (track lastHeartbeatTime) -- Modify: `cmd/agent_start.go` (wire MeterProvider + heartbeat gauge) -- Modify: `cmd/agent_setup.go` (wire MeterProvider) -- Modify: `cmd/start.go` (wire in combined mode) - -- [ ] **Step 1: Add OTEL meter fields to Agent** - -In `internal/agent/types.go`, add fields for the OTEL instruments: - -```go -import ( - "go.opentelemetry.io/otel/metric" -) - -// In Agent struct: -jobsProcessed metric.Int64Counter -jobsActive metric.Int64UpDownCounter -jobDuration metric.Float64Histogram -``` - -- [ ] **Step 2: Add `SetMeterProvider()` to Agent** - -In `internal/agent/agent.go`: - -```go -// SetMeterProvider creates OTEL instruments for job metrics. -func (a *Agent) SetMeterProvider( - mp *sdkmetric.MeterProvider, -) { - meter := mp.Meter("osapi-agent") - - a.jobsProcessed, _ = meter.Int64Counter( - "osapi_jobs_processed_total", - metric.WithDescription("Total jobs processed"), - ) - a.jobsActive, _ = meter.Int64UpDownCounter( - "osapi_jobs_active", - metric.WithDescription("Currently executing jobs"), - ) - a.jobDuration, _ = meter.Float64Histogram( - "osapi_job_duration_seconds", - metric.WithDescription("Job execution duration in seconds"), - ) -} -``` - -- [ ] **Step 3: Instrument `handleJobMessage`** - -In `internal/agent/handler.go`, around the job processing: - -At the start of job processing (after "Write started event"): - -```go -if a.jobsActive != nil { - a.jobsActive.Add(ctx, 1) -} -``` - -After processing completes (both success and failure paths): - -```go -if a.jobsActive != nil { - a.jobsActive.Add(ctx, -1) -} -if a.jobDuration != nil { - a.jobDuration.Record(ctx, time.Since(startTime).Seconds()) -} -if a.jobsProcessed != nil { - status := "completed" - if response.Status == job.StatusFailed { - status = "failed" - } - a.jobsProcessed.Add(ctx, 1, - metric.WithAttributes(attribute.String("status", status))) -} -``` - -- [ ] **Step 4: Track `lastHeartbeatTime`** - -In `internal/agent/types.go`, add: - -```go -lastHeartbeatTime atomic.Value // stores time.Time -``` - -In `internal/agent/heartbeat.go`, after successful KV put (line 189): - -```go -a.lastHeartbeatTime.Store(time.Now()) -``` - -In `internal/agent/agent.go`, add accessor: - -```go -// LastHeartbeatTime returns the timestamp of the last successful -// heartbeat write. Returns zero time if no heartbeat has been written. -func (a *Agent) LastHeartbeatTime() time.Time { - if t, ok := a.lastHeartbeatTime.Load().(time.Time); ok { - return t - } - return time.Time{} -} -``` - -- [ ] **Step 5: Wire agent metrics in cmd** - -In `cmd/agent_start.go` (and `cmd/start.go` for combined mode), after creating -the metrics server: - -```go -if metricsServer != nil { - a.SetMeterProvider(metricsServer.MeterProvider()) - - // Register heartbeat age gauge - metricsServer.Registry().MustRegister( - prometheus.NewGaugeFunc( - prometheus.GaugeOpts{ - Name: "osapi_heartbeat_age_seconds", - Help: "Seconds since last successful heartbeat write.", - }, - func() float64 { - t := a.LastHeartbeatTime() - if t.IsZero() { - return 0 - } - return time.Since(t).Seconds() - }, - ), - ) -} -``` - -- [ ] **Step 6: Test agent metrics** - -Add test for `SetMeterProvider` in `agent_public_test.go` — verify it doesn't -panic and instruments are created. - -Add test for `LastHeartbeatTime` — returns zero before heartbeat, non-zero -after. - -- [ ] **Step 7: Run tests and verify coverage** - -```bash -go test -count=1 ./internal/agent/... ./cmd/... -``` - -- [ ] **Step 8: Commit** - -```bash -git add internal/agent/ cmd/ -git commit -m "feat: add agent job and heartbeat metrics" -``` - ---- - -## Chunk 3: Documentation - -### Task 6: Update documentation - -**Files:** - -- Modify: `docs/docs/sidebar/features/metrics.md` -- Modify: `docs/docs/sidebar/features/health-checks.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` - -- [ ] **Step 1: Update metrics.md** - -Rewrite the "What It Exposes" section and add new sections: - -**Health Probes section** (new, after Endpoints): - -Document that each metrics server port also serves `/health` (liveness, -always 200) and `/health/ready` (readiness, 200 or 503). No authentication -required. Useful for Kubernetes probes: - -```yaml -livenessProbe: - httpGet: - path: /health - port: 9091 -readinessProbe: - httpGet: - path: /health/ready - port: 9091 -``` - -**Application Metrics Reference section** (new, replace "What It Exposes"): - -Table of all custom metrics with columns: Metric, Type, Labels, Component, -Description. Include all metrics from the spec: - -- `osapi_component_up` (gauge, all) -- `osapi_api_requests_total` (counter, controller) -- `osapi_api_request_duration_seconds` (histogram, controller) -- `osapi_jobs_created_total` (counter, controller) -- `osapi_jobs_processed_total` (counter, agent) -- `osapi_jobs_active` (gauge, agent) -- `osapi_job_duration_seconds` (histogram, agent) -- `osapi_heartbeat_age_seconds` (gauge, agent) - -Plus note that Go runtime and process metrics are always included. - -- [ ] **Step 2: Update health-checks.md** - -Add a note in the "Endpoints" section or a new "Metrics Server Health Probes" -section explaining that `/health` and `/health/ready` are also available on each -component's metrics port (9090, 9091, 9092) without authentication. -Cross-reference the metrics page. - -- [ ] **Step 3: Update configuration.md** - -In `docs/docs/sidebar/usage/configuration.md`, add a note to each metrics server -section (`controller.metrics`, `agent.metrics`, `nats.server.metrics`) that the -metrics port also serves `/health` and `/health/ready` endpoints for liveness -and readiness probes. - -- [ ] **Step 4: Run prettier** - -```bash -npx prettier docs/docs/sidebar/features/metrics.md --write \ - --config docs/prettier.config.js -npx prettier docs/docs/sidebar/features/health-checks.md --write \ - --config docs/prettier.config.js -npx prettier docs/docs/sidebar/usage/configuration.md --write \ - --config docs/prettier.config.js -``` - -- [ ] **Step 5: Commit** - -```bash -git add docs/ -git commit -m "docs: add health probes and application metrics reference" -``` - ---- - -### Task 7: Verify - -- [ ] **Step 1: Full build and test** - -```bash -go build ./... && go test -count=1 ./... -``` - -- [ ] **Step 2: Lint** - -```bash -just go::vet -``` - -- [ ] **Step 3: Coverage gaps check** - -```bash -just go::unit-cov-gaps -``` - -Expect only the pre-existing defense-in-depth gaps (docker, file, job). - -- [ ] **Step 4: Format check** - -```bash -just go::fmt-check && just docs::fmt-check -``` diff --git a/docs/plans/2026-03-25-registration-pattern-design.md b/docs/plans/2026-03-25-registration-pattern-design.md deleted file mode 100644 index 0d0a391a8..000000000 --- a/docs/plans/2026-03-25-registration-pattern-design.md +++ /dev/null @@ -1,297 +0,0 @@ -# Registration Pattern Design - -## Goal - -Eliminate centralized lists that grow with each new component. Adding a new -provider, operation, or infrastructure bucket should require changing 1-2 files, -not 5-10. - -## Problem - -Adding a new provider today touches: - -1. `agent/types.go` — add field -2. `agent/agent.go` — add parameter + WireProviderFacts entry -3. `agent/factory.go` — add return value + creation logic -4. `agent/processor.go` — add switch case -5. `cmd/agent_setup.go` — unpack tuple + pass to New() -6. `job/client/types.go` — add 2-4 interface methods -7. `job/client/*.go` — implement methods -8. Regenerate mocks - -The JobClient interface has 60+ methods that are thin wrappers around 2 internal -functions. KV bucket creation manually lists every bucket name even though -they're all in osapi.yaml. - -## Design - -### 1. Agent Provider Registry - -Replace individual provider fields, parameters, and switch dispatch with a -registry that providers register into at construction time. - -**Registry type:** - -```go -// internal/agent/registry.go -type ProcessorFunc func(job.Request) (json.RawMessage, error) - -type ProviderRegistry struct { - processors map[string]ProcessorFunc - providers []any // for WireProviderFacts -} - -func NewProviderRegistry() *ProviderRegistry { - return &ProviderRegistry{ - processors: make(map[string]ProcessorFunc), - } -} - -func (r *ProviderRegistry) Register( - category string, - provider any, - processFn ProcessorFunc, -) { - r.processors[category] = processFn - r.providers = append(r.providers, provider) -} - -func (r *ProviderRegistry) Dispatch( - req job.Request, -) (json.RawMessage, error) { - fn, ok := r.processors[req.Category] - if !ok { - return nil, fmt.Errorf("unsupported category: %s", req.Category) - } - return fn(req) -} - -func (r *ProviderRegistry) AllProviders() []any { - return r.providers -} -``` - -**Processor functions become standalone closures**, not Agent methods. They -capture their provider dependency: - -```go -// internal/agent/processor_schedule.go -func NewScheduleProcessor( - cronProvider cron.Provider, - logger *slog.Logger, -) ProcessorFunc { - return func(req job.Request) (json.RawMessage, error) { - // dispatch cron sub-operations using cronProvider - } -} -``` - -**Agent construction simplifies:** - -```go -func New( - appFs avfs.VFS, - appConfig config.Config, - logger *slog.Logger, - jobClient jobclient.JobClient, - streamName string, - registry *ProviderRegistry, - processProvider process.Provider, - registryKV jetstream.KeyValue, - factsKV jetstream.KeyValue, -) *Agent { - a := &Agent{...} - provider.WireProviderFacts(a.GetFacts, registry.AllProviders()...) - return a -} -``` - -**Setup wiring (one file, one place per provider):** - -```go -// cmd/agent_setup.go -registry := agent.NewProviderRegistry() - -// Node providers -hostProvider := host.NewDebianProvider() -diskProvider := disk.NewDebianProvider(log) -// ... -registry.Register("node", agent.NewNodeProcessor( - hostProvider, diskProvider, memProvider, loadProvider, -), hostProvider, diskProvider, memProvider, loadProvider) - -// Docker -dockerProvider := docker.New() -registry.Register("docker", agent.NewDockerProcessor( - dockerProvider, log, -), dockerProvider) - -// Cron -cronProvider := cron.NewDebianProvider(...) -registry.Register("schedule", agent.NewScheduleProcessor( - cronProvider, log, -), cronProvider) - -agent.New(..., registry, ...) -``` - -Wait — `Register` takes one provider but some categories have multiple (node has -host, disk, mem, load). The registry needs to accept multiple providers for -FactsAware wiring: - -```go -func (r *ProviderRegistry) Register( - category string, - processFn ProcessorFunc, - providers ...any, -) { - r.processors[category] = processFn - r.providers = append(r.providers, providers...) -} -``` - -**What this eliminates:** - -- `agent/types.go` — no more provider fields (registry holds them) -- `agent/agent.go` — no more 15-parameter constructor -- `agent/factory.go` — deleted (providers created in setup) -- `agent/processor.go` switch — replaced by registry.Dispatch() -- Each `processor_*.go` — methods on Agent → standalone functions - -**What remains:** - -- `cmd/agent_setup.go` — still creates providers and registers them (this is the - ONE place you add a new provider) -- Each `processor_*.go` — still exists as a standalone function - -### 2. JobClient Interface Simplification - -Replace 60+ typed methods with 4 generic ones: - -```go -type JobClient interface { - Query( - ctx context.Context, - target string, - category string, - operation string, - data any, - ) (*job.Response, error) - - QueryBroadcast( - ctx context.Context, - target string, - category string, - operation string, - data any, - ) (string, map[string]*job.Response, map[string]string, error) - - Modify( - ctx context.Context, - target string, - category string, - operation string, - data any, - ) (*job.Response, error) - - ModifyBroadcast( - ctx context.Context, - target string, - category string, - operation string, - data any, - ) (string, map[string]*job.Response, map[string]string, error) -} -``` - -**API handlers change from:** - -```go -resp, err := s.JobClient.ModifyDockerCreate(ctx, hostname, data) -``` - -**To:** - -```go -resp, err := s.JobClient.Modify( - ctx, hostname, "docker", job.OperationDockerCreate, data) -``` - -The operation constants remain typed — `job.OperationDockerCreate` is still a -constant string. Typos are caught by tests, not the compiler. - -**What this eliminates:** - -- `job/client/types.go` — 60 methods → 4 -- `job/client/modify_docker.go`, `query_node.go`, etc. — deleted (the generic - methods handle all operations) -- Mock regeneration — only 4 methods to mock, never changes -- `job/client/schedule_cron.go`, `modify_command.go`, etc. — deleted - -**What remains:** - -- `job/client/client.go` — implements the 4 generic methods -- Operation constants — still needed for the string arguments - -### 3. Config-Driven Infrastructure - -Add methods to the config struct that iterate infrastructure: - -```go -// internal/config/nats.go -func (n NATSConfig) AllKVBucketConfigs() []KVBucketConfig { - return []KVBucketConfig{ - n.KV, n.Audit, n.Registry, n.Facts, n.State, n.FileState, - } -} - -func (n NATSConfig) AllObjectStoreConfigs() []ObjectStoreConfig { - var configs []ObjectStoreConfig - if n.Objects.Bucket != "" { - configs = append(configs, n.Objects) - } - return configs -} -``` - -Then `controller_setup.go` iterates: - -```go -for _, cfg := range appConfig.NATS.AllKVBucketConfigs() { - // create or update bucket -} -``` - -**What this eliminates:** - -- Manual `add(appConfig.NATS.Xxx.Bucket)` calls in setup -- Forgetting to add new buckets to the creation list - -**What remains:** - -- The config struct still has named fields (needed for typed access) -- `AllKVBucketConfigs()` method needs updating when new buckets are added (but - it's ONE place, not scattered across setup code) - -## Scope - -| Change | Files eliminated | Files simplified | New files | -| ------------------------ | --------------------- | ------------------------------------------ | ----------- | -| Provider registry | factory.go deleted | agent.go, types.go, processor.go, setup.go | registry.go | -| JobClient simplification | ~8 typed method files | types.go (60→4 methods), all handler files | None | -| Config iteration | None | controller_setup.go | None | - -## What This Does NOT Change - -- SDK typed methods — consumers still call `c.Docker.Create()` -- OpenAPI specs — response schemas unchanged -- CLI commands — unchanged -- Operation/permission constants — still explicit declarations -- Provider implementations — unchanged (they don't know about the registry) - -## Migration Order - -1. **Provider registry** first — biggest win, self-contained -2. **JobClient simplification** second — touches many handler files but is - mechanical (replace method name with generic call) -3. **Config iteration** third — smallest, independent diff --git a/docs/plans/2026-03-25-registration-pattern.md b/docs/plans/2026-03-25-registration-pattern.md deleted file mode 100644 index 2fd6a40aa..000000000 --- a/docs/plans/2026-03-25-registration-pattern.md +++ /dev/null @@ -1,489 +0,0 @@ -# Registration Pattern Implementation Plan - -> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development -> (if subagents available) or superpowers:executing-plans to implement this -> plan. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Eliminate centralized lists that grow with each new component, -replacing them with a provider registry, a simplified 4-method JobClient -interface, and config-driven infrastructure iteration. - -**Architecture:** Providers register themselves with a registry at construction -time. The agent dispatches via registry map lookup instead of a switch -statement. The JobClient shrinks from 60+ typed wrapper methods to 4 generic -methods. Config structs expose iteration methods for KV buckets. - -**Tech Stack:** Go 1.25, testify/suite, gomock - ---- - -## Chunk 1: Provider Registry - -### Task 1: Create the ProviderRegistry type - -**Files:** - -- Create: `internal/agent/registry.go` -- Create: `internal/agent/registry_public_test.go` - -- [ ] **Step 1: Create registry.go** - -```go -package agent - -import ( - "encoding/json" - "fmt" - - "github.com/osapi-io/osapi/internal/job" -) - -// ProcessorFunc handles job requests for a category. -type ProcessorFunc func(job.Request) (json.RawMessage, error) - -// ProviderRegistry holds registered providers and their processors. -type ProviderRegistry struct { - processors map[string]ProcessorFunc - providers []any -} - -// NewProviderRegistry creates an empty registry. -func NewProviderRegistry() *ProviderRegistry { - return &ProviderRegistry{ - processors: make(map[string]ProcessorFunc), - } -} - -// Register adds a category processor and its providers to the registry. -// Providers are collected for FactsAware wiring. -func (r *ProviderRegistry) Register( - category string, - processFn ProcessorFunc, - providers ...any, -) { - r.processors[category] = processFn - r.providers = append(r.providers, providers...) -} - -// Dispatch routes a job request to the registered processor for its -// category. Returns an error if the category is not registered. -func (r *ProviderRegistry) Dispatch( - req job.Request, -) (json.RawMessage, error) { - fn, ok := r.processors[req.Category] - if !ok { - return nil, fmt.Errorf( - "unsupported job category: %s", req.Category) - } - return fn(req) -} - -// AllProviders returns all registered providers for FactsAware wiring. -func (r *ProviderRegistry) AllProviders() []any { - return r.providers -} -``` - -- [ ] **Step 2: Write tests** - -Test Register, Dispatch (success + unknown category), AllProviders. - -- [ ] **Step 3: Verify** - -Run: `go test ./internal/agent/... -count=1` - -- [ ] **Step 4: Commit** - -```bash -git commit -m "feat: add ProviderRegistry for category-based dispatch" -``` - -### Task 2: Convert processor methods to standalone functions - -Each `processor_*.go` file has methods on `*Agent` that access `a.xxxProvider`. -Convert them to factory functions that return a `ProcessorFunc` closure -capturing the provider. - -**Files:** - -- Modify: `internal/agent/processor.go` -- Modify: `internal/agent/processor_schedule.go` -- Modify: `internal/agent/processor_docker.go` -- Modify: `internal/agent/processor_file.go` -- Modify: `internal/agent/processor_command.go` - -For each processor file, the pattern changes from: - -```go -// Before: method on Agent -func (a *Agent) processScheduleOperation( - jobRequest job.Request, -) (json.RawMessage, error) { - if a.cronProvider == nil { ... } - // uses a.cronProvider -} -``` - -To: - -```go -// After: factory returning ProcessorFunc closure -func NewScheduleProcessor( - cronProvider cron.Provider, - logger *slog.Logger, -) ProcessorFunc { - return func(req job.Request) (json.RawMessage, error) { - if cronProvider == nil { ... } - // uses cronProvider directly (captured) - } -} -``` - -The sub-dispatch functions (processCronList, processCronGet, etc.) become local -functions or closures within the processor. - -Do this for all 6 processor files: - -- `processor.go` — `processNodeOperation` → `NewNodeProcessor` (captures host, - disk, mem, load, netinfo providers) -- `processor_schedule.go` — → `NewScheduleProcessor` (captures cron) -- `processor_docker.go` — → `NewDockerProcessor` (captures docker) -- `processor_file.go` — → `NewFileProcessor` (captures file) -- `processor_command.go` — → `NewCommandProcessor` (captures command) -- Network operations in processor.go — → `NewNetworkProcessor` (captures dns, - ping) - -- [ ] **Step 1: Convert processor_schedule.go** - -- [ ] **Step 2: Convert processor_docker.go** - -- [ ] **Step 3: Convert processor_file.go** - -- [ ] **Step 4: Convert processor_command.go** - -- [ ] **Step 5: Split processor.go into NewNodeProcessor and - NewNetworkProcessor** - -- [ ] **Step 6: Remove the switch in processJobOperation — replace with - registry.Dispatch()** - -- [ ] **Step 7: Update tests for all processor files** - -The existing tests mock the agent and call processor methods. They need to call -the standalone factory functions instead. - -- [ ] **Step 8: Verify all tests pass** - -Run: `go test ./internal/agent/... -count=1` - -- [ ] **Step 9: Commit** - -### Task 3: Simplify Agent struct and constructor - -**Files:** - -- Modify: `internal/agent/types.go` -- Modify: `internal/agent/agent.go` -- Delete: `internal/agent/factory.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Remove provider fields from Agent struct** - -Replace 12 provider fields with one `registry *ProviderRegistry`. Keep -`processProvider` (it's telemetry, not a job processor). - -- [ ] **Step 2: Simplify agent.New()** - -From 17 parameters to ~8: - -```go -func New( - appFs avfs.VFS, - appConfig config.Config, - logger *slog.Logger, - jobClient jobclient.JobClient, - streamName string, - registry *ProviderRegistry, - processProvider process.Provider, - registryKV jetstream.KeyValue, - factsKV jetstream.KeyValue, -) *Agent -``` - -WireProviderFacts uses `registry.AllProviders()`. - -- [ ] **Step 3: Delete factory.go** - -The factory returned a tuple of providers. Now providers are created directly in -`agent_setup.go` and registered individually. - -- [ ] **Step 4: Update agent_setup.go** - -```go -registry := agent.NewProviderRegistry() - -// Create and register each provider category -hostProv := host.NewDebianProvider() -diskProv := disk.NewDebianProvider(log) -memProv := mem.NewDebianProvider() -loadProv := load.NewDebianProvider() -registry.Register("node", - agent.NewNodeProcessor(hostProv, diskProv, memProv, loadProv, log), - hostProv, diskProv, memProv, loadProv) - -dnsProv := dns.NewDebianProvider(log, execManager) -pingProv := ping.NewDebianProvider() -registry.Register("network", - agent.NewNetworkProcessor(dnsProv, pingProv, log), - dnsProv, pingProv) - -commandProv := command.New(log, execManager) -registry.Register("command", - agent.NewCommandProcessor(commandProv, log), - commandProv) - -// ... file, docker, cron same pattern - -a := agent.New(appFs, appConfig, log, jobClient, streamName, - registry, process.New(), registryKV, factsKV) -``` - -- [ ] **Step 5: Update all agent test files** - -Tests that create agents with mocked providers need to use the registry pattern -instead of passing individual providers. - -- [ ] **Step 6: Verify everything** - -Run: `go build ./... && go test ./... -count=1 -short` - -- [ ] **Step 7: Commit** - ---- - -## Chunk 2: JobClient Interface Simplification - -### Task 4: Add generic methods to JobClient - -**Files:** - -- Modify: `internal/job/client/types.go` -- Modify: `internal/job/client/client.go` - -- [ ] **Step 1: Add 4 generic methods to interface** - -```go -// Generic job dispatch methods. All typed wrapper methods delegate -// to these. New operations use these directly — no need to add -// methods to the interface. -Query( - ctx context.Context, - target string, - category string, - operation string, - data any, -) (string, *job.Response, error) - -QueryBroadcast( - ctx context.Context, - target string, - category string, - operation string, - data any, -) (string, map[string]*job.Response, map[string]string, error) - -Modify( - ctx context.Context, - target string, - category string, - operation string, - data any, -) (string, *job.Response, error) - -ModifyBroadcast( - ctx context.Context, - target string, - category string, - operation string, - data any, -) (string, map[string]*job.Response, map[string]string, error) -``` - -- [ ] **Step 2: Implement in client.go** - -Each method: marshal data → build job.Request → call -publishAndWait/publishAndCollect → process results/errors → return. - -- [ ] **Step 3: Write tests for generic methods** - -- [ ] **Step 4: Commit** - -### Task 5: Migrate API handlers to generic JobClient methods - -**Files:** - -- Modify: all handler files in `internal/controller/api/node/` -- Modify: all handler files in `internal/controller/api/docker/` -- Modify: all handler files in `internal/controller/api/schedule/` - -For each handler, change from: - -```go -resp, err := s.JobClient.ModifyDockerCreate(ctx, hostname, data) -``` - -To: - -```go -_, resp, err := s.JobClient.Modify( - ctx, hostname, "docker", job.OperationDockerCreate, data) -``` - -And for broadcast: - -```go -jobID, results, errs, err := s.JobClient.ModifyBroadcast( - ctx, target, "docker", job.OperationDockerCreate, data) -``` - -This is mechanical — same transformation for every handler. - -- [ ] **Step 1: Migrate node handlers (7 files)** -- [ ] **Step 2: Migrate docker handlers (9 files)** -- [ ] **Step 3: Migrate schedule handlers (5 files)** -- [ ] **Step 4: Migrate file handlers (3 files)** -- [ ] **Step 5: Migrate command handlers (2 files)** -- [ ] **Step 6: Migrate network handlers (3 files)** -- [ ] **Step 7: Update all handler tests** -- [ ] **Step 8: Verify: `go test ./internal/controller/api/... -count=1`** -- [ ] **Step 9: Commit** - -### Task 6: Remove typed wrapper methods - -**Files:** - -- Delete: `internal/job/client/query.go` -- Delete: `internal/job/client/query_node.go` -- Delete: `internal/job/client/modify.go` -- Delete: `internal/job/client/modify_command.go` -- Delete: `internal/job/client/modify_docker.go` -- Delete: `internal/job/client/schedule_cron.go` -- Delete: `internal/job/client/file.go` -- Modify: `internal/job/client/types.go` — remove old method signatures -- Regenerate: `internal/job/mocks/job_client.gen.go` - -- [ ] **Step 1: Remove typed methods from interface** -- [ ] **Step 2: Delete implementation files** -- [ ] **Step 3: Delete typed method test files** -- [ ] **Step 4: Regenerate mocks** -- [ ] **Step 5: Verify: `go build ./... && go test ./... -count=1 -short`** -- [ ] **Step 6: Commit** - ---- - -## Chunk 3: Config-Driven Infrastructure - -### Task 7: Add config iteration methods - -**Files:** - -- Modify: `internal/config/types.go` -- Create: `internal/config/nats.go` (or add to existing) -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Add AllKVBucketConfigs method** - -Read the NATS config struct first. Add a method that returns all KV bucket -configurations: - -```go -func (n NATSConfig) AllKVConfigs() []KVConfig { - return []KVConfig{ - {Name: "kv", Bucket: n.KV.Bucket, TTL: n.KV.TTL, ...}, - {Name: "response", Bucket: n.KV.ResponseBucket, ...}, - {Name: "audit", Bucket: n.Audit.Bucket, ...}, - {Name: "registry", Bucket: n.Registry.Bucket, ...}, - {Name: "facts", Bucket: n.Facts.Bucket, ...}, - {Name: "state", Bucket: n.State.Bucket, ...}, - {Name: "file_state", Bucket: n.FileState.Bucket, ...}, - } -} -``` - -- [ ] **Step 2: Update controller_setup.go to iterate** - -Replace manual `add()` calls with loop: - -```go -for _, cfg := range appConfig.NATS.AllKVConfigs() { - if cfg.Bucket != "" { - add(cfg.Bucket) - } -} -``` - -- [ ] **Step 3: Write tests** -- [ ] **Step 4: Verify** -- [ ] **Step 5: Commit** - -### Task 8: Update CLAUDE.md - -**Files:** - -- Modify: `CLAUDE.md` - -- [ ] **Step 1: Update provider guide** - -Update Step 0 to describe the registry pattern instead of manual field/parameter -wiring. Document: - -- How to create a provider and register it -- How to write a NewXxxProcessor factory function -- That adding a provider is ONE change in agent_setup.go -- That JobClient is generic — no new methods needed - -- [ ] **Step 2: Commit** - ---- - -## Chunk 4: Verification - -### Task 9: Full verification - -- [ ] **Step 1:** `go build ./...` -- [ ] **Step 2:** `just go::unit` -- [ ] **Step 3:** `just go::vet` -- [ ] **Step 4:** Verify adding a hypothetical provider requires only - agent_setup.go + processor file + provider package - ---- - -## Files Summary - -| Action | File | -| -------- | --------------------------------------------------------------- | -| Create | `internal/agent/registry.go` | -| Create | `internal/agent/registry_public_test.go` | -| Rewrite | `internal/agent/processor.go` → uses registry.Dispatch | -| Rewrite | `internal/agent/processor_schedule.go` → NewScheduleProcessor | -| Rewrite | `internal/agent/processor_docker.go` → NewDockerProcessor | -| Rewrite | `internal/agent/processor_file.go` → NewFileProcessor | -| Rewrite | `internal/agent/processor_command.go` → NewCommandProcessor | -| Create | `internal/agent/processor_network.go` (split from processor.go) | -| Simplify | `internal/agent/types.go` — 12 fields → 1 registry | -| Simplify | `internal/agent/agent.go` — 17 params → 8 | -| Delete | `internal/agent/factory.go` | -| Rewrite | `cmd/agent_setup.go` — uses registry | -| Add | `internal/job/client/client.go` — 4 generic methods | -| Simplify | `internal/job/client/types.go` — 60→4 interface methods | -| Delete | `internal/job/client/query.go` | -| Delete | `internal/job/client/query_node.go` | -| Delete | `internal/job/client/modify.go` | -| Delete | `internal/job/client/modify_command.go` | -| Delete | `internal/job/client/modify_docker.go` | -| Delete | `internal/job/client/schedule_cron.go` | -| Delete | `internal/job/client/file.go` | -| Modify | All handler files (~29) — use generic JobClient | -| Add | `internal/config/` — AllKVConfigs method | -| Modify | `cmd/controller_setup.go` — iterate config | -| Update | `CLAUDE.md` | diff --git a/docs/plans/2026-03-25-unified-broadcast-responses-design.md b/docs/plans/2026-03-25-unified-broadcast-responses-design.md deleted file mode 100644 index c2bf123a6..000000000 --- a/docs/plans/2026-03-25-unified-broadcast-responses-design.md +++ /dev/null @@ -1,197 +0,0 @@ -# Unified Broadcast Response Design - -## Goal - -Standardize all node-targeted API responses so every operation supports -broadcast (`_all`, label selectors) and returns a uniform collection response. -Every result item carries `hostname` and `error` fields. Single-target and -broadcast operations return the same response shape. Update CLAUDE.md so future -providers follow the pattern from day one. - -## Problem - -18 of 29 node-targeted operations lack broadcast support. Docker (9 ops), File -(3 ops), Cron mutations (3 ops), and Cron get have no broadcast path — they -silently route `_all` through the single-target path and return one random -agent's response. Users targeting a fleet see results from one host with no -indication others were skipped. - -Response types are also inconsistent: some return collections, some return flat -objects. Some have `hostname` on result items, some don't. - -## Design - -### Uniform Response Shape - -Every node-targeted operation returns: - -```json -{ - "job_id": "550e8400-...", - "results": [ - { - "hostname": "web-01", - "error": "", - ...domain-specific fields... - }, - { - "hostname": "web-02", - "error": "operation not supported on this OS family" - } - ] -} -``` - -- Single-target (`_any`, hostname): collection with 1 result -- Broadcast (`_all`, label): collection with N results -- Failed/skipped agents: result entry with `hostname` + `error`, empty domain - fields - -### Handler Pattern - -Every handler follows: - -```go -func (s *Handler) PostOperation(ctx, request) { - validate(request) - hostname := request.Hostname - - if job.IsBroadcastTarget(hostname) { - return s.postOperationBroadcast(ctx, hostname, ...) - } - - // Single target path. - resp, err := s.JobClient.SingleMethod(ctx, hostname, ...) - // Wrap in collection with 1 result. - return collectionResponse(resp.JobID, []ResultItem{ - {Hostname: resp.Hostname, ...fields...}, - }) -} - -func (s *Handler) postOperationBroadcast(ctx, target, ...) { - jobID, results, errs, err := s.JobClient.BroadcastMethod(...) - - var items []ResultItem - for _, r := range results { - items = append(items, ResultItem{Hostname: r.Hostname, ...}) - } - for host, errMsg := range errs { - items = append(items, ResultItem{Hostname: host, Error: errMsg}) - } - - return collectionResponse(jobID, items) -} -``` - -### Changes Per Domain - -#### Docker (9 operations) - -Response types already have `hostname` and `error` on result items. Already -return collections. Need: - -1. Job client: 9 `*Broadcast` methods -2. Handlers: 9 `IsBroadcastTarget` checks + broadcast functions -3. No schema changes — response shapes are correct - -Operations: create, list, inspect, start, stop, remove, exec, pull, -image-remove. - -#### File (3 operations: deploy, undeploy, status) - -Deploy and undeploy responses are flat (not collections). Need: - -1. Convert `FileDeployResponse` and `FileUndeployResponse` to collection pattern - with `job_id` + `results[]` -2. Add `error` field to deploy/undeploy result items -3. Job client: 3 `*Broadcast` methods -4. Handlers: 3 `IsBroadcastTarget` checks + broadcast functions -5. Update SDK types and CLI output - -File status already has a collection-compatible shape. - -#### Cron (4 operations: get, create, update, delete) - -Cron list already has broadcast. The other 4 need it: - -1. Add `hostname` to `CronCreateResponse`, `CronUpdateResponse`, - `CronDeleteResponse` -2. Convert `CronEntryResponse` (get) to collection pattern -3. Convert mutation responses to collection pattern -4. Job client: 4 `*Broadcast` methods -5. Handlers: 4 `IsBroadcastTarget` checks + broadcast functions -6. Update SDK types and CLI output - -### Job Client Broadcast Pattern - -Every broadcast method follows the same pattern. For query operations: - -```go -func (c *Client) QueryDockerListBroadcast( - ctx context.Context, - target string, - ..., -) (string, map[string]*DockerListResult, map[string]string, error) { - // Build request, call publishAndCollect, process responses. -} -``` - -For modify operations: - -```go -func (c *Client) ModifyDockerCreateBroadcast( - ctx context.Context, - target string, - ..., -) (string, map[string]*DockerResult, map[string]string, error) { - // Build request, call publishAndCollect, process responses. -} -``` - -Return signature: `(jobID, resultsByHost, errorsByHost, error)`. Matches the -pattern used by existing broadcast methods like `QueryNodeHostnameBroadcast`. - -### SDK Changes - -No new types needed for Docker (already have hostname + error). - -For File and Cron mutations, add `Hostname` field to existing SDK result types -where missing. The SDK `Collection[T]` type already handles the `job_id` + -`results[]` envelope. - -### CLI Changes - -All CLI commands should use `BuildBroadcastTable` for collection responses. -Commands that currently use `PrintKV` for single results (cron -create/update/delete, file deploy/undeploy) switch to table output showing -HOSTNAME + STATUS + domain fields. - -### CLAUDE.md Update - -Add to "Adding a New API Domain" section: - -> **Broadcast support (MANDATORY):** Every operation that targets a node -> (`/node/{hostname}/...`) MUST support broadcast. The handler checks -> `job.IsBroadcastTarget(hostname)` and routes to a broadcast function. The job -> client has both a single-target and `*Broadcast` method for each operation. -> All responses use a collection envelope with `job_id` + `results[]`. Every -> result item includes `hostname` and `error` fields. Single-target returns 1 -> result in the collection; broadcast returns N results. - -### Scope - -| Domain | Operations | Schema changes | New broadcast methods | -| ------ | ---------- | ------------------------------------- | --------------------- | -| Docker | 9 | none | 9 | -| File | 3 | deploy+undeploy → collection | 3 | -| Cron | 4 | get/create/update/delete → collection | 4 | -| Total | 16 | 5 schemas | 16 | - -### What This Does NOT Change - -- Node query operations (7) — already have full broadcast -- Command exec/shell (2) — already have full broadcast -- Network DNS/ping (3) — already have full broadcast -- Cron list — already has broadcast -- File upload/list/get/delete — not node-targeted (Object Store ops) -- Health, Audit, Job, Agent endpoints — not node-targeted diff --git a/docs/plans/2026-03-27-container-dns-provider-design.md b/docs/plans/2026-03-27-container-dns-provider-design.md deleted file mode 100644 index 24f099f74..000000000 --- a/docs/plans/2026-03-27-container-dns-provider-design.md +++ /dev/null @@ -1,120 +0,0 @@ -# Container DNS Provider + Container Detection - -## Problem - -When the OSAPI agent runs inside a Docker container on a Debian-based image, the -DNS provider fails because `resolvectl` is not available. Containers use -`/etc/resolv.conf` directly — there is no systemd-resolved. DNS writes are -managed by the container runtime (Docker, Kubernetes), not the agent. - -## Design - -Two changes: a new `DebianDocker` DNS provider that reads `/etc/resolv.conf` -instead of calling `resolvectl`, and a `containerized` built-in fact so -providers and consumers can detect container environments. - -### 1. Container Detection — `platform.IsContainer()` - -**File:** `pkg/sdk/platform/container.go` - -Add an `IsContainer() bool` function that checks for `/.dockerenv` file -existence. Use an injectable function variable (`ContainerCheckFn`) following -the existing `HostInfoFn` pattern for testability. - -**File:** `pkg/sdk/platform/container_public_test.go` - -Test both container and non-container paths by overriding `ContainerCheckFn`. - -### 2. DebianDocker DNS Provider - -Three new files in `internal/provider/network/dns/`: - -**`debian_docker.go`** — Provider struct and constructor. - -```go -type DebianDocker struct { - provider.FactsAware - logger *slog.Logger - fs avfs.VFS -} - -func NewDebianDockerProvider( - logger *slog.Logger, - fs avfs.VFS, -) *DebianDocker -``` - -No exec manager — this provider only reads files via avfs. - -Compile-time check: `var _ Provider = (*DebianDocker)(nil)` - -**`debian_docker_get_resolv_conf_by_interface.go`** — Get implementation. - -`GetResolvConfByInterface` reads `/etc/resolv.conf` via avfs and parses -`nameserver` and `search` lines. The `interfaceName` parameter is accepted but -ignored — containers have a single global DNS configuration. - -Returns `GetResult` with `DNSServers` and `SearchDomains` populated from the -file contents. Returns `["."]` for search domains if none are found (matching -the Debian provider convention). - -**`debian_docker_update_resolv_conf_by_interface.go`** — Update implementation. - -`UpdateResolvConfByInterface` returns `provider.ErrUnsupported`. DNS in -containers is managed by the container runtime, not the agent. - -### 3. Containerized Built-in Fact - -**`internal/facts/keys.go`** — Add `KeyContainerized = "containerized"` constant -with description "Whether the agent is running inside a container". - -**`internal/job/types.go`** — Add `Containerized bool` field to -`FactsRegistration`. - -**`internal/agent/facts.go`** — Call `platform.IsContainer()` during facts -collection and set `Containerized` on the registration. - -**`internal/agent/factref.go`** — Add resolver case for `@fact.containerized` -that returns the boolean value from `FactsRegistration.Containerized`. - -### 4. Agent Setup Wiring - -**`cmd/agent_setup.go`** — Update the DNS provider switch: - -```go -case "debian": - if platform.IsContainer() { - dnsProvider = dns.NewDebianDockerProvider(log, appFs) - } else { - dnsProvider = dns.NewDebianProvider(log, execManager) - } -``` - -All other providers remain unchanged — host, disk, mem, load, ping all work -inside containers already since they read `/proc` directly. - -### 5. Test Files - -Each new production file gets a matching `*_public_test.go`: - -- `pkg/sdk/platform/container_public_test.go` -- `internal/provider/network/dns/debian_docker_public_test.go` -- `internal/provider/network/dns/debian_docker_get_resolv_conf_by_interface_public_test.go` -- `internal/provider/network/dns/debian_docker_update_resolv_conf_by_interface_public_test.go` - -All tests use testify/suite with table-driven patterns. DNS tests use -`memfs.New()` for filesystem mocking. - -Facts-related changes are covered by updating existing test files: - -- `internal/facts/keys_public_test.go` — add `containerized` to key list -- `internal/agent/facts_public_test.go` — verify `Containerized` is set -- `internal/agent/factref_public_test.go` — test `@fact.containerized` - resolution - -## Out of Scope - -- Non-Debian container images (no `redhat_docker` etc. until needed) -- Container detection for podman, LXC, or other runtimes (Docker only for now) -- Changes to other providers (host, disk, mem, load work in containers already) -- SDK or CLI changes (the fact surfaces automatically through existing paths) diff --git a/docs/plans/2026-03-27-container-dns-provider.md b/docs/plans/2026-03-27-container-dns-provider.md deleted file mode 100644 index fbf4b3455..000000000 --- a/docs/plans/2026-03-27-container-dns-provider.md +++ /dev/null @@ -1,871 +0,0 @@ -# Container DNS Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add container detection to the platform package and a `DebianDocker` -DNS provider that reads `/etc/resolv.conf` for containerized agents. - -**Architecture:** A new `platform.IsContainer()` function detects Docker -containers via `/.dockerenv`. The `DebianDocker` DNS provider uses `avfs.VFS` to -parse `/etc/resolv.conf` for Get (ignoring the interface parameter) and returns -`ErrUnsupported` for Update. A new `containerized` built-in fact key exposes the -container state. Agent setup wires the container check into DNS provider -selection. - -**Tech Stack:** Go, avfs (memfs for tests), testify/suite, gopsutil - -**Spec:** `docs/superpowers/specs/2026-03-27-container-dns-provider-design.md` - ---- - -### Task 1: Container Detection — `platform.IsContainer()` - -**Files:** - -- Create: `pkg/sdk/platform/container.go` -- Create: `pkg/sdk/platform/container_public_test.go` - -- [ ] **Step 1: Write the failing test** - -Create `pkg/sdk/platform/container_public_test.go`: - -```go -package platform_test - -import ( - "testing" - - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/pkg/sdk/platform" -) - -type ContainerPublicTestSuite struct { - suite.Suite -} - -func (s *ContainerPublicTestSuite) TearDownSubTest() { - platform.ContainerCheckFn = platform.DefaultContainerCheck -} - -func (s *ContainerPublicTestSuite) TestIsContainer() { - tests := []struct { - name string - checkFn func() bool - want bool - }{ - { - name: "when inside a Docker container", - checkFn: func() bool { - return true - }, - want: true, - }, - { - name: "when not inside a container", - checkFn: func() bool { - return false - }, - want: false, - }, - } - - for _, tc := range tests { - s.Run(tc.name, func() { - platform.ContainerCheckFn = tc.checkFn - - got := platform.IsContainer() - - s.Equal(tc.want, got) - }) - } -} - -func TestContainerPublicTestSuite(t *testing.T) { - suite.Run(t, new(ContainerPublicTestSuite)) -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `go test -run TestContainerPublicTestSuite -v ./pkg/sdk/platform/...` -Expected: FAIL — `ContainerCheckFn`, `DefaultContainerCheck`, `IsContainer` not -defined. - -- [ ] **Step 3: Write minimal implementation** - -Create `pkg/sdk/platform/container.go`: - -```go -package platform - -import "os" - -// DefaultContainerCheck checks for the presence of /.dockerenv -// to determine if the process is running inside a Docker container. -func DefaultContainerCheck() bool { - _, err := os.Stat("/.dockerenv") - return err == nil -} - -// ContainerCheckFn is the function used to detect container environments. -// Override in tests to simulate different environments. -var ContainerCheckFn = DefaultContainerCheck - -// IsContainer reports whether the current process is running inside -// a container (currently Docker only). -func IsContainer() bool { - return ContainerCheckFn() -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `go test -run TestContainerPublicTestSuite -v ./pkg/sdk/platform/...` -Expected: PASS - -- [ ] **Step 5: Commit** - -``` -feat(platform): add IsContainer() for Docker container detection -``` - ---- - -### Task 2: DebianDocker DNS Provider — Struct and Update Stub - -**Files:** - -- Create: `internal/provider/network/dns/debian_docker.go` -- Create: - `internal/provider/network/dns/debian_docker_update_resolv_conf_by_interface.go` -- Create: `internal/provider/network/dns/debian_docker_public_test.go` - -- [ ] **Step 1: Write the failing test** - -Create `internal/provider/network/dns/debian_docker_public_test.go`: - -```go -package dns_test - -import ( - "log/slog" - "os" - "testing" - - "github.com/avfs/avfs" - "github.com/avfs/avfs/vfs/memfs" - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/provider" - "github.com/osapi-io/osapi/internal/provider/network/dns" -) - -type DebianDockerPublicTestSuite struct { - suite.Suite - - logger *slog.Logger - fs avfs.VFS -} - -func (s *DebianDockerPublicTestSuite) SetupTest() { - s.logger = slog.New(slog.NewTextHandler(os.Stdout, nil)) - s.fs = memfs.New() -} - -func (s *DebianDockerPublicTestSuite) SetupSubTest() { - s.SetupTest() -} - -func (s *DebianDockerPublicTestSuite) TestUpdateResolvConfByInterface() { - tests := []struct { - name string - }{ - { - name: "returns ErrUnsupported for container", - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - p := dns.NewDebianDockerProvider(s.logger, s.fs) - result, err := p.UpdateResolvConfByInterface( - []string{"8.8.8.8"}, - []string{"example.com"}, - "eth0", - ) - - s.Error(err) - s.Nil(result) - s.ErrorIs(err, provider.ErrUnsupported) - }) - } -} - -func TestDebianDockerPublicTestSuite(t *testing.T) { - suite.Run(t, new(DebianDockerPublicTestSuite)) -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: -`go test -run TestDebianDockerPublicTestSuite -v ./internal/provider/network/dns/...` -Expected: FAIL — `NewDebianDockerProvider` not defined. - -- [ ] **Step 3: Write minimal implementation** - -Create `internal/provider/network/dns/debian_docker.go`: - -```go -package dns - -import ( - "log/slog" - - "github.com/avfs/avfs" - - "github.com/osapi-io/osapi/internal/provider" -) - -// Compile-time check: DebianDocker must satisfy Provider and FactsSetter. -var _ Provider = (*DebianDocker)(nil) -var _ provider.FactsSetter = (*DebianDocker)(nil) - -// DebianDocker implements the DNS Provider interface for Debian-family -// systems running inside Docker containers. It reads DNS configuration -// from /etc/resolv.conf directly (no systemd-resolved). Updates are -// not supported because container DNS is managed by the runtime. -type DebianDocker struct { - provider.FactsAware - - logger *slog.Logger - fs avfs.VFS -} - -// NewDebianDockerProvider factory to create a new DebianDocker instance. -func NewDebianDockerProvider( - logger *slog.Logger, - fs avfs.VFS, -) *DebianDocker { - return &DebianDocker{ - logger: logger.With(slog.String("subsystem", "provider.dns.container")), - fs: fs, - } -} -``` - -Create -`internal/provider/network/dns/debian_docker_update_resolv_conf_by_interface.go`: - -```go -package dns - -import ( - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// UpdateResolvConfByInterface returns ErrUnsupported for container -// environments. DNS configuration in containers is managed by the -// container runtime (Docker, Kubernetes), not the agent. -func (d *DebianDocker) UpdateResolvConfByInterface( - _ []string, - _ []string, - _ string, -) (*UpdateResult, error) { - return nil, fmt.Errorf("dns (container): %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: -`go test -run TestDebianDockerPublicTestSuite -v ./internal/provider/network/dns/...` -Expected: FAIL — `GetResolvConfByInterface` not implemented yet (compile-time -check). Add a temporary stub to unblock: - -The compile-time `var _ Provider = (*DebianDocker)(nil)` will fail because -`GetResolvConfByInterface` is not yet implemented. To unblock, add a placeholder -in `debian_docker_get_resolv_conf_by_interface.go` that panics — this will be -replaced in Task 3. Alternatively, remove the `var _ Provider` line and add it -back in Task 3. - -The simpler approach: temporarily comment out -`var _ Provider = (*DebianDocker)(nil)` in `debian_docker.go`. Re-add it in Task -3 when Get is implemented. Keep only -`var _ provider.FactsSetter = (*DebianDocker)(nil)`. - -Run again: -`go test -run TestDebianDockerPublicTestSuite -v ./internal/provider/network/dns/...` -Expected: PASS - -- [ ] **Step 5: Commit** - -``` -feat(dns): add DebianDocker provider struct and Update stub -``` - ---- - -### Task 3: DebianDocker DNS Provider — Get (resolv.conf parsing) - -**Files:** - -- Create: - `internal/provider/network/dns/debian_docker_get_resolv_conf_by_interface.go` -- Modify: `internal/provider/network/dns/debian_docker.go` (re-add - `var _ Provider` check) -- Modify: `internal/provider/network/dns/debian_docker_public_test.go` (add Get - tests) - -- [ ] **Step 1: Write the failing tests** - -Add to `internal/provider/network/dns/debian_docker_public_test.go` — insert -this method before the `TestUpdateResolvConfByInterface` method: - -```go -func (s *DebianDockerPublicTestSuite) TestGetResolvConfByInterface() { - tests := []struct { - name string - setupFS func(fs avfs.VFS) - interfaceName string - want *dns.GetResult - wantErr bool - errContains string - }{ - { - name: "when resolv.conf has servers and search domains", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte( - "# Generated by Docker\n"+ - "nameserver 127.0.0.11\n"+ - "nameserver 8.8.8.8\n"+ - "search example.com local.lan\n"+ - "options ndots:0\n", - ), 0o644) - }, - interfaceName: "eth0", - want: &dns.GetResult{ - DNSServers: []string{"127.0.0.11", "8.8.8.8"}, - SearchDomains: []string{"example.com", "local.lan"}, - }, - }, - { - name: "when resolv.conf has only nameservers", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte( - "nameserver 8.8.8.8\n"+ - "nameserver 8.8.4.4\n", - ), 0o644) - }, - interfaceName: "eth0", - want: &dns.GetResult{ - DNSServers: []string{"8.8.8.8", "8.8.4.4"}, - SearchDomains: []string{"."}, - }, - }, - { - name: "when resolv.conf has IPv6 nameservers", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte( - "nameserver 2001:4860:4860::8888\n"+ - "nameserver 2001:4860:4860::8844\n", - ), 0o644) - }, - interfaceName: "any-interface", - want: &dns.GetResult{ - DNSServers: []string{"2001:4860:4860::8888", "2001:4860:4860::8844"}, - SearchDomains: []string{"."}, - }, - }, - { - name: "when resolv.conf has comments and blank lines", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte( - "# This is a comment\n"+ - "\n"+ - "nameserver 1.1.1.1\n"+ - "# Another comment\n"+ - "search test.local\n"+ - "\n", - ), 0o644) - }, - interfaceName: "eth0", - want: &dns.GetResult{ - DNSServers: []string{"1.1.1.1"}, - SearchDomains: []string{"test.local"}, - }, - }, - { - name: "when resolv.conf does not exist", - setupFS: func(fs avfs.VFS) { - // Don't create the file - }, - interfaceName: "eth0", - wantErr: true, - errContains: "failed to read /etc/resolv.conf", - }, - { - name: "when resolv.conf is empty", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte(""), 0o644) - }, - interfaceName: "eth0", - want: &dns.GetResult{ - DNSServers: nil, - SearchDomains: []string{"."}, - }, - }, - { - name: "when interface parameter is ignored", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte( - "nameserver 10.0.0.1\n", - ), 0o644) - }, - interfaceName: "completely-ignored", - want: &dns.GetResult{ - DNSServers: []string{"10.0.0.1"}, - SearchDomains: []string{"."}, - }, - }, - { - name: "when multiple search lines uses last one", - setupFS: func(fs avfs.VFS) { - _ = avfs.MkdirAll(fs, "/etc", 0o755) - _ = fs.WriteFile("/etc/resolv.conf", []byte( - "nameserver 8.8.8.8\n"+ - "search first.com\n"+ - "search second.com third.com\n", - ), 0o644) - }, - interfaceName: "eth0", - want: &dns.GetResult{ - DNSServers: []string{"8.8.8.8"}, - SearchDomains: []string{"second.com", "third.com"}, - }, - }, - } - - for _, tc := range tests { - s.Run(tc.name, func() { - tc.setupFS(s.fs) - - p := dns.NewDebianDockerProvider(s.logger, s.fs) - got, err := p.GetResolvConfByInterface(tc.interfaceName) - - if tc.wantErr { - s.Error(err) - s.Contains(err.Error(), tc.errContains) - } else { - s.NoError(err) - s.Equal(tc.want, got) - } - }) - } -} -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: -`go test -run TestDebianDockerPublicTestSuite/TestGetResolvConfByInterface -v ./internal/provider/network/dns/...` -Expected: FAIL — `GetResolvConfByInterface` not defined on `DebianDocker`. - -- [ ] **Step 3: Write minimal implementation** - -Create -`internal/provider/network/dns/debian_docker_get_resolv_conf_by_interface.go`: - -```go -package dns - -import ( - "bufio" - "fmt" - "strings" -) - -const resolvConfPath = "/etc/resolv.conf" - -// GetResolvConfByInterface reads DNS configuration from /etc/resolv.conf. -// The interfaceName parameter is accepted but ignored — containers have -// a single global DNS configuration managed by the container runtime. -func (d *DebianDocker) GetResolvConfByInterface( - _ string, -) (*GetResult, error) { - f, err := d.fs.Open(resolvConfPath) - if err != nil { - return nil, fmt.Errorf("failed to read %s: %w", resolvConfPath, err) - } - defer f.Close() - - result := &GetResult{} - - scanner := bufio.NewScanner(f) - for scanner.Scan() { - line := strings.TrimSpace(scanner.Text()) - - if line == "" || strings.HasPrefix(line, "#") { - continue - } - - fields := strings.Fields(line) - if len(fields) < 2 { - continue - } - - switch fields[0] { - case "nameserver": - result.DNSServers = append(result.DNSServers, fields[1]) - case "search": - result.SearchDomains = fields[1:] - } - } - - if err := scanner.Err(); err != nil { - return nil, fmt.Errorf("failed to parse %s: %w", resolvConfPath, err) - } - - if len(result.SearchDomains) == 0 { - result.SearchDomains = []string{"."} - } - - return result, nil -} -``` - -Re-add the Provider compile-time check in `debian_docker.go`: - -```go -var _ Provider = (*DebianDocker)(nil) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: -`go test -run TestDebianDockerPublicTestSuite -v ./internal/provider/network/dns/...` -Expected: PASS (both Get and Update tests) - -- [ ] **Step 5: Commit** - -``` -feat(dns): add DebianDocker Get implementation for resolv.conf parsing -``` - ---- - -### Task 4: Add `containerized` Built-in Fact Key - -**Files:** - -- Modify: `internal/facts/keys.go` -- Modify: `internal/facts/keys_public_test.go` -- Modify: `internal/controller/api/facts/facts_keys_get.go` -- Modify: `internal/controller/api/facts/facts_keys_get_public_test.go` - -- [ ] **Step 1: Update keys.go — add the constant** - -In `internal/facts/keys.go`, add `KeyContainerized` to the const block: - -```go -const ( - KeyInterfacePrimary = "interface.primary" - KeyHostname = "hostname" - KeyArch = "arch" - KeyKernel = "kernel" - KeyFQDN = "fqdn" - KeyContainerized = "containerized" -) -``` - -Add `KeyContainerized` to `BuiltInKeys()`: - -```go -func BuiltInKeys() []string { - return []string{ - KeyInterfacePrimary, - KeyHostname, - KeyArch, - KeyKernel, - KeyFQDN, - KeyContainerized, - } -} -``` - -Add `KeyContainerized` to the `IsKnownKey` switch: - -```go -case KeyInterfacePrimary, KeyHostname, KeyArch, KeyKernel, KeyFQDN, KeyContainerized: - return true -``` - -- [ ] **Step 2: Update the keys test** - -In `internal/facts/keys_public_test.go`, update `TestBuiltInKeys`: - -```go -{ - name: "when called returns all six built-in keys", - validateFunc: func(keys []string) { - s.Len(keys, 6) - s.Contains(keys, facts.KeyInterfacePrimary) - s.Contains(keys, facts.KeyHostname) - s.Contains(keys, facts.KeyArch) - s.Contains(keys, facts.KeyKernel) - s.Contains(keys, facts.KeyFQDN) - s.Contains(keys, facts.KeyContainerized) - }, -}, -``` - -Add a test row to `TestIsKnownKey`: - -```go -{ - name: "when containerized", - key: facts.KeyContainerized, - wantOK: true, -}, -``` - -- [ ] **Step 3: Update the facts keys API handler** - -In `internal/controller/api/facts/facts_keys_get.go`, add the description: - -```go -var builtInDescriptions = map[string]string{ - factskeys.KeyInterfacePrimary: "Primary network interface name", - factskeys.KeyHostname: "Agent hostname", - factskeys.KeyArch: "CPU architecture", - factskeys.KeyKernel: "Kernel version", - factskeys.KeyFQDN: "Fully qualified domain name", - factskeys.KeyContainerized: "Whether the agent is running inside a container", -} -``` - -- [ ] **Step 4: Update the facts keys API test** - -In `internal/controller/api/facts/facts_keys_get_public_test.go`, update the -test that checks the count of keys returned (if there is a count assertion, -update from 5 to 6). - -- [ ] **Step 5: Run tests** - -Run: `go test -v ./internal/facts/... ./internal/controller/api/facts/...` -Expected: PASS - -- [ ] **Step 6: Commit** - -``` -feat(facts): add containerized built-in fact key -``` - ---- - -### Task 5: Add `Containerized` to `FactsRegistration` and Fact Resolution - -**Files:** - -- Modify: `internal/job/types.go` -- Modify: `internal/agent/factref.go` -- Modify: `internal/agent/factref_public_test.go` - -- [ ] **Step 1: Add field to FactsRegistration** - -In `internal/job/types.go`, add `Containerized` to the `FactsRegistration` -struct: - -```go -type FactsRegistration struct { - Architecture string `json:"architecture,omitempty"` - KernelVersion string `json:"kernel_version,omitempty"` - CPUCount int `json:"cpu_count,omitempty"` - FQDN string `json:"fqdn,omitempty"` - ServiceMgr string `json:"service_mgr,omitempty"` - PackageMgr string `json:"package_mgr,omitempty"` - Containerized bool `json:"containerized"` - Interfaces []NetworkInterface `json:"interfaces,omitempty"` - PrimaryInterface string `json:"primary_interface,omitempty"` - Routes []Route `json:"routes,omitempty"` - Facts map[string]any `json:"facts,omitempty"` -} -``` - -Note: `Containerized bool` does NOT use `omitempty` — the field must always be -present (false is meaningful). - -- [ ] **Step 2: Add fact resolution for containerized** - -In `internal/agent/factref.go`, add a case to the `lookupFact` switch: - -```go -case facts.KeyContainerized: - if f.Containerized { - return "true", nil - } - return "false", nil -``` - -Insert after the `facts.KeyFQDN` case. - -- [ ] **Step 3: Add tests for @fact.containerized resolution** - -In `internal/agent/factref_public_test.go`, add two test rows to the -`TestResolveFacts` table: - -```go -{ - name: "when containerized is true", - params: map[string]any{ - "in_container": "@fact.containerized", - }, - facts: &job.FactsRegistration{ - Containerized: true, - }, - hostname: "web-01", - validateFunc: func(result map[string]any) { - s.Equal("true", result["in_container"]) - }, -}, -{ - name: "when containerized is false", - params: map[string]any{ - "in_container": "@fact.containerized", - }, - facts: &job.FactsRegistration{ - Containerized: false, - }, - hostname: "web-01", - validateFunc: func(result map[string]any) { - s.Equal("false", result["in_container"]) - }, -}, -``` - -- [ ] **Step 4: Run tests** - -Run: `go test -v ./internal/agent/... ./internal/job/...` Expected: PASS - -- [ ] **Step 5: Commit** - -``` -feat(facts): add Containerized field to FactsRegistration and fact resolver -``` - ---- - -### Task 6: Wire Container Detection into Facts Collection - -**Files:** - -- Modify: `internal/agent/facts.go` -- Modify: `internal/agent/facts_public_test.go` (if test covers writeFacts - fields) - -- [ ] **Step 1: Add platform.IsContainer() call to writeFacts** - -In `internal/agent/facts.go`, add the import: - -```go -"github.com/osapi-io/osapi/pkg/sdk/platform" -``` - -In the `writeFacts` method, add after `reg := job.FactsRegistration{}`: - -```go -reg.Containerized = platform.IsContainer() -``` - -- [ ] **Step 2: Run tests** - -Run: `go test -v ./internal/agent/...` Expected: PASS (existing tests should -still pass — `IsContainer()` returns false on the test host, matching the -default zero value) - -- [ ] **Step 3: Commit** - -``` -feat(agent): collect containerized fact during facts refresh -``` - ---- - -### Task 7: Wire DebianDocker DNS Provider into Agent Setup - -**Files:** - -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Update the DNS provider switch** - -In `cmd/agent_setup.go`, update the DNS provider selection: - -```go -// --- Network providers --- -var dnsProvider dns.Provider -switch plat { -case "debian": - if platform.IsContainer() { - dnsProvider = dns.NewDebianDockerProvider(log, appFs) - } else { - dnsProvider = dns.NewDebianProvider(log, execManager) - } -case "darwin": - dnsProvider = dns.NewDarwinProvider(log, execManager) -default: - dnsProvider = dns.NewLinuxProvider() -} -``` - -No new imports needed — `dns` and `platform` are already imported. - -- [ ] **Step 2: Verify build** - -Run: `go build ./...` Expected: Compiles with no errors. - -- [ ] **Step 3: Run all tests** - -Run: `just go::unit` Expected: PASS - -- [ ] **Step 4: Run lint** - -Run: `just go::vet` Expected: Clean - -- [ ] **Step 5: Commit** - -``` -feat(agent): wire DebianDocker DNS provider for containerized agents -``` - ---- - -### Task 8: Format and Final Verification - -- [ ] **Step 1: Format code** - -Run: `just go::fmt` - -- [ ] **Step 2: Run full test suite** - -Run: `just test` Expected: PASS (lint + unit + coverage) - -- [ ] **Step 3: Commit any formatting changes** - -If `just go::fmt` produced changes: - -``` -style: format new container DNS provider files -``` diff --git a/docs/plans/2026-03-28-hostname-update.md b/docs/plans/2026-03-28-hostname-update.md deleted file mode 100644 index 66a53fbc8..000000000 --- a/docs/plans/2026-03-28-hostname-update.md +++ /dev/null @@ -1,987 +0,0 @@ -# Hostname Update Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a full-stack `hostname update` feature — from provider through -API, SDK, CLI, and docs — so operators can change a node's hostname via -`hostnamectl set-hostname`. - -**Architecture:** The Debian provider calls `hostnamectl set-hostname` via -`exec.Manager`, checking the current hostname first for idempotency. Darwin, -Linux, and DebianDocker providers return `ErrUnsupported`. The API exposes -`PUT /node/{hostname}/hostname` following the DNS update pattern. The CLI adds -`client node hostname update --name `. - -**Tech Stack:** Go, exec.Manager (for hostnamectl), oapi-codegen, testify/suite, -gomock - ---- - -### Task 1: Provider — Add SetHostname to Interface and Types - -**Files:** - -- Modify: `internal/provider/node/host/types.go` - -- [ ] **Step 1: Add SetHostnameResult and SetHostname to the interface** - -In `internal/provider/node/host/types.go`, add the result type after the -existing `Result` struct: - -```go -// SetHostnameResult represents the outcome of a hostname set operation. -type SetHostnameResult struct { - // Changed indicates whether the hostname was actually modified. - Changed bool `json:"changed"` -} -``` - -Add `SetHostname` to the `Provider` interface: - -```go -// SetHostname sets the system hostname. -SetHostname(name string) (*SetHostnameResult, error) -``` - -- [ ] **Step 2: Verify build fails** - -Run: `go build ./...` Expected: FAIL — all Provider implementations missing -`SetHostname`. - -- [ ] **Step 3: Commit** - -``` -feat(host): add SetHostname to Provider interface -``` - ---- - -### Task 2: Provider — Debian SetHostname Implementation - -**Files:** - -- Modify: `internal/provider/node/host/debian.go` (add `exec.Manager` field) -- Create: `internal/provider/node/host/debian_set_hostname.go` -- Create: `internal/provider/node/host/debian_set_hostname_public_test.go` -- Modify: `cmd/agent_setup.go` (pass `execManager` to `NewDebianProvider`) - -- [ ] **Step 1: Add exec.Manager to Debian struct** - -In `internal/provider/node/host/debian.go`, add the import and field: - -```go -import ( - "os" - "os/exec" - "runtime" - - "github.com/shirou/gopsutil/v4/host" - - "github.com/osapi-io/osapi/internal/exec" - "github.com/osapi-io/osapi/internal/provider" -) -``` - -Note: the import alias must avoid collision with `os/exec`. Use the full path -`iexec "github.com/osapi-io/osapi/internal/exec"` if needed, or since the field -type is an interface, use the `exec.Manager` type directly. Check how the dns -package handles this — it imports `"github.com/osapi-io/osapi/internal/exec"` -and the field type is `exec.Manager`. - -Add `execManager` field to the `Debian` struct: - -```go -type Debian struct { - provider.FactsAware - - InfoFn func() (*host.InfoStat, error) - HostnameFn func() (string, error) - NumCPUFn func() int - StatFn func(name string) (os.FileInfo, error) - LookPathFn func(file string) (string, error) - execManager iexec.Manager -} -``` - -Update `NewDebianProvider` to accept `exec.Manager`: - -```go -func NewDebianProvider( - execManager iexec.Manager, -) *Debian { - return &Debian{ - InfoFn: host.Info, - HostnameFn: os.Hostname, - NumCPUFn: runtime.NumCPU, - StatFn: os.Stat, - LookPathFn: exec.LookPath, - execManager: execManager, - } -} -``` - -Update `cmd/agent_setup.go` to pass `execManager`: - -```go -case "debian": - hostProvider = nodeHost.NewDebianProvider(execManager) -``` - -- [ ] **Step 2: Write the failing test** - -Create `internal/provider/node/host/debian_set_hostname_public_test.go`: - -```go -package host_test - -import ( - "log/slog" - "os" - "testing" - - "github.com/golang/mock/gomock" - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/exec/mocks" - "github.com/osapi-io/osapi/internal/provider/node/host" -) - -type DebianSetHostnamePublicTestSuite struct { - suite.Suite - ctrl *gomock.Controller - - logger *slog.Logger -} - -func (s *DebianSetHostnamePublicTestSuite) SetupTest() { - s.ctrl = gomock.NewController(s.T()) - s.logger = slog.New(slog.NewTextHandler(os.Stdout, nil)) -} - -func (s *DebianSetHostnamePublicTestSuite) SetupSubTest() { - s.SetupTest() -} - -func (s *DebianSetHostnamePublicTestSuite) TearDownTest() { - s.ctrl.Finish() -} - -func (s *DebianSetHostnamePublicTestSuite) TestSetHostname() { - tests := []struct { - name string - setupMock func() *mocks.MockManager - hostname string - wantErr bool - errContains string - validateFunc func(*host.SetHostnameResult) - }{ - { - name: "when hostname changes", - setupMock: func() *mocks.MockManager { - mock := mocks.NewPlainMockManager(s.ctrl) - mock.EXPECT(). - RunCmd("hostnamectl", []string{"hostname"}). - Return("old-hostname\n", nil) - mock.EXPECT(). - RunCmd("hostnamectl", []string{"set-hostname", "new-hostname"}). - Return("", nil) - return mock - }, - hostname: "new-hostname", - validateFunc: func(r *host.SetHostnameResult) { - s.True(r.Changed) - }, - }, - { - name: "when hostname already set returns unchanged", - setupMock: func() *mocks.MockManager { - mock := mocks.NewPlainMockManager(s.ctrl) - mock.EXPECT(). - RunCmd("hostnamectl", []string{"hostname"}). - Return("same-hostname\n", nil) - return mock - }, - hostname: "same-hostname", - validateFunc: func(r *host.SetHostnameResult) { - s.False(r.Changed) - }, - }, - { - name: "when hostnamectl hostname errors", - setupMock: func() *mocks.MockManager { - mock := mocks.NewPlainMockManager(s.ctrl) - mock.EXPECT(). - RunCmd("hostnamectl", []string{"hostname"}). - Return("", fmt.Errorf("command not found")) - return mock - }, - hostname: "new-hostname", - wantErr: true, - errContains: "failed to get current hostname", - }, - { - name: "when hostnamectl set-hostname errors", - setupMock: func() *mocks.MockManager { - mock := mocks.NewPlainMockManager(s.ctrl) - mock.EXPECT(). - RunCmd("hostnamectl", []string{"hostname"}). - Return("old-hostname\n", nil) - mock.EXPECT(). - RunCmd("hostnamectl", []string{"set-hostname", "new-hostname"}). - Return("", fmt.Errorf("permission denied")) - return mock - }, - hostname: "new-hostname", - wantErr: true, - errContains: "failed to set hostname", - }, - } - - for _, tc := range tests { - s.Run(tc.name, func() { - mock := tc.setupMock() - - p := host.NewDebianProvider(mock) - result, err := p.SetHostname(tc.hostname) - - if tc.wantErr { - s.Error(err) - s.Contains(err.Error(), tc.errContains) - } else { - s.NoError(err) - tc.validateFunc(result) - } - }) - } -} - -func TestDebianSetHostnamePublicTestSuite(t *testing.T) { - suite.Run(t, new(DebianSetHostnamePublicTestSuite)) -} -``` - -Note: add `"fmt"` to imports for the test. - -- [ ] **Step 3: Run test to verify it fails** - -Run: -`go test -run TestDebianSetHostnamePublicTestSuite -v ./internal/provider/node/host/...` -Expected: FAIL — `SetHostname` not defined. - -- [ ] **Step 4: Write implementation** - -Create `internal/provider/node/host/debian_set_hostname.go`: - -```go -package host - -import ( - "fmt" - "strings" -) - -// SetHostname sets the system hostname using hostnamectl. -// It checks the current hostname first and returns Changed: false -// if the hostname is already set to the requested value. -func (u *Debian) SetHostname( - name string, -) (*SetHostnameResult, error) { - current, err := u.execManager.RunCmd("hostnamectl", []string{"hostname"}) - if err != nil { - return nil, fmt.Errorf("failed to get current hostname: %w", err) - } - - if strings.TrimSpace(current) == name { - return &SetHostnameResult{Changed: false}, nil - } - - if _, err := u.execManager.RunCmd("hostnamectl", []string{"set-hostname", name}); err != nil { - return nil, fmt.Errorf("failed to set hostname: %w", err) - } - - return &SetHostnameResult{Changed: true}, nil -} -``` - -- [ ] **Step 5: Run tests** - -Run: -`go test -run TestDebianSetHostnamePublicTestSuite -v ./internal/provider/node/host/...` -Expected: PASS - -- [ ] **Step 6: Commit** - -``` -feat(host): add Debian SetHostname via hostnamectl -``` - ---- - -### Task 3: Provider — Darwin, Linux, and DebianDocker Stubs - -**Files:** - -- Create: `internal/provider/node/host/darwin_set_hostname.go` -- Create: `internal/provider/node/host/linux_set_hostname.go` -- Modify: existing Darwin/Linux test files to add SetHostname test - -For DebianDocker: the host provider is not container-aware the same way DNS is. -The container check happens at agent_setup time — if `platform.IsContainer()`, -pass a nil `execManager` or use the existing pattern where the Debian provider -is still used but `hostnamectl` will fail naturally inside Docker. The cleaner -approach: since `hostnamectl` won't exist in Docker containers, the Debian -provider's `SetHostname` will return an error from `RunCmd`. This is acceptable -— the error message will be clear -(`failed to get current hostname: exec: "hostnamectl": executable file not found`). -No separate DebianDocker host provider is needed. - -- [ ] **Step 1: Create Darwin stub** - -Create `internal/provider/node/host/darwin_set_hostname.go`: - -```go -package host - -import ( - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// SetHostname returns ErrUnsupported on Darwin. -// Darwin is a development platform only; mutations are not supported. -func (d *Darwin) SetHostname( - _ string, -) (*SetHostnameResult, error) { - return nil, fmt.Errorf("host: %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 2: Create Linux stub** - -Create `internal/provider/node/host/linux_set_hostname.go`: - -```go -package host - -import ( - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// SetHostname returns ErrUnsupported on generic Linux. -func (l *Linux) SetHostname( - _ string, -) (*SetHostnameResult, error) { - return nil, fmt.Errorf("host: %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 3: Add tests for both stubs** - -Add `TestSetHostname` methods to the existing Darwin and Linux test suites (in -their respective `*_public_test.go` files). Follow the pattern in -`internal/provider/network/dns/linux_public_test.go` — single-row table testing -`ErrUnsupported`. - -- [ ] **Step 4: Verify build and tests** - -Run: `go build ./... && go test -v ./internal/provider/node/host/...` Expected: -PASS - -- [ ] **Step 5: Commit** - -``` -feat(host): add Darwin and Linux SetHostname stubs (ErrUnsupported) -``` - ---- - -### Task 4: Job Operation and Agent Processor - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` (add `OpNodeHostnameUpdate`) -- Modify: `internal/job/types.go` (add `OperationNodeHostnameUpdate`) -- Modify: `internal/agent/processor.go` (handle hostname update in processor) - -- [ ] **Step 1: Add operation constant** - -In `pkg/sdk/client/operations.go`, add after `OpNodeHostnameGet`: - -```go -OpNodeHostnameUpdate JobOperation = "node.hostname.update" -``` - -In `internal/job/types.go`, add after `OperationNodeHostnameGet`: - -```go -OperationNodeHostnameUpdate = client.OpNodeHostnameUpdate -``` - -- [ ] **Step 2: Update processor to handle hostname update** - -In `internal/agent/processor.go`, change the `hostname` case to sub-dispatch: - -```go -case "hostname": - if req.Operation == job.OperationNodeHostnameUpdate { - return setNodeHostname(hostProvider, req, logger) - } - return getNodeHostname(hostProvider, appConfig, logger) -``` - -Add the `setNodeHostname` function after `getNodeHostname`: - -```go -// setNodeHostname sets the node hostname via the host provider. -func setNodeHostname( - hostProvider nodeHost.Provider, - req job.Request, - logger *slog.Logger, -) (json.RawMessage, error) { - logger.Debug("executing host.SetHostname") - - var data struct { - Hostname string `json:"hostname"` - } - if err := json.Unmarshal(req.Data, &data); err != nil { - return nil, fmt.Errorf("invalid hostname update data: %w", err) - } - - result, err := hostProvider.SetHostname(data.Hostname) - if err != nil { - return nil, err - } - - resp := map[string]interface{}{ - "hostname": data.Hostname, - "changed": result.Changed, - } - - return json.Marshal(resp) -} -``` - -- [ ] **Step 3: Run tests** - -Run: `go build ./... && go test -v ./internal/agent/...` Expected: PASS - -- [ ] **Step 4: Commit** - -``` -feat(agent): add hostname update operation to node processor -``` - ---- - -### Task 5: OpenAPI Spec — PUT /node/{hostname}/hostname - -**Files:** - -- Modify: `internal/controller/api/node/gen/api.yaml` - -- [ ] **Step 1: Add PUT endpoint and schemas** - -In `internal/controller/api/node/gen/api.yaml`, add `put:` under the existing -`/node/{hostname}/hostname` path (after the `get:` block, before the next path): - -```yaml -put: - summary: Update node hostname - description: Set the system hostname on the target node. - tags: - - node_operations - operationId: PutNodeHostname - security: - - BearerAuth: - - node:write - parameters: - - $ref: '#/components/parameters/Hostname' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/HostnameUpdateRequest' - responses: - '202': - description: Hostname update accepted. - content: - application/json: - schema: - $ref: '#/components/schemas/HostnameUpdateCollectionResponse' - '400': - description: Invalid input. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '401': - description: Unauthorized - API key required - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden - Insufficient permissions - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Error updating hostname. - content: - application/json: - schema: - $ref: '../../common/gen/api.yaml#/components/schemas/ErrorResponse' -``` - -Add the request and response schemas to `components/schemas`: - -```yaml -HostnameUpdateRequest: - type: object - properties: - hostname: - type: string - x-oapi-codegen-extra-tags: - validate: required,min=1,max=253 - description: The new hostname to set. - example: 'web-01' - required: - - hostname - -HostnameUpdateResultItem: - type: object - properties: - hostname: - type: string - description: The hostname of the agent. - status: - type: string - enum: [ok, failed] - changed: - type: boolean - description: Whether the hostname was actually modified. - error: - type: string - required: - - hostname - - status - -HostnameUpdateCollectionResponse: - type: object - properties: - job_id: - type: string - format: uuid - description: The job ID used to process this request. - example: '550e8400-e29b-41d4-a716-446655440000' - results: - type: array - items: - $ref: '#/components/schemas/HostnameUpdateResultItem' - required: - - results -``` - -- [ ] **Step 2: Regenerate code** - -Run: `just generate` - -- [ ] **Step 3: Verify generated code compiles** - -Run: `go build ./...` Expected: FAIL — new `PutNodeHostname` method required by -`StrictServerInterface` but not implemented yet. - -- [ ] **Step 4: Commit** - -``` -feat(api): add PUT /node/{hostname}/hostname OpenAPI spec -``` - ---- - -### Task 6: API Handler — PutNodeHostname - -**Files:** - -- Create: `internal/controller/api/node/node_hostname_put.go` -- Create: `internal/controller/api/node/node_hostname_put_public_test.go` - -- [ ] **Step 1: Write the handler** - -Create `internal/controller/api/node/node_hostname_put.go` following the -`network_dns_put_by_interface.go` pattern exactly. The handler: - -1. Validates the target hostname via `validateHostname()` -2. Validates the request body via `validation.Struct(request.Body)` -3. Checks `job.IsBroadcastTarget()` and routes accordingly -4. Single target: calls `s.JobClient.Modify()` with category `"node"` and - operation `job.OperationNodeHostnameUpdate` -5. Broadcast: calls `s.JobClient.ModifyBroadcast()` with same -6. Returns 202 with `HostnameUpdateCollectionResponse` - -Data passed to the job: - -```go -data := map[string]any{ - "hostname": request.Body.Hostname, -} -``` - -- [ ] **Step 2: Write tests** - -Create `internal/controller/api/node/node_hostname_put_public_test.go` with -table-driven tests covering: success (single target), success (broadcast), -validation error (empty hostname), bad target hostname, and job client error. -Follow the existing test patterns in -`internal/controller/api/node/node_hostname_get_public_test.go`. - -Include `TestPutNodeHostnameHTTP` and `TestPutNodeHostnameRBACHTTP` methods for -wiring tests through the full Echo middleware stack. - -- [ ] **Step 3: Run tests** - -Run: `go test -v ./internal/controller/api/node/...` Expected: PASS - -- [ ] **Step 4: Commit** - -``` -feat(api): add PutNodeHostname handler with broadcast support -``` - ---- - -### Task 7: Server Wiring and Permissions - -**Files:** - -- Modify: `internal/controller/api/handler.go` (if handler registration needs - updating — check if the node handler auto-registers all methods) -- Modify: `internal/config/permissions.go` or equivalent — add `node:write` - permission to the `admin` and `write` roles - -- [ ] **Step 1: Check if handler registration is automatic** - -The node handler already implements `StrictServerInterface`. After regenerating, -the new `PutNodeHostname` method is automatically included in the handler. Check -if any explicit route registration is needed. - -- [ ] **Step 2: Add node:write permission to roles** - -Check where permissions are defined (likely in the security/auth middleware -configuration or in `internal/config/`). Add `node:write` to the `admin` and -`write` roles. This is the permission declared in the OpenAPI spec's -`BearerAuth` security for the PUT endpoint. - -- [ ] **Step 3: Verify build and tests** - -Run: `go build ./... && go test -v ./internal/controller/api/...` Expected: PASS - -- [ ] **Step 4: Commit** - -``` -feat(auth): add node:write permission for hostname update -``` - ---- - -### Task 8: SDK — SetHostname Method - -**Files:** - -- Modify: `pkg/sdk/client/node.go` (add `SetHostname` method) -- Modify: `pkg/sdk/client/node_types.go` (add `HostnameUpdateResult` type) -- Modify: `pkg/sdk/client/gen/` (regenerate SDK client from combined spec) - -- [ ] **Step 1: Regenerate SDK client** - -Run: `go generate ./pkg/sdk/client/gen/...` - -- [ ] **Step 2: Add SDK types** - -In `pkg/sdk/client/node_types.go`, add: - -```go -// HostnameUpdateResult represents a hostname update result from a single agent. -type HostnameUpdateResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Error string `json:"error,omitempty"` - Changed bool `json:"changed"` -} -``` - -Add the `gen→SDK` conversion function: - -```go -func hostnameUpdateCollectionFromGen( - r *gen.HostnameUpdateCollectionResponse, -) Collection[HostnameUpdateResult] { - // ... follow existing pattern from hostnameCollectionFromGen -} -``` - -- [ ] **Step 3: Add SetHostname method** - -In `pkg/sdk/client/node.go`, add: - -```go -// SetHostname updates the hostname on the target node. -func (s *NodeService) SetHostname( - ctx context.Context, - target string, - name string, -) (*Response[Collection[HostnameUpdateResult]], error) { - body := gen.HostnameUpdateRequest{ - Hostname: name, - } - - resp, err := s.client.PutNodeHostnameWithResponse(ctx, target, body) - if err != nil { - return nil, fmt.Errorf("set hostname: %w", err) - } - - if err := checkError( - resp.StatusCode(), - resp.JSON400, - resp.JSON401, - resp.JSON403, - resp.JSON500, - ); err != nil { - return nil, err - } - - if resp.JSON202 == nil { - return nil, &UnexpectedStatusError{APIError{ - StatusCode: resp.StatusCode(), - Message: "nil response body", - }} - } - - return NewResponse( - hostnameUpdateCollectionFromGen(resp.JSON202), - resp.Body, - ), nil -} -``` - -- [ ] **Step 4: Run tests** - -Run: `go build ./... && go test -v ./pkg/sdk/client/...` Expected: PASS - -- [ ] **Step 5: Commit** - -``` -feat(sdk): add SetHostname method to NodeService -``` - ---- - -### Task 9: CLI — client node hostname update - -**Files:** - -- Create: `cmd/client_node_hostname_update.go` - -- [ ] **Step 1: Create the CLI command** - -Create `cmd/client_node_hostname_update.go` following the pattern of -`cmd/client_node_network_dns_update.go`: - -```go -var clientNodeHostnameUpdateCmd = &cobra.Command{ - Use: "update", - Short: "Update the node's hostname", - Long: `Set a new hostname on the target node using hostnamectl.`, - Run: func(cmd *cobra.Command, _ []string) { - ctx := cmd.Context() - host, _ := cmd.Flags().GetString("target") - name, _ := cmd.Flags().GetString("name") - - resp, err := sdkClient.Node.SetHostname(ctx, host, name) - if err != nil { - cli.HandleError(err, logger) - return - } - - if jsonOutput { - fmt.Println(string(resp.RawJSON())) - return - } - - if resp.Data.JobID != "" { - fmt.Println() - cli.PrintKV("Job ID", resp.Data.JobID) - } - - results := make([]cli.ResultRow, 0, len(resp.Data.Results)) - for _, r := range resp.Data.Results { - var errPtr *string - if r.Error != "" { - errPtr = &r.Error - } - results = append(results, cli.ResultRow{ - Hostname: r.Hostname, - Error: errPtr, - Fields: []string{fmt.Sprintf("%t", r.Changed)}, - }) - } - headers, rows := cli.BuildBroadcastTable(results, []string{"CHANGED"}) - cli.PrintCompactTable([]cli.Section{{Headers: headers, Rows: rows}}) - }, -} - -func init() { - clientNodeHostnameCmd.AddCommand(clientNodeHostnameUpdateCmd) - clientNodeHostnameUpdateCmd.Flags().String("name", "", "New hostname to set (required)") - _ = clientNodeHostnameUpdateCmd.MarkFlagRequired("name") -} -``` - -- [ ] **Step 2: Verify it works** - -Run: `go build ./... && go run main.go client node hostname update --help` -Expected: Shows help with `--name` and `--target` flags. - -- [ ] **Step 3: Commit** - -``` -feat(cli): add client node hostname update command -``` - ---- - -### Task 10: Documentation Updates - -**Files:** - -- Modify: `docs/docs/sidebar/usage/cli/client/node/hostname.md` -- Modify: `docs/docs/sidebar/sdk/client/node.md` -- Modify: `docs/docs/sidebar/sdk/orchestrator/operations/node-hostname.md` -- Modify: `examples/sdk/client/node.go` -- Modify: `docs/docs/sidebar/usage/configuration.md` (add `node:write` to - permissions table) - -- [ ] **Step 1: Update CLI docs** - -In `docs/docs/sidebar/usage/cli/client/node/hostname.md`, add the update section -after the existing get examples: - -```markdown -## Update - -Set the hostname on the target node: - -\`\`\`bash $ osapi client node hostname update --name web-01 - -Job ID: 550e8400-e29b-41d4-a716-446655440000 - -HOSTNAME CHANGED web-01 true \`\`\` - -When targeting all hosts: - -\`\`\`bash $ osapi client node hostname update --name web-01 --target \_all -\`\`\` - -### Flags - -| Flag | Description | Default | -| -------------- | -------------------------------------------------------- | ------- | -| `--name` | New hostname to set (required) | | -| `-T, --target` | Target: `_any`, `_all`, hostname, or label (`group:web`) | `_any` | -``` - -- [ ] **Step 2: Update SDK docs** - -In `docs/docs/sidebar/sdk/client/node.md`, add `SetHostname` to the Node Info -methods table and add a usage example. - -In `docs/docs/sidebar/sdk/orchestrator/operations/node-hostname.md`, add a -section for `node.hostname.update`. - -- [ ] **Step 3: Update SDK example** - -In `examples/sdk/client/node.go`, add a `SetHostname` example after the Hostname -get block: - -```go -// Set hostname (uncomment to run — this mutates the system) -// setResp, err := c.Node.SetHostname(ctx, "web-01", "new-hostname") -// if err != nil { -// log.Fatalf("set hostname: %v", err) -// } -// fmt.Printf("Set hostname changed: %t\n", setResp.Data.Results[0].Changed) -``` - -- [ ] **Step 4: Update permissions docs** - -In `docs/docs/sidebar/usage/configuration.md`, add `node:write` to the `admin` -and `write` role permission lists. - -- [ ] **Step 5: Commit** - -``` -docs: add hostname update to CLI, SDK, and configuration docs -``` - ---- - -### Task 11: Integration Test - -**Files:** - -- Modify: `test/integration/node_test.go` - -- [ ] **Step 1: Add hostname update integration test** - -Add a test case to the existing node test suite. Guard with `skipWrite(s.T())` -since this is a mutation: - -```go -{ - name: "updates hostname", - args: []string{"client", "node", "hostname", "update", "--name", currentHostname, "--json"}, - validateFunc: func(stdout string, exitCode int) { - skipWrite(s.T()) - s.Require().Equal(0, exitCode) - // Parse JSON response and verify changed field - }, -}, -``` - -Use the current hostname to ensure idempotency (Changed: false). - -- [ ] **Step 2: Commit** - -``` -test(integration): add hostname update integration test -``` - ---- - -### Task 12: Format and Final Verification - -- [ ] **Step 1: Format** - -Run: `just go::fmt` - -- [ ] **Step 2: Lint** - -Run: `just go::vet` Expected: 0 issues - -- [ ] **Step 3: Full test suite** - -Run: `just go::unit` Expected: PASS - -- [ ] **Step 4: Commit any formatting changes** - -``` -style: format hostname update files -``` diff --git a/docs/plans/2026-03-29-sysctl-provider-design.md b/docs/plans/2026-03-29-sysctl-provider-design.md deleted file mode 100644 index db165eda0..000000000 --- a/docs/plans/2026-03-29-sysctl-provider-design.md +++ /dev/null @@ -1,264 +0,0 @@ -# Sysctl Provider Design - -## Overview - -Add kernel parameter management (sysctl) to OSAPI. Users can query, set, and -remove sysctl parameters on managed nodes. Every set operation is persistent -(writes to `/etc/sysctl.d/` and applies immediately) — there is no runtime-only -mode. - -## Architecture - -Sysctl is a **meta-provider** under `internal/provider/node/sysctl/`. It -delegates file writes to `file.Deployer` for SHA tracking, idempotency, and -drift detection. After deploying the conf file, it calls `sysctl -p ` to -apply the change at runtime. - -- **Category**: `node` (alongside host, disk, mem, load) -- **Path prefix**: `/node/{hostname}/sysctl` -- **Permissions**: `sysctl:read`, `sysctl:write` -- **Provider type**: meta-provider (delegates to file provider) - -## Provider Interface - -```go -// Package sysctl provides kernel parameter management via /etc/sysctl.d/. -package sysctl - -type Provider interface { - List(ctx context.Context) ([]Entry, error) - Get(ctx context.Context, key string) (*Entry, error) - Set(ctx context.Context, entry Entry) (*SetResult, error) - Delete(ctx context.Context, key string) (*DeleteResult, error) -} -``` - -No separate Create/Update — `Set` is idempotent. If the key's conf file exists -with the same value, `Changed: false`. If different or new, deploy and apply. - -## Data Types - -```go -type Entry struct { - Key string `json:"key"` // e.g., "net.ipv4.ip_forward" - Value string `json:"value"` // e.g., "1" -} - -type SetResult struct { - Key string `json:"key"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type DeleteResult struct { - Key string `json:"key"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -## File Layout - -Each managed key gets its own conf file: - -- **Path**: `/etc/sysctl.d/osapi-{sanitized-key}.conf` -- **Content**: `{key} = {value}\n` -- **Sanitization**: dots remain in the filename (e.g., - `osapi-net.ipv4.ip_forward.conf`) - -The file provider tracks each file in the file-state KV bucket. Domain-specific -metadata (key, value) is stored in the `FileState.Metadata` map. - -## Operations Flow - -### Set - -1. Validate key format (dotted kernel parameter name) -2. Generate conf file content: `{key} = {value}\n` -3. Deploy via `file.Deployer.Deploy()` — handles SHA comparison, idempotency, - and state persistence -4. If changed, apply via `sysctl -p /etc/sysctl.d/osapi-{key}.conf` -5. Return `SetResult` with `Changed` reflecting whether the file was actually - modified - -### Delete - -1. Look up key in file-state KV to verify it is managed -2. Undeploy via `file.Deployer.Undeploy()` — removes file and state -3. Apply defaults via `sysctl --system` to reload all conf files -4. Return `DeleteResult` - -### List - -1. Scan file-state KV for sysctl-managed files (filter by metadata) -2. For each managed key, read current runtime value via `sysctl -n {key}` -3. Return list of `Entry` with current runtime values - -### Get - -1. Look up key in file-state KV -2. Read current runtime value via `sysctl -n {key}` -3. Return `Entry` - -## API Endpoints - -| Method | Path | Permission | Description | -| -------- | ------------------------------- | -------------- | ------------------------------------- | -| `GET` | `/node/{hostname}/sysctl` | `sysctl:read` | List managed sysctl entries | -| `GET` | `/node/{hostname}/sysctl/{key}` | `sysctl:read` | Get a single entry by key | -| `POST` | `/node/{hostname}/sysctl` | `sysctl:write` | Set a sysctl key (idempotent) | -| `DELETE` | `/node/{hostname}/sysctl/{key}` | `sysctl:write` | Remove managed entry, restore default | - -All endpoints support broadcast targeting (`_all`, `_any`, hostname, label -selectors). - -### Response Shape - -All node-targeted operations return the standard collection response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "key": "net.ipv4.ip_forward", - "value": "1", - "error": "" - }, - { - "hostname": "web-02", - "key": "net.ipv4.ip_forward", - "value": "1", - "error": "" - } - ] -} -``` - -Set/Delete results include `changed` field. Single-target returns 1 result; -broadcast returns N results. - -### POST Request Body - -```json -{ - "key": "net.ipv4.ip_forward", - "value": "1" -} -``` - -### Validation - -- `key`: required, must match sysctl key format (dotted name, e.g., - `net.ipv4.ip_forward`) -- `value`: required, non-empty string -- Path parameter `{key}` on GET/DELETE uses the dotted key name directly - -## Platform Implementations - -| Platform | Implementation | -| -------- | ----------------------------------------------------- | -| Debian | Full — delegates to file provider, applies via sysctl | -| Darwin | Returns `ErrUnsupported` for all methods | -| Linux | Returns `ErrUnsupported` for all methods | - -### Container Behavior - -No `DebianDocker` variant is needed. Unlike hostname or DNS, sysctl works the -same inside containers — reads always succeed (host kernel values), and writes -succeed or fail based on container capabilities. The standard Debian provider -handles both cases: if the agent lacks permissions, `sysctl -w` returns an error -and the provider reports it in the result. - -### Debian Dependencies - -- `file.Deployer` — for conf file deployment and state tracking -- `jetstream.KeyValue` — file-state KV for listing managed entries -- `exec.Manager` — for running `sysctl -p` and `sysctl -n` commands -- `avfs.VFS` — filesystem access - -## Orchestrator Integration - -The OSAPI API is single-key CRUD. The orchestrator DSL handles batching — a -single sysctl block in the DSL can declare multiple keys, and the orchestrator -iterates, calling the API once per key. This is the same pattern used for cron -entries. - -```yaml -# Example orchestrator DSL (future work in osapi-orchestrator) -sysctl: - - key: net.ipv4.ip_forward - value: '1' - - key: net.core.somaxconn - value: '4096' -``` - -## Files to Create/Modify - -### New Files - -``` -internal/provider/node/sysctl/ - types.go — Provider interface + Entry, SetResult, DeleteResult - debian.go — Debian implementation (meta-provider) - darwin.go — macOS stub - linux.go — Generic Linux stub - mocks/ - generate.go — //go:generate mockgen directive - -internal/controller/api/sysctl/ - types.go — Handler struct + dependency interfaces - sysctl.go — New() factory + interface check - sysctl_list_get.go — GET /sysctl handler - sysctl_get.go — GET /sysctl/{key} handler - sysctl_set.go — POST /sysctl handler - sysctl_delete.go — DELETE /sysctl/{key} handler - validate.go — Input validation helpers - gen/ - api.yaml — OpenAPI spec - cfg.yaml — oapi-codegen config - generate.go — //go:generate directive - -internal/agent/processor_sysctl.go — Processor wiring - -pkg/sdk/client/ - sysctl.go — SysctlService methods - sysctl_types.go — SDK result types + gen conversions - -cmd/ - client_sysctl.go — Parent command - client_sysctl_list.go — list subcommand - client_sysctl_get.go — get subcommand - client_sysctl_set.go — set subcommand - client_sysctl_delete.go — delete subcommand - -examples/sdk/client/sysctl.go — SDK example - -docs/docs/sidebar/features/sysctl.md — Feature docs -docs/docs/sidebar/usage/cli/client/sysctl/sysctl.md — CLI parent -docs/docs/sidebar/usage/cli/client/sysctl/list.md — CLI list -docs/docs/sidebar/usage/cli/client/sysctl/get.md — CLI get -docs/docs/sidebar/usage/cli/client/sysctl/set.md — CLI set -docs/docs/sidebar/usage/cli/client/sysctl/delete.md — CLI delete - -test/integration/sysctl_test.go — Integration tests -``` - -### Modified Files - -``` -internal/job/types.go — Add OperationSysctl* constants -internal/agent/processor_sysctl.go — New processor -cmd/agent_setup.go — Create and register sysctl provider -internal/controller/api/types.go — Add sysctlHandler field -internal/controller/api/handler.go — Wire handler in CreateHandlers -internal/controller/api/handler_sysctl.go — GetSysctlHandler method -cmd/controller_start.go — Initialize sysctl handler -pkg/sdk/client/osapi.go — Wire SysctlService -pkg/sdk/client/permissions.go — Add PermSysctlRead/Write -internal/authtoken/permissions.go — Re-export + add to roles -docs/docusaurus.config.ts — Add to Features navbar -docs/docs/sidebar/usage/configuration.md — Add permissions -docs/docs/sidebar/architecture/system-architecture.md — Add endpoints -``` diff --git a/docs/plans/2026-03-29-sysctl-provider.md b/docs/plans/2026-03-29-sysctl-provider.md deleted file mode 100644 index 6605445bc..000000000 --- a/docs/plans/2026-03-29-sysctl-provider.md +++ /dev/null @@ -1,1491 +0,0 @@ -# Sysctl Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add kernel parameter management (sysctl) as a meta-provider that -delegates file writes to `file.Deployer` for SHA tracking and idempotency, with -full API/CLI/SDK support. - -**Architecture:** Sysctl is a meta-provider under -`internal/provider/node/sysctl/` that generates `/etc/sysctl.d/osapi-{key}.conf` -files and applies them via `sysctl -p`. It follows the cron provider pattern: -delegate to `file.Deployer`, store domain metadata in `FileState.Metadata`, scan -file-state KV to list managed entries. The API lives under -`/node/{hostname}/sysctl` with broadcast support. - -**Tech Stack:** Go 1.25, Echo, oapi-codegen (strict-server), NATS JetStream KV, -gomock, testify/suite, avfs - -**Coverage baseline:** 99.9% — must remain at or above this after -implementation. - ---- - -## File Map - -### New Files - -``` -internal/provider/node/sysctl/ - types.go — Provider interface + Entry, SetResult, DeleteResult - debian.go — Debian meta-provider (delegates to file.Deployer) - darwin.go — macOS stub (ErrUnsupported) - linux.go — Generic Linux stub (ErrUnsupported) - export_test.go — Expose unexported vars for test package - mocks/ - generate.go — //go:generate mockgen directive - -internal/agent/processor_sysctl.go — NewSysctlProcessor factory + operation dispatch - -internal/controller/api/sysctl/ - types.go — Handler struct + dependencies - sysctl.go — New() factory + interface check - sysctl_list_get.go — GET /sysctl handler + broadcast - sysctl_get.go — GET /sysctl/{key} handler + broadcast - sysctl_set.go — POST /sysctl handler + broadcast - sysctl_delete.go — DELETE /sysctl/{key} handler + broadcast - validate.go — validateHostname + validateSysctlKey - sysctl_list_get_public_test.go — Tests for list handler - sysctl_get_public_test.go — Tests for get handler - sysctl_set_public_test.go — Tests for set handler - sysctl_delete_public_test.go — Tests for delete handler - gen/ - api.yaml — OpenAPI spec - cfg.yaml — oapi-codegen config - generate.go — //go:generate directive - -internal/controller/api/handler_sysctl.go — GetSysctlHandler() method - -pkg/sdk/client/ - sysctl.go — SysctlService methods - sysctl_types.go — SDK result types + gen→SDK conversions - sysctl_public_test.go — SDK service tests - sysctl_types_public_test.go — SDK type conversion tests - -cmd/ - client_node_sysctl.go — Parent command - client_node_sysctl_list.go — list subcommand - client_node_sysctl_get.go — get subcommand - client_node_sysctl_set.go — set subcommand - client_node_sysctl_delete.go — delete subcommand - -examples/sdk/client/sysctl.go — SDK example - -test/integration/sysctl_test.go — Integration smoke tests - -docs/docs/sidebar/features/sysctl.md -docs/docs/sidebar/usage/cli/client/node/sysctl/sysctl.md -docs/docs/sidebar/usage/cli/client/node/sysctl/list.md -docs/docs/sidebar/usage/cli/client/node/sysctl/get.md -docs/docs/sidebar/usage/cli/client/node/sysctl/set.md -docs/docs/sidebar/usage/cli/client/node/sysctl/delete.md -``` - -### Modified Files - -``` -pkg/sdk/client/operations.go — Add OpSysctl* constants -pkg/sdk/client/permissions.go — Add PermSysctlRead/Write -pkg/sdk/client/osapi.go — Wire SysctlService into Client -internal/job/types.go — Add OperationSysctl* re-exports -internal/authtoken/permissions.go — Re-export + add to roles -internal/controller/api/types.go — Add sysctlHandler field (if needed) -internal/controller/api/handler.go — Wire in CreateHandlers (if needed) -cmd/controller_setup.go — Append GetSysctlHandler + register provider -cmd/agent_setup.go — Create + register sysctl provider -internal/controller/api/handler_public_test.go — Add TestGetSysctlHandler -docs/docusaurus.config.ts — Add to Features navbar -docs/docs/sidebar/usage/configuration.md — Add sysctl permissions -``` - ---- - -## Task 1: SDK Constants (Operations + Permissions) - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `pkg/sdk/client/permissions.go` - -- [ ] **Step 1: Add sysctl operation constants** - -In `pkg/sdk/client/operations.go`, add after the Schedule/Cron block: - -```go -// Sysctl operations. -const ( - OpSysctlList JobOperation = "node.sysctl.list" - OpSysctlGet JobOperation = "node.sysctl.get" - OpSysctlSet JobOperation = "node.sysctl.set" - OpSysctlDelete JobOperation = "node.sysctl.delete" -) -``` - -- [ ] **Step 2: Add sysctl permission constants** - -In `pkg/sdk/client/permissions.go`, add after `PermCronWrite`: - -```go - PermSysctlRead Permission = "sysctl:read" - PermSysctlWrite Permission = "sysctl:write" -``` - -- [ ] **Step 3: Re-export in internal/job/types.go** - -Add after the Schedule/Cron operations block: - -```go -// Sysctl operations. -const ( - OperationSysctlList = client.OpSysctlList - OperationSysctlGet = client.OpSysctlGet - OperationSysctlSet = client.OpSysctlSet - OperationSysctlDelete = client.OpSysctlDelete -) -``` - -- [ ] **Step 4: Re-export permissions in internal/authtoken/permissions.go** - -Add constants: - -```go - PermSysctlRead = client.PermSysctlRead - PermSysctlWrite = client.PermSysctlWrite -``` - -Add to `AllPermissions` slice: - -```go - PermSysctlRead, - PermSysctlWrite, -``` - -Add to `DefaultRolePermissions`: - -- `RoleAdmin`: add `PermSysctlRead, PermSysctlWrite` -- `RoleWrite`: add `PermSysctlRead, PermSysctlWrite` -- `RoleRead`: add `PermSysctlRead` - -- [ ] **Step 5: Verify it compiles** - -Run: `go build ./...` Expected: clean build - -- [ ] **Step 6: Commit** - -```bash -git add pkg/sdk/client/operations.go pkg/sdk/client/permissions.go \ - internal/job/types.go internal/authtoken/permissions.go -git commit -m "feat(sysctl): add operation and permission constants" -``` - ---- - -## Task 2: Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/sysctl/types.go` -- Create: `internal/provider/node/sysctl/darwin.go` -- Create: `internal/provider/node/sysctl/linux.go` -- Create: `internal/provider/node/sysctl/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -```go -// Package sysctl provides kernel parameter management via /etc/sysctl.d/. -// It is a meta-provider that delegates file writes to the file provider -// for SHA tracking, idempotency, and drift detection. -package sysctl - -import "context" - -// Provider implements the methods to manage sysctl entries. -type Provider interface { - // List returns all osapi-managed sysctl entries with current runtime values. - List(ctx context.Context) ([]Entry, error) - // Get returns a single sysctl entry by key with current runtime value. - Get(ctx context.Context, key string) (*Entry, error) - // Set deploys a sysctl conf file and applies it. Idempotent. - Set(ctx context.Context, entry Entry) (*SetResult, error) - // Delete removes a managed sysctl conf file and reloads defaults. - Delete(ctx context.Context, key string) (*DeleteResult, error) -} - -// Entry represents a sysctl kernel parameter. -type Entry struct { - Key string `json:"key"` - Value string `json:"value"` -} - -// SetResult represents the outcome of a sysctl set operation. -type SetResult struct { - Key string `json:"key"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// DeleteResult represents the outcome of a sysctl delete operation. -type DeleteResult struct { - Key string `json:"key"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Create darwin.go** - -```go -package sysctl - -import ( - "context" - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// Darwin implements the sysctl Provider interface for macOS. -// All methods return ErrUnsupported. -type Darwin struct{} - -// NewDarwinProvider factory to create a new Darwin instance. -func NewDarwinProvider() *Darwin { - return &Darwin{} -} - -// List returns ErrUnsupported on macOS. -func (d *Darwin) List( - _ context.Context, -) ([]Entry, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} - -// Get returns ErrUnsupported on macOS. -func (d *Darwin) Get( - _ context.Context, - _ string, -) (*Entry, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} - -// Set returns ErrUnsupported on macOS. -func (d *Darwin) Set( - _ context.Context, - _ Entry, -) (*SetResult, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} - -// Delete returns ErrUnsupported on macOS. -func (d *Darwin) Delete( - _ context.Context, - _ string, -) (*DeleteResult, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 3: Create linux.go** (same pattern as darwin.go with Linux - struct/factory) - -```go -package sysctl - -import ( - "context" - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// Linux implements the sysctl Provider interface for generic Linux. -// All methods return ErrUnsupported. -type Linux struct{} - -// NewLinuxProvider factory to create a new Linux instance. -func NewLinuxProvider() *Linux { - return &Linux{} -} - -// List returns ErrUnsupported on generic Linux. -func (l *Linux) List( - _ context.Context, -) ([]Entry, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} - -// Get returns ErrUnsupported on generic Linux. -func (l *Linux) Get( - _ context.Context, - _ string, -) (*Entry, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} - -// Set returns ErrUnsupported on generic Linux. -func (l *Linux) Set( - _ context.Context, - _ Entry, -) (*SetResult, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} - -// Delete returns ErrUnsupported on generic Linux. -func (l *Linux) Delete( - _ context.Context, - _ string, -) (*DeleteResult, error) { - return nil, fmt.Errorf("sysctl: %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 4: Create mocks/generate.go** - -```go -// Package mocks contains generated mocks for the sysctl provider. -package mocks - -//go:generate go tool github.com/golang/mock/mockgen -source=../types.go -destination=provider.gen.go -package=mocks -``` - -- [ ] **Step 5: Generate mocks** - -Run: `go generate ./internal/provider/node/sysctl/mocks/...` Expected: -`mocks/provider.gen.go` created - -- [ ] **Step 6: Verify it compiles** - -Run: `go build ./...` Expected: clean build - -- [ ] **Step 7: Commit** - -```bash -git add internal/provider/node/sysctl/ -git commit -m "feat(sysctl): add provider interface and platform stubs" -``` - ---- - -## Task 3: Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/sysctl/debian.go` -- Create: `internal/provider/node/sysctl/export_test.go` -- Test: `internal/provider/node/sysctl/debian_public_test.go` -- Test: `internal/provider/node/sysctl/darwin_public_test.go` -- Test: `internal/provider/node/sysctl/linux_public_test.go` - -The Debian provider is a meta-provider. It: - -- Generates conf file content from key/value pairs -- Delegates file writes to `file.Deployer` -- Stores `key` and `value` in `FileState.Metadata` -- Uses `exec.Manager` to run `sysctl -p` and `sysctl -n` -- Scans file-state KV to list managed entries - -- [ ] **Step 1: Write failing tests for Darwin and Linux stubs** - -Create `internal/provider/node/sysctl/darwin_public_test.go`: - -```go -package sysctl_test - -import ( - "context" - "testing" - - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/provider" - "github.com/osapi-io/osapi/internal/provider/node/sysctl" -) - -type DarwinPublicTestSuite struct { - suite.Suite - provider *sysctl.Darwin -} - -func (s *DarwinPublicTestSuite) SetupTest() { - s.provider = sysctl.NewDarwinProvider() -} - -func (s *DarwinPublicTestSuite) TestAllMethodsReturnErrUnsupported() { - tests := []struct { - name string - fn func() error - }{ - { - name: "List", - fn: func() error { - _, err := s.provider.List(context.Background()) - return err - }, - }, - { - name: "Get", - fn: func() error { - _, err := s.provider.Get(context.Background(), "net.ipv4.ip_forward") - return err - }, - }, - { - name: "Set", - fn: func() error { - _, err := s.provider.Set(context.Background(), sysctl.Entry{ - Key: "net.ipv4.ip_forward", - Value: "1", - }) - return err - }, - }, - { - name: "Delete", - fn: func() error { - _, err := s.provider.Delete(context.Background(), "net.ipv4.ip_forward") - return err - }, - }, - } - - for _, tt := range tests { - s.Run(tt.name, func() { - err := tt.fn() - s.Require().Error(err) - s.ErrorIs(err, provider.ErrUnsupported) - }) - } -} - -func TestDarwinPublicTestSuite(t *testing.T) { - suite.Run(t, new(DarwinPublicTestSuite)) -} -``` - -Create `internal/provider/node/sysctl/linux_public_test.go` with the same -pattern using `Linux`/`NewLinuxProvider`. - -- [ ] **Step 2: Run stub tests to verify they pass** - -Run: `go test -v ./internal/provider/node/sysctl/...` Expected: all pass - -- [ ] **Step 3: Write failing tests for Debian provider** - -Create `internal/provider/node/sysctl/debian_public_test.go` with test suite: - -```go -package sysctl_test - -type DebianPublicTestSuite struct { - suite.Suite - mockCtrl *gomock.Controller - mockDeployer *filemocks.MockDeployer - mockExec *execmocks.MockManager - mockStateKV *natsmocks.MockKeyValue - provider *sysctl.Debian - ctx context.Context -} -``` - -Tests to cover for each method: - -**TestList:** - -- success with managed entries (mock stateKV.ListKeys, stateKV.Get for each, - exec for runtime values) -- empty list (no managed keys) -- stateKV.ListKeys error -- exec error reading runtime value (still returns entry with empty value) - -**TestGet:** - -- success (entry exists in stateKV, runtime value read via exec) -- not found (stateKV returns no entry for the key) -- exec error reading value - -**TestSet:** - -- success creates new entry (deploy returns Changed=true, exec sysctl -p - succeeds) -- idempotent no change (deploy returns Changed=false, skip sysctl -p) -- deploy error -- sysctl -p error (file deployed but apply failed) - -**TestDelete:** - -- success (undeploy succeeds, sysctl --system succeeds) -- not found (entry not in stateKV) -- undeploy error -- sysctl --system error - -Each method is ONE suite method with all scenarios as table rows. - -- [ ] **Step 4: Run tests to verify they fail** - -Run: `go test -v ./internal/provider/node/sysctl/...` Expected: FAIL — `Debian` -type does not exist yet - -- [ ] **Step 5: Implement debian.go** - -Create `internal/provider/node/sysctl/debian.go`: - -Key implementation details: - -- Struct embeds `provider.FactsAware`, holds `logger`, `fs`, - `fileDeployer file.Deployer`, `stateKV jetstream.KeyValue`, - `execManager exec.Manager`, `hostname string` -- Compile-time checks: `var _ Provider = (*Debian)(nil)` and - `var _ provider.FactsSetter = (*Debian)(nil)` -- `NewDebianProvider(logger, fs, fileDeployer, stateKV, execManager, hostname) *Debian` -- `confPath(key)` returns `/etc/sysctl.d/osapi-{key}.conf` -- `confContent(key, value)` returns `{key} = {value}\n` -- `buildMetadata(entry)` returns - `map[string]string{"key": entry.Key, "value": entry.Value}` -- `isManagedFile(stateKey)` checks stateKV for existence and metadata containing - "key" -- `Set` deploys via `file.Deployer.Deploy()` with `ContentType: "raw"`, then - runs `sysctl -p ` if changed -- `Delete` undeploys via `file.Deployer.Undeploy()`, then runs `sysctl --system` -- `List` scans stateKV keys matching hostname prefix, filters for sysctl - metadata, reads runtime values -- `Get` looks up single key in stateKV, reads runtime value - -For deploying, the content is generated inline (not from object store), so use a -`DeployRequest` with the content embedded. Check how the cron provider generates -content — it uses `ObjectName` pointing to an object store entry. For sysctl, -the content is trivial (`key = value\n`), so we need to check if `file.Deployer` -supports inline content or if we need to write to the object store first. - -Read `internal/provider/file/types.go` and `internal/provider/file/deploy.go` to -understand the `DeployRequest` fields. The `ObjectName` field references an -object in NATS Object Store. For sysctl, we may need to: - -1. Write the content to a temp object in the object store, then deploy from it, - OR -2. Add support for inline content in `DeployRequest` - -Since the cron provider always deploys from object store objects, and the sysctl -content is a single line, the simplest approach is to write the content directly -to the filesystem (bypassing `file.Deployer`) and manage state in the KV -manually. However, that loses SHA tracking and idempotency. - -**Alternative approach**: The Debian provider can write the conf file directly -using `avfs.VFS`, track state in the file-state KV itself, and use `sysctl -p` -to apply. This is simpler than using `file.Deployer` for trivial content. The -provider manages its own state entries in the same KV bucket using the same key -format (`hostname.sha256(path)`). - -The implementer should read the cron provider's `debian.go` carefully to decide -which approach fits best. If `file.Deployer.Deploy()` requires an object store -reference, use the direct-write approach with manual KV state tracking. If it -can accept inline content, use the deployer. - -- [ ] **Step 6: Create export_test.go for testing internals** - -```go -package sysctl - -// SetConfPath overrides the confPath function for testing. -var SetConfPath = func(fn func(string) string) { - confPathFn = fn -} - -// ResetConfPath restores the default confPath function. -var ResetConfPath = func() { - confPathFn = defaultConfPath -} -``` - -Adjust as needed based on which internal functions need test injection. - -- [ ] **Step 7: Run all tests** - -Run: `go test -v ./internal/provider/node/sysctl/...` Expected: all pass - -- [ ] **Step 8: Check coverage** - -Run: `go test -coverprofile=cover.out ./internal/provider/node/sysctl/...` -Expected: 100% on non-generated code - -- [ ] **Step 9: Commit** - -```bash -git add internal/provider/node/sysctl/ -git commit -m "feat(sysctl): implement Debian meta-provider with tests" -``` - ---- - -## Task 4: Agent Processor - -**Files:** - -- Create: `internal/agent/processor_sysctl.go` -- Test: `internal/agent/processor_sysctl_public_test.go` - -- [ ] **Step 1: Write failing tests** - -Create `internal/agent/processor_sysctl_public_test.go` following the pattern in -`processor_schedule_public_test.go`: - -Test all operations: - -- `sysctl.list` — success, provider error -- `sysctl.get` — success, unmarshal error, provider error -- `sysctl.set` — success, unmarshal error, provider error -- `sysctl.delete` — success, unmarshal error, provider error -- unknown sub-operation — error - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `go test -run TestProcessorSysctl -v ./internal/agent/...` Expected: FAIL - -- [ ] **Step 3: Implement processor_sysctl.go** - -```go -package agent - -import ( - "context" - "encoding/json" - "fmt" - "log/slog" - "strings" - - "github.com/osapi-io/osapi/internal/job" - "github.com/osapi-io/osapi/internal/provider/node/sysctl" -) - -// NewSysctlProcessor creates a processor for sysctl operations. -func NewSysctlProcessor( - provider sysctl.Provider, - logger *slog.Logger, -) ProcessorFunc { - return func(req job.Request) (json.RawMessage, error) { - if provider == nil { - return nil, fmt.Errorf("sysctl provider not available") - } - - parts := strings.Split(req.Operation, ".") - if len(parts) < 3 { - return nil, fmt.Errorf("invalid sysctl operation: %s", req.Operation) - } - subOp := parts[2] - - ctx := context.Background() - - switch subOp { - case "list": - return processSysctlList(ctx, provider, logger) - case "get": - return processSysctlGet(ctx, provider, logger, req) - case "set": - return processSysctlSet(ctx, provider, logger, req) - case "delete": - return processSysctlDelete(ctx, provider, logger, req) - default: - return nil, fmt.Errorf("unsupported sysctl operation: %s", req.Operation) - } - } -} - -func processSysctlList( - ctx context.Context, - provider sysctl.Provider, - logger *slog.Logger, -) (json.RawMessage, error) { - logger.Debug("executing sysctl.list") - - entries, err := provider.List(ctx) - if err != nil { - return nil, err - } - - return json.Marshal(entries) -} - -func processSysctlGet( - ctx context.Context, - provider sysctl.Provider, - logger *slog.Logger, - req job.Request, -) (json.RawMessage, error) { - var data struct { - Key string `json:"key"` - } - if err := json.Unmarshal(req.Data, &data); err != nil { - return nil, fmt.Errorf("unmarshal sysctl get data: %w", err) - } - - logger.Debug("executing sysctl.get", slog.String("key", data.Key)) - - entry, err := provider.Get(ctx, data.Key) - if err != nil { - return nil, err - } - - return json.Marshal(entry) -} - -func processSysctlSet( - ctx context.Context, - provider sysctl.Provider, - logger *slog.Logger, - req job.Request, -) (json.RawMessage, error) { - var entry sysctl.Entry - if err := json.Unmarshal(req.Data, &entry); err != nil { - return nil, fmt.Errorf("unmarshal sysctl set data: %w", err) - } - - logger.Debug("executing sysctl.set", slog.String("key", entry.Key)) - - result, err := provider.Set(ctx, entry) - if err != nil { - return nil, err - } - - return json.Marshal(result) -} - -func processSysctlDelete( - ctx context.Context, - provider sysctl.Provider, - logger *slog.Logger, - req job.Request, -) (json.RawMessage, error) { - var data struct { - Key string `json:"key"` - } - if err := json.Unmarshal(req.Data, &data); err != nil { - return nil, fmt.Errorf("unmarshal sysctl delete data: %w", err) - } - - logger.Debug("executing sysctl.delete", slog.String("key", data.Key)) - - result, err := provider.Delete(ctx, data.Key) - if err != nil { - return nil, err - } - - return json.Marshal(result) -} -``` - -- [ ] **Step 4: Run tests** - -Run: `go test -run TestProcessorSysctl -v ./internal/agent/...` Expected: all -pass - -- [ ] **Step 5: Commit** - -```bash -git add internal/agent/processor_sysctl.go internal/agent/processor_sysctl_public_test.go -git commit -m "feat(sysctl): add agent processor with tests" -``` - ---- - -## Task 5: Agent Wiring - -**Files:** - -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Add sysctl provider creation in setupAgent** - -In `cmd/agent_setup.go`, after the cron provider creation -(`createCronProvider`), add: - -```go - // --- Sysctl provider --- - sysctlProvider := createSysctlProvider(log, appFs, fileProvider, fileStateKV, execManager, hostname) -``` - -Add the `createSysctlProvider` helper function: - -```go -func createSysctlProvider( - log *slog.Logger, - fs avfs.VFS, - fileProvider fileProv.Provider, - fileStateKV jetstream.KeyValue, - execManager exec.Manager, - hostname string, -) sysctlProv.Provider { - plat := platform.Detect() - - switch plat { - case "debian": - if fileProvider == nil { - log.Warn("file provider not available, sysctl operations disabled") - return sysctlProv.NewLinuxProvider() - } - return sysctlProv.NewDebianProvider(log, fs, fileProvider, fileStateKV, execManager, hostname) - case "darwin": - return sysctlProv.NewDarwinProvider() - default: - return sysctlProv.NewLinuxProvider() - } -} -``` - -Add the import: - -```go -sysctlProv "github.com/osapi-io/osapi/internal/provider/node/sysctl" -``` - -Note: `execManager` is already created earlier in `setupAgent` as -`execManager := exec.New(log)`. If it doesn't exist, check if it's named -`execManager` or just created inline. The sysctl provider needs it for -`sysctl -p` and `sysctl -n` commands. - -- [ ] **Step 2: Register sysctl processor in the registry** - -After `registry.Register("schedule", ...)`, add: - -```go - registry.Register("sysctl", - agent.NewSysctlProcessor(sysctlProvider, log), - sysctlProvider, - ) -``` - -Note: The sysctl operations use `node.sysctl.*` format. The registry dispatches -on the first segment. Since the operation format is `node.sysctl.list`, the -registry key should match how the processor splits the operation. Check if the -node processor already handles `node.*` operations. If so, the sysctl processor -should be integrated INTO the node processor OR use a different category key. -Read `internal/agent/processor_node.go` to confirm. - -If the node processor dispatches `node.hostname.*`, `node.disk.*`, etc., then -sysctl should be added as another case in the node processor rather than a -separate registry entry. In that case, modify `processor_node.go` to handle -`sysctl` sub-operations and delegate to the sysctl provider. The registry key -would remain `"node"`. - -The implementer MUST read `internal/agent/processor_node.go` to determine the -correct integration approach. - -- [ ] **Step 3: Verify it compiles** - -Run: `go build ./...` Expected: clean build - -- [ ] **Step 4: Commit** - -```bash -git add cmd/agent_setup.go internal/agent/processor_node.go # or processor_sysctl.go -git commit -m "feat(sysctl): wire provider into agent" -``` - ---- - -## Task 6: OpenAPI Spec + Code Generation - -**Files:** - -- Create: `internal/controller/api/sysctl/gen/api.yaml` -- Create: `internal/controller/api/sysctl/gen/cfg.yaml` -- Create: `internal/controller/api/sysctl/gen/generate.go` - -- [ ] **Step 1: Create api.yaml** - -Follow the cron spec structure -(`internal/controller/api/schedule/gen/api.yaml`). - -Paths: - -- `GET /node/{hostname}/sysctl` — list, security `sysctl:read`, responses - 200/401/403/500 -- `POST /node/{hostname}/sysctl` — set, security `sysctl:write`, responses - 200/400/401/403/500 -- `GET /node/{hostname}/sysctl/{key}` — get, security `sysctl:read`, responses - 200/401/403/404/500 -- `DELETE /node/{hostname}/sysctl/{key}` — delete, security `sysctl:write`, - responses 200/401/403/404/500 - -Parameters: - -- `Hostname` — same as cron spec (reuse `$ref` to common) -- `SysctlKey` — path param, `type: string`, `minLength: 1`, - `pattern: '^[a-z0-9._]+$'` - -Request schemas: - -- `SysctlSetRequest` — required: `key`, `value`. Both strings with validate - tags. - -Response schemas: - -- `SysctlEntry` — hostname (required), status (required, enum - ok/failed/skipped), key, value, error -- `SysctlMutationResult` — hostname (required), status (required, enum - ok/failed/skipped), key, changed, error -- `SysctlCollectionResponse` — job_id (uuid), results (array of SysctlEntry) -- `SysctlGetResponse` — same structure as collection -- `SysctlSetResponse` — job_id (uuid), results (array of SysctlMutationResult) -- `SysctlDeleteResponse` — job_id (uuid), results (array of - SysctlMutationResult) - -- [ ] **Step 2: Create cfg.yaml** - -```yaml ---- -package: gen -output: sysctl.gen.go -generate: - models: true - echo-server: true - strict-server: true -import-mapping: - ../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/controller/api/common/gen -output-options: - skip-prune: true -``` - -- [ ] **Step 3: Create generate.go** - -```go -// Package gen contains generated code for the sysctl API. -package gen - -//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen -config cfg.yaml api.yaml -``` - -- [ ] **Step 4: Generate code** - -Run: `go generate ./internal/controller/api/sysctl/gen/...` Expected: -`sysctl.gen.go` created - -- [ ] **Step 5: Verify it compiles** - -Run: `go build ./internal/controller/api/sysctl/...` Expected: clean build - -- [ ] **Step 6: Commit** - -```bash -git add internal/controller/api/sysctl/gen/ -git commit -m "feat(sysctl): add OpenAPI spec and generate code" -``` - ---- - -## Task 7: API Handler Implementation - -**Files:** - -- Create: `internal/controller/api/sysctl/types.go` -- Create: `internal/controller/api/sysctl/sysctl.go` -- Create: `internal/controller/api/sysctl/validate.go` -- Create: `internal/controller/api/sysctl/sysctl_list_get.go` -- Create: `internal/controller/api/sysctl/sysctl_get.go` -- Create: `internal/controller/api/sysctl/sysctl_set.go` -- Create: `internal/controller/api/sysctl/sysctl_delete.go` -- Test: `internal/controller/api/sysctl/sysctl_list_get_public_test.go` -- Test: `internal/controller/api/sysctl/sysctl_get_public_test.go` -- Test: `internal/controller/api/sysctl/sysctl_set_public_test.go` -- Test: `internal/controller/api/sysctl/sysctl_delete_public_test.go` - -Follow the cron handler pattern exactly. Each handler file: - -1. Validates hostname via `validateHostname()` -2. Validates request body via `validation.Struct()` (for POST) -3. Checks `job.IsBroadcastTarget()` and routes to broadcast function -4. Calls `JobClient.Query` (reads) or `JobClient.Modify` (writes) with category - `"node"` and the sysctl operation constant -5. Handles skipped/failed status -6. Converts provider results to gen response types - -The category string passed to `JobClient.Query/Modify` MUST match the registry -key used in `agent_setup.go`. If sysctl is registered under the `"node"` -category (integrated into the node processor), use `"node"`. If it's a separate -registry entry, use `"sysctl"`. - -- [ ] **Step 1: Create types.go, sysctl.go, validate.go** - -Follow the schedule handler pattern. `validate.go` should include both -`validateHostname` and `validateSysctlKey` (validates the dotted key format). - -- [ ] **Step 2: Write failing tests for list handler** - -Follow `cron_list_get_public_test.go` pattern with table-driven tests covering: - -- success single target -- success broadcast -- skipped status -- query error -- invalid hostname - -Include `TestSysctlListGetHTTP` and `TestSysctlListGetRBACHTTP` methods. - -- [ ] **Step 3: Implement list handler** - -- [ ] **Step 4: Run list tests** - -Run: `go test -run TestSysctlListGet -v ./internal/controller/api/sysctl/...` - -- [ ] **Step 5: Repeat steps 2-4 for get, set, delete handlers** - -Each handler test file needs table-driven tests + HTTP wiring + RBAC tests. - -- [ ] **Step 6: Run all handler tests** - -Run: `go test -v ./internal/controller/api/sysctl/...` Expected: all pass - -- [ ] **Step 7: Commit** - -```bash -git add internal/controller/api/sysctl/ -git commit -m "feat(sysctl): implement API handlers with tests" -``` - ---- - -## Task 8: Server Wiring - -**Files:** - -- Create: `internal/controller/api/handler_sysctl.go` -- Modify: `cmd/controller_setup.go` -- Modify: `internal/controller/api/handler_public_test.go` - -- [ ] **Step 1: Create handler_sysctl.go** - -Follow `handler_schedule.go` pattern: - -```go -func (s *Server) GetSysctlHandler( - jobClient client.JobClient, -) []func(e *echo.Echo) { - var tokenManager TokenValidator = authtoken.New(s.logger) - - sysctlHandler := sysctlAPI.New(s.logger, jobClient) - - strictHandler := sysctlGen.NewStrictHandler( - sysctlHandler, - []sysctlGen.StrictMiddlewareFunc{ - func(handler strictecho.StrictEchoHandlerFunc, _ string) strictecho.StrictEchoHandlerFunc { - return scopeMiddleware( - handler, - tokenManager, - s.appConfig.Controller.API.Security.SigningKey, - sysctlGen.BearerAuthScopes, - s.customRoles, - ) - }, - }, - ) - - return []func(e *echo.Echo){ - func(e *echo.Echo) { - sysctlGen.RegisterHandlers(e, strictHandler) - }, - } -} -``` - -- [ ] **Step 2: Wire in controller_setup.go** - -Add after `GetScheduleHandler`: - -```go -handlers = append(handlers, sm.GetSysctlHandler(jc)...) -``` - -Add the interface method to the `HandlerFactory` interface (or wherever -`GetScheduleHandler` is declared): - -```go -GetSysctlHandler(jobClient jobclient.JobClient) []func(e *echo.Echo) -``` - -- [ ] **Step 3: Add test in handler_public_test.go** - -Add `TestGetSysctlHandler` following the `TestGetScheduleHandler` pattern. - -- [ ] **Step 4: Run tests** - -Run: `go test -v ./internal/controller/api/...` Expected: all pass - -- [ ] **Step 5: Verify full build** - -Run: `go build ./...` Expected: clean build - -- [ ] **Step 6: Commit** - -```bash -git add internal/controller/api/handler_sysctl.go \ - internal/controller/api/handler_public_test.go \ - cmd/controller_setup.go -git commit -m "feat(sysctl): wire handler into server" -``` - ---- - -## Task 9: Regenerate Combined Spec - -**Files:** - -- Modified by generation: `internal/controller/api/gen/api.yaml` -- Modified by generation: `pkg/sdk/client/gen/` - -- [ ] **Step 1: Regenerate combined spec** - -Run: `just generate` Expected: combined `api.yaml` includes sysctl paths, SDK -client regenerated - -- [ ] **Step 2: Verify build** - -Run: `go build ./...` Expected: clean build - -- [ ] **Step 3: Commit** - -```bash -git add internal/controller/api/gen/ pkg/sdk/client/gen/ -git commit -m "chore: regenerate combined spec with sysctl endpoints" -``` - ---- - -## Task 10: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/sysctl.go` -- Create: `pkg/sdk/client/sysctl_types.go` -- Modify: `pkg/sdk/client/osapi.go` -- Test: `pkg/sdk/client/sysctl_public_test.go` -- Test: `pkg/sdk/client/sysctl_types_public_test.go` - -- [ ] **Step 1: Create sysctl_types.go** - -Define SDK types (never expose gen types): - -```go -package client - -// SysctlEntryResult represents a sysctl entry from a query operation. -type SysctlEntryResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Key string `json:"key,omitempty"` - Value string `json:"value,omitempty"` - Error string `json:"error,omitempty"` -} - -// SysctlMutationResult represents the result of a sysctl set or delete. -type SysctlMutationResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Key string `json:"key,omitempty"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// SysctlSetOpts contains options for setting a sysctl parameter. -type SysctlSetOpts struct { - // Key is the sysctl parameter name (e.g., "net.ipv4.ip_forward"). Required. - Key string - // Value is the parameter value. Required. - Value string -} -``` - -Add gen→SDK conversion functions following the cron pattern. - -- [ ] **Step 2: Create sysctl.go** - -```go -package client - -// SysctlService provides sysctl management operations. -type SysctlService struct { - client *gen.ClientWithResponses -} -``` - -Methods: `SysctlList`, `SysctlGet`, `SysctlSet`, `SysctlDelete` — each following -the schedule service pattern with proper error checking, nil response guards, -and response wrapping. - -- [ ] **Step 3: Wire into osapi.go** - -Add `Sysctl *SysctlService` to `Client` struct and initialize in `New()`: - -```go -c.Sysctl = &SysctlService{client: httpClient} -``` - -- [ ] **Step 4: Write tests** - -Create `pkg/sdk/client/sysctl_public_test.go` using `httptest.Server` mocks. -Cover all status code paths (200, 400, 401, 403, 404, 500), nil response body, -transport errors. - -Create `pkg/sdk/client/sysctl_types_public_test.go` testing conversion -functions. - -- [ ] **Step 5: Run tests** - -Run: `go test -v ./pkg/sdk/client/...` Expected: all pass, 100% coverage on -sysctl files - -- [ ] **Step 6: Commit** - -```bash -git add pkg/sdk/client/sysctl.go pkg/sdk/client/sysctl_types.go \ - pkg/sdk/client/osapi.go \ - pkg/sdk/client/sysctl_public_test.go pkg/sdk/client/sysctl_types_public_test.go -git commit -m "feat(sysctl): add SDK service with tests" -``` - ---- - -## Task 11: CLI Commands - -**Files:** - -- Create: `cmd/client_node_sysctl.go` -- Create: `cmd/client_node_sysctl_list.go` -- Create: `cmd/client_node_sysctl_get.go` -- Create: `cmd/client_node_sysctl_set.go` -- Create: `cmd/client_node_sysctl_delete.go` - -- [ ] **Step 1: Create parent command** - -```go -var clientNodeSysctlCmd = &cobra.Command{ - Use: "sysctl", - Short: "Manage kernel parameters", -} - -func init() { - clientNodeCmd.AddCommand(clientNodeSysctlCmd) -} -``` - -- [ ] **Step 2: Create list command** - -Follow `client_node_schedule_cron_list.go` pattern: - -- Call `sdkClient.Sysctl.SysctlList(ctx, host)` -- Handle `--json` output -- Build table with fields: KEY, VALUE -- Use `cli.BuildBroadcastTable` + `cli.PrintCompactTable` - -- [ ] **Step 3: Create get command** - -Flags: `--key` (required) - -- Call `sdkClient.Sysctl.SysctlGet(ctx, host, key)` -- Same output pattern - -- [ ] **Step 4: Create set command** - -Flags: `--key` (required), `--value` (required) - -- Build `client.SysctlSetOpts{Key: key, Value: value}` -- Call `sdkClient.Sysctl.SysctlSet(ctx, host, opts)` -- Use `cli.BuildMutationTable` with CHANGED field - -- [ ] **Step 5: Create delete command** - -Flags: `--key` (required) - -- Call `sdkClient.Sysctl.SysctlDelete(ctx, host, key)` -- Mutation table output - -- [ ] **Step 6: Verify CLI builds** - -Run: `go build ./cmd/...` Expected: clean build - -- [ ] **Step 7: Commit** - -```bash -git add cmd/client_node_sysctl*.go -git commit -m "feat(sysctl): add CLI commands" -``` - ---- - -## Task 12: SDK Example - -**Files:** - -- Create: `examples/sdk/client/sysctl.go` - -- [ ] **Step 1: Create example** - -Follow the conventions in CLAUDE.md (one domain per file, self-contained, print -results, handle errors inline, under ~100 lines): - -```go -package main - -import ( - "context" - "fmt" - "log" - - "github.com/osapi-io/osapi/pkg/sdk/client" -) - -func sysctlExample() { - c, err := client.New("http://localhost:8080", - client.WithBearerToken("..."), - ) - if err != nil { - log.Fatalf("create client: %v", err) - } - - ctx := context.Background() - - // List managed sysctl entries - listResp, err := c.Sysctl.SysctlList(ctx, "_any") - if err != nil { - log.Fatalf("sysctl list: %v", err) - } - fmt.Printf("Managed entries: %d\n", len(listResp.Data.Results)) - for _, e := range listResp.Data.Results { - fmt.Printf(" %s = %s\n", e.Key, e.Value) - } - - // Set a sysctl parameter - setResp, err := c.Sysctl.SysctlSet(ctx, "_any", client.SysctlSetOpts{ - Key: "net.ipv4.ip_forward", - Value: "1", - }) - if err != nil { - log.Fatalf("sysctl set: %v", err) - } - if first := setResp.Data.First(); first != nil { - fmt.Printf("Set %s: changed=%v\n", first.Key, first.Changed) - } -} -``` - -- [ ] **Step 2: Commit** - -```bash -git add examples/sdk/client/sysctl.go -git commit -m "feat(sysctl): add SDK example" -``` - ---- - -## Task 13: Documentation - -**Files:** - -- Create: `docs/docs/sidebar/features/sysctl.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/sysctl/sysctl.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/sysctl/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/sysctl/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/sysctl/set.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/sysctl/delete.md` -- Modify: `docs/docusaurus.config.ts` -- Modify: `docs/docs/sidebar/usage/configuration.md` - -- [ ] **Step 1: Create feature page** - -Follow the template from `docs/docs/sidebar/features/cron-management.md`: -overview, how it works, operations table, CLI examples, permissions, platforms. - -- [ ] **Step 2: Create CLI doc pages** - -Parent page with ``, one page per subcommand with usage examples. - -- [ ] **Step 3: Update docusaurus.config.ts** - -Add sysctl to the Features navbar dropdown. - -- [ ] **Step 4: Update configuration.md** - -Add `sysctl:read` and `sysctl:write` to the permissions/roles tables. - -- [ ] **Step 5: Commit** - -```bash -git add docs/ -git commit -m "docs(sysctl): add feature and CLI documentation" -``` - ---- - -## Task 14: Integration Tests - -**Files:** - -- Create: `test/integration/sysctl_test.go` - -- [ ] **Step 1: Create integration smoke tests** - -Follow the `node_test.go` pattern. Guard writes with `skipWrite(s.T())`. - -```go -//go:build integration - -package integration_test - -type SysctlSmokeSuite struct { - suite.Suite -} - -func (s *SysctlSmokeSuite) TestSysctlList() { - // Test --json output, verify results array -} - -func TestSysctlSmokeSuite(t *testing.T) { - suite.Run(t, new(SysctlSmokeSuite)) -} -``` - -- [ ] **Step 2: Commit** - -```bash -git add test/integration/sysctl_test.go -git commit -m "test(sysctl): add integration smoke tests" -``` - ---- - -## Task 15: Final Verification - -- [ ] **Step 1: Regenerate all** - -Run: `just generate` Expected: clean - -- [ ] **Step 2: Build** - -Run: `go build ./...` Expected: clean - -- [ ] **Step 3: Run all unit tests** - -Run: `just go::unit` Expected: all pass - -- [ ] **Step 4: Check coverage** - -Run: `just go::unit-cov` Expected: >= 99.9% total coverage - -- [ ] **Step 5: Lint** - -Run: `just go::vet` Expected: clean - -- [ ] **Step 6: Format** - -Run: `just go::fmt` Expected: no changes - -- [ ] **Step 7: Ready check** - -Run: `just ready` Expected: all green - -- [ ] **Step 8: Final commit (if formatting changed anything)** - -```bash -git add -A -git commit -m "chore(sysctl): formatting and lint fixes" -``` diff --git a/docs/plans/2026-03-30-api-directory-restructure-design.md b/docs/plans/2026-03-30-api-directory-restructure-design.md deleted file mode 100644 index 42beae946..000000000 --- a/docs/plans/2026-03-30-api-directory-restructure-design.md +++ /dev/null @@ -1,260 +0,0 @@ -# API Directory Restructure Design - -## Overview - -Reorganize `internal/controller/api/` so that all node-targeted API handler -packages live under `api/node/`, matching the URL path structure -(`/node/{hostname}/...`). Read-only single-endpoint handlers stay flat in -`api/node/`. Domains with mutations (CRUD) get their own subdirectory with -`gen/`, `types.go`, and handler files. - -## Motivation - -Currently, node-targeted domains are scattered at the top level of `api/`: -`api/sysctl/`, `api/schedule/`, `api/docker/`, while `api/node/` holds hostname, -disk, mem, load, status, uptime, os, DNS, ping, command, and file handlers in -one package. Every one of these routes to `/node/{hostname}/...`, so they belong -under `api/node/`. - -As more domains are added (power, process, user, ntp, ssh), the flat top-level -layout will grow confusing. Nesting under `api/node/` keeps things organized and -makes it clear what's node-targeted vs controller-only. - -## Design Rule - -- **Read-only, single-endpoint handlers** stay flat in `api/node/` (disk, - memory, load, status, uptime, os) -- **Domains with mutations** (create, update, delete) get their own subdirectory - under `api/node/` with `gen/`, `types.go`, handler files, and tests - -## Directory Structure - -### Before - -``` -internal/controller/api/ - handler.go - handler_agent.go - handler_audit.go - handler_docker.go - handler_facts.go - handler_file.go - handler_health.go - handler_node.go - handler_schedule.go - handler_sysctl.go - handler_public_test.go - types.go - agent/ - audit/ - common/ - docker/ ← node-targeted, at top level - facts/ - file/ - health/ - job/ - node/ ← catch-all for many domains - schedule/ ← node-targeted, at top level - sysctl/ ← node-targeted, at top level - gen/ -``` - -### After - -``` -internal/controller/api/ - handler.go - handler_node.go ← flat read-only node handlers - handler_node_hostname.go ← renamed from part of handler_node.go - handler_node_sysctl.go ← renamed from handler_sysctl.go - handler_node_schedule.go ← renamed from handler_schedule.go - handler_node_docker.go ← renamed from handler_docker.go - handler_node_command.go ← new (split from handler_node.go) - handler_node_file.go ← new (split from handler_node.go) - handler_node_network.go ← new (split from handler_node.go) - handler_agent.go - handler_audit.go - handler_facts.go - handler_file.go ← controller-only file CRUD - handler_health.go - handler_job.go ← renamed from handler_node.go job part - handler_public_test.go - types.go - node/ - gen/ ← OpenAPI spec for read-only node endpoints - node.go ← factory for flat read-only handlers - types.go ← shared types - validate.go ← validateHostname (shared by all node subpkgs) - disk_get.go - memory_get.go - load_get.go - status_get.go - uptime_get.go - os_get.go - *_public_test.go - hostname/ ← has PUT, gets own dir - gen/ - types.go - hostname.go - hostname_get.go - hostname_put.go - validate.go - *_public_test.go - sysctl/ ← moved from api/sysctl/ - gen/ - types.go - sysctl.go - sysctl_list_get.go - sysctl_get.go - sysctl_create.go - sysctl_update.go - sysctl_delete.go - validate.go - *_public_test.go - schedule/ ← moved from api/schedule/ - gen/ - types.go - schedule.go - cron_list_get.go - cron_get.go - cron_create.go - cron_update.go - cron_delete.go - validate.go - *_public_test.go - docker/ ← moved from api/docker/ - gen/ - types.go - docker.go - container_list.go - container_create.go - container_inspect.go - container_start.go - container_stop.go - container_remove.go - container_exec.go - container_pull.go - container_image_remove.go - validate.go - *_public_test.go - command/ ← split from api/node/ - gen/ - types.go - command.go - exec_post.go - shell_post.go - validate.go - *_public_test.go - file/ ← split from api/node/ (deploy/undeploy/status) - gen/ - types.go - file.go - deploy_post.go - undeploy_post.go - status_post.go - validate.go - *_public_test.go - network/ ← split from api/node/ (dns/ping) - gen/ - types.go - network.go - dns_get.go - dns_put.go - ping_post.go - validate.go - *_public_test.go - agent/ ← unchanged - audit/ ← unchanged - common/ ← unchanged - facts/ ← unchanged - file/ ← unchanged (controller-only file CRUD) - health/ ← unchanged - job/ ← unchanged - gen/ ← combined spec -``` - -## Handler Shim Pattern - -Shims stay in `api/` as methods on `Server`. They are renamed to match the -nested path: - -| Before | After | -| --------------------- | ----------------------------- | -| `handler_node.go` | split into multiple files | -| `handler_sysctl.go` | `handler_node_sysctl.go` | -| `handler_schedule.go` | `handler_node_schedule.go` | -| `handler_docker.go` | `handler_node_docker.go` | -| (in handler_node.go) | `handler_node_hostname.go` | -| (in handler_node.go) | `handler_node_command.go` | -| (in handler_node.go) | `handler_node_file.go` | -| (in handler_node.go) | `handler_node_network.go` | -| (new) | `handler_node.go` (read-only) | - -Shims stay in `api/` because: - -- They are methods on `Server` which owns middleware config -- Moving them into domain packages would couple domains to auth/middleware - internals -- Domain packages stay clean: only know about `JobClient` and their own `gen` - types - -## OpenAPI Spec Split - -The current monolithic `api/node/gen/api.yaml` must be split. Each subdirectory -gets its own `gen/api.yaml` containing only its endpoints. The read-only node -endpoints (disk, mem, load, status, uptime, os) stay in `api/node/gen/api.yaml`. - -The combined spec (`api/gen/api.yaml`) is still generated by `redocly join` from -all individual specs — no change to how it works, just more input specs. - -## Validation Sharing - -`validateHostname()` is needed by every node sub-package. Options: - -1. Each sub-package defines its own copy (current cron/sysctl pattern — simple, - no cross-package imports) -2. Put it in `api/node/validate.go` and import from sub-packages - -Option 1 is simpler and avoids circular imports. Each sub-package already has -its own `validate.go` with `validateHostname()`. This is a one-liner that calls -`validation.Var()` — duplication is acceptable. - -## Controller-Only Packages - -These packages are NOT under `/node/{hostname}/` and stay at the top level: - -| Package | Routes | -| --------- | ------------------------------------ | -| `agent/` | `/node` (list/get/drain/undrain) | -| `job/` | `/job/...` | -| `health/` | `/health/...` | -| `audit/` | `/audit/...` | -| `file/` | `/file/...` (upload/list/get/delete) | -| `facts/` | `/facts/...` | - -## What Does NOT Change - -- Provider directory structure — stays as-is -- Agent processor structure — stays as-is -- SDK client structure — stays as-is -- CLI command structure — stays as-is -- Job operation constants — stay as-is -- URL paths — stay exactly the same -- Handler logic — no changes to handler implementations, only package paths and - imports - -## Migration Approach - -This is a purely mechanical refactor: - -1. Create new directory structure -2. Move files, update package declarations -3. Update imports across the codebase -4. Split the monolithic `api/node/gen/api.yaml` into per-domain specs -5. Regenerate all specs -6. Rename shim files -7. Update `controller_setup.go` method names -8. Run tests, lint, verify - -No logic changes anywhere. Every handler, test, and validation function stays -identical — only the package path changes. diff --git a/docs/plans/2026-03-30-api-directory-restructure.md b/docs/plans/2026-03-30-api-directory-restructure.md deleted file mode 100644 index 708d1e3fc..000000000 --- a/docs/plans/2026-03-30-api-directory-restructure.md +++ /dev/null @@ -1,834 +0,0 @@ -# API Directory Restructure Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Move all node-targeted API handler packages under `api/node/` to -mirror the URL path structure, splitting the monolithic `api/node/` package into -domain-specific sub-packages. - -**Architecture:** Each node-targeted domain gets its own sub-package under -`api/node/` with its own `gen/` directory and OpenAPI spec. Domains that already -have their own packages (sysctl, schedule, docker) are moved. Domains currently -embedded in the monolithic `api/node/` package (hostname, network, command, -file) are split out. Read-only single-GET endpoints (disk, memory, load, status, -uptime, os) remain flat in `api/node/`. Handler shims stay in `api/` with names -updated to match the nested path. - -**Tech Stack:** Go 1.25, Echo, oapi-codegen (strict-server), redocly join - -**Coverage baseline:** 99.9% — must remain at or above this. - ---- - -## File Map - -### Packages That Move (directory rename + import update) - -``` -api/sysctl/ → api/node/sysctl/ -api/schedule/ → api/node/schedule/ -api/docker/ → api/node/docker/ -``` - -### Packages That Split Out of api/node/ - -``` -api/node/hostname/ ← new (from node_hostname_get.go, node_hostname_put.go) -api/node/network/ ← new (from network_dns_*.go, network_ping_*.go) -api/node/command/ ← new (from command_exec_post.go, command_shell_post.go) -api/node/file/ ← new (from file_deploy_post.go, file_undeploy_post.go, file_status_post.go) -``` - -### Files Remaining in api/node/ (read-only, flat) - -``` -api/node/ - gen/ ← slimmed OpenAPI spec (disk, mem, load, status, uptime, os only) - node.go ← factory (slimmed — only read-only handlers) - types.go ← shared types (slimmed) - validate.go ← validateHostname (kept — each sub-pkg also has its own copy) - node_disk_get.go - node_memory_get.go - node_load_get.go - node_status_get.go - node_uptime_get.go - node_os_get.go - export_test.go - *_public_test.go ← tests for the above -``` - -### Handler Shim Renames in api/ - -``` -handler_node.go → handler_node.go (slimmed — read-only only) - + handler_node_hostname.go (new) - + handler_node_network.go (new) - + handler_node_command.go (new) - + handler_node_file.go (new) -handler_sysctl.go → handler_node_sysctl.go (import path change) -handler_schedule.go → handler_node_schedule.go (import path change) -handler_docker.go → handler_node_docker.go (import path change) -``` - -### Unchanged Packages - -``` -api/agent/ api/audit/ api/common/ -api/facts/ api/file/ api/health/ -api/job/ api/gen/ -``` - ---- - -## Task Order - -The safest approach is to do each domain independently so the project compiles -and tests pass after every task. Order: - -1. Move sysctl (already its own package — simplest) -2. Move schedule (already its own package) -3. Move docker (already its own package) -4. Split hostname out of node/ -5. Split network out of node/ -6. Split command out of node/ -7. Split file out of node/ -8. Slim down remaining node/ package -9. Regenerate combined spec -10. Update CLAUDE.md -11. Final verification - ---- - -## Task 1: Move sysctl under node/ - -**Files:** - -- Move: `internal/controller/api/sysctl/` → - `internal/controller/api/node/sysctl/` -- Rename: `internal/controller/api/handler_sysctl.go` → - `internal/controller/api/handler_node_sysctl.go` -- Modify: all files that import `api/sysctl` or `api/sysctl/gen` -- Modify: `internal/controller/api/handler_public_test.go` - -- [ ] **Step 1: Move the directory** - -```bash -git mv internal/controller/api/sysctl internal/controller/api/node/sysctl -``` - -- [ ] **Step 2: Rename the handler shim** - -```bash -git mv internal/controller/api/handler_sysctl.go internal/controller/api/handler_node_sysctl.go -``` - -- [ ] **Step 3: Update import paths** - -Search the entire codebase for the old import path and update: - -```bash -# Find all files importing the old path -grep -r "controller/api/sysctl" --include="*.go" . -``` - -Update every occurrence: - -- `github.com/osapi-io/osapi/internal/controller/api/sysctl` → - `github.com/osapi-io/osapi/internal/controller/api/node/sysctl` -- `github.com/osapi-io/osapi/internal/controller/api/sysctl/gen` → - `github.com/osapi-io/osapi/internal/controller/api/node/sysctl/gen` - -Files that will need import updates: - -- `internal/controller/api/handler_node_sysctl.go` -- `internal/controller/api/handler_public_test.go` -- `cmd/controller_setup.go` - -- [ ] **Step 4: Update the import-mapping in sysctl's cfg.yaml** - -The `cfg.yaml` has a relative import-mapping for the common spec. After moving -one level deeper, the relative path changes: - -Read `internal/controller/api/node/sysctl/gen/cfg.yaml` and update: - -```yaml -import-mapping: - ../../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/controller/api/common/gen -``` - -(Was `../../common/gen/api.yaml` — now one level deeper.) - -Also update the `$ref` paths in `api.yaml` that reference -`../../common/gen/api.yaml` — they become `../../../common/gen/api.yaml`. - -- [ ] **Step 5: Rename the handler method** - -In `handler_node_sysctl.go`, rename `GetSysctlHandler` to -`GetNodeSysctlHandler`. - -Update the call site in `cmd/controller_setup.go`: - -```go -// Before: -handlers = append(handlers, sm.GetSysctlHandler(jc)...) -// After: -handlers = append(handlers, sm.GetNodeSysctlHandler(jc)...) -``` - -Update the interface in `cmd/controller_setup.go` if `GetSysctlHandler` is -declared there. - -Update the test in `handler_public_test.go`: - -- Rename `TestGetSysctlHandler` → `TestGetNodeSysctlHandler` -- Update the method call inside the test - -- [ ] **Step 6: Regenerate the sysctl spec** - -```bash -go generate ./internal/controller/api/node/sysctl/gen/... -``` - -- [ ] **Step 7: Verify build and tests** - -```bash -go build ./... -go test ./internal/controller/api/... ./cmd/... -``` - -- [ ] **Step 8: Commit** - -```bash -git add -A -git commit -m "refactor(api): move sysctl handlers under node/" -``` - ---- - -## Task 2: Move schedule under node/ - -**Files:** - -- Move: `internal/controller/api/schedule/` → - `internal/controller/api/node/schedule/` -- Rename: `internal/controller/api/handler_schedule.go` → - `internal/controller/api/handler_node_schedule.go` - -Follow the exact same steps as Task 1: - -- [ ] **Step 1: Move the directory** - -```bash -git mv internal/controller/api/schedule internal/controller/api/node/schedule -``` - -- [ ] **Step 2: Rename the handler shim** - -```bash -git mv internal/controller/api/handler_schedule.go internal/controller/api/handler_node_schedule.go -``` - -- [ ] **Step 3: Update import paths** - -Search for `controller/api/schedule` and update to -`controller/api/node/schedule` in all `.go` files. Files that will need updates: - -- `internal/controller/api/handler_node_schedule.go` -- `internal/controller/api/handler_public_test.go` -- `cmd/controller_setup.go` - -- [ ] **Step 4: Update cfg.yaml and api.yaml relative paths** - -In `internal/controller/api/node/schedule/gen/cfg.yaml`, update the -import-mapping relative path (one level deeper). - -In `internal/controller/api/node/schedule/gen/api.yaml`, update all `$ref` paths -to common spec (one level deeper). - -- [ ] **Step 5: Rename the handler method** - -In `handler_node_schedule.go`: - -- Rename `GetScheduleHandler` → `GetNodeScheduleHandler` - -Update in `cmd/controller_setup.go`: - -```go -handlers = append(handlers, sm.GetNodeScheduleHandler(jc)...) -``` - -Update the interface declaration if it exists. - -Update in `handler_public_test.go`: - -- Rename `TestGetScheduleHandler` → `TestGetNodeScheduleHandler` - -- [ ] **Step 6: Regenerate** - -```bash -go generate ./internal/controller/api/node/schedule/gen/... -``` - -- [ ] **Step 7: Verify build and tests** - -```bash -go build ./... -go test ./internal/controller/api/... ./cmd/... -``` - -- [ ] **Step 8: Commit** - -```bash -git add -A -git commit -m "refactor(api): move schedule handlers under node/" -``` - ---- - -## Task 3: Move docker under node/ - -**Files:** - -- Move: `internal/controller/api/docker/` → - `internal/controller/api/node/docker/` -- Rename: `internal/controller/api/handler_docker.go` → - `internal/controller/api/handler_node_docker.go` - -Follow the exact same pattern as Tasks 1-2: - -- [ ] **Step 1: Move the directory** - -```bash -git mv internal/controller/api/docker internal/controller/api/node/docker -``` - -- [ ] **Step 2: Rename the handler shim** - -```bash -git mv internal/controller/api/handler_docker.go internal/controller/api/handler_node_docker.go -``` - -- [ ] **Step 3: Update import paths** - -Search for `controller/api/docker` and update to `controller/api/node/docker`. - -- [ ] **Step 4: Update cfg.yaml and api.yaml relative paths** - -One level deeper for the common spec reference. - -- [ ] **Step 5: Rename the handler method** - -`GetDockerHandler` → `GetNodeDockerHandler` in shim, controller_setup.go, -interface, and test. - -- [ ] **Step 6: Regenerate** - -```bash -go generate ./internal/controller/api/node/docker/gen/... -``` - -- [ ] **Step 7: Verify build and tests** - -```bash -go build ./... -go test ./internal/controller/api/... ./cmd/... -``` - -- [ ] **Step 8: Commit** - -```bash -git add -A -git commit -m "refactor(api): move docker handlers under node/" -``` - ---- - -## Task 4: Split hostname out of node/ - -This is the first split task. The current `api/node/` package has -`node_hostname_get.go` and `node_hostname_put.go` that need to become their own -package at `api/node/hostname/`. - -**Files:** - -- Create: `internal/controller/api/node/hostname/` -- Create: `internal/controller/api/node/hostname/gen/api.yaml` -- Create: `internal/controller/api/node/hostname/gen/cfg.yaml` -- Create: `internal/controller/api/node/hostname/gen/generate.go` -- Move + modify: `node_hostname_get.go` → `hostname/hostname_get.go` -- Move + modify: `node_hostname_put.go` → `hostname/hostname_put.go` -- Move + modify: corresponding `*_public_test.go` files -- Create: `internal/controller/api/node/hostname/types.go` -- Create: `internal/controller/api/node/hostname/hostname.go` -- Create: `internal/controller/api/node/hostname/validate.go` -- Create: `internal/controller/api/handler_node_hostname.go` -- Modify: `internal/controller/api/node/gen/api.yaml` (remove hostname - endpoints) -- Modify: `internal/controller/api/node/node.go` (remove hostname from - interface) - -- [ ] **Step 1: Extract hostname paths from node's api.yaml** - -Read `internal/controller/api/node/gen/api.yaml`. Find the hostname endpoints -(GET and PUT on `/node/{hostname}/hostname`). Extract them into a new -`internal/controller/api/node/hostname/gen/api.yaml`. - -Include the hostname-specific request/response schemas. Reference common schemas -via `$ref` to `../../../common/gen/api.yaml`. - -Remove the hostname endpoints from the node spec. - -- [ ] **Step 2: Create cfg.yaml and generate.go** - -```yaml -# cfg.yaml ---- -package: gen -output: hostname.gen.go -generate: - models: true - echo-server: true - strict-server: true -import-mapping: - ../../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/controller/api/common/gen -output-options: - skip-prune: true -``` - -```go -// generate.go -package gen - -//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen -config cfg.yaml api.yaml -``` - -- [ ] **Step 3: Generate code** - -```bash -go generate ./internal/controller/api/node/hostname/gen/... -``` - -- [ ] **Step 4: Create hostname.go (factory + interface check)** - -```go -package hostname - -import ( - "log/slog" - - client "github.com/osapi-io/osapi/internal/job/client" - gen "github.com/osapi-io/osapi/internal/controller/api/node/hostname/gen" -) - -var _ gen.StrictServerInterface = (*Hostname)(nil) - -func New( - logger *slog.Logger, - jobClient client.JobClient, - appConfig config.Config, -) *Hostname { - return &Hostname{ - JobClient: jobClient, - logger: logger.With(slog.String("subsystem", "api.hostname")), - appConfig: appConfig, - } -} -``` - -Check what the existing hostname handlers need (the PUT handler may need -`appConfig` for labels). Read the existing handler code to determine the -constructor signature. - -- [ ] **Step 5: Create types.go and validate.go** - -```go -// types.go -package hostname - -type Hostname struct { - JobClient client.JobClient - logger *slog.Logger - appConfig config.Config -} -``` - -```go -// validate.go -package hostname - -func validateHostname(hostname string) (string, bool) { - return validation.Var(hostname, "required,min=1,valid_target") -} -``` - -- [ ] **Step 6: Move and update handler files** - -Move `node_hostname_get.go` → `hostname/hostname_get.go`: - -- Change `package node` → `package hostname` -- Update receiver type from `(s *Node)` → `(s *Hostname)` -- Update gen type references from `gen.GetNodeHostname*` to the new generated - types (check the generated interface) -- Update import paths - -Same for `node_hostname_put.go` and all corresponding test files. - -- [ ] **Step 7: Create handler shim** - -Create `internal/controller/api/handler_node_hostname.go` following the existing -shim pattern. Method: `GetNodeHostnameHandler`. - -- [ ] **Step 8: Remove hostname from node's StrictServerInterface** - -The node `gen/api.yaml` no longer has hostname endpoints, so after regeneration -the node's `StrictServerInterface` won't include them. Remove the hostname -methods from `node.go` and update `handler_node.go`. - -Regenerate the node spec: - -```bash -go generate ./internal/controller/api/node/gen/... -``` - -- [ ] **Step 9: Wire in controller_setup.go** - -Add the new handler call: - -```go -handlers = append(handlers, sm.GetNodeHostnameHandler(jc)...) -``` - -- [ ] **Step 10: Update handler_public_test.go** - -Add `TestGetNodeHostnameHandler` test. - -- [ ] **Step 11: Verify build and tests** - -```bash -go build ./... -go test ./internal/controller/api/... ./cmd/... -``` - -- [ ] **Step 12: Commit** - -```bash -git add -A -git commit -m "refactor(api): split hostname handlers into node/hostname/" -``` - ---- - -## Task 5: Split network out of node/ - -Same pattern as Task 4. Extract DNS and ping endpoints. - -**Files to move from api/node/:** - -- `network_dns_get_by_interface.go` -- `network_dns_put_by_interface.go` -- `network_ping_post.go` -- Corresponding `*_public_test.go` files - -**New package:** `internal/controller/api/node/network/` - -- [ ] **Step 1: Extract network paths from node's api.yaml** - -Find DNS (GET/PUT `/node/{hostname}/network/dns/{interfaceName}`) and ping (POST -`/node/{hostname}/network/ping`) endpoints. Create -`api/node/network/gen/api.yaml`. - -Remove from node spec. - -- [ ] **Step 2: Create cfg.yaml, generate.go, generate code** - -- [ ] **Step 3: Create network.go, types.go, validate.go** - -Factory: `New(logger, jobClient) *Network` Struct fields: `JobClient`, `logger` - -- [ ] **Step 4: Move and update handler files** - -Change package, receiver type, gen references, imports. - -- [ ] **Step 5: Create handler shim handler_node_network.go** - -Method: `GetNodeNetworkHandler` - -- [ ] **Step 6: Regenerate node spec, wire in controller_setup.go** - -- [ ] **Step 7: Update handler_public_test.go** - -- [ ] **Step 8: Verify build and tests** - -- [ ] **Step 9: Commit** - -```bash -git add -A -git commit -m "refactor(api): split network handlers into node/network/" -``` - ---- - -## Task 6: Split command out of node/ - -Same pattern. Extract exec and shell endpoints. - -**Files to move:** - -- `command_exec_post.go` -- `command_shell_post.go` -- Corresponding `*_public_test.go` files - -**New package:** `internal/controller/api/node/command/` - -- [ ] **Step 1-8:** Follow the same steps as Task 5. - -Extract POST `/node/{hostname}/command/exec` and POST -`/node/{hostname}/command/shell`. - -Factory: `New(logger, jobClient) *Command` Handler shim: `GetNodeCommandHandler` - -- [ ] **Step 9: Commit** - -```bash -git add -A -git commit -m "refactor(api): split command handlers into node/command/" -``` - ---- - -## Task 7: Split file (deploy/undeploy/status) out of node/ - -Same pattern. Extract the node-targeted file operations (NOT the controller-only -`api/file/` package which stays at the top level). - -**Files to move:** - -- `file_deploy_post.go` -- `file_undeploy_post.go` -- `file_status_post.go` -- Corresponding `*_public_test.go` files - -**New package:** `internal/controller/api/node/filedeploy/` - -Note: Cannot use `api/node/file/` because Go doesn't allow a package `file` -under `node` when there's already an `api/file/` package — the import paths -would be unambiguous (`api/node/file` vs `api/file`) but the package name `file` -would collide in files that import both. Use `filedeploy` to avoid confusion. - -Actually — check if any handler file imports both `api/file` and the node file -handlers. If not, `api/node/file/` is fine since Go resolves by full import -path. The package name would be `file` in both cases but they're different -packages. Import aliases handle any collision: `nodeFile "...api/node/file"`. - -The implementer should check whether `api/node/file/` or `api/node/filedeploy/` -is cleaner. Read existing code to see if there are cross-imports. - -- [ ] **Step 1-8:** Follow the same steps as Task 5. - -Extract POST endpoints for deploy, undeploy, status under -`/node/{hostname}/file/...`. - -Factory: `New(logger, jobClient) *File` Handler shim: `GetNodeFileHandler` - -Rename existing `handler_file.go` (controller-only file CRUD) to be explicit — -it's already named `handler_file.go` and handles `api/file/` which is NOT -moving. Just make sure the new shim is `handler_node_file.go` to avoid -confusion. - -- [ ] **Step 9: Commit** - -```bash -git add -A -git commit -m "refactor(api): split file deploy handlers into node/file/" -``` - ---- - -## Task 8: Slim down remaining node/ package - -After tasks 4-7, the `api/node/` package should only contain the read-only GET -handlers: disk, memory, load, status, uptime, os. - -**Files:** - -- Modify: `internal/controller/api/node/node.go` — slim factory -- Modify: `internal/controller/api/node/types.go` — remove unused types -- Modify: `internal/controller/api/node/gen/api.yaml` — should only have 6 GET - endpoints -- Modify: `internal/controller/api/handler_node.go` — slim to read-only -- Delete: any leftover moved files - -- [ ] **Step 1: Verify node/ only has read-only handlers** - -List files in `internal/controller/api/node/`. Should only have: - -``` -gen/ -node.go -types.go -validate.go -export_test.go -node_disk_get.go -node_memory_get.go -node_load_get.go -node_status_get.go -node_uptime_get.go -node_os_get.go -*_public_test.go (for the above) -``` - -And subdirectories: `hostname/`, `sysctl/`, `schedule/`, `docker/`, `network/`, -`command/`, `file/`. - -- [ ] **Step 2: Regenerate the slimmed node spec** - -```bash -go generate ./internal/controller/api/node/gen/... -``` - -- [ ] **Step 3: Clean up node.go** - -Remove any methods from the `Node` struct that were moved to sub-packages. The -`Node` struct should only implement the slimmed `gen.StrictServerInterface` with -read-only methods. - -- [ ] **Step 4: Clean up types.go** - -Remove any types that are no longer used (they moved to sub-package types.go -files). - -- [ ] **Step 5: Update handler_node.go** - -The `GetNodeHandler` method should only register the read-only node routes. The -other handlers are now separate shims (`GetNodeHostnameHandler`, etc.). - -- [ ] **Step 6: Verify build and tests** - -```bash -go build ./... -go test ./internal/controller/api/... ./cmd/... -``` - -- [ ] **Step 7: Commit** - -```bash -git add -A -git commit -m "refactor(api): slim node package to read-only handlers" -``` - ---- - -## Task 9: Regenerate combined spec - -- [ ] **Step 1: Run just generate** - -```bash -just generate -``` - -This runs `redocly join` across all individual specs to produce the combined -`api/gen/api.yaml` and regenerates the SDK client. - -- [ ] **Step 2: Verify build** - -```bash -go build ./... -``` - -- [ ] **Step 3: Commit** - -```bash -git add -A -git commit -m "chore: regenerate combined spec after API restructure" -``` - ---- - -## Task 10: Update CLAUDE.md - -- [ ] **Step 1: Update the "Adding a New API Domain" guide** - -Update Step 1 (OpenAPI Spec) to reference the nested path: - -``` -internal/controller/api/node/{domain}/gen/ -``` - -Update Step 2 (Handler Implementation) to reference: - -``` -internal/controller/api/node/{domain}/ -``` - -Update Step 3 (Server Wiring) to reference: - -``` -handler_node_{domain}.go -GetNode{Domain}Handler -``` - -Update the file structure examples throughout to show the nested layout. - -- [ ] **Step 2: Update the architecture quick reference** - -Update the `internal/controller/api/` description to reflect the nested -structure. - -- [ ] **Step 3: Commit** - -```bash -git add CLAUDE.md -git commit -m "docs: update CLAUDE.md for nested API directory structure" -``` - ---- - -## Task 11: Final Verification - -- [ ] **Step 1: Full regeneration** - -```bash -just generate -``` - -- [ ] **Step 2: Build** - -```bash -go build ./... -``` - -- [ ] **Step 3: All unit tests** - -```bash -just go::unit -``` - -- [ ] **Step 4: Coverage** - -```bash -just go::unit-cov -``` - -Expected: >= 99.9% - -- [ ] **Step 5: Lint** - -```bash -just go::vet -``` - -- [ ] **Step 6: Format** - -```bash -just go::fmt -``` - -- [ ] **Step 7: Commit any fixes** - -```bash -git add -A -git commit -m "chore: formatting and lint fixes after restructure" -``` diff --git a/docs/plans/2026-03-30-ntp-timezone-provider-design.md b/docs/plans/2026-03-30-ntp-timezone-provider-design.md deleted file mode 100644 index 159465eb1..000000000 --- a/docs/plans/2026-03-30-ntp-timezone-provider-design.md +++ /dev/null @@ -1,365 +0,0 @@ -# NTP + Timezone Provider Design - -## Overview - -Add NTP server management and timezone configuration to OSAPI. Two separate -providers under `provider/node/`, two separate API packages under `api/node/`. -NTP manages chrony server configuration via drop-in files. Timezone reads and -sets the system timezone via timedatectl. - -## NTP Provider - -### Architecture - -Direct provider at `internal/provider/node/ntp/`. Manages a chrony drop-in -config file at `/etc/chrony/sources.d/osapi-ntp.sources`. Reads sync status and -configured sources via `chronyc` commands. Applies changes via -`chronyc reload sources`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/ntp` -- **Permissions**: `ntp:read`, `ntp:write` -- **Provider type**: direct (writes config, runs commands) - -### Provider Interface - -```go -type Provider interface { - Get(ctx context.Context) (*Status, error) - Create(ctx context.Context, config Config) (*CreateResult, error) - Update(ctx context.Context, config Config) (*UpdateResult, error) - Delete(ctx context.Context) (*DeleteResult, error) -} -``` - -### Data Types - -```go -type Config struct { - Servers []string `json:"servers"` -} - -type Status struct { - Synchronized bool `json:"synchronized"` - Stratum int `json:"stratum,omitempty"` - Offset string `json:"offset,omitempty"` - CurrentSource string `json:"current_source,omitempty"` - Servers []string `json:"servers,omitempty"` -} - -type CreateResult struct { - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type UpdateResult struct { - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type DeleteResult struct { - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -### Debian Implementation - -**Config file**: `/etc/chrony/sources.d/osapi-ntp.sources` - -Content format (one server per line): - -``` -server 0.pool.ntp.org iburst -server 1.pool.ntp.org iburst -server time.google.com iburst -``` - -**Operations:** - -- **Get**: Parse `chronyc tracking` for sync state (synchronized, stratum, - offset, current source). Parse `chronyc sources` for configured server list. - Always succeeds — returns current state regardless of whether osapi manages - the config. -- **Create**: Write the drop-in file with the server list. Fail if the osapi - drop-in already exists. Run `chronyc reload sources`. -- **Update**: Overwrite the drop-in file. Fail if the osapi drop-in does not - exist. Idempotent — compare content, skip write if unchanged - (`Changed: false`). Run `chronyc reload sources` if changed. -- **Delete**: Remove the drop-in file. Fail if not found. Run - `chronyc reload sources` to revert to default sources. - -**Idempotency**: SHA-based comparison of generated file content, same approach -as sysctl provider. - -### Container Behavior - -No `DebianDocker` variant needed. NTP is a host-level concern — containers -inherit the host's time. If the agent runs in a container, chronyc is unlikely -to be available and operations return the standard error. - -### API Endpoints - -| Method | Path | Permission | Description | -| -------- | ---------------------- | ----------- | ------------------------- | -| `GET` | `/node/{hostname}/ntp` | `ntp:read` | Get sync status + servers | -| `POST` | `/node/{hostname}/ntp` | `ntp:write` | Create managed NTP config | -| `PUT` | `/node/{hostname}/ntp` | `ntp:write` | Update server list | -| `DELETE` | `/node/{hostname}/ntp` | `ntp:write` | Remove managed config | - -All endpoints support broadcast targeting. - -### Response Shape - -GET response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "synchronized": true, - "stratum": 2, - "offset": "+0.003s", - "current_source": "0.pool.ntp.org", - "servers": ["0.pool.ntp.org", "1.pool.ntp.org"] - } - ] -} -``` - -POST/PUT/DELETE response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "changed": true - } - ] -} -``` - -POST/PUT request body: - -```json -{ - "servers": ["0.pool.ntp.org", "1.pool.ntp.org", "time.google.com"] -} -``` - ---- - -## Timezone Provider - -### Architecture - -Direct provider at `internal/provider/node/timezone/`. Reads and sets the system -timezone via `timedatectl`. No config files — this is a direct system call. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/timezone` -- **Permissions**: `timezone:read`, `timezone:write` -- **Provider type**: direct - -### Provider Interface - -```go -type Provider interface { - Get(ctx context.Context) (*Info, error) - Update(ctx context.Context, timezone string) (*UpdateResult, error) -} -``` - -No Create or Delete — timezone always exists on the system. Only GET (read) and -PUT (update). - -### Data Types - -```go -type Info struct { - Timezone string `json:"timezone"` - UTCOffset string `json:"utc_offset,omitempty"` -} - -type UpdateResult struct { - Timezone string `json:"timezone"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -### Debian Implementation - -- **Get**: Run `timedatectl show -p Timezone --value` for timezone name. Run - `date +%:z` or parse timedatectl output for UTC offset. -- **Update**: Run `timedatectl set-timezone `. Idempotent — read current - timezone first, skip if already set (`Changed: false`). Validate timezone name - against `/usr/share/zoneinfo/` or `timedatectl list-timezones`. - -### Container Behavior - -No `DebianDocker` variant needed. Containers inherit the host timezone. If the -agent runs in a container, `timedatectl` may not be available — operations -return the standard error. - -### API Endpoints - -| Method | Path | Permission | Description | -| ------ | --------------------------- | ---------------- | ------------ | -| `GET` | `/node/{hostname}/timezone` | `timezone:read` | Get timezone | -| `PUT` | `/node/{hostname}/timezone` | `timezone:write` | Set timezone | - -All endpoints support broadcast targeting. - -### Response Shape - -GET response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "timezone": "America/New_York", - "utc_offset": "-04:00" - } - ] -} -``` - -PUT request body: - -```json -{ - "timezone": "America/New_York" -} -``` - -PUT response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "timezone": "America/New_York", - "changed": true - } - ] -} -``` - ---- - -## Platform Implementations - -| Provider | Debian | Darwin | Linux | -| -------- | ----------- | -------------- | -------------- | -| NTP | chronyc | ErrUnsupported | ErrUnsupported | -| Timezone | timedatectl | ErrUnsupported | ErrUnsupported | - ---- - -## Files to Create/Modify - -### New Files - -``` -internal/provider/node/ntp/ - types.go - debian.go - darwin.go - linux.go - mocks/generate.go - -internal/provider/node/timezone/ - types.go - debian.go - darwin.go - linux.go - mocks/generate.go - -internal/agent/processor_ntp.go -internal/agent/processor_timezone.go - -internal/controller/api/node/ntp/ - gen/ (api.yaml, cfg.yaml, generate.go) - types.go - ntp.go - ntp_get.go - ntp_create.go - ntp_update.go - ntp_delete.go - handler.go - validate.go - *_public_test.go - -internal/controller/api/node/timezone/ - gen/ (api.yaml, cfg.yaml, generate.go) - types.go - timezone.go - timezone_get.go - timezone_update.go - handler.go - validate.go - *_public_test.go - -pkg/sdk/client/ - ntp.go - ntp_types.go - timezone.go - timezone_types.go - *_public_test.go - -cmd/ - client_node_ntp.go - client_node_ntp_get.go - client_node_ntp_create.go - client_node_ntp_update.go - client_node_ntp_delete.go - client_node_timezone.go - client_node_timezone_get.go - client_node_timezone_update.go - -examples/sdk/client/ntp.go -examples/sdk/client/timezone.go - -test/integration/ntp_test.go -test/integration/timezone_test.go - -docs/docs/sidebar/features/ntp.md -docs/docs/sidebar/features/timezone.md -docs/docs/sidebar/usage/cli/client/node/ntp/... -docs/docs/sidebar/usage/cli/client/node/timezone/... -docs/docs/sidebar/sdk/client/ntp.md -docs/docs/sidebar/sdk/client/timezone.md -``` - -### Modified Files - -``` -pkg/sdk/client/operations.go — Add OpNtp*, OpTimezone* -pkg/sdk/client/permissions.go — Add PermNtp*, PermTimezone* -pkg/sdk/client/osapi.go — Wire services -internal/job/types.go — Re-export operations -internal/authtoken/permissions.go — Re-export + add to roles -internal/agent/processor.go — Add ntp/timezone to node processor -cmd/agent_setup.go — Create + register providers -cmd/controller_setup.go — Register handlers -docs/docusaurus.config.ts — Add to Features navbar -docs/docs/sidebar/usage/configuration.md — Add permissions -docs/docs/sidebar/features/authentication.md — Add permissions -docs/docs/sidebar/architecture/api-guidelines.md — Add endpoints -docs/docs/sidebar/architecture/architecture.md — Add feature links -CLAUDE.md — Update provider list -``` diff --git a/docs/plans/2026-03-30-ntp-timezone-provider.md b/docs/plans/2026-03-30-ntp-timezone-provider.md deleted file mode 100644 index a77a73d15..000000000 --- a/docs/plans/2026-03-30-ntp-timezone-provider.md +++ /dev/null @@ -1,1019 +0,0 @@ -# NTP + Timezone Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add NTP server management (chrony) and timezone configuration -(timedatectl) as node providers with full API/CLI/SDK support. - -**Architecture:** Two independent direct providers under `provider/node/`. NTP -manages `/etc/chrony/sources.d/osapi.sources` and reads status via `chronyc`. -Timezone reads/sets via `timedatectl`. Both integrate into the node processor, -each with its own API package under `api/node/`, SDK service, and CLI commands. - -**Tech Stack:** Go 1.25, Echo, oapi-codegen (strict-server), NATS JetStream, -gomock, testify/suite, avfs - -**Coverage baseline:** 99.9% — must remain at or above this. - ---- - -## File Map - -### NTP — New Files - -``` -internal/provider/node/ntp/ - types.go — Provider interface + Status, Config, results - debian.go — Chrony implementation - darwin.go — macOS stub (ErrUnsupported) - linux.go — Generic Linux stub (ErrUnsupported) - export_test.go — Expose unexported vars for testing - mocks/generate.go — //go:generate mockgen directive - -internal/agent/processor_ntp.go — NTP operation dispatch - -internal/controller/api/node/ntp/ - gen/ (api.yaml, cfg.yaml, generate.go) - types.go — Handler struct - ntp.go — New() factory + interface check - ntp_get.go — GET handler + broadcast - ntp_create.go — POST handler + broadcast - ntp_update.go — PUT handler + broadcast - ntp_delete.go — DELETE handler + broadcast - handler.go — Handler() self-registration - validate.go — validateHostname - *_public_test.go — Tests for all handlers + RBAC - -pkg/sdk/client/ - ntp.go — NTPService methods - ntp_types.go — SDK result types + conversions - ntp_public_test.go - ntp_types_public_test.go - -cmd/ - client_node_ntp.go — Parent command - client_node_ntp_get.go — get subcommand - client_node_ntp_create.go — create subcommand - client_node_ntp_update.go — update subcommand - client_node_ntp_delete.go — delete subcommand - -examples/sdk/client/ntp.go -test/integration/ntp_test.go -docs/docs/sidebar/features/ntp.md -docs/docs/sidebar/usage/cli/client/node/ntp/ntp.md -docs/docs/sidebar/usage/cli/client/node/ntp/get.md -docs/docs/sidebar/usage/cli/client/node/ntp/create.md -docs/docs/sidebar/usage/cli/client/node/ntp/update.md -docs/docs/sidebar/usage/cli/client/node/ntp/delete.md -docs/docs/sidebar/sdk/client/ntp.md -``` - -### Timezone — New Files - -``` -internal/provider/node/timezone/ - types.go — Provider interface + Info, UpdateResult - debian.go — timedatectl implementation - darwin.go — macOS stub (ErrUnsupported) - linux.go — Generic Linux stub (ErrUnsupported) - mocks/generate.go - -internal/agent/processor_timezone.go - -internal/controller/api/node/timezone/ - gen/ (api.yaml, cfg.yaml, generate.go) - types.go - timezone.go - timezone_get.go — GET handler + broadcast - timezone_update.go — PUT handler + broadcast - handler.go - validate.go - *_public_test.go - -pkg/sdk/client/ - timezone.go - timezone_types.go - timezone_public_test.go - timezone_types_public_test.go - -cmd/ - client_node_timezone.go — Parent command - client_node_timezone_get.go - client_node_timezone_update.go - -examples/sdk/client/timezone.go -test/integration/timezone_test.go -docs/docs/sidebar/features/timezone.md -docs/docs/sidebar/usage/cli/client/node/timezone/timezone.md -docs/docs/sidebar/usage/cli/client/node/timezone/get.md -docs/docs/sidebar/usage/cli/client/node/timezone/update.md -docs/docs/sidebar/sdk/client/timezone.md -``` - -### Modified Files (shared) - -``` -pkg/sdk/client/operations.go — Add OpNtp*, OpTimezone* -pkg/sdk/client/permissions.go — Add PermNtp*, PermTimezone* -pkg/sdk/client/osapi.go — Wire NTP + Timezone services -pkg/sdk/client/export_test.go — Add conversion bridges -internal/job/types.go — Re-export operations -internal/authtoken/permissions.go — Re-export + add to roles -internal/agent/processor.go — Add ntp/timezone cases to node processor -internal/agent/export_test.go — Export new providers for tests -internal/agent/fixture_public_test.go — Add provider params -cmd/agent_setup.go — Create + register providers -cmd/controller_setup.go — Register handlers -docs/docusaurus.config.ts — Add to Features navbar -docs/docs/sidebar/usage/configuration.md — Add permissions -docs/docs/sidebar/features/authentication.md — Add to permissions tables -docs/docs/sidebar/architecture/api-guidelines.md — Add endpoint table -docs/docs/sidebar/architecture/architecture.md — Add feature links -CLAUDE.md — Update provider list -``` - ---- - -## Task 1: SDK Constants (Operations + Permissions) - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/job/types.go` -- Modify: `internal/authtoken/permissions.go` - -- [ ] **Step 1: Add NTP and timezone operation constants** - -In `pkg/sdk/client/operations.go`, add after the Sysctl block: - -```go -// NTP operations. -const ( - OpNtpGet JobOperation = "node.ntp.get" - OpNtpCreate JobOperation = "node.ntp.create" - OpNtpUpdate JobOperation = "node.ntp.update" - OpNtpDelete JobOperation = "node.ntp.delete" -) - -// Timezone operations. -const ( - OpTimezoneGet JobOperation = "node.timezone.get" - OpTimezoneUpdate JobOperation = "node.timezone.update" -) -``` - -- [ ] **Step 2: Add permission constants** - -In `pkg/sdk/client/permissions.go`, add after `PermSysctlWrite`: - -```go - PermNtpRead Permission = "ntp:read" - PermNtpWrite Permission = "ntp:write" - PermTimezoneRead Permission = "timezone:read" - PermTimezoneWrite Permission = "timezone:write" -``` - -- [ ] **Step 3: Re-export in internal/job/types.go** - -Add after the Sysctl operations block: - -```go -// NTP operations. -const ( - OperationNtpGet = client.OpNtpGet - OperationNtpCreate = client.OpNtpCreate - OperationNtpUpdate = client.OpNtpUpdate - OperationNtpDelete = client.OpNtpDelete -) - -// Timezone operations. -const ( - OperationTimezoneGet = client.OpTimezoneGet - OperationTimezoneUpdate = client.OpTimezoneUpdate -) -``` - -- [ ] **Step 4: Re-export permissions in internal/authtoken/permissions.go** - -Add constants, add to `AllPermissions`, add to `DefaultRolePermissions`: - -- `RoleAdmin`: add all four -- `RoleWrite`: add all four -- `RoleRead`: add `PermNtpRead` and `PermTimezoneRead` only - -- [ ] **Step 5: Verify build** - -Run: `go build ./...` - -- [ ] **Step 6: Commit** - -```bash -git add pkg/sdk/client/operations.go pkg/sdk/client/permissions.go \ - internal/job/types.go internal/authtoken/permissions.go -git commit -m "feat(ntp,timezone): add operation and permission constants" -``` - ---- - -## Task 2: NTP Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/ntp/types.go` -- Create: `internal/provider/node/ntp/darwin.go` -- Create: `internal/provider/node/ntp/linux.go` -- Create: `internal/provider/node/ntp/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -```go -// Package ntp provides NTP server management via chrony. -package ntp - -import "context" - -// Provider implements the methods to manage NTP configuration. -type Provider interface { - // Get returns current NTP sync status and configured servers. - Get(ctx context.Context) (*Status, error) - // Create deploys a managed NTP server configuration. Fails if already managed. - Create(ctx context.Context, config Config) (*CreateResult, error) - // Update replaces the managed NTP server configuration. Fails if not managed. - Update(ctx context.Context, config Config) (*UpdateResult, error) - // Delete removes the managed NTP server configuration. - Delete(ctx context.Context) (*DeleteResult, error) -} - -// Config represents an NTP server configuration to deploy. -type Config struct { - Servers []string `json:"servers"` -} - -// Status represents the current NTP sync state and configured servers. -type Status struct { - Synchronized bool `json:"synchronized"` - Stratum int `json:"stratum,omitempty"` - Offset string `json:"offset,omitempty"` - CurrentSource string `json:"current_source,omitempty"` - Servers []string `json:"servers,omitempty"` -} - -// CreateResult represents the outcome of an NTP config create operation. -type CreateResult struct { - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// UpdateResult represents the outcome of an NTP config update operation. -type UpdateResult struct { - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// DeleteResult represents the outcome of an NTP config delete operation. -type DeleteResult struct { - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Create darwin.go and linux.go** - -Both return `fmt.Errorf("ntp: %w", provider.ErrUnsupported)` for all methods. -Follow the sysctl stub pattern exactly. Add license headers. - -- [ ] **Step 3: Create mocks/generate.go** - -```go -package mocks - -//go:generate go tool github.com/golang/mock/mockgen -source=../types.go -destination=provider.gen.go -package=mocks -``` - -- [ ] **Step 4: Generate mocks and verify build** - -```bash -go generate ./internal/provider/node/ntp/mocks/... -go build ./... -``` - -- [ ] **Step 5: Commit** - -```bash -git add internal/provider/node/ntp/ -git commit -m "feat(ntp): add provider interface and platform stubs" -``` - ---- - -## Task 3: NTP Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/ntp/debian.go` -- Create: `internal/provider/node/ntp/export_test.go` -- Create: `internal/provider/node/ntp/debian_public_test.go` -- Create: `internal/provider/node/ntp/darwin_public_test.go` -- Create: `internal/provider/node/ntp/linux_public_test.go` - -The Debian NTP provider: - -- Writes `/etc/chrony/sources.d/osapi.sources` with server entries -- Reads status via `chronyc tracking` (parse output for sync state) -- Reads sources via `chronyc sources` (parse output for server list) -- Applies changes via `chronyc reload sources` -- Uses SHA-based idempotency (same approach as sysctl) - -- [ ] **Step 1: Write tests for Darwin and Linux stubs** - -Follow the sysctl pattern: one suite per platform, verify all methods return -`ErrUnsupported`. - -- [ ] **Step 2: Write failing tests for Debian provider** - -Suite: `DebianPublicTestSuite` with mocks for `exec.Manager` and `avfs.VFS`. - -**TestGet:** - -- success (parse chronyc tracking + sources output) -- chronyc tracking error -- chronyc sources error - -**TestCreate:** - -- success (file doesn't exist, writes it, runs reload) -- already exists error -- write error (failfs) -- reload error (non-fatal, still returns success) -- idempotent (same content, Changed: false) - -**TestUpdate:** - -- success (file exists, overwrites, runs reload) -- not managed error (file doesn't exist) -- write error (failfs) -- idempotent (same content, Changed: false) - -**TestDelete:** - -- success (file exists, removes it, runs reload) -- not found error -- remove error (failfs) - -- [ ] **Step 3: Implement debian.go** - -```go -var ( - _ Provider = (*Debian)(nil) - _ provider.FactsSetter = (*Debian)(nil) -) - -type Debian struct { - provider.FactsAware - logger *slog.Logger - fs avfs.VFS - execManager exec.Manager -} - -func NewDebianProvider( - logger *slog.Logger, - fs avfs.VFS, - execManager exec.Manager, -) *Debian { - return &Debian{ - logger: logger.With(slog.String("subsystem", "provider.ntp")), - fs: fs, - execManager: execManager, - } -} -``` - -Key implementation details: - -- Config file path: `/etc/chrony/sources.d/osapi.sources` -- Config content format: `server iburst\n` per server -- **Get**: run `chronyc tracking` and parse output for `Leap status`, `Stratum`, - `System time` (offset), `Reference ID`. Run `chronyc sources` and parse for - server addresses. Return Status struct. -- **Create**: check if osapi.sources exists → error if yes. Write file. Run - `chronyc reload sources`. -- **Update**: check if osapi.sources exists → error if no. Compare SHA of new - content → skip if same. Write file. Run `chronyc reload sources`. -- **Delete**: check if exists → error if no. Remove file. Run - `chronyc reload sources`. - -Read `internal/provider/node/sysctl/debian.go` as the reference for the file -write + idempotency pattern. - -- [ ] **Step 4: Create export_test.go** - -Expose any unexported variables needed for testing (e.g., `marshalJSON` or -chronyc command path overrides). - -- [ ] **Step 5: Verify all tests pass with 100% coverage** - -```bash -go test -coverprofile=/tmp/c.out -v ./internal/provider/node/ntp/... -go tool cover -func=/tmp/c.out | grep ntp -``` - -- [ ] **Step 6: Commit** - -```bash -git add internal/provider/node/ntp/ -git commit -m "feat(ntp): implement Debian chrony provider with tests" -``` - ---- - -## Task 4: NTP Agent Processor + Wiring - -**Files:** - -- Create: `internal/agent/processor_ntp.go` -- Create: `internal/agent/processor_ntp_public_test.go` -- Modify: `internal/agent/processor.go` — add ntp provider param + case -- Modify: `cmd/agent_setup.go` — create + register provider - -- [ ] **Step 1: Write processor tests** - -Test all operations: ntp.get, ntp.create, ntp.update, ntp.delete, unsupported, -nil provider. Follow `processor_sysctl_public_test.go` pattern. - -- [ ] **Step 2: Implement processor_ntp.go** - -```go -func processNtpOperation( - ntpProvider ntp.Provider, - logger *slog.Logger, - jobRequest job.Request, -) (json.RawMessage, error) { - // nil check, parse sub-operation, dispatch -} -``` - -Sub-operations: get (no data), create (unmarshal Config), update (unmarshal -Config), delete (no data). - -- [ ] **Step 3: Add to node processor** - -In `processor.go`, add `ntpProvider ntp.Provider` parameter to -`NewNodeProcessor` and add `case "ntp":`. - -- [ ] **Step 4: Wire in agent_setup.go** - -Create `createNtpProvider` function. Add provider to `NewNodeProcessor` call and -`registry.Register` providers list. - -NTP provider needs: logger, fs (avfs.VFS), execManager. No KV needed — it -manages files directly like sysctl. - -- [ ] **Step 5: Fix existing tests** - -Add nil NTP provider parameter to any test that calls `NewNodeProcessor`. - -- [ ] **Step 6: Verify all tests pass** - -```bash -go test ./internal/agent/... ./cmd/... -``` - -- [ ] **Step 7: Commit** - -```bash -git add internal/agent/ cmd/agent_setup.go -git commit -m "feat(ntp): add agent processor and wiring" -``` - ---- - -## Task 5: NTP OpenAPI Spec + API Handlers - -**Files:** - -- Create: `internal/controller/api/node/ntp/gen/api.yaml` -- Create: `internal/controller/api/node/ntp/gen/cfg.yaml` -- Create: `internal/controller/api/node/ntp/gen/generate.go` -- Create: `internal/controller/api/node/ntp/types.go` -- Create: `internal/controller/api/node/ntp/ntp.go` -- Create: `internal/controller/api/node/ntp/validate.go` -- Create: `internal/controller/api/node/ntp/ntp_get.go` -- Create: `internal/controller/api/node/ntp/ntp_create.go` -- Create: `internal/controller/api/node/ntp/ntp_update.go` -- Create: `internal/controller/api/node/ntp/ntp_delete.go` -- Create: `internal/controller/api/node/ntp/handler.go` -- Create: all `*_public_test.go` files - -- [ ] **Step 1: Create OpenAPI spec** - -Paths: - -- `GET /node/{hostname}/ntp` — operationId: `GetNodeNtp`, security: `ntp:read` -- `POST /node/{hostname}/ntp` — operationId: `PostNodeNtp`, security: - `ntp:write` -- `PUT /node/{hostname}/ntp` — operationId: `PutNodeNtp`, security: `ntp:write` -- `DELETE /node/{hostname}/ntp` — operationId: `DeleteNodeNtp`, security: - `ntp:write` - -Schemas: - -- `NtpCreateRequest` — required: servers (array of strings) -- `NtpUpdateRequest` — required: servers (array of strings) -- `NtpStatusEntry` — hostname (req), status (req, enum ok/failed/skipped), - synchronized, stratum, offset, current_source, servers, error -- `NtpMutationResult` — hostname (req), status (req), changed, error -- Collection responses wrapping results + job_id - -Reference common ErrorResponse via `../../../common/gen/api.yaml`. - -- [ ] **Step 2: Create cfg.yaml, generate.go, generate code** - -- [ ] **Step 3: Create handler struct, factory, validate** - -Follow sysctl pattern. Handler struct has JobClient + logger. - -- [ ] **Step 4: Implement all handlers with broadcast support** - -Follow sysctl handler pattern exactly. Category is `"node"`. Operations are -`job.OperationNtpGet`, etc. - -- [ ] **Step 5: Create handler.go (self-registration)** - -Follow sysctl handler.go pattern. - -- [ ] **Step 6: Write tests for all handlers** - -Each handler needs: success, broadcast, skipped, error, not-found (for -update/delete), validation (for create/update). Include RBAC HTTP tests. - -- [ ] **Step 7: Wire in controller_setup.go** - -```go -handlers = append(handlers, ntpAPI.Handler(log, jc, signingKey, customRoles)...) -``` - -- [ ] **Step 8: Verify** - -```bash -go build ./... -go test ./internal/controller/api/node/ntp/... ./cmd/... -``` - -- [ ] **Step 9: Commit** - -```bash -git add internal/controller/api/node/ntp/ cmd/controller_setup.go -git commit -m "feat(ntp): add OpenAPI spec, API handlers, and server wiring" -``` - ---- - -## Task 6: NTP SDK Service - -**Files:** - -- Create: `pkg/sdk/client/ntp.go` -- Create: `pkg/sdk/client/ntp_types.go` -- Create: `pkg/sdk/client/ntp_public_test.go` -- Create: `pkg/sdk/client/ntp_types_public_test.go` -- Modify: `pkg/sdk/client/osapi.go` -- Modify: `pkg/sdk/client/export_test.go` - -- [ ] **Step 1: Create SDK types** - -```go -type NtpStatusResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Synchronized bool `json:"synchronized,omitempty"` - Stratum int `json:"stratum,omitempty"` - Offset string `json:"offset,omitempty"` - CurrentSource string `json:"current_source,omitempty"` - Servers []string `json:"servers,omitempty"` - Error string `json:"error,omitempty"` -} - -type NtpMutationResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type NtpCreateOpts struct { - Servers []string -} - -type NtpUpdateOpts struct { - Servers []string -} -``` - -Add gen→SDK conversion functions. - -- [ ] **Step 2: Create SDK service** - -Methods: `NtpGet`, `NtpCreate`, `NtpUpdate`, `NtpDelete`. Follow sysctl service -pattern. - -- [ ] **Step 3: Wire into osapi.go** - -- [ ] **Step 4: Write tests** - -Cover all status code paths. Use httptest.Server mocks. - -- [ ] **Step 5: Verify** - -```bash -go test -v ./pkg/sdk/client/... -``` - -- [ ] **Step 6: Commit** - -```bash -git add pkg/sdk/client/ -git commit -m "feat(ntp): add SDK service with tests" -``` - ---- - -## Task 7: NTP CLI Commands - -**Files:** - -- Create: `cmd/client_node_ntp.go` -- Create: `cmd/client_node_ntp_get.go` -- Create: `cmd/client_node_ntp_create.go` -- Create: `cmd/client_node_ntp_update.go` -- Create: `cmd/client_node_ntp_delete.go` - -- [ ] **Step 1: Create parent command** - -```go -var clientNodeNtpCmd = &cobra.Command{ - Use: "ntp", - Short: "Manage NTP configuration", -} - -func init() { - clientNodeCmd.AddCommand(clientNodeNtpCmd) -} -``` - -- [ ] **Step 2: Create get command** - -No extra flags. Call `sdkClient.Ntp.NtpGet(ctx, host)`. Table fields: -SYNCHRONIZED, STRATUM, OFFSET, SOURCE, SERVERS. Format `Servers` as -comma-separated string. - -- [ ] **Step 3: Create create command** - -Flags: `--servers` (required, string slice). Call -`sdkClient.Ntp.NtpCreate(ctx, host, opts)`. Mutation table output. - -- [ ] **Step 4: Create update command** - -Flags: `--servers` (required, string slice). Call -`sdkClient.Ntp.NtpUpdate(ctx, host, opts)`. - -- [ ] **Step 5: Create delete command** - -No extra flags. Call `sdkClient.Ntp.NtpDelete(ctx, host)`. - -- [ ] **Step 6: Verify build** - -- [ ] **Step 7: Commit** - -```bash -git add cmd/client_node_ntp*.go -git commit -m "feat(ntp): add CLI commands" -``` - ---- - -## Task 8: NTP Docs + Example + Integration Test - -**Files:** - -- Create: `examples/sdk/client/ntp.go` -- Create: `test/integration/ntp_test.go` -- Create: `docs/docs/sidebar/features/ntp.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/ntp/ntp.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/ntp/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/ntp/create.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/ntp/update.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/ntp/delete.md` -- Create: `docs/docs/sidebar/sdk/client/ntp.md` - -- [ ] **Step 1: Create SDK example** - -Follow sysctl example pattern: list status, create config. - -- [ ] **Step 2: Create integration test** - -Follow sysctl integration test pattern: `NtpSmokeSuite` with `TestNtpGet`. - -- [ ] **Step 3: Create feature doc** - -Follow cron-management.md template. - -- [ ] **Step 4: Create CLI doc pages** - -Parent with ``, one page per subcommand. - -- [ ] **Step 5: Create SDK doc page** - -Follow sysctl SDK doc pattern. - -- [ ] **Step 6: Commit** - -```bash -git add examples/ test/ docs/ -git commit -m "feat(ntp): add docs, SDK example, and integration tests" -``` - ---- - -## Task 9: Timezone Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/timezone/types.go` -- Create: `internal/provider/node/timezone/darwin.go` -- Create: `internal/provider/node/timezone/linux.go` -- Create: `internal/provider/node/timezone/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -```go -// Package timezone provides system timezone management via timedatectl. -package timezone - -import "context" - -// Provider implements the methods to manage the system timezone. -type Provider interface { - // Get returns the current system timezone. - Get(ctx context.Context) (*Info, error) - // Update sets the system timezone. Idempotent. - Update(ctx context.Context, timezone string) (*UpdateResult, error) -} - -// Info represents the current timezone configuration. -type Info struct { - Timezone string `json:"timezone"` - UTCOffset string `json:"utc_offset,omitempty"` -} - -// UpdateResult represents the outcome of a timezone update. -type UpdateResult struct { - Timezone string `json:"timezone"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Create darwin.go and linux.go stubs** - -All methods return `fmt.Errorf("timezone: %w", provider.ErrUnsupported)`. - -- [ ] **Step 3: Create mocks and generate** - -- [ ] **Step 4: Verify and commit** - -```bash -git add internal/provider/node/timezone/ -git commit -m "feat(timezone): add provider interface and platform stubs" -``` - ---- - -## Task 10: Timezone Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/timezone/debian.go` -- Create: `internal/provider/node/timezone/debian_public_test.go` -- Create: `internal/provider/node/timezone/darwin_public_test.go` -- Create: `internal/provider/node/timezone/linux_public_test.go` - -- [ ] **Step 1: Write stub tests** - -- [ ] **Step 2: Write Debian tests** - -**TestGet:** - -- success (parse timedatectl output) -- timedatectl error - -**TestUpdate:** - -- success (different timezone, runs timedatectl set-timezone) -- idempotent (same timezone, Changed: false) -- timedatectl error -- invalid timezone (validate against known list or let timedatectl fail) - -- [ ] **Step 3: Implement debian.go** - -```go -type Debian struct { - provider.FactsAware - logger *slog.Logger - execManager exec.Manager -} - -func NewDebianProvider( - logger *slog.Logger, - execManager exec.Manager, -) *Debian -``` - -- **Get**: run `timedatectl show -p Timezone --value` for name, `date +%:z` for - UTC offset. -- **Update**: read current timezone first (for idempotency), run - `timedatectl set-timezone ` if different. - -No file management needed — timedatectl handles everything. - -- [ ] **Step 4: Verify 100% coverage and commit** - -```bash -git add internal/provider/node/timezone/ -git commit -m "feat(timezone): implement Debian timedatectl provider with tests" -``` - ---- - -## Task 11: Timezone Agent Processor + Wiring - -**Files:** - -- Create: `internal/agent/processor_timezone.go` -- Create: `internal/agent/processor_timezone_public_test.go` -- Modify: `internal/agent/processor.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Write processor tests** - -Operations: timezone.get, timezone.update, unsupported, nil provider. - -- [ ] **Step 2: Implement processor_timezone.go** - -Two sub-operations: get (no data), update (unmarshal `{"timezone": "..."}` from -Data). - -- [ ] **Step 3: Add to node processor and agent_setup** - -Add `timezoneProvider timezone.Provider` to `NewNodeProcessor`. Add -`case "timezone":`. - -Create `createTimezoneProvider` in agent_setup.go. Needs: logger, execManager. - -- [ ] **Step 4: Fix existing tests, verify, commit** - -```bash -git add internal/agent/ cmd/agent_setup.go -git commit -m "feat(timezone): add agent processor and wiring" -``` - ---- - -## Task 12: Timezone OpenAPI Spec + API Handlers - -**Files:** - -- Create: `internal/controller/api/node/timezone/gen/...` -- Create: `internal/controller/api/node/timezone/*.go` -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Create OpenAPI spec** - -Paths: - -- `GET /node/{hostname}/timezone` — `GetNodeTimezone`, security: `timezone:read` -- `PUT /node/{hostname}/timezone` — `PutNodeTimezone`, security: - `timezone:write` - -Schemas: - -- `TimezoneUpdateRequest` — required: timezone (string, validate: required) -- `TimezoneEntry` — hostname (req), status (req), timezone, utc_offset, error -- `TimezoneMutationResult` — hostname (req), status (req), timezone, changed, - error -- Collection responses - -- [ ] **Step 2: Create handlers + handler.go + tests** - -Follow NTP handler pattern. Two handlers: get and update. Include RBAC HTTP -tests. - -- [ ] **Step 3: Wire in controller_setup.go** - -- [ ] **Step 4: Verify and commit** - -```bash -git add internal/controller/api/node/timezone/ cmd/controller_setup.go -git commit -m "feat(timezone): add OpenAPI spec, API handlers, and server wiring" -``` - ---- - -## Task 13: Timezone SDK Service - -**Files:** - -- Create: `pkg/sdk/client/timezone.go` -- Create: `pkg/sdk/client/timezone_types.go` -- Create: `pkg/sdk/client/timezone_public_test.go` -- Create: `pkg/sdk/client/timezone_types_public_test.go` -- Modify: `pkg/sdk/client/osapi.go` - -- [ ] **Step 1: Create types, service, wire, test** - -Two methods: `TimezoneGet`, `TimezoneUpdate`. `TimezoneUpdateOpts` has one -field: `Timezone string`. - -- [ ] **Step 2: Verify and commit** - -```bash -git add pkg/sdk/client/ -git commit -m "feat(timezone): add SDK service with tests" -``` - ---- - -## Task 14: Timezone CLI + Docs - -**Files:** - -- Create: `cmd/client_node_timezone.go` -- Create: `cmd/client_node_timezone_get.go` -- Create: `cmd/client_node_timezone_update.go` -- Create: `examples/sdk/client/timezone.go` -- Create: `test/integration/timezone_test.go` -- Create: `docs/docs/sidebar/features/timezone.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/timezone/*.md` -- Create: `docs/docs/sidebar/sdk/client/timezone.md` - -- [ ] **Step 1: Create CLI commands** - -Get: table fields TIMEZONE, UTC_OFFSET. Update: flags `--timezone` (required). - -- [ ] **Step 2: Create SDK example, integration test, docs** - -- [ ] **Step 3: Verify and commit** - -```bash -git add cmd/ examples/ test/ docs/ -git commit -m "feat(timezone): add CLI commands, docs, and integration tests" -``` - ---- - -## Task 15: Shared Docs + Regeneration + Final Verification - -**Files:** - -- Modify: `docs/docusaurus.config.ts` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/features/authentication.md` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `CLAUDE.md` - -- [ ] **Step 1: Update shared docs** - -Add NTP and timezone to: - -- Features navbar dropdown -- Permissions/roles tables in configuration.md and authentication.md -- Endpoint table in api-guidelines.md -- Feature links in architecture.md -- Provider list in CLAUDE.md - -- [ ] **Step 2: Regenerate combined spec** - -```bash -just generate -``` - -- [ ] **Step 3: Full verification** - -```bash -go build ./... -just go::unit -just go::unit-cov # must be >= 99.9% -just go::vet -just go::fmt -``` - -- [ ] **Step 4: Commit** - -```bash -git add -A -git commit -m "docs(ntp,timezone): update shared docs and regenerate specs" -``` diff --git a/docs/plans/2026-03-31-certificate-management-provider-design.md b/docs/plans/2026-03-31-certificate-management-provider-design.md deleted file mode 100644 index ef00bc37b..000000000 --- a/docs/plans/2026-03-31-certificate-management-provider-design.md +++ /dev/null @@ -1,161 +0,0 @@ -# Certificate Management Provider Design - -## Overview - -Add CA certificate management to OSAPI. Deploy custom CA certificates to the -system trust store alongside default system CAs, and remove them. Uses -`file.Deployer` for SHA-tracked deployment and `update-ca-certificates` to -rebuild the trust bundle. Read-only listing of both system and custom CAs. - -## Architecture - -Meta provider at `internal/provider/node/certificate/`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/certificate/ca` -- **Permissions**: `certificate:read`, `certificate:write` -- **Provider type**: meta (file.Deployer + exec.Manager) - -## Provider Interface - -```go -type Provider interface { - List(ctx context.Context) ([]Entry, error) - Create(ctx context.Context, entry Entry) (*CreateResult, error) - Update(ctx context.Context, entry Entry) (*UpdateResult, error) - Delete(ctx context.Context, name string) (*DeleteResult, error) -} -``` - -## Data Types - -```go -type Entry struct { - Name string `json:"name"` - Source string `json:"source"` // "system" or "custom" - Object string `json:"object,omitempty"` -} - -type CreateResult struct { - Changed bool `json:"changed"` - Name string `json:"name"` -} - -type UpdateResult struct { - Changed bool `json:"changed"` - Name string `json:"name"` -} - -type DeleteResult struct { - Changed bool `json:"changed"` - Name string `json:"name"` -} -``` - -## Debian Implementation - -Custom CA certs are deployed to -`/usr/local/share/ca-certificates/osapi-{name}.crt` via `file.Deployer`. After -every create, update, or delete, the provider runs `update-ca-certificates` to -rebuild the system trust bundle. - -- **List**: Walk `/usr/share/ca-certificates/` for system CAs (strip path prefix - and `.crt` extension for name). Query file state KV for entries with `osapi-` - prefix for custom CAs. Return both with `source` field. -- **Create**: Deploy PEM from Object Store to - `/usr/local/share/ca-certificates/osapi-{name}.crt` via `file.Deployer` with - mode `0644`. Store metadata `{"source": "custom"}` in FileState. Run - `update-ca-certificates`. -- **Update**: Same as create but for an existing entry. The `file.Deployer` - compares SHA — if content unchanged, returns `changed: false` and skips - `update-ca-certificates`. -- **Delete**: Undeploy via `file.Deployer`, run `update-ca-certificates`. - -## Platform Implementations - -| Platform | Implementation | -| -------- | -------------------------------------- | -| Debian | file.Deployer + update-ca-certificates | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -No container check — CA cert management works in Docker containers. -`update-ca-certificates` is available and the trust store is writable. - -## API Endpoints - -| Method | Path | Permission | Description | -| -------- | ---------------------------------------- | ------------------- | --------------------- | -| `GET` | `/node/{hostname}/certificate/ca` | `certificate:read` | List CA certs | -| `POST` | `/node/{hostname}/certificate/ca` | `certificate:write` | Add custom CA cert | -| `PUT` | `/node/{hostname}/certificate/ca/{name}` | `certificate:write` | Update custom CA cert | -| `DELETE` | `/node/{hostname}/certificate/ca/{name}` | `certificate:write` | Remove custom CA cert | - -All endpoints support broadcast targeting. - -### POST/PUT Request Body - -```json -{ - "name": "internal-corp-ca", - "object": "corp-ca.pem" -} -``` - -`object` references an existing Object Store upload containing the PEM-encoded -CA certificate. For PUT, `name` comes from the path parameter. - -### Response Shape (List) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "certificates": [ - { "name": "DigiCert_Global_Root_G2", "source": "system" }, - { "name": "internal-corp-ca", "source": "custom" } - ] - } - ] -} -``` - -### Response Shape (Create/Update/Delete) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "name": "internal-corp-ca", - "changed": true - } - ] -} -``` - -## SDK - -```go -client.Certificate.List(ctx, host) -client.Certificate.Create(ctx, host, opts) -client.Certificate.Update(ctx, host, name, opts) -client.Certificate.Delete(ctx, host, name) -``` - -`CertificateCreateOpts` / `CertificateUpdateOpts` with `Name` and `Object` -fields. - -## Permissions - -- `certificate:read` — list CA certificates. Added to admin, write, and read - roles. -- `certificate:write` — create, update, delete custom CA certificates. Added to - admin and write roles. diff --git a/docs/plans/2026-03-31-certificate-management-provider.md b/docs/plans/2026-03-31-certificate-management-provider.md deleted file mode 100644 index 116fb8fe7..000000000 --- a/docs/plans/2026-03-31-certificate-management-provider.md +++ /dev/null @@ -1,1361 +0,0 @@ -# Certificate Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add CA certificate management to OSAPI — deploy custom CA certificates -to the system trust store via `file.Deployer`, list system + custom CAs, update -and remove custom CAs. - -**Architecture:** Meta provider at `internal/provider/node/certificate/` using -`file.Deployer` for SHA-tracked deployment and `exec.Manager` for -`update-ca-certificates`. Registered as its own agent category (`certificate`) -following the cron/schedule pattern. Four API endpoints under -`/node/{hostname}/certificate/ca`. - -**Tech Stack:** Go, file.Deployer, exec.Manager, update-ca-certificates, -oapi-codegen strict-server - ---- - -## File Structure - -### Provider Layer - -- Create: `internal/provider/node/certificate/types.go` — Provider interface + - domain types -- Create: `internal/provider/node/certificate/debian.go` — Debian implementation - with file.Deployer -- Create: `internal/provider/node/certificate/debian_list.go` — List - implementation (walks system + custom CA dirs) -- Create: `internal/provider/node/certificate/darwin.go` — macOS stub -- Create: `internal/provider/node/certificate/linux.go` — generic Linux stub -- Create: `internal/provider/node/certificate/mocks/generate.go` — mockgen - directive -- Test: `internal/provider/node/certificate/debian_public_test.go` -- Test: `internal/provider/node/certificate/debian_list_public_test.go` -- Test: `internal/provider/node/certificate/darwin_public_test.go` -- Test: `internal/provider/node/certificate/linux_public_test.go` - -### Agent Layer - -- Create: `internal/agent/processor_certificate.go` — certificate operation - dispatcher -- Modify: `cmd/agent_setup.go` — create certificate provider factory, register - as own category -- Test: `internal/agent/processor_certificate_public_test.go` - -### API Layer - -- Create: `internal/controller/api/node/certificate/gen/api.yaml` — OpenAPI spec -- Create: `internal/controller/api/node/certificate/gen/cfg.yaml` — oapi-codegen - config -- Create: `internal/controller/api/node/certificate/gen/generate.go` — - go:generate -- Create: `internal/controller/api/node/certificate/types.go` — handler struct -- Create: `internal/controller/api/node/certificate/certificate.go` — New(), - compile-time check -- Create: `internal/controller/api/node/certificate/validate.go` — - validateHostname -- Create: `internal/controller/api/node/certificate/ca_list_get.go` — list - handler -- Create: `internal/controller/api/node/certificate/ca_create_post.go` — create - handler -- Create: `internal/controller/api/node/certificate/ca_update_put.go` — update - handler -- Create: `internal/controller/api/node/certificate/ca_delete.go` — delete - handler -- Create: `internal/controller/api/node/certificate/handler.go` — Handler() - registration -- Modify: `cmd/controller_setup.go` — register certificate handler -- Test: `internal/controller/api/node/certificate/ca_list_get_public_test.go` -- Test: `internal/controller/api/node/certificate/ca_create_post_public_test.go` -- Test: `internal/controller/api/node/certificate/ca_update_put_public_test.go` -- Test: `internal/controller/api/node/certificate/ca_delete_public_test.go` -- Test: `internal/controller/api/node/certificate/handler_public_test.go` - -### Operations & Permissions - -- Modify: `pkg/sdk/client/operations.go` — add certificate operation constants -- Modify: `internal/job/types.go` — add certificate operation aliases -- Modify: `pkg/sdk/client/permissions.go` — add `PermCertificateRead`, - `PermCertificateWrite` -- Modify: `internal/authtoken/permissions.go` — add to all roles - -### SDK Layer - -- Create: `pkg/sdk/client/certificate.go` — CertificateService methods -- Create: `pkg/sdk/client/certificate_types.go` — SDK result types - - conversions -- Modify: `pkg/sdk/client/osapi.go` — add Certificate field -- Test: `pkg/sdk/client/certificate_public_test.go` -- Test: `pkg/sdk/client/certificate_types_public_test.go` - -### CLI Layer - -- Create: `cmd/client_node_certificate.go` — parent command -- Create: `cmd/client_node_certificate_list.go` — list subcommand -- Create: `cmd/client_node_certificate_create.go` — create subcommand -- Create: `cmd/client_node_certificate_update.go` — update subcommand -- Create: `cmd/client_node_certificate_delete.go` — delete subcommand - -### Documentation - -- Create: `docs/docs/sidebar/features/certificate-management.md` — feature page -- Create: `docs/docs/sidebar/usage/cli/client/node/certificate/certificate.md` — - CLI landing -- Create: `docs/docs/sidebar/usage/cli/client/node/certificate/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/certificate/create.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/certificate/update.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/certificate/delete.md` -- Create: `docs/docs/sidebar/sdk/client/management/certificate.md` — SDK doc -- Create: `examples/sdk/client/certificate.go` — SDK example -- Modify: `docs/docs/sidebar/features/features.md` — add cert row -- Modify: `docs/docs/sidebar/features/authentication.md` — add permissions to - role tables -- Modify: `docs/docs/sidebar/usage/configuration.md` — add permissions -- Modify: `docs/docs/sidebar/architecture/architecture.md` — add feature link -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` — add endpoints -- Modify: `docs/docusaurus.config.ts` — add to dropdowns -- Modify: `docs/docs/sidebar/sdk/client/client.md` — add service to table - -### Integration Test - -- Create: `test/integration/certificate_test.go` — smoke test - ---- - -### Task 1: Provider Interface and Types - -**Files:** - -- Create: `internal/provider/node/certificate/types.go` - -- [ ] **Step 1: Create provider interface and types** - -```go -// Package certificate provides CA certificate management operations. -package certificate - -import ( - "context" -) - -// Provider implements CA certificate management operations. -type Provider interface { - // List returns all CA certificates (system and custom). - List(ctx context.Context) ([]Entry, error) - // Create deploys a new custom CA certificate to the trust store. - Create(ctx context.Context, entry Entry) (*CreateResult, error) - // Update redeploys an existing custom CA certificate. - Update(ctx context.Context, entry Entry) (*UpdateResult, error) - // Delete removes a custom CA certificate from the trust store. - Delete(ctx context.Context, name string) (*DeleteResult, error) -} - -// Entry represents a CA certificate. -type Entry struct { - Name string `json:"name"` - Source string `json:"source,omitempty"` // "system" or "custom" - Object string `json:"object,omitempty"` -} - -// CreateResult represents the outcome of a CA certificate creation. -type CreateResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// UpdateResult represents the outcome of a CA certificate update. -type UpdateResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// DeleteResult represents the outcome of a CA certificate deletion. -type DeleteResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Verify it compiles** - -Run: `go build ./internal/provider/node/certificate/...` - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/node/certificate/types.go -git commit -m "feat(certificate): add provider interface and types" -``` - ---- - -### Task 2: Platform Stubs (Darwin + Linux) - -**Files:** - -- Create: `internal/provider/node/certificate/darwin.go` -- Create: `internal/provider/node/certificate/linux.go` -- Test: `internal/provider/node/certificate/darwin_public_test.go` -- Test: `internal/provider/node/certificate/linux_public_test.go` - -- [ ] **Step 1: Write stub tests** - -Create `darwin_public_test.go` and `linux_public_test.go` with testify/suite. -Test all four methods (List, Create, Update, Delete) return -`provider.ErrUnsupported`. Follow the pattern in -`internal/provider/node/log/darwin_public_test.go` — one suite method per -provider method, all in a single table. - -- [ ] **Step 2: Run tests to verify they fail** - -Run: `go test -v ./internal/provider/node/certificate/...` - -- [ ] **Step 3: Implement stubs** - -Create `darwin.go`: - -```go -type Darwin struct{} - -func NewDarwinProvider() *Darwin { return &Darwin{} } - -// All methods return: -// fmt.Errorf("certificate: %w", provider.ErrUnsupported) -``` - -Create `linux.go` — same pattern with `Linux` struct. - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `go test -v ./internal/provider/node/certificate/...` - -- [ ] **Step 5: Commit** - -```bash -git add internal/provider/node/certificate/ -git commit -m "feat(certificate): add darwin and linux provider stubs" -``` - ---- - -### Task 3: Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/certificate/debian.go` -- Create: `internal/provider/node/certificate/debian_list.go` -- Create: `internal/provider/node/certificate/mocks/generate.go` -- Test: `internal/provider/node/certificate/debian_public_test.go` -- Test: `internal/provider/node/certificate/debian_list_public_test.go` - -This is a meta provider following the cron pattern. Read -`internal/provider/scheduled/cron/debian.go` as the primary reference. - -- [ ] **Step 1: Create mock generator** - -Create `internal/provider/node/certificate/mocks/generate.go`: - -```go -package mocks - -//go:generate go tool github.com/golang/mock/mockgen -source=../types.go -destination=provider.gen.go -package=mocks -``` - -Run: `go generate ./internal/provider/node/certificate/mocks/...` - -- [ ] **Step 2: Write Debian provider tests** - -Create `debian_public_test.go` with testify/suite and gomock. The Debian struct -needs: - -- `provider.FactsAware` embedded -- `logger *slog.Logger` -- `fs avfs.VFS` (for listing system CAs) -- `fileDeployer file.Deployer` (mocked) -- `stateKV jetstream.KeyValue` (mocked) -- `execManager exec.Manager` (mocked, for `update-ca-certificates`) -- `hostname string` - -**TestCreate** — table-driven with cases: - -- success: mock fileDeployer.Deploy with path - `/usr/local/share/ca-certificates/osapi-mycert.crt`, mode `0644`, returns - `Changed: true`. Mock execManager.RunCmd for `update-ca-certificates`. Verify - result. -- already exists: mock fs.Stat on the path returns nil (file exists). Verify - error "already exists". -- deploy error: mock fileDeployer.Deploy returns error. -- update-ca-certificates error: deploy succeeds, RunCmd fails. -- invalid name (empty): verify error. -- invalid name (special chars): verify error. - -**TestUpdate** — table-driven: - -- success: mock fs.Stat finds file, mock fileDeployer.Deploy returns - `Changed: true`, mock RunCmd succeeds. -- not found: mock fs.Stat returns error. Verify "does not exist". -- deploy error. -- update with same content: Deploy returns `Changed: false`, skip - `update-ca-certificates`. - -**TestDelete** — table-driven: - -- success: mock fs.Stat finds file, mock fileDeployer.Undeploy returns - `Changed: true`, mock RunCmd succeeds. -- not found: returns `Changed: false`, no error. -- undeploy error. - -- [ ] **Step 3: Write List tests** - -Create `debian_list_public_test.go` with testify/suite. Use `memfs.New()` for -filesystem. Set up: - -- `/usr/share/ca-certificates/mozilla/DigiCert.crt` (system) -- `/usr/local/share/ca-certificates/osapi-mycert.crt` (custom, with matching - file state KV entry) -- `/usr/local/share/ca-certificates/manual.crt` (not managed — no file state - entry) - -**TestList** — table-driven: - -- success with system + custom certs -- empty directories -- fs.ReadDir error on system dir -- custom cert without file state (skipped) - -- [ ] **Step 4: Implement debian.go** - -```go -package certificate - -import ( - "context" - "fmt" - "log/slog" - "regexp" - - "github.com/avfs/avfs" - "github.com/nats-io/nats.go/jetstream" - - "github.com/osapi-io/osapi/internal/exec" - "github.com/osapi-io/osapi/internal/provider" - "github.com/osapi-io/osapi/internal/provider/file" -) - -const ( - systemCADir = "/usr/share/ca-certificates" - customCADir = "/usr/local/share/ca-certificates" - managedPrefix = "osapi-" -) - -var validName = regexp.MustCompile(`^[a-zA-Z0-9_-]+$`) - -// Compile-time checks. -var ( - _ Provider = (*Debian)(nil) - _ provider.FactsSetter = (*Debian)(nil) -) - -type Debian struct { - provider.FactsAware - logger *slog.Logger - fs avfs.VFS - fileDeployer file.Deployer - stateKV jetstream.KeyValue - execManager exec.Manager - hostname string -} - -func NewDebianProvider( - logger *slog.Logger, - fs avfs.VFS, - fileDeployer file.Deployer, - stateKV jetstream.KeyValue, - execManager exec.Manager, - hostname string, -) *Debian { - return &Debian{ - logger: logger.With(slog.String("subsystem", "provider.certificate")), - fs: fs, - fileDeployer: fileDeployer, - stateKV: stateKV, - execManager: execManager, - hostname: hostname, - } -} - -func (d *Debian) Create( - ctx context.Context, - entry Entry, -) (*CreateResult, error) { - if err := validateName(entry.Name); err != nil { - return nil, err - } - - filePath := customCADir + "/" + managedPrefix + entry.Name + ".crt" - - // Check if already exists on disk. - if _, err := d.fs.Stat(filePath); err == nil { - return nil, fmt.Errorf( - "certificate %q already exists", - entry.Name, - ) - } - - d.logger.Debug("executing certificate.Create", - slog.String("name", entry.Name), - ) - - result, err := d.fileDeployer.Deploy(ctx, file.DeployRequest{ - ObjectName: entry.Object, - Path: filePath, - Mode: "0644", - ContentType: "raw", - Metadata: map[string]string{ - "source": "custom", - }, - }) - if err != nil { - return nil, fmt.Errorf("create certificate: %w", err) - } - - if result.Changed { - if _, err := d.execManager.RunCmd( - "update-ca-certificates", - nil, - ); err != nil { - return nil, fmt.Errorf( - "update-ca-certificates: %w", - err, - ) - } - } - - return &CreateResult{ - Name: entry.Name, - Changed: result.Changed, - }, nil -} - -func (d *Debian) Update( - ctx context.Context, - entry Entry, -) (*UpdateResult, error) { - if err := validateName(entry.Name); err != nil { - return nil, err - } - - filePath := customCADir + "/" + managedPrefix + entry.Name + ".crt" - - if _, err := d.fs.Stat(filePath); err != nil { - return nil, fmt.Errorf( - "certificate %q does not exist", - entry.Name, - ) - } - - d.logger.Debug("executing certificate.Update", - slog.String("name", entry.Name), - ) - - result, err := d.fileDeployer.Deploy(ctx, file.DeployRequest{ - ObjectName: entry.Object, - Path: filePath, - Mode: "0644", - ContentType: "raw", - Metadata: map[string]string{ - "source": "custom", - }, - }) - if err != nil { - return nil, fmt.Errorf("update certificate: %w", err) - } - - if result.Changed { - if _, err := d.execManager.RunCmd( - "update-ca-certificates", - nil, - ); err != nil { - return nil, fmt.Errorf( - "update-ca-certificates: %w", - err, - ) - } - } - - return &UpdateResult{ - Name: entry.Name, - Changed: result.Changed, - }, nil -} - -func (d *Debian) Delete( - ctx context.Context, - name string, -) (*DeleteResult, error) { - if err := validateName(name); err != nil { - return nil, err - } - - filePath := customCADir + "/" + managedPrefix + name + ".crt" - - if _, err := d.fs.Stat(filePath); err != nil { - return &DeleteResult{ - Name: name, - Changed: false, - }, nil - } - - d.logger.Debug("executing certificate.Delete", - slog.String("name", name), - ) - - result, err := d.fileDeployer.Undeploy( - ctx, - file.UndeployRequest{Path: filePath}, - ) - if err != nil { - return nil, fmt.Errorf("delete certificate: %w", err) - } - - if result.Changed { - if _, err := d.execManager.RunCmd( - "update-ca-certificates", - nil, - ); err != nil { - return nil, fmt.Errorf( - "update-ca-certificates: %w", - err, - ) - } - } - - return &DeleteResult{ - Name: name, - Changed: result.Changed, - }, nil -} - -func validateName(name string) error { - if name == "" { - return fmt.Errorf("invalid certificate name: empty") - } - if !validName.MatchString(name) { - return fmt.Errorf( - "invalid certificate name %q: must match %s", - name, - validName.String(), - ) - } - return nil -} -``` - -- [ ] **Step 5: Implement debian_list.go** - -```go -package certificate - -import ( - "context" - "encoding/json" - "io/fs" - "path/filepath" - "strings" - - "github.com/osapi-io/osapi/internal/job" - "github.com/osapi-io/osapi/internal/provider/file" -) - -// List returns all CA certificates — both system CAs from -// /usr/share/ca-certificates/ and custom CAs managed by OSAPI -// from /usr/local/share/ca-certificates/. -func (d *Debian) List( - ctx context.Context, -) ([]Entry, error) { - d.logger.Debug("executing certificate.List") - - var result []Entry - - // Walk system CA directory for system certs. - systemEntries, err := d.listSystemCAs() - if err != nil { - return nil, fmt.Errorf("list system CAs: %w", err) - } - result = append(result, systemEntries...) - - // List custom CAs from file state KV. - customEntries := d.listCustomCAs(ctx) - result = append(result, customEntries...) - - return result, nil -} - -// listSystemCAs walks /usr/share/ca-certificates/ and returns -// entries with source "system". -func (d *Debian) listSystemCAs() ([]Entry, error) { - var entries []Entry - - err := d.fs.WalkDir( - systemCADir, - func(path string, dirEntry fs.DirEntry, err error) error { - if err != nil { - return nil // skip unreadable entries - } - if dirEntry.IsDir() { - return nil - } - if !strings.HasSuffix(path, ".crt") { - return nil - } - - // Strip base dir and .crt extension for name. - rel, _ := filepath.Rel(systemCADir, path) - name := strings.TrimSuffix(rel, ".crt") - - entries = append(entries, Entry{ - Name: name, - Source: "system", - }) - - return nil - }, - ) - if err != nil { - return nil, err - } - - return entries, nil -} - -// listCustomCAs reads custom CAs from the filesystem, checking -// the file-state KV to confirm they are OSAPI-managed. -func (d *Debian) listCustomCAs( - ctx context.Context, -) []Entry { - var entries []Entry - - dirEntries, err := d.fs.ReadDir(customCADir) - if err != nil { - return entries - } - - for _, dirEntry := range dirEntries { - if dirEntry.IsDir() { - continue - } - - name := dirEntry.Name() - if !strings.HasPrefix(name, managedPrefix) { - continue - } - if !strings.HasSuffix(name, ".crt") { - continue - } - - path := customCADir + "/" + name - - // Verify this is managed via file state KV. - stateKey := file.BuildStateKey(d.hostname, path) - kvEntry, err := d.stateKV.Get(ctx, stateKey) - if err != nil { - continue - } - - var state job.FileState - if err := json.Unmarshal( - kvEntry.Value(), - &state, - ); err != nil { - continue - } - if state.UndeployedAt != "" { - continue - } - - // Strip osapi- prefix and .crt suffix for clean name. - cleanName := strings.TrimPrefix(name, managedPrefix) - cleanName = strings.TrimSuffix(cleanName, ".crt") - - entries = append(entries, Entry{ - Name: cleanName, - Source: "custom", - Object: state.ObjectName, - }) - } - - return entries -} -``` - -Note: `debian_list.go` needs `"fmt"` in its imports for the `List` method error -wrapping. - -- [ ] **Step 6: Run tests to verify they pass** - -Run: `go test -v ./internal/provider/node/certificate/...` - -- [ ] **Step 7: Verify 100% coverage** - -Run: - -```bash -go test -coverprofile=/tmp/cert_prov.cov \ - ./internal/provider/node/certificate/... && \ - go tool cover -func=/tmp/cert_prov.cov | \ - grep -v "100.0%" | grep -v "mocks" -``` - -Fix any gaps before proceeding. - -- [ ] **Step 8: Commit** - -```bash -git add internal/provider/node/certificate/ -git commit -m "feat(certificate): add meta provider with file.Deployer" -``` - ---- - -### Task 4: Operations, Permissions, and Agent Wiring - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `internal/job/types.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/authtoken/permissions.go` -- Create: `internal/agent/processor_certificate.go` -- Modify: `cmd/agent_setup.go` -- Test: `internal/agent/processor_certificate_public_test.go` - -- [ ] **Step 1: Add operation constants** - -In `pkg/sdk/client/operations.go`, add after Log operations: - -```go -// Certificate operations. -const ( - OpCertificateCAList JobOperation = "certificate.ca.list" - OpCertificateCACreate JobOperation = "certificate.ca.create" - OpCertificateCAUpdate JobOperation = "certificate.ca.update" - OpCertificateCADelete JobOperation = "certificate.ca.delete" -) -``` - -In `internal/job/types.go`, add corresponding aliases: - -```go -// Certificate operations. -const ( - OperationCertificateCAList = client.OpCertificateCAList - OperationCertificateCACreate = client.OpCertificateCACreate - OperationCertificateCAUpdate = client.OpCertificateCAUpdate - OperationCertificateCADelete = client.OpCertificateCADelete -) -``` - -- [ ] **Step 2: Add permission constants** - -In `pkg/sdk/client/permissions.go`, add: - -```go - PermCertificateRead Permission = "certificate:read" - PermCertificateWrite Permission = "certificate:write" -``` - -In `internal/authtoken/permissions.go`: - -- Add constants `PermCertificateRead`, `PermCertificateWrite` -- Add both to `AllPermissions` -- Add `PermCertificateRead`, `PermCertificateWrite` to admin role -- Add `PermCertificateRead`, `PermCertificateWrite` to write role -- Add `PermCertificateRead` to read role - -- [ ] **Step 3: Write processor tests** - -Create `internal/agent/processor_certificate_public_test.go`. Follow the -`processor_schedule_public_test.go` pattern — the certificate provider gets its -own `NewCertificateProcessor`. - -**TestProcessCertificateOperation** — dispatch-level: - -- nil provider returns error -- invalid operation format -- unsupported sub-operation - -**TestProcessCertificateCAList** — table-driven: - -- success -- provider error - -**TestProcessCertificateCACreate** — table-driven: - -- success -- unmarshal error -- provider error - -**TestProcessCertificateCAUpdate** — table-driven: - -- success -- unmarshal error -- provider error - -**TestProcessCertificateCADelete** — table-driven: - -- success -- unmarshal error -- provider error - -- [ ] **Step 4: Implement processor** - -Create `internal/agent/processor_certificate.go`. Follow `processor_schedule.go` -pattern: - -```go -func NewCertificateProcessor( - certProvider certProv.Provider, - logger *slog.Logger, -) ProcessorFunc { - return func(req job.Request) (json.RawMessage, error) { - if certProvider == nil { - return nil, fmt.Errorf( - "certificate provider not available", - ) - } - baseOperation := strings.Split( - req.Operation, ".")[0] - switch baseOperation { - case "ca": - return processCertificateCAOperation( - certProvider, logger, req) - default: - return nil, fmt.Errorf( - "unsupported certificate operation: %s", - req.Operation) - } - } -} -``` - -`processCertificateCAOperation` splits on `.` to get sub-op (`list`, `create`, -`update`, `delete`) and dispatches. - -- [ ] **Step 5: Wire in agent_setup.go** - -Add import: - -```go -certProv "github.com/osapi-io/osapi/internal/provider/node/certificate" -``` - -Add factory function `createCertificateProvider` — on Debian, needs -`fileProvider`, `fileStateKV`, `execManager`, `hostname`. If -`fileProvider == nil`, log warning and return Linux stub. No container check. -Darwin/Linux return stubs. - -Register as its own category: - -```go -registry.Register("certificate", - agent.NewCertificateProcessor(certProvider, log), - certProvider, -) -``` - -- [ ] **Step 6: Run tests and verify** - -```bash -go test -v ./internal/agent/... -go build ./... -``` - -- [ ] **Step 7: Verify 100% coverage on processor** - -```bash -go test -coverprofile=/tmp/cert_proc.cov \ - ./internal/agent/... && \ - go tool cover -func=/tmp/cert_proc.cov | \ - grep "processor_certificate" -``` - -- [ ] **Step 8: Commit** - -```bash -git add pkg/sdk/client/operations.go internal/job/types.go \ - pkg/sdk/client/permissions.go \ - internal/authtoken/permissions.go \ - internal/agent/processor_certificate.go \ - internal/agent/processor_certificate_public_test.go \ - cmd/agent_setup.go -git commit -m "feat(certificate): add operations, permissions, and agent wiring" -``` - ---- - -### Task 5: OpenAPI Spec and Code Generation - -**Files:** - -- Create: `internal/controller/api/node/certificate/gen/api.yaml` -- Create: `internal/controller/api/node/certificate/gen/cfg.yaml` -- Create: `internal/controller/api/node/certificate/gen/generate.go` - -- [ ] **Step 1: Create OpenAPI spec** - -Read `internal/controller/api/node/schedule/gen/api.yaml` as the reference. -Create `api.yaml` with: - -- Tag: `certificate_operations`, displayName `Node/Certificate` -- Paths: - - `GET /node/{hostname}/certificate/ca` (operationId: `GetNodeCertificateCa`, - security: `certificate:read`) - - `POST /node/{hostname}/certificate/ca` (operationId: - `PostNodeCertificateCa`, security: `certificate:write`) - - `PUT /node/{hostname}/certificate/ca/{name}` (operationId: - `PutNodeCertificateCa`, security: `certificate:write`) - - `DELETE /node/{hostname}/certificate/ca/{name}` (operationId: - `DeleteNodeCertificateCa`, security: `certificate:write`) -- Parameters: Hostname (path), CertName (path, `name`) -- Request body for POST: `CertificateCACreateRequest` with `name` (required, - validate: required,min=1) and `object` (required, validate: required,min=1) -- Request body for PUT: `CertificateCAUpdateRequest` with `object` (required, - validate: required,min=1). Name comes from path. -- Schemas: - - `CertificateCAInfo` — name (string), source (string enum system/custom), - object (string) - - `CertificateCAEntry` — hostname, status (ok/failed/skipped), certificates - (array of CertificateCAInfo), error - - `CertificateCACollectionResponse` — job_id, results (array of - CertificateCAEntry) - - `CertificateCAMutationEntry` — hostname, status, name, changed (boolean), - error - - `CertificateCAMutationResponse` — job_id, results (array of - CertificateCAMutationEntry) -- Responses: 200, 400 (for POST/PUT), 401, 403, 404 (for PUT/DELETE), 500 - -- [ ] **Step 2: Create cfg.yaml and generate.go** - -`cfg.yaml`: - -```yaml -package: gen -output: certificate.gen.go -generate: - models: true - echo-server: true - strict-server: true -import-mapping: - ../../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/controller/api/common/gen -output-options: - skip-prune: true -``` - -`generate.go`: - -```go -package gen -//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen -config cfg.yaml api.yaml -``` - -- [ ] **Step 3: Generate code and rebuild combined spec** - -```bash -go generate ./internal/controller/api/node/certificate/gen/... -just generate -go build ./... -``` - -- [ ] **Step 4: Commit** - -```bash -git add internal/controller/api/node/certificate/gen/ -git commit -m "feat(certificate): add OpenAPI spec and generated code" -``` - ---- - -### Task 6: API Handler Implementation - -**Files:** - -- Create all handler files under `internal/controller/api/node/certificate/` -- Modify: `cmd/controller_setup.go` -- Test: all `*_public_test.go` files - -- [ ] **Step 1: Create handler scaffolding** - -Create `types.go`, `certificate.go`, `validate.go`, `handler.go` following the -same pattern as `internal/controller/api/node/schedule/`. The handler struct is -`Certificate` with `JobClient` and `logger`. - -Compile-time check: - -```go -var _ gen.StrictServerInterface = (*Certificate)(nil) -``` - -Subsystem: `"api.certificate"`. - -- [ ] **Step 2: Implement list handler** - -Create `ca_list_get.go` — `GetNodeCertificateCa` method. - -- Validate hostname -- If broadcast: `QueryBroadcast` with category `"certificate"` and - `job.OperationCertificateCAList` -- Single target: `Query` with same -- Parse response: unmarshal `[]certProv.Entry` from resp.Data, convert to - `[]gen.CertificateCAInfo` -- Return `gen.GetNodeCertificateCa200JSONResponse` - -- [ ] **Step 3: Implement create handler** - -Create `ca_create_post.go` — `PostNodeCertificateCa` method. - -- Validate hostname -- Validate request body -- Build entry from request body (name + object) -- If broadcast: `ModifyBroadcast` with category `"certificate"` and - `job.OperationCertificateCACreate` -- Single target: `Modify` with same -- Parse mutation response (name, changed) -- Handle 400 (validation), 401, 403, 500 - -- [ ] **Step 4: Implement update handler** - -Create `ca_update_put.go` — `PutNodeCertificateCa` method. - -- Validate hostname -- Name from path param `request.Name` -- Object from request body -- Build entry with name + object -- `Modify` with `job.OperationCertificateCAUpdate` -- Handle 404 (not found) - -- [ ] **Step 5: Implement delete handler** - -Create `ca_delete.go` — `DeleteNodeCertificateCa` method. - -- Validate hostname -- Name from path param -- `Modify` with `job.OperationCertificateCADelete` and - `map[string]string{"name": name}` - -- [ ] **Step 6: Write handler tests** - -Create test files for all four handlers. Follow the pattern in -`internal/controller/api/node/schedule/cron_create_public_test.go` and similar. -Each test file needs: - -- Table-driven tests with success, error, skipped, broadcast cases -- HTTP wiring tests (`TestXxxHTTP`) -- RBAC tests (`TestXxxRBACHTTP`) — 401, 403, 200 - -- [ ] **Step 7: Register in controller_setup.go** - -Add import: - -```go -certAPI "github.com/osapi-io/osapi/internal/controller/api/node/certificate" -``` - -Add: - -```go -handlers = append(handlers, - certAPI.Handler(log, jc, signingKey, customRoles)...) -``` - -- [ ] **Step 8: Run tests and verify coverage** - -```bash -go test -v ./internal/controller/api/node/certificate/... -go build ./... -go test -coverprofile=/tmp/cert_handler.cov \ - ./internal/controller/api/node/certificate/... && \ - go tool cover -func=/tmp/cert_handler.cov | \ - grep -v "100.0%" | grep -v "gen/" -``` - -- [ ] **Step 9: Commit** - -```bash -git add internal/controller/api/node/certificate/ \ - cmd/controller_setup.go -git commit -m "feat(certificate): add API handlers with broadcast support" -``` - ---- - -### Task 7: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/certificate.go` -- Create: `pkg/sdk/client/certificate_types.go` -- Modify: `pkg/sdk/client/osapi.go` -- Test: `pkg/sdk/client/certificate_public_test.go` -- Test: `pkg/sdk/client/certificate_types_public_test.go` - -- [ ] **Step 1: Implement types** - -Create `certificate_types.go` with: - -- `CertificateCAResult` — Hostname, Status, Certificates ([]CertificateCA), - Error -- `CertificateCA` — Name, Source, Object -- `CertificateCAMutationResult` — Hostname, Status, Name, Changed, Error -- `CertificateCreateOpts` — Name, Object -- `CertificateUpdateOpts` — Object -- Conversion functions from gen types - -- [ ] **Step 2: Implement service** - -Create `certificate.go` with `CertificateService`: - -- `List(ctx, hostname)` → `*Response[Collection[CertificateCAResult]]` -- `Create(ctx, hostname, opts CertificateCreateOpts)` → - `*Response[Collection[CertificateCAMutationResult]]` -- `Update(ctx, hostname, name, opts CertificateUpdateOpts)` → - `*Response[Collection[CertificateCAMutationResult]]` -- `Delete(ctx, hostname, name)` → - `*Response[Collection[CertificateCAMutationResult]]` - -Each method: build gen params/body, call generated client, checkError, nil -guard, convert, return. - -- [ ] **Step 3: Wire in osapi.go** - -Add `Certificate *CertificateService` to Client struct and initialize in -`New()`. - -- [ ] **Step 4: Regenerate SDK client** - -```bash -go generate ./pkg/sdk/client/gen/... -``` - -- [ ] **Step 5: Write tests** - -`certificate_public_test.go` — httptest.Server tests for all 4 methods, covering -200, 400, 401, 403, 404, 500, nil body, transport error. - -`certificate_types_public_test.go` — conversion function tests. - -- [ ] **Step 6: Verify 100% coverage** - -```bash -go test -coverprofile=/tmp/cert_sdk.cov \ - ./pkg/sdk/client/... && \ - go tool cover -func=/tmp/cert_sdk.cov | \ - grep "certificate" | grep -v "100.0%" -``` - -- [ ] **Step 7: Commit** - -```bash -git add pkg/sdk/client/certificate.go \ - pkg/sdk/client/certificate_types.go \ - pkg/sdk/client/certificate_public_test.go \ - pkg/sdk/client/certificate_types_public_test.go \ - pkg/sdk/client/osapi.go pkg/sdk/client/gen/ -git commit -m "feat(certificate): add SDK service with tests" -``` - ---- - -### Task 8: CLI Commands - -**Files:** - -- Create: `cmd/client_node_certificate.go` -- Create: `cmd/client_node_certificate_list.go` -- Create: `cmd/client_node_certificate_create.go` -- Create: `cmd/client_node_certificate_update.go` -- Create: `cmd/client_node_certificate_delete.go` - -- [ ] **Step 1: Create parent command** - -```go -var clientNodeCertificateCmd = &cobra.Command{ - Use: "certificate", - Short: "Manage CA certificates", -} - -func init() { - clientNodeCmd.AddCommand(clientNodeCertificateCmd) -} -``` - -- [ ] **Step 2: Create list subcommand** - -`client_node_certificate_list.go`: - -- Calls `sdkClient.Certificate.List(ctx, host)` -- Table headers: `NAME`, `SOURCE` -- Uses `BuildBroadcastTable` - -- [ ] **Step 3: Create create subcommand** - -`client_node_certificate_create.go`: - -- Flags: `--name` (required), `--object` (required) -- Calls `sdkClient.Certificate.Create(ctx, host, opts)` -- Uses `BuildMutationTable` with headers `NAME`, `CHANGED` - -- [ ] **Step 4: Create update subcommand** - -`client_node_certificate_update.go`: - -- Flags: `--name` (required), `--object` (required) -- Calls `sdkClient.Certificate.Update(ctx, host, name, opts)` -- Uses `BuildMutationTable` - -- [ ] **Step 5: Create delete subcommand** - -`client_node_certificate_delete.go`: - -- Flags: `--name` (required) -- Calls `sdkClient.Certificate.Delete(ctx, host, name)` -- Uses `BuildMutationTable` - -- [ ] **Step 6: Verify build** - -```bash -go build ./... -``` - -- [ ] **Step 7: Commit** - -```bash -git add cmd/client_node_certificate*.go -git commit -m "feat(certificate): add CLI commands for CA cert management" -``` - ---- - -### Task 9: Documentation and SDK Example - -**Files:** - -- Create all doc files listed in File Structure -- Modify all cross-reference files - -- [ ] **Step 1: Create feature page** - -`docs/docs/sidebar/features/certificate-management.md`: - -- How It Works (List, Create, Update, Delete) -- Operations table (4 operations) -- CLI Usage examples -- Broadcast Support -- Supported Platforms (Debian: Full, Darwin: Skipped, Linux: Skipped) -- No container restriction -- Permissions: `certificate:read` (list), `certificate:write` (create, update, - delete) -- Related links - -- [ ] **Step 2: Create CLI doc pages** - -Landing page + list.md, create.md, update.md, delete.md with usage, flags, and -output examples. - -- [ ] **Step 3: Create SDK doc page** - -`docs/docs/sidebar/sdk/client/management/certificate.md`: - -- Methods table (List, Create, Update, Delete) -- Request/result types -- Usage examples -- Permissions - -- [ ] **Step 4: Create SDK example** - -`examples/sdk/client/certificate.go` — demonstrate List, Create, Delete with -cleanup-first pattern. Under ~100 lines. - -- [ ] **Step 5: Update cross-references** - -- `features/features.md` — add row -- `features/authentication.md` — add `certificate:read`, `certificate:write` to - role tables -- `usage/configuration.md` — add permissions -- `architecture/architecture.md` — add feature link -- `architecture/api-guidelines.md` — add 4 endpoint rows -- `docusaurus.config.ts` — add to Features + SDK dropdowns -- `sdk/client/client.md` — add Certificate to Management table - -- [ ] **Step 6: Commit** - -```bash -git add docs/ examples/sdk/client/certificate.go -git commit -m "docs: add certificate management feature docs, SDK example, and cross-references" -``` - ---- - -### Task 10: Integration Test - -**Files:** - -- Create: `test/integration/certificate_test.go` - -- [ ] **Step 1: Write integration test** - -`//go:build integration` tag. Test: - -- `osapi client node certificate list --target _any --json` — verify JSON with - results array containing system certs -- Optionally test create/delete if `OSAPI_INTEGRATION_WRITES=1` is set (guarded - by `skipWrite`) - -- [ ] **Step 2: Commit** - -```bash -git add test/integration/certificate_test.go -git commit -m "test(certificate): add integration test" -``` - ---- - -### Task 11: Final Verification - -- [ ] **Step 1: Run full suite** - -```bash -just generate -go build ./... -just go::unit -just go::vet -``` - -- [ ] **Step 2: Verify coverage on all new code** - -```bash -go test -coverprofile=/tmp/cert_all.cov \ - ./internal/provider/node/certificate/... \ - ./internal/agent/... \ - ./internal/controller/api/node/certificate/... \ - ./pkg/sdk/client/... -go tool cover -func=/tmp/cert_all.cov | \ - grep "certificate" | grep -v "100.0%" | \ - grep -v "mocks\|gen/" -``` - -All new certificate code must be at 100%. - -- [ ] **Step 3: Commit any fixes** - -```bash -git add -A -git commit -m "chore(certificate): fix formatting and lint" -``` diff --git a/docs/plans/2026-03-31-log-management-provider-design.md b/docs/plans/2026-03-31-log-management-provider-design.md deleted file mode 100644 index 7d27bac1e..000000000 --- a/docs/plans/2026-03-31-log-management-provider-design.md +++ /dev/null @@ -1,123 +0,0 @@ -# Log Viewing Provider Design - -## Overview - -Add log viewing to OSAPI. Query systemd journal entries with optional filtering -by lines, time range, and priority. Read-only — no write operations. Uses -`journalctl --output=json` for structured parsing. - -## Architecture - -Direct provider at `internal/provider/node/log/`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/log` -- **Permissions**: `log:read` -- **Provider type**: direct (exec.Manager) - -## Provider Interface - -```go -type Provider interface { - Query(ctx context.Context, opts QueryOpts) ([]Entry, error) - QueryUnit(ctx context.Context, unit string, opts QueryOpts) ([]Entry, error) -} -``` - -## Data Types - -```go -type QueryOpts struct { - Lines int `json:"lines,omitempty"` - Since string `json:"since,omitempty"` - Priority string `json:"priority,omitempty"` -} - -type Entry struct { - Timestamp string `json:"timestamp"` - Unit string `json:"unit,omitempty"` - Priority string `json:"priority"` - Message string `json:"message"` - PID int `json:"pid,omitempty"` - Hostname string `json:"hostname,omitempty"` -} -``` - -## Debian Implementation - -- **Query**: run `journalctl --output=json -n ` with optional - `--since=` and `--priority=`. Parse JSON lines output — each - line is a JSON object with fields `__REALTIME_TIMESTAMP`, `SYSLOG_IDENTIFIER`, - `PRIORITY`, `MESSAGE`, `_PID`, `_HOSTNAME`. -- **QueryUnit**: same but with `-u ` flag. - -Default `lines` is 100 if not specified. `since` uses journalctl format (e.g., -`"1 hour ago"`, `"2026-03-31"`). `priority` uses journalctl levels (0-7 or names -like `err`, `warning`). - -## Platform Implementations - -| Platform | Implementation | -| -------- | ------------------------ | -| Debian | journalctl --output=json | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -Return `ErrUnsupported` in containers — `journalctl` requires systemd which -isn't available in containers. - -## API Endpoints - -| Method | Path | Permission | Description | -| ------ | ---------------------------------- | ---------- | ------------------------ | -| `GET` | `/node/{hostname}/log` | `log:read` | Query journal entries | -| `GET` | `/node/{hostname}/log/unit/{name}` | `log:read` | Query entries for a unit | - -All endpoints support broadcast targeting. - -### Query Parameters - -| Param | Type | Default | Description | -| ---------- | ------- | ------- | -------------------------------------- | -| `lines` | integer | 100 | Number of entries to return | -| `since` | string | | Time filter (e.g., "1 hour ago") | -| `priority` | string | | Minimum priority (emerg..debug or 0-7) | - -### Response Shape - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "entries": [ - { - "timestamp": "2026-03-31T22:30:45.123Z", - "unit": "nginx.service", - "priority": "info", - "message": "Started nginx", - "pid": 1234, - "hostname": "web-01" - } - ] - } - ] -} -``` - -## SDK - -```go -client.Log.Query(ctx, host, opts) -client.Log.QueryUnit(ctx, host, unit, opts) -``` - -`LogQueryOpts` struct with optional `Lines`, `Since`, `Priority`. - -## Permissions - -- `log:read` — query journal entries. Added to admin, write, and read roles. diff --git a/docs/plans/2026-03-31-log-management-provider.md b/docs/plans/2026-03-31-log-management-provider.md deleted file mode 100644 index 5421a5836..000000000 --- a/docs/plans/2026-03-31-log-management-provider.md +++ /dev/null @@ -1,2771 +0,0 @@ -# Log Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add read-only log viewing to OSAPI via `journalctl --output=json`, -with optional filtering by lines, time range, and priority. - -**Architecture:** Direct provider at `internal/provider/node/log/` using -`exec.Manager` to run `journalctl`. Two API endpoints: query all journal entries -and query by unit name. Read-only with `log:read` permission added to all -built-in roles. - -**Tech Stack:** Go, exec.Manager, journalctl JSON output, oapi-codegen -strict-server - ---- - -## File Structure - -### Provider Layer - -- Create: `internal/provider/node/log/types.go` — Provider interface + domain - types -- Create: `internal/provider/node/log/debian.go` — journalctl implementation -- Create: `internal/provider/node/log/debian_query.go` — shared query logic -- Create: `internal/provider/node/log/darwin.go` — macOS stub -- Create: `internal/provider/node/log/linux.go` — generic Linux stub -- Create: `internal/provider/node/log/mocks/generate.go` — mockgen directive -- Test: `internal/provider/node/log/debian_public_test.go` -- Test: `internal/provider/node/log/darwin_public_test.go` -- Test: `internal/provider/node/log/linux_public_test.go` - -### Agent Layer - -- Create: `internal/agent/processor_log.go` — log operation dispatcher -- Modify: `internal/agent/processor.go` — add `log` case + logProvider param -- Modify: `cmd/agent_setup.go` — create log provider factory, wire into registry -- Test: `internal/agent/processor_log_public_test.go` - -### API Layer - -- Create: `internal/controller/api/node/log/gen/api.yaml` — OpenAPI spec -- Create: `internal/controller/api/node/log/gen/cfg.yaml` — oapi-codegen config -- Create: `internal/controller/api/node/log/gen/generate.go` — go:generate -- Create: `internal/controller/api/node/log/types.go` — handler struct -- Create: `internal/controller/api/node/log/log.go` — New(), compile-time check -- Create: `internal/controller/api/node/log/validate.go` — validateHostname -- Create: `internal/controller/api/node/log/log_query_get.go` — query handler -- Create: `internal/controller/api/node/log/log_unit_get.go` — query unit - handler -- Create: `internal/controller/api/node/log/handler.go` — Handler() registration -- Modify: `cmd/controller_setup.go` — register log handler -- Test: `internal/controller/api/node/log/log_query_get_public_test.go` -- Test: `internal/controller/api/node/log/log_unit_get_public_test.go` -- Test: `internal/controller/api/node/log/handler_public_test.go` - -### Operations & Permissions - -- Modify: `pkg/sdk/client/operations.go` — add log operation constants -- Modify: `internal/job/types.go` — add log operation aliases -- Modify: `pkg/sdk/client/permissions.go` — add `PermLogRead` -- Modify: `internal/authtoken/permissions.go` — add `PermLogRead` to all roles - -### SDK Layer - -- Create: `pkg/sdk/client/log.go` — LogService methods -- Create: `pkg/sdk/client/log_types.go` — SDK result types + conversions -- Modify: `pkg/sdk/client/osapi.go` — add Log field -- Test: `pkg/sdk/client/log_public_test.go` -- Test: `pkg/sdk/client/log_types_public_test.go` - -### CLI Layer - -- Create: `cmd/client_node_log.go` — parent command -- Create: `cmd/client_node_log_query.go` — query subcommand -- Create: `cmd/client_node_log_unit.go` — query-unit subcommand - -### Documentation - -- Create: `docs/docs/sidebar/features/log-management.md` — feature page -- Create: `docs/docs/sidebar/usage/cli/client/node/log/log.md` — CLI landing -- Create: `docs/docs/sidebar/usage/cli/client/node/log/query.md` — query CLI doc -- Create: `docs/docs/sidebar/usage/cli/client/node/log/unit.md` — unit CLI doc -- Create: `docs/docs/sidebar/sdk/client/operations/log.md` — SDK doc -- Create: `examples/sdk/client/log.go` — SDK example -- Modify: `docs/docs/sidebar/features/features.md` — add log to table -- Modify: `docs/docs/sidebar/features/authentication.md` — add log:read to roles -- Modify: `docs/docs/sidebar/usage/configuration.md` — add log:read to - permissions -- Modify: `docs/docs/sidebar/architecture/architecture.md` — add log feature - link -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` — add log endpoints -- Modify: `docs/docusaurus.config.ts` — add SDK dropdown + features dropdown - -### Integration Test - -- Create: `test/integration/log_test.go` — smoke test - ---- - -### Task 1: Provider Interface and Types - -**Files:** - -- Create: `internal/provider/node/log/types.go` - -- [ ] **Step 1: Create provider interface and types** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -// Package log provides log viewing operations. -package log - -import ( - "context" -) - -// Provider implements log viewing operations. -type Provider interface { - // Query returns journal entries with optional filtering. - Query(ctx context.Context, opts QueryOpts) ([]Entry, error) - // QueryUnit returns journal entries for a specific systemd unit. - QueryUnit(ctx context.Context, unit string, opts QueryOpts) ([]Entry, error) -} - -// QueryOpts contains optional filters for log queries. -type QueryOpts struct { - Lines int `json:"lines,omitempty"` - Since string `json:"since,omitempty"` - Priority string `json:"priority,omitempty"` -} - -// Entry represents a single journal entry. -type Entry struct { - Timestamp string `json:"timestamp"` - Unit string `json:"unit,omitempty"` - Priority string `json:"priority"` - Message string `json:"message"` - PID int `json:"pid,omitempty"` - Hostname string `json:"hostname,omitempty"` -} -``` - -- [ ] **Step 2: Verify it compiles** - -Run: `go build ./internal/provider/node/log/...` Expected: PASS - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/node/log/types.go -git commit -m "feat(log): add provider interface and types" -``` - ---- - -### Task 2: Platform Stubs (Darwin + Linux) - -**Files:** - -- Create: `internal/provider/node/log/darwin.go` -- Create: `internal/provider/node/log/linux.go` -- Test: `internal/provider/node/log/darwin_public_test.go` -- Test: `internal/provider/node/log/linux_public_test.go` - -- [ ] **Step 1: Write darwin stub tests** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log_test - -import ( - "context" - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/suite" - - logProv "github.com/osapi-io/osapi/internal/provider/node/log" - "github.com/osapi-io/osapi/internal/provider" -) - -type DarwinPublicTestSuite struct { - suite.Suite - - provider *logProv.Darwin -} - -func (s *DarwinPublicTestSuite) SetupTest() { - s.provider = logProv.NewDarwinProvider() -} - -func (s *DarwinPublicTestSuite) TestQuery() { - _, err := s.provider.Query(context.Background(), logProv.QueryOpts{}) - - assert.ErrorIs(s.T(), err, provider.ErrUnsupported) -} - -func (s *DarwinPublicTestSuite) TestQueryUnit() { - _, err := s.provider.QueryUnit(context.Background(), "nginx.service", logProv.QueryOpts{}) - - assert.ErrorIs(s.T(), err, provider.ErrUnsupported) -} - -func TestDarwinPublicTestSuite(t *testing.T) { - suite.Run(t, new(DarwinPublicTestSuite)) -} -``` - -- [ ] **Step 2: Write linux stub tests** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log_test - -import ( - "context" - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/suite" - - logProv "github.com/osapi-io/osapi/internal/provider/node/log" - "github.com/osapi-io/osapi/internal/provider" -) - -type LinuxPublicTestSuite struct { - suite.Suite - - provider *logProv.Linux -} - -func (s *LinuxPublicTestSuite) SetupTest() { - s.provider = logProv.NewLinuxProvider() -} - -func (s *LinuxPublicTestSuite) TestQuery() { - _, err := s.provider.Query(context.Background(), logProv.QueryOpts{}) - - assert.ErrorIs(s.T(), err, provider.ErrUnsupported) -} - -func (s *LinuxPublicTestSuite) TestQueryUnit() { - _, err := s.provider.QueryUnit(context.Background(), "nginx.service", logProv.QueryOpts{}) - - assert.ErrorIs(s.T(), err, provider.ErrUnsupported) -} - -func TestLinuxPublicTestSuite(t *testing.T) { - suite.Run(t, new(LinuxPublicTestSuite)) -} -``` - -- [ ] **Step 3: Run tests to verify they fail** - -Run: `go test -v ./internal/provider/node/log/...` Expected: FAIL — Darwin and -Linux types don't exist yet - -- [ ] **Step 4: Implement darwin stub** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log - -import ( - "context" - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// Darwin implements the Provider interface for Darwin (macOS). -// All methods return ErrUnsupported as log viewing is not available on macOS. -type Darwin struct{} - -// NewDarwinProvider factory to create a new Darwin instance. -func NewDarwinProvider() *Darwin { - return &Darwin{} -} - -// Query returns ErrUnsupported on Darwin. -func (d *Darwin) Query( - _ context.Context, - _ QueryOpts, -) ([]Entry, error) { - return nil, fmt.Errorf("log: %w", provider.ErrUnsupported) -} - -// QueryUnit returns ErrUnsupported on Darwin. -func (d *Darwin) QueryUnit( - _ context.Context, - _ string, - _ QueryOpts, -) ([]Entry, error) { - return nil, fmt.Errorf("log: %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 5: Implement linux stub** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log - -import ( - "context" - "fmt" - - "github.com/osapi-io/osapi/internal/provider" -) - -// Linux implements the Provider interface for generic Linux. -// All methods return ErrUnsupported as this is a generic Linux stub. -type Linux struct{} - -// NewLinuxProvider factory to create a new Linux instance. -func NewLinuxProvider() *Linux { - return &Linux{} -} - -// Query returns ErrUnsupported on generic Linux. -func (l *Linux) Query( - _ context.Context, - _ QueryOpts, -) ([]Entry, error) { - return nil, fmt.Errorf("log: %w", provider.ErrUnsupported) -} - -// QueryUnit returns ErrUnsupported on generic Linux. -func (l *Linux) QueryUnit( - _ context.Context, - _ string, - _ QueryOpts, -) ([]Entry, error) { - return nil, fmt.Errorf("log: %w", provider.ErrUnsupported) -} -``` - -- [ ] **Step 6: Run tests to verify they pass** - -Run: `go test -v ./internal/provider/node/log/...` Expected: PASS — all 4 tests -pass - -- [ ] **Step 7: Commit** - -```bash -git add internal/provider/node/log/darwin.go internal/provider/node/log/linux.go \ - internal/provider/node/log/darwin_public_test.go internal/provider/node/log/linux_public_test.go -git commit -m "feat(log): add darwin and linux provider stubs" -``` - ---- - -### Task 3: Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/log/debian.go` -- Create: `internal/provider/node/log/debian_query.go` -- Create: `internal/provider/node/log/mocks/generate.go` -- Test: `internal/provider/node/log/debian_public_test.go` - -- [ ] **Step 1: Create mock generator** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -// Package mocks provides mock implementations for testing. -package mocks - -//go:generate go tool github.com/golang/mock/mockgen -source=../types.go -destination=provider.gen.go -package=mocks -``` - -- [ ] **Step 2: Generate mocks** - -Run: `go generate ./internal/provider/node/log/mocks/...` Expected: generates -`provider.gen.go` - -- [ ] **Step 3: Write debian provider tests** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log_test - -import ( - "context" - "fmt" - "log/slog" - "os" - "testing" - - "github.com/golang/mock/gomock" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/suite" - - "github.com/osapi-io/osapi/internal/exec" - execMocks "github.com/osapi-io/osapi/internal/exec/mocks" - logProv "github.com/osapi-io/osapi/internal/provider/node/log" -) - -type DebianPublicTestSuite struct { - suite.Suite - - mockCtrl *gomock.Controller - mockExecManager *execMocks.MockManager - provider *logProv.Debian - ctx context.Context - logger *slog.Logger -} - -func (s *DebianPublicTestSuite) SetupTest() { - s.mockCtrl = gomock.NewController(s.T()) - s.mockExecManager = execMocks.NewMockManager(s.mockCtrl) - s.logger = slog.New(slog.NewTextHandler(os.Stdout, nil)) - s.provider = logProv.NewDebianProvider(s.logger, s.mockExecManager) - s.ctx = context.Background() -} - -func (s *DebianPublicTestSuite) TearDownTest() { - s.mockCtrl.Finish() -} - -func (s *DebianPublicTestSuite) TestQuery() { - journalLine1 := `{"__REALTIME_TIMESTAMP":"1711929045123456","SYSLOG_IDENTIFIER":"nginx","PRIORITY":"6","MESSAGE":"Started nginx","_PID":"1234","_HOSTNAME":"web-01"}` - journalLine2 := `{"__REALTIME_TIMESTAMP":"1711929046000000","SYSLOG_IDENTIFIER":"sshd","PRIORITY":"4","MESSAGE":"Connection closed","_PID":"5678","_HOSTNAME":"web-01"}` - journalOutput := journalLine1 + "\n" + journalLine2 + "\n" - - tests := []struct { - name string - opts logProv.QueryOpts - setupMock func() - validateFunc func(entries []logProv.Entry, err error) - }{ - { - name: "default query with no options", - opts: logProv.QueryOpts{}, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{"--output=json", "-n", "100"}). - Return(journalOutput, nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Len(s.T(), entries, 2) - assert.Equal(s.T(), "nginx", entries[0].Unit) - assert.Equal(s.T(), "info", entries[0].Priority) - assert.Equal(s.T(), "Started nginx", entries[0].Message) - assert.Equal(s.T(), 1234, entries[0].PID) - assert.Equal(s.T(), "web-01", entries[0].Hostname) - assert.Equal(s.T(), "sshd", entries[1].Unit) - assert.Equal(s.T(), "warning", entries[1].Priority) - }, - }, - { - name: "query with all options", - opts: logProv.QueryOpts{ - Lines: 50, - Since: "1 hour ago", - Priority: "err", - }, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{ - "--output=json", - "-n", "50", - "--since=1 hour ago", - "--priority=err", - }). - Return(journalOutput, nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Len(s.T(), entries, 2) - }, - }, - { - name: "query with custom lines", - opts: logProv.QueryOpts{Lines: 10}, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{"--output=json", "-n", "10"}). - Return(journalOutput, nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Len(s.T(), entries, 2) - }, - }, - { - name: "exec error", - opts: logProv.QueryOpts{}, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{"--output=json", "-n", "100"}). - Return("", fmt.Errorf("exec failed")) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.Error(s.T(), err) - assert.Nil(s.T(), entries) - assert.Contains(s.T(), err.Error(), "log: query") - }, - }, - { - name: "empty output", - opts: logProv.QueryOpts{}, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{"--output=json", "-n", "100"}). - Return("", nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Empty(s.T(), entries) - }, - }, - { - name: "malformed JSON line skipped", - opts: logProv.QueryOpts{}, - setupMock: func() { - output := "not json\n" + journalLine1 + "\n" - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{"--output=json", "-n", "100"}). - Return(output, nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Len(s.T(), entries, 1) - assert.Equal(s.T(), "nginx", entries[0].Unit) - }, - }, - } - - for _, tc := range tests { - s.Run(tc.name, func() { - tc.setupMock() - entries, err := s.provider.Query(s.ctx, tc.opts) - tc.validateFunc(entries, err) - }) - } -} - -func (s *DebianPublicTestSuite) TestQueryUnit() { - journalLine := `{"__REALTIME_TIMESTAMP":"1711929045123456","SYSLOG_IDENTIFIER":"nginx","PRIORITY":"6","MESSAGE":"Started nginx","_PID":"1234","_HOSTNAME":"web-01"}` - - tests := []struct { - name string - unit string - opts logProv.QueryOpts - setupMock func() - validateFunc func(entries []logProv.Entry, err error) - }{ - { - name: "query unit with defaults", - unit: "nginx.service", - opts: logProv.QueryOpts{}, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{ - "--output=json", - "-u", "nginx.service", - "-n", "100", - }). - Return(journalLine+"\n", nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Len(s.T(), entries, 1) - assert.Equal(s.T(), "nginx", entries[0].Unit) - }, - }, - { - name: "query unit with all options", - unit: "sshd.service", - opts: logProv.QueryOpts{ - Lines: 25, - Since: "2026-03-31", - Priority: "warning", - }, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", []string{ - "--output=json", - "-u", "sshd.service", - "-n", "25", - "--since=2026-03-31", - "--priority=warning", - }). - Return(journalLine+"\n", nil) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.NoError(s.T(), err) - assert.Len(s.T(), entries, 1) - }, - }, - { - name: "exec error", - unit: "nginx.service", - opts: logProv.QueryOpts{}, - setupMock: func() { - s.mockExecManager.EXPECT(). - RunCmd("journalctl", gomock.Any()). - Return("", fmt.Errorf("exec failed")) - }, - validateFunc: func(entries []logProv.Entry, err error) { - assert.Error(s.T(), err) - assert.Nil(s.T(), entries) - assert.Contains(s.T(), err.Error(), "log: query unit") - }, - }, - } - - for _, tc := range tests { - s.Run(tc.name, func() { - tc.setupMock() - entries, err := s.provider.QueryUnit(s.ctx, tc.unit, tc.opts) - tc.validateFunc(entries, err) - }) - } -} - -func TestDebianPublicTestSuite(t *testing.T) { - suite.Run(t, new(DebianPublicTestSuite)) -} -``` - -- [ ] **Step 4: Run tests to verify they fail** - -Run: `go test -v ./internal/provider/node/log/...` Expected: FAIL — Debian type -doesn't exist yet - -- [ ] **Step 5: Implement debian.go** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log - -import ( - "context" - "fmt" - "log/slog" - - "github.com/osapi-io/osapi/internal/exec" - "github.com/osapi-io/osapi/internal/provider" -) - -// Compile-time checks. -var ( - _ Provider = (*Debian)(nil) - _ provider.FactsSetter = (*Debian)(nil) -) - -// Debian implements the Provider interface for Debian-family systems. -type Debian struct { - provider.FactsAware - logger *slog.Logger - execManager exec.Manager -} - -// NewDebianProvider factory to create a new Debian instance. -func NewDebianProvider( - logger *slog.Logger, - execManager exec.Manager, -) *Debian { - return &Debian{ - logger: logger.With(slog.String("subsystem", "provider.log")), - execManager: execManager, - } -} - -// Query returns journal entries with optional filtering. -func (d *Debian) Query( - _ context.Context, - opts QueryOpts, -) ([]Entry, error) { - d.logger.Debug("executing log.Query") - - args := buildArgs(opts) - - output, err := d.execManager.RunCmd("journalctl", args) - if err != nil { - return nil, fmt.Errorf("log: query: %w", err) - } - - return parseJournalOutput(output, d.logger), nil -} - -// QueryUnit returns journal entries for a specific systemd unit. -func (d *Debian) QueryUnit( - _ context.Context, - unit string, - opts QueryOpts, -) ([]Entry, error) { - d.logger.Debug("executing log.QueryUnit", - slog.String("unit", unit), - ) - - args := buildUnitArgs(unit, opts) - - output, err := d.execManager.RunCmd("journalctl", args) - if err != nil { - return nil, fmt.Errorf("log: query unit: %w", err) - } - - return parseJournalOutput(output, d.logger), nil -} -``` - -- [ ] **Step 6: Implement debian_query.go** - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package log - -import ( - "encoding/json" - "fmt" - "log/slog" - "strconv" - "strings" - "time" -) - -// journalEntry represents the raw JSON output from journalctl --output=json. -type journalEntry struct { - RealtimeTimestamp string `json:"__REALTIME_TIMESTAMP"` - SyslogIdentifier string `json:"SYSLOG_IDENTIFIER"` - Priority string `json:"PRIORITY"` - Message string `json:"MESSAGE"` - PID string `json:"_PID"` - Hostname string `json:"_HOSTNAME"` -} - -// priorityNames maps journalctl numeric priorities to human-readable names. -var priorityNames = map[string]string{ - "0": "emerg", - "1": "alert", - "2": "crit", - "3": "err", - "4": "warning", - "5": "notice", - "6": "info", - "7": "debug", -} - -// buildArgs constructs the journalctl command arguments for a general query. -func buildArgs( - opts QueryOpts, -) []string { - args := []string{"--output=json"} - - lines := opts.Lines - if lines <= 0 { - lines = 100 - } - args = append(args, "-n", fmt.Sprintf("%d", lines)) - - if opts.Since != "" { - args = append(args, "--since="+opts.Since) - } - - if opts.Priority != "" { - args = append(args, "--priority="+opts.Priority) - } - - return args -} - -// buildUnitArgs constructs the journalctl command arguments for a unit query. -func buildUnitArgs( - unit string, - opts QueryOpts, -) []string { - args := []string{"--output=json", "-u", unit} - - lines := opts.Lines - if lines <= 0 { - lines = 100 - } - args = append(args, "-n", fmt.Sprintf("%d", lines)) - - if opts.Since != "" { - args = append(args, "--since="+opts.Since) - } - - if opts.Priority != "" { - args = append(args, "--priority="+opts.Priority) - } - - return args -} - -// parseJournalOutput parses the JSON lines output from journalctl. -func parseJournalOutput( - output string, - logger *slog.Logger, -) []Entry { - lines := strings.Split(strings.TrimSpace(output), "\n") - var entries []Entry - - for _, line := range lines { - line = strings.TrimSpace(line) - if line == "" { - continue - } - - var je journalEntry - if err := json.Unmarshal([]byte(line), &je); err != nil { - logger.Debug("skipping malformed journal line", - slog.String("error", err.Error()), - ) - continue - } - - entries = append(entries, journalEntryToEntry(je)) - } - - return entries -} - -// journalEntryToEntry converts a raw journal entry to the provider Entry type. -func journalEntryToEntry( - je journalEntry, -) Entry { - ts := parseTimestamp(je.RealtimeTimestamp) - priority := priorityNames[je.Priority] - if priority == "" { - priority = je.Priority - } - - var pid int - if je.PID != "" { - pid, _ = strconv.Atoi(je.PID) - } - - return Entry{ - Timestamp: ts, - Unit: je.SyslogIdentifier, - Priority: priority, - Message: je.Message, - PID: pid, - Hostname: je.Hostname, - } -} - -// parseTimestamp converts a journalctl __REALTIME_TIMESTAMP (microseconds -// since epoch) to RFC3339 format. -func parseTimestamp( - usec string, -) string { - us, err := strconv.ParseInt(usec, 10, 64) - if err != nil { - return usec - } - - t := time.UnixMicro(us).UTC() - - return t.Format(time.RFC3339Nano) -} -``` - -- [ ] **Step 7: Run tests to verify they pass** - -Run: `go test -v ./internal/provider/node/log/...` Expected: PASS — all tests -pass - -- [ ] **Step 8: Commit** - -```bash -git add internal/provider/node/log/debian.go internal/provider/node/log/debian_query.go \ - internal/provider/node/log/debian_public_test.go \ - internal/provider/node/log/mocks/ -git commit -m "feat(log): add debian provider with journalctl parsing" -``` - ---- - -### Task 4: Operations, Permissions, and Agent Wiring - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` — add log operations -- Modify: `internal/job/types.go` — add log operation aliases -- Modify: `pkg/sdk/client/permissions.go` — add `PermLogRead` -- Modify: `internal/authtoken/permissions.go` — add `PermLogRead` to all roles -- Create: `internal/agent/processor_log.go` — log dispatcher -- Modify: `internal/agent/processor.go` — add `log` case + logProvider param -- Modify: `cmd/agent_setup.go` — create log provider, wire into registry -- Test: `internal/agent/processor_log_public_test.go` - -- [ ] **Step 1: Add operation constants to SDK** - -Add to `pkg/sdk/client/operations.go` after the Package operations block: - -```go -// Log operations. -const ( - OpLogQuery JobOperation = "node.log.query" - OpLogQueryUnit JobOperation = "node.log.queryUnit" -) -``` - -- [ ] **Step 2: Add operation aliases to job types** - -Add to `internal/job/types.go` after the Package operations block: - -```go -// Log operations. -const ( - OperationLogQuery = client.OpLogQuery - OperationLogQueryUnit = client.OpLogQueryUnit -) -``` - -- [ ] **Step 3: Add permission constant to SDK** - -Add to `pkg/sdk/client/permissions.go` after the Package permissions: - -```go - PermLogRead Permission = "log:read" -``` - -- [ ] **Step 4: Add permission to authtoken** - -Add to `internal/authtoken/permissions.go`: - -1. Add constant after PackageWrite: - -```go - PermLogRead = client.PermLogRead -``` - -2. Add to `AllPermissions` slice: - -```go - PermLogRead, -``` - -3. Add to admin role after `PermPackageWrite`: - -```go - PermLogRead, -``` - -4. Add to write role after `PermPackageWrite`: - -```go - PermLogRead, -``` - -5. Add to read role after `PermPackageRead`: - -```go - PermLogRead, -``` - -- [ ] **Step 5: Write agent processor tests** - -Create `internal/agent/processor_log_public_test.go`. The tests exercise -`processLogOperation` via the node processor's `log` case. Follow the same -pattern as `processor_process_public_test.go` — table-driven tests with gomock -for the log provider mock. Test cases: - -- `log.query` with default opts (empty data) -- `log.query` with all options (lines, since, priority) -- `log.queryUnit` with unit name -- unsupported sub-operation (`log.invalid`) -- invalid operation format (`log` with no sub-op) -- nil log provider - -The tests construct a `job.Request` with `Operation: "log.query"` (note: the -node processor strips the base operation from the dotted format `"hostname.get"` -→ `"hostname"`, but for log the operation string passed to `processLogOperation` -is already `"log.query"`, `"log.queryUnit"` etc. The node processor matches on -`baseOperation` which is `"log"`, then delegates to `processLogOperation` which -splits on `.` to get the sub-op). - -- [ ] **Step 6: Run tests to verify they fail** - -Run: `go test -run TestProcessorLogPublicTestSuite -v ./internal/agent/...` -Expected: FAIL — `processLogOperation` doesn't exist - -- [ ] **Step 7: Implement processor_log.go** - -Create `internal/agent/processor_log.go`: - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -package agent - -import ( - "context" - "encoding/json" - "fmt" - "log/slog" - "strings" - - "github.com/osapi-io/osapi/internal/job" - logProv "github.com/osapi-io/osapi/internal/provider/node/log" -) - -// processLogOperation dispatches log sub-operations. -func processLogOperation( - logProvider logProv.Provider, - logger *slog.Logger, - jobRequest job.Request, -) (json.RawMessage, error) { - if logProvider == nil { - return nil, fmt.Errorf("log provider not available") - } - - parts := strings.Split(jobRequest.Operation, ".") - if len(parts) < 2 { - return nil, fmt.Errorf("invalid log operation: %s", jobRequest.Operation) - } - subOp := parts[1] - - ctx := context.Background() - - switch subOp { - case "query": - return processLogQuery(ctx, logProvider, logger, jobRequest) - case "queryUnit": - return processLogQueryUnit(ctx, logProvider, logger, jobRequest) - default: - return nil, fmt.Errorf("unsupported log operation: %s", jobRequest.Operation) - } -} - -// processLogQuery handles the log.query operation. -func processLogQuery( - ctx context.Context, - logProvider logProv.Provider, - logger *slog.Logger, - jobRequest job.Request, -) (json.RawMessage, error) { - logger.Debug("executing log.Query") - - var opts logProv.QueryOpts - if jobRequest.Data != nil { - _ = json.Unmarshal(jobRequest.Data, &opts) - } - - result, err := logProvider.Query(ctx, opts) - if err != nil { - return nil, err - } - - return json.Marshal(result) -} - -// processLogQueryUnit handles the log.queryUnit operation. -func processLogQueryUnit( - ctx context.Context, - logProvider logProv.Provider, - logger *slog.Logger, - jobRequest job.Request, -) (json.RawMessage, error) { - logger.Debug("executing log.QueryUnit") - - var data struct { - Unit string `json:"unit"` - logProv.QueryOpts - } - if err := json.Unmarshal(jobRequest.Data, &data); err != nil { - return nil, fmt.Errorf("unmarshal log query unit data: %w", err) - } - - result, err := logProvider.QueryUnit(ctx, data.Unit, data.QueryOpts) - if err != nil { - return nil, err - } - - return json.Marshal(result) -} -``` - -- [ ] **Step 8: Wire log into NewNodeProcessor** - -Add `logProvider logProv.Provider` parameter to `NewNodeProcessor` in -`internal/agent/processor.go`. Add the import: - -```go -logProv "github.com/osapi-io/osapi/internal/provider/node/log" -``` - -Add the case in the switch: - -```go - case "log": - return processLogOperation(logProvider, logger, req) -``` - -- [ ] **Step 9: Create log provider factory in agent_setup.go** - -Add import: - -```go -logProv "github.com/osapi-io/osapi/internal/provider/node/log" -``` - -Add factory function: - -```go -// createLogProvider creates a platform-specific log provider. On Debian, the -// log provider reads journal entries via journalctl. In containers, journalctl -// is not available — returns ErrUnsupported. On other platforms, all operations -// return ErrUnsupported. -func createLogProvider( - log *slog.Logger, - execManager exec.Manager, -) logProv.Provider { - plat := platform.Detect() - - switch plat { - case "debian": - if platform.IsContainer() { - log.Info("running in container, log operations disabled") - return logProv.NewLinuxProvider() - } - return logProv.NewDebianProvider(log, execManager) - case "darwin": - return logProv.NewDarwinProvider() - default: - return logProv.NewLinuxProvider() - } -} -``` - -Add to `setupAgent` after `packageProvider`: - -```go - // --- Log provider --- - logProvider := createLogProvider(log, execManager) -``` - -Add `logProvider` to the `NewNodeProcessor` call and the providers list in -`registry.Register("node", ...)`. - -- [ ] **Step 10: Run tests** - -Run: `go test -v ./internal/agent/... && go build ./...` Expected: PASS - -- [ ] **Step 11: Commit** - -```bash -git add pkg/sdk/client/operations.go internal/job/types.go \ - pkg/sdk/client/permissions.go internal/authtoken/permissions.go \ - internal/agent/processor_log.go internal/agent/processor_log_public_test.go \ - internal/agent/processor.go cmd/agent_setup.go -git commit -m "feat(log): add operations, permissions, and agent wiring" -``` - ---- - -### Task 5: OpenAPI Spec and Code Generation - -**Files:** - -- Create: `internal/controller/api/node/log/gen/api.yaml` -- Create: `internal/controller/api/node/log/gen/cfg.yaml` -- Create: `internal/controller/api/node/log/gen/generate.go` - -- [ ] **Step 1: Create OpenAPI spec** - -Create `internal/controller/api/node/log/gen/api.yaml`: - -```yaml -# Copyright (c) 2026 John Dewey -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to -# deal in the Software without restriction, including without limitation the -# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -# sell copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -# DEALINGS IN THE SOFTWARE. - ---- -openapi: 3.0.0 -info: - title: Log Management API - version: 1.0.0 -tags: - - name: log_operations - x-displayName: Node/Log - description: Log viewing operations on a target node. - -paths: - /node/{hostname}/log: - get: - summary: Query journal entries - description: > - Query systemd journal entries on the target node with optional filtering - by lines, time range, and priority. - tags: - - log_operations - operationId: GetNodeLog - security: - - BearerAuth: - - log:read - parameters: - - $ref: '#/components/parameters/Hostname' - - name: lines - in: query - required: false - description: Number of entries to return (default 100). - x-oapi-codegen-extra-tags: - validate: omitempty,min=1,max=10000 - schema: - type: integer - default: 100 - minimum: 1 - maximum: 10000 - - name: since - in: query - required: false - description: > - Time filter in journalctl format (e.g., "1 hour ago", "2026-03-31"). - schema: - type: string - - name: priority - in: query - required: false - description: > - Minimum priority level (0-7 or name: emerg, alert, crit, err, - warning, notice, info, debug). - schema: - type: string - responses: - '200': - description: Journal entries. - content: - application/json: - schema: - $ref: '#/components/schemas/LogCollectionResponse' - '401': - description: Unauthorized - API key required - content: - application/json: - schema: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden - Insufficient permissions - content: - application/json: - schema: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Error querying journal. - content: - application/json: - schema: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - - /node/{hostname}/log/unit/{name}: - get: - summary: Query journal entries for a unit - description: > - Query systemd journal entries for a specific unit on the target node - with optional filtering. - tags: - - log_operations - operationId: GetNodeLogUnit - security: - - BearerAuth: - - log:read - parameters: - - $ref: '#/components/parameters/Hostname' - - $ref: '#/components/parameters/UnitName' - - name: lines - in: query - required: false - description: Number of entries to return (default 100). - x-oapi-codegen-extra-tags: - validate: omitempty,min=1,max=10000 - schema: - type: integer - default: 100 - minimum: 1 - maximum: 10000 - - name: since - in: query - required: false - description: > - Time filter in journalctl format (e.g., "1 hour ago", "2026-03-31"). - schema: - type: string - - name: priority - in: query - required: false - description: > - Minimum priority level (0-7 or name: emerg, alert, crit, err, - warning, notice, info, debug). - schema: - type: string - responses: - '200': - description: Journal entries for the unit. - content: - application/json: - schema: - $ref: '#/components/schemas/LogCollectionResponse' - '401': - description: Unauthorized - API key required - content: - application/json: - schema: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '403': - description: Forbidden - Insufficient permissions - content: - application/json: - schema: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - '500': - description: Error querying journal. - content: - application/json: - schema: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - -# -- Reusable components -- - -components: - parameters: - Hostname: - name: hostname - in: path - required: true - description: > - Target agent hostname, reserved routing value (_any, _all), or label - selector (key:value). - # NOTE: x-oapi-codegen-extra-tags on path params do not generate - # validate tags in strict-server mode. Validation is handled - # manually in handlers via validateHostname(). - x-oapi-codegen-extra-tags: - validate: required,min=1,valid_target - schema: - type: string - minLength: 1 - - UnitName: - name: name - in: path - required: true - description: > - Systemd unit name (e.g., nginx.service, sshd.service). - # NOTE: x-oapi-codegen-extra-tags on path params do not generate - # validate tags in strict-server mode. Validation is handled - # manually in the handler. - x-oapi-codegen-extra-tags: - validate: required,min=1 - schema: - type: string - minLength: 1 - - securitySchemes: - BearerAuth: - type: http - scheme: bearer - bearerFormat: JWT - - schemas: - ErrorResponse: - $ref: '../../../common/gen/api.yaml#/components/schemas/ErrorResponse' - - # -- Response schemas -- - - LogEntryInfo: - type: object - description: A single journal entry. - properties: - timestamp: - type: string - description: Entry timestamp in RFC3339 format. - example: '2026-03-31T22:30:45.123456Z' - unit: - type: string - description: Systemd unit or syslog identifier. - example: 'nginx.service' - priority: - type: string - description: Priority level name. - example: 'info' - message: - type: string - description: Log message. - example: 'Started nginx' - pid: - type: integer - description: Process ID that generated the entry. - example: 1234 - hostname: - type: string - description: Hostname where the entry originated. - example: 'web-01' - - LogResultEntry: - type: object - description: Log query result for a single agent. - properties: - hostname: - type: string - description: The hostname of the agent. - status: - type: string - enum: [ok, failed, skipped] - description: The status of the operation for this host. - entries: - type: array - description: Journal entries from this agent. - items: - $ref: '#/components/schemas/LogEntryInfo' - error: - type: string - description: Error message if the agent failed. - required: - - hostname - - status - - # -- Collection responses -- - - LogCollectionResponse: - type: object - properties: - job_id: - type: string - format: uuid - description: The job ID used to process this request. - example: '550e8400-e29b-41d4-a716-446655440000' - results: - type: array - items: - $ref: '#/components/schemas/LogResultEntry' - required: - - results -``` - -- [ ] **Step 2: Create oapi-codegen config** - -Create `internal/controller/api/node/log/gen/cfg.yaml`: - -```yaml -# Copyright (c) 2026 John Dewey -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to -# deal in the Software without restriction, including without limitation the -# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -# sell copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -# DEALINGS IN THE SOFTWARE. - ---- -package: gen -output: log.gen.go -generate: - models: true - echo-server: true - strict-server: true -import-mapping: - ../../../common/gen/api.yaml: github.com/osapi-io/osapi/internal/controller/api/common/gen -output-options: - # to make sure that all types are generated - skip-prune: true -``` - -- [ ] **Step 3: Create generate.go** - -Create `internal/controller/api/node/log/gen/generate.go`: - -```go -// Copyright (c) 2026 John Dewey - -// Permission is hereby granted, free of charge, to any person obtaining a copy -// of this software and associated documentation files (the "Software"), to -// deal in the Software without restriction, including without limitation the -// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the Software is -// furnished to do so, subject to the following conditions: - -// The above copyright notice and this permission notice shall be included in -// all copies or substantial portions of the Software. - -// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -// DEALINGS IN THE SOFTWARE. - -// Package gen contains generated code for the log API. -package gen - -//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen -config cfg.yaml api.yaml -``` - -- [ ] **Step 4: Generate code** - -Run: `go generate ./internal/controller/api/node/log/gen/...` Expected: -generates `log.gen.go` - -- [ ] **Step 5: Regenerate combined spec** - -Run: `just generate` Expected: combined spec updated, all code regenerates - -- [ ] **Step 6: Commit** - -```bash -git add internal/controller/api/node/log/gen/ -git commit -m "feat(log): add OpenAPI spec and generated code" -``` - ---- - -### Task 6: API Handler Implementation - -**Files:** - -- Create: `internal/controller/api/node/log/types.go` -- Create: `internal/controller/api/node/log/log.go` -- Create: `internal/controller/api/node/log/validate.go` -- Create: `internal/controller/api/node/log/log_query_get.go` -- Create: `internal/controller/api/node/log/log_unit_get.go` -- Create: `internal/controller/api/node/log/handler.go` -- Modify: `cmd/controller_setup.go` -- Test: `internal/controller/api/node/log/log_query_get_public_test.go` -- Test: `internal/controller/api/node/log/log_unit_get_public_test.go` -- Test: `internal/controller/api/node/log/handler_public_test.go` - -- [ ] **Step 1: Create handler types, factory, and validate** - -Create `internal/controller/api/node/log/types.go`: - -```go -package log - -import ( - "log/slog" - - "github.com/osapi-io/osapi/internal/job/client" -) - -// Log implementation of the Log APIs operations. -type Log struct { - // JobClient provides job-based operations for log management. - JobClient client.JobClient - logger *slog.Logger -} -``` - -Create `internal/controller/api/node/log/log.go`: - -```go -package log - -import ( - "log/slog" - - "github.com/osapi-io/osapi/internal/controller/api/node/log/gen" - "github.com/osapi-io/osapi/internal/job/client" -) - -// ensure that we've conformed to the `StrictServerInterface` with a compile-time check -var _ gen.StrictServerInterface = (*Log)(nil) - -// New factory to create a new instance. -func New( - logger *slog.Logger, - jobClient client.JobClient, -) *Log { - return &Log{ - JobClient: jobClient, - logger: logger.With(slog.String("subsystem", "api.log")), - } -} -``` - -Create `internal/controller/api/node/log/validate.go`: - -```go -package log - -import "github.com/osapi-io/osapi/internal/validation" - -// validateHostname validates a hostname path parameter using the shared -// validator. Returns the error message and false if invalid. -// -// This exists because oapi-codegen does not generate validate tags on -// path parameters in strict-server mode (upstream limitation). -func validateHostname( - hostname string, -) (string, bool) { - return validation.Var(hostname, "required,min=1,valid_target") -} -``` - -(All files need full license headers — follow existing patterns.) - -- [ ] **Step 2: Write handler tests for GetNodeLog** - -Create `internal/controller/api/node/log/log_query_get_public_test.go` — follow -the same pattern as `process_list_get_public_test.go`. Test cases: - -- success (single target) -- skipped (single target) -- broadcast success -- broadcast with failed/skipped hosts -- validation error (invalid hostname) -- job client error -- TestGetNodeLogHTTP (raw HTTP through middleware) -- TestGetNodeLogRBACHTTP (auth: 401, 403, 200) - -The test should mock `s.mockJobClient.EXPECT().Query(...)` with category -`"node"` and operation `job.OperationLogQuery`. The handler must pass query -params (`lines`, `since`, `priority`) as JSON data. - -- [ ] **Step 3: Write handler tests for GetNodeLogUnit** - -Create `internal/controller/api/node/log/log_unit_get_public_test.go` — same -pattern. Test cases: - -- success (single target) -- skipped (single target) -- broadcast success -- validation error -- job client error -- TestGetNodeLogUnitHTTP -- TestGetNodeLogUnitRBACHTTP - -The handler must pass `unit` (from path param) and query params as JSON data -with operation `job.OperationLogQueryUnit`. - -- [ ] **Step 4: Implement GetNodeLog handler** - -Create `internal/controller/api/node/log/log_query_get.go`: - -```go -package log - -import ( - "context" - "encoding/json" - "log/slog" - - "github.com/google/uuid" - - "github.com/osapi-io/osapi/internal/controller/api/node/log/gen" - "github.com/osapi-io/osapi/internal/job" - logProv "github.com/osapi-io/osapi/internal/provider/node/log" -) - -// GetNodeLog queries journal entries on a target node. -func (s *Log) GetNodeLog( - ctx context.Context, - request gen.GetNodeLogRequestObject, -) (gen.GetNodeLogResponseObject, error) { - if errMsg, ok := validateHostname(request.Hostname); !ok { - return gen.GetNodeLog500JSONResponse{Error: &errMsg}, nil - } - - hostname := request.Hostname - - // Defense in depth: current query fields use omitempty so validation - // always passes, but guards against future field additions. - if errMsg, ok := validation.Struct(request.Params); !ok { - return gen.GetNodeLog500JSONResponse{Error: &errMsg}, nil - } - - s.logger.Debug("log query", - slog.String("target", hostname), - slog.Bool("broadcast", job.IsBroadcastTarget(hostname)), - ) - - opts := logProv.QueryOpts{} - if request.Params.Lines != nil { - opts.Lines = *request.Params.Lines - } - if request.Params.Since != nil { - opts.Since = *request.Params.Since - } - if request.Params.Priority != nil { - opts.Priority = *request.Params.Priority - } - - data, _ := json.Marshal(opts) - - if job.IsBroadcastTarget(hostname) { - return s.getNodeLogBroadcast(ctx, hostname, data) - } - - jobID, resp, err := s.JobClient.Query(ctx, hostname, "node", job.OperationLogQuery, data) - if err != nil { - errMsg := err.Error() - return gen.GetNodeLog500JSONResponse{Error: &errMsg}, nil - } - - if resp.Status == job.StatusSkipped { - e := resp.Error - jobUUID := uuid.MustParse(jobID) - return gen.GetNodeLog200JSONResponse{ - JobId: &jobUUID, - Results: []gen.LogResultEntry{ - { - Hostname: resp.Hostname, - Status: gen.LogResultEntryStatusSkipped, - Error: &e, - }, - }, - }, nil - } - - entries := logEntriesFromResponse(resp) - jobUUID := uuid.MustParse(jobID) - - return gen.GetNodeLog200JSONResponse{ - JobId: &jobUUID, - Results: []gen.LogResultEntry{ - { - Hostname: resp.Hostname, - Status: gen.LogResultEntryStatusOk, - Entries: &entries, - }, - }, - }, nil -} - -// logEntriesFromResponse extracts LogEntryInfo slice from a job response. -func logEntriesFromResponse( - resp *job.Response, -) []gen.LogEntryInfo { - var provEntries []logProv.Entry - if resp.Data != nil { - _ = json.Unmarshal(resp.Data, &provEntries) - } - - result := make([]gen.LogEntryInfo, 0, len(provEntries)) - for _, e := range provEntries { - result = append(result, logEntryToGen(e)) - } - - return result -} - -// logEntryToGen converts a provider Entry to a gen.LogEntryInfo. -func logEntryToGen( - e logProv.Entry, -) gen.LogEntryInfo { - ts := e.Timestamp - unit := e.Unit - priority := e.Priority - message := e.Message - pid := e.PID - hostname := e.Hostname - - return gen.LogEntryInfo{ - Timestamp: &ts, - Unit: stringPtrOrNil(unit), - Priority: &priority, - Message: &message, - Pid: intPtrOrNil(pid), - Hostname: stringPtrOrNil(hostname), - } -} - -func stringPtrOrNil(s string) *string { - if s == "" { - return nil - } - return &s -} - -func intPtrOrNil(i int) *int { - if i == 0 { - return nil - } - return &i -} - -// getNodeLogBroadcast handles broadcast targets for log query. -func (s *Log) getNodeLogBroadcast( - ctx context.Context, - target string, - data json.RawMessage, -) (gen.GetNodeLogResponseObject, error) { - jobID, responses, err := s.JobClient.QueryBroadcast( - ctx, - target, - "node", - job.OperationLogQuery, - data, - ) - if err != nil { - errMsg := err.Error() - return gen.GetNodeLog500JSONResponse{Error: &errMsg}, nil - } - - var items []gen.LogResultEntry - for host, resp := range responses { - item := gen.LogResultEntry{ - Hostname: host, - } - switch resp.Status { - case job.StatusFailed: - item.Status = gen.LogResultEntryStatusFailed - e := resp.Error - item.Error = &e - case job.StatusSkipped: - item.Status = gen.LogResultEntryStatusSkipped - e := resp.Error - item.Error = &e - default: - item.Status = gen.LogResultEntryStatusOk - entries := logEntriesFromResponse(resp) - item.Entries = &entries - } - items = append(items, item) - } - - jobUUID := uuid.MustParse(jobID) - return gen.GetNodeLog200JSONResponse{ - JobId: &jobUUID, - Results: items, - }, nil -} -``` - -Note: The import for `validation` is -`"github.com/osapi-io/osapi/internal/validation"`. All files need full license -headers. - -- [ ] **Step 5: Implement GetNodeLogUnit handler** - -Create `internal/controller/api/node/log/log_unit_get.go` — same pattern as -`log_query_get.go` but adds unit from `request.Name` path param. Passes -`{"unit":"...","lines":...,"since":"...","priority":"..."}` as job data. Uses -`job.OperationLogQueryUnit`. - -- [ ] **Step 6: Implement handler.go** - -Create `internal/controller/api/node/log/handler.go` — same pattern as -`internal/controller/api/node/process/handler.go`: - -```go -package log - -import ( - "log/slog" - - "github.com/labstack/echo/v4" - strictecho "github.com/oapi-codegen/runtime/strictmiddleware/echo" - - "github.com/osapi-io/osapi/internal/authtoken" - "github.com/osapi-io/osapi/internal/controller/api" - gen "github.com/osapi-io/osapi/internal/controller/api/node/log/gen" - "github.com/osapi-io/osapi/internal/job/client" -) - -// Handler returns Log route registration functions. -func Handler( - logger *slog.Logger, - jobClient client.JobClient, - signingKey string, - customRoles map[string][]string, -) []func(e *echo.Echo) { - var tokenManager api.TokenValidator = authtoken.New(logger) - - logHandler := New(logger, jobClient) - - strictHandler := gen.NewStrictHandler( - logHandler, - []gen.StrictMiddlewareFunc{ - func(handler strictecho.StrictEchoHandlerFunc, _ string) strictecho.StrictEchoHandlerFunc { - return api.ScopeMiddleware( - handler, - tokenManager, - signingKey, - gen.BearerAuthScopes, - customRoles, - ) - }, - }, - ) - - return []func(e *echo.Echo){ - func(e *echo.Echo) { - gen.RegisterHandlers(e, strictHandler) - }, - } -} -``` - -- [ ] **Step 7: Register handler in controller_setup.go** - -Add import: - -```go -logAPI "github.com/osapi-io/osapi/internal/controller/api/node/log" -``` - -Add after the `packageAPI.Handler(...)` line: - -```go - handlers = append(handlers, logAPI.Handler(log, jc, signingKey, customRoles)...) -``` - -- [ ] **Step 8: Write handler_public_test.go** - -Test route registration and middleware execution (same pattern as other handler -tests). - -- [ ] **Step 9: Run tests** - -Run: `go test -v ./internal/controller/api/node/log/... && go build ./...` -Expected: PASS - -- [ ] **Step 10: Commit** - -```bash -git add internal/controller/api/node/log/ cmd/controller_setup.go -git commit -m "feat(log): add API handlers with broadcast support" -``` - ---- - -### Task 7: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/log.go` -- Create: `pkg/sdk/client/log_types.go` -- Modify: `pkg/sdk/client/osapi.go` -- Test: `pkg/sdk/client/log_public_test.go` -- Test: `pkg/sdk/client/log_types_public_test.go` - -- [ ] **Step 1: Write SDK service tests** - -Create `pkg/sdk/client/log_public_test.go` — test with `httptest.Server`. Test -cases for `Query` and `QueryUnit`: - -- success (200) -- auth error (401, 403) -- server error (500) -- nil response body -- transport error - -- [ ] **Step 2: Write SDK types tests** - -Create `pkg/sdk/client/log_types_public_test.go` — test conversion functions: - -- `logCollectionFromGen` with full data -- `logCollectionFromGen` with error entries -- `logEntryInfoFromGen` field mapping -- Nil/empty fields - -- [ ] **Step 3: Implement log_types.go** - -```go -package client - -import ( - "github.com/osapi-io/osapi/pkg/sdk/client/gen" -) - -// LogEntryResult represents the result of a log query for one host. -type LogEntryResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Entries []LogEntry `json:"entries,omitempty"` - Error string `json:"error,omitempty"` -} - -// LogEntry represents a single journal entry. -type LogEntry struct { - Timestamp string `json:"timestamp,omitempty"` - Unit string `json:"unit,omitempty"` - Priority string `json:"priority,omitempty"` - Message string `json:"message,omitempty"` - PID int `json:"pid,omitempty"` - Hostname string `json:"hostname,omitempty"` -} - -// LogQueryOpts contains options for log query operations. -type LogQueryOpts struct { - Lines *int - Since *string - Priority *string -} - -// logCollectionFromGen converts a gen.LogCollectionResponse -// to a Collection[LogEntryResult]. -func logCollectionFromGen( - g *gen.LogCollectionResponse, -) Collection[LogEntryResult] { - results := make([]LogEntryResult, 0, len(g.Results)) - for _, r := range g.Results { - results = append(results, logEntryResultFromGen(r)) - } - - return Collection[LogEntryResult]{ - Results: results, - JobID: jobIDFromGen(g.JobId), - } -} - -// logEntryResultFromGen converts a gen.LogResultEntry to a LogEntryResult. -func logEntryResultFromGen( - r gen.LogResultEntry, -) LogEntryResult { - result := LogEntryResult{ - Hostname: r.Hostname, - Status: string(r.Status), - Error: derefString(r.Error), - } - - if r.Entries != nil { - entries := make([]LogEntry, 0, len(*r.Entries)) - for _, e := range *r.Entries { - entries = append(entries, logEntryInfoFromGen(e)) - } - result.Entries = entries - } - - return result -} - -// logEntryInfoFromGen converts a gen.LogEntryInfo to a LogEntry. -func logEntryInfoFromGen( - e gen.LogEntryInfo, -) LogEntry { - return LogEntry{ - Timestamp: derefString(e.Timestamp), - Unit: derefString(e.Unit), - Priority: derefString(e.Priority), - Message: derefString(e.Message), - PID: derefInt(e.Pid), - Hostname: derefString(e.Hostname), - } -} -``` - -- [ ] **Step 4: Implement log.go** - -```go -package client - -import ( - "context" - "fmt" - - "github.com/osapi-io/osapi/pkg/sdk/client/gen" -) - -// LogService provides log viewing operations. -type LogService struct { - client *gen.ClientWithResponses -} - -// Query returns journal entries from the target host. -func (s *LogService) Query( - ctx context.Context, - hostname string, - opts LogQueryOpts, -) (*Response[Collection[LogEntryResult]], error) { - params := &gen.GetNodeLogParams{ - Lines: opts.Lines, - Since: opts.Since, - Priority: opts.Priority, - } - - resp, err := s.client.GetNodeLogWithResponse(ctx, hostname, params) - if err != nil { - return nil, fmt.Errorf("log query: %w", err) - } - - if err := checkError( - resp.StatusCode(), - resp.JSON401, - resp.JSON403, - resp.JSON500, - ); err != nil { - return nil, err - } - - if resp.JSON200 == nil { - return nil, &UnexpectedStatusError{APIError{ - StatusCode: resp.StatusCode(), - Message: "nil response body", - }} - } - - return NewResponse(logCollectionFromGen(resp.JSON200), resp.Body), nil -} - -// QueryUnit returns journal entries for a specific unit on the target host. -func (s *LogService) QueryUnit( - ctx context.Context, - hostname string, - unit string, - opts LogQueryOpts, -) (*Response[Collection[LogEntryResult]], error) { - params := &gen.GetNodeLogUnitParams{ - Lines: opts.Lines, - Since: opts.Since, - Priority: opts.Priority, - } - - resp, err := s.client.GetNodeLogUnitWithResponse(ctx, hostname, unit, params) - if err != nil { - return nil, fmt.Errorf("log query unit: %w", err) - } - - if err := checkError( - resp.StatusCode(), - resp.JSON401, - resp.JSON403, - resp.JSON500, - ); err != nil { - return nil, err - } - - if resp.JSON200 == nil { - return nil, &UnexpectedStatusError{APIError{ - StatusCode: resp.StatusCode(), - Message: "nil response body", - }} - } - - return NewResponse(logCollectionFromGen(resp.JSON200), resp.Body), nil -} -``` - -- [ ] **Step 5: Wire LogService in osapi.go** - -Add field to Client struct: - -```go - // Log provides log viewing operations (query journal entries). - Log *LogService -``` - -Add initialization in `New()`: - -```go - c.Log = &LogService{client: httpClient} -``` - -- [ ] **Step 6: Regenerate SDK client** - -Run: `go generate ./pkg/sdk/client/gen/...` Expected: SDK client picks up log -endpoints - -- [ ] **Step 7: Run tests** - -Run: `go test -v ./pkg/sdk/client/...` Expected: PASS - -- [ ] **Step 8: Commit** - -```bash -git add pkg/sdk/client/log.go pkg/sdk/client/log_types.go \ - pkg/sdk/client/log_public_test.go pkg/sdk/client/log_types_public_test.go \ - pkg/sdk/client/osapi.go pkg/sdk/client/gen/ -git commit -m "feat(log): add SDK service with tests" -``` - ---- - -### Task 8: CLI Commands - -**Files:** - -- Create: `cmd/client_node_log.go` -- Create: `cmd/client_node_log_query.go` -- Create: `cmd/client_node_log_unit.go` - -- [ ] **Step 1: Create parent command** - -Create `cmd/client_node_log.go`: - -```go -package cmd - -import ( - "github.com/spf13/cobra" -) - -// clientNodeLogCmd represents the clientNodeLog command. -var clientNodeLogCmd = &cobra.Command{ - Use: "log", - Short: "View journal logs", -} - -func init() { - clientNodeCmd.AddCommand(clientNodeLogCmd) -} -``` - -- [ ] **Step 2: Create query subcommand** - -Create `cmd/client_node_log_query.go`: - -```go -package cmd - -import ( - "fmt" - - "github.com/spf13/cobra" - - "github.com/osapi-io/osapi/internal/cli" - "github.com/osapi-io/osapi/pkg/sdk/client" -) - -// clientNodeLogQueryCmd represents the log query command. -var clientNodeLogQueryCmd = &cobra.Command{ - Use: "query", - Short: "Query journal entries", - Long: `Query systemd journal entries on the target node.`, - Run: func(cmd *cobra.Command, _ []string) { - ctx := cmd.Context() - host, _ := cmd.Flags().GetString("target") - lines, _ := cmd.Flags().GetInt("lines") - since, _ := cmd.Flags().GetString("since") - priority, _ := cmd.Flags().GetString("priority") - - opts := client.LogQueryOpts{} - if cmd.Flags().Changed("lines") { - opts.Lines = &lines - } - if since != "" { - opts.Since = &since - } - if priority != "" { - opts.Priority = &priority - } - - resp, err := sdkClient.Log.Query(ctx, host, opts) - if err != nil { - cli.HandleError(err, logger) - return - } - - if jsonOutput { - fmt.Println(string(resp.RawJSON())) - return - } - - if resp.Data.JobID != "" { - fmt.Println() - cli.PrintKV("Job ID", resp.Data.JobID) - fmt.Println() - } - - results := make([]cli.ResultRow, 0) - for _, r := range resp.Data.Results { - if r.Error != "" { - var errPtr *string - e := r.Error - errPtr = &e - results = append(results, cli.ResultRow{ - Hostname: r.Hostname, - Status: r.Status, - Error: errPtr, - }) - - continue - } - - for _, entry := range r.Entries { - message := entry.Message - if len(message) > 80 { - message = message[:77] + "..." - } - - results = append(results, cli.ResultRow{ - Hostname: r.Hostname, - Status: r.Status, - Fields: []string{ - entry.Timestamp, - entry.Priority, - entry.Unit, - message, - }, - }) - } - } - headers, rows := cli.BuildBroadcastTable( - results, - []string{"TIMESTAMP", "PRIORITY", "UNIT", "MESSAGE"}, - ) - cli.PrintCompactTable([]cli.Section{{Headers: headers, Rows: rows}}) - }, -} - -func init() { - clientNodeLogCmd.AddCommand(clientNodeLogQueryCmd) - - clientNodeLogQueryCmd.PersistentFlags(). - Int("lines", 100, "Number of entries to return") - clientNodeLogQueryCmd.PersistentFlags(). - String("since", "", "Time filter (e.g., \"1 hour ago\")") - clientNodeLogQueryCmd.PersistentFlags(). - String("priority", "", "Minimum priority (emerg..debug or 0-7)") -} -``` - -- [ ] **Step 3: Create query-unit subcommand** - -Create `cmd/client_node_log_unit.go`: - -```go -package cmd - -import ( - "fmt" - - "github.com/spf13/cobra" - - "github.com/osapi-io/osapi/internal/cli" - "github.com/osapi-io/osapi/pkg/sdk/client" -) - -// clientNodeLogUnitCmd represents the log unit command. -var clientNodeLogUnitCmd = &cobra.Command{ - Use: "unit", - Short: "Query journal entries for a unit", - Long: `Query systemd journal entries for a specific unit on the target node.`, - Run: func(cmd *cobra.Command, _ []string) { - ctx := cmd.Context() - host, _ := cmd.Flags().GetString("target") - unit, _ := cmd.Flags().GetString("name") - lines, _ := cmd.Flags().GetInt("lines") - since, _ := cmd.Flags().GetString("since") - priority, _ := cmd.Flags().GetString("priority") - - opts := client.LogQueryOpts{} - if cmd.Flags().Changed("lines") { - opts.Lines = &lines - } - if since != "" { - opts.Since = &since - } - if priority != "" { - opts.Priority = &priority - } - - resp, err := sdkClient.Log.QueryUnit(ctx, host, unit, opts) - if err != nil { - cli.HandleError(err, logger) - return - } - - if jsonOutput { - fmt.Println(string(resp.RawJSON())) - return - } - - if resp.Data.JobID != "" { - fmt.Println() - cli.PrintKV("Job ID", resp.Data.JobID) - fmt.Println() - } - - results := make([]cli.ResultRow, 0) - for _, r := range resp.Data.Results { - if r.Error != "" { - var errPtr *string - e := r.Error - errPtr = &e - results = append(results, cli.ResultRow{ - Hostname: r.Hostname, - Status: r.Status, - Error: errPtr, - }) - - continue - } - - for _, entry := range r.Entries { - message := entry.Message - if len(message) > 80 { - message = message[:77] + "..." - } - - results = append(results, cli.ResultRow{ - Hostname: r.Hostname, - Status: r.Status, - Fields: []string{ - entry.Timestamp, - entry.Priority, - entry.Unit, - message, - }, - }) - } - } - headers, rows := cli.BuildBroadcastTable( - results, - []string{"TIMESTAMP", "PRIORITY", "UNIT", "MESSAGE"}, - ) - cli.PrintCompactTable([]cli.Section{{Headers: headers, Rows: rows}}) - }, -} - -func init() { - clientNodeLogCmd.AddCommand(clientNodeLogUnitCmd) - - clientNodeLogUnitCmd.PersistentFlags(). - String("name", "", "Systemd unit name (required)") - clientNodeLogUnitCmd.PersistentFlags(). - Int("lines", 100, "Number of entries to return") - clientNodeLogUnitCmd.PersistentFlags(). - String("since", "", "Time filter (e.g., \"1 hour ago\")") - clientNodeLogUnitCmd.PersistentFlags(). - String("priority", "", "Minimum priority (emerg..debug or 0-7)") - - _ = clientNodeLogUnitCmd.MarkPersistentFlagRequired("name") -} -``` - -- [ ] **Step 4: Build and verify** - -Run: `go build ./... && go run main.go client node log --help` Expected: shows -`query` and `unit` subcommands - -- [ ] **Step 5: Commit** - -```bash -git add cmd/client_node_log.go cmd/client_node_log_query.go cmd/client_node_log_unit.go -git commit -m "feat(log): add CLI commands for journal log viewing" -``` - ---- - -### Task 9: Documentation and SDK Example - -**Files:** - -- Create: `docs/docs/sidebar/features/log-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/log/log.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/log/query.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/log/unit.md` -- Create: `docs/docs/sidebar/sdk/client/operations/log.md` -- Create: `examples/sdk/client/log.go` -- Modify: `docs/docs/sidebar/features/features.md` -- Modify: `docs/docs/sidebar/features/authentication.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` -- Modify: `docs/docusaurus.config.ts` - -- [ ] **Step 1: Create feature page** - -Create `docs/docs/sidebar/features/log-management.md` following the -process-management.md template. Include: - -- How It Works (Query, QueryUnit) -- Operations table -- CLI Usage examples -- Broadcast Support section -- Supported Platforms table (Debian: Full, Darwin: Skipped, Linux: Skipped) -- Container Behavior (skipped — journalctl requires systemd) -- Permissions table (log:read for all operations) -- Related links - -- [ ] **Step 2: Create CLI doc pages** - -Create landing page `docs/docs/sidebar/usage/cli/client/node/log/log.md`: - -```markdown ---- -sidebar_position: 1 ---- - -# Log - - -``` - -Create `query.md` and `unit.md` pages with usage examples, flags, and output -samples. - -- [ ] **Step 3: Create SDK doc page** - -Create `docs/docs/sidebar/sdk/client/operations/log.md` following existing SDK -doc patterns. Document `Query` and `QueryUnit` methods with code examples. - -- [ ] **Step 4: Create SDK example** - -Create `examples/sdk/client/log.go` — demonstrate `Query` and `QueryUnit` with -error handling and result printing. Under ~100 lines. - -- [ ] **Step 5: Update cross-references** - -Update these files to add log management: - -- `features/features.md` — add row to features table -- `features/authentication.md` — add `log:read` to all three role tables -- `usage/configuration.md` — add `log:read` to permissions comments and role - tables -- `architecture/architecture.md` — add log feature link -- `architecture/api-guidelines.md` — add log endpoint rows to path pattern table -- `docusaurus.config.ts` — add to Features dropdown and SDK dropdown - -- [ ] **Step 6: Commit** - -```bash -git add docs/ examples/sdk/client/log.go -git commit -m "docs: add log management feature docs, SDK example, and cross-references" -``` - ---- - -### Task 10: Integration Test - -**Files:** - -- Create: `test/integration/log_test.go` - -- [ ] **Step 1: Write integration test** - -Create `test/integration/log_test.go` with `//go:build integration` tag. Follow -the pattern of existing integration tests. Test: - -- `osapi client node log query --target _any --json` → verify JSON output -- `osapi client node log unit --target _any --name sshd.service --json` → verify - JSON output or graceful error - -- [ ] **Step 2: Commit** - -```bash -git add test/integration/log_test.go -git commit -m "test(log): add integration test" -``` - ---- - -### Task 11: Final Verification - -- [ ] **Step 1: Run full test suite** - -```bash -just generate -go build ./... -just go::unit -just go::vet -``` - -Expected: all pass, lint clean - -- [ ] **Step 2: Commit any fixes** - -If `just generate` produces diffs (combined spec, formatting), commit them: - -```bash -git add -A -git commit -m "chore(log): regenerate specs and fix formatting" -``` diff --git a/docs/plans/2026-03-31-package-management-provider-design.md b/docs/plans/2026-03-31-package-management-provider-design.md deleted file mode 100644 index 93c09b9cc..000000000 --- a/docs/plans/2026-03-31-package-management-provider-design.md +++ /dev/null @@ -1,180 +0,0 @@ -# Package Management Provider Design - -## Overview - -Add package management to OSAPI. List installed packages, get details, install, -remove, refresh package sources, and list available updates. Uses `apt-get` and -`dpkg-query` via `exec.Manager`. Provider package is `apt` (the tool), API path -is `/node/{hostname}/package` (the concept). - -## Architecture - -Direct provider at `internal/provider/node/apt/`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/package` -- **Permissions**: `package:read`, `package:write` -- **Provider type**: direct (exec.Manager) - -## Provider Interface - -```go -type Provider interface { - List(ctx context.Context) ([]Package, error) - Get(ctx context.Context, name string) (*Package, error) - Install(ctx context.Context, name string) (*Result, error) - Remove(ctx context.Context, name string) (*Result, error) - Update(ctx context.Context) (*Result, error) - ListUpdates(ctx context.Context) ([]Update, error) -} -``` - -## Data Types - -```go -type Package struct { - Name string `json:"name"` - Version string `json:"version"` - Description string `json:"description,omitempty"` - Status string `json:"status"` - Size int64 `json:"size,omitempty"` -} - -type Update struct { - Name string `json:"name"` - CurrentVersion string `json:"current_version"` - NewVersion string `json:"new_version"` -} - -type Result struct { - Name string `json:"name,omitempty"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -## Debian Implementation - -- **List**: `dpkg-query -W -f` with format string to get name, version, - description, status, installed size. Parse tab-separated output. Filter for - `install ok installed` status. -- **Get**: same query filtered to one package. Error if not found. -- **Install**: `apt-get install -y `. Return Changed: true. -- **Remove**: `apt-get remove -y `. Return Changed: true. -- **Update**: `apt-get update`. Refreshes package index. Return Changed: true. -- **ListUpdates**: `apt list --upgradable`. Parse output for package name, - current version, and new version. - -## Platform Implementations - -| Platform | Implementation | -| -------- | -------------------- | -| Debian | apt-get / dpkg-query | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -Return `ErrUnsupported` in containers. - -## API Endpoints - -| Method | Path | Permission | Description | -| -------- | --------------------------------- | --------------- | ----------------------- | -| `GET` | `/node/{hostname}/package` | `package:read` | List installed packages | -| `GET` | `/node/{hostname}/package/{name}` | `package:read` | Get package details | -| `POST` | `/node/{hostname}/package` | `package:write` | Install a package | -| `DELETE` | `/node/{hostname}/package/{name}` | `package:write` | Remove a package | -| `POST` | `/node/{hostname}/package/update` | `package:write` | Refresh package sources | -| `GET` | `/node/{hostname}/package/update` | `package:read` | List available updates | - -All endpoints support broadcast targeting. - -### POST Install Request Body - -```json -{ - "name": "nginx" -} -``` - -Name is required. - -### Response Shapes - -List response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "packages": [ - { - "name": "nginx", - "version": "1.24.0-1", - "status": "installed", - "size": 1234567 - } - ] - } - ] -} -``` - -Install/Remove/Update response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "name": "nginx", - "changed": true - } - ] -} -``` - -List updates response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "updates": [ - { - "name": "openssl", - "current_version": "3.0.11-1", - "new_version": "3.0.13-1" - } - ] - } - ] -} -``` - -## SDK - -```go -client.Package.List(ctx, host) -client.Package.Get(ctx, host, name) -client.Package.Install(ctx, host, name) -client.Package.Remove(ctx, host, name) -client.Package.Update(ctx, host) -client.Package.ListUpdates(ctx, host) -``` - -## Permissions - -- `package:read` — list, get, list updates. Added to admin, write, and read - roles. -- `package:write` — install, remove, update sources. Added to admin and write - roles. diff --git a/docs/plans/2026-03-31-package-management-provider.md b/docs/plans/2026-03-31-package-management-provider.md deleted file mode 100644 index dd96d9ca5..000000000 --- a/docs/plans/2026-03-31-package-management-provider.md +++ /dev/null @@ -1,473 +0,0 @@ -# Package Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add package management (list, get, install, remove, update sources, -list updates) via apt as a node provider with full API/CLI/SDK support. - -**Architecture:** Direct provider at `provider/node/apt/` using `apt-get` and -`dpkg-query` via `exec.Manager`. API path is `/node/{hostname}/package`. One SDK -service (`PackageService`). Permissions: `package:read` (all roles), -`package:write` (admin + write). - -**Tech Stack:** Go 1.25, Echo, oapi-codegen (strict-server), gomock, -testify/suite - -**Coverage baseline:** 99.9% — must remain at or above this. - ---- - -## Task 1: SDK Constants (Operations + Permissions) - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/job/types.go` -- Modify: `internal/authtoken/permissions.go` - -- [ ] **Step 1: Add package operation constants** - -In `pkg/sdk/client/operations.go`: - -```go -// Package operations. -const ( - OpPackageList JobOperation = "node.package.list" - OpPackageGet JobOperation = "node.package.get" - OpPackageInstall JobOperation = "node.package.install" - OpPackageRemove JobOperation = "node.package.remove" - OpPackageUpdate JobOperation = "node.package.update" - OpPackageListUpdates JobOperation = "node.package.listUpdates" -) -``` - -- [ ] **Step 2: Add permission constants** - -```go - PermPackageRead Permission = "package:read" - PermPackageWrite Permission = "package:write" -``` - -- [ ] **Step 3: Re-export in internal/job/types.go** - -- [ ] **Step 4: Re-export permissions in internal/authtoken/permissions.go** - -Add to `DefaultRolePermissions`: - -- `RoleAdmin`: both -- `RoleWrite`: both -- `RoleRead`: `PermPackageRead` only - -- [ ] **Step 5: Verify and commit** - -```bash -go build ./... -git commit -m "feat(package): add operation and permission constants" -``` - ---- - -## Task 2: Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/apt/types.go` -- Create: `internal/provider/node/apt/darwin.go` -- Create: `internal/provider/node/apt/linux.go` -- Create: `internal/provider/node/apt/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -Package name is `apt`. Provider interface with 6 methods: List, Get, Install, -Remove, Update, ListUpdates. - -Data types: Package, Update, Result (see design spec). - -- [ ] **Step 2: Create darwin.go and linux.go stubs** - -All methods return `fmt.Errorf("package: %w", provider.ErrUnsupported)`. - -- [ ] **Step 3: Create mocks and generate** - -- [ ] **Step 4: Verify and commit** - -```bash -go generate ./internal/provider/node/apt/mocks/... -go build ./... -git commit -m "feat(package): add provider interface and platform stubs" -``` - ---- - -## Task 3: Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/apt/debian.go` -- Create: `internal/provider/node/apt/debian_public_test.go` -- Create: `internal/provider/node/apt/darwin_public_test.go` -- Create: `internal/provider/node/apt/linux_public_test.go` - -The Debian provider uses `exec.Manager` for all operations. - -```go -type Debian struct { - provider.FactsAware - logger *slog.Logger - execManager exec.Manager -} - -func NewDebianProvider( - logger *slog.Logger, - execManager exec.Manager, -) *Debian -``` - -### Operations - -- **List**: run - `dpkg-query -W -f '${Package}\t${Version}\t${binary:Summary}\t${db:Status-Abbrev}\t${Installed-Size}\n'`. - Parse tab-separated output. Filter lines where status starts with `ii` - (installed). -- **Get**: same command with package name argument. Error if not installed. -- **Install**: run `apt-get install -y `. Return - `Result{Name: name, Changed: true}`. -- **Remove**: run `apt-get remove -y `. Return - `Result{Name: name, Changed: true}`. -- **Update**: run `apt-get update`. Return `Result{Changed: true}`. -- **ListUpdates**: run `apt list --upgradable 2>/dev/null`. Parse output lines - like `package/source version [upgradable from: oldversion]`. - -### Tests - -Mock `exec.Manager` with gomock. - -**TestList:** success (parse dpkg output), exec error, empty output **TestGet:** -success, not found, exec error **TestInstall:** success, exec error (package not -found) **TestRemove:** success, exec error **TestUpdate:** success, exec error -**TestListUpdates:** success (parse apt list output), no updates, exec error - -Stub tests: Darwin and Linux return ErrUnsupported. - -Target: 100% coverage. - -- [ ] **Step 1: Write stub tests** -- [ ] **Step 2: Write Debian tests** -- [ ] **Step 3: Implement debian.go** -- [ ] **Step 4: Verify 100% coverage and commit** - -```bash -go test -coverprofile=/tmp/c.out ./internal/provider/node/apt/... -git commit -m "feat(package): implement Debian apt provider with tests" -``` - ---- - -## Task 4: Agent Processor + Wiring - -**Files:** - -- Create: `internal/agent/processor_package.go` -- Create: `internal/agent/processor_package_public_test.go` -- Modify: `internal/agent/processor.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Create processor with tests** - -Six sub-operations: package.list, package.get, package.install, package.remove, -package.update, package.listUpdates. - -Get/install/remove unmarshal `{"name": "..."}` from Data. -List/update/listUpdates need no data. - -- [ ] **Step 2: Add to node processor** - -Add `packageProvider apt.Provider` param to `NewNodeProcessor`. Add -`case "package":` dispatch. - -- [ ] **Step 3: Wire in agent_setup.go** - -```go -func createPackageProvider( - log *slog.Logger, - execManager exec.Manager, -) aptProv.Provider -``` - -Container check: return ErrUnsupported in containers. - -- [ ] **Step 4: Fix existing tests and verify** - -```bash -go test ./internal/agent/... ./cmd/... -git commit -m "feat(package): add agent processor and wiring" -``` - ---- - -## Task 5: OpenAPI Spec + API Handlers - -**Files:** - -- Create: `internal/controller/api/node/package/gen/api.yaml` -- Create: `internal/controller/api/node/package/gen/cfg.yaml` -- Create: `internal/controller/api/node/package/gen/generate.go` -- Create: `internal/controller/api/node/package/types.go` -- Create: `internal/controller/api/node/package/package.go` -- Create: `internal/controller/api/node/package/validate.go` -- Create: `internal/controller/api/node/package/package_list_get.go` -- Create: `internal/controller/api/node/package/package_get.go` -- Create: `internal/controller/api/node/package/package_install.go` -- Create: `internal/controller/api/node/package/package_remove.go` -- Create: `internal/controller/api/node/package/package_update_post.go` -- Create: `internal/controller/api/node/package/package_update_get.go` -- Create: `internal/controller/api/node/package/handler.go` -- Create: test files for each handler -- Modify: `cmd/controller_setup.go` - -Note: the API handler package is `package` which is a Go reserved word. Use -`pkg` as the Go package name: `package pkg` at the top of each file. Import -alias in controller_setup.go: `packageAPI "...api/node/package"`. - -Actually — Go does NOT allow `package` as a directory name for a package -declaration. The directory can be named `package` but the Go package must be -something else. Use `package pkgmgmt` or just put it under a different directory -name. - -Simplest: name the directory `pkg` to match the Go package name. So: -`internal/controller/api/node/pkg/`. But `pkg` is confusing. - -Better: `internal/controller/api/node/package/` with `package packagemgmt` at -the top. Import as `packageAPI "...api/node/package"`. - -Wait — Go actually CAN have a directory named `package` with a different package -declaration. The directory name doesn't need to match the Go package name. Use -directory `package/` with `package packageapi` or similar. - -The implementer should read how this is handled and choose the cleanest -approach. Follow the pattern: directory name matches the URL path segment -(`package`), Go package name avoids the reserved word. - -- [ ] **Step 1: Create OpenAPI spec** - -Six paths under `/node/{hostname}/package`: - -- `GET /` — list installed, security: `package:read` -- `POST /` — install, security: `package:write` -- `GET /{name}` — get details, security: `package:read` -- `DELETE /{name}` — remove, security: `package:write` -- `POST /update` — refresh sources, security: `package:write` -- `GET /update` — list updates, security: `package:read` - -Request schemas: - -- `PackageInstallRequest` — name (required, validate required) - -Response schemas: - -- `PackageEntry` — hostname, status, packages (array of PackageInfo) -- `PackageInfo` — name, version, description, status, size -- `PackageMutationResult` — hostname, status, name, changed, error -- `UpdateEntry` — hostname, status, updates (array of UpdateInfo) -- `UpdateInfo` — name, current_version, new_version - -- [ ] **Step 2: Generate code and implement handlers** - -List/Get use `JobClient.Query`. Install/Remove/Update use `JobClient.Modify`. -ListUpdates uses `JobClient.Query`. - -Category: `"node"`. - -- [ ] **Step 3: Create handler.go and write tests with RBAC** - -- [ ] **Step 4: Wire in controller_setup.go and verify** - -```bash -go build ./... -go test ./internal/controller/api/node/package/... ./cmd/... -git commit -m "feat(package): add OpenAPI spec, API handlers, and server wiring" -``` - ---- - -## Task 6: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/package.go` -- Create: `pkg/sdk/client/package_types.go` -- Create: `pkg/sdk/client/package_public_test.go` -- Create: `pkg/sdk/client/package_types_public_test.go` -- Modify: `pkg/sdk/client/osapi.go` - -Note: `package.go` is fine as a filename — Go filenames don't conflict with -reserved words, only package declarations do. - -- [ ] **Step 1: Create types** - -```go -type PackageInfoResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Packages []PackageInfo `json:"packages,omitempty"` - Error string `json:"error,omitempty"` -} - -type PackageInfo struct { - Name string `json:"name,omitempty"` - Version string `json:"version,omitempty"` - Description string `json:"description,omitempty"` - Status string `json:"status,omitempty"` - Size int64 `json:"size,omitempty"` -} - -type PackageMutationResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Name string `json:"name,omitempty"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type PackageUpdateResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Updates []UpdateInfo `json:"updates,omitempty"` - Error string `json:"error,omitempty"` -} - -type UpdateInfo struct { - Name string `json:"name,omitempty"` - CurrentVersion string `json:"current_version,omitempty"` - NewVersion string `json:"new_version,omitempty"` -} -``` - -- [ ] **Step 2: Create service** - -```go -type PackageService struct { - client *gen.ClientWithResponses -} -``` - -Methods (clean verbs): - -- `List(ctx, hostname)` -- `Get(ctx, hostname, name)` -- `Install(ctx, hostname, name)` -- `Remove(ctx, hostname, name)` -- `Update(ctx, hostname)` -- `ListUpdates(ctx, hostname)` - -Wire as `Package *PackageService` in Client. - -Run `just generate` first. - -- [ ] **Step 3: Write tests and verify** - -```bash -go test ./pkg/sdk/client/... -git commit -m "feat(package): add SDK service with tests" -``` - ---- - -## Task 7: CLI Commands - -**Files:** - -- Create: `cmd/client_node_package.go` — parent -- Create: `cmd/client_node_package_list.go` -- Create: `cmd/client_node_package_get.go` -- Create: `cmd/client_node_package_install.go` -- Create: `cmd/client_node_package_remove.go` -- Create: `cmd/client_node_package_update.go` -- Create: `cmd/client_node_package_updates.go` - -- [ ] **Step 1: Create parent + list/get commands** - -List: table fields NAME, VERSION, STATUS, SIZE. Format SIZE with -`cli.FormatBytes`. - -Get: flag `--name` (required). Same table. - -- [ ] **Step 2: Create install/remove commands** - -Install: flag `--name` (required). Mutation table. Remove: flag `--name` -(required). Mutation table. - -- [ ] **Step 3: Create update + list-updates commands** - -Update (refresh sources): no extra flags. Mutation table. List updates: table -fields NAME, CURRENT, NEW. - -The CLI command for listing updates could be `updates` as a subcommand: -`osapi client node package updates`. - -- [ ] **Step 4: Verify and commit** - -```bash -go build ./... -git commit -m "feat(package): add CLI commands" -``` - ---- - -## Task 8: Docs + Example + Integration Test - -**Files:** - -- Create: `examples/sdk/client/package.go` -- Create: `test/integration/package_test.go` -- Create: `docs/docs/sidebar/features/package-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/package.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/install.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/remove.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/update.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/package/updates.md` -- Create: `docs/docs/sidebar/sdk/client/system-config/package.md` -- Modify: shared docs (features table, auth, config, api guidelines, - architecture, client.md, docusaurus.config.ts) - -SDK doc goes under `system-config/` category — package management is system -configuration alongside sysctl, ntp, timezone. - -- [ ] **Step 1: Create SDK example** - -Demonstrate List and Get. Don't demonstrate Install in example (destructive). - -- [ ] **Step 2: Create integration test** - -`PackageSmokeSuite` with `TestPackageList` (read-only). Guard install/remove -with `skipWrite`. - -- [ ] **Step 3: Create feature page + CLI docs + SDK doc** - -- [ ] **Step 4: Update all shared docs** - -Features table, auth permissions, config roles, API guidelines (6 endpoints), -architecture feature link, client.md system-config table, docusaurus dropdowns -(Features + SDK). - -- [ ] **Step 5: Regenerate and verify** - -```bash -just generate -go build ./... -just go::unit -just go::unit-cov # >= 99.9% -just go::vet -``` - -- [ ] **Step 6: Commit** - -```bash -git commit -m "feat(package): add docs, SDK example, and integration tests" -``` diff --git a/docs/plans/2026-03-31-power-management-provider-design.md b/docs/plans/2026-03-31-power-management-provider-design.md deleted file mode 100644 index 342800ebf..000000000 --- a/docs/plans/2026-03-31-power-management-provider-design.md +++ /dev/null @@ -1,131 +0,0 @@ -# Power Management Provider Design - -## Overview - -Add power management (reboot/shutdown) to OSAPI. Two action operations on a -direct provider — no persistent resources, no file management. The provider runs -the `shutdown` command with a minimum 5-second implicit delay so the agent can -complete its job response lifecycle before the system goes down. - -## Architecture - -Direct provider at `internal/provider/node/power/`. Action operations only (no -CRUD). - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/power` -- **Permissions**: `power:execute` -- **Provider type**: direct - -## Provider Interface - -```go -type Provider interface { - Reboot(ctx context.Context, opts Opts) (*Result, error) - Shutdown(ctx context.Context, opts Opts) (*Result, error) -} -``` - -## Data Types - -```go -type Opts struct { - Delay int `json:"delay,omitempty"` - Message string `json:"message,omitempty"` -} - -type Result struct { - Action string `json:"action"` - Delay int `json:"delay"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- `Delay` — seconds before the action. Minimum 5 seconds enforced by the - provider regardless of what the user requests. This gives the agent time to - write the job result, send the response, and run graceful shutdown. -- `Message` — optional human-readable reason. Logged by the agent before - executing. - -## Debian Implementation - -- **Reboot**: run `shutdown -r` with the computed delay -- **Shutdown**: run `shutdown -h` with the computed delay -- Actual delay = `max(userDelay, 5)` seconds -- Provider returns immediately with `changed: true` and the actual delay applied -- The agent completes its KV write and response lifecycle before the system goes - down -- Use `exec.Manager` for running the shutdown command - -## Platform Implementations - -| Platform | Implementation | -| -------- | -------------------- | -| Debian | `shutdown -r` / `-h` | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -No container variant needed — power management doesn't make sense inside a -container. - -## API Endpoints - -| Method | Path | Permission | Description | -| ------ | --------------------------------- | --------------- | ------------- | -| `POST` | `/node/{hostname}/power/reboot` | `power:execute` | Reboot node | -| `POST` | `/node/{hostname}/power/shutdown` | `power:execute` | Shutdown node | - -All endpoints support broadcast targeting. - -### Request Body (optional) - -```json -{ - "delay": 60, - "message": "Scheduled maintenance" -} -``` - -Both fields optional. If omitted, immediate action (with the 5-second minimum -implicit delay). - -### Validation - -- `delay`: integer, min 0, optional -- `message`: string, optional - -### Response Shape - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "action": "reboot", - "delay": 60, - "changed": true - } - ] -} -``` - -## SDK - -```go -client.Power.Reboot(ctx, host, opts) -client.Power.Shutdown(ctx, host, opts) -``` - -`PowerOpts` struct with optional `Delay` and `Message` fields. Both methods -return `*Response[Collection[PowerResult]]`. - -## Permission - -`power:execute` — action permission, same pattern as `command:execute`. No read -permission exists for this domain. - -Added to admin role only (not write or read) — power operations are destructive -and should require explicit authorization. diff --git a/docs/plans/2026-03-31-power-management-provider.md b/docs/plans/2026-03-31-power-management-provider.md deleted file mode 100644 index 52cd3281a..000000000 --- a/docs/plans/2026-03-31-power-management-provider.md +++ /dev/null @@ -1,428 +0,0 @@ -# Power Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add power management (reboot/shutdown) as action operations with a -minimum 5-second implicit delay for agent lifecycle completion. - -**Architecture:** Direct provider at `provider/node/power/` with two action -methods (Reboot, Shutdown). Integrates into the node processor. API has two POST -endpoints under `/node/{hostname}/power/`. SDK exposes `client.Power.Reboot()` -and `client.Power.Shutdown()`. Permission is `power:execute` (admin only). - -**Tech Stack:** Go 1.25, Echo, oapi-codegen (strict-server), gomock, -testify/suite - -**Coverage baseline:** 99.9% — must remain at or above this. - ---- - -## Task 1: SDK Constants (Operations + Permissions) - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/job/types.go` -- Modify: `internal/authtoken/permissions.go` - -- [ ] **Step 1: Add power operation constants** - -In `pkg/sdk/client/operations.go`: - -```go -// Power operations. -const ( - OpPowerReboot JobOperation = "node.power.reboot" - OpPowerShutdown JobOperation = "node.power.shutdown" -) -``` - -- [ ] **Step 2: Add permission constant** - -In `pkg/sdk/client/permissions.go`: - -```go - PermPowerExecute Permission = "power:execute" -``` - -- [ ] **Step 3: Re-export in internal/job/types.go** - -```go -// Power operations. -const ( - OperationPowerReboot = client.OpPowerReboot - OperationPowerShutdown = client.OpPowerShutdown -) -``` - -- [ ] **Step 4: Re-export permission in internal/authtoken/permissions.go** - -Add constant, add to `AllPermissions`, add to `DefaultRolePermissions` for -`RoleAdmin` ONLY (not write or read — power is destructive). - -- [ ] **Step 5: Verify and commit** - -```bash -go build ./... -git commit -m "feat(power): add operation and permission constants" -``` - ---- - -## Task 2: Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/power/types.go` -- Create: `internal/provider/node/power/darwin.go` -- Create: `internal/provider/node/power/linux.go` -- Create: `internal/provider/node/power/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -```go -package power - -import "context" - -// Provider implements power management operations. -type Provider interface { - Reboot(ctx context.Context, opts Opts) (*Result, error) - Shutdown(ctx context.Context, opts Opts) (*Result, error) -} - -// Opts contains optional parameters for power operations. -type Opts struct { - Delay int `json:"delay,omitempty"` - Message string `json:"message,omitempty"` -} - -// Result represents the outcome of a power operation. -type Result struct { - Action string `json:"action"` - Delay int `json:"delay"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Create darwin.go and linux.go stubs** - -Both return `fmt.Errorf("power: %w", provider.ErrUnsupported)`. - -- [ ] **Step 3: Create mocks and generate** - -- [ ] **Step 4: Verify and commit** - -```bash -go generate ./internal/provider/node/power/mocks/... -go build ./... -git commit -m "feat(power): add provider interface and platform stubs" -``` - ---- - -## Task 3: Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/power/debian.go` -- Create: `internal/provider/node/power/debian_public_test.go` -- Create: `internal/provider/node/power/darwin_public_test.go` -- Create: `internal/provider/node/power/linux_public_test.go` - -The Debian provider: - -- Enforces minimum 5-second delay: `actualDelay := max(userDelay, 5)` -- Runs shutdown command in background so the provider returns before the system - goes down -- Reboot: `shutdown -r +N` or `sleep N && shutdown -r now &` -- Shutdown: `shutdown -h +N` or `sleep N && shutdown -h now &` -- Logs the message before executing if provided - -- [ ] **Step 1: Write stub tests (Darwin, Linux)** - -Verify all methods return `ErrUnsupported`. - -- [ ] **Step 2: Write Debian tests** - -Test cases for Reboot and Shutdown: - -- success with default delay (5 seconds) -- success with user delay > 5 -- success with user delay < 5 (clamped to 5) -- success with message -- exec error -- verify Changed is always true on success - -- [ ] **Step 3: Implement debian.go** - -```go -type Debian struct { - provider.FactsAware - logger *slog.Logger - execManager exec.Manager -} - -func NewDebianProvider( - logger *slog.Logger, - execManager exec.Manager, -) *Debian -``` - -- [ ] **Step 4: Verify 100% coverage and commit** - -```bash -go test -coverprofile=/tmp/c.out ./internal/provider/node/power/... -git commit -m "feat(power): implement Debian power provider with tests" -``` - ---- - -## Task 4: Agent Processor + Wiring - -**Files:** - -- Create: `internal/agent/processor_power.go` -- Create: `internal/agent/processor_power_public_test.go` -- Modify: `internal/agent/processor.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Create processor with tests** - -Two sub-operations: `power.reboot` and `power.shutdown`. Both unmarshal -`power.Opts` from `jobRequest.Data` (data may be nil for default opts). - -- [ ] **Step 2: Add to node processor** - -Add `powerProvider power.Provider` parameter to `NewNodeProcessor`. Add -`case "power":` dispatch. - -- [ ] **Step 3: Wire in agent_setup.go** - -Create `createPowerProvider` function. Add to `NewNodeProcessor` call and -`registry.Register` providers list. - -- [ ] **Step 4: Fix existing tests and verify** - -```bash -go test ./internal/agent/... ./cmd/... -git commit -m "feat(power): add agent processor and wiring" -``` - ---- - -## Task 5: OpenAPI Spec + API Handlers - -**Files:** - -- Create: `internal/controller/api/node/power/gen/api.yaml` -- Create: `internal/controller/api/node/power/gen/cfg.yaml` -- Create: `internal/controller/api/node/power/gen/generate.go` -- Create: `internal/controller/api/node/power/types.go` -- Create: `internal/controller/api/node/power/power.go` -- Create: `internal/controller/api/node/power/validate.go` -- Create: `internal/controller/api/node/power/reboot_post.go` -- Create: `internal/controller/api/node/power/shutdown_post.go` -- Create: `internal/controller/api/node/power/handler.go` -- Create: test files for each handler -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Create OpenAPI spec** - -Two POST paths: - -- `POST /node/{hostname}/power/reboot` — `PostNodePowerReboot`, security: - `power:execute` -- `POST /node/{hostname}/power/shutdown` — `PostNodePowerShutdown`, security: - `power:execute` - -Request body (shared, optional): - -- `PowerRequest` — delay (integer, min 0), message (string) - -Response schemas: - -- `PowerResult` — hostname (req), status (req, enum ok/failed/skipped), action, - delay, changed, error -- `PowerRebootResponse` / `PowerShutdownResponse` — job_id + results array - -- [ ] **Step 2: Generate code and create handler struct** - -- [ ] **Step 3: Implement both handlers with broadcast support** - -Category `"node"`, operations `job.OperationPowerReboot` and -`job.OperationPowerShutdown`. Use `JobClient.Modify` (these are state-changing -actions). - -- [ ] **Step 4: Create handler.go (self-registration)** - -- [ ] **Step 5: Write tests with RBAC for both handlers** - -- [ ] **Step 6: Wire in controller_setup.go and verify** - -```bash -go build ./... -go test ./internal/controller/api/node/power/... ./cmd/... -git commit -m "feat(power): add OpenAPI spec, API handlers, and server wiring" -``` - ---- - -## Task 6: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/power.go` -- Create: `pkg/sdk/client/power_types.go` -- Create: `pkg/sdk/client/power_public_test.go` -- Create: `pkg/sdk/client/power_types_public_test.go` -- Modify: `pkg/sdk/client/osapi.go` - -- [ ] **Step 1: Create types** - -```go -type PowerResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Action string `json:"action,omitempty"` - Delay int `json:"delay,omitempty"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type PowerOpts struct { - Delay int - Message string -} -``` - -- [ ] **Step 2: Create service** - -```go -type PowerService struct { - client *gen.ClientWithResponses -} -``` - -Methods: `Reboot(ctx, hostname, opts)` and `Shutdown(ctx, hostname, opts)`. Both -return `*Response[Collection[PowerResult]]`. - -- [ ] **Step 3: Wire into osapi.go, write tests** - -Run `just generate` first to get the combined spec updated. - -- [ ] **Step 4: Verify and commit** - -```bash -go test ./pkg/sdk/client/... -git commit -m "feat(power): add SDK service with tests" -``` - ---- - -## Task 7: CLI Commands - -**Files:** - -- Create: `cmd/client_node_power.go` -- Create: `cmd/client_node_power_reboot.go` -- Create: `cmd/client_node_power_shutdown.go` - -- [ ] **Step 1: Create parent command** - -```go -var clientNodePowerCmd = &cobra.Command{ - Use: "power", - Short: "Manage power state", -} - -func init() { - clientNodeCmd.AddCommand(clientNodePowerCmd) -} -``` - -- [ ] **Step 2: Create reboot command** - -Flags: `--delay` (int, optional), `--message` (string, optional). Call -`sdkClient.Power.Reboot(ctx, host, opts)`. Mutation table output with ACTION and -DELAY columns. - -- [ ] **Step 3: Create shutdown command** - -Same flags. Call `sdkClient.Power.Shutdown(ctx, host, opts)`. - -- [ ] **Step 4: Verify and commit** - -```bash -go build ./... -git commit -m "feat(power): add CLI commands" -``` - ---- - -## Task 8: Docs + Example + Integration Test - -**Files:** - -- Create: `examples/sdk/client/power.go` -- Create: `test/integration/power_test.go` -- Create: `docs/docs/sidebar/features/power-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/power/power.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/power/reboot.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/power/shutdown.md` -- Create: `docs/docs/sidebar/sdk/client/power.md` -- Modify: `docs/docs/sidebar/features/features.md` -- Modify: `docs/docs/sidebar/features/authentication.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `docs/docusaurus.config.ts` (Features + SDK dropdowns) - -- [ ] **Step 1: Create SDK example** - -Demonstrate `client.Power.Reboot()` with delay and message. - -- [ ] **Step 2: Create integration test** - -`PowerSmokeSuite` — read-only test only (don't actually reboot in CI). Test that -the endpoint responds (even if skipped on macOS). - -- [ ] **Step 3: Create feature page** - -`power-management.md` — what it does, how the delay works, CLI examples, -permissions (admin only), platforms. - -- [ ] **Step 4: Create CLI doc pages** - -Directory with landing page + reboot.md + shutdown.md. - -- [ ] **Step 5: Create SDK doc page** - -Title: `# Power`. Methods: `Reboot`, `Shutdown`. - -- [ ] **Step 6: Update all shared docs** - -Features table, authentication permissions, configuration roles table, API -guidelines endpoints, architecture feature link, docusaurus dropdowns -(Features + SDK). - -- [ ] **Step 7: Regenerate and verify** - -```bash -just generate -go build ./... -just go::unit -just go::unit-cov # >= 99.9% -just go::vet -``` - -- [ ] **Step 8: Commit** - -```bash -git commit -m "feat(power): add docs, SDK example, and integration tests" -``` diff --git a/docs/plans/2026-03-31-process-management-provider-design.md b/docs/plans/2026-03-31-process-management-provider-design.md deleted file mode 100644 index 24511979e..000000000 --- a/docs/plans/2026-03-31-process-management-provider-design.md +++ /dev/null @@ -1,158 +0,0 @@ -# Process Management Provider Design - -## Overview - -Add process management to OSAPI. List running processes, get details by PID, and -send signals (TERM, KILL, HUP, etc.). Direct provider using gopsutil for process -info and syscall.Kill for signaling. - -## Architecture - -Direct provider at `internal/provider/node/process/`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/process` -- **Permissions**: `process:read`, `process:execute` -- **Provider type**: direct (gopsutil + syscall) - -## Provider Interface - -```go -type Provider interface { - List(ctx context.Context) ([]Info, error) - Get(ctx context.Context, pid int) (*Info, error) - Signal(ctx context.Context, pid int, signal string) (*SignalResult, error) -} -``` - -## Data Types - -```go -type Info struct { - PID int `json:"pid"` - Name string `json:"name"` - User string `json:"user"` - State string `json:"state"` - CPUPercent float64 `json:"cpu_percent"` - MemPercent float32 `json:"mem_percent"` - MemRSS int64 `json:"mem_rss"` - Command string `json:"command"` - StartTime string `json:"start_time"` -} - -type SignalResult struct { - PID int `json:"pid"` - Signal string `json:"signal"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -## Debian Implementation - -- **List**: `gopsutil/process.Processes()` to get all PIDs, collect info per - process (name, user, state, CPU%, mem%, RSS, command, start time). Return - `[]Info`. -- **Get**: `gopsutil/process.NewProcess(pid)` and read info. Return error if PID - doesn't exist. -- **Signal**: validate signal name against allowed set (TERM, KILL, HUP, INT, - USR1, USR2). Convert to `syscall.Signal`. Call `syscall.Kill(pid, sig)`. - Return `SignalResult{Changed: true}`. - -## Platform Implementations - -| Platform | Implementation | -| -------- | ----------------------- | -| Debian | gopsutil + syscall.Kill | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -Add `platform.IsContainer()` check in `agent_setup.go`. Process management in -containers returns ErrUnsupported — process management is the host's concern. - -## API Endpoints - -| Method | Path | Permission | Description | -| ------ | --------------------------------------- | ----------------- | ------------------ | -| `GET` | `/node/{hostname}/process` | `process:read` | List all processes | -| `GET` | `/node/{hostname}/process/{pid}` | `process:read` | Get process by PID | -| `POST` | `/node/{hostname}/process/{pid}/signal` | `process:execute` | Send signal to PID | - -All endpoints support broadcast targeting. - -### POST Request Body - -```json -{ - "signal": "TERM" -} -``` - -Signal is required. Valid values: TERM, KILL, HUP, INT, USR1, USR2. Validated -via `x-oapi-codegen-extra-tags` with `oneof`. - -### Response Shapes - -List response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "processes": [ - { - "pid": 1, - "name": "systemd", - "user": "root", - "state": "S", - "cpu_percent": 0.1, - "mem_percent": 0.5, - "mem_rss": 12345678, - "command": "/sbin/init", - "start_time": "2026-03-30T10:00:00Z" - } - ] - } - ] -} -``` - -Get response — same shape but `processes` has one entry. - -Signal response: - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "pid": 1234, - "signal": "TERM", - "changed": true - } - ] -} -``` - -## SDK - -```go -client.Process.List(ctx, host) -client.Process.Get(ctx, host, pid) -client.Process.Signal(ctx, host, pid, opts) -``` - -`ProcessSignalOpts` has one required field: `Signal string`. - -## Permissions - -- `process:read` — list and get. Added to admin, write, and read roles. -- `process:execute` — send signals. Added to admin role only (destructive - operation). diff --git a/docs/plans/2026-03-31-process-management-provider.md b/docs/plans/2026-03-31-process-management-provider.md deleted file mode 100644 index c3c9494db..000000000 --- a/docs/plans/2026-03-31-process-management-provider.md +++ /dev/null @@ -1,522 +0,0 @@ -# Process Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add process listing, details, and signal sending as a node provider -with full API/CLI/SDK support. - -**Architecture:** Direct provider at `provider/node/process/` using -`gopsutil/process` for reading process info and `syscall.Kill` for signaling. -Three endpoints: list all, get by PID, signal by PID. Integrates into the node -processor. Permission: `process:read` (all roles) and `process:execute` (admin -only). - -**Tech Stack:** Go 1.25, gopsutil/v4, Echo, oapi-codegen (strict-server), -gomock, testify/suite - -**Coverage baseline:** 99.9% — must remain at or above this. - ---- - -## Task 1: SDK Constants (Operations + Permissions) - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/job/types.go` -- Modify: `internal/authtoken/permissions.go` - -- [ ] **Step 1: Add process operation constants** - -In `pkg/sdk/client/operations.go`: - -```go -// Process operations. -const ( - OpProcessList JobOperation = "node.process.list" - OpProcessGet JobOperation = "node.process.get" - OpProcessSignal JobOperation = "node.process.signal" -) -``` - -- [ ] **Step 2: Add permission constants** - -In `pkg/sdk/client/permissions.go`: - -```go - PermProcessRead Permission = "process:read" - PermProcessExecute Permission = "process:execute" -``` - -- [ ] **Step 3: Re-export in internal/job/types.go** - -```go -// Process operations. -const ( - OperationProcessList = client.OpProcessList - OperationProcessGet = client.OpProcessGet - OperationProcessSignal = client.OpProcessSignal -) -``` - -- [ ] **Step 4: Re-export permissions in internal/authtoken/permissions.go** - -Add constants. Add to `AllPermissions`. Add to `DefaultRolePermissions`: - -- `RoleAdmin`: add `PermProcessRead` and `PermProcessExecute` -- `RoleWrite`: add `PermProcessRead` only -- `RoleRead`: add `PermProcessRead` only - -- [ ] **Step 5: Verify and commit** - -```bash -go build ./... -git commit -m "feat(process): add operation and permission constants" -``` - ---- - -## Task 2: Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/process/types.go` -- Create: `internal/provider/node/process/darwin.go` -- Create: `internal/provider/node/process/linux.go` -- Create: `internal/provider/node/process/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -```go -package process - -import "context" - -// Provider implements process management operations. -type Provider interface { - List(ctx context.Context) ([]Info, error) - Get(ctx context.Context, pid int) (*Info, error) - Signal(ctx context.Context, pid int, signal string) (*SignalResult, error) -} - -type Info struct { - PID int `json:"pid"` - Name string `json:"name"` - User string `json:"user"` - State string `json:"state"` - CPUPercent float64 `json:"cpu_percent"` - MemPercent float32 `json:"mem_percent"` - MemRSS int64 `json:"mem_rss"` - Command string `json:"command"` - StartTime string `json:"start_time"` -} - -type SignalResult struct { - PID int `json:"pid"` - Signal string `json:"signal"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Create darwin.go and linux.go stubs** - -All methods return `fmt.Errorf("process: %w", provider.ErrUnsupported)`. - -- [ ] **Step 3: Create mocks and generate** - -- [ ] **Step 4: Verify and commit** - -```bash -go generate ./internal/provider/node/process/mocks/... -go build ./... -git commit -m "feat(process): add provider interface and platform stubs" -``` - ---- - -## Task 3: Debian Provider Implementation - -**Files:** - -- Create: `internal/provider/node/process/debian.go` -- Create: `internal/provider/node/process/debian_public_test.go` -- Create: `internal/provider/node/process/darwin_public_test.go` -- Create: `internal/provider/node/process/linux_public_test.go` - -The Debian provider uses `gopsutil/v4/process` for reading process info and -`syscall.Kill` for sending signals. - -- [ ] **Step 1: Write stub tests** - -Verify Darwin and Linux return `ErrUnsupported` for all methods. - -- [ ] **Step 2: Write Debian tests** - -The provider wraps gopsutil — for testability, the provider should accept an -interface that wraps gopsutil calls so it can be mocked. Alternatively, use the -export_test.go pattern to swap the gopsutil functions. - -**TestList:** - -- success (returns process list) -- gopsutil error - -**TestGet:** - -- success (PID exists) -- PID not found -- gopsutil error - -**TestSignal:** - -- success with TERM -- success with KILL -- invalid signal name -- PID not found (kill returns ESRCH) -- permission denied (kill returns EPERM) - -- [ ] **Step 3: Implement debian.go** - -```go -var ( - _ Provider = (*Debian)(nil) - _ provider.FactsSetter = (*Debian)(nil) -) - -type Debian struct { - provider.FactsAware - logger *slog.Logger -} - -func NewDebianProvider( - logger *slog.Logger, -) *Debian -``` - -Key implementation: - -- **List**: call `process.Processes()`, for each PID gather info with - `p.Name()`, `p.Username()`, `p.Status()`, `p.CPUPercent()`, - `p.MemoryPercent()`, `p.MemoryInfo()` (for RSS), `p.Cmdline()`, - `p.CreateTime()`. Skip processes that error (permission denied on some PIDs is - normal). -- **Get**: `process.NewProcess(int32(pid))`, gather same info. Return error if - PID doesn't exist. -- **Signal**: validate signal name against allowed map. Use - `syscall.Kill(pid, sig)`. Handle `ESRCH` (no such process) and `EPERM` - (permission denied) errors. - -Allowed signals map: - -```go -var allowedSignals = map[string]syscall.Signal{ - "TERM": syscall.SIGTERM, - "KILL": syscall.SIGKILL, - "HUP": syscall.SIGHUP, - "INT": syscall.SIGINT, - "USR1": syscall.SIGUSR1, - "USR2": syscall.SIGUSR2, -} -``` - -- [ ] **Step 4: Verify 100% coverage and commit** - -```bash -go test -coverprofile=/tmp/c.out ./internal/provider/node/process/... -git commit -m "feat(process): implement Debian process provider with tests" -``` - ---- - -## Task 4: Agent Processor + Wiring - -**Files:** - -- Create: `internal/agent/processor_process.go` -- Create: `internal/agent/processor_process_public_test.go` -- Modify: `internal/agent/processor.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Create processor with tests** - -Three sub-operations: - -- `process.list` — no data, call `provider.List(ctx)` -- `process.get` — unmarshal `{"pid": 1234}`, call `provider.Get(ctx, pid)` -- `process.signal` — unmarshal `{"pid": 1234, "signal": "TERM"}`, call - `provider.Signal(ctx, pid, signal)` - -- [ ] **Step 2: Add to node processor** - -Add `processProvider process.Provider` parameter to `NewNodeProcessor`. Add -`case "process":` dispatch. - -- [ ] **Step 3: Wire in agent_setup.go** - -Create `createProcessProvider` function. Add container check. Process provider -only needs `logger` (no exec manager, no fs). - -```go -func createProcessProvider( - log *slog.Logger, -) processProv.Provider { - plat := platform.Detect() - switch plat { - case "debian": - if platform.IsContainer() { - log.Info("running in container, process operations disabled") - return processProv.NewLinuxProvider() - } - return processProv.NewDebianProvider(log) - case "darwin": - return processProv.NewDarwinProvider() - default: - return processProv.NewLinuxProvider() - } -} -``` - -- [ ] **Step 4: Fix existing tests and verify** - -```bash -go test ./internal/agent/... ./cmd/... -git commit -m "feat(process): add agent processor and wiring" -``` - ---- - -## Task 5: OpenAPI Spec + API Handlers - -**Files:** - -- Create: `internal/controller/api/node/process/gen/api.yaml` -- Create: `internal/controller/api/node/process/gen/cfg.yaml` -- Create: `internal/controller/api/node/process/gen/generate.go` -- Create: `internal/controller/api/node/process/types.go` -- Create: `internal/controller/api/node/process/process.go` -- Create: `internal/controller/api/node/process/validate.go` -- Create: `internal/controller/api/node/process/process_list_get.go` -- Create: `internal/controller/api/node/process/process_get.go` -- Create: `internal/controller/api/node/process/process_signal_post.go` -- Create: `internal/controller/api/node/process/handler.go` -- Create: test files for each handler -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Create OpenAPI spec** - -Three paths: - -- `GET /node/{hostname}/process` — `GetNodeProcess`, security: `process:read`. - Response: `ProcessCollectionResponse` -- `GET /node/{hostname}/process/{pid}` — `GetNodeProcessByPid`, security: - `process:read`. PID is integer path param. Response: `ProcessGetResponse` -- `POST /node/{hostname}/process/{pid}/signal` — `PostNodeProcessSignal`, - security: `process:execute`. Request body: `ProcessSignalRequest` with signal - (required, enum of TERM/KILL/HUP/INT/USR1/USR2, validate - `required,oneof=...`). Response: `ProcessSignalResponse` - -Schemas: - -- `ProcessEntry` — hostname (req), status (req, enum ok/failed/skipped), - processes (array of ProcessInfo), error -- `ProcessInfo` — pid, name, user, state, cpu_percent, mem_percent, mem_rss, - command, start_time -- `ProcessSignalResult` — hostname (req), status (req), pid, signal, changed, - error - -- [ ] **Step 2: Generate code and create handlers** - -- [ ] **Step 3: Implement all handlers with broadcast support** - -Category `"node"`. Use `JobClient.Query` for list/get (reads), -`JobClient.Modify` for signal (state change). - -- [ ] **Step 4: Create handler.go (self-registration)** - -- [ ] **Step 5: Write tests with RBAC** - -- [ ] **Step 6: Wire in controller_setup.go and verify** - -```bash -go build ./... -go test ./internal/controller/api/node/process/... ./cmd/... -git commit -m "feat(process): add OpenAPI spec, API handlers, and server wiring" -``` - ---- - -## Task 6: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/process.go` -- Create: `pkg/sdk/client/process_types.go` -- Create: `pkg/sdk/client/process_public_test.go` -- Create: `pkg/sdk/client/process_types_public_test.go` -- Modify: `pkg/sdk/client/osapi.go` - -- [ ] **Step 1: Create types** - -```go -type ProcessInfoResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Processes []ProcessInfo `json:"processes,omitempty"` - Error string `json:"error,omitempty"` -} - -type ProcessInfo struct { - PID int `json:"pid"` - Name string `json:"name,omitempty"` - User string `json:"user,omitempty"` - State string `json:"state,omitempty"` - CPUPercent float64 `json:"cpu_percent,omitempty"` - MemPercent float32 `json:"mem_percent,omitempty"` - MemRSS int64 `json:"mem_rss,omitempty"` - Command string `json:"command,omitempty"` - StartTime string `json:"start_time,omitempty"` -} - -type ProcessSignalResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - PID int `json:"pid,omitempty"` - Signal string `json:"signal,omitempty"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type ProcessSignalOpts struct { - Signal string -} -``` - -- [ ] **Step 2: Create service** - -```go -type ProcessService struct { - client *gen.ClientWithResponses -} -``` - -Methods: `List(ctx, hostname)`, `Get(ctx, hostname, pid)`, -`Signal(ctx, hostname, pid, opts)`. - -- [ ] **Step 3: Wire into osapi.go, run `just generate`, write tests** - -- [ ] **Step 4: Verify and commit** - -```bash -go test ./pkg/sdk/client/... -git commit -m "feat(process): add SDK service with tests" -``` - ---- - -## Task 7: CLI Commands - -**Files:** - -- Create: `cmd/client_node_process.go` -- Create: `cmd/client_node_process_list.go` -- Create: `cmd/client_node_process_get.go` -- Create: `cmd/client_node_process_signal.go` - -- [ ] **Step 1: Create parent command** - -- [ ] **Step 2: Create list command** - -No extra flags. Table fields: PID, NAME, USER, STATE, CPU%, MEM%, COMMAND. Use -`cli.BuildBroadcastTable` + `cli.PrintCompactTable`. Format CPU% and MEM% with -`fmt.Sprintf("%.1f%%", val)`. - -- [ ] **Step 3: Create get command** - -Flag: `--pid` (int, required). Same table output as list. - -- [ ] **Step 4: Create signal command** - -Flags: `--pid` (int, required), `--signal` (string, required). Mutation table -output with PID and SIGNAL fields. - -- [ ] **Step 5: Verify and commit** - -```bash -go build ./... -git commit -m "feat(process): add CLI commands" -``` - ---- - -## Task 8: Docs + Example + Integration Test - -**Files:** - -- Create: `examples/sdk/client/process.go` -- Create: `test/integration/process_test.go` -- Create: `docs/docs/sidebar/features/process-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/process/process.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/process/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/process/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/process/signal.md` -- Create: `docs/docs/sidebar/sdk/client/management/process.md` -- Modify: `docs/docs/sidebar/features/features.md` -- Modify: `docs/docs/sidebar/features/authentication.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `docs/docs/sidebar/sdk/client/client.md` -- Modify: `docs/docusaurus.config.ts` (Features + SDK dropdowns) - -SDK doc goes under `management/` category (same as Agent, Job, Health, Audit — -process management is an operational concern). - -- [ ] **Step 1: Create SDK example** - -Demonstrate `client.Process.List()` and `client.Process.Get()`. Don't -demonstrate Signal in the example (destructive). - -- [ ] **Step 2: Create integration test** - -`ProcessSmokeSuite` with `TestProcessList`. Guard signal test with `skipWrite`. - -- [ ] **Step 3: Create feature page** - -`process-management.md` — list, get, signal operations. CLI examples. -Permissions. Platforms. - -- [ ] **Step 4: Create CLI doc pages** - -Directory with landing page + list.md, get.md, signal.md. - -- [ ] **Step 5: Create SDK doc page** - -Under `management/`. Title: `# Process`. Methods: List, Get, Signal. Add to -`client.md` Management table. - -- [ ] **Step 6: Update all shared docs** - -Features table, authentication permissions, configuration roles, API guidelines -endpoints, architecture feature link, docusaurus dropdowns (Features + SDK under -Management group). - -- [ ] **Step 7: Regenerate and verify** - -```bash -just generate -go build ./... -just go::unit -just go::unit-cov # >= 99.9% -just go::vet -``` - -- [ ] **Step 8: Commit** - -```bash -git commit -m "feat(process): add docs, SDK example, and integration tests" -``` diff --git a/docs/plans/2026-03-31-user-management-provider-design.md b/docs/plans/2026-03-31-user-management-provider-design.md deleted file mode 100644 index 5f583902e..000000000 --- a/docs/plans/2026-03-31-user-management-provider-design.md +++ /dev/null @@ -1,187 +0,0 @@ -# User & Group Management Provider Design - -## Overview - -Add user and group management to OSAPI. CRUD operations for local system users -and groups. Uses `useradd`, `usermod`, `userdel`, `groupadd`, `groupmod`, -`groupdel`, and `chpasswd` via `exec.Manager`. Never exposes password hashes. - -## Architecture - -Direct provider at `internal/provider/node/user/`. One provider package with two -sets of methods (users and groups). Two API path prefixes under -`/node/{hostname}/`: `user/` and `group/`. - -- **Category**: `node` -- **Permissions**: `user:read`, `user:write` -- **Provider type**: direct (exec.Manager) - -## Provider Interface - -```go -type Provider interface { - // Users - ListUsers(ctx context.Context) ([]User, error) - GetUser(ctx context.Context, name string) (*User, error) - CreateUser(ctx context.Context, opts CreateUserOpts) (*UserResult, error) - UpdateUser(ctx context.Context, name string, opts UpdateUserOpts) (*UserResult, error) - DeleteUser(ctx context.Context, name string) (*UserResult, error) - ChangePassword(ctx context.Context, name string, password string) (*UserResult, error) - - // Groups - ListGroups(ctx context.Context) ([]Group, error) - GetGroup(ctx context.Context, name string) (*Group, error) - CreateGroup(ctx context.Context, opts CreateGroupOpts) (*GroupResult, error) - UpdateGroup(ctx context.Context, name string, opts UpdateGroupOpts) (*GroupResult, error) - DeleteGroup(ctx context.Context, name string) (*GroupResult, error) -} -``` - -## Data Types - -```go -type User struct { - Name string `json:"name"` - UID int `json:"uid"` - GID int `json:"gid"` - Home string `json:"home"` - Shell string `json:"shell"` - Groups []string `json:"groups,omitempty"` - Locked bool `json:"locked"` -} - -type CreateUserOpts struct { - Name string `json:"name"` - UID int `json:"uid,omitempty"` - GID int `json:"gid,omitempty"` - Home string `json:"home,omitempty"` - Shell string `json:"shell,omitempty"` - Groups []string `json:"groups,omitempty"` - Password string `json:"password,omitempty"` - System bool `json:"system,omitempty"` -} - -type UpdateUserOpts struct { - Shell string `json:"shell,omitempty"` - Home string `json:"home,omitempty"` - Groups []string `json:"groups,omitempty"` - Lock *bool `json:"lock,omitempty"` -} - -type Group struct { - Name string `json:"name"` - GID int `json:"gid"` - Members []string `json:"members,omitempty"` -} - -type CreateGroupOpts struct { - Name string `json:"name"` - GID int `json:"gid,omitempty"` - System bool `json:"system,omitempty"` -} - -type UpdateGroupOpts struct { - Members []string `json:"members,omitempty"` -} - -type UserResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type GroupResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -## Debian Implementation - -- **ListUsers**: parse `/etc/passwd`, filter UID >= 1000 by default (skip system - users like root, daemon, etc.) -- **GetUser**: parse `/etc/passwd` for specific user, read groups via - `id -Gn ` -- **CreateUser**: `useradd` with flags for UID (`-u`), home (`-d`), shell - (`-s`), groups (`-G`), system (`-r`), create home (`-m`). If password - provided, pipe to `chpasswd` after creation. -- **UpdateUser**: `usermod` with flags for shell (`-s`), home (`-d -m`), groups - (`-G`). Lock via `usermod -L`, unlock via `usermod -U`. -- **DeleteUser**: `userdel -r` (removes home directory) -- **ChangePassword**: echo `name:password` and pipe to `chpasswd` -- **ListGroups**: parse `/etc/group` -- **GetGroup**: parse `/etc/group` for specific group -- **CreateGroup**: `groupadd` with optional GID (`-g`), system (`-r`) -- **UpdateGroup**: `groupmod` for membership. Use `gpasswd -M` to set member - list. -- **DeleteGroup**: `groupdel` - -Never expose password hashes — only read from `/etc/passwd` (which doesn't -contain hashes), not `/etc/shadow`. - -## Platform Implementations - -| Platform | Implementation | -| -------- | -------------------------------------------------- | -| Debian | useradd/usermod/userdel/groupadd/groupmod/groupdel | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -Return `ErrUnsupported` in containers — user/group management is the host's -concern. - -## API Endpoints - -### User Endpoints - -| Method | Path | Permission | Description | -| -------- | --------------------------------------- | ------------ | --------------- | -| `GET` | `/node/{hostname}/user` | `user:read` | List users | -| `GET` | `/node/{hostname}/user/{name}` | `user:read` | Get user | -| `POST` | `/node/{hostname}/user` | `user:write` | Create user | -| `PUT` | `/node/{hostname}/user/{name}` | `user:write` | Update user | -| `DELETE` | `/node/{hostname}/user/{name}` | `user:write` | Delete user | -| `POST` | `/node/{hostname}/user/{name}/password` | `user:write` | Change password | - -### Group Endpoints - -| Method | Path | Permission | Description | -| -------- | ------------------------------- | ------------ | ------------ | -| `GET` | `/node/{hostname}/group` | `user:read` | List groups | -| `GET` | `/node/{hostname}/group/{name}` | `user:read` | Get group | -| `POST` | `/node/{hostname}/group` | `user:write` | Create group | -| `PUT` | `/node/{hostname}/group/{name}` | `user:write` | Update group | -| `DELETE` | `/node/{hostname}/group/{name}` | `user:write` | Delete group | - -All endpoints support broadcast targeting. - -## SDK - -Two SDK services sharing the same permissions: - -```go -// UserService -client.User.List(ctx, host) -client.User.Get(ctx, host, name) -client.User.Create(ctx, host, opts) -client.User.Update(ctx, host, name, opts) -client.User.Delete(ctx, host, name) -client.User.ChangePassword(ctx, host, name, password) - -// GroupService -client.Group.List(ctx, host) -client.Group.Get(ctx, host, name) -client.Group.Create(ctx, host, opts) -client.Group.Update(ctx, host, name, opts) -client.Group.Delete(ctx, host, name) -``` - -## Permissions - -- `user:read` — list and get (users and groups). Added to admin, write, and read - roles. -- `user:write` — create, update, delete, change password. Added to admin and - write roles. diff --git a/docs/plans/2026-03-31-user-management-provider.md b/docs/plans/2026-03-31-user-management-provider.md deleted file mode 100644 index 8839c59f6..000000000 --- a/docs/plans/2026-03-31-user-management-provider.md +++ /dev/null @@ -1,566 +0,0 @@ -# User & Group Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add user and group management as a node provider with CRUD operations, -password changes, and full API/CLI/SDK support. - -**Architecture:** Direct provider at `provider/node/user/` using `useradd`, -`usermod`, `userdel`, `groupadd`, `groupmod`, `groupdel`, and `chpasswd` via -`exec.Manager`. Parses `/etc/passwd` and `/etc/group` for reads. Two API path -prefixes (`user/` and `group/`), two SDK services (`UserService` and -`GroupService`). Permissions: `user:read` (all roles), `user:write` (admin + -write). - -**Tech Stack:** Go 1.25, Echo, oapi-codegen (strict-server), gomock, -testify/suite, avfs - -**Coverage baseline:** 99.9% — must remain at or above this. - ---- - -## Task 1: SDK Constants (Operations + Permissions) - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/job/types.go` -- Modify: `internal/authtoken/permissions.go` - -- [ ] **Step 1: Add user and group operation constants** - -In `pkg/sdk/client/operations.go`: - -```go -// User operations. -const ( - OpUserList JobOperation = "node.user.list" - OpUserGet JobOperation = "node.user.get" - OpUserCreate JobOperation = "node.user.create" - OpUserUpdate JobOperation = "node.user.update" - OpUserDelete JobOperation = "node.user.delete" - OpUserChangePassword JobOperation = "node.user.password" -) - -// Group operations. -const ( - OpGroupList JobOperation = "node.group.list" - OpGroupGet JobOperation = "node.group.get" - OpGroupCreate JobOperation = "node.group.create" - OpGroupUpdate JobOperation = "node.group.update" - OpGroupDelete JobOperation = "node.group.delete" -) -``` - -- [ ] **Step 2: Add permission constants** - -```go - PermUserRead Permission = "user:read" - PermUserWrite Permission = "user:write" -``` - -- [ ] **Step 3: Re-export in internal/job/types.go** - -- [ ] **Step 4: Re-export permissions in internal/authtoken/permissions.go** - -Add to `DefaultRolePermissions`: - -- `RoleAdmin`: `PermUserRead` + `PermUserWrite` -- `RoleWrite`: `PermUserRead` + `PermUserWrite` -- `RoleRead`: `PermUserRead` only - -- [ ] **Step 5: Verify and commit** - -```bash -go build ./... -git commit -m "feat(user): add operation and permission constants" -``` - ---- - -## Task 2: Provider Interface + Platform Stubs - -**Files:** - -- Create: `internal/provider/node/user/types.go` -- Create: `internal/provider/node/user/darwin.go` -- Create: `internal/provider/node/user/linux.go` -- Create: `internal/provider/node/user/mocks/generate.go` - -- [ ] **Step 1: Create types.go** - -Provider interface with 11 methods (6 user + 5 group). All data types: User, -Group, CreateUserOpts, UpdateUserOpts, CreateGroupOpts, UpdateGroupOpts, -UserResult, GroupResult. See design spec for exact type definitions. - -- [ ] **Step 2: Create darwin.go and linux.go stubs** - -All 11 methods return `fmt.Errorf("user: %w", provider.ErrUnsupported)`. - -- [ ] **Step 3: Create mocks and generate** - -- [ ] **Step 4: Verify and commit** - -```bash -go generate ./internal/provider/node/user/mocks/... -go build ./... -git commit -m "feat(user): add provider interface and platform stubs" -``` - ---- - -## Task 3: Debian Provider — User Operations - -**Files:** - -- Create: `internal/provider/node/user/debian.go` -- Create: `internal/provider/node/user/debian_user.go` -- Create: `internal/provider/node/user/debian_public_test.go` -- Create: `internal/provider/node/user/darwin_public_test.go` -- Create: `internal/provider/node/user/linux_public_test.go` - -Split the Debian implementation across two files for readability: `debian.go` -for the struct/constructor + group methods, `debian_user.go` for user methods. -Or organize by concern: `debian.go` for struct/constructor, `debian_user.go` for -user ops, `debian_group.go` for group ops. - -- [ ] **Step 1: Write stub tests (Darwin, Linux)** - -Verify all 11 methods return `ErrUnsupported`. - -- [ ] **Step 2: Write Debian user tests** - -Test cases for each user method: - -**TestListUsers:** - -- success (parse /etc/passwd, filter UID >= 1000) -- parse error -- empty result (no non-system users) - -**TestGetUser:** - -- success (user exists) -- user not found -- exec error (id -Gn fails) - -**TestCreateUser:** - -- success with minimal opts (name only) -- success with all opts (UID, home, shell, groups, password, system) -- useradd error (user already exists) -- password set after creation - -**TestUpdateUser:** - -- success changing shell -- success changing groups -- success locking user -- success unlocking user -- usermod error - -**TestDeleteUser:** - -- success -- user not found -- userdel error - -**TestChangePassword:** - -- success -- chpasswd error - -- [ ] **Step 3: Implement debian.go + debian_user.go** - -```go -type Debian struct { - provider.FactsAware - logger *slog.Logger - fs avfs.VFS - execManager exec.Manager -} - -func NewDebianProvider( - logger *slog.Logger, - fs avfs.VFS, - execManager exec.Manager, -) *Debian -``` - -Use `avfs.VFS` for reading `/etc/passwd` and `/etc/group` (testable with memfs). -Use `exec.Manager` for running useradd/usermod/etc. - -Parsing `/etc/passwd`: each line is `name:x:uid:gid:gecos:home:shell`. Filter -UID >= 1000 for ListUsers. GetUser reads all UIDs. - -For ChangePassword: construct `name:password` string and pipe to `chpasswd`. -Read exec.Manager to understand how to pass stdin. If exec.Manager doesn't -support stdin, use `exec.Manager.RunCmd("chpasswd", []string{})` with the input -as a separate call, or use `usermod --password $(openssl passwd -6 password)`. - -- [ ] **Step 4: Verify user tests pass with 100% coverage** - ---- - -## Task 4: Debian Provider — Group Operations - -**Files:** - -- Create: `internal/provider/node/user/debian_group.go` -- Modify: `internal/provider/node/user/debian_public_test.go` (add group tests) - -- [ ] **Step 1: Write Debian group tests** - -**TestListGroups:** - -- success (parse /etc/group) -- parse error - -**TestGetGroup:** - -- success -- group not found - -**TestCreateGroup:** - -- success with name only -- success with GID and system flag -- groupadd error - -**TestUpdateGroup:** - -- success updating members -- gpasswd error - -**TestDeleteGroup:** - -- success -- groupdel error - -- [ ] **Step 2: Implement debian_group.go** - -Parsing `/etc/group`: each line is `name:x:gid:member1,member2`. - -- [ ] **Step 3: Verify all tests pass with 100% coverage** - -```bash -go test -coverprofile=/tmp/c.out ./internal/provider/node/user/... -git commit -m "feat(user): implement Debian user and group provider with tests" -``` - ---- - -## Task 5: Agent Processor + Wiring - -**Files:** - -- Create: `internal/agent/processor_user.go` -- Create: `internal/agent/processor_user_public_test.go` -- Modify: `internal/agent/processor.go` -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Create processor with tests** - -Two base operations dispatching to sub-operations: - -`user.*` operations: - -- `user.list` — no data, call `provider.ListUsers(ctx)` -- `user.get` — unmarshal `{"name": "..."}`, call `provider.GetUser` -- `user.create` — unmarshal `CreateUserOpts`, call `provider.CreateUser` -- `user.update` — unmarshal `{"name": "...", ...opts}`, call - `provider.UpdateUser` -- `user.delete` — unmarshal `{"name": "..."}`, call `provider.DeleteUser` -- `user.password` — unmarshal `{"name": "...", "password": "..."}`, call - `provider.ChangePassword` - -`group.*` operations: - -- `group.list` — no data, call `provider.ListGroups(ctx)` -- `group.get` — unmarshal `{"name": "..."}`, call `provider.GetGroup` -- `group.create` — unmarshal `CreateGroupOpts`, call `provider.CreateGroup` -- `group.update` — unmarshal `{"name": "...", ...opts}`, call - `provider.UpdateGroup` -- `group.delete` — unmarshal `{"name": "..."}`, call `provider.DeleteGroup` - -- [ ] **Step 2: Add to node processor** - -Add `userProvider user.Provider` to `NewNodeProcessor`. Add `case "user":` and -`case "group":` dispatch — both route to the same provider but different -methods. - -- [ ] **Step 3: Wire in agent_setup.go** - -```go -func createUserProvider( - log *slog.Logger, - fs avfs.VFS, - execManager exec.Manager, -) userProv.Provider -``` - -Container check: return `ErrUnsupported` in containers. - -- [ ] **Step 4: Fix existing tests and verify** - -```bash -go test ./internal/agent/... ./cmd/... -git commit -m "feat(user): add agent processor and wiring" -``` - ---- - -## Task 6: OpenAPI Spec + User API Handlers - -**Files:** - -- Create: `internal/controller/api/node/user/gen/api.yaml` -- Create: `internal/controller/api/node/user/gen/cfg.yaml` -- Create: `internal/controller/api/node/user/gen/generate.go` -- Create: `internal/controller/api/node/user/types.go` -- Create: `internal/controller/api/node/user/user.go` -- Create: `internal/controller/api/node/user/validate.go` -- Create: `internal/controller/api/node/user/user_list_get.go` -- Create: `internal/controller/api/node/user/user_get.go` -- Create: `internal/controller/api/node/user/user_create.go` -- Create: `internal/controller/api/node/user/user_update.go` -- Create: `internal/controller/api/node/user/user_delete.go` -- Create: `internal/controller/api/node/user/user_password.go` -- Create: `internal/controller/api/node/user/handler.go` -- Create: test files for each handler -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Create OpenAPI spec** - -The spec includes BOTH user and group endpoints. Six user paths: - -- `GET /node/{hostname}/user` — list users -- `POST /node/{hostname}/user` — create user -- `GET /node/{hostname}/user/{name}` — get user -- `PUT /node/{hostname}/user/{name}` — update user -- `DELETE /node/{hostname}/user/{name}` — delete user -- `POST /node/{hostname}/user/{name}/password` — change password - -Five group paths: - -- `GET /node/{hostname}/group` — list groups -- `POST /node/{hostname}/group` — create group -- `GET /node/{hostname}/group/{name}` — get group -- `PUT /node/{hostname}/group/{name}` — update group -- `DELETE /node/{hostname}/group/{name}` — delete group - -All user endpoints use `user:read` or `user:write`. Group endpoints use the same -permissions. - -Request/response schemas for users and groups. - -- [ ] **Step 2: Generate code and implement user handlers** - -Category `"node"`. User operations use `job.OperationUser*`. User list/get use -`JobClient.Query`. User create/update/delete/ password use `JobClient.Modify`. - -- [ ] **Step 3: Create handler.go (self-registration)** - -- [ ] **Step 4: Write tests with RBAC for user handlers** - -- [ ] **Step 5: Wire in controller_setup.go** - -```bash -go build ./... -go test ./internal/controller/api/node/user/... ./cmd/... -git commit -m "feat(user): add user OpenAPI spec and API handlers" -``` - ---- - -## Task 7: Group API Handlers - -**Files:** - -- Create: `internal/controller/api/node/user/group_list_get.go` -- Create: `internal/controller/api/node/user/group_get.go` -- Create: `internal/controller/api/node/user/group_create.go` -- Create: `internal/controller/api/node/user/group_update.go` -- Create: `internal/controller/api/node/user/group_delete.go` -- Create: test files for each handler - -Group handlers live in the same `api/node/user/` package as user handlers — they -share the same OpenAPI spec and handler struct. - -- [ ] **Step 1: Implement group handlers with broadcast support** - -Category `"node"`. Group operations use `job.OperationGroup*`. - -- [ ] **Step 2: Write tests with RBAC** - -- [ ] **Step 3: Verify** - -```bash -go test ./internal/controller/api/node/user/... -git commit -m "feat(user): add group API handlers with tests" -``` - ---- - -## Task 8: SDK Services - -**Files:** - -- Create: `pkg/sdk/client/user.go` -- Create: `pkg/sdk/client/user_types.go` -- Create: `pkg/sdk/client/user_public_test.go` -- Create: `pkg/sdk/client/user_types_public_test.go` -- Create: `pkg/sdk/client/group.go` -- Create: `pkg/sdk/client/group_types.go` -- Create: `pkg/sdk/client/group_public_test.go` -- Create: `pkg/sdk/client/group_types_public_test.go` -- Modify: `pkg/sdk/client/osapi.go` - -Two SDK services: - -**UserService:** - -- `List(ctx, hostname)` -- `Get(ctx, hostname, name)` -- `Create(ctx, hostname, opts)` -- `Update(ctx, hostname, name, opts)` -- `Delete(ctx, hostname, name)` -- `ChangePassword(ctx, hostname, name, password)` - -**GroupService:** - -- `List(ctx, hostname)` -- `Get(ctx, hostname, name)` -- `Create(ctx, hostname, opts)` -- `Update(ctx, hostname, name, opts)` -- `Delete(ctx, hostname, name)` - -Wire both into `osapi.go`: `User *UserService`, `Group *GroupService`. - -Run `just generate` before creating SDK services. - -- [ ] **Step 1: Create user SDK service + types + tests** -- [ ] **Step 2: Create group SDK service + types + tests** -- [ ] **Step 3: Wire into osapi.go** -- [ ] **Step 4: Verify** - -```bash -go test ./pkg/sdk/client/... -git commit -m "feat(user): add SDK services with tests" -``` - ---- - -## Task 9: CLI Commands - -**Files:** - -- Create: `cmd/client_node_user.go` — parent -- Create: `cmd/client_node_user_list.go` -- Create: `cmd/client_node_user_get.go` -- Create: `cmd/client_node_user_create.go` -- Create: `cmd/client_node_user_update.go` -- Create: `cmd/client_node_user_delete.go` -- Create: `cmd/client_node_user_password.go` -- Create: `cmd/client_node_group.go` — parent -- Create: `cmd/client_node_group_list.go` -- Create: `cmd/client_node_group_get.go` -- Create: `cmd/client_node_group_create.go` -- Create: `cmd/client_node_group_update.go` -- Create: `cmd/client_node_group_delete.go` - -- [ ] **Step 1: Create user CLI commands** - -User list: table fields NAME, UID, GID, HOME, SHELL, GROUPS, LOCKED. User get: -flags `--name` (required). Same table. User create: flags `--name` (required), -`--uid`, `--gid`, `--home`, `--shell`, `--groups` (string slice), `--password`, -`--system`. User update: flags `--name` (required), `--shell`, `--home`, -`--groups` (string slice), `--lock`, `--unlock`. User delete: flags `--name` -(required). User password: flags `--name` (required), `--password` (required). - -- [ ] **Step 2: Create group CLI commands** - -Group list: table fields NAME, GID, MEMBERS. Group get: flags `--name` -(required). Group create: flags `--name` (required), `--gid`, `--system`. Group -update: flags `--name` (required), `--members` (string slice). Group delete: -flags `--name` (required). - -- [ ] **Step 3: Verify build** - -```bash -go build ./... -git commit -m "feat(user): add CLI commands" -``` - ---- - -## Task 10: Docs + Examples + Integration Test - -**Files:** - -- Create: `examples/sdk/client/user.go` -- Create: `examples/sdk/client/group.go` -- Create: `test/integration/user_test.go` -- Create: `docs/docs/sidebar/features/user-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/user.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/create.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/update.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/delete.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/password.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/group/group.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/group/list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/group/get.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/group/create.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/group/update.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/group/delete.md` -- Create: `docs/docs/sidebar/sdk/client/management/user.md` -- Create: `docs/docs/sidebar/sdk/client/management/group.md` -- Modify: shared docs (features table, auth, config, api guidelines, - architecture, client.md, docusaurus.config.ts) - -- [ ] **Step 1: Create SDK examples (user.go, group.go)** - -User example: list users, get by name. Group example: list groups, create a -group. - -- [ ] **Step 2: Create integration test** - -`UserSmokeSuite` with `TestUserList` (read-only) and `TestGroupList` -(read-only). Guard writes with `skipWrite`. - -- [ ] **Step 3: Create feature page + CLI docs** - -Feature page: `user-management.md`. CLI docs as directories with landing pages. - -- [ ] **Step 4: Create SDK doc pages** - -Under `management/`: `user.md` and `group.md`. Add both to `client.md` -Management table. - -- [ ] **Step 5: Update all shared docs** - -Features table, auth permissions, config roles, API guidelines (11 endpoints), -architecture feature link, docusaurus dropdowns (Features + SDK under Management -group). - -- [ ] **Step 6: Regenerate and verify** - -```bash -just generate -go build ./... -just go::unit -just go::unit-cov # >= 99.9% -just go::vet -``` - -- [ ] **Step 7: Commit** - -```bash -git commit -m "feat(user): add docs, SDK examples, and integration tests" -``` diff --git a/docs/plans/2026-04-01-audit-stream-migration-plan.md b/docs/plans/2026-04-01-audit-stream-migration-plan.md deleted file mode 100644 index 1c7b15261..000000000 --- a/docs/plans/2026-04-01-audit-stream-migration-plan.md +++ /dev/null @@ -1,807 +0,0 @@ -# Audit Stream Migration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace the audit KV store with a JetStream stream for chronological -ordering and efficient pagination, add `trace_id` to audit entries. - -**Architecture:** The audit `StreamStore` receives a `jetstream.Stream` handle -(for reads: `GetLastMsgForSubject`, `OrderedConsumer`, `Info`) and uses -`nc.Publish()` (via a `Publisher` interface) for writes. The `Store` interface -is unchanged — all consumers (handlers, middleware, export) work without -modification. Config changes from KV bucket fields to stream fields. - -**Tech Stack:** Go, NATS JetStream streams, OpenTelemetry trace context - ---- - -### Task 1: Update config types and YAML - -**Files:** - -- Modify: `internal/config/types.go:124-132` -- Modify: `configs/osapi.yaml` (default config) -- Modify: `configs/osapi.nerd.yaml` (dev config) - -- [ ] **Step 1: Update the NATSAudit config struct** - -Replace the KV bucket config with stream config: - -```go -// NATSAudit configuration for the audit log stream. -type NATSAudit struct { - // Stream is the JetStream stream name for audit log entries. - Stream string `mapstructure:"stream"` - // Subject is the base subject prefix for audit messages. - Subject string `mapstructure:"subject"` - MaxAge string `mapstructure:"max_age"` // e.g. "720h" (30 days) - MaxBytes int64 `mapstructure:"max_bytes"` - Storage string `mapstructure:"storage"` // "file" or "memory" - Replicas int `mapstructure:"replicas"` -} -``` - -- [ ] **Step 2: Update osapi.yaml configs** - -In both `configs/osapi.yaml` and `configs/osapi.nerd.yaml`, change the -`nats.audit` section: - -```yaml -nats: - audit: - stream: 'AUDIT' - subject: 'audit' - max_age: '720h' - max_bytes: 52428800 - storage: 'file' - replicas: 1 -``` - -- [ ] **Step 3: Verify it compiles** - -Run: `go build ./...` - -Expect: compile errors in files that reference `NATSAudit.Bucket` and -`NATSAudit.TTL` — that's expected, we fix them in subsequent tasks. - -- [ ] **Step 4: Commit** - -``` -chore(config): rename audit config from KV bucket to stream -``` - ---- - -### Task 2: Update CLI config builder and NATS setup - -**Files:** - -- Modify: `internal/cli/nats.go:128-142` -- Modify: `internal/cli/nats_public_test.go` (update test for renamed function) -- Modify: `cmd/nats_setup.go:150-155` - -- [ ] **Step 1: Replace BuildAuditKVConfig with BuildAuditStreamConfig** - -In `internal/cli/nats.go`, replace `BuildAuditKVConfig`: - -```go -// BuildAuditStreamConfig builds a jetstream.StreamConfig from audit -// config values. -func BuildAuditStreamConfig( - namespace string, - auditCfg config.NATSAudit, -) jetstream.StreamConfig { - streamName := job.ApplyNamespaceToInfraName( - namespace, - auditCfg.Stream, - ) - subject := job.ApplyNamespaceToSubjects( - namespace, - auditCfg.Subject, - ) - maxAge, _ := time.ParseDuration(auditCfg.MaxAge) - - return jetstream.StreamConfig{ - Name: streamName, - Subjects: []string{subject + ".>"}, - MaxAge: maxAge, - MaxBytes: auditCfg.MaxBytes, - Storage: ParseJetstreamStorageType(auditCfg.Storage), - Replicas: auditCfg.Replicas, - Discard: jetstream.DiscardOld, - } -} -``` - -- [ ] **Step 2: Update the test for the renamed function** - -In `internal/cli/nats_public_test.go`, update the test that exercises -`BuildAuditKVConfig` to test `BuildAuditStreamConfig` instead. The test should -verify stream name, subjects with `.>` suffix, max age, storage type, and -replicas. - -- [ ] **Step 3: Update nats_setup.go to create stream instead of KV** - -In `cmd/nats_setup.go`, replace the audit KV bucket creation block: - -```go -if appConfig.NATS.Audit.Stream != "" { - auditStreamConfig := cli.BuildAuditStreamConfig( - namespace, - appConfig.NATS.Audit, - ) - if err := nc.CreateOrUpdateStreamWithConfig( - ctx, - auditStreamConfig, - ); err != nil { - return fmt.Errorf( - "create audit stream %s: %w", - auditStreamConfig.Name, - err, - ) - } -} -``` - -- [ ] **Step 4: Run tests and verify build** - -Run: `go test ./internal/cli/... -count=1` Run: `go build ./...` - -Expect: cli tests pass, build still has errors in controller_setup.go (expected -— fixed in Task 4). - -- [ ] **Step 5: Commit** - -``` -feat(audit): replace KV bucket setup with stream creation -``` - ---- - -### Task 3: Add TraceID to audit entry and OpenAPI spec - -**Files:** - -- Modify: `internal/audit/types.go:27-48` -- Modify: `internal/controller/api/audit/gen/api.yaml:190-247` -- Modify: `internal/controller/api/audit/audit_list.go:75-94` (mapEntryToGen) -- Modify: `internal/controller/api/middleware_audit.go` -- Modify: `internal/controller/api/export_test.go` (if needed) -- Modify: `pkg/sdk/client/audit_types.go` - -- [ ] **Step 1: Add TraceID to the audit Entry struct** - -In `internal/audit/types.go`, add the field: - -```go -type Entry struct { - ID string `json:"id"` - Timestamp time.Time `json:"timestamp"` - User string `json:"user"` - Roles []string `json:"roles"` - Method string `json:"method"` - Path string `json:"path"` - OperationID string `json:"operation_id,omitempty"` - SourceIP string `json:"source_ip"` - ResponseCode int `json:"response_code"` - DurationMs int64 `json:"duration_ms"` - TraceID string `json:"trace_id,omitempty"` -} -``` - -- [ ] **Step 2: Add trace_id to the OpenAPI spec** - -In `internal/controller/api/audit/gen/api.yaml`, add `trace_id` to the -`AuditEntry` schema properties (after `duration_ms`): - -```yaml -trace_id: - type: string - description: OpenTelemetry trace ID for correlation. - example: '4bf92f3577b34da6a3ce929d0e0e4736' -``` - -Do NOT add it to `required` — it's optional (empty when tracing is disabled). - -- [ ] **Step 3: Regenerate OpenAPI code** - -Run: `just generate` - -- [ ] **Step 4: Update mapEntryToGen in audit_list.go** - -Add the `TraceID` mapping in `mapEntryToGen`: - -```go -if e.TraceID != "" { - entry.TraceId = &e.TraceID -} -``` - -- [ ] **Step 5: Add trace ID extraction to the audit middleware** - -In `internal/controller/api/middleware_audit.go`, add the import for -`go.opentelemetry.io/otel/trace` and extract the trace ID: - -```go -spanCtx := trace.SpanContextFromContext( - c.Request().Context(), -) -if spanCtx.HasTraceID() { - entry.TraceID = spanCtx.TraceID().String() -} -``` - -Add this after building the `entry` struct and before the goroutine that writes -it. - -- [ ] **Step 6: Add TraceID to SDK audit types** - -In `pkg/sdk/client/audit_types.go`, add to `AuditEntry`: - -```go -TraceID string `json:"trace_id,omitempty"` -``` - -Update `auditEntryFromGen` to map the field: - -```go -if g.TraceId != nil { - a.TraceID = *g.TraceId -} -``` - -- [ ] **Step 7: Run tests** - -Run: `go test ./internal/controller/api/audit/... -count=1` Run: -`go test ./internal/controller/api/ -run Audit -count=1` Run: -`go test ./pkg/sdk/client/ -run Audit -count=1` - -Expect: all pass. The middleware test uses a hand-written spy that already -accepts the new field (it stores the full `Entry`). The trace ID will be empty -in tests since there's no OTel span — that's correct. - -- [ ] **Step 8: Commit** - -``` -feat(audit): add trace_id field for OpenTelemetry correlation -``` - ---- - -### Task 4: Implement the stream store - -**Files:** - -- Create: `internal/audit/stream_store.go` -- Create: `internal/audit/stream_store_public_test.go` -- Modify: `internal/audit/export_test.go` (keep marshalJSON export) -- Delete: `internal/audit/kv_store.go` -- Delete: `internal/audit/kv_store_test.go` -- Delete: `internal/audit/kv_store_public_test.go` - -- [ ] **Step 1: Create the StreamStore** - -Create `internal/audit/stream_store.go`: - -```go -package audit - -import ( - "context" - "encoding/json" - "fmt" - "log/slog" - - "github.com/nats-io/nats.go/jetstream" -) - -// ensure StreamStore implements Store at compile time. -var _ Store = (*StreamStore)(nil) - -// marshalJSON is a package-level variable for testing the marshal -// error path. -var marshalJSON = json.Marshal - -// Publisher publishes messages to a NATS subject. -type Publisher interface { - Publish( - ctx context.Context, - subject string, - data []byte, - ) error -} - -// StreamStore implements Store backed by a NATS JetStream stream. -type StreamStore struct { - stream jetstream.Stream - publisher Publisher - subject string - logger *slog.Logger -} - -// NewStreamStore creates a new StreamStore with the given -// dependencies. The subject is the base prefix (e.g., "audit"); -// messages are published to "audit.{id}". -func NewStreamStore( - logger *slog.Logger, - stream jetstream.Stream, - publisher Publisher, - subject string, -) *StreamStore { - return &StreamStore{ - stream: stream, - publisher: publisher, - subject: subject, - logger: logger.With(slog.String("subsystem", "audit")), - } -} - -// Write persists an audit entry to the stream. -func (s *StreamStore) Write( - ctx context.Context, - entry Entry, -) error { - data, err := marshalJSON(entry) - if err != nil { - return fmt.Errorf("marshal audit entry: %w", err) - } - - subject := s.subject + "." + entry.ID - if err := s.publisher.Publish(ctx, subject, data); err != nil { - return fmt.Errorf("publish audit entry: %w", err) - } - - return nil -} - -// Get retrieves a single audit entry by ID using subject lookup. -func (s *StreamStore) Get( - ctx context.Context, - id string, -) (*Entry, error) { - subject := s.subject + "." + id - - msg, err := s.stream.GetLastMsgForSubject(ctx, subject) - if err != nil { - return nil, fmt.Errorf( - "get audit entry: not found: %w", - err, - ) - } - - var entry Entry - if err := json.Unmarshal(msg.Data, &entry); err != nil { - return nil, fmt.Errorf("unmarshal audit entry: %w", err) - } - - return &entry, nil -} - -// List retrieves audit entries with pagination, newest first. -// Uses the stream's message count for total and an ordered consumer -// for efficient sequential reads. -func (s *StreamStore) List( - ctx context.Context, - limit int, - offset int, -) ([]Entry, int, error) { - info, err := s.stream.Info(ctx) - if err != nil { - return nil, 0, fmt.Errorf("get stream info: %w", err) - } - - total := int(info.State.Msgs) - if total == 0 || offset >= total { - return []Entry{}, total, nil - } - - // For newest-first: read from the end. - // We want entries at positions [total-offset-limit .. total-offset) - // mapped to stream sequences [first .. last]. - startIdx := total - offset - limit - if startIdx < 0 { - startIdx = 0 - } - count := total - offset - startIdx - - startSeq := info.State.FirstSeq + uint64(startIdx) - - consumer, err := s.stream.OrderedConsumer( - ctx, - jetstream.OrderedConsumerConfig{ - DeliverPolicy: jetstream.DeliverByStartSequencePolicy, - OptStartSeq: startSeq, - }, - ) - if err != nil { - return nil, 0, fmt.Errorf("create ordered consumer: %w", err) - } - - entries := make([]Entry, 0, count) - - fetchCtx, cancel := context.WithCancel(ctx) - defer cancel() - - batch, err := consumer.Fetch(count, jetstream.FetchMaxWait(fetchTimeout)) - if err != nil { - return nil, 0, fmt.Errorf("fetch audit entries: %w", err) - } - - for msg := range batch.Messages() { - var entry Entry - if err := json.Unmarshal(msg.Data(), &entry); err != nil { - s.logger.Warn( - "failed to unmarshal audit entry", - slog.String("error", err.Error()), - ) - - continue - } - - entries = append(entries, entry) - } - - if batchErr := batch.Error(); batchErr != nil { - s.logger.Warn( - "batch fetch error", - slog.String("error", batchErr.Error()), - ) - } - - _ = fetchCtx - - // Reverse for newest-first order. - for i, j := 0, len(entries)-1; i < j; i, j = i+1, j-1 { - entries[i], entries[j] = entries[j], entries[i] - } - - return entries, total, nil -} - -// ListAll retrieves all audit entries, newest first. -func (s *StreamStore) ListAll( - ctx context.Context, -) ([]Entry, error) { - info, err := s.stream.Info(ctx) - if err != nil { - return nil, fmt.Errorf("get stream info: %w", err) - } - - total := int(info.State.Msgs) - if total == 0 { - return []Entry{}, nil - } - - consumer, err := s.stream.OrderedConsumer( - ctx, - jetstream.OrderedConsumerConfig{ - DeliverPolicy: jetstream.DeliverAllPolicy, - }, - ) - if err != nil { - return nil, fmt.Errorf("create ordered consumer: %w", err) - } - - entries := make([]Entry, 0, total) - - batch, err := consumer.Fetch( - total, - jetstream.FetchMaxWait(fetchTimeout), - ) - if err != nil { - return nil, fmt.Errorf("fetch audit entries: %w", err) - } - - for msg := range batch.Messages() { - var entry Entry - if err := json.Unmarshal(msg.Data(), &entry); err != nil { - s.logger.Warn( - "failed to unmarshal audit entry", - slog.String("error", err.Error()), - ) - - continue - } - - entries = append(entries, entry) - } - - if batchErr := batch.Error(); batchErr != nil { - s.logger.Warn( - "batch fetch error", - slog.String("error", batchErr.Error()), - ) - } - - // Reverse for newest-first order. - for i, j := 0, len(entries)-1; i < j; i, j = i+1, j-1 { - entries[i], entries[j] = entries[j], entries[i] - } - - return entries, nil -} -``` - -Also add a `fetchTimeout` constant at the top of the file: - -```go -const fetchTimeout = 5 * time.Second -``` - -And add `"time"` to the imports. - -- [ ] **Step 2: Update export_test.go** - -The `export_test.go` file currently exports `SetMarshalJSON` / -`ResetMarshalJSON` for the `marshalJSON` var in `kv_store.go`. Since -`stream_store.go` declares the same `marshalJSON` var, the export test file -works unchanged. Verify it still compiles. - -- [ ] **Step 3: Write the stream store tests** - -Create `internal/audit/stream_store_public_test.go`. Use gomock to mock -`jetstream.Stream` and the `Publisher` interface. Follow the exact same test -structure as `kv_store_public_test.go`: - -Test `Write`: - -- successfully publishes entry (mock publisher expects - `Publish(ctx, "audit.{id}", data)`) -- returns error when publish fails -- returns error when marshal fails (via `SetMarshalJSON`) - -Test `Get`: - -- successfully gets entry (mock stream `GetLastMsgForSubject` returns - `RawStreamMsg` with valid JSON data) -- returns error containing "not found" when subject not found -- returns error when unmarshal fails (bad JSON data) - -Test `List`: - -- returns all entries newest-first when within limit -- applies pagination correctly (offset + limit) -- returns empty when offset exceeds total -- returns empty for empty stream (Info returns `State.Msgs == 0`) -- returns error when stream info fails -- skips entries when unmarshal fails (bad JSON in batch) - -Test `ListAll`: - -- returns all entries newest-first -- returns empty for empty stream -- returns error when stream info fails -- skips entries when unmarshal fails - -For mocking `jetstream.Stream`: create a mock interface in -`internal/audit/mocks/` using `go:generate mockgen`. The mock needs `Info`, -`GetLastMsgForSubject`, and `OrderedConsumer` methods. - -For mocking the `Publisher` interface: add a `go:generate mockgen` directive for -the `Publisher` interface defined in `stream_store.go`. - -For mocking `jetstream.Consumer` (returned by `OrderedConsumer`): mock its -`Fetch` method which returns `MessageBatch`. Mock `MessageBatch` for its -`Messages()` channel and `Error()` method. - -Target: **100% coverage** on `stream_store.go`. - -- [ ] **Step 4: Delete old KV store files** - -Delete: - -- `internal/audit/kv_store.go` -- `internal/audit/kv_store_test.go` -- `internal/audit/kv_store_public_test.go` - -- [ ] **Step 5: Regenerate mocks** - -Update `internal/audit/mocks/generate.go` to generate mocks for the new -interfaces: - -```go -//go:generate go tool github.com/golang/mock/mockgen -source=../store.go -destination=store.gen.go -package=mocks -//go:generate go tool github.com/golang/mock/mockgen -source=../stream_store.go -destination=publisher.gen.go -package=mocks -mock_names=Publisher=MockPublisher -``` - -Run: `go generate ./internal/audit/mocks/...` - -Also generate mocks for the `jetstream.Stream` and `jetstream.Consumer` -interfaces used in tests. These can live in `internal/audit/mocks/` or use -`gomock`'s reflect mode for the `jetstream` package interfaces. Check how -existing tests in the codebase mock `jetstream` interfaces (e.g., the -`job/mocks` package) and follow the same pattern. - -- [ ] **Step 6: Run tests and check coverage** - -Run: `go test ./internal/audit/... -count=1 -coverprofile=/tmp/audit.out` Run: -`go tool cover -func=/tmp/audit.out | grep stream_store` - -Expect: 100% coverage on `stream_store.go`. - -- [ ] **Step 7: Commit** - -``` -feat(audit): implement stream-based audit store -``` - ---- - -### Task 5: Wire stream store in controller setup - -**Files:** - -- Modify: `cmd/controller_setup.go:640-658` - -- [ ] **Step 1: Replace createAuditStore to use stream** - -Replace the `createAuditStore` function: - -```go -func createAuditStore( - ctx context.Context, - log *slog.Logger, - nc NATSClient, - namespace string, -) (audit.Store, []api.Option) { - if appConfig.NATS.Audit.Stream == "" { - return nil, nil - } - - auditStreamConfig := cli.BuildAuditStreamConfig( - namespace, - appConfig.NATS.Audit, - ) - if err := nc.CreateOrUpdateStreamWithConfig( - ctx, - auditStreamConfig, - ); err != nil { - cli.LogFatal(log, "failed to create audit stream", err) - } - - streamName := job.ApplyNamespaceToInfraName( - namespace, - appConfig.NATS.Audit.Stream, - ) - stream, err := nc.Stream(ctx, streamName) - if err != nil { - cli.LogFatal(log, "failed to get audit stream", err) - } - - subject := job.ApplyNamespaceToSubjects( - namespace, - appConfig.NATS.Audit.Subject, - ) - - store := audit.NewStreamStore(log, stream, nc, subject) - - return store, []api.Option{api.WithAuditStore(store)} -} -``` - -Note: `nc` (the `NATSClient`) satisfies `audit.Publisher` since it has a -`Publish(ctx, subject, data) error` method. Verify the method signature matches. -If the `NATSClient` interface's `Publish` method signature matches -`audit.Publisher`, pass it directly. If not, create a thin adapter. - -- [ ] **Step 2: Verify build** - -Run: `go build ./...` - -Expect: clean build. - -- [ ] **Step 3: Run full test suite** - -Run: `just go::unit` - -Expect: all tests pass. - -- [ ] **Step 4: Commit** - -``` -feat(audit): wire stream store in controller setup -``` - ---- - -### Task 6: Update documentation - -**Files:** - -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/features/audit-logging.md` - -- [ ] **Step 1: Update configuration.md** - -Replace the `nats.audit` section in the full reference YAML: - -```yaml -audit: - # JetStream stream name for audit log entries. - stream: 'AUDIT' - # Base subject prefix for audit messages. - subject: 'audit' - # Maximum age of audit entries (Go duration). Default 30 days. - max_age: '720h' - # Maximum total size of the audit stream in bytes. - max_bytes: 52428800 # 50 MiB - # Storage backend: "file" or "memory". - storage: 'file' - # Number of stream replicas. - replicas: 1 -``` - -Update the `nats.audit` section reference table: - -| Key | Type | Description | -| ----------- | ------ | ------------------------------------ | -| `stream` | string | JetStream stream name for audit logs | -| `subject` | string | Base subject prefix for audit msgs | -| `max_age` | string | Maximum entry age (Go duration) | -| `max_bytes` | int | Maximum stream size in bytes | -| `storage` | string | `"file"` or `"memory"` | -| `replicas` | int | Number of stream replicas | - -Update the environment variable table — replace `OSAPI_NATS_AUDIT_BUCKET` and -`OSAPI_NATS_AUDIT_TTL` with `OSAPI_NATS_AUDIT_STREAM`, -`OSAPI_NATS_AUDIT_SUBJECT`, and `OSAPI_NATS_AUDIT_MAX_AGE`. - -- [ ] **Step 2: Update audit-logging.md if it mentions KV** - -Check `docs/docs/sidebar/features/audit-logging.md` for references to "KV -bucket" or "bucket" and update to "stream". Add a note about the `trace_id` -field. - -- [ ] **Step 3: Commit** - -``` -docs(audit): update config reference for stream migration -``` - ---- - -### Task 7: Final verification - -- [ ] **Step 1: Run full test suite with coverage** - -```bash -go test ./internal/audit/... -coverprofile=/tmp/audit.out -count=1 -go tool cover -func=/tmp/audit.out | grep -v mocks -``` - -Verify 100% on `stream_store.go`. - -```bash -go test ./internal/controller/api/audit/... -count=1 -go test ./internal/controller/api/ -run Audit -count=1 -go test ./pkg/sdk/client/ -run Audit -count=1 -``` - -All must pass. - -- [ ] **Step 2: Build and lint** - -```bash -go build ./... -just go::vet -``` - -- [ ] **Step 3: Verify no references to old KV audit remain** - -```bash -grep -r "AuditKV\|BuildAuditKVConfig\|NewKVStore\|audit.*bucket\|kv_store" \ - --include="*.go" internal/ cmd/ pkg/ | grep -v _test.go | grep -v mocks -``` - -Expect: no matches. - -- [ ] **Step 4: Run docs formatting** - -```bash -just docs::fmt-check -``` - -Fix any formatting issues. diff --git a/docs/plans/2026-04-01-audit-stream-migration.md b/docs/plans/2026-04-01-audit-stream-migration.md deleted file mode 100644 index 78a8d6a8e..000000000 --- a/docs/plans/2026-04-01-audit-stream-migration.md +++ /dev/null @@ -1,152 +0,0 @@ -# Audit Stream Migration - -Migrate the audit store from NATS KV to a JetStream stream for chronological -ordering and efficient pagination. - -## Problem - -Audit entries are stored in a NATS KV bucket keyed by random UUID v4. `List()` -fetches all keys into memory, sorts them (incorrectly — UUIDs don't sort -chronologically), then paginates. With 1400+ entries and a 30-day TTL, this gets -progressively slower and returns entries in random order. - -## Solution - -Replace the KV bucket with a JetStream stream. Use ULIDs as message subjects for -chronological ordering and direct lookup. Add `trace_id` to audit entries for -OpenTelemetry correlation. - -## Design - -### Config Change - -```yaml -# Before -nats: - audit: - bucket: 'audit-log' - ttl: '720h' - max_bytes: 52428800 - storage: 'file' - replicas: 1 - -# After -nats: - audit: - stream: 'AUDIT' - subject: 'audit' - max_age: '720h' - max_bytes: 52428800 - storage: 'file' - replicas: 1 -``` - -Fields renamed: `bucket` -> `stream`, `ttl` -> `max_age`. New field: `subject` -(base subject for audit messages). Drop `bucket` entirely. - -### Audit Entry - -Add one field to `Entry`: - -```go -type Entry struct { - ID string `json:"id"` - Timestamp time.Time `json:"timestamp"` - User string `json:"user"` - Roles []string `json:"roles,omitempty"` - Method string `json:"method"` - Path string `json:"path"` - SourceIP string `json:"source_ip"` - ResponseCode int `json:"response_code"` - DurationMs int64 `json:"duration_ms"` - OperationID string `json:"operation_id,omitempty"` - TraceID string `json:"trace_id,omitempty"` // NEW -} -``` - -The `ID` field changes from UUID to ULID. This is the only breaking change — no -backward compatibility needed. - -### Store Interface - -The `Store` interface stays the same: - -```go -type Store interface { - Write(ctx context.Context, entry Entry) error - Get(ctx context.Context, id string) (*Entry, error) - List(ctx context.Context, limit int, offset int) ([]Entry, int, error) - ListAll(ctx context.Context) ([]Entry, error) -} -``` - -### Stream Store Operations - -| Operation | Implementation | -| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Write | `js.Publish("audit.{ulid}", data)` | -| Get | `stream.GetMsg(ctx, &GetMsgRequest{NextFor: "audit.{id}"})` | -| List | `stream.Info()` for total count; ordered consumer with `DeliverByStartSequence` for pagination; read newest-first by computing start sequence from total - offset | -| ListAll | Ordered consumer from sequence 1, read all messages forward | -| Count | `stream.Info().State.Msgs` | - -### Middleware Change - -Extract trace ID from OpenTelemetry span context in the audit middleware: - -```go -spanCtx := trace.SpanContextFromContext(c.Request().Context()) -if spanCtx.HasTraceID() { - entry.TraceID = spanCtx.TraceID().String() -} -``` - -### Files Changed - -Production code: - -- `internal/audit/types.go` — add `TraceID` field to `Entry` -- `internal/audit/stream_store.go` — new stream-based `Store` impl -- `internal/audit/kv_store.go` — delete -- `internal/audit/mocks/` — regenerate -- `internal/config/types.go` — update audit config struct -- `internal/controller/api/middleware_audit.go` — add trace ID, use ULID -- `cmd/nats_setup.go` — create stream instead of KV bucket -- `cmd/controller_setup.go` — wire stream store -- `internal/controller/api/audit/gen/api.yaml` — add `trace_id` field -- `internal/controller/api/audit/audit_list.go` — update `mapEntryToGen` -- `internal/controller/api/audit/audit_get.go` — update if needed -- `pkg/sdk/client/audit_types.go` — add `TraceID` to SDK types -- `docs/docs/sidebar/usage/configuration.md` — update config reference - -Test code: - -- `internal/audit/stream_store_public_test.go` — new, 100% coverage -- `internal/audit/kv_store_test.go` — delete -- `internal/audit/kv_store_public_test.go` — delete -- `internal/controller/api/middleware_audit_public_test.go` — update -- Update all existing test files that reference changed types - -### Coverage Baseline - -All files below are currently at 100% coverage. The new implementation must -maintain 100%: - -| File | Current | -| --------------------------------------------- | ----------------------------- | -| `internal/audit/kv_store.go` | 100% -> new `stream_store.go` | -| `internal/audit/export/` | 100% | -| `internal/controller/api/audit/` | 100% | -| `internal/controller/api/middleware_audit.go` | 100% | -| `pkg/sdk/client/audit.go` | 100% | -| `pkg/sdk/client/audit_types.go` | 100% | - -### Not Changing - -- `internal/audit/export/` — export uses `ListAll()` via the `Store` interface, - no changes needed -- OpenAPI spec for audit list/get/export — response shapes stay the same, just - add `trace_id` field -- CLI commands — they consume SDK types, pick up `trace_id` via `--json` - automatically -- Job KV, registry KV, state KV, facts KV — no changes diff --git a/docs/plans/2026-04-01-service-management-provider-design.md b/docs/plans/2026-04-01-service-management-provider-design.md deleted file mode 100644 index 8deb9bd89..000000000 --- a/docs/plans/2026-04-01-service-management-provider-design.md +++ /dev/null @@ -1,260 +0,0 @@ -# Service Management Provider Design - -## Overview - -Add systemd service management to OSAPI. List and inspect services, control them -(start/stop/restart/enable/disable), and manage custom unit files via Object -Store deployment. Hybrid provider — direct for control operations, meta for unit -file CRUD. - -## Architecture - -Hybrid provider at `internal/provider/node/service/`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/service` -- **Permissions**: `service:read`, `service:write` -- **Provider type**: hybrid (exec.Manager + file.Deployer) - -## Provider Interface - -```go -type Provider interface { - // Read - List(ctx context.Context) ([]Info, error) - Get(ctx context.Context, name string) (*Info, error) - // Unit file CRUD (meta provider pattern) - Create(ctx context.Context, entry Entry) (*CreateResult, error) - Update(ctx context.Context, entry Entry) (*UpdateResult, error) - Delete(ctx context.Context, name string) (*DeleteResult, error) - // Control actions (direct provider pattern) - Start(ctx context.Context, name string) (*ActionResult, error) - Stop(ctx context.Context, name string) (*ActionResult, error) - Restart(ctx context.Context, name string) (*ActionResult, error) - Enable(ctx context.Context, name string) (*ActionResult, error) - Disable(ctx context.Context, name string) (*ActionResult, error) -} -``` - -## Data Types - -```go -type Info struct { - Name string `json:"name"` - Status string `json:"status"` - Enabled bool `json:"enabled"` - Description string `json:"description,omitempty"` - PID int `json:"pid,omitempty"` -} - -type Entry struct { - Name string `json:"name"` - Object string `json:"object,omitempty"` -} - -type CreateResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type UpdateResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type DeleteResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type ActionResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` -} -``` - -## Debian Implementation - -The Debian struct needs: - -- `provider.FactsAware` embedded -- `logger *slog.Logger` -- `fs avfs.VFS` — unit file existence checks -- `fileDeployer file.Deployer` — unit file deployment -- `stateKV jetstream.KeyValue` — managed file tracking -- `execManager exec.Manager` — systemctl commands -- `hostname string` — state key construction - -### Read Operations - -- **List**: Run `systemctl list-units --type=service --all --output=json`. Parse - JSON output into `[]Info`. Each entry maps `ActiveState` to status and - `UnitFileState` to enabled. -- **Get**: Run - `systemctl show {name} --property=ActiveState,UnitFileState,Description,MainPID`. - Parse key=value output into `Info`. - -### Unit File CRUD (Meta Provider) - -- **Create**: Deploy unit file from Object Store to - `/etc/systemd/system/osapi-{name}.service` via `file.Deployer` with mode - `0644`. Run `systemctl daemon-reload`. Fails if unit already exists. -- **Update**: Redeploy unit file to same path. `file.Deployer` compares SHA — if - unchanged, returns `changed: false` and skips `daemon-reload`. If object not - specified, preserve existing (read from state KV). -- **Delete**: Stop and disable the service first (best-effort), undeploy via - `file.Deployer`, run `systemctl daemon-reload`. - -### Control Actions (Direct Provider) - -- **Start**: Check current state via `systemctl is-active {name}`. If already - active, return `changed: false`. Otherwise run `systemctl start {name}`. -- **Stop**: Check if active. If already inactive, return `changed: false`. - Otherwise run `systemctl stop {name}`. -- **Restart**: Always run `systemctl restart {name}`, return `changed: true`. No - idempotency check — restart is always intentional. -- **Enable**: Check via `systemctl is-enabled {name}`. If already enabled, - return `changed: false`. Otherwise run `systemctl enable {name}`. -- **Disable**: Check if enabled. If already disabled, return `changed: false`. - Otherwise run `systemctl disable {name}`. - -## Platform Implementations - -| Platform | Implementation | -| -------- | ------------------------- | -| Debian | systemctl + file.Deployer | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -Return `ErrUnsupported` in containers — systemctl requires systemd which isn't -available in standard containers. - -## API Endpoints - -| Method | Path | Permission | Description | -| -------- | ----------------------------------------- | --------------- | ------------------- | -| `GET` | `/node/{hostname}/service` | `service:read` | List all services | -| `GET` | `/node/{hostname}/service/{name}` | `service:read` | Get service details | -| `POST` | `/node/{hostname}/service` | `service:write` | Create unit file | -| `PUT` | `/node/{hostname}/service/{name}` | `service:write` | Update unit file | -| `DELETE` | `/node/{hostname}/service/{name}` | `service:write` | Delete unit file | -| `POST` | `/node/{hostname}/service/{name}/start` | `service:write` | Start service | -| `POST` | `/node/{hostname}/service/{name}/stop` | `service:write` | Stop service | -| `POST` | `/node/{hostname}/service/{name}/restart` | `service:write` | Restart service | -| `POST` | `/node/{hostname}/service/{name}/enable` | `service:write` | Enable at boot | -| `POST` | `/node/{hostname}/service/{name}/disable` | `service:write` | Disable at boot | - -All endpoints support broadcast targeting. - -### POST Request Body (Create) - -```json -{ - "name": "my-app", - "object": "my-app-unit" -} -``` - -`object` references an existing Object Store upload containing the systemd unit -file content. The provider writes it to -`/etc/systemd/system/osapi-{name}.service`. - -### PUT Request Body (Update) - -```json -{ - "object": "my-app-unit-v2" -} -``` - -Name comes from path parameter. Object is the new unit file content from Object -Store. - -### Response Shape (List) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "services": [ - { - "name": "nginx.service", - "status": "active", - "enabled": true, - "description": "A high performance web server", - "pid": 1234 - } - ] - } - ] -} -``` - -### Response Shape (Get) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "service": { - "name": "nginx.service", - "status": "active", - "enabled": true, - "description": "A high performance web server", - "pid": 1234 - } - } - ] -} -``` - -### Response Shape (Actions + CRUD) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "name": "nginx.service", - "changed": true - } - ] -} -``` - -## SDK - -```go -client.Service.List(ctx, host) -client.Service.Get(ctx, host, name) -client.Service.Create(ctx, host, opts) -client.Service.Update(ctx, host, name, opts) -client.Service.Delete(ctx, host, name) -client.Service.Start(ctx, host, name) -client.Service.Stop(ctx, host, name) -client.Service.Restart(ctx, host, name) -client.Service.Enable(ctx, host, name) -client.Service.Disable(ctx, host, name) -``` - -`ServiceCreateOpts` with `Name` and `Object` fields. `ServiceUpdateOpts` with -`Object` field. - -## Permissions - -- `service:read` — list and get. Added to admin, write, and read roles. -- `service:write` — create, update, delete, start, stop, restart, enable, - disable. Added to admin and write roles. diff --git a/docs/plans/2026-04-01-service-management-provider.md b/docs/plans/2026-04-01-service-management-provider.md deleted file mode 100644 index 16c534fd8..000000000 --- a/docs/plans/2026-04-01-service-management-provider.md +++ /dev/null @@ -1,921 +0,0 @@ -# Service Management Provider Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add systemd service management to OSAPI — list, inspect, -start/stop/restart, enable/disable services, and manage custom unit files via -Object Store deployment. - -**Architecture:** Hybrid provider at `internal/provider/node/service/` using -`exec.Manager` for systemctl control operations and `file.Deployer` for unit -file CRUD. Registered under the `node` agent category. Container check enabled — -systemctl requires systemd. - -**Tech Stack:** Go, exec.Manager, file.Deployer, avfs.VFS, systemctl, -oapi-codegen strict-server - ---- - -## File Structure - -### Provider Layer - -- Create: `internal/provider/node/service/types.go` — Provider interface + - domain types -- Create: `internal/provider/node/service/debian.go` — Debian struct, - constructor, compile-time checks -- Create: `internal/provider/node/service/debian_list.go` — List implementation - (systemctl list-units) -- Create: `internal/provider/node/service/debian_get.go` — Get implementation - (systemctl show) -- Create: `internal/provider/node/service/debian_action.go` — - Start/Stop/Restart/Enable/Disable -- Create: `internal/provider/node/service/debian_unit.go` — Unit file - Create/Update/Delete via file.Deployer -- Create: `internal/provider/node/service/darwin.go` — macOS stub -- Create: `internal/provider/node/service/linux.go` — generic Linux stub -- Create: `internal/provider/node/service/mocks/generate.go` -- Test: `internal/provider/node/service/debian_list_public_test.go` -- Test: `internal/provider/node/service/debian_get_public_test.go` -- Test: `internal/provider/node/service/debian_action_public_test.go` -- Test: `internal/provider/node/service/debian_unit_public_test.go` -- Test: `internal/provider/node/service/darwin_public_test.go` -- Test: `internal/provider/node/service/linux_public_test.go` - -### Agent Layer - -- Create: `internal/agent/processor_service.go` — service operation dispatcher -- Modify: `internal/agent/processor.go` — add `service` case to NewNodeProcessor -- Modify: `cmd/agent_setup.go` — create service provider factory, wire into - registry -- Test: `internal/agent/processor_service_public_test.go` - -### Operations & Permissions - -- Modify: `pkg/sdk/client/operations.go` — add service operation constants -- Modify: `internal/job/types.go` — add service operation aliases -- Modify: `pkg/sdk/client/permissions.go` — add `PermServiceRead`, - `PermServiceWrite` -- Modify: `internal/authtoken/permissions.go` — add to all roles - -### API Layer - -- Create: `internal/controller/api/node/service/gen/api.yaml` — OpenAPI spec -- Create: `internal/controller/api/node/service/gen/cfg.yaml` -- Create: `internal/controller/api/node/service/gen/generate.go` -- Create: `internal/controller/api/node/service/types.go` — handler struct -- Create: `internal/controller/api/node/service/service.go` — New(), - compile-time check -- Create: `internal/controller/api/node/service/validate.go` — validateHostname -- Create: `internal/controller/api/node/service/service_list_get.go` — list - handler -- Create: `internal/controller/api/node/service/service_get.go` — get handler -- Create: `internal/controller/api/node/service/service_create_post.go` — create - handler -- Create: `internal/controller/api/node/service/service_update_put.go` — update - handler -- Create: `internal/controller/api/node/service/service_delete.go` — delete - handler -- Create: `internal/controller/api/node/service/service_start_post.go` — start - handler -- Create: `internal/controller/api/node/service/service_stop_post.go` — stop - handler -- Create: `internal/controller/api/node/service/service_restart_post.go` — - restart handler -- Create: `internal/controller/api/node/service/service_enable_post.go` — enable - handler -- Create: `internal/controller/api/node/service/service_disable_post.go` — - disable handler -- Create: `internal/controller/api/node/service/handler.go` — Handler() - registration -- Modify: `cmd/controller_setup.go` — register service handler -- Test: one `*_public_test.go` per handler file + handler test - -### SDK Layer - -- Create: `pkg/sdk/client/service.go` — ServiceService methods -- Create: `pkg/sdk/client/service_types.go` — SDK result types + conversions -- Modify: `pkg/sdk/client/osapi.go` — add Service field -- Test: `pkg/sdk/client/service_public_test.go` -- Test: `pkg/sdk/client/service_types_public_test.go` - -### CLI Layer - -- Create: `cmd/client_node_service.go` — parent command -- Create: `cmd/client_node_service_list.go` -- Create: `cmd/client_node_service_get.go` -- Create: `cmd/client_node_service_create.go` -- Create: `cmd/client_node_service_update.go` -- Create: `cmd/client_node_service_delete.go` -- Create: `cmd/client_node_service_start.go` -- Create: `cmd/client_node_service_stop.go` -- Create: `cmd/client_node_service_restart.go` -- Create: `cmd/client_node_service_enable.go` -- Create: `cmd/client_node_service_disable.go` - -### Documentation - -- Create: `docs/docs/sidebar/features/service-management.md` — feature page -- Create: `docs/docs/sidebar/usage/cli/client/node/service/service.md` — CLI - landing -- Create: CLI doc pages for all 10 subcommands -- Create: `docs/docs/sidebar/sdk/client/operations/service.md` — SDK doc -- Create: `examples/sdk/client/service.go` — SDK example -- Modify: `docs/docs/sidebar/features/features.md` -- Modify: `docs/docs/sidebar/features/authentication.md` -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` -- Modify: `docs/docusaurus.config.ts` -- Modify: `docs/docs/sidebar/sdk/client/client.md` - -### Integration Test - -- Create: `test/integration/service_test.go` - ---- - -### Task 1: Provider Interface and Types - -**Files:** - -- Create: `internal/provider/node/service/types.go` - -- [ ] **Step 1: Create provider interface and types** - -```go -// Package service provides systemd service management operations. -package service - -import "context" - -// Provider implements systemd service management operations. -type Provider interface { - // Read - List(ctx context.Context) ([]Info, error) - Get(ctx context.Context, name string) (*Info, error) - // Unit file CRUD (meta provider pattern) - Create(ctx context.Context, entry Entry) (*CreateResult, error) - Update(ctx context.Context, entry Entry) (*UpdateResult, error) - Delete(ctx context.Context, name string) (*DeleteResult, error) - // Control actions (direct provider pattern) - Start(ctx context.Context, name string) (*ActionResult, error) - Stop(ctx context.Context, name string) (*ActionResult, error) - Restart(ctx context.Context, name string) (*ActionResult, error) - Enable(ctx context.Context, name string) (*ActionResult, error) - Disable(ctx context.Context, name string) (*ActionResult, error) -} - -// Info represents a systemd service. -type Info struct { - Name string `json:"name"` - Status string `json:"status"` - Enabled bool `json:"enabled"` - Description string `json:"description,omitempty"` - PID int `json:"pid,omitempty"` -} - -// Entry represents a unit file deployment request. -type Entry struct { - Name string `json:"name"` - Object string `json:"object,omitempty"` -} - -// CreateResult represents the outcome of a unit file creation. -type CreateResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// UpdateResult represents the outcome of a unit file update. -type UpdateResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// DeleteResult represents the outcome of a unit file deletion. -type DeleteResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// ActionResult represents the outcome of a service control action. -type ActionResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` -} -``` - -- [ ] **Step 2: Verify it compiles** - -Run: `go build ./internal/provider/node/service/...` - -- [ ] **Step 3: Commit** - -```bash -git add internal/provider/node/service/types.go -git commit -m "feat(service): add provider interface and types" -``` - ---- - -### Task 2: Platform Stubs (Darwin + Linux) - -**Files:** - -- Create: `internal/provider/node/service/darwin.go` -- Create: `internal/provider/node/service/linux.go` -- Test: `internal/provider/node/service/darwin_public_test.go` -- Test: `internal/provider/node/service/linux_public_test.go` - -- [ ] **Step 1: Write stub tests** - -All 10 methods must return `provider.ErrUnsupported`. One suite method per -provider method, all in a single table. Follow -`internal/provider/node/certificate/darwin_public_test.go`. - -- [ ] **Step 2: Implement stubs** - -All methods return `fmt.Errorf("service: %w", provider.ErrUnsupported)`. - -- [ ] **Step 3: Create mocks** - -Create `internal/provider/node/service/mocks/generate.go` and run -`go generate ./internal/provider/node/service/mocks/...`. - -- [ ] **Step 4: Run tests** - -Run: `go test -v ./internal/provider/node/service/...` - -- [ ] **Step 5: Commit** - -```bash -git add internal/provider/node/service/ -git commit -m "feat(service): add darwin and linux stubs" -``` - ---- - -### Task 3: Debian Implementation — Read Operations - -**Files:** - -- Create: `internal/provider/node/service/debian.go` -- Create: `internal/provider/node/service/debian_list.go` -- Create: `internal/provider/node/service/debian_get.go` -- Test: `internal/provider/node/service/debian_list_public_test.go` -- Test: `internal/provider/node/service/debian_get_public_test.go` - -- [ ] **Step 1: Implement debian.go** - -Debian struct with all dependencies: - -```go -type Debian struct { - provider.FactsAware - logger *slog.Logger - fs avfs.VFS - fileDeployer file.Deployer - stateKV jetstream.KeyValue - execManager exec.Manager - hostname string -} -``` - -Constructor -`NewDebianProvider(logger, fs, fileDeployer, stateKV, execManager, hostname)` -with subsystem `"provider.service"`. - -Compile-time checks for Provider and FactsSetter. - -- [ ] **Step 2: Implement debian_list.go** - -`List` runs `systemctl list-units --type=service --all --output=json`. Parse -JSON output — each entry has `unit` (name), `active` (state), `sub` (sub-state), -`description`. Map `active` to status and determine enabled via a separate -`systemctl is-enabled {name}` call (or batch via -`systemctl list-unit-files --type=service --output=json`). - -For efficiency, use two commands: - -1. `systemctl list-units --type=service --all --output=json` for active/inactive - status -2. `systemctl list-unit-files --type=service --output=json` for enabled/disabled - status - -Merge the two into `[]Info`. - -- [ ] **Step 3: Implement debian_get.go** - -`Get` runs -`systemctl show {name}.service --property=ActiveState,UnitFileState,Description,MainPID`. -Parse key=value output. Map `ActiveState` to status, `UnitFileState=enabled` to -`Enabled: true`, `MainPID` to PID (0 if inactive), `Description` to description. - -If service not found (exit code non-zero), return error. - -- [ ] **Step 4: Write tests** - -**TestList** — table-driven with gomock for execManager: - -- success (mock returns JSON for 2 services) -- exec error on list-units -- exec error on list-unit-files -- empty service list -- malformed JSON - -**TestGet** — table-driven: - -- success (active, enabled service) -- success (inactive, disabled) -- service not found (exec error) -- malformed output - -Use `memfs.New()` for fs (not needed for read ops but required by the struct). - -- [ ] **Step 5: Verify 100% coverage** - -- [ ] **Step 6: Commit** - -```bash -git add internal/provider/node/service/ -git commit -m "feat(service): add list and get operations" -``` - ---- - -### Task 4: Debian Implementation — Control Actions - -**Files:** - -- Create: `internal/provider/node/service/debian_action.go` -- Test: `internal/provider/node/service/debian_action_public_test.go` - -- [ ] **Step 1: Implement control actions** - -Each action checks current state for idempotency, then runs the systemctl -command: - -**Start**: `systemctl is-active {name}` → if "active", return `changed: false`. -Otherwise `systemctl start {name}`. - -**Stop**: `systemctl is-active {name}` → if not "active", return -`changed: false`. Otherwise `systemctl stop {name}`. - -**Restart**: Always `systemctl restart {name}`, return `changed: true`. No -idempotency — restart is intentional. - -**Enable**: `systemctl is-enabled {name}` → if "enabled", return -`changed: false`. Otherwise `systemctl enable {name}`. - -**Disable**: `systemctl is-enabled {name}` → if not "enabled", return -`changed: false`. Otherwise `systemctl disable {name}`. - -All actions validate the service name first using the same `validateName` regex -from the certificate provider: `^[a-zA-Z0-9_@.-]+$` (service names allow dots -and `@`). - -Error wrapping: `fmt.Errorf("service: start: %w", err)` etc. - -- [ ] **Step 2: Write tests** - -**TestStart** — table-driven: - -- success (is-active returns "inactive", start succeeds) -- already active → changed: false -- start error -- is-active check error -- invalid name - -**TestStop** — table-driven: - -- success (is-active returns "active", stop succeeds) -- already stopped → changed: false -- stop error -- invalid name - -**TestRestart** — table-driven: - -- success (always changed: true) -- restart error -- invalid name - -**TestEnable** — table-driven: - -- success (is-enabled returns "disabled", enable succeeds) -- already enabled → changed: false -- enable error -- invalid name - -**TestDisable** — table-driven: - -- success (is-enabled returns "enabled", disable succeeds) -- already disabled → changed: false -- disable error -- invalid name - -- [ ] **Step 3: Verify 100% coverage** - -- [ ] **Step 4: Commit** - -```bash -git add internal/provider/node/service/ -git commit -m "feat(service): add start/stop/restart/enable/disable" -``` - ---- - -### Task 5: Debian Implementation — Unit File CRUD - -**Files:** - -- Create: `internal/provider/node/service/debian_unit.go` -- Test: `internal/provider/node/service/debian_unit_public_test.go` - -Follow the certificate provider pattern exactly. Read -`internal/provider/node/certificate/debian.go`. - -- [ ] **Step 1: Implement unit file CRUD** - -**Create**: Validate name. Check file doesn't exist at -`/etc/systemd/system/osapi-{name}.service`. Deploy via `fileDeployer.Deploy` -with mode `0644`, contentType `raw`, metadata `{"source": "custom"}`. If -changed, run `systemctl daemon-reload`. - -**Update**: Validate name. Check file EXISTS. Deploy same path with new object. -If object empty, preserve from state KV (like cron). If changed, run -`daemon-reload`. - -**Delete**: Validate name. Check file exists (if not, return changed: false). -Stop and disable service first (best-effort — log warnings on error). Undeploy -via `fileDeployer.Undeploy`. If changed, run `daemon-reload`. - -Helper: `daemonReload()` runs `systemctl daemon-reload` via execManager. - -Helper: `isManagedFile(ctx, path)` checks file state KV, same pattern as -cron/certificate. - -Helper: `buildEntryFromState(ctx, name, path)` reads state KV and reconstructs -Entry. - -- [ ] **Step 2: Write tests** - -Use gomock for fileDeployer, stateKV, execManager. Use `memfs.New()` for -filesystem. - -**TestCreate** — table-driven: - -- success (deploy + daemon-reload) -- already exists (fs.Stat succeeds → error) -- deploy error -- daemon-reload error -- invalid name -- deploy returns changed:false → skip daemon-reload - -**TestUpdate** — table-driven: - -- success (stat finds file, deploy + daemon-reload) -- not found → error -- deploy error -- unchanged (changed:false, skip daemon-reload) -- daemon-reload error -- invalid name -- preserve object when not specified - -**TestDelete** — table-driven: - -- success (stop + disable + undeploy + daemon-reload) -- not found → changed:false -- undeploy error -- daemon-reload error -- stop/disable failures are non-fatal -- invalid name - -- [ ] **Step 3: Verify 100% coverage** - -- [ ] **Step 4: Commit** - -```bash -git add internal/provider/node/service/ -git commit -m "feat(service): add unit file CRUD via file.Deployer" -``` - ---- - -### Task 6: Operations, Permissions, and Agent Wiring - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `internal/job/types.go` -- Modify: `pkg/sdk/client/permissions.go` -- Modify: `internal/authtoken/permissions.go` -- Create: `internal/agent/processor_service.go` -- Modify: `internal/agent/processor.go` -- Modify: `cmd/agent_setup.go` -- Test: `internal/agent/processor_service_public_test.go` - -- [ ] **Step 1: Add operation constants** - -```go -// Service operations. -const ( - OpServiceList JobOperation = "node.service.list" - OpServiceGet JobOperation = "node.service.get" - OpServiceCreate JobOperation = "node.service.create" - OpServiceUpdate JobOperation = "node.service.update" - OpServiceDelete JobOperation = "node.service.delete" - OpServiceStart JobOperation = "node.service.start" - OpServiceStop JobOperation = "node.service.stop" - OpServiceRestart JobOperation = "node.service.restart" - OpServiceEnable JobOperation = "node.service.enable" - OpServiceDisable JobOperation = "node.service.disable" -) -``` - -Plus aliases in `internal/job/types.go`. - -- [ ] **Step 2: Add permissions** - -`PermServiceRead` and `PermServiceWrite`. Add to AllPermissions. Add both to -admin, both to write, read-only to read role. - -- [ ] **Step 3: Implement processor** - -`processServiceOperation` dispatches 10 sub-operations. Follow -`processor_certificate.go` for CRUD and `processor_power.go` for action sub-ops. - -Sub-ops: `list`, `get`, `create`, `update`, `delete`, `start`, `stop`, -`restart`, `enable`, `disable`. - -For list: no data needed. For get/start/stop/restart/enable/ disable: unmarshal -`{"name":"..."}`. For create/update: unmarshal `service.Entry`. - -- [ ] **Step 4: Wire into node processor** - -Add `serviceProvider service.Provider` parameter to `NewNodeProcessor`. Add -`case "service"` to the switch. - -- [ ] **Step 5: Wire in agent_setup.go** - -Create `createServiceProvider` — on Debian, needs `fileProvider`, `fileStateKV`, -`execManager`, `hostname`. Container check: return Linux stub if -`platform.IsContainer()`. If `fileProvider == nil`, log warning, return Linux -stub. - -- [ ] **Step 6: Write processor tests** - -**TestProcessServiceOperation** — dispatch-level table: - -- nil provider, invalid operation, unsupported sub-op - -One suite method per sub-operation (list, get, create, update, delete, start, -stop, restart, enable, disable). Each with success, unmarshal error (where -applicable), and provider error cases. - -- [ ] **Step 7: Verify 100% coverage** - -- [ ] **Step 8: Commit** - -```bash -git commit -m "feat(service): add operations, permissions, and agent wiring" -``` - ---- - -### Task 7: OpenAPI Spec and Code Generation - -**Files:** - -- Create: `internal/controller/api/node/service/gen/api.yaml` -- Create: `internal/controller/api/node/service/gen/cfg.yaml` -- Create: `internal/controller/api/node/service/gen/generate.go` - -- [ ] **Step 1: Create OpenAPI spec** - -10 endpoints. Parameters: Hostname, ServiceName (path). - -Request schemas: - -- `ServiceCreateRequest` — name + object (both required, validate tags) -- `ServiceUpdateRequest` — object (required, validate tag) - -Response schemas: - -- `ServiceInfo` — name, status, enabled, description, pid -- `ServiceListEntry` — hostname, status (ok/failed/skipped), services (array of - ServiceInfo), error -- `ServiceGetEntry` — hostname, status, service (ServiceInfo), error -- `ServiceMutationEntry` — hostname, status, name, changed, error -- `ServiceListResponse` — job_id, results (ServiceListEntry[]) -- `ServiceGetResponse` — job_id, results (ServiceGetEntry[]) -- `ServiceMutationResponse` — job_id, results (ServiceMutationEntry[]) - -Action endpoints (start/stop/restart/enable/disable) use POST with no request -body. They share the `ServiceMutationResponse`. - -Security: `service:read` for GET, `service:write` for POST/PUT/DELETE. - -- [ ] **Step 2: Generate code and rebuild** - -```bash -go generate ./internal/controller/api/node/service/gen/... -just generate -go build ./... -``` - -- [ ] **Step 3: Commit** - -```bash -git commit -m "feat(service): add OpenAPI spec and generated code" -``` - ---- - -### Task 8: API Handler Implementation - -**Files:** - -- Create all handler files under `internal/controller/api/node/service/` -- Modify: `cmd/controller_setup.go` - -- [ ] **Step 1: Create handler scaffolding** - -`types.go`, `service.go` (New + compile-time check, subsystem `"api.service"`), -`validate.go`, `handler.go`. - -- [ ] **Step 2: Implement list handler** - -`service_list_get.go` — Query with `"node"` category, -`job.OperationServiceList`. Parse `[]serviceProv.Info` from response. Broadcast -support. - -- [ ] **Step 3: Implement get handler** - -`service_get.go` — Query with `job.OperationServiceGet` and data -`{"name": name}`. Parse single `serviceProv.Info`. - -- [ ] **Step 4: Implement CRUD handlers** - -`service_create_post.go` — Modify with `job.OperationServiceCreate`. Validate -body. Parse mutation response. - -`service_update_put.go` — Modify with `job.OperationServiceUpdate`. Name from -path param. Handle 404. - -`service_delete.go` — Modify with `job.OperationServiceDelete`. - -- [ ] **Step 5: Implement action handlers** - -`service_start_post.go`, `service_stop_post.go`, `service_restart_post.go`, -`service_enable_post.go`, `service_disable_post.go` — all Modify with no request -body, name from path. Data: `{"name": name}`. - -- [ ] **Step 6: Register in controller_setup.go** - -Add import and handler registration. - -- [ ] **Step 7: Write tests** - -One test file per handler file. Each needs success, error, skipped, broadcast, -validation, HTTP wiring, and RBAC tests (401/403/200). One suite method per -handler function, all scenarios as table rows. - -- [ ] **Step 8: Verify 100% coverage** - -- [ ] **Step 9: Commit** - -```bash -git commit -m "feat(service): add API handlers with broadcast support" -``` - ---- - -### Task 9: SDK Service - -**Files:** - -- Create: `pkg/sdk/client/service.go` -- Create: `pkg/sdk/client/service_types.go` -- Modify: `pkg/sdk/client/osapi.go` -- Test: `pkg/sdk/client/service_public_test.go` -- Test: `pkg/sdk/client/service_types_public_test.go` - -- [ ] **Step 1: Implement types** - -```go -type ServiceInfoResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Services []ServiceInfo `json:"services,omitempty"` - Error string `json:"error,omitempty"` -} - -type ServiceInfo struct { - Name string `json:"name,omitempty"` - Status string `json:"status,omitempty"` - Enabled bool `json:"enabled"` - Description string `json:"description,omitempty"` - PID int `json:"pid,omitempty"` -} - -type ServiceGetResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Service *ServiceInfo `json:"service,omitempty"` - Error string `json:"error,omitempty"` -} - -type ServiceMutationResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type ServiceCreateOpts struct { - Name string - Object string -} - -type ServiceUpdateOpts struct { - Object string -} -``` - -Plus conversion functions. - -- [ ] **Step 2: Implement service methods** - -10 methods on `ServiceService`: - -- `List(ctx, hostname)` → `*Response[Collection[ServiceInfoResult]]` -- `Get(ctx, hostname, name)` → `*Response[Collection[ServiceGetResult]]` -- `Create(ctx, hostname, opts)` → `*Response[Collection[ServiceMutationResult]]` -- `Update(ctx, hostname, name, opts)` → - `*Response[Collection[ServiceMutationResult]]` -- `Delete(ctx, hostname, name)` → `*Response[Collection[ServiceMutationResult]]` -- `Start/Stop/Restart/Enable/Disable(ctx, hostname, name)` → - `*Response[Collection[ServiceMutationResult]]` - -Error wrapping: `"service list: %w"`, `"service start: %w"`, etc. - -- [ ] **Step 3: Wire in osapi.go** - -Add `Service *ServiceService` to Client and init in New(). - -- [ ] **Step 4: Regenerate SDK client** - -- [ ] **Step 5: Write tests** - -httptest.Server tests for all 10 methods. Each covers 200, 401, 403, 500, nil -body, transport error. Create/Update also cover 400. Update/Delete also -cover 404. - -Conversion function tests for all converters. - -- [ ] **Step 6: Verify 100% coverage** - -- [ ] **Step 7: Commit** - -```bash -git commit -m "feat(service): add SDK service with tests" -``` - ---- - -### Task 10: CLI Commands - -**Files:** - -- Create: 11 CLI command files - -- [ ] **Step 1: Create parent command** - -`cmd/client_node_service.go` — `Use: "service"`, -`Short: "Manage systemd services"`. Register under `clientNodeCmd`. - -- [ ] **Step 2: Create list command** - -`cmd/client_node_service_list.go` — table headers: `NAME`, `STATUS`, `ENABLED`, -`DESCRIPTION`. Uses `BuildBroadcastTable`. - -- [ ] **Step 3: Create get command** - -`cmd/client_node_service_get.go` — `--name` flag (required). Shows single -service details. Uses `BuildBroadcastTable` with `NAME`, `STATUS`, `ENABLED`, -`DESCRIPTION`, `PID`. - -- [ ] **Step 4: Create CRUD commands** - -`client_node_service_create.go` — `--name`, `--object` (both required). Uses -`BuildMutationTable`. - -`client_node_service_update.go` — `--name`, `--object` (both required). - -`client_node_service_delete.go` — `--name` (required). - -- [ ] **Step 5: Create action commands** - -`client_node_service_start.go`, `client_node_service_stop.go`, -`client_node_service_restart.go`, `client_node_service_enable.go`, -`client_node_service_disable.go` — each has `--name` (required). Uses -`BuildMutationTable` with `NAME`, `CHANGED`. - -- [ ] **Step 6: Verify build** - -```bash -go build ./... -``` - -- [ ] **Step 7: Commit** - -```bash -git commit -m "feat(service): add CLI commands for service management" -``` - ---- - -### Task 11: Documentation and SDK Example - -**Files:** - -- Create all doc files listed in File Structure -- Modify all cross-reference files - -- [ ] **Step 1: Create feature page** - -`docs/docs/sidebar/features/service-management.md` with: - -- How It Works: List, Get, Start/Stop/Restart, Enable/Disable, - Create/Update/Delete unit files -- Operations table (10 operations) -- CLI Usage examples -- Broadcast Support -- Supported Platforms (Debian: Full, Darwin/Linux: Skipped) -- Container Behavior (ErrUnsupported in containers) -- Permissions - -- [ ] **Step 2: Create CLI doc pages** - -Landing page + 10 subcommand docs. - -- [ ] **Step 3: Create SDK doc + example** - -SDK doc at `docs/docs/sidebar/sdk/client/operations/service.md`. Example at -`examples/sdk/client/service.go`. - -- [ ] **Step 4: Update cross-references** - -features.md, authentication.md, configuration.md, architecture.md, -api-guidelines.md, docusaurus.config.ts, client.md. - -- [ ] **Step 5: Commit** - -```bash -git commit -m "docs: add service management feature docs, SDK example, and cross-references" -``` - ---- - -### Task 12: Integration Test and Final Verification - -- [ ] **Step 1: Create integration test** - -`test/integration/service_test.go` — test -`osapi client node service list --target _any --json`. - -- [ ] **Step 2: Run full suite** - -```bash -just generate -go build ./... -just go::unit -just go::vet -``` - -- [ ] **Step 3: Verify 100% coverage on all new code** - -```bash -go test -coverprofile=/tmp/svc.cov \ - ./internal/provider/node/service/... \ - ./internal/agent/... \ - ./internal/controller/api/node/service/... \ - ./pkg/sdk/client/... -go tool cover -func=/tmp/svc.cov | \ - grep "service" | grep -v "100.0%" | \ - grep -v "mocks\|gen/" -``` - -- [ ] **Step 4: Commit any fixes** - -```bash -git commit -m "chore(service): fix formatting and lint" -``` diff --git a/docs/plans/2026-04-01-ssh-key-management-provider-design.md b/docs/plans/2026-04-01-ssh-key-management-provider-design.md deleted file mode 100644 index 8744bc569..000000000 --- a/docs/plans/2026-04-01-ssh-key-management-provider-design.md +++ /dev/null @@ -1,158 +0,0 @@ -# SSH Key Management Provider Design - -## Overview - -Add SSH authorized key management to OSAPI. List, add, and remove SSH public -keys in a user's `~/.ssh/authorized_keys` file. Extends the existing user -provider — no new provider package or permissions. Manages any key regardless of -who added it. - -## Architecture - -Extends the existing user provider at `internal/provider/node/user/`. - -- **Category**: `node` -- **Path prefix**: `/node/{hostname}/user/{name}/ssh-key` -- **Permissions**: `user:read` (list), `user:write` (add, remove) -- **Provider type**: direct-write (avfs.VFS) - -No state tracking, no file.Deployer. The provider reads and writes -`authorized_keys` directly. The orchestrator is responsible for desired-state -management. - -## Provider Interface Additions - -Added to the existing `user.Provider` interface: - -```go -ListKeys(ctx context.Context, username string) ([]SSHKey, error) -AddKey(ctx context.Context, username string, key SSHKey) (*SSHKeyResult, error) -RemoveKey(ctx context.Context, username string, fingerprint string) (*SSHKeyResult, error) -``` - -## Data Types - -```go -type SSHKey struct { - Type string `json:"type"` - Fingerprint string `json:"fingerprint"` - Comment string `json:"comment,omitempty"` -} - -type SSHKeyResult struct { - Changed bool `json:"changed"` -} -``` - -## Debian Implementation - -The provider resolves the user's home directory from `/etc/passwd` (already -parsed by the user provider), then operates on `~/.ssh/authorized_keys`. - -- **ListKeys**: Read `authorized_keys`, parse each non-empty, non-comment line - into type + base64 key + comment. Compute SHA256 fingerprint from decoded key - bytes. Return all entries. -- **AddKey**: Check if key already exists by fingerprint. If present, return - `changed: false`. Otherwise append the raw public key line. Create `~/.ssh/` - (mode `0700`) and `authorized_keys` (mode `0600`) if they don't exist. Set - ownership to the target user via `exec.Manager` (`chown user:user`). -- **RemoveKey**: Read file, filter out the line matching the fingerprint, - rewrite file. Return `changed: false` if fingerprint not found. Return error - if file doesn't exist. - -## Platform Implementations - -| Platform | Implementation | -| -------- | ---------------------- | -| Debian | Direct file read/write | -| Darwin | ErrUnsupported | -| Linux | ErrUnsupported | - -## Container Behavior - -No container check — SSH key management works in containers. - -## API Endpoints - -| Method | Path | Permission | Description | -| -------- | ---------------------------------------------------- | ------------ | -------------------- | -| `GET` | `/node/{hostname}/user/{name}/ssh-key` | `user:read` | List authorized keys | -| `POST` | `/node/{hostname}/user/{name}/ssh-key` | `user:write` | Add a key | -| `DELETE` | `/node/{hostname}/user/{name}/ssh-key/{fingerprint}` | `user:write` | Remove a key | - -All endpoints support broadcast targeting. - -### POST Request Body - -```json -{ - "key": "ssh-ed25519 AAAA... user@host" -} -``` - -The full public key line as it would appear in `authorized_keys`. - -### Response Shape (List) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "keys": [ - { - "type": "ssh-ed25519", - "fingerprint": "SHA256:abc123...", - "comment": "john@laptop" - } - ] - } - ] -} -``` - -### Response Shape (Add/Remove) - -```json -{ - "job_id": "...", - "results": [ - { - "hostname": "web-01", - "status": "ok", - "changed": true - } - ] -} -``` - -## SDK - -```go -client.User.ListKeys(ctx, host, username) -client.User.AddKey(ctx, host, username, opts) -client.User.RemoveKey(ctx, host, username, fingerprint) -``` - -`SSHKeyAddOpts` struct with `Key` field (the full public key string). - -## CLI - -```bash -osapi client node user ssh-key list --target web-01 --name john -osapi client node user ssh-key add --target web-01 --name john \ - --key "ssh-ed25519 AAAA... john@laptop" -osapi client node user ssh-key remove --target web-01 --name john \ - --fingerprint "SHA256:abc123..." -``` - -## Permissions - -Reuses existing permissions — no new permissions needed. - -- `user:read` — list keys -- `user:write` — add and remove keys - -These are already in all built-in roles. diff --git a/docs/plans/2026-04-01-ssh-key-management-provider.md b/docs/plans/2026-04-01-ssh-key-management-provider.md deleted file mode 100644 index 3c49f5931..000000000 --- a/docs/plans/2026-04-01-ssh-key-management-provider.md +++ /dev/null @@ -1,1223 +0,0 @@ -# SSH Key Management Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add SSH authorized key management to OSAPI — list, add, and remove SSH -public keys in a user's `~/.ssh/authorized_keys` file by extending the existing -user provider. - -**Architecture:** Extends `internal/provider/node/user/` with three new methods -(ListKeys, AddKey, RemoveKey). New SSH key endpoints added to the existing user -OpenAPI spec. Operations dispatched via a new `sshKey` case in the node -processor. Reuses existing `user:read`/`user:write` permissions. No new provider -package, agent category, or permissions needed. - -**Tech Stack:** Go, avfs.VFS, crypto/sha256 for fingerprints, encoding/base64 -for key decoding, oapi-codegen strict-server - ---- - -## File Structure - -### Provider Layer - -- Modify: `internal/provider/node/user/types.go` — add SSHKey, SSHKeyResult - types + 3 methods to Provider interface -- Create: `internal/provider/node/user/debian_ssh_key.go` — Debian - implementation (list/add/remove authorized_keys) -- Modify: `internal/provider/node/user/darwin.go` — add 3 stub methods -- Modify: `internal/provider/node/user/linux.go` — add 3 stub methods -- Test: `internal/provider/node/user/debian_ssh_key_public_test.go` -- Modify: `internal/provider/node/user/darwin_public_test.go` — add stub tests -- Modify: `internal/provider/node/user/linux_public_test.go` — add stub tests - -### Agent Layer - -- Create: `internal/agent/processor_ssh_key.go` — SSH key operation dispatcher -- Modify: `internal/agent/processor.go` — add `sshKey` case to NewNodeProcessor -- Test: `internal/agent/processor_ssh_key_public_test.go` - -### Operations - -- Modify: `pkg/sdk/client/operations.go` — add SSH key operation constants -- Modify: `internal/job/types.go` — add SSH key operation aliases - -### API Layer - -- Modify: `internal/controller/api/node/user/gen/api.yaml` — add 3 ssh-key - endpoints + schemas -- Create: `internal/controller/api/node/user/ssh_key_list_get.go` — list handler -- Create: `internal/controller/api/node/user/ssh_key_add_post.go` — add handler -- Create: `internal/controller/api/node/user/ssh_key_remove_delete.go` — remove - handler -- Test: `internal/controller/api/node/user/ssh_key_list_get_public_test.go` -- Test: `internal/controller/api/node/user/ssh_key_add_post_public_test.go` -- Test: `internal/controller/api/node/user/ssh_key_remove_delete_public_test.go` - -### SDK Layer - -- Modify: `pkg/sdk/client/user.go` — add ListKeys, AddKey, RemoveKey methods -- Modify: `pkg/sdk/client/user_types.go` — add SSHKey result types + conversions -- Modify: `pkg/sdk/client/user_public_test.go` — add tests -- Modify: `pkg/sdk/client/user_types_public_test.go` — add conversion tests - -### CLI Layer - -- Create: `cmd/client_node_user_ssh_key.go` — parent command -- Create: `cmd/client_node_user_ssh_key_list.go` — list subcommand -- Create: `cmd/client_node_user_ssh_key_add.go` — add subcommand -- Create: `cmd/client_node_user_ssh_key_remove.go` — remove subcommand - -### Documentation - -- Modify: `docs/docs/sidebar/features/user-management.md` — add SSH key section -- Create: `docs/docs/sidebar/usage/cli/client/node/user/ssh-key.md` — CLI - landing -- Create: `docs/docs/sidebar/usage/cli/client/node/user/ssh-key-list.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/ssh-key-add.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/user/ssh-key-remove.md` -- Modify: `docs/docs/sidebar/sdk/client/management/user.md` — add SSH key - methods -- Modify: `examples/sdk/client/user.go` — add SSH key demo -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` — add endpoints - ---- - -### Task 1: Provider Types and Stubs - -**Files:** - -- Modify: `internal/provider/node/user/types.go` -- Modify: `internal/provider/node/user/darwin.go` -- Modify: `internal/provider/node/user/linux.go` -- Modify: `internal/provider/node/user/darwin_public_test.go` -- Modify: `internal/provider/node/user/linux_public_test.go` - -- [ ] **Step 1: Add types to types.go** - -Add to `internal/provider/node/user/types.go`: - -```go -// SSHKey represents an SSH authorized key entry. -type SSHKey struct { - Type string `json:"type"` - Fingerprint string `json:"fingerprint"` - Comment string `json:"comment,omitempty"` -} - -// SSHKeyResult represents the result of an SSH key mutation. -type SSHKeyResult struct { - Changed bool `json:"changed"` -} -``` - -Add 3 methods to the Provider interface: - -```go - // ListKeys returns SSH authorized keys for a user. - ListKeys(ctx context.Context, username string) ([]SSHKey, error) - // AddKey adds an SSH authorized key for a user. - AddKey(ctx context.Context, username string, key SSHKey) (*SSHKeyResult, error) - // RemoveKey removes an SSH authorized key by fingerprint. - RemoveKey(ctx context.Context, username string, fingerprint string) (*SSHKeyResult, error) -``` - -- [ ] **Step 2: Add stub methods to darwin.go and linux.go** - -Add to both `darwin.go` and `linux.go`: - -```go -// ListKeys returns ErrUnsupported on Darwin/Linux. -func (d *Darwin) ListKeys( - _ context.Context, - _ string, -) ([]SSHKey, error) { - return nil, fmt.Errorf("user: %w", provider.ErrUnsupported) -} - -// AddKey returns ErrUnsupported on Darwin/Linux. -func (d *Darwin) AddKey( - _ context.Context, - _ string, - _ SSHKey, -) (*SSHKeyResult, error) { - return nil, fmt.Errorf("user: %w", provider.ErrUnsupported) -} - -// RemoveKey returns ErrUnsupported on Darwin/Linux. -func (d *Darwin) RemoveKey( - _ context.Context, - _ string, - _ string, -) (*SSHKeyResult, error) { - return nil, fmt.Errorf("user: %w", provider.ErrUnsupported) -} -``` - -(Same for Linux struct.) - -- [ ] **Step 3: Add stub tests** - -Add test cases to the existing test tables in `darwin_public_test.go` and -`linux_public_test.go` for ListKeys, AddKey, and RemoveKey all returning -ErrUnsupported. - -- [ ] **Step 4: Regenerate mocks** - -Run: `go generate ./internal/provider/node/user/mocks/...` - -- [ ] **Step 5: Run tests** - -Run: `go test -v ./internal/provider/node/user/...` Expected: all pass, new stub -tests included - -- [ ] **Step 6: Commit** - -```bash -git add internal/provider/node/user/ -git commit -m "feat(user): add SSH key types and platform stubs" -``` - ---- - -### Task 2: Debian SSH Key Implementation - -**Files:** - -- Create: `internal/provider/node/user/debian_ssh_key.go` -- Test: `internal/provider/node/user/debian_ssh_key_public_test.go` - -- [ ] **Step 1: Write tests** - -Create `debian_ssh_key_public_test.go` with testify/suite. Use `memfs.New()` for -filesystem and gomock for exec.Manager. - -Set up a memfs with `/etc/passwd` containing: - -``` -root:x:0:0:root:/root:/bin/bash -john:x:1000:1000:John:/home/john:/bin/bash -``` - -**TestListKeys** — table-driven: - -- success (authorized_keys with 2 keys, verify type + fingerprint - - comment) -- user not found in /etc/passwd → error -- no authorized_keys file → empty list, no error -- empty authorized_keys → empty list -- lines with comments and blank lines skipped -- malformed key line skipped (logged as debug) - -**TestAddKey** — table-driven: - -- success (creates .ssh dir + file, appends key) -- key already exists (same fingerprint) → changed: false -- user not found → error -- creates .ssh dir with 0700 if missing -- creates authorized_keys with 0600 if missing -- appends to existing file - -**TestRemoveKey** — table-driven: - -- success (rewrites file without matching key) -- fingerprint not found → changed: false -- user not found → error -- no authorized_keys file → changed: false -- file becomes empty after removal (still valid) - -- [ ] **Step 2: Implement debian_ssh_key.go** - -```go -package user - -import ( - "context" - "crypto/sha256" - "encoding/base64" - "fmt" - "log/slog" - "strings" -) - -// ListKeys returns SSH authorized keys for a user. -func (d *Debian) ListKeys( - _ context.Context, - username string, -) ([]SSHKey, error) { - d.logger.Debug("executing user.ListKeys", - slog.String("username", username), - ) - - home, err := d.userHomeDir(username) - if err != nil { - return nil, fmt.Errorf("ssh key: list: %w", err) - } - - authKeysPath := home + "/.ssh/authorized_keys" - - content, err := d.fs.ReadFile(authKeysPath) - if err != nil { - // No file = no keys, not an error. - return []SSHKey{}, nil - } - - return parseAuthorizedKeys(string(content), d.logger), nil -} - -// AddKey adds an SSH authorized key for a user. -func (d *Debian) AddKey( - _ context.Context, - username string, - key SSHKey, -) (*SSHKeyResult, error) { - d.logger.Debug("executing user.AddKey", - slog.String("username", username), - ) - - home, err := d.userHomeDir(username) - if err != nil { - return nil, fmt.Errorf("ssh key: add: %w", err) - } - - sshDir := home + "/.ssh" - authKeysPath := sshDir + "/authorized_keys" - - // Ensure .ssh directory exists. - if err := d.fs.MkdirAll(sshDir, 0o700); err != nil { - return nil, fmt.Errorf( - "ssh key: create .ssh dir: %w", err) - } - - // Read existing keys to check for duplicates. - existing, _ := d.fs.ReadFile(authKeysPath) - existingKeys := parseAuthorizedKeys( - string(existing), d.logger) - - for _, ek := range existingKeys { - if ek.Fingerprint == key.Fingerprint { - return &SSHKeyResult{Changed: false}, nil - } - } - - // Build the key line from the SSHKey fields. - keyLine := key.Type + " " + - base64.StdEncoding.EncodeToString(/* raw key bytes */) - // Actually, the API receives the full key line in a - // dedicated field. See the AddKey handler — it passes - // the raw key line. The provider should store the raw - // public key line. - - // Append key to file. - f, err := d.fs.OpenFile( - authKeysPath, - os.O_APPEND|os.O_CREATE|os.O_WRONLY, - 0o600, - ) - if err != nil { - return nil, fmt.Errorf( - "ssh key: open authorized_keys: %w", err) - } - defer f.Close() - - if _, err := f.Write( - []byte(key.RawLine + "\n"), - ); err != nil { - return nil, fmt.Errorf( - "ssh key: write key: %w", err) - } - - // Set ownership. - if _, err := d.execManager.RunCmd( - "chown", - []string{"-R", username + ":" + username, sshDir}, - ); err != nil { - d.logger.Warn("failed to set .ssh ownership", - slog.String("error", err.Error()), - ) - } - - return &SSHKeyResult{Changed: true}, nil -} - -// RemoveKey removes an SSH authorized key by fingerprint. -func (d *Debian) RemoveKey( - _ context.Context, - username string, - fingerprint string, -) (*SSHKeyResult, error) { - d.logger.Debug("executing user.RemoveKey", - slog.String("username", username), - slog.String("fingerprint", fingerprint), - ) - - home, err := d.userHomeDir(username) - if err != nil { - return nil, fmt.Errorf("ssh key: remove: %w", err) - } - - authKeysPath := home + "/.ssh/authorized_keys" - - content, err := d.fs.ReadFile(authKeysPath) - if err != nil { - return &SSHKeyResult{Changed: false}, nil - } - - lines := strings.Split(string(content), "\n") - var newLines []string - found := false - - for _, line := range lines { - trimmed := strings.TrimSpace(line) - if trimmed == "" || strings.HasPrefix(trimmed, "#") { - newLines = append(newLines, line) - continue - } - - fp := fingerprintFromLine(trimmed) - if fp == fingerprint { - found = true - continue // skip this line - } - newLines = append(newLines, line) - } - - if !found { - return &SSHKeyResult{Changed: false}, nil - } - - newContent := strings.Join(newLines, "\n") - if err := d.fs.WriteFile( - authKeysPath, []byte(newContent), 0o600, - ); err != nil { - return nil, fmt.Errorf( - "ssh key: write authorized_keys: %w", err) - } - - return &SSHKeyResult{Changed: true}, nil -} - -// userHomeDir resolves a user's home directory from -// /etc/passwd. -func (d *Debian) userHomeDir( - username string, -) (string, error) { - content, err := d.fs.ReadFile("/etc/passwd") - if err != nil { - return "", fmt.Errorf( - "read /etc/passwd: %w", err) - } - - for _, line := range strings.Split( - string(content), "\n") { - fields := strings.Split(line, ":") - if len(fields) >= 6 && fields[0] == username { - return fields[5], nil - } - } - - return "", fmt.Errorf("user %q not found", username) -} - -// parseAuthorizedKeys parses an authorized_keys file content -// into SSHKey entries. -func parseAuthorizedKeys( - content string, - logger *slog.Logger, -) []SSHKey { - var keys []SSHKey - - for _, line := range strings.Split(content, "\n") { - line = strings.TrimSpace(line) - if line == "" || strings.HasPrefix(line, "#") { - continue - } - - parts := strings.Fields(line) - if len(parts) < 2 { - logger.Debug("skipping malformed key line", - slog.String("line", line), - ) - continue - } - - keyType := parts[0] - keyData := parts[1] - comment := "" - if len(parts) >= 3 { - comment = strings.Join(parts[2:], " ") - } - - fp := computeFingerprint(keyData) - if fp == "" { - logger.Debug("skipping key with invalid base64", - slog.String("line", line), - ) - continue - } - - keys = append(keys, SSHKey{ - Type: keyType, - Fingerprint: fp, - Comment: comment, - }) - } - - return keys -} - -// computeFingerprint computes SHA256 fingerprint from base64- -// encoded key data. -func computeFingerprint( - keyData string, -) string { - decoded, err := base64.StdEncoding.DecodeString(keyData) - if err != nil { - return "" - } - - hash := sha256.Sum256(decoded) - - return "SHA256:" + - base64.RawStdEncoding.EncodeToString(hash[:]) -} - -// fingerprintFromLine extracts fingerprint from a key line. -func fingerprintFromLine( - line string, -) string { - parts := strings.Fields(line) - if len(parts) < 2 { - return "" - } - - return computeFingerprint(parts[1]) -} -``` - -**IMPORTANT**: The SSHKey type needs a `RawLine` field to store the full public -key string for AddKey. Update the types: - -```go -type SSHKey struct { - Type string `json:"type"` - Fingerprint string `json:"fingerprint"` - Comment string `json:"comment,omitempty"` - RawLine string `json:"raw_line,omitempty"` -} -``` - -The API handler populates `RawLine` from the POST body's `key` field. The -provider uses `RawLine` to append to `authorized_keys`. ListKeys does NOT -populate `RawLine` (we don't expose raw key data in list responses — just type, -fingerprint, comment). - -- [ ] **Step 3: Run tests** - -Run: `go test -v ./internal/provider/node/user/...` Expected: all pass - -- [ ] **Step 4: Verify 100% coverage on new file** - -```bash -go test -coverprofile=/tmp/ssh.cov \ - ./internal/provider/node/user/... && \ - go tool cover -func=/tmp/ssh.cov | \ - grep "debian_ssh_key" -``` - -All functions must be 100%. - -- [ ] **Step 5: Commit** - -```bash -git add internal/provider/node/user/ -git commit -m "feat(user): add SSH key management to debian provider" -``` - ---- - -### Task 3: Operations and Agent Processor - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `internal/job/types.go` -- Create: `internal/agent/processor_ssh_key.go` -- Modify: `internal/agent/processor.go` — add `sshKey` case -- Test: `internal/agent/processor_ssh_key_public_test.go` - -- [ ] **Step 1: Add operation constants** - -In `pkg/sdk/client/operations.go`, add after User operations: - -```go -// SSH Key operations. -const ( - OpSSHKeyList JobOperation = "node.sshKey.list" - OpSSHKeyAdd JobOperation = "node.sshKey.add" - OpSSHKeyRemove JobOperation = "node.sshKey.remove" -) -``` - -In `internal/job/types.go`, add corresponding aliases: - -```go -// SSH Key operations. -const ( - OperationSSHKeyList = client.OpSSHKeyList - OperationSSHKeyAdd = client.OpSSHKeyAdd - OperationSSHKeyRemove = client.OpSSHKeyRemove -) -``` - -- [ ] **Step 2: Write processor tests** - -Create `internal/agent/processor_ssh_key_public_test.go`. The processor -dispatches to the existing `userProvider` (same as user/group operations). Test -via `NewNodeProcessor`. - -**TestProcessSSHKeyOperation** — dispatch-level table: - -- nil user provider → error -- invalid operation format -- unsupported sub-operation - -**TestProcessSSHKeyList** — table-driven: - -- success (returns keys) -- unmarshal error (invalid JSON) -- provider error - -**TestProcessSSHKeyAdd** — table-driven: - -- success -- unmarshal error -- provider error - -**TestProcessSSHKeyRemove** — table-driven: - -- success -- unmarshal error -- provider error - -One suite method per function, ALL scenarios as table rows. - -- [ ] **Step 3: Implement processor_ssh_key.go** - -```go -func processSshKeyOperation( - userProvider user.Provider, - logger *slog.Logger, - jobRequest job.Request, -) (json.RawMessage, error) { - if userProvider == nil { - return nil, fmt.Errorf( - "user provider not available") - } - - parts := strings.Split(jobRequest.Operation, ".") - if len(parts) < 2 { - return nil, fmt.Errorf( - "invalid sshKey operation: %s", - jobRequest.Operation) - } - subOp := parts[1] - - ctx := context.Background() - - switch subOp { - case "list": - return processSshKeyList( - ctx, userProvider, logger, jobRequest) - case "add": - return processSshKeyAdd( - ctx, userProvider, logger, jobRequest) - case "remove": - return processSshKeyRemove( - ctx, userProvider, logger, jobRequest) - default: - return nil, fmt.Errorf( - "unsupported sshKey operation: %s", - jobRequest.Operation) - } -} -``` - -Each sub-handler unmarshals username (and key data for add, fingerprint for -remove) from `jobRequest.Data`, calls the provider, and marshals the result. - -- [ ] **Step 4: Wire into node processor** - -In `internal/agent/processor.go`, add case to the `NewNodeProcessor` switch: - -```go - case "sshKey": - return processSshKeyOperation( - userProvider, logger, req) -``` - -- [ ] **Step 5: Run tests and verify coverage** - -```bash -go test -v ./internal/agent/... -go build ./... -go test -coverprofile=/tmp/ssh_proc.cov \ - ./internal/agent/... && \ - go tool cover -func=/tmp/ssh_proc.cov | \ - grep "processor_ssh_key" -``` - -- [ ] **Step 6: Commit** - -```bash -git add pkg/sdk/client/operations.go \ - internal/job/types.go \ - internal/agent/processor_ssh_key.go \ - internal/agent/processor_ssh_key_public_test.go \ - internal/agent/processor.go -git commit -m "feat(user): add SSH key operations and agent processor" -``` - ---- - -### Task 4: OpenAPI Spec Update and Code Generation - -**Files:** - -- Modify: `internal/controller/api/node/user/gen/api.yaml` - -- [ ] **Step 1: Add endpoints to existing user OpenAPI spec** - -Add to `internal/controller/api/node/user/gen/api.yaml` after the password -endpoint section: - -```yaml -# -- SSH Key management ------------------------------------------------ - -/node/{hostname}/user/{name}/ssh-key: - get: - summary: List SSH authorized keys - description: > - List SSH authorized keys for a user on the target node. - tags: - - user_operations - operationId: GetNodeUserSshKey - security: - - BearerAuth: - - user:read - parameters: - - $ref: '#/components/parameters/Hostname' - - $ref: '#/components/parameters/UserName' - responses: - '200': - description: List of SSH authorized keys. - content: - application/json: - schema: - $ref: '#/components/schemas/SSHKeyCollectionResponse' - '401': ... - '403': ... - '500': ... - - post: - summary: Add SSH authorized key - description: > - Add an SSH authorized key for a user on the target node. - tags: - - user_operations - operationId: PostNodeUserSshKey - security: - - BearerAuth: - - user:write - parameters: - - $ref: '#/components/parameters/Hostname' - - $ref: '#/components/parameters/UserName' - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/SSHKeyAddRequest' - responses: - '200': - description: Key added. - content: - application/json: - schema: - $ref: '#/components/schemas/SSHKeyMutationResponse' - '400': ... - '401': ... - '403': ... - '500': ... - -/node/{hostname}/user/{name}/ssh-key/{fingerprint}: - delete: - summary: Remove SSH authorized key - description: > - Remove an SSH authorized key by fingerprint. - tags: - - user_operations - operationId: DeleteNodeUserSshKey - security: - - BearerAuth: - - user:write - parameters: - - $ref: '#/components/parameters/Hostname' - - $ref: '#/components/parameters/UserName' - - $ref: '#/components/parameters/SSHKeyFingerprint' - responses: - '200': - description: Key removed. - content: - application/json: - schema: - $ref: '#/components/schemas/SSHKeyMutationResponse' - '401': ... - '403': ... - '500': ... -``` - -Add schemas: - -```yaml -SSHKeyAddRequest: - type: object - required: - - key - properties: - key: - type: string - description: > - Full SSH public key line (e.g., "ssh-ed25519 AAAA... user@host"). - x-oapi-codegen-extra-tags: - validate: required,min=1 - -SSHKeyInfo: - type: object - properties: - type: - type: string - example: 'ssh-ed25519' - fingerprint: - type: string - example: 'SHA256:abc123...' - comment: - type: string - example: 'john@laptop' - -SSHKeyEntry: - type: object - properties: - hostname: - type: string - status: - type: string - enum: [ok, failed, skipped] - keys: - type: array - items: - $ref: '#/components/schemas/SSHKeyInfo' - error: - type: string - required: - - hostname - - status - -SSHKeyMutationEntry: - type: object - properties: - hostname: - type: string - status: - type: string - enum: [ok, failed, skipped] - changed: - type: boolean - error: - type: string - required: - - hostname - - status - -SSHKeyCollectionResponse: - type: object - properties: - job_id: - type: string - format: uuid - results: - type: array - items: - $ref: '#/components/schemas/SSHKeyEntry' - required: - - results - -SSHKeyMutationResponse: - type: object - properties: - job_id: - type: string - format: uuid - results: - type: array - items: - $ref: '#/components/schemas/SSHKeyMutationEntry' - required: - - results -``` - -Add parameter: - -```yaml -SSHKeyFingerprint: - name: fingerprint - in: path - required: true - description: SSH key SHA256 fingerprint. - x-oapi-codegen-extra-tags: - validate: required,min=1 - schema: - type: string - minLength: 1 -``` - -- [ ] **Step 2: Generate code and rebuild** - -```bash -go generate ./internal/controller/api/node/user/gen/... -just generate -go build ./... -``` - -- [ ] **Step 3: Commit** - -```bash -git add internal/controller/api/node/user/gen/ \ - internal/controller/api/gen/ \ - pkg/sdk/client/gen/ -git commit -m "feat(user): add SSH key endpoints to OpenAPI spec" -``` - ---- - -### Task 5: API Handler Implementation - -**Files:** - -- Create: `internal/controller/api/node/user/ssh_key_list_get.go` -- Create: `internal/controller/api/node/user/ssh_key_add_post.go` -- Create: `internal/controller/api/node/user/ssh_key_remove_delete.go` -- Test: all 3 `*_public_test.go` files - -- [ ] **Step 1: Implement list handler** - -`GetNodeUserSshKey` method on the existing `User` handler struct: - -- Validate hostname -- username from `request.Name` -- Query with category `"node"`, operation `job.OperationSSHKeyList`, data - `{"username": username}` -- Parse response: unmarshal `[]userProv.SSHKey`, convert to `[]gen.SSHKeyInfo` -- Broadcast support - -- [ ] **Step 2: Implement add handler** - -`PostNodeUserSshKey`: - -- Validate hostname, body (`key` field) -- Parse the raw key line to extract type, fingerprint, comment -- Build `userProv.SSHKey{Type, Fingerprint, Comment, RawLine}` -- Modify with `job.OperationSSHKeyAdd`, data includes `username` + the SSHKey - struct -- Parse mutation response - -- [ ] **Step 3: Implement remove handler** - -`DeleteNodeUserSshKey`: - -- Validate hostname -- fingerprint from `request.Fingerprint` -- Modify with `job.OperationSSHKeyRemove`, data - `{"username": username, "fingerprint": fingerprint}` -- Parse mutation response - -- [ ] **Step 4: Write tests** - -Each handler test file needs: success, skipped, broadcast, validation error, job -error, HTTP wiring, RBAC (401/403/200). One suite method per handler, all -scenarios as table rows. - -- [ ] **Step 5: Run tests and verify coverage** - -```bash -go test -v ./internal/controller/api/node/user/... -go test -coverprofile=/tmp/ssh_h.cov \ - ./internal/controller/api/node/user/... && \ - go tool cover -func=/tmp/ssh_h.cov | \ - grep "ssh_key" | grep -v "100.0%" -``` - -- [ ] **Step 6: Commit** - -```bash -git add internal/controller/api/node/user/ -git commit -m "feat(user): add SSH key API handlers with broadcast support" -``` - ---- - -### Task 6: SDK Service Extension - -**Files:** - -- Modify: `pkg/sdk/client/user.go` -- Modify: `pkg/sdk/client/user_types.go` -- Modify: `pkg/sdk/client/user_public_test.go` -- Modify: `pkg/sdk/client/user_types_public_test.go` - -- [ ] **Step 1: Add types** - -In `user_types.go`, add: - -```go -type SSHKeyInfoResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Keys []SSHKeyInfo `json:"keys,omitempty"` - Error string `json:"error,omitempty"` -} - -type SSHKeyInfo struct { - Type string `json:"type,omitempty"` - Fingerprint string `json:"fingerprint,omitempty"` - Comment string `json:"comment,omitempty"` -} - -type SSHKeyMutationResult struct { - Hostname string `json:"hostname"` - Status string `json:"status"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -type SSHKeyAddOpts struct { - Key string -} -``` - -Add conversion functions. - -- [ ] **Step 2: Add methods to UserService** - -In `user.go`, add: - -```go -func (s *UserService) ListKeys( - ctx context.Context, - hostname string, - username string, -) (*Response[Collection[SSHKeyInfoResult]], error) - -func (s *UserService) AddKey( - ctx context.Context, - hostname string, - username string, - opts SSHKeyAddOpts, -) (*Response[Collection[SSHKeyMutationResult]], error) - -func (s *UserService) RemoveKey( - ctx context.Context, - hostname string, - username string, - fingerprint string, -) (*Response[Collection[SSHKeyMutationResult]], error) -``` - -- [ ] **Step 3: Regenerate SDK client** - -```bash -go generate ./pkg/sdk/client/gen/... -``` - -- [ ] **Step 4: Write tests** - -Add tests to existing test files (or create new `user_ssh_key_public_test.go` / -`user_ssh_key_types_public_test.go` if the existing files are already large). -Follow existing patterns with httptest.Server. - -- [ ] **Step 5: Verify 100% coverage** - -```bash -go test -coverprofile=/tmp/ssh_sdk.cov \ - ./pkg/sdk/client/... && \ - go tool cover -func=/tmp/ssh_sdk.cov | \ - grep "user" | grep -v "100.0%" -``` - -- [ ] **Step 6: Commit** - -```bash -git add pkg/sdk/client/ -git commit -m "feat(user): add SSH key SDK methods with tests" -``` - ---- - -### Task 7: CLI Commands - -**Files:** - -- Create: `cmd/client_node_user_ssh_key.go` -- Create: `cmd/client_node_user_ssh_key_list.go` -- Create: `cmd/client_node_user_ssh_key_add.go` -- Create: `cmd/client_node_user_ssh_key_remove.go` - -- [ ] **Step 1: Create parent command** - -```go -var clientNodeUserSshKeyCmd = &cobra.Command{ - Use: "ssh-key", - Short: "Manage SSH authorized keys", -} - -func init() { - clientNodeUserCmd.AddCommand(clientNodeUserSshKeyCmd) -} -``` - -Wait — check whether `clientNodeUserCmd` exists. Look at -`cmd/client_node_user.go` for the parent. - -- [ ] **Step 2: Create list subcommand** - -Flags: `--name` (username, required) - -- Calls `sdkClient.User.ListKeys(ctx, host, name)` -- Table headers: `TYPE`, `FINGERPRINT`, `COMMENT` -- Uses `BuildBroadcastTable` - -- [ ] **Step 3: Create add subcommand** - -Flags: `--name` (required), `--key` (required, full public key line) - -- Calls `sdkClient.User.AddKey(ctx, host, name, opts)` -- Uses `BuildMutationTable` with headers `CHANGED` - -- [ ] **Step 4: Create remove subcommand** - -Flags: `--name` (required), `--fingerprint` (required) - -- Calls `sdkClient.User.RemoveKey(ctx, host, name, fp)` -- Uses `BuildMutationTable` - -- [ ] **Step 5: Verify build** - -```bash -go build ./... -``` - -- [ ] **Step 6: Commit** - -```bash -git add cmd/client_node_user_ssh_key*.go -git commit -m "feat(user): add SSH key CLI commands" -``` - ---- - -### Task 8: Documentation - -**Files:** - -- Modify: `docs/docs/sidebar/features/user-management.md` -- Create: CLI doc pages for ssh-key commands -- Modify: `docs/docs/sidebar/sdk/client/management/user.md` -- Modify: `examples/sdk/client/user.go` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` - -- [ ] **Step 1: Update feature page** - -Add SSH Key Management section to -`docs/docs/sidebar/features/user-management.md`: - -- How It Works (list, add, remove) -- Add to Operations table -- Add CLI examples for ssh-key subcommands -- Note: uses existing `user:read`/`user:write` permissions - -- [ ] **Step 2: Create CLI doc pages** - -Create landing page + list.md, add.md, remove.md under -`docs/docs/sidebar/usage/cli/client/node/user/`. - -- [ ] **Step 3: Update SDK doc** - -Add ListKeys, AddKey, RemoveKey to the user SDK doc page with code examples and -result type tables. - -- [ ] **Step 4: Update SDK example** - -Add SSH key demo to `examples/sdk/client/user.go`. - -- [ ] **Step 5: Update api-guidelines** - -Add endpoint rows: - -``` -| `/node/{hostname}/user/{name}/ssh-key` | User | -| `/node/{hostname}/user/{name}/ssh-key/{fingerprint}` | User | -``` - -- [ ] **Step 6: Commit** - -```bash -git add docs/ examples/ -git commit -m "docs: add SSH key management to user docs and SDK example" -``` - ---- - -### Task 9: Integration Test and Final Verification - -**Files:** - -- Modify or create: `test/integration/user_test.go` (add SSH key tests) - -- [ ] **Step 1: Add integration test** - -Add SSH key list test to the existing user integration test file (or create new -if it doesn't exist). Test: - -- `osapi client node user ssh-key list --target _any --name root --json` - -- [ ] **Step 2: Run full suite** - -```bash -just generate -go build ./... -just go::unit -just go::vet -``` - -- [ ] **Step 3: Verify coverage** - -```bash -go test -coverprofile=/tmp/ssh_all.cov \ - ./internal/provider/node/user/... \ - ./internal/agent/... \ - ./internal/controller/api/node/user/... \ - ./pkg/sdk/client/... -go tool cover -func=/tmp/ssh_all.cov | \ - grep "ssh_key\|ssh_key" | grep -v "100.0%" | \ - grep -v "mocks\|gen/" -``` - -- [ ] **Step 4: Commit any fixes** - -```bash -git add -A -git commit -m "chore(user): fix formatting and lint" -``` diff --git a/docs/plans/2026-04-02-agent-privilege-escalation-design.md b/docs/plans/2026-04-02-agent-privilege-escalation-design.md deleted file mode 100644 index 4f3a1f589..000000000 --- a/docs/plans/2026-04-02-agent-privilege-escalation-design.md +++ /dev/null @@ -1,281 +0,0 @@ -# Agent Privilege Escalation Design - -Run the OSAPI agent as an unprivileged user with config-driven sudo escalation -for write operations, Linux capabilities for direct file access, and preflight -verification at startup. - -## Problem - -The agent runs as root by default. This grants full system access to a -network-facing process that accepts jobs from NATS. A compromised agent (or a -malicious job) has unrestricted access to the host. The guiding principles call -for least-privilege mode. - -## Solution - -Split command execution into read and write paths. Reads run unprivileged. -Writes run through `sudo` when configured. The agent verifies its privileges at -startup and refuses to start if the configuration doesn't match the system -state. - -## Config - -```yaml -agent: - privilege_escalation: - sudo: true - capabilities: true - preflight: true -``` - -| Field | Type | Default | Description | -| -------------- | ---- | ------- | ------------------------------------------ | -| `sudo` | bool | false | Prepend `sudo` to write commands | -| `capabilities` | bool | false | Verify Linux capabilities at startup | -| `preflight` | bool | false | Run privilege checks before accepting jobs | - -When all fields are false (or the section is absent), the agent behaves as -before — commands run as the current user. - -## Exec Manager Interface - -Add `RunPrivilegedCmd` to the `Manager` interface: - -```go -type Manager interface { - RunCmd(name string, args []string) (string, error) - RunPrivilegedCmd(name string, args []string) (string, error) - RunCmdFull(name string, args []string, cwd string, timeout int) (*CmdResult, error) -} -``` - -The `Exec` struct gains a `sudo bool` field: - -```go -func (e *Exec) RunPrivilegedCmd( - name string, - args []string, -) (string, error) { - if e.sudo { - args = append([]string{name}, args...) - name = "sudo" - } - return e.RunCmdImpl(name, args, "") -} -``` - -When `sudo` is false, `RunPrivilegedCmd` is identical to `RunCmd`. - -## Provider Changes - -Every provider write operation changes from `RunCmd` to `RunPrivilegedCmd`. Read -operations stay on `RunCmd`. The providers themselves don't know or care whether -sudo is enabled — the exec manager handles it. - -```go -// Read — always unprivileged -output, _ := d.execManager.RunCmd("systemctl", []string{"is-active", name}) - -// Write — elevated when configured -_, err := d.execManager.RunPrivilegedCmd("systemctl", []string{"start", name}) -``` - -Tests enforce this: the mock `Manager` has both methods. If a write operation -calls `RunCmd` instead of `RunPrivilegedCmd`, the mock expectation fails. - -## Command Classification - -### Write operations (use `RunPrivilegedCmd`) - -| Command | Domain | -| -------------------------- | ----------- | -| `systemctl start/stop/…` | Service | -| `systemctl daemon-reload` | Service | -| `sysctl -p`, `--system` | Sysctl | -| `timedatectl set-timezone` | Timezone | -| `hostnamectl set-hostname` | Hostname | -| `chronyc reload sources` | NTP | -| `useradd`, `usermod` | User | -| `userdel -r` | User | -| `groupadd`, `groupdel` | Group | -| `gpasswd -M` | Group | -| `chown -R` | SSH Key | -| `apt-get install/remove` | Package | -| `apt-get update` | Package | -| `update-ca-certificates` | Certificate | -| `shutdown -r/-h` | Power | -| `sh -c "echo … chpasswd"` | User | - -### Read operations (use `RunCmd`) - -| Command | Domain | -| --------------------------- | -------- | -| `systemctl list-units` | Service | -| `systemctl list-unit-files` | Service | -| `systemctl show` | Service | -| `systemctl is-active` | Service | -| `systemctl is-enabled` | Service | -| `sysctl -n` | Sysctl | -| `timedatectl show` | Timezone | -| `hostnamectl hostname` | Hostname | -| `journalctl` | Log | -| `chronyc tracking` | NTP | -| `chronyc sources -c` | NTP | -| `id -Gn` | User | -| `passwd -S` | User | -| `dpkg-query` | Package | -| `apt list --upgradable` | Package | -| `date +%:z` | Timezone | - -## Preflight Checks - -Run during `agent start` before the agent subscribes to NATS. Checks are -sequential: sudo first, then capabilities. If any check fails, the agent logs -the failure and exits with a non-zero status. - -### Sudo verification - -For each write command, run `sudo -n --version` (or equivalent no-op -flag). The `-n` flag makes sudo fail immediately if a password would be -required. If the command doesn't support `--version`, use -`sudo -n which ` as a fallback. - -### Capability verification - -Read `/proc/self/status`, parse the `CapEff` hexadecimal bitmask, and check that -required capability bits are set: - -| Capability | Bit | Purpose | -| --------------------- | --- | ------------------------------- | -| `CAP_DAC_READ_SEARCH` | 2 | Read restricted files | -| `CAP_DAC_OVERRIDE` | 1 | Write files regardless of owner | -| `CAP_FOWNER` | 3 | Change file ownership | -| `CAP_KILL` | 5 | Signal any process | - -### Output format - -``` -OSAPI Agent Preflight Check -───────────────────────────── -Sudo access: - ✓ systemctl ✓ sysctl ✓ timedatectl - ✓ hostnamectl ✓ chronyc ✓ useradd - ✓ usermod ✓ userdel ✓ groupadd - ✓ groupdel ✓ gpasswd ✓ chown - ✓ apt-get ✓ shutdown ✓ update-ca-certificates - ✗ sh (sudoers rule missing) - -Capabilities: - ✓ CAP_DAC_READ_SEARCH ✓ CAP_DAC_OVERRIDE - ✓ CAP_FOWNER ✓ CAP_KILL - -Result: FAILED (1 error) - - sudo: sh not configured in /etc/sudoers.d/osapi-agent -``` - -## Deployment Artifacts - -### Sudoers drop-in (`/etc/sudoers.d/osapi-agent`) - -```sudoers -# Service management -osapi ALL=(root) NOPASSWD: /usr/bin/systemctl start * -osapi ALL=(root) NOPASSWD: /usr/bin/systemctl stop * -osapi ALL=(root) NOPASSWD: /usr/bin/systemctl restart * -osapi ALL=(root) NOPASSWD: /usr/bin/systemctl enable * -osapi ALL=(root) NOPASSWD: /usr/bin/systemctl disable * -osapi ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload - -# Kernel parameters -osapi ALL=(root) NOPASSWD: /usr/sbin/sysctl -p * -osapi ALL=(root) NOPASSWD: /usr/sbin/sysctl --system - -# Timezone -osapi ALL=(root) NOPASSWD: /usr/bin/timedatectl set-timezone * - -# Hostname -osapi ALL=(root) NOPASSWD: /usr/bin/hostnamectl set-hostname * - -# NTP -osapi ALL=(root) NOPASSWD: /usr/bin/chronyc reload sources - -# User and group management -osapi ALL=(root) NOPASSWD: /usr/sbin/useradd * -osapi ALL=(root) NOPASSWD: /usr/sbin/usermod * -osapi ALL=(root) NOPASSWD: /usr/sbin/userdel * -osapi ALL=(root) NOPASSWD: /usr/sbin/groupadd * -osapi ALL=(root) NOPASSWD: /usr/sbin/groupdel * -osapi ALL=(root) NOPASSWD: /usr/bin/gpasswd * -osapi ALL=(root) NOPASSWD: /usr/bin/chown * -osapi ALL=(root) NOPASSWD: /bin/sh -c echo * - -# Package management -osapi ALL=(root) NOPASSWD: /usr/bin/apt-get install * -osapi ALL=(root) NOPASSWD: /usr/bin/apt-get remove * -osapi ALL=(root) NOPASSWD: /usr/bin/apt-get update - -# Certificate trust store -osapi ALL=(root) NOPASSWD: /usr/sbin/update-ca-certificates - -# Power management -osapi ALL=(root) NOPASSWD: /sbin/shutdown * -``` - -### Linux capabilities - -```bash -sudo setcap \ - 'cap_dac_read_search+ep cap_dac_override+ep cap_fowner+ep cap_kill+ep' \ - /usr/local/bin/osapi -``` - -### Systemd unit file - -```ini -[Unit] -Description=OSAPI Agent -After=network.target - -[Service] -Type=simple -User=osapi -Group=osapi -ExecStart=/usr/local/bin/osapi agent start -Restart=always -RestartSec=5 -AmbientCapabilities=CAP_DAC_READ_SEARCH CAP_DAC_OVERRIDE CAP_FOWNER CAP_KILL -CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_DAC_OVERRIDE CAP_FOWNER CAP_KILL -SecureBits=keep-caps -NoNewPrivileges=no -PrivateTmp=true - -[Install] -WantedBy=multi-user.target -``` - -## Not Changing - -- Controller and NATS server — already run unprivileged, no changes needed -- The `command exec` and `command shell` endpoints — these execute arbitrary - user-provided commands, so they inherit whatever privileges the agent has. - They are gated by the `command:execute` permission in RBAC. -- Docker provider — talks to the Docker API socket, not system commands. The - `osapi` user needs to be in the `docker` group. - -## Files Changed - -- `internal/config/types.go` — add `PrivilegeEscalation` struct -- `internal/exec/manager.go` — add `RunPrivilegedCmd` to interface -- `internal/exec/types.go` — add `sudo bool` to `Exec` struct -- `internal/exec/run_privileged_cmd.go` — new file, implementation -- `internal/exec/run_privileged_cmd_public_test.go` — tests -- `internal/exec/mocks/` — regenerate -- `internal/agent/preflight.go` — new file, sudo + caps verification -- `internal/agent/preflight_public_test.go` — tests -- `internal/agent/agent.go` — call preflight during `Start()` -- `cmd/agent_setup.go` — pass `sudo` bool to exec manager -- Every provider `debian*.go` file — change write `RunCmd` to `RunPrivilegedCmd` - (~37 call sites) -- Every provider `debian*_public_test.go` — update mock expectations -- `docs/docs/sidebar/usage/configuration.md` — add config reference -- `docs/docs/sidebar/features/` — add agent hardening feature page diff --git a/docs/plans/2026-04-02-agent-privilege-escalation.md b/docs/plans/2026-04-02-agent-privilege-escalation.md deleted file mode 100644 index a3b2f7867..000000000 --- a/docs/plans/2026-04-02-agent-privilege-escalation.md +++ /dev/null @@ -1,989 +0,0 @@ -# Agent Privilege Escalation Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add config-driven sudo escalation for write commands, Linux capability -verification, and startup preflight checks to the OSAPI agent so it can run as -an unprivileged user. - -**Architecture:** The exec `Manager` interface gains `RunPrivilegedCmd` which -prepends `sudo` when configured. Providers call it for write operations. At -startup, the agent runs preflight checks to verify sudo and capabilities before -accepting jobs. - -**Tech Stack:** Go, Linux capabilities (`/proc/self/status`), sudo - ---- - -### Task 1: Add privilege escalation config - -**Files:** - -- Modify: `internal/config/types.go:353-374` -- Modify: `configs/osapi.yaml` -- Modify: `configs/osapi.nerd.yaml` -- Modify: `configs/osapi.local.yaml` - -- [ ] **Step 1: Add PrivilegeEscalation struct and wire into AgentConfig** - -In `internal/config/types.go`, add a new struct before `AgentConfig`: - -```go -// PrivilegeEscalation configuration for least-privilege agent mode. -type PrivilegeEscalation struct { - // Sudo prepends "sudo" to write commands when true. - Sudo bool `mapstructure:"sudo"` - // Capabilities verifies Linux capabilities at startup when true. - Capabilities bool `mapstructure:"capabilities"` - // Preflight runs privilege checks before accepting jobs when true. - Preflight bool `mapstructure:"preflight"` -} -``` - -Add the field to `AgentConfig`: - -```go -type AgentConfig struct { - // ... existing fields ... - // PrivilegeEscalation configures least-privilege agent mode. - PrivilegeEscalation PrivilegeEscalation `mapstructure:"privilege_escalation,omitempty"` -} -``` - -- [ ] **Step 2: Add config to YAML files** - -In `configs/osapi.yaml`, `configs/osapi.nerd.yaml`, and -`configs/osapi.local.yaml`, add to the `agent:` section (disabled by default): - -```yaml -# Least-privilege mode. When enabled, the agent runs as an -# unprivileged user and uses sudo for write operations. -# privilege_escalation: -# sudo: false -# capabilities: false -# preflight: false -``` - -- [ ] **Step 3: Verify build** - -Run: `go build ./...` - -- [ ] **Step 4: Commit** - -``` -feat(agent): add privilege escalation config -``` - ---- - -### Task 2: Add RunPrivilegedCmd to exec manager - -**Files:** - -- Modify: `internal/exec/manager.go` -- Modify: `internal/exec/types.go` -- Modify: `internal/exec/exec.go` -- Create: `internal/exec/run_privileged_cmd.go` -- Create: `internal/exec/run_privileged_cmd_public_test.go` -- Modify: `internal/exec/mocks/generate.go` - -- [ ] **Step 1: Add RunPrivilegedCmd to the Manager interface** - -In `internal/exec/manager.go`: - -```go -type Manager interface { - // RunCmd executes the provided command with arguments, using the - // current working directory. Use for read operations. - RunCmd( - name string, - args []string, - ) (string, error) - - // RunPrivilegedCmd executes a command with privilege escalation. - // When sudo is enabled in config, prepends "sudo" to the command. - // When sudo is disabled, behaves identically to RunCmd. - // Use for write operations that modify system state. - RunPrivilegedCmd( - name string, - args []string, - ) (string, error) - - // RunCmdFull executes a command with separate stdout/stderr capture, - // an optional working directory, and a timeout in seconds. - RunCmdFull( - name string, - args []string, - cwd string, - timeout int, - ) (*CmdResult, error) -} -``` - -- [ ] **Step 2: Add sudo field to Exec struct and update constructor** - -In `internal/exec/types.go`, add the `sudo` field: - -```go -type Exec struct { - logger *slog.Logger - sudo bool -} -``` - -In `internal/exec/exec.go`, update the constructor: - -```go -func New( - logger *slog.Logger, - sudo bool, -) *Exec { - return &Exec{ - logger: logger.With(slog.String("subsystem", "exec")), - sudo: sudo, - } -} -``` - -- [ ] **Step 3: Create RunPrivilegedCmd implementation** - -Create `internal/exec/run_privileged_cmd.go`: - -```go -package exec - -// RunPrivilegedCmd executes a command with privilege escalation. -// When sudo is enabled, the command is run via "sudo". When disabled, -// it behaves identically to RunCmd. -func (e *Exec) RunPrivilegedCmd( - name string, - args []string, -) (string, error) { - if e.sudo { - args = append([]string{name}, args...) - name = "sudo" - } - - return e.RunCmdImpl(name, args, "") -} -``` - -- [ ] **Step 4: Write tests** - -Create `internal/exec/run_privileged_cmd_public_test.go`. Test: - -- When sudo is false, RunPrivilegedCmd behaves like RunCmd (runs the command - directly) -- When sudo is true, the command is prefixed with sudo (verify the actual args - passed to the underlying exec) - -Since `RunCmdImpl` calls `os/exec.Command`, use a test helper pattern: override -the command execution to capture what would be run. Look at how -`run_cmd_public_test.go` tests `RunCmd` and follow the same pattern. - -- [ ] **Step 5: Update agent_setup.go to pass sudo config** - -In `cmd/agent_setup.go`, change the `exec.New` call: - -```go -// Before: -execManager := exec.New(log) - -// After: -execManager := exec.New( - log, - appConfig.Agent.PrivilegeEscalation.Sudo, -) -``` - -- [ ] **Step 6: Regenerate mocks** - -Run: `go generate ./internal/exec/mocks/...` - -- [ ] **Step 7: Fix compilation errors** - -The mock `Manager` now requires `RunPrivilegedCmd`. Any existing test that -constructs a mock `Manager` may need updating. Run `go build ./...` and fix any -compile errors. - -- [ ] **Step 8: Run tests** - -Run: `go test ./internal/exec/... -count=1` Run: `go build ./...` - -- [ ] **Step 9: Commit** - -``` -feat(exec): add RunPrivilegedCmd with config-driven sudo -``` - ---- - -### Task 3: Add preflight checks - -**Files:** - -- Create: `internal/agent/preflight.go` -- Create: `internal/agent/preflight_public_test.go` -- Modify: `internal/agent/server.go` - -- [ ] **Step 1: Create preflight types and runner** - -Create `internal/agent/preflight.go`: - -```go -package agent - -import ( - "bufio" - "encoding/hex" - "fmt" - "log/slog" - "os" - "strings" - - "github.com/osapi-io/osapi/internal/exec" -) - -// PreflightResult holds the outcome of a single preflight check. -type PreflightResult struct { - Name string - Passed bool - Error string -} - -// sudoCommands lists the binaries that require sudo access. -var sudoCommands = []string{ - "systemctl", - "sysctl", - "timedatectl", - "hostnamectl", - "chronyc", - "useradd", - "usermod", - "userdel", - "groupadd", - "groupdel", - "gpasswd", - "chown", - "apt-get", - "shutdown", - "update-ca-certificates", - "sh", -} - -// requiredCapabilities maps capability names to their bit positions -// in the CapEff bitmask from /proc/self/status. -var requiredCapabilities = map[string]uint{ - "CAP_DAC_OVERRIDE": 1, - "CAP_DAC_READ_SEARCH": 2, - "CAP_FOWNER": 3, - "CAP_KILL": 5, -} - -// procStatusPath is the path to read for capability detection. -// Overridden in tests. -var procStatusPath = "/proc/self/status" - -// RunPreflight checks sudo access and capabilities. Returns all -// results and whether all checks passed. -func RunPreflight( - logger *slog.Logger, - execManager exec.Manager, - checkSudo bool, - checkCaps bool, -) ([]PreflightResult, bool) { - var results []PreflightResult - allPassed := true - - if checkSudo { - sudoResults := checkSudoAccess(logger, execManager) - results = append(results, sudoResults...) - for _, r := range sudoResults { - if !r.Passed { - allPassed = false - } - } - } - - if checkCaps { - capResults := checkCapabilities(logger) - results = append(results, capResults...) - for _, r := range capResults { - if !r.Passed { - allPassed = false - } - } - } - - return results, allPassed -} - -// checkSudoAccess verifies that sudo -n works for each required -// command. Uses "sudo -n which " which is a no-op that -// tests sudo access without side effects. -func checkSudoAccess( - logger *slog.Logger, - execManager exec.Manager, -) []PreflightResult { - results := make([]PreflightResult, 0, len(sudoCommands)) - - for _, cmd := range sudoCommands { - _, err := execManager.RunCmd( - "sudo", - []string{"-n", "which", cmd}, - ) - - result := PreflightResult{ - Name: "sudo:" + cmd, - Passed: err == nil, - } - if err != nil { - result.Error = fmt.Sprintf( - "sudo access denied for %s: %s", - cmd, - err.Error(), - ) - } - - logger.Debug( - "preflight sudo check", - slog.String("command", cmd), - slog.Bool("passed", result.Passed), - ) - - results = append(results, result) - } - - return results -} - -// checkCapabilities reads /proc/self/status and verifies that -// required capability bits are set in the effective capability mask. -func checkCapabilities( - logger *slog.Logger, -) []PreflightResult { - capEff, err := readCapEff() - if err != nil { - return []PreflightResult{{ - Name: "capabilities", - Passed: false, - Error: fmt.Sprintf("failed to read capabilities: %s", err), - }} - } - - results := make([]PreflightResult, 0, len(requiredCapabilities)) - - for name, bit := range requiredCapabilities { - hasCap := (capEff>>bit)&1 == 1 - result := PreflightResult{ - Name: "cap:" + name, - Passed: hasCap, - } - if !hasCap { - result.Error = fmt.Sprintf("%s not set", name) - } - - logger.Debug( - "preflight capability check", - slog.String("capability", name), - slog.Bool("passed", hasCap), - ) - - results = append(results, result) - } - - return results -} - -// readCapEff reads the effective capability bitmask from -// /proc/self/status. -func readCapEff() (uint64, error) { - f, err := os.Open(procStatusPath) - if err != nil { - return 0, fmt.Errorf("open %s: %w", procStatusPath, err) - } - defer func() { _ = f.Close() }() - - scanner := bufio.NewScanner(f) - for scanner.Scan() { - line := scanner.Text() - if strings.HasPrefix(line, "CapEff:") { - hexStr := strings.TrimSpace( - strings.TrimPrefix(line, "CapEff:"), - ) - bytes, err := hex.DecodeString(hexStr) - if err != nil { - return 0, fmt.Errorf( - "decode CapEff %q: %w", - hexStr, - err, - ) - } - // Convert big-endian bytes to uint64. - var val uint64 - for _, b := range bytes { - val = (val << 8) | uint64(b) - } - return val, nil - } - } - - return 0, fmt.Errorf("CapEff not found in %s", procStatusPath) -} -``` - -- [ ] **Step 2: Write preflight tests** - -Create `internal/agent/preflight_public_test.go`. Use testify/suite with -table-driven tests. Test: - -**TestCheckSudoAccess:** - -- All commands pass (mock RunCmd returns nil for all sudo -n which calls) -- One command fails (mock RunCmd returns error for one) -- Multiple commands fail - -**TestCheckCapabilities:** - -- All capabilities present (write a fake /proc/self/status file with full - CapEff, override `procStatusPath` via export_test.go) -- Missing capability (write CapEff without a required bit) -- Cannot read file (set procStatusPath to nonexistent path) - -**TestRunPreflight:** - -- Both sudo and caps enabled and pass → allPassed true -- Sudo fails → allPassed false -- Caps fails → allPassed false -- Both disabled → empty results, allPassed true - -Create `internal/agent/export_test.go` (or add to existing) to expose -`procStatusPath` for testing: - -```go -package agent - -func SetProcStatusPath(p string) { procStatusPath = p } -func ResetProcStatusPath() { procStatusPath = "/proc/self/status" } -``` - -- [ ] **Step 3: Wire preflight into agent Start()** - -In `internal/agent/server.go`, add preflight check after hostname determination -but before starting heartbeat: - -```go -func (a *Agent) Start() { - a.ctx, a.cancel = context.WithCancel(context.Background()) - a.startedAt = time.Now() - a.state = job.AgentStateReady - - a.logger.Info("starting node agent") - - a.hostname, _ = job.GetAgentHostname(a.appConfig.Agent.Hostname) - - // Run preflight checks if configured. - pe := a.appConfig.Agent.PrivilegeEscalation - if pe.Preflight { - results, ok := RunPreflight( - a.logger, - a.execManager, - pe.Sudo, - pe.Capabilities, - ) - if !ok { - for _, r := range results { - if !r.Passed { - a.logger.Error( - "preflight check failed", - slog.String("check", r.Name), - slog.String("error", r.Error), - ) - } - } - a.logger.Error("preflight failed, agent cannot start") - a.cancel() - return - } - a.logger.Info("preflight checks passed") - } - - // ... rest of Start() unchanged ... -``` - -The agent needs access to the exec manager. Check if it's already on the `Agent` -struct. If not, add an `execManager exec.Manager` field and pass it from -`agent_setup.go`. - -- [ ] **Step 4: Run tests** - -Run: `go test ./internal/agent/... -count=1` Run: `go build ./...` - -- [ ] **Step 5: Commit** - -``` -feat(agent): add preflight checks for sudo and capabilities -``` - ---- - -### Task 4: Migrate service provider to RunPrivilegedCmd - -**Files:** - -- Modify: `internal/provider/node/service/debian_action.go` -- Modify: `internal/provider/node/service/debian_unit.go` -- Modify: `internal/provider/node/service/debian_action_public_test.go` -- Modify: `internal/provider/node/service/debian_unit_public_test.go` - -- [ ] **Step 1: Update write calls in debian_action.go** - -Change these `RunCmd` calls to `RunPrivilegedCmd`: - -```go -// Start — line 49: "systemctl start" is a write -d.execManager.RunPrivilegedCmd("systemctl", []string{"start", unitName}) - -// Stop — "systemctl stop" is a write -d.execManager.RunPrivilegedCmd("systemctl", []string{"stop", unitName}) - -// Restart — "systemctl restart" is a write -d.execManager.RunPrivilegedCmd("systemctl", []string{"restart", unitName}) - -// Enable — "systemctl enable" is a write -d.execManager.RunPrivilegedCmd("systemctl", []string{"enable", unitName}) - -// Disable — "systemctl disable" is a write -d.execManager.RunPrivilegedCmd("systemctl", []string{"disable", unitName}) -``` - -Keep these as `RunCmd` (reads): - -- `systemctl is-active` (Start, Stop) -- `systemctl is-enabled` (Enable, Disable) - -- [ ] **Step 2: Update write calls in debian_unit.go** - -```go -// Delete — "systemctl stop" and "systemctl disable" are writes -d.execManager.RunPrivilegedCmd("systemctl", []string{"stop", unitName}) -d.execManager.RunPrivilegedCmd("systemctl", []string{"disable", unitName}) - -// daemonReload — "systemctl daemon-reload" is a write -d.execManager.RunPrivilegedCmd("systemctl", []string{"daemon-reload"}) -``` - -- [ ] **Step 3: Update test mock expectations** - -In `debian_action_public_test.go`, change all mock expectations for write -commands from `RunCmd` to `RunPrivilegedCmd`. Keep read command expectations on -`RunCmd`. - -In `debian_unit_public_test.go`, change mock expectations for `systemctl stop`, -`systemctl disable`, `systemctl daemon-reload` from `RunCmd` to -`RunPrivilegedCmd`. - -- [ ] **Step 4: Run tests** - -Run: `go test ./internal/provider/node/service/... -count=1` - -- [ ] **Step 5: Commit** - -``` -refactor(service): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 5: Migrate sysctl provider - -**Files:** - -- Modify: `internal/provider/node/sysctl/debian.go` -- Modify: `internal/provider/node/sysctl/debian_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `sysctl -p ` (apply config) -- `sysctl --system` (reload all) - -Keep as `RunCmd`: - -- `sysctl -n ` (read parameter) - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/sysctl/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(sysctl): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 6: Migrate hostname provider - -**Files:** - -- Modify: `internal/provider/node/host/debian_update_hostname.go` -- Modify: `internal/provider/node/host/debian_update_hostname_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `hostnamectl set-hostname ` - -Keep as `RunCmd` (in other host files): - -- `hostnamectl hostname` (read) - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/host/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(host): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 7: Migrate timezone provider - -**Files:** - -- Modify: `internal/provider/node/timezone/debian.go` -- Modify: `internal/provider/node/timezone/debian_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `timedatectl set-timezone ` - -Keep as `RunCmd`: - -- `timedatectl show -p Timezone --value` (read) -- `date +%:z` (read) - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/timezone/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(timezone): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 8: Migrate NTP provider - -**Files:** - -- Modify: `internal/provider/node/ntp/debian.go` -- Modify: `internal/provider/node/ntp/debian_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `chronyc reload sources` - -Keep as `RunCmd`: - -- `chronyc tracking` (read) -- `chronyc sources -c` (read) - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/ntp/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(ntp): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 9: Migrate user provider - -**Files:** - -- Modify: `internal/provider/node/user/debian_user.go` -- Modify: `internal/provider/node/user/debian_group.go` -- Modify: `internal/provider/node/user/debian_ssh_key.go` -- Modify: `internal/provider/node/user/debian_user_public_test.go` -- Modify: `internal/provider/node/user/debian_group_public_test.go` -- Modify: `internal/provider/node/user/debian_ssh_key_public_test.go` - -- [ ] **Step 1: Update write calls in debian_user.go** - -Change to `RunPrivilegedCmd`: - -- `useradd --create-home ...` -- `usermod ...` (all variants) -- `userdel -r ...` -- `sh -c "echo ... | chpasswd"` - -Keep as `RunCmd`: - -- `id -Gn ` (read) -- `passwd -S ` (read) - -- [ ] **Step 2: Update write calls in debian_group.go** - -Change to `RunPrivilegedCmd`: - -- `groupadd ...` -- `groupdel ...` -- `gpasswd -M ...` - -- [ ] **Step 3: Update write calls in debian_ssh_key.go** - -Change to `RunPrivilegedCmd`: - -- `chown -R ...` - -- [ ] **Step 4: Update all test mock expectations** - -- [ ] **Step 5: Run tests** - -Run: `go test ./internal/provider/node/user/... -count=1` - -- [ ] **Step 6: Commit** - -``` -refactor(user): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 10: Migrate package provider - -**Files:** - -- Modify: `internal/provider/node/apt/debian.go` -- Modify: `internal/provider/node/apt/debian_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `apt-get install -y ...` -- `apt-get remove -y ...` -- `apt-get update` - -Keep as `RunCmd`: - -- `dpkg-query ...` (read) -- `apt list --upgradable` (read) - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/apt/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(apt): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 11: Migrate power provider - -**Files:** - -- Modify: `internal/provider/node/power/debian.go` -- Modify: `internal/provider/node/power/debian_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `shutdown -r ...` (reboot) -- `shutdown -h ...` (halt) - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/power/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(power): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 12: Migrate certificate provider - -**Files:** - -- Modify: `internal/provider/node/certificate/debian.go` -- Modify: `internal/provider/node/certificate/debian_public_test.go` - -- [ ] **Step 1: Update write calls** - -Change to `RunPrivilegedCmd`: - -- `update-ca-certificates` - -- [ ] **Step 2: Update test mock expectations** - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/node/certificate/... -count=1` - -- [ ] **Step 4: Commit** - -``` -refactor(certificate): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 13: Migrate DNS provider - -**Files:** - -- Modify: - `internal/provider/network/dns/debian_update_resolv_conf_by_interface.go` (or - equivalent write file) -- Modify: corresponding test file - -- [ ] **Step 1: Check if DNS uses exec for writes** - -Read the DNS provider to determine if it executes commands for writes. The DNS -provider may use `resolvectl` or write files directly. If it uses `RunCmd` for -writes, change to `RunPrivilegedCmd`. If it only writes files via `avfs`, no -changes needed (file writes are covered by capabilities). - -- [ ] **Step 2: Update if needed and run tests** - -Run: `go test ./internal/provider/network/dns/... -count=1` - -- [ ] **Step 3: Commit (if changes made)** - -``` -refactor(dns): use RunPrivilegedCmd for write operations -``` - ---- - -### Task 14: Update documentation - -**Files:** - -- Modify: `docs/docs/sidebar/usage/configuration.md` -- Create: `docs/docs/sidebar/features/agent-hardening.md` - -- [ ] **Step 1: Add config reference to configuration.md** - -In the agent section of configuration.md, add: - -```yaml -# Least-privilege mode for running the agent as an unprivileged user. -privilege_escalation: - # Prepend "sudo" to write commands. - sudo: false - # Verify Linux capabilities at startup. - capabilities: false - # Run privilege checks before accepting jobs. - preflight: false -``` - -Add the environment variable mappings: - -| Config Key | Environment Variable | -| ----------------------------------------- | ----------------------------------------------- | -| `agent.privilege_escalation.sudo` | `OSAPI_AGENT_PRIVILEGE_ESCALATION_SUDO` | -| `agent.privilege_escalation.capabilities` | `OSAPI_AGENT_PRIVILEGE_ESCALATION_CAPABILITIES` | -| `agent.privilege_escalation.preflight` | `OSAPI_AGENT_PRIVILEGE_ESCALATION_PREFLIGHT` | - -- [ ] **Step 2: Create agent hardening feature page** - -Create `docs/docs/sidebar/features/agent-hardening.md` with: - -- Overview of least-privilege mode -- Configuration reference -- Sudoers drop-in (copy from spec) -- Capabilities setup (copy from spec) -- Systemd unit file (copy from spec) -- Preflight output example -- Command privilege reference tables (copy from spec) - -- [ ] **Step 3: Update docusaurus.config.ts** - -Add "Agent Hardening" to the Features navbar dropdown. - -- [ ] **Step 4: Commit** - -``` -docs: add agent hardening feature page and config reference -``` - ---- - -### Task 15: Final verification - -- [ ] **Step 1: Run full test suite** - -```bash -just go::unit -``` - -All tests must pass. - -- [ ] **Step 2: Build and lint** - -```bash -go build ./... -just go::vet -``` - -- [ ] **Step 3: Verify no stray RunCmd calls for write operations** - -Check each write command from the spec against the codebase to confirm it uses -`RunPrivilegedCmd`: - -```bash -grep -rn 'RunCmd.*"systemctl".*"start\|stop\|restart\|enable\|disable\|daemon-reload"' \ - internal/provider/ --include="*.go" | grep -v _test.go | grep -v RunPrivilegedCmd -``` - -Repeat for other write commands (`useradd`, `usermod`, `apt-get`, `shutdown`, -`sysctl -p`, etc.). Expect: no matches. - -- [ ] **Step 4: Verify read commands stayed on RunCmd** - -```bash -grep -rn 'RunPrivilegedCmd.*"systemctl".*"list-units\|list-unit-files\|show\|is-active\|is-enabled"' \ - internal/provider/ --include="*.go" | grep -v _test.go -``` - -Expect: no matches (reads should not use RunPrivilegedCmd). diff --git a/docs/plans/2026-04-03-network-interface-route.md b/docs/plans/2026-04-03-network-interface-route.md deleted file mode 100644 index 95f5873ae..000000000 --- a/docs/plans/2026-04-03-network-interface-route.md +++ /dev/null @@ -1,682 +0,0 @@ -# Network Interface & Route Management Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use -> superpowers:subagent-driven-development (recommended) or -> superpowers:executing-plans to implement this plan task-by-task. Steps use -> checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add full CRUD for network interface configuration and route management -via Netplan drop-in files, following the direct-write provider pattern -established by sysctl and DNS. - -**Architecture:** A new `netplan` provider in -`internal/provider/network/netplan/` handles interface and route CRUD. It -generates Netplan YAML, writes drop-in files with `osapi-` prefix, validates -with `netplan generate`, and applies with `netplan apply`. List/get for -interfaces reuses the existing `netinfo` provider for system state. The shared -`netplan.ApplyConfig` helper (from DNS migration) handles the write → validate → -apply flow. - -**Tech Stack:** Go, Netplan YAML, NATS JetStream KV, avfs - -**Baseline coverage:** 99.9% — must not regress. - ---- - -### Task 1: Interface and route provider types - -**Files:** - -- Create: `internal/provider/network/netplan/types.go` -- Create: `internal/provider/network/netplan/mocks/generate.go` - -- [ ] **Step 1: Define the provider interfaces and types** - -Create `internal/provider/network/netplan/types.go`: - -```go -package netplan - -import "context" - -// InterfaceProvider manages network interface configuration via Netplan. -type InterfaceProvider interface { - List(ctx context.Context) ([]InterfaceEntry, error) - Get(ctx context.Context, name string) (*InterfaceEntry, error) - Create(ctx context.Context, entry InterfaceEntry) (*InterfaceResult, error) - Update(ctx context.Context, entry InterfaceEntry) (*InterfaceResult, error) - Delete(ctx context.Context, name string) (*InterfaceResult, error) -} - -// RouteProvider manages route configuration via Netplan. -type RouteProvider interface { - List(ctx context.Context) ([]RouteListEntry, error) - Get(ctx context.Context, interfaceName string) (*RouteEntry, error) - Create(ctx context.Context, entry RouteEntry) (*RouteResult, error) - Update(ctx context.Context, entry RouteEntry) (*RouteResult, error) - Delete(ctx context.Context, interfaceName string) (*RouteResult, error) -} - -// InterfaceEntry represents a managed interface configuration. -type InterfaceEntry struct { - Name string `json:"name"` - DHCP4 *bool `json:"dhcp4,omitempty"` - DHCP6 *bool `json:"dhcp6,omitempty"` - Addresses []string `json:"addresses,omitempty"` - Gateway4 string `json:"gateway4,omitempty"` - Gateway6 string `json:"gateway6,omitempty"` - MTU int `json:"mtu,omitempty"` - MACAddress string `json:"mac_address,omitempty"` - WakeOnLAN *bool `json:"wakeonlan,omitempty"` - Managed bool `json:"managed,omitempty"` -} - -// InterfaceResult is the outcome of a create/update/delete. -type InterfaceResult struct { - Name string `json:"name"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} - -// RouteEntry represents managed routes for an interface. -type RouteEntry struct { - Interface string `json:"interface"` - Routes []Route `json:"routes"` -} - -// Route is a single route definition. -type Route struct { - To string `json:"to"` - Via string `json:"via"` - Metric int `json:"metric,omitempty"` -} - -// RouteListEntry is a route from the system routing table. -type RouteListEntry struct { - Destination string `json:"destination"` - Gateway string `json:"gateway"` - Interface string `json:"interface"` - Mask string `json:"mask,omitempty"` - Metric int `json:"metric,omitempty"` - Flags string `json:"flags,omitempty"` -} - -// RouteResult is the outcome of a route create/update/delete. -type RouteResult struct { - Interface string `json:"interface"` - Changed bool `json:"changed"` - Error string `json:"error,omitempty"` -} -``` - -- [ ] **Step 2: Add mock generation** - -Create `internal/provider/network/netplan/mocks/generate.go`: - -```go -package mocks - -//go:generate go tool github.com/golang/mock/mockgen -source=../types.go -destination=provider.gen.go -package=mocks -``` - -Run: `go generate ./internal/provider/network/netplan/mocks/...` - -- [ ] **Step 3: Verify build** - -Run: `go build ./...` - -- [ ] **Step 4: Commit** - -``` -feat(netplan): add interface and route provider types -``` - ---- - -### Task 2: Interface provider implementation - -**Files:** - -- Create: `internal/provider/network/netplan/interface.go` -- Create: `internal/provider/network/netplan/interface_public_test.go` -- Create: `internal/provider/network/netplan/debian.go` -- Create: `internal/provider/network/netplan/darwin.go` -- Create: `internal/provider/network/netplan/linux.go` - -- [ ] **Step 1: Create the Debian interface provider** - -Create `internal/provider/network/netplan/debian.go` with the struct and -constructor: - -```go -type Debian struct { - provider.FactsAware - logger *slog.Logger - fs avfs.VFS - stateKV jetstream.KeyValue - execManager exec.Manager - hostname string - netinfo netinfo.Provider -} - -func NewDebianProvider( - logger *slog.Logger, - fs avfs.VFS, - stateKV jetstream.KeyValue, - execManager exec.Manager, - hostname string, - netinfo netinfo.Provider, -) *Debian -``` - -Compile-time checks for `InterfaceProvider`, `RouteProvider`, and -`provider.FactsSetter`. - -Create `darwin.go` and `linux.go` stubs returning `ErrUnsupported`. - -- [ ] **Step 2: Implement interface CRUD** - -Create `internal/provider/network/netplan/interface.go`: - -**List** — delegates to `netinfo.GetInterfaces()`. For each interface, check if -an `osapi-{name}.yaml` file exists to set `Managed: true`. - -**Get** — delegates to `netinfo.GetInterfaces()`, filters by name. Checks -managed status. - -**Create** — validates name, checks file doesn't exist, generates Netplan YAML, -calls `ApplyConfig`. - -**Update** — validates name, checks file exists, generates Netplan YAML, calls -`ApplyConfig`. - -**Delete** — calls `RemoveConfig`. - -YAML generation function `generateInterfaceYAML(entry InterfaceEntry)`: - -```yaml -network: - version: 2 - ethernets: - eth0: - dhcp4: false - dhcp6: false - addresses: - - 10.0.0.5/24 - gateway4: 10.0.0.1 - mtu: 1500 -``` - -File path: `/etc/netplan/osapi-{name}.yaml` - -- [ ] **Step 3: Write interface tests** - -Create `internal/provider/network/netplan/interface_public_test.go`. Use -testify/suite, table-driven, validateFunc. Use `memfs`, gomock. - -Test each method: List (with managed + unmanaged), Get (found, not found), -Create (success, already exists, generate fails), Update (success, not found), -Delete (success, not found). YAML generation tests for each field combination. - -Target: 100% coverage on `interface.go`. - -- [ ] **Step 4: Run tests** - -Run: `go test ./internal/provider/network/netplan/... -count=1` - -- [ ] **Step 5: Commit** - -``` -feat(netplan): implement interface CRUD with Netplan -``` - ---- - -### Task 3: Route provider implementation - -**Files:** - -- Create: `internal/provider/network/netplan/route.go` -- Create: `internal/provider/network/netplan/route_public_test.go` - -- [ ] **Step 1: Implement route CRUD** - -Create `internal/provider/network/netplan/route.go`: - -**List** — delegates to `netinfo.GetRoutes()`. Converts `RouteResult` to -`RouteListEntry`. - -**Get** — reads the managed route file from state KV or disk for the given -interface. Parses the YAML to extract routes. - -**Create** — validates interface name, checks file doesn't exist, validates no -default route in list, generates YAML, calls `ApplyConfig`. - -**Update** — same as create but file must exist. - -**Delete** — validates no default route in managed routes, calls `RemoveConfig`. - -YAML generation function `generateRouteYAML(entry RouteEntry)`: - -```yaml -network: - version: 2 - ethernets: - eth0: - routes: - - to: 10.1.0.0/16 - via: 10.0.0.1 - metric: 100 -``` - -File path: `/etc/netplan/osapi-{interface}-routes.yaml` - -Default route protection: reject create/update if any route has `To` of -`0.0.0.0/0`, `::/0`, or `default`. - -- [ ] **Step 2: Write route tests** - -Create `internal/provider/network/netplan/route_public_test.go`. - -Test each method. Include default route protection tests (reject `0.0.0.0/0`). -YAML generation tests. - -Target: 100% coverage on `route.go`. - -- [ ] **Step 3: Run tests** - -Run: `go test ./internal/provider/network/netplan/... -count=1` - -- [ ] **Step 4: Commit** - -``` -feat(netplan): implement route CRUD with default route protection -``` - ---- - -### Task 4: Job operations and agent processor - -**Files:** - -- Modify: `pkg/sdk/client/operations.go` -- Modify: `internal/job/types.go` -- Create: `internal/agent/processor_interface.go` -- Create: `internal/agent/processor_route.go` -- Create: `internal/agent/processor_interface_public_test.go` -- Create: `internal/agent/processor_route_public_test.go` -- Modify: `internal/agent/processor_network.go` - -- [ ] **Step 1: Add operation constants** - -In `pkg/sdk/client/operations.go`, add: - -```go -// Network interface operations. -const ( - OpNetworkInterfaceList JobOperation = "interface.list" - OpNetworkInterfaceGet JobOperation = "interface.get" - OpNetworkInterfaceCreate JobOperation = "interface.create" - OpNetworkInterfaceUpdate JobOperation = "interface.update" - OpNetworkInterfaceDelete JobOperation = "interface.delete" -) - -// Network route operations. -const ( - OpNetworkRouteList JobOperation = "route.list" - OpNetworkRouteGet JobOperation = "route.get" - OpNetworkRouteCreate JobOperation = "route.create" - OpNetworkRouteUpdate JobOperation = "route.update" - OpNetworkRouteDelete JobOperation = "route.delete" -) -``` - -Mirror in `internal/job/types.go`. - -- [ ] **Step 2: Create interface processor** - -Create `internal/agent/processor_interface.go` with `processInterfaceOperation` -that dispatches list/get/create/update/delete to the provider. Follow existing -processor patterns (e.g., `processor_sysctl.go`). - -- [ ] **Step 3: Create route processor** - -Create `internal/agent/processor_route.go` with `processRouteOperation`. Same -pattern. - -- [ ] **Step 4: Wire into network processor** - -In `internal/agent/processor_network.go`, update `NewNetworkProcessor` to accept -`InterfaceProvider` and `RouteProvider`. Add `case "interface"` and -`case "route"` to the switch. - -- [ ] **Step 5: Write processor tests** - -Create test files for both processors. Follow existing patterns. - -- [ ] **Step 6: Run tests** - -Run: `go test ./internal/agent/... -count=1` Run: `go build ./...` - -- [ ] **Step 7: Commit** - -``` -feat(network): add interface and route agent processors -``` - ---- - -### Task 5: Agent wiring - -**Files:** - -- Modify: `cmd/agent_setup.go` - -- [ ] **Step 1: Create and register providers** - -In `cmd/agent_setup.go`, create the Netplan provider (Debian only, -ErrUnsupported on other platforms) and pass it to `NewNetworkProcessor` -alongside the existing DNS and ping providers. - -The Netplan provider needs `fs`, `stateKV`, `execManager`, `hostname`, and -`netinfoProvider` — all already available in agent setup. - -- [ ] **Step 2: Verify build** - -Run: `go build ./...` - -- [ ] **Step 3: Commit** - -``` -feat(network): wire interface and route providers in agent -``` - ---- - -### Task 6: OpenAPI spec and code generation - -**Files:** - -- Modify: `internal/controller/api/node/network/gen/api.yaml` - -- [ ] **Step 1: Add interface endpoints to the OpenAPI spec** - -Add to the existing network spec: - -``` -GET /node/{hostname}/network/interface -GET /node/{hostname}/network/interface/{name} -POST /node/{hostname}/network/interface/{name} -PUT /node/{hostname}/network/interface/{name} -DELETE /node/{hostname}/network/interface/{name} -``` - -Request body for POST/PUT (`InterfaceConfigRequest`): - -- `dhcp4` (bool, omitempty) -- `dhcp6` (bool, omitempty) -- `addresses` ([]string, omitempty, dive, cidr) -- `gateway4` (string, omitempty, ipv4) -- `gateway6` (string, omitempty, ipv6) -- `mtu` (int, omitempty, min=68, max=9000) -- `mac_address` (string, omitempty) -- `wakeonlan` (bool, omitempty) - -All with `x-oapi-codegen-extra-tags` validate tags. - -- [ ] **Step 2: Add route endpoints** - -``` -GET /node/{hostname}/network/route -GET /node/{hostname}/network/route/{interface} -POST /node/{hostname}/network/route/{interface} -PUT /node/{hostname}/network/route/{interface} -DELETE /node/{hostname}/network/route/{interface} -``` - -Request body for POST/PUT (`RouteConfigRequest`): - -- `routes` ([]RouteItem, required, min=1) - - - `to` (string, required, cidr) - - `via` (string, required, ip) - - `metric` (int, omitempty, min=0) - -- [ ] **Step 3: Add DELETE for DNS** - -Add `DELETE /node/{hostname}/network/dns` endpoint to remove managed DNS config. - -- [ ] **Step 4: Regenerate code** - -Run: `just generate` - -- [ ] **Step 5: Commit** - -``` -feat(network): add interface, route, and DNS delete to OpenAPI spec -``` - ---- - -### Task 7: API handlers - -**Files:** - -- Create: `internal/controller/api/node/network/interface_list_get.go` -- Create: `internal/controller/api/node/network/interface_get.go` -- Create: `internal/controller/api/node/network/interface_create_post.go` -- Create: `internal/controller/api/node/network/interface_update_put.go` -- Create: `internal/controller/api/node/network/interface_delete.go` -- Create: `internal/controller/api/node/network/route_list_get.go` -- Create: `internal/controller/api/node/network/route_get.go` -- Create: `internal/controller/api/node/network/route_create_post.go` -- Create: `internal/controller/api/node/network/route_update_put.go` -- Create: `internal/controller/api/node/network/route_delete.go` -- Create: `internal/controller/api/node/network/dns_delete.go` -- Create: corresponding `*_public_test.go` for each handler - -- [ ] **Step 1: Implement interface handlers** - -Follow existing handler patterns (e.g., `sysctl` domain). Each handler: - -- Validates hostname -- Validates request body (for POST/PUT) -- Calls `JobClient.Query`/`Modify` or broadcast variants -- Returns collection response - -- [ ] **Step 2: Implement route handlers** - -Same pattern. Route create/update validates request body. - -- [ ] **Step 3: Implement DNS delete handler** - -Calls `JobClient.Modify` with the delete operation. - -- [ ] **Step 4: Write handler tests** - -Unit tests + HTTP wiring tests + RBAC tests for each endpoint. Follow existing -test patterns in the network package. - -- [ ] **Step 5: Update handler.go** - -Update `Handler()` function — it should already pick up new endpoints from the -regenerated `StrictServerInterface`. Verify the compile-time check passes. - -- [ ] **Step 6: Run tests** - -Run: `go test ./internal/controller/api/node/network/... -count=1` Run: -`go build ./...` - -- [ ] **Step 7: Commit** - -``` -feat(network): add interface, route, and DNS delete handlers -``` - ---- - -### Task 8: SDK service - -**Files:** - -- Create: `pkg/sdk/client/interface.go` -- Create: `pkg/sdk/client/interface_types.go` -- Create: `pkg/sdk/client/interface_public_test.go` -- Create: `pkg/sdk/client/interface_types_public_test.go` -- Create: `pkg/sdk/client/route.go` -- Create: `pkg/sdk/client/route_types.go` -- Create: `pkg/sdk/client/route_public_test.go` -- Create: `pkg/sdk/client/route_types_public_test.go` -- Modify: `pkg/sdk/client/dns.go` (add Delete method) -- Modify: `pkg/sdk/client/osapi.go` (add Interface and Route services) - -- [ ] **Step 1: Create interface SDK service** - -`InterfaceService` with List, Get, Create, Update, Delete methods. Follow -existing SDK patterns (e.g., `SysctlService`). - -- [ ] **Step 2: Create route SDK service** - -`RouteService` with List, Get, Create, Update, Delete methods. - -- [ ] **Step 3: Add DNS Delete to SDK** - -Add `Delete(ctx, target)` method to `DNSService`. - -- [ ] **Step 4: Wire into Client** - -Add `Interface *InterfaceService` and `Route *RouteService` fields to the -`Client` struct in `osapi.go`. - -- [ ] **Step 5: Regenerate SDK client** - -Run: `go generate ./pkg/sdk/client/gen/...` - -- [ ] **Step 6: Write tests** - -100% coverage on all SDK service methods and type conversions. - -- [ ] **Step 7: Commit** - -``` -feat(sdk): add Interface and Route services -``` - ---- - -### Task 9: CLI commands - -**Files:** - -- Create: `cmd/client_node_network_interface.go` -- Create: `cmd/client_node_network_interface_list.go` -- Create: `cmd/client_node_network_interface_get.go` -- Create: `cmd/client_node_network_interface_create.go` -- Create: `cmd/client_node_network_interface_update.go` -- Create: `cmd/client_node_network_interface_delete.go` -- Create: `cmd/client_node_network_route.go` -- Create: `cmd/client_node_network_route_list.go` -- Create: `cmd/client_node_network_route_get.go` -- Create: `cmd/client_node_network_route_create.go` -- Create: `cmd/client_node_network_route_update.go` -- Create: `cmd/client_node_network_route_delete.go` -- Create: `cmd/client_node_network_dns_delete.go` - -- [ ] **Step 1: Create interface CLI commands** - -Parent `interface` command under `client node network`. Subcommands: list, get, -create, update, delete. Follow existing CLI patterns (flags for each field, -`--json` support, `printKV`/`printStyledTable`). - -- [ ] **Step 2: Create route CLI commands** - -Parent `route` command. Subcommands: list, get, create, update, delete. -`--route` flag accepts `to:via:metric` format for each route. - -- [ ] **Step 3: Create DNS delete command** - -`osapi client node network dns delete --target HOST` - -- [ ] **Step 4: Verify build** - -Run: `go build ./...` - -- [ ] **Step 5: Commit** - -``` -feat(cli): add interface, route, and DNS delete commands -``` - ---- - -### Task 10: Documentation - -**Files:** - -- Create: `docs/docs/sidebar/features/network-interface-management.md` -- Create: `docs/docs/sidebar/usage/cli/client/node/network/interface/` -- Create: `docs/docs/sidebar/usage/cli/client/node/network/route/` -- Modify: `docs/docs/sidebar/features/network-management.md` -- Modify: `docs/docs/sidebar/features/features.md` -- Modify: `docs/docs/sidebar/architecture/architecture.md` -- Modify: `docs/docs/sidebar/architecture/api-guidelines.md` -- Modify: `docs/docusaurus.config.ts` -- Create: SDK doc pages and examples - -- [ ] **Step 1: Create feature page** - -Network interface and route management feature page with: - -- Overview, how it works, Netplan drop-in pattern -- CLI examples for each operation -- Safety rules (default route protection) -- Managed file reference - -- [ ] **Step 2: Create CLI doc pages** - -One page per CLI subcommand with example output. - -- [ ] **Step 3: Update cross-references** - -Features page, architecture, API guidelines, navbar dropdown. - -- [ ] **Step 4: Create SDK docs and examples** - -SDK doc page for Interface and Route services. Example files. - -- [ ] **Step 5: Commit** - -``` -docs: add network interface and route management documentation -``` - ---- - -### Task 11: Final verification - -- [ ] **Step 1: Run full test suite** - -```bash -just go::unit -``` - -- [ ] **Step 2: Build and lint** - -```bash -go build ./... -just go::vet -``` - -- [ ] **Step 3: Coverage check** - -```bash -just go::unit-cov 2>&1 | tail -1 -``` - -Must be >= 99.9%. - -- [ ] **Step 4: Cross-layer consistency check** - -Verify the interface and route domains appear in all the same places as existing -domains (grep for "sysctl" across the codebase and confirm "interface" and -"route" appear in the same files). diff --git a/docs/plans/cli-output-audit.md b/docs/plans/cli-output-audit.md deleted file mode 100644 index a17809e09..000000000 --- a/docs/plans/cli-output-audit.md +++ /dev/null @@ -1,95 +0,0 @@ -# CLI Output Audit — 100-char Width Target - -## Goal - -Standardize all CLI table output to fit within ~100 characters. Merge -STATUS/CHANGED/ERROR into a single STATUS column. Show error details in a -separate section below the table. - -## STATUS Column Values - -| Value | Meaning | -| --------- | -------------------------------------- | -| `ok` | Succeeded, no change needed | -| `changed` | Succeeded and modified system state | -| `skip` | Agent can't perform (unsupported OS) | -| `err` | Operation failed (details shown below) | - -## Table Format - -``` - Job ID: ... - - HOSTNAME STATUS NAME UID HOME - nerd ok root 0 /root - nerd ok retr0h 1000 /home/retr0h - mac skip - - Errors: - mac operation not supported on this OS family -``` - -## Core Changes - -### BuildBroadcastTable + BuildMutationTable - -- Always show HOSTNAME + STATUS -- STATUS = `ok` | `changed` | `skip` | `err` -- Remove ERROR column from table -- Remove CHANGED column (merged into STATUS) -- Return error list separately for rendering below table - -### PrintCompactTable - -- Accept optional errors section -- Render errors below table when present - -## Per-Command Column Audit - -### Keep as-is (already compact) - -| Command | Columns | Est. | -| ---------------- | -------------------------------- | ---- | -| hostname get | (none — just HOSTNAME + STATUS) | ~40 | -| uptime get | UPTIME | ~45 | -| os get | DISTRIBUTION, VERSION | ~55 | -| memory get | TOTAL, USED, FREE, USAGE | ~65 | -| load get | LOAD (1m), LOAD (5m), LOAD (15m) | ~65 | -| timezone get | TIMEZONE, UTC_OFFSET | ~55 | -| sysctl get/list | KEY, VALUE | ~60 | -| group get/list | NAME, GID, MEMBERS | ~60 | -| certificate list | NAME, SOURCE | ~55 | -| log source | SOURCE | ~45 | - -### Trim columns - -| Command | Current | Proposed | Dropped | -| ------------------ | -------------------------------------------------------- | ------------------------------------- | --------------------------- | -| hostname get | LABELS | (none) | LABELS (use node get) | -| user list | NAME, UID, GID, HOME, SHELL, GROUPS, LOCKED | NAME, UID, HOME, SHELL, GROUPS | GID, LOCKED | -| user get | NAME, UID, GID, HOME, SHELL, GROUPS, LOCKED | NAME, UID, HOME, SHELL, GROUPS | GID, LOCKED | -| process list | PID, NAME, USER, STATE, CPU%, MEM%, COMMAND | PID, NAME, USER, STATE, CPU%, COMMAND | MEM% | -| process get | PID, NAME, USER, STATE, CPU%, MEM%, COMMAND | PID, NAME, USER, STATE, CPU%, COMMAND | MEM% | -| service get | NAME, STATUS, ENABLED, DESCRIPTION, PID | NAME, STATUS, ENABLED, DESCRIPTION | PID | -| service list | NAME, STATUS, ENABLED, DESCRIPTION | NAME, STATUS, ENABLED | DESCRIPTION | -| cron list | NAME, SOURCE, SCHEDULE, OBJECT, USER | NAME, SCHEDULE, OBJECT, USER | SOURCE | -| ntp get | SYNCHRONIZED, STRATUM, OFFSET, SOURCE, SERVERS | SYNCHRONIZED, SOURCE, SERVERS | STRATUM, OFFSET | -| ping | AVG RTT, MIN RTT, MAX RTT, PACKET LOSS, PACKETS RECEIVED | AVG RTT, MIN RTT, MAX RTT, LOSS | PACKETS RECEIVED | -| docker list | ID, NAME, IMAGE, STATE, CREATED | NAME, IMAGE, STATE | ID, CREATED | -| docker inspect | ID, NAME, IMAGE, ... | NAME, IMAGE, STATE | (trim to essentials) | -| disk get | MOUNT, TOTAL, USED, FREE, USAGE | MOUNT, TOTAL, USED, USAGE | FREE | -| log query/unit | TIMESTAMP, PRIORITY, UNIT, MESSAGE | TIMESTAMP, UNIT, MESSAGE | PRIORITY | -| command exec/shell | STDOUT, STDERR, EXIT CODE, DURATION | EXIT CODE, STDOUT | STDERR (separate), DURATION | -| status get | UPTIME, LOAD (1m), MEMORY USED | UPTIME, LOAD, MEM | shorten headers | -| package list | NAME, VERSION, STATUS, SIZE | NAME, VERSION, STATUS | SIZE | -| file status | PATH, STATUS, SHA256 | PATH, STATUS | SHA256 (use --json) | -| ssh key list | TYPE, FINGERPRINT, COMMENT | TYPE, FINGERPRINT | COMMENT | - -## Implementation Order - -1. Rewrite `BuildBroadcastTable` — unified STATUS, separate errors -2. Rewrite `BuildMutationTable` — same pattern (or merge into one) -3. Update `PrintCompactTable` — render errors section -4. Update tests for new table format -5. Trim columns per command (one commit per domain group) -6. Update CLI docs diff --git a/docs/plans/ui-pki-agent-identity-design.md b/docs/plans/ui-pki-agent-identity-design.md deleted file mode 100644 index c5f703c9b..000000000 --- a/docs/plans/ui-pki-agent-identity-design.md +++ /dev/null @@ -1,148 +0,0 @@ -# UI PKI & Agent Identity — Design Spec - -Date: 2026-04-12 - -## Overview - -Update the React management dashboard to show agent identity (machine ID, -fingerprint), PKI enrollment state (Pending), and missing job status colors. Add -a pending agents admin section with accept/reject actions. Register PKI commands -in the `:` command bar. - -## 1. Agent Card Updates - -**File:** `ui/src/components/domain/agent-card.tsx` - -### Machine ID - -Show truncated machine ID below the hostname. Use the existing `Text` component -with `variant="muted"` and a `title` attribute for the full value on hover. -Clicking copies the full machine ID to clipboard. - -``` -┌─────────────────────────────────────┐ -│ 🖥 web-01 Ready │ -│ a1b2c3d4-e5f6... Pending │ -│ SHA256:4fee... │ -│ │ -│ CPU: 0.42 MEM: 4.2/8 GB UP: 3d │ -│ Ubuntu 24.04 / amd64 / 4 cpu │ -│ ● facts ● heartbeat ● pki │ -│ ⚠ DiskPressure │ -│ group:web.dev.us-east │ -│ [Drain] │ -└─────────────────────────────────────┘ -``` - -### Fingerprint - -Show fingerprint when present (PKI enabled). Truncate to ~20 chars with `...`. -Use `Text variant="muted"`. Only shown when `agent.fingerprint` is non-empty. - -### Pending State - -Add `Pending` to `stateVariant()`: - -```typescript -case "Pending": - return "pending" as const; -``` - -When state is `Pending`: - -- Show `Pending` badge (yellow, uses existing `pending` variant) -- Hide drain/undrain buttons -- Show `Text variant="muted"`: "Awaiting PKI enrollment" - -## 2. Job Status Colors - -**File:** `ui/src/pages/jobs.tsx` - -Update `statusBadgeVariant()` to handle all job statuses: - -```typescript -function statusBadgeVariant(status?: string) { - switch (status) { - case 'completed': - return 'ready' as const; - case 'failed': - case 'partial_failure': - return 'error' as const; - case 'processing': - case 'acknowledged': - case 'started': - return 'running' as const; - case 'submitted': - return 'pending' as const; - case 'skipped': - return 'muted' as const; - case 'retried': - return 'pending' as const; - default: - return 'muted' as const; - } -} -``` - -## 3. Pending Agents Section - -### Location - -Add a "Pending Agents" section to the Dashboard page, above or below the -existing agent cards. Only visible when there are pending agents AND the user -has `agent:write` permission. - -### Component - -**New file:** `ui/src/components/domain/pending-agent-card.tsx` - -A card showing: - -- Machine ID (full) -- Hostname -- Fingerprint (full SHA256) -- Requested time (relative, e.g., "5m ago") -- Accept button (green) -- Reject button (red/muted) - -Uses existing components: `Card`, `Badge variant="pending"`, `Button`, `Text`. - -### Data - -Use the generated SDK function `getAgentsPending()` from the agent operations -module. Poll on the same interval as the dashboard health data. - -### Accept/Reject - -Call `acceptAgent(hostname)` and `rejectAgent(hostname)` from the generated SDK. -On success, refresh the pending list. Show a brief success/error state on the -button. - -## 4. Command Bar - -**File:** Register commands in the Dashboard page (`ui/src/pages/dashboard.tsx`) - -Commands: - -- `pending` — scroll to or expand the pending agents section -- `accept ` — accept a pending agent by hostname -- `reject ` — reject a pending agent by hostname - -Commands are registered via `useCommands()` hook. Accept/reject commands need a -hostname argument — use the command bar's input to extract it. - -## 5. Components Summary - -| Component | File | New/Modify | -| -------------------- | ------------------------ | ---------- | -| `AgentCard` | `agent-card.tsx` | Modify | -| `PendingAgentCard` | `pending-agent-card.tsx` | New | -| `statusBadgeVariant` | `jobs.tsx` | Modify | -| Dashboard | `dashboard.tsx` | Modify | -| Command registration | `dashboard.tsx` | Modify | - -## Non-Goals - -- Full agent detail page (separate feature) -- PKI configuration UI (config is YAML-only) -- Key rotation UI (CLI-only for now)