From bcdaf084051ff80c26c4fdd556bee97b91fa89fd Mon Sep 17 00:00:00 2001 From: eshulman2 Date: Sun, 6 Sep 2026 13:54:03 +0300 Subject: [PATCH] docs: clarify Forge operations and workflow configuration --- README.md | 3 +- docs/architecture/control-plane.md | 118 ++++++ .../forge-2.0-control-plane-design.md | 364 ------------------ docs/architecture/index.md | 1 + docs/architecture/internals.md | 3 + docs/architecture/overview.md | 3 + docs/forge-2.0-control-plane-guide.md | 321 --------------- docs/guide/bug-workflow.md | 6 +- docs/guide/feature-workflow.md | 6 +- docs/guide/labels.md | 2 + docs/guide/pr-commands.md | 4 +- docs/guide/task-workflow.md | 5 +- docs/index.md | 5 +- docs/operations.md | 126 ++++++ docs/reference/api.md | 17 + docs/reference/config.md | 35 ++ docs/reference/declarative-workflows.md | 43 +++ docs/reference/proposals.md | 5 + zensical.toml | 2 + 19 files changed, 372 insertions(+), 697 deletions(-) create mode 100644 docs/architecture/control-plane.md delete mode 100644 docs/architecture/forge-2.0-control-plane-design.md delete mode 100644 docs/forge-2.0-control-plane-guide.md create mode 100644 docs/operations.md diff --git a/README.md b/README.md index af6d27af6..5f270b770 100644 --- a/README.md +++ b/README.md @@ -225,7 +225,8 @@ See [Getting Started](https://Forge-sdlc.github.io/forge/getting-started/) for t - [Feature Workflow](https://Forge-sdlc.github.io/forge/guide/feature-workflow/): Understand the feature pipeline and approval gates. - [Bug Workflow](https://Forge-sdlc.github.io/forge/guide/bug-workflow/): Understand triage, RCA, fix options, and bug implementation. - [Task Workflow](https://Forge-sdlc.github.io/forge/guide/task-workflow/): Understand standalone Task and Epic implementation. -- [PR Commands](https://Forge-sdlc.github.io/forge/guide/pr-commands/): Rebase PRs and handle CI gate skips. +- [Workflow Management](https://Forge-sdlc.github.io/forge/guide/labels/): Control approvals, revisions, draft review, retries, and PR commands. +- [Operations](https://Forge-sdlc.github.io/forge/operations/): Run services, inspect execution, recover effects, and respond to incidents. - [Configuration Reference](https://Forge-sdlc.github.io/forge/reference/config/): Environment variables and project configuration. - [Architecture](https://Forge-sdlc.github.io/forge/architecture/): Versioned workflows, reconciliation, typed stations, durable effects, and execution inspection. diff --git a/docs/architecture/control-plane.md b/docs/architecture/control-plane.md new file mode 100644 index 000000000..71560cc8d --- /dev/null +++ b/docs/architecture/control-plane.md @@ -0,0 +1,118 @@ +# Control-plane architecture + +Forge coordinates distributed systems that can retry, race, fail mid-write, or +disagree. Its control plane makes the ownership of external facts, workflow +decisions, agent work, and provider mutations explicit. + +```mermaid +flowchart LR + E[Provider event] --> A[Ingress adapter] + A --> O[Versioned observation] + O --> L[Reconciliation ledger] + L -->|accepted| C[Workflow command] + L -->|duplicate, stale, conflict| T[Execution timeline] + C --> P[Pinned definition and checkpoint] + P --> N[Graph node] + N --> S[Typed station] + S --> R[Validated reducer] + R --> P + N --> F[Effect command] + F --> J[Durable effect journal] + J --> X[Provider executor] +``` + +## Ownership model + +| Record | Owner | Purpose | +| --- | --- | --- | +| Observation | Jira, source control, or poller | A versioned external fact and delivery identity. | +| Observation decision | Reconciliation ledger | Whether evidence is accepted, duplicate, stale, or conflicting. | +| Workflow command | Command boundary | The semantic operation requested by accepted evidence. | +| Definition and checkpoint | Forge | The immutable workflow revision, saved position, and workflow-owned state. | +| Station attempt | Forge | A bounded request, validated result, and owned state update. | +| Effect record | Effect journal | Intent, lease, attempts, provider evidence, and idempotent external write. | +| Execution timeline | Read-model projection | An explainable view of why a run is waiting, blocked, or complete. | + +External systems own their facts. Forge owns how those facts are reconciled and +which process transition they authorize. A webhook payload must not mutate +`current_node`, definition identity, approval state, or effect-journal fields. + +## Main boundaries + +### Observations and commands + +Ingress adapters normalize webhook and poller payloads into stable observations. +The ledger is the precondition for command handling, so duplicate and +out-of-order delivery converges instead of advancing a workflow twice. Commands +are then checked against the active definition's transition policy. + +### Pinned declarative definitions + +Definitions provide the workflow topology: entry, nodes, edges, routed +branches, joins, and flow execution settings. Each workflow instance pins its +name, revision, digest, and canonical definition. Publishing a new definition +changes future selection only; it cannot silently alter an in-flight ticket. + +The trusted catalog provides authority: node identity, station contracts, +allowed effects, mandatory policies, preconditions, and observation behavior. +Project authors configure flow with a declarative definition; they cannot grant +new authority. See [declarative workflows](../reference/declarative-workflows.md). + +### Typed stations + +A projector creates a versioned, narrow station request from owned checkpoint +fields. A station performs one bounded operation such as artifact generation, +triage, review, task routing, implementation input resolution, or sandbox +execution. Its reducer validates the outcome and writes only fields assigned to +that station. + +Planning and review run on the host. Implementation runs in a rootless Podman +container. Containers do not receive Jira, Redis, or source-control +credentials; provider writes return through effects. + +### Durable effects and read models + +Nodes request external writes through `EffectCommand` values. The effect journal +persists intent before contacting a provider, leases one executor, and retains +attempt/provider evidence. Recovery reuses the effect's idempotency identity, +which avoids repeating agent work after a crash. + +Read models join the checkpoint, definition, observations, commands, station +attempts, and effects. They are diagnostic projections, never an alternate +control path. See [operations](../operations.md) for recovery procedures. + +## Extending Forge safely + +There are two distinct extension paths. + +### Project administrators: declarative configuration + +Project configuration can select a built-in or published workflow and compose +catalog-registered nodes, routers, transitions, joins, concurrency, and resume +mappings. It can configure repositories, proposal review, skills, and model +policy. It cannot execute Python or shell code, store credentials, make provider +calls, add an effect operation, or relax trusted policies. + +### Core maintainers: trusted capabilities + +Adding a lifecycle capability requires code and contract changes in this order: + +1. Classify the change as an external observation, semantic command, bounded + station operation, effect, or topology change. +2. Define or update the versioned domain/station/structured-output contract. +3. Add an ingress adapter and command mapping when external evidence begins the + behavior; provide stable identities and reconciliation semantics. +4. Add a narrow projector, station, reducer, and contract tests for work that + consumes or changes workflow state. +5. Add a stable effect operation and executor for every provider write, with + idempotency, retry, and provider-evidence tests. +6. Register the trusted node/router/station/effect authority in the catalog, + then change the built-in definition or supported flow accordingly. +7. Increment workflow revisions when topology changes, validate/render/diff the + definition, and simulate migration for saved positions. +8. Ensure the observation, command, station, and effect evidence appears in the + execution timeline; test duplicate delivery and worker restart recovery. + +Avoid shortcuts: direct provider writes from nodes, webhook-driven graph +updates, unrestricted agent access to checkpoints, or Python branches that +silently change topology reintroduce the failure modes these boundaries prevent. diff --git a/docs/architecture/forge-2.0-control-plane-design.md b/docs/architecture/forge-2.0-control-plane-design.md deleted file mode 100644 index 7259a7e82..000000000 --- a/docs/architecture/forge-2.0-control-plane-design.md +++ /dev/null @@ -1,364 +0,0 @@ -# Forge 2.0 control-plane architecture for developers - -This is the developer and architect guide to the control-plane stack delivered -by PRs 324–332. It explains the design intent, the ownership rules that replace -the pre-2.0 implementation style, and how to extend Forge without bypassing -the new correctness boundaries. - -For release, deployment, and operator procedures, see -[`../forge-2.0-control-plane-guide.md`](../forge-2.0-control-plane-guide.md). - -## The architectural shift - -Pre-2.0 Forge was principally a LangGraph application: queue deliveries entered -the worker, nodes interpreted provider payloads, invoked agents, mutated the -checkpoint, and called Jira or GitHub as needed. That worked for the golden -path, but process correctness was distributed across node code, event handlers, -provider adapters, and retry logic. - -Forge 2.0 makes Forge a **durable workflow control plane**. LangGraph remains -the graph execution adapter; it is no longer the public definition of a Forge -process. The following records are now first-class architectural boundaries: - -| Record | Source of truth | What it answers | -| --- | --- | --- | -| Observation | Jira, source control, or poller | What external fact was received? | -| Observation decision | Reconciliation ledger | Is that fact accepted, duplicate, stale, or conflicting? | -| Workflow command | Command boundary | What state transition or exceptional operation does the accepted fact request? | -| Pinned definition/checkpoint | Forge | Which immutable process revision and position owns this run? | -| Station attempt | Forge | What bounded operation was requested, and what validated result did it return? | -| Effect record | Effect journal | Which external write was intended, attempted, and observed? | -| Execution read model/timeline | Forge projection | Why is this run at its current state? | - -The key rule is: **external systems own their facts; Forge owns interpretation -and process state.** A Jira/GitHub payload cannot directly set `current_node`, -pause/retry state, workflow identity, or an effect-journal field. - -```mermaid -flowchart LR - E[Provider event] --> A[Ingress adapter] - A --> O[Versioned Observation] - O --> L[Reconciliation ledger] - L -->|accepted| C[Workflow command] - L -->|duplicate/stale/conflict| T[Timeline only] - C --> P[Pinned definition + transition policy] - P --> N[Graph node] - N --> S[Typed station] - S --> R[Validated reducer] - R --> P - N --> F[Effect command] - F --> J[Durable effect journal] - J --> X[Provider executor] -``` - -## Dependency and PR map - -The stack's dependency order is not entirely numerical: - -``` -324 -> 325 -> 326 -> 327 -> 328 -> 331 -> 329 -> 330 -> 332 -``` - -PR 331 is an ancestor of PR 329; PR 330 depends on 329; PR 332 depends on 330. -Architecturally, the stack implements eight layers: - -| Layer | PR | Design responsibility | -| --- | --- | --- | -| 1 | 324 | Versioned domain contracts and provider-neutral source-control observations | -| 2 | 325 | Normalized ingress and semantic workflow commands | -| 3 | 326 | Durable, recoverable external effects | -| 4 | 327 | Typed station/projection/reducer execution boundary | -| 5 | 328 | Governed, declarative, pinned process definitions | -| 6 | 331 | Observation reconciliation across webhook and poller ingress | -| 7 | 329 | Execution read models, timeline, and Org Pulse projection | -| 8 | 330/332 | Removal of legacy paths; definitions as sole topology; strict structured outputs | - -## Layer 1: domain contracts and provider neutrality (PR 324) - -`src/forge/domain/` establishes versioned Pydantic contracts for observations, -commands, effects, identities, interactions, and stations. Code crossing a -system boundary should use these contracts rather than an ad-hoc `dict` or a -provider SDK object. - -Source control is now represented by contracts and adapters under -`src/forge/integrations/source_control/`. GitHub is an adapter implementation, -not a workflow dependency. Its events are adapted into `Observation` values -with a stable resource identity, provider revision, facts, correlation, and a -delivery identity. - -### Consequences for new providers - -To support another source-control provider, add a conforming adapter and -observation mapping. Do not add provider-specific conditionals to a workflow -node. The adapter must define stable resource and delivery identities and, -where the provider supports it, ordering/revision metadata. If the provider -cannot provide safe ordering data, the system should surface ambiguity instead -of inventing an ordering rule. - -## Layer 2: commands are the only ingress into process control (PR 325) - -Ingress adapters under `src/forge/orchestrator/event_adapters/` turn raw queue -events into observations. `command_handlers.py` and the command-operation -boundary derive a `WorkflowCommand` only after an observation is reconciled. -This applies to Jira approval labels, comment commands, retry requests, review -events, CI results, and exceptional actions such as `/forge rebase`. - -This separates three decisions that were formerly easy to conflate: - -1. Did the provider report a coherent, sufficiently new external fact? -2. What semantic action does that fact express? -3. Does the currently pinned workflow allow that action at its saved position? - -### Development rule - -Never advance a graph from a webhook handler, a poller payload, or a label -parser. Add/extend an observation adapter, command derivation, and the relevant -workflow transition policy. An unrecognized command must remain observable but -must not mutate workflow state. - -## Layer 3: external writes are durable effects (PR 326) - -`src/forge/effects/` replaces best-effort direct mutation with the effect -journal/executor/service pattern: - -1. Construct a stable `EffectCommand` with workflow identity, operation, - target, and idempotency identity. -2. Submit it to the Redis-backed journal *before* contacting a provider. -3. Claim a lease so only one executor owns the attempt. -4. Execute the registered provider executor. -5. Persist the resulting provider evidence, status, and attempt history. - -The journal states distinguish pending/running work from retryable, -precondition, terminal, and successful results. Transient failures use bounded -backoff. A recovery sweep executes due records. A workflow-critical effect can -wait for a concurrent sweep owner to settle; it does not treat exclusive lease -ownership as a failure. Retryable and terminal outcomes nevertheless fail -closed and prevent an unsafe process advance. - -### Why idempotency is non-negotiable - -The sequence “provider mutated successfully, worker died before checkpoint -write” is unavoidable in distributed systems. Retrying the agent or node may -create duplicate comments, branches, issues, or PRs. Retrying a stable effect -identity instead converges on the intended mutation and preserves its history. - -### Development rule - -Do not instantiate a Jira/GitHub/repository client to make a workflow-visible -write from a node, station, or command handler. Register an executor and emit -an allowed `EffectCommand` through `effect_runtime`. Reads may still use the -appropriate adapter. New effect operations must be declared in the trusted -effect catalog and granted only to the relevant trusted nodes. - -## Layer 4: typed stations narrow agent and execution authority (PR 327) - -The station runtime separates orchestration from work execution: - -- A **projector** selects the permitted fields from the checkpoint and creates - a versioned `StationRequest`. -- A **station** performs one bounded operation: approval classification, - artifact generation, triage, task routing, agent operation, implementation - input resolution, sandbox execution, or persistence. -- A **reducer** validates the `StationOutcome` and applies only the state fields - owned by that station. - -Station definitions and the registry live under `src/forge/workflow/stations/`; -projectors and reducers live in their corresponding packages. The node remains -responsible for orchestration and routing, not for open-ended provider access. - -This preserves the product split: planning/review agents execute on the host; -implementation runs in the rootless Podman sandbox. The sandbox receives -repository/model execution material but not Jira, Redis, or source-control -credentials. Provider writes return to the host durable-effect boundary. - -### Development rule - -When adding a new agent or sandbox operation, define a typed input/output -contract first. Keep it narrow, version it, project only owned input, and add a -reducer that rejects malformed or unauthorized output. Do not pass the entire -LangGraph state into a new agent as a convenience. - -## Layer 5: workflows are governed, declarative processes (PR 328) - -`src/forge/workflow/declarative/` owns definition parsing, validation, -publication, resolution, manifest generation, catalog lookup, compilation, and -migration analysis. Built-in Feature, Bug, and Task Takeover definitions are -canonical JSON artifacts in `definitions/`; human-authored project definitions -are YAML or JSON and are published as canonical JSON to Jira project properties. - -A definition is intentionally flow-only: - -```yaml -metadata: - name: prd-only - revision: 1 -spec: - state: feature - entry: generate_prd - steps: - generate_prd: - next: prd_approval_gate - prd_approval_gate: - route: route_prd_approval - branches: - generate_spec: __end__ - regenerate_prd: generate_prd - __end__: __end__ -``` - -The definition determines topology: state profile, entry, nodes, fixed edges, -routed branches, dynamic fan-out, joins, and retry/concurrency flow settings. -The trusted catalog determines execution authority: node identity, station -contract, effect capabilities, required policies, preconditions, and -observation behavior. A project author cannot grant itself an effect capability -or weaken a mandatory policy by editing YAML. - -Each created workflow pins the selected definition name, revision, digest, and -canonical payload in the checkpoint. Publication changes what future tickets -select; it does not silently rewrite a running ticket. - -### Definition authoring rule - -Run `forge workflow catalog ` before authoring and -use only reported nodes/routers. Then run `validate`, `render`, and `diff`. -Every static router outcome must appear in its `branches` map. If a saved node -is renamed/removed, increment `metadata.revision`, provide -`spec.resume.fromRevisions`, and run `simulate-migration` against real or -representative checkpoints. Valid YAML alone does not prove resumption safety. - -## Layer 6: reconciliation makes multiple ingress sources converge (PR 331) - -The Redis observation ledger is the precondition for command interpretation. -It tracks resource identity, delivery identity, provider revision, decision, -drift class, and per-run history. Equivalent webhook and poller deliveries -therefore become one logical external observation. - -The ledger classifies observations as accepted, duplicate, stale, or conflict. -It also blocks attempts by an external payload to assert workflow-owned facts. -An orderable provider revision is preferred. When a provider supplies opaque or -unversioned revisions that cannot safely be ordered, Forge records the reason -and does not pretend the event is newer. - -### Architectural consequence - -At-least-once delivery is expected, not exceptional. A new integration must be -correct under duplicate delivery, out-of-order delivery, and a poller/webhook -race. “The handler is idempotent in practice” is insufficient: it must produce -a stable observation identity and let the ledger make the ordering decision. - -## Layer 7: read models are projections, not an alternate control path (PR 329) - -`src/forge/read_models/` composes independent durable records into an -operator-facing execution model and timeline. It does not execute nodes or read -current Jira labels to infer process state. Its sources are the pinned -definition/checkpoint, observation decisions, station attempts, and effects. - -The gateway exposes: - -```text -GET /api/v1/workflows/{ticket_key}/execution -GET /api/v1/workflows/{ticket_key}/execution/timeline -GET /api/v1/org-pulse/workflows/{ticket_key} -GET /api/v1/effects/workflow/{run_id} -GET /api/v1/effects/{idempotency_key} -POST /api/v1/effects/{idempotency_key}/replay -``` - -The APIs are protected by `FORGE_OPERATOR_TOKEN`. The replay endpoint is the -only mutation endpoint in this group, and it only requeues an eligible terminal -effect; it does not rerun a station or advance the graph. - -### Development rule - -Add diagnostic data at its authoritative boundary (observation decision, -station attempt, effect result, checkpoint transition), then project it into -the timeline. Do not add an API endpoint that recalculates workflow state from -provider data or performs hidden recovery work in a GET request. - -## Layer 8: the cutover is intentionally strict (PRs 330 and 332) - -The final PRs remove Phase 8 compatibility paths, move observation transitions -behind policy, make definitions the sole source of graph topology, complete -built-in effect-capability declarations, and enforce structured model outputs. -Structured stages use Pydantic output contracts with provider-native structured -output and validated fallback strategy; narrative artifacts remain Markdown. - -The strictness is intentional: - -- an undeclared router outcome is an error, not an inferred transition; -- an unknown workflow label/definition blocks instead of falling back; -- invalid structured output is rejected and retried/escalated according to the - workflow rather than parsed optimistically; -- a node cannot emit an effect operation absent from its trusted capability set; -- external facts cannot overwrite process-owned checkpoint facts. - -This is a Forge 2.0 major-version boundary. Old direct-mutation extensions and -legacy checkpoints should be drained, resolved, or explicitly migrated rather -than assumed resumable. - -## How to implement a change after the cutover - -Use this sequence for a new lifecycle capability: - -1. **Classify the boundary.** Is it a provider fact, a semantic user/provider - command, a workflow topology change, a bounded station operation, or an - external effect? One feature can require several, but do not collapse them. -2. **Define versioned contracts.** Add/change domain, station, or structured - output models before implementation. -3. **Adapt and reconcile ingress.** For external input, create a stable - observation and command mapping, then define how transition policy consumes - the command. -4. **Use a station for work.** Add projector, station, reducer, and contract - tests. Preserve ownership boundaries in the reducer. -5. **Use an effect for writes.** Add a stable operation, executor, catalog - capability, and retry/idempotency tests. -6. **Change topology declaratively.** Update the trusted catalog/built-in - definition as appropriate, increment definition revision, and validate, - render, diff, and simulate migration. -7. **Expose evidence.** Ensure the resulting observation/command/station/effect - is visible in the execution timeline and that operator errors are actionable. -8. **Test convergence.** Cover replay, duplicate/out-of-order ingress, worker - restart around effects, unauthorized output/effects, and pinned-definition - behavior—not only the happy-path graph execution. - -## Design mistakes the new architecture is meant to prevent - -| Avoid | Use instead | Reason | -| --- | --- | --- | -| Calling Jira/GitHub directly in a node | Effect command and registered executor | Provides idempotency, recovery, audit, and authority checks. | -| Routing directly from a webhook | Observation -> ledger -> command -> policy | Makes duplicate and stale delivery safe. | -| Adding a Python branch to change a workflow's flow | Definition/catalog change | Keeps topology inspectable, versioned, and migratable. | -| Giving an agent the full checkpoint | Projected station request | Limits authority and makes output ownership auditable. | -| Treating an LLM JSON string as trusted | Structured output contract plus reducer validation | Rejects malformed or unauthorized state changes. | -| Retrying the whole workflow after a provider failure | Replay/repair the individual effect | Avoids repeating agent work and duplicating writes. | -| Deriving status from live Jira labels in an API | Durable execution read model | Preserves causal process history. | - -## Integration fixes validated with the stack - -The integration branch contains production fixes found while exercising all -three workflows. These are not separate architectural layers, but they clarify -how the boundaries should work in practice: - -- Gate resumption schedules the definition's declared next transition. -- Workspace setup and implementation persistence declare their required - repository effect capabilities. -- Shared `implement_work` resolves and persists a repository-scoped work unit - for both task-based and taskless execution. -- A merged PR is a terminal source-control observation even when its head SHA - is no longer available or differs from a tracked head. -- Feature decomposition keeps its draft in checkpoint state, avoiding a second - Jira attachment authority. -- Repository labels are reconciled without removing/readding a retained label, - which would defeat durable-effect deduplication. -- Bug RCA structured output selects one configured repository and only then - writes the matching `repo:/` label. - -## Architectural bottom line - -The new system has more explicit components because Forge is coordinating -unreliable distributed actors: providers, queues, agents, containers, and -workers. Those components are not optional abstraction layers. They assign one -owner to each fact, decision, mutation, and transition. Future development -should preserve that separation; bypassing it may restore a short path locally, -but reintroduces the duplicate, crash-recovery, and unexplained-state failures -the Forge 2.0 stack is designed to eliminate. diff --git a/docs/architecture/index.md b/docs/architecture/index.md index 43b8b2c50..83e70e117 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -7,6 +7,7 @@ For workflow details, see the [Feature](../guide/feature-workflow.md), [Bug](../ | Part | Contents | |------|----------| +| [Control-plane architecture](control-plane.md) | Ownership boundaries and safe project/core extension paths | | [System and components](overview.md) | Control-plane structure and component responsibilities | | [Runtime internals](internals.md) | State authority, reconciliation, stations, effects, and security | | [Reference](reference.md) | Architectural decisions, known limitations, workflow lifecycles | diff --git a/docs/architecture/internals.md b/docs/architecture/internals.md index 372362d2c..e14b8e928 100644 --- a/docs/architecture/internals.md +++ b/docs/architecture/internals.md @@ -1,5 +1,8 @@ # Runtime internals +This page is the component-level reference. Read [Control-plane architecture](control-plane.md) +first for the system ownership model and the safe extension paths. + ## State and correctness boundaries Forge deliberately keeps four kinds of durable state separate: diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index abb878b4d..de3251008 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -5,6 +5,9 @@ systems remain authoritative for their domain facts. Forge owns process interpre versioned process definition, each run's pinned definition and position, transition decisions, station attempts, and external-effect history. +For ownership rules and the distinction between project configuration and core +extensions, see [Control-plane architecture](control-plane.md). + ```mermaid flowchart LR External["Jira / source control"] -->|webhook or poller observation| Gateway["Gateway"] diff --git a/docs/forge-2.0-control-plane-guide.md b/docs/forge-2.0-control-plane-guide.md deleted file mode 100644 index 445b63aa5..000000000 --- a/docs/forge-2.0-control-plane-guide.md +++ /dev/null @@ -1,321 +0,0 @@ -# Forge 2.0 control-plane change guide - -This guide describes the stacked control-plane change set in PRs 324–332 and -the operational model it introduces. It is written for operators and project -administrators upgrading from the pre-2.0 Forge runtime. - -## What this release is - -Forge 2.0 changes the execution architecture, not the product goal. A managed -Jira ticket still drives planning, implementation, CI repair, and human review. -The change is that Forge now records and governs every boundary between an -incoming provider event and an external mutation. - -Before this change set, the worker interpreted webhook/poller payloads and -called workflow nodes and provider clients directly. That made normal operation -simple, but left important questions difficult to answer after a retry, worker -crash, duplicate delivery, or workflow-definition change: - -- Was this event already handled, and is it newer than the last one? -- What workflow transition did it authorize? -- Did a Jira update, branch push, or pull-request creation happen before the - worker stopped? -- Can an operator retry one failed provider write without rerunning an agent? -- Which exact workflow definition was the ticket executing? - -Forge 2.0 supplies durable answers to those questions. It is a control plane -over the existing Jira, source-control, agent, sandbox, Redis, gateway, and -worker runtime. - -## Merge order - -The pull requests are a dependency stack, but the final portion is not ordered -by PR number. The required ancestry order is: - -``` -324 -> 325 -> 326 -> 327 -> 328 -> 331 -> 329 -> 330 -> 332 -``` - -PR 331 is an ancestor of PR 329, PR 329 is an ancestor of PR 330, and PR 330 -is an ancestor of PR 332. Merging 329 before 331 would create an avoidable -stacking conflict or duplicate ancestry situation. - -## Runtime model after the change - -```mermaid -flowchart LR - P[Jira, GitHub, poller] --> G[Gateway] - G --> Q[Redis Streams] - Q --> W[Worker] - W --> O[Observation ledger] - O --> C[Validated command] - C --> D[Pinned workflow definition] - D --> S[Typed station] - S --> A[Agent or Podman sandbox] - D --> E[Durable effect journal] - E --> P - W --> R[Execution read model and timeline] -``` - -The gateway remains intentionally thin: it authenticates and queues ingress. -The worker owns reconciliation, command interpretation, workflow execution, -and effect recovery. The poller remains an external peer ingress service; it is -not replaced and no new poller process is introduced by this stack. - -## New logical services and storage - -There are no new mandatory containers in `docker-compose.yml`. Redis, the -gateway, and the host-side worker remain the deployment units. The following -new **logical services** run inside the gateway/worker process and persist to -Redis: - -| Component | Runs in | Purpose | -| --- | --- | --- | -| Observation ledger | Worker | Deduplicates and orders Jira/GitHub/poller observations and records why one was accepted, stale, duplicate, or conflicting. | -| Command boundary | Worker | Converts accepted observations and exceptional user actions into typed, validated commands before workflow state changes. | -| Durable effect service | Worker | Journals Jira, source-control, and repository write intent; leases execution; retries transient failures; and permits targeted replay. | -| Typed station runtime | Worker | Runs a bounded operation using a versioned request/output contract rather than giving a node the whole checkpoint and unrestricted provider access. | -| Definition registry/compiler | Worker and CLI | Resolves an immutable built-in or published definition, validates it against the trusted catalog, and compiles it to LangGraph. | -| Execution read model/timeline | Gateway API and worker | Produces an operator view of process position, observations, station attempts, effects, waiting, blocking, and migration status. | - -Redis therefore becomes more than the queue and LangGraph checkpoint store. It -also stores observation decisions, effect records and scheduling indexes, -definition/pinning data, and execution timeline records. Preserve Redis during -the release; deleting it discards recovery and audit history. - -## What each PR introduces - -### PR 324 — versioned workflow domain contracts - -Establishes the provider-neutral vocabulary used by the rest of the stack: -identities, observations, commands, interactions, effects, and stations. It -adapts GitHub events into source-control observations and moves implementation -input resolution behind a typed station. - -Practical effect: Forge stops treating an inbound webhook payload as workflow -control data. It first turns it into a stable, typed description of an external -fact. This also begins provider-neutral source-control support: GitHub is the -current adapter, while workflow code addresses a source-control contract rather -than GitHub-specific objects. - -### PR 325 — command interpretation boundary - -Normalizes all ingress at the worker boundary, then derives and persists a -semantic command before applying a state transition. Jira labels/comments, -retries, and exceptional commands use the same model; source-control review -enrichment is isolated from graph execution. - -Practical effect: a label or comment no longer changes checkpoint state merely -because it arrived. The command must be recognized and allowed by the active -workflow transition policy. Ignored events retain a durable identity and an -explanation instead of being invisible. - -### PR 326 — durable external effects - -Introduces the durable effect journal and executors for Jira, source control, -repository pushes, files, notifications, and workflow relationship updates. -Required writes are submitted before a provider call, executed under an -exclusive lease, retain attempt/provider evidence, and retry with bounded -backoff. Terminal effects can be explicitly replayed by an operator. - -Practical effect: a worker failure between deciding to create a PR and receiving -the provider response no longer requires rerunning the planning or implementation -agent. Forge recovers the individual mutation by its stable idempotency key. A -required effect that is retryable or terminal still blocks forward progress: -Forge fails closed rather than assuming the side effect happened. - -### PR 327 — typed station boundary - -Moves agent calls, approval handling, planning, task routing, sandbox execution, -review inference, and post-merge persistence behind typed stations. Projectors -construct the narrow request a station may see; reducers validate the station -outcome and apply only fields that station owns. - -Practical effect: workflow nodes become orchestration code rather than direct -provider/agent clients. Planning and review stay host-side; implementation stays -in the existing rootless Podman sandbox. Sandboxes still do not receive Jira, -Redis, or source-control credentials. This is a safety and testability boundary, -not a new agent service to run. - -### PR 328 — governed, declarative workflows - -Adds versioned process definitions, a trusted node/router/station catalog, -definition validation, publication/activation controls, manifests, and migration -simulation. Built-in Feature, Bug, and Task Takeover workflows are published as -immutable JSON artifacts and compiled from their topology. - -Practical effect: flow topology is no longer implicitly defined by Python graph -wiring. Each run pins a definition name, revision, digest, state profile, and -position. A later publication cannot silently alter an in-flight ticket. -Administrators may publish a constrained YAML/JSON workflow with registered -nodes only; they cannot add Python, shell, credentials, provider calls, or new -effect authority. The trusted catalog—not the definition author—assigns allowed -effects, station contracts, preconditions, and observation policy. - -### PR 331 — webhook and poller reconciliation - -Adds the convergent observation ledger and reconciles every observation before -command interpretation. Equivalent deliveries from a webhook and the poller -converge; stale revisions, contradictory revisions, and attempts to set -workflow-owned facts are recorded and rejected. - -Practical effect: ingress is at-least-once, but its workflow consequences are -convergent. A duplicate check-suite, review, or ticket event will not advance a -workflow twice. Forge intentionally refuses to guess when a provider supplies no -orderable revision or sends contradictory facts; that produces an observable -conflict for an operator instead of unsafe state movement. - -### PR 329 — execution read models and timeline - -Adds durable execution read models and timeline construction, plus a compact -Org Pulse projection. It joins the pinned manifest/checkpoint with command and -observation decisions, station attempts, and effect history. - -Practical effect: operators no longer need to reconstruct a workflow from -worker logs and Jira comments. They can inspect why a ticket is waiting, -blocked, stale, conflicted, or migration-ineligible, and see the relevant -attempts and external writes in order. - -### PR 330 — remove compatibility execution paths - -Removes the transitional Phase 8 compatibility paths and makes policy-governed -observation transitions authoritative. Legacy direct paths are deliberately no -longer available. - -Practical effect: this is the principal Forge 2.0 compatibility boundary. The -new control-plane rules are not optional fallbacks. Do not expect a pre-2.0 -checkpoint or custom extension that relied on direct node/provider mutation to -continue executing unchanged. - -### PR 332 — complete the cutover and enforce structured output - -Makes definitions the sole source of graph topology and keeps them flow-only. -Completes built-in effect capability declarations and governed side effects, -adds strict structured agent-output validation, accepts valid JSON-array triage -output, and verifies built-in workflow revision behavior. - -Practical effect: an undeclared router result, an invalid structured artifact, -or a node attempting an unauthorized effect is an explicit failure, not a best -effort continuation. This explains errors such as `router returned undeclared -outcome`: they reveal a definition/catalog contract mismatch that must be fixed -rather than guessed around. - -## Changes to the three workflows - -The Feature, Bug, and Task Takeover lifecycles remain familiar to users: -approvals still pause work and CI/human-review signals still govern pull-request -completion. Their execution semantics are now shared and durable. - -| Area | Prior behavior | Forge 2.0 behavior | -| --- | --- | --- | -| Approval and revision | Ingress-specific node handling could directly advance a graph. | A reconciled observation produces a validated command that the pinned definition allows or rejects. | -| Agent work | Nodes could call agents/providers with broad state context. | Typed stations use a bounded request, typed outcome, and reducer-owned state fields. | -| Jira/SCM writes | A crash could leave uncertainty or cause a duplicate on retry. | Each write has an effect identity, journal status, attempts, provider evidence, and controlled replay. | -| PR/CI/review events | Webhook and poller events could be processed as separate deliveries. | Both are observations of the same resource and reconcile before workflow interpretation. | -| Workflow changes | Runtime graph wiring was the effective definition. | A run pins an immutable definition revision and digest. | -| Diagnostics | Jira and logs were the primary reconstruction tools. | Read-only execution/timeline APIs expose durable process evidence. | - -The implementation path is additionally standardized around the `implement_work` -node. It resolves a scoped work unit from the current task, repository-specific -task/epic plan, general plan, spec, RCA, PRD, or root ticket in that order, and -persists its identity and artifact digests. This is why taskless and task-based -workflows can share one safe implementation engine. - -## New operator and administrator interfaces - -### Workflow-definition CLI - -```bash -forge workflow catalog feature -forge workflow validate workflow.yaml -forge workflow render workflow.yaml -forge workflow diff previous.yaml current.yaml -forge workflow simulate-migration previous.yaml current.yaml instances.json -forge workflow publish MYPROJECT workflow.yaml -forge workflow list MYPROJECT -forge workflow show MYPROJECT workflow-name -forge workflow show-history MYPROJECT workflow-name -``` - -Use YAML as the authoring format. Publication stores canonical JSON in the Jira -project property and requires Jira project/global administration permission. -Select a published workflow with `forge:workflow:`; without that label, -Forge uses the built-in ticket-type workflow. Multiple workflow labels, an -unknown definition, or invalid content block execution rather than falling back. - -### Operator APIs - -These read/operate on durable state and require `FORGE_OPERATOR_TOKEN` as a -Bearer token. Without it the operator API is disabled (503); a missing or bad -token receives 401. - -```text -GET /api/v1/workflows/{ticket_key}/execution -GET /api/v1/workflows/{ticket_key}/execution/timeline?cursor=0&limit=50 -GET /api/v1/org-pulse/workflows/{ticket_key} -GET /api/v1/effects/workflow/{run_id} -GET /api/v1/effects/{idempotency_key} -POST /api/v1/effects/{idempotency_key}/replay -``` - -Effect replay is intentional operator recovery for a terminal effect; it does -not rerun an agent or blindly advance the workflow. - -## Release and operational implications - -1. Treat this as a major-version cutover. The stack removes legacy compatibility - paths. Drain or explicitly resolve active pre-2.0 workflows before deploying - it. Do not assume they can resume under the new runtime. - -2. Keep the existing services running: Redis, the Forge gateway, the host worker, - and (where used) forge-poller. Restart the gateway and worker after deploying - the code so the new routes, catalog, and worker dependencies are loaded. The - worker continues to own the Podman runtime. - -3. Preserve Redis. It now holds recovery/audit records as well as queues and - checkpoints. Do not apply a broad Redis flush as part of deployment. - -4. Configure `FORGE_OPERATOR_TOKEN` before relying on the new inspection or - replay APIs. Restrict the token to trusted operators; replay performs an - external provider mutation. - -5. Continue to configure production webhook secrets. The gateway validates Jira - and source-control signatures when their secrets are configured; production - should configure both. Poller deliveries must use the intended forwarding - configuration. - -6. Review custom definitions and integrations. Definitions must be flow-only and - use catalog-listed names. Provider mutations must cross the effect service; - old extensions that call Jira/GitHub or mutate checkpoints directly must be - migrated to stations/effects/command handlers. - -7. Update runbooks and alerts. A `blocked` workflow may now mean an explicit - effect precondition/terminal failure, invalid station output, unrecognized - command, stale/conflicting observation, or a missing definition route. Use - the execution timeline before retriggering a ticket. - -## Integration hardening applied during validation - -The integration branch also includes follow-up fixes discovered by exercising -Feature, Task Takeover, and Bug workflows. They should ship with the stack: - -- approval-gate resume schedules the intended next transition; -- required effects wait briefly for a concurrent recovery sweep that owns the - same idempotent write, while still failing closed on retryable/terminal errors; -- workspace setup and shared implementation have the declared effect authority - required for repository persistence; -- merged pull requests reconcile as a terminal event without depending on an - otherwise unknowable head SHA; -- feature decomposition no longer needs a duplicate Jira draft attachment—the - workflow checkpoint is authoritative; -- task-plan revisions reconcile repository labels safely; and -- bug RCA now selects one configured repository in structured output and writes - the matching `repo:/` Jira label only after validation. - -## Bottom line - -Forge still automates the same delivery workflow. Forge 2.0 makes its control -decisions, external writes, and operational evidence explicit, durable, and -inspectable. The cost is stricter contracts and a real major-version migration -boundary; the benefit is safe recovery and explainability when distributed -events, providers, agents, and workers inevitably retry or disagree. diff --git a/docs/guide/bug-workflow.md b/docs/guide/bug-workflow.md index 391bf50ab..5396eb3a5 100644 --- a/docs/guide/bug-workflow.md +++ b/docs/guide/bug-workflow.md @@ -57,7 +57,7 @@ Forge immediately acknowledges the ticket and evaluates it against a 7-field com ### 2. RCA Analysis -Forge spawns a container that clones the relevant repo(s) and performs hypothesis-driven codebase exploration: +Forge selects one configured repository for the RCA, applies its `repo:/` label after validation, and then performs hypothesis-driven codebase exploration against that repository: - Forms ranked candidate root causes - Investigates each with `grep`, `git blame`, and file reads @@ -83,7 +83,7 @@ Plain comments (no prefix) are ignored by the workflow. ### 4. Planning -After option selection, Forge spawns a container to produce a concrete implementation plan covering specific files, tests, and order of operations. Each involved repository is tagged `repo:/` for automatic task decomposition. +After option selection, Forge produces a concrete implementation plan covering specific files, tests, and order of operations. The selected repository is tagged `repo:/` for validated task decomposition; a revision does not remove that retained repository assignment. The plan is posted as a Jira comment and `forge:plan-pending` is set. @@ -156,3 +156,5 @@ Bug fix PRs are created from fork branches. If `main` advances while the PR is o | `forge:retry` | Human | Resume from the failed node | See [Jira Labels](labels.md) for the full reference. + +For blocked effects, provider-event conflicts, and safe replay, see [Operations](../operations.md). diff --git a/docs/guide/feature-workflow.md b/docs/guide/feature-workflow.md index 61469911b..f1ff143f6 100644 --- a/docs/guide/feature-workflow.md +++ b/docs/guide/feature-workflow.md @@ -68,7 +68,7 @@ Forge breaks the feature into logical epics — high-level areas of work that ma By default, Forge uses an interactive **Draft Review Flow** at this stage (unless YOLO mode is active): 1. Instead of creating Jira tickets immediately, Forge stores the proposed epics in durable workflow state. -2. Forge posts a markdown table comment on the Feature ticket outlining the proposed Epics. +2. Forge posts a markdown table comment on the Feature ticket outlining the proposed Epics. The comment is a review view; the workflow checkpoint is the authoritative draft. 3. The workflow pauses at `plan_approval_gate`. **Human action:** Review the epic plan draft. You have several options at this stage: @@ -104,7 +104,7 @@ Forge generates granular implementation tasks scoped to individual repositories. By default, Forge uses an interactive **Draft Review Flow** at this stage (unless YOLO mode is active): 1. Instead of creating Jira tickets immediately, Forge stores the proposed tasks in durable workflow state. -2. Forge posts a markdown table comment on the Feature ticket outlining the proposed Tasks. +2. Forge posts a markdown table comment on the Feature ticket outlining the proposed Tasks. The comment is a review view; the workflow checkpoint is the authoritative draft. 3. The workflow pauses at `task_approval_gate`. **Human action:** Review the task draft. You have several options at this stage: @@ -225,7 +225,7 @@ If a stage fails, Forge: 1. Sets the `forge:blocked` label 2. Posts a comment tagging the reporter and assignee with the error -To retry, add the `forge:retry` label. Forge resumes from the exact node that failed — not from the beginning. +To retry, add the `forge:retry` label. Forge resumes from the exact node that failed — not from the beginning. For an unresolved provider write, conflicting event, or effect replay, inspect the [operations guide](../operations.md) before retrying. !!! tip "CI retries" If CI fix attempts are exhausted, `forge:retry` resets the attempt counter for a fresh budget of retries. diff --git a/docs/guide/labels.md b/docs/guide/labels.md index 96188122a..a9fc6e25a 100644 --- a/docs/guide/labels.md +++ b/docs/guide/labels.md @@ -82,6 +82,8 @@ For stages using the draft-based review flow (Epic Plan and Tasks), you can post **Handling failures:** When `forge:blocked` appears, read the Forge comment for the error. Fix the underlying issue if needed, then add `forge:retry`. +For failures involving provider writes, duplicate/stale/conflicting events, or operator effect replay, use the [operations guide](../operations.md) before adding `forge:retry`. Retry resumes from Forge's durable saved position; it is not a request to rerun the whole lifecycle. + **Resetting contested reviews:** If the workflow is paused at `review_response_gate` due to contested comments, adding `forge:retry` will transition the workflow back to `human_review_gate`, clearing the contested comments and resetting the review state to await a fresh review. !!! warning "Don't remove `forge:managed`" diff --git a/docs/guide/pr-commands.md b/docs/guide/pr-commands.md index 7403e0359..22952dd39 100644 --- a/docs/guide/pr-commands.md +++ b/docs/guide/pr-commands.md @@ -71,11 +71,11 @@ Merge `main` into the PR branch and resolve any merge conflicts using AI. Use th ## When Commands Are Active -**Skip-gate and unskip-gate** only work when Forge's workflow is in a CI stage: +**Skip-gate and unskip-gate** only work when Forge's workflow is at a CI or review gate: -- `wait_for_ci_gate` - `ci_evaluator` - `attempt_ci_fix` +- `human_review_gate` **Rebase** works from any workflow stage where a PR exists in the workflow state. diff --git a/docs/guide/task-workflow.md b/docs/guide/task-workflow.md index 4e6c9423c..65e42361f 100644 --- a/docs/guide/task-workflow.md +++ b/docs/guide/task-workflow.md @@ -91,7 +91,7 @@ Review the posted implementation plan before Forge changes code. | Ask a question | Comment with `?` or `@forge ask` | | Leave paused | Keep `forge:plan-pending` | -Question comments route to Q&A mode and return to the same approval gate. Revision comments regenerate the plan with the previous plan and your feedback in context. +Question comments route to Q&A mode and return to the same approval gate. Revision comments regenerate the plan with the previous plan and your feedback in context. Forge retains the validated `repo:/` assignment across a plan revision; it does not remove and re-add the same label as a revision side effect. !!! tip "YOLO mode" If `forge:yolo` is present, Forge auto-approves the task plan and proceeds to workspace setup. Human PR review still remains a gate. @@ -187,5 +187,4 @@ If a stage fails, Forge records the error, sets `forge:blocked`, and posts an er To resume, add `forge:retry`. Forge resumes from the saved `current_node` instead of restarting the workflow from triage. For CI failures, retrying gives the workflow a fresh CI-fix budget. -Planning and triage have bounded retry behavior. Qualitative review also has a bounded retry loop and will proceed to PR creation after the retry cap when it has a non-adequate verdict, preserving that state for reviewers. - +Planning and triage have bounded retry behavior. Qualitative review also has a bounded retry loop and will proceed to PR creation after the retry cap when it has a non-adequate verdict, preserving that state for reviewers. Use [Operations](../operations.md) when the Jira error indicates an unresolved effect or conflicting provider observation rather than an implementation issue. diff --git a/docs/index.md b/docs/index.md index 3e13afdac..b256532b4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -47,9 +47,12 @@ graph TD - [Feature Workflow](guide/feature-workflow.md) — How features flow through Forge - [Bug Workflow](guide/bug-workflow.md) — How bug diagnosis and implementation flow through Forge - [Task Workflow](guide/task-workflow.md) — How standalone Tasks and Epics become PRs +- [Workflow Management](guide/labels.md) — Approvals, revisions, retries, and Jira/PR controls +- [Operations](operations.md) — Services, execution inspection, recovery, and incident response - [Developer Guide](developer-guide.md) — Full local development reference - [Architecture](architecture/index.md) — How workflow state, reconciliation, stations, and effects fit together - [Declarative Workflows](reference/declarative-workflows.md) — Compose registered Forge stages safely +- [Configuration](reference/config.md) — Repository, model, proposal, and deployment settings - [Skills System](skills/index.md) — Customize Forge for your stack - [Contributing](dev/contributing.md) — How to contribute @@ -91,4 +94,4 @@ Tailored for already-scoped, standalone tickets or task takeovers that bypass th : Customizable per-project AI behavior. Override only what's specific to your stack; defaults cover the rest. **Resumable Workflows** -: LangGraph checkpoints state to Redis after every step. Use `forge:retry` to resume from the exact node that failed. +: Forge records checkpoints, observations, and external effects durably. Use `forge:retry` to resume from the saved failed position after addressing the cause; use the [operations guide](operations.md) to inspect execution evidence first. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 000000000..cb70ad284 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,126 @@ +# Operations and workflow management + +Forge is a durable, event-driven workflow service. This guide explains what +operators run, how a ticket advances, and how to diagnose and safely recover a +workflow without repeating completed agent work or provider writes. + +## Runtime services + +| Component | Responsibility | +| --- | --- | +| Gateway | Validates Jira and source-control webhooks and places ingress on Redis Streams. It does not make workflow decisions. | +| Worker | Reconciles observations, validates commands, executes the ticket's pinned workflow, runs recovery, and writes the execution timeline. | +| Redis | Stores queue entries, LangGraph checkpoints, observation decisions, effect records, definition/pinning data, and timeline records. | +| Poller | A peer ingress source that observes Jira and source-control changes and forwards them to Forge. It improves recovery; it is not replaced by the gateway. | +| Podman | Runs implementation work in short-lived, rootless containers. The worker remains responsible for their lifecycle. | + +The observation ledger, command boundary, definition registry, station runtime, +effect journal, and execution read model are logical control-plane services. +They run with the gateway/worker deployment rather than as additional required +containers. + +## Deploying and changing configuration + +Run Redis, the gateway, and at least one worker. Run forge-poller where polling +is part of the deployment. After deploying code or changing service +configuration, restart the gateway and workers so routes, registries, and +settings are reloaded. + +Preserve Redis during upgrades. It contains recovery and audit records in +addition to queue messages and checkpoints. A Redis flush destroys the evidence +needed to determine whether an external write occurred and can make active +workflows unrecoverable. + +`repos.yaml` is loaded once per process. Restart the gateway and worker after +changing it; there is no runtime registry reload. See [configuration](reference/config.md) +for repository, proposal, model, and environment settings. + +## How incoming events become workflow work + +Webhooks and poller events are at-least-once deliveries. Forge converts each +delivery into an observation, then records a reconciliation decision before +interpreting it: + +| Decision | Meaning | Operator action | +| --- | --- | --- | +| Accepted | The provider fact is new and coherent. | No action; the resulting command may advance the workflow. | +| Duplicate | The same provider fact arrived again. | No action. It is intentionally convergent. | +| Stale | A provider revision is older than known evidence. | Investigate provider ordering only if it is unexpected. | +| Conflict | The event cannot safely be ordered or contradicts accepted evidence. | Inspect the timeline and provider record; do not force the workflow forward. | + +An accepted observation becomes a command only when it is meaningful at the +saved position of the ticket's pinned workflow. Jira/GitHub facts never set a +workflow position directly. + +## Inspecting execution + +Set `FORGE_OPERATOR_TOKEN` to enable execution and Org Pulse inspection. Set +`EFFECT_OPERATOR_TOKEN` separately to enable durable-effect inspection and +replay. Requests require the corresponding value as a Bearer token. When the +relevant token is not configured the interface returns `503`; a missing or +invalid token returns `401`. + +```text +GET /api/v1/workflows/{ticket_key}/execution +GET /api/v1/workflows/{ticket_key}/execution/timeline?cursor=0&limit=50 +GET /api/v1/org-pulse/workflows/{ticket_key} + +# Requires EFFECT_OPERATOR_TOKEN +GET /api/v1/effects/workflow/{run_id} +GET /api/v1/effects/{idempotency_key} +POST /api/v1/effects/{idempotency_key}/replay +``` + +Start with the execution view, then use the timeline to find the accepted +observation, command decision, station attempt, or effect that explains the +current wait or failure. These read endpoints do not advance a workflow or +recompute state from live provider data. + +## Effects, retries, and replay + +Every workflow-visible Jira, source-control, or repository mutation is a +durable effect. Forge records the intent, leases execution, calls the provider, +and retains attempt and provider evidence under a stable idempotency key. + +- **Pending/running effect:** a worker or recovery sweep owns the write. Wait + for it to settle; a concurrent lease is not a second failure. +- **Retryable failure:** Forge retries with bounded backoff. The workflow fails + closed while a required effect remains unresolved. +- **Terminal/precondition failure:** correct the underlying provider, + repository, or configuration problem before recovery. +- **Replay:** `POST .../replay` requeues an eligible terminal effect. It repeats + only that provider mutation; it does not rerun an agent or advance the graph. + +Use replay only after confirming the intended write and correcting its cause. +Do not use it as a substitute for inspecting the effect history. + +## Managing blocked workflows + +When Forge cannot safely continue, it adds `forge:blocked` and posts a Jira +comment with the failure. Common causes include an unresolved required effect, +an invalid station result, an unrecognized command, conflicting provider +evidence, a missing workflow route, or incomplete repository configuration. + +1. Read the Jira error and execution timeline. +2. Correct the root cause (for example, repository configuration or provider + permission), or replay the specific eligible terminal effect. +3. Add `forge:retry` to request a fresh run from the saved failed position. + +`forge:retry` does not restart the ticket from intake. It clears the blocked +state after command validation and resumes from the durable checkpoint. It also +resets a depleted CI-fix budget. At `review_response_gate`, it clears contested +review comments and returns the ticket to human review. + +See [Jira labels](guide/labels.md) and [PR commands](guide/pr-commands.md) for +the human controls available to each workflow. + +## Monitoring and incident response + +Use the execution timeline for ticket-specific causality, worker/gateway logs +for process diagnostics, and the existing Prometheus, Langfuse, and Grafana +views for throughput, latency, model use, CI behavior, and system health. + +During an incident, preserve Redis, record the ticket/run/effect identities, +and inspect provider evidence before retrying. Restarting a worker is safe when +durable records are intact; blindly re-running agents or deleting state is not +an equivalent recovery procedure. diff --git a/docs/reference/api.md b/docs/reference/api.md index 0f71d809e..cc9771f8c 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -138,6 +138,23 @@ request/sample histograms by design. Worker metrics are available separately at `http://localhost:8001/metrics`. +### Durable effect API + +Effect inspection and replay are deliberately separate from execution reads. +They require `EFFECT_OPERATOR_TOKEN` as a Bearer token; the routes return `503` +when it is unset and `401` for a missing or invalid token. + +```http +GET /api/v1/effects/workflow/{run_id} +GET /api/v1/effects/{idempotency_key} +POST /api/v1/effects/{idempotency_key}/replay +``` + +Replay is an operator recovery action. It requeues one eligible terminal effect +using its existing idempotency identity; it does not rerun an agent or advance a +workflow. Inspect the effect's attempts and provider evidence before replaying. +See [Operations](../operations.md) for effect states and blocked-workflow triage. + ## Webhook Configuration ### Jira diff --git a/docs/reference/config.md b/docs/reference/config.md index c7f7a3e5f..9f22968e5 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -228,6 +228,16 @@ global stage mapping, then the global default, and finally the legacy |----------|---------|-------------| | `REDIS_URL` | `redis://localhost:6380/0` | Redis connection URL | +### Operator APIs + +| Variable | Description | +| --- | --- | +| `FORGE_OPERATOR_TOKEN` | Bearer token required for execution and Org Pulse read APIs. The routes are disabled when it is empty. | +| `EFFECT_OPERATOR_TOKEN` | Bearer token required for durable-effect inspection and replay APIs. The routes are disabled when it is unset. | + +Use distinct values when different operators should have workflow-read versus +effect-replay authority. See [operations](../operations.md) for recovery rules. + ## Per-Project Repository Configuration !!! warning "Production requirement" @@ -264,6 +274,31 @@ curl -X PUT \ -d '"org/repo1"' ``` +Repository labels on managed tickets use `repo:/`. Forge validates +that assignment against the project's configured repositories before workspace +setup or implementation. A missing or invalid assignment blocks the workflow +instead of selecting a repository implicitly. + +If the deployment uses `FORGE_REPOS_CONFIG_PATH` to load a `repos.yaml` +registry, the process caches that registry for its lifetime. Restart the gateway +and every worker after changing the file. See [operations](../operations.md) +for the safe deployment and recovery model. + +## Proposal review configuration + +Projects can opt into GitHub pull-request review for PRDs and specifications. +Set `forge.prd_proposals_repo` to an `owner/repo` repository and optionally set +`forge.prd_proposals_path` to a base directory. Forge then creates `prd.md` and +`design.md` under `{path}/{TICKET}/` on separate proposal branches; merge is +approval and review feedback requests regeneration. + +Use `forge project-setup MYPROJ --prd-proposals-repo owner/repo` to configure +the repository and `--prd-proposals-path path` to configure the base path. Set +either option to an empty value to remove/reset it. When project configuration +is not required, `PRD_PROPOSALS_REPO` and `PRD_PROPOSALS_PATH` provide global +fallbacks. See [proposal review](proposals.md) for the distinction between this +workflow behavior and core-project design proposals. + ## Local Development Overrides Use these to skip the Jira project property requirement during local development: diff --git a/docs/reference/declarative-workflows.md b/docs/reference/declarative-workflows.md index 686ed76c9..4938f8df7 100644 --- a/docs/reference/declarative-workflows.md +++ b/docs/reference/declarative-workflows.md @@ -4,6 +4,10 @@ Forge project administrators can compose the nodes and state profiles shipped wi project-specific workflows. Definitions are selected by Jira label, validated before compilation, and compiled into LangGraph graphs at runtime. They cannot import Python or define expressions. +This is project-level configuration, not a plugin system. A definition controls flow; trusted +Forge code controls authority. To add a provider adapter, a station, a node, or an effect executor, +use the [core-maintainer extension path](../architecture/control-plane.md#core-maintainers-trusted-capabilities). + ## Author and publish Create a YAML file locally: @@ -39,6 +43,17 @@ forge workflow validate workflow.yaml forge workflow publish MYPROJ workflow.yaml ``` +Before authoring a definition, inspect the supported catalog and before activating a revision, +inspect the resulting topology and migration impact: + +```bash +forge workflow catalog feature +forge workflow validate workflow.yaml --json +forge workflow render workflow.yaml +forge workflow diff previous.yaml workflow.yaml +forge workflow simulate-migration previous.yaml workflow.yaml instances.json +``` + Publishing stores canonical JSON in the `forge.workflow.prd-only` Jira project property. Jira requires the credentials used by the command to have global or project administration permission. The canonical value must fit Jira's 32,768-byte project-property limit. @@ -134,3 +149,31 @@ forge workflow list MYPROJ forge workflow show MYPROJ prd-only forge workflow show-history MYPROJ prd-only ``` + +## Configuration boundary + +Project authors can configure these flow-level choices: + +- the built-in state profile (`feature`, `bug`, or `task_takeover`); +- registered steps, fixed edges, router branches, joins, dynamic fan-out, retry + bounds, and concurrency; and +- revision/resume mappings for explicitly migratable saved positions. + +The trusted catalog, not a project definition, owns node kind, station contract, +effect operations, required and mandatory policies, observation policy, +preconditions, provider credentials, and external command handling. A definition +cannot grant a node permission to push a branch, create a Jira issue, bypass an +approval, or make arbitrary network, shell, or Python calls. + +## Safe revision checklist + +1. Start from a built-in definition with the same state profile. +2. Run `forge workflow catalog STATE` and use only listed nodes and routers. +3. Increment `metadata.revision` for every content change. +4. Validate and render the candidate definition. +5. Diff it against the active revision. +6. When a saved node changes, add `spec.resume.fromRevisions` and run migration + simulation against representative active instances. +7. Publish only after reviewing the resulting canonical JSON and migration + result. Existing tickets remain pinned to their prior definition unless a + declared migration applies. diff --git a/docs/reference/proposals.md b/docs/reference/proposals.md index b01180d6f..5a4912d02 100644 --- a/docs/reference/proposals.md +++ b/docs/reference/proposals.md @@ -1,5 +1,10 @@ # Proposals +This page describes design proposals for changes to Forge itself. It is separate +from [PRD and specification proposal review](config.md#proposal-review-configuration), +which lets a Jira project review generated planning artifacts in a GitHub +repository. + Forge uses a lightweight proposal process for significant changes. Proposals live in the [`proposals/`](https://github.com/forge-sdlc/forge/tree/main/proposals) directory of the repository. ## Creating a Proposal diff --git a/zensical.toml b/zensical.toml index 8ffb80304..739e4c165 100644 --- a/zensical.toml +++ b/zensical.toml @@ -9,6 +9,7 @@ edit_uri = "edit/main/docs/" nav = [ {"Home" = "index.md"}, {"Getting Started" = "getting-started.md"}, + {"Operations" = "operations.md"}, {"Product Roadmap" = "roadmap.md"}, {"User Guide" = [ {"Feature Workflow" = "guide/feature-workflow.md"}, @@ -26,6 +27,7 @@ nav = [ {"Overview" = "developer-guide.md"}, {"Architecture" = [ {"Overview" = "architecture/index.md"}, + {"Control-plane Architecture" = "architecture/control-plane.md"}, {"System and Components" = "architecture/overview.md"}, {"Runtime Internals" = "architecture/internals.md"}, {"Structured Model Output" = "architecture/structured-output.md"},