diff --git a/docs/architecture/option-b-completion-plan.md b/docs/architecture/option-b-completion-plan.md index 5b7fa8aba..0ce7dbaeb 100644 --- a/docs/architecture/option-b-completion-plan.md +++ b/docs/architecture/option-b-completion-plan.md @@ -107,7 +107,7 @@ Exit gate: every supported station runs through the same typed boundary locally ## Phase 5 — Versioned process definitions and governance -PR: #328. Status: partial. +PR: #328. Status: complete. Purpose: make the golden path explicit, inspectable, versioned, and enforceable rather than implicit in Python topology. diff --git a/docs/architecture/phase-5-process-definition-plan.md b/docs/architecture/phase-5-process-definition-plan.md new file mode 100644 index 000000000..329c40d0f --- /dev/null +++ b/docs/architecture/phase-5-process-definition-plan.md @@ -0,0 +1,59 @@ +# Phase 5 implementation plan: explicit process definition + +**Status:** Complete + +**Depends on:** Versioned contracts and the Phase 4 station boundary + +**Goal:** Make the executable process inspectable without reading predicates, Python +functions or LangGraph state. The Forge-owned definition remains authoritative; +LangGraph is one compiler target rather than the process model itself. + +## Delivery slices + +1. **Runtime-independent manifest.** Compile workflow YAML into a canonical process + manifest containing node roles, station contracts, gates and labelled transitions. +2. **Visualization.** Render the same validated manifest as Mermaid or JSON for review, + documentation and Org Pulse integration. +3. **Change impact.** Compare revisions and report added/removed nodes, changed routing, + contract changes and missing resume mappings that can strand in-flight work. +4. **Golden-path publication.** Publish Forge's supported feature, bug and task-takeover + definitions as versioned manifests rather than retaining topology only in Python. +5. **Governance and rollout.** Validate mandatory gates/contracts, compatibility policy, + supported extension points and revision rollout before publication. + +## Delivered + +PR #328 now ships feature, bug, and task-takeover as checked-in canonical process +artifacts. JSON inspection, Mermaid rendering, LangGraph compilation, revision comparison, +and runtime selection consume those same artifacts and digests; Python graph builders are +no longer the default topology authority. + +The compiler validates mandatory gates and policies, registered station contracts, +complete routing outcomes, effect capabilities, joins, bounded fan-out, retry limits, +reachability, and safe cycles. Compiled execution enforces declared effect capabilities, +dynamic targets, concurrency limits, and retry bounds. + +Each new instance persists the complete immutable definition identity and resumes against +that pinned artifact. Revision adoption is an explicit migration operation. Change impact +uses patch/compatible/migratable/breaking classifications, and the migration simulator +reports eligibility for each active checkpoint before rollout. + +Publication is project-scoped and immutable. Publish, activate, and rollback are separate +CAS-protected, append-only audited decisions; the CLI cannot overwrite or delete active +history. The same behavior is verified against a real Redis server. + +The governance and rollout requirements for these definitions are specified in the +[Workflow-definition governance policy](phase-5-workflow-definition-governance.md). The +policy covers golden paths, custom definitions, ownership and review, mandatory contracts, +effect capabilities, immutable publication/activation, compatibility, migration, and +operational evidence. + +## Exit evidence + +- Built-in artifact round-trip and digest snapshots prove the packaged process is the + process compiled by the default runtime. +- Governance tests cover mandatory gates/contracts/outcomes, effect authorization, + fan-out cardinality, retries, immutable publication, CAS activation, and rollback. +- Pinning and migration tests prove active instances cannot silently adopt a changed + definition and produce deterministic per-instance dry-run reports. +- Workflow, architecture, and status-transition regression suites remain green. diff --git a/docs/architecture/phase-5-workflow-definition-governance.md b/docs/architecture/phase-5-workflow-definition-governance.md new file mode 100644 index 000000000..77404b0b8 --- /dev/null +++ b/docs/architecture/phase-5-workflow-definition-governance.md @@ -0,0 +1,201 @@ +# Workflow-definition governance policy + +**Status:** Active + +This policy governs every Forge `WorkflowDefinition`, whether it is shipped by Forge or +published by a project administrator. A definition is a release artifact: it is compiled, +validated, reviewed, and published as canonical JSON before it can be selected by a new +workflow instance. LangGraph is an execution target and cannot weaken these rules. + +## Definition classes + +Forge has two classes of definitions: + +| Class | Owner and purpose | Permitted change path | +| --- | --- | --- | +| **Golden path** | Forge owns the feature, bug, and task-takeover processes, including their state profiles, required gates, station contracts, and effect policy. | A Forge code/release change publishes the definition through the same compiler used for custom definitions. Golden paths are the compatibility baseline. | +| **Custom** | A project administrator composes a supported state profile from Forge-registered stations, routers, gates, joins, and extension points. The project owns its operational choice; Forge owns the runtime contracts and safety policy. | Validate and publish through the workflow-definition API/CLI. Custom definitions cannot add Python, expressions, arbitrary imports, or unregistered topology. | + +Custom definitions may omit optional golden-path stages, but may not bypass a mandatory +policy, contract, approval boundary, or effect restriction. A custom definition that needs a +new station, router, state profile, or effect capability is an extension proposal, not a +custom YAML change; it requires a Forge-reviewed extension and a new registered catalog +entry first. + +## Ownership and review + +Every definition has one accountable owner and a named operational contact in its release +metadata. Ownership is not delegated by merely granting project administration permission. + +- Forge maintainers own golden-path definitions, the catalog of stations/routers/gates, + compatibility rules, and the mandatory policy set. +- The project administrator owns a custom definition's intent, project selection label, + rollout scope, migration decision, and incident response contact. +- The platform/operator team owns publication storage, activation controls, revision + retention, audit logs, and rollback execution. +- Security/reliability reviewers must approve any change to effect capabilities, external + writes, approval semantics, joins/concurrency, or recovery behavior. + +The required review level is determined before publication: + +1. A documentation-only or description change still requires the definition owner and one + peer reviewer. +2. A compatible topology or routing change requires the owner, a Forge workflow reviewer, + and an automated validation report. +3. A policy, contract, capability, migration, or breaking change requires the owner, a + Forge maintainer, and a security/reliability reviewer. The release record must include + an impact report, migration simulation, rollout/rollback plan, and named approvers. + +No author may self-approve a change that they classify as breaking. Emergency publication +is permitted only to restore service or remove an unsafe capability; it must preserve an +immutable revision and receive retrospective review within one business day. + +## Mandatory policies and contracts + +Publication fails closed unless the definition declares a supported state profile and passes +all of the following checks: + +- Every node, router, gate, join, and extension is in the Forge registry for that profile; + every station contract and version matches the state profile's registered binding, and + every routed outcome is covered by the reviewed routing contract. +- Every station outcome has an explicit route or terminal handling. Unknown outcomes, + implicit fall-through, unreachable nodes, unbounded transitions, and unguarded cycles are + rejected. Cycles must cross an approved human or CI pause boundary and have a bounded + retry policy. +- Required preconditions are declared and evaluated before a station can request an + external effect. A missing repository, workspace, pull request, approval, or other + structural input blocks the station rather than being inferred. +- Required gates remain present for the profile. For example, artifact approval and the + implementation/review/CI boundaries cannot be replaced with a direct edge to an + external write. A definition must explicitly declare whether code changes, a pull + request, CI, and human review are expected when those capabilities are optional. +- Join steps declare `all` or `any` and are rejected without multiple incoming paths. + Dynamic fan-out declares every target and a maximum cardinality; runtime routing rejects + undeclared targets or branch counts above that bound. +- The compiler emits a canonical manifest and digest; the review report includes the + rendered topology, contract versions, policy decisions, effect capabilities, and change + impact against the prior revision. + +These checks apply identically to built-in and custom definitions. A project cannot turn a +mandatory policy off through metadata or an extension point. + +## Effect capabilities and extension points + +Definitions request named capabilities, not provider clients. The initial finite allowlist is: + +- `jira.comment`, `jira.labels`, and `jira.status` for workflow signalling; +- `jira.issue_content`, `jira.issue_lifecycle`, and `jira.issue_structure` for bounded + issue mutations; +- `jira.project_configuration` for explicitly governed project metadata; and +- `source_control.branch`, `source_control.commit`, `source_control.pull_request`, and + `source_control.review` for repository and review mutations. + +Each capability is scoped to the workflow instance, repository/project identity, station +contract, and effect idempotency key. Read-only observations do not grant a write +capability. A station may request only capabilities declared by its catalog entry and the +definition; the durable effect journal remains the only path to an external mutation. +Custom definitions may use existing capabilities subject to project policy. They may not +introduce provider-specific operations, arbitrary HTTP, credentials, shell execution, or +an effect implementation in YAML. Supported extension points are registered station +contracts, routers, gates, join strategies, state-profile fields, and durable effect +capability descriptors. Every extension documents its input/output schema, failure and +retry behavior, authorization scope, idempotency key, and compatibility class. + +## Immutable publication and activation + +Publication and activation are separate operations: + +1. The author submits a definition with a strictly increasing revision. Forge canonicalizes + it, validates it, computes its digest, and stores the complete artifact and validation + report as `published`. +2. Publication is rejected if content changes without a revision increment, if the digest + already identifies different content, or if mandatory review/evidence is missing. +3. An operator or release automation explicitly activates one published revision for a + project/definition name, optionally with a canary scope and start/end time. Activation + affects only new instances unless an approved migration is separately executed. +4. Each instance stores the definition name, revision, digest, and activation context at + creation. Resume uses that pinned immutable artifact; deleting or replacing the active + pointer cannot change an in-flight instance. +5. Published revisions are immutable and retained for the maximum checkpoint lifetime plus + the audit-retention period. Rollback activates an earlier revision; it never edits or + reuses a revision number. + +Activation is blocked when validation, compatibility, migration, or canary evidence is +missing. A removed definition remains readable for pinned instances until they finish, +expire, or are explicitly migrated. + +## Compatibility classification + +The impact report assigns exactly one class to every revision change: + +| Class | Examples | Existing instances | Rollout requirement | +| --- | --- | --- | --- | +| **Patch** | Metadata/description change, or a non-executable canonicalization that preserves digest-relevant behavior. | No migration; pinned instances continue unchanged. | Normal review and validation. | +| **Compatible** | Add an unreachable optional node/branch, add an optional field with a default, or add a backward-compatible station contract. | Continue on their pinned revision; opt-in migration only if a resume map is supplied. | Impact report and canary activation. | +| **Migratable** | Rename/replace a node with equivalent state, reorder work after a safe boundary, or change a contract with a deterministic state conversion. | Remain pinned until an approved migration maps every affected checkpoint. | Dry-run simulation, per-instance eligibility, operator approval, and rollback window. | +| **Breaking** | Remove a reachable node/outcome, alter state meaning, mandatory gate, effect capability, join semantics, or contract incompatibly. | Never silently adopt. Pause or complete on the old revision, or use an explicitly approved migration. | Security/reliability review, migration or drain plan, canary, and explicit activation decision. | + +If classification is uncertain, use the more restrictive class. A revision rollback is +breaking for instances that have already observed the newer topology unless a compatibility +analysis proves otherwise. + +## Migration and resume mappings + +Before activating a migratable or breaking revision, the owner supplies a mapping for every +checkpoint shape that can exist in production. A mapping identifies old revision and node, +new revision and node, state-field conversions/defaults, outstanding gate/effect behavior, +and whether the instance is eligible. The migration simulator must exercise completed, +waiting, retrying, fan-out, join, and failure states and report unmapped or ambiguous cases. + +Migration is transactional per instance: acquire the workflow lock, validate the pinned +artifact and mapping, write a migration event and new checkpoint, then release the lock. +An effect that is pending or indeterminate is not replayed merely because a node was +renamed; its original effect identity and result remain authoritative. Ineligible +instances stay on the old revision or are placed in an operator-visible blocked state. +Resume mappings are part of the immutable revision artifact and cannot be supplied after +activation without publishing a new revision. + +## Deprecation and breaking changes + +Deprecation is announced with a replacement revision, owner, end-of-new-instance date, +checkpoint drain deadline, and migration instructions. During deprecation, new instances +may be blocked or routed to the replacement, but pinned instances continue while the old +artifact is retained. Force-expiring an instance requires an incident/owner decision and an +audit record of its recovery or data-loss implications. + +Breaking changes require a migration or an explicit drain. The release record must state +which instances are affected, how approvals and effects are preserved, how a failed +migration is recovered, and when the old revision can be retired. Removing the canonical +artifact, changing its digest, or silently adopting a new revision is never a valid +breaking-change procedure. + +## Rollout, rollback, and audit evidence + +The operator records a pre-activation snapshot, validation output, rendered manifest, +compatibility classification, migration simulation, approvers, target scope, canary +metrics, and rollback trigger. Canary activation starts with a bounded project or instance +cohort and must observe error rate, blocked/resume rate, station contract failures, effect +retries, and unexpected routes before expansion. + +Rollback means activating a previously published immutable revision and stopping further +migration. It does not rewrite checkpoints or cancel durable effects. If instances were +migrated, the rollback record must include a reverse mapping or leave those instances on +the migrated revision while new instances use the prior one. Indeterminate external +effects are reconciled through the effect journal before retry or compensation. + +Operational audit evidence is append-only and queryable by definition name, revision, +digest, project, and workflow instance. At minimum retain: + +- author, owner, reviewers, approvers, timestamps, source commit, canonical artifact, and + validation/compiler version; +- publication and activation/deactivation events, target scope, canary observations, + policy decisions, and rollback trigger; +- instance creation/resume with pinned revision, migration eligibility and mapping, + migration result, blocked reason, and operator action; and +- station contract decisions, transition/outcome decisions, join results, effect IDs and + attempts/results, and links to incident or recovery records. + +Forge publication and activation require an actor and reason, retain the canonical +artifact and impact report, and record the decision append-only. Organizational release +automation is responsible for attaching the additional review, canary, and incident links +required above to that actor/reason evidence before invoking the governed API. diff --git a/src/forge/cli.py b/src/forge/cli.py index ba64bc39c..42972b51a 100644 --- a/src/forge/cli.py +++ b/src/forge/cli.py @@ -1701,9 +1701,57 @@ def main(argv: list[str] | None = None) -> int: workflow_validate.add_argument("file") workflow_validate.add_argument("--json", action="store_true", help="Print canonical JSON") + workflow_render = workflow_subparsers.add_parser( + "render", help="Render a validated workflow process manifest" + ) + workflow_render.add_argument("file") + workflow_render.add_argument("--format", choices=("mermaid", "json"), default="mermaid") + + workflow_diff = workflow_subparsers.add_parser( + "diff", help="Report structural and in-flight impact between revisions" + ) + workflow_diff.add_argument("previous") + workflow_diff.add_argument("current") + + workflow_simulate = workflow_subparsers.add_parser( + "simulate-migration", + help="Dry-run a definition change against active instance snapshots", + ) + workflow_simulate.add_argument("previous") + workflow_simulate.add_argument("current") + workflow_simulate.add_argument("instances", help="JSON array of active checkpoint snapshots") + workflow_publish = workflow_subparsers.add_parser("publish", help="Publish a YAML workflow") workflow_publish.add_argument("project_key") workflow_publish.add_argument("file") + workflow_publish.add_argument("--actor", default="forge-cli") + workflow_publish.add_argument("--reason", default="CLI publication") + + workflow_activate = workflow_subparsers.add_parser( + "activate", help="Activate an already-published workflow revision" + ) + workflow_activate.add_argument("project_key") + workflow_activate.add_argument("name") + workflow_activate.add_argument("revision", type=int) + workflow_activate.add_argument("--actor", default="forge-cli") + workflow_activate.add_argument("--reason", default="CLI activation") + workflow_activate.add_argument( + "--expected-active-digest", + help="Fail if the active definition digest has changed since it was read", + ) + + workflow_rollback = workflow_subparsers.add_parser( + "rollback", help="Activate a previously published compatible revision" + ) + workflow_rollback.add_argument("project_key") + workflow_rollback.add_argument("name") + workflow_rollback.add_argument("revision", type=int) + workflow_rollback.add_argument("--actor", default="forge-cli") + workflow_rollback.add_argument("--reason", default="CLI rollback") + workflow_rollback.add_argument( + "--expected-active-digest", + help="Fail if the active definition digest has changed since it was read", + ) workflow_show = workflow_subparsers.add_parser("show", help="Show one project workflow") workflow_show.add_argument("project_key") @@ -1713,6 +1761,13 @@ def main(argv: list[str] | None = None) -> int: workflow_list = workflow_subparsers.add_parser("list", help="List project workflows") workflow_list.add_argument("project_key") + workflow_history = workflow_subparsers.add_parser( + "show-history", help="Show immutable publication and rollout audit history" + ) + workflow_history.add_argument("project_key") + workflow_history.add_argument("name") + workflow_history.add_argument("--json", action="store_true") + workflow_delete = workflow_subparsers.add_parser("delete", help="Delete a project workflow") workflow_delete.add_argument("project_key") workflow_delete.add_argument("name") diff --git a/src/forge/orchestrator/worker.py b/src/forge/orchestrator/worker.py index bf3e6d7fb..a15d34310 100644 --- a/src/forge/orchestrator/worker.py +++ b/src/forge/orchestrator/worker.py @@ -619,14 +619,23 @@ async def _process_workflow(self, message: QueueMessage) -> None: and existing_state and existing_state.values ): + values = dict(existing_state.values) + # A pinned artifact is immutable: validate it and continue on + # that exact graph. Revision adoption belongs to the explicit + # migration operation, never to ordinary event handling. try: - migrated = workflow_instance.migrate_state(dict(existing_state.values)) + status = workflow_instance.pin_status(values) + if status == "pinned": + workflow_instance.validate_pinned_state(values) + elif status == "legacy_unpinned": + # Compatibility for checkpoints predating definition + # pinning is explicit and auditable in the state. + pinned = workflow_instance.pin_legacy_state(values) + await compiled_workflow.aupdate_state(config, pinned) + existing_state = await compiled_workflow.aget_state(config) except Exception as exc: await self._report_custom_workflow_configuration_error(ticket_key, str(exc)) return - if migrated != existing_state.values: - await compiled_workflow.aupdate_state(config, migrated) - existing_state = await compiled_workflow.aget_state(config) # Debug logging for checkpoint state logger.debug(f"Existing state for {ticket_key}: {existing_state}") @@ -2060,7 +2069,13 @@ async def _find_workflow_by_state(self, ticket_key: str) -> tuple[Any, Any]: async def _resolve_custom_workflow( self, ticket_key: str, labels: list[str] ) -> DeclarativeWorkflow | None: - """Resolve a pinned custom identity or the workflow selected by a ticket label.""" + """Resolve a pinned identity or a workflow selected by a label. + + Pinned checkpoints carry their canonical artifact, so resuming one does + not consult the mutable Jira project property. Identity-only checkpoints + use the publication store and fail closed if that exact artifact is + unavailable. + """ raw_checkpoint: dict[str, Any] | None = None config = {"configurable": {"thread_id": ticket_key}} with contextlib.suppress(Exception): @@ -2077,9 +2092,32 @@ async def _resolve_custom_workflow( if not workflow_name: return None + revision = values.get("workflow_definition_revision", values.get("workflow_revision")) + digest = values.get("workflow_definition_digest", values.get("workflow_digest")) + canonical = values.get("workflow_definition") + if revision is not None or digest is not None or canonical is not None: + from forge.workflow.declarative.publication import DefinitionPublisher + + return await load_project_workflow( + None, + str(project_key), + str(workflow_name), + pinned_revision=int(revision) if revision is not None else None, + pinned_digest=str(digest) if digest is not None else None, + pinned_definition=canonical, + definition_reader=DefinitionPublisher(str(project_key)), + ) + jira = JiraClient() try: - return await load_project_workflow(jira, str(project_key), str(workflow_name)) + from forge.workflow.declarative.publication import DefinitionPublisher + + return await load_project_workflow( + jira, + str(project_key), + str(workflow_name), + definition_reader=DefinitionPublisher(str(project_key)), + ) finally: await jira.close() @@ -2271,13 +2309,38 @@ async def run_single_ticket(ticket_key: str) -> dict[str, Any]: issue.labels ) if workflow_name: - workflow_instance: Any = await load_project_workflow( - jira, + from forge.workflow.declarative.publication import DefinitionPublisher + + workflow_instance: Any + project_key = ( checkpoint_values.get("workflow_project_key") or issue.project_key - or ticket_key.split("-", 1)[0], - workflow_name, + or ticket_key.split("-", 1)[0] + ) + revision = checkpoint_values.get( + "workflow_definition_revision", checkpoint_values.get("workflow_revision") ) + digest = checkpoint_values.get( + "workflow_definition_digest", checkpoint_values.get("workflow_digest") + ) + canonical = checkpoint_values.get("workflow_definition") + if revision is not None or digest is not None or canonical is not None: + workflow_instance = await load_project_workflow( + None, + project_key, + workflow_name, + pinned_revision=int(revision) if revision is not None else None, + pinned_digest=str(digest) if digest is not None else None, + pinned_definition=canonical, + definition_reader=DefinitionPublisher(project_key), + ) + else: + workflow_instance = await load_project_workflow( + jira, + project_key, + workflow_name, + definition_reader=DefinitionPublisher(project_key), + ) if not workflow_instance.supports_ticket_type(ticket_type): raise ValueError( f"workflow '{workflow_name}' is incompatible with ticket type " @@ -2316,7 +2379,12 @@ async def run_single_ticket(ticket_key: str) -> dict[str, Any]: **initial_state, } if checkpoint_values: - initial_state = workflow_instance.migrate_state(checkpoint_values) + status = workflow_instance.pin_status(checkpoint_values) + if status == "pinned": + workflow_instance.validate_pinned_state(checkpoint_values) + initial_state = dict(checkpoint_values) + elif status == "legacy_unpinned": + initial_state = workflow_instance.pin_legacy_state(checkpoint_values) # Use ticket_key as thread_id for checkpointing config: dict[str, Any] = checkpoint_config diff --git a/src/forge/workflow/base.py b/src/forge/workflow/base.py index f5de2dcb8..bd12dceb3 100644 --- a/src/forge/workflow/base.py +++ b/src/forge/workflow/base.py @@ -131,9 +131,16 @@ class BaseState(TypedDict, total=False): workflow_name: str workflow_revision: int workflow_digest: str + # Canonical names for the immutable process artifact. The shorter + # workflow_* fields above remain for checkpoint compatibility. + workflow_definition_revision: int + workflow_definition_digest: str + workflow_definition: dict[str, Any] + workflow_pin_status: str workflow_state_profile: str workflow_project_key: str workflow_transition_count: int + workflow_node_attempts: dict[str, int] # Generic node-contract capabilities and durable precondition audit trail. # Missing capability keys preserve legacy inference; explicit booleans are diff --git a/src/forge/workflow/declarative/__init__.py b/src/forge/workflow/declarative/__init__.py index 44c353673..77a772b70 100644 --- a/src/forge/workflow/declarative/__init__.py +++ b/src/forge/workflow/declarative/__init__.py @@ -2,13 +2,43 @@ from forge.workflow.declarative.compiler import DeclarativeWorkflowCompiler from forge.workflow.declarative.loader import load_workflow_file, load_workflow_value +from forge.workflow.declarative.manifest import ( + ProcessChangeClassification, + ProcessChangeImpact, + ProcessInstanceSnapshot, + ProcessManifest, + ProcessMigrationClassification, + ProcessMigrationSimulation, + build_process_manifest, + compare_process_definitions, + render_mermaid, + simulate_process_migration, +) from forge.workflow.declarative.models import WorkflowDefinition +from forge.workflow.declarative.publication import ( + DefinitionPublisher, + InMemoryDefinitionPublisher, + PublicationDecision, +) from forge.workflow.declarative.workflow import DeclarativeWorkflow __all__ = [ "DeclarativeWorkflow", "DeclarativeWorkflowCompiler", + "ProcessChangeClassification", + "ProcessChangeImpact", + "ProcessInstanceSnapshot", + "ProcessManifest", + "ProcessMigrationClassification", + "ProcessMigrationSimulation", "WorkflowDefinition", "load_workflow_file", "load_workflow_value", + "build_process_manifest", + "compare_process_definitions", + "render_mermaid", + "simulate_process_migration", + "DefinitionPublisher", + "InMemoryDefinitionPublisher", + "PublicationDecision", ] diff --git a/src/forge/workflow/declarative/builtins.py b/src/forge/workflow/declarative/builtins.py new file mode 100644 index 000000000..174f1acf9 --- /dev/null +++ b/src/forge/workflow/declarative/builtins.py @@ -0,0 +1,108 @@ +"""Versioned definitions for Forge-supported golden paths. + +The built-in workflows are checked-in process artifacts. Keeping the source +documents separate from this adapter makes the artifacts inspectable and +ensures the runtime cannot silently invent or mutate workflow topology. +""" + +from __future__ import annotations + +import json +from importlib import resources +from typing import Any + +from forge.models.workflow import TicketType +from forge.workflow.declarative.loader import load_workflow_value +from forge.workflow.declarative.models import WorkflowDefinition +from forge.workflow.declarative.workflow import DeclarativeWorkflow + +POLICY = "forge-contracts-v1" +JIRA_EFFECTS = ("jira.*",) +SC_EFFECTS = ("source_control.*",) + + +def _load_builtin_definition(name: str) -> WorkflowDefinition: + """Load and validate a checked-in built-in process artifact by name.""" + resource = resources.files("forge.workflow.declarative.definitions").joinpath(f"{name}.json") + try: + value = json.loads(resource.read_text(encoding="utf-8")) + except FileNotFoundError as exc: + raise RuntimeError(f"missing built-in workflow artifact: {name}") from exc + except json.JSONDecodeError as exc: + raise RuntimeError(f"invalid built-in workflow artifact: {name}") from exc + definition = load_workflow_value(value) + if definition.metadata.name != name: + raise RuntimeError( + f"built-in workflow artifact {name!r} declares name {definition.metadata.name!r}" + ) + return definition + + +def builtin_feature_definition() -> WorkflowDefinition: + """Return the immutable feature golden-path definition.""" + return _load_builtin_definition("feature") + + +def builtin_definitions() -> tuple[WorkflowDefinition, ...]: + return ( + builtin_feature_definition(), + builtin_bug_definition(), + builtin_task_takeover_definition(), + ) + + +class FeatureGoldenWorkflow(DeclarativeWorkflow): + """Default Feature/Story runtime compiled from the published process model.""" + + name = "feature" + description = "Full SDLC workflow compiled from the versioned feature definition" + + def __init__(self) -> None: + super().__init__(builtin_feature_definition(), "BUILTIN") + + @property + def cache_key(self) -> str: + return f"builtin:{self.name}:{self.definition.metadata.revision}:{self.definition.digest}" + + def matches(self, ticket_type: TicketType, _labels: list[str], _event: dict[str, Any]) -> bool: + return ticket_type in {TicketType.FEATURE, TicketType.STORY} + + +def builtin_bug_definition() -> WorkflowDefinition: + """Return the immutable bug-fix golden-path definition.""" + return _load_builtin_definition("bug") + + +class BugGoldenWorkflow(DeclarativeWorkflow): + name = "bug" + description = "Bug-fix workflow compiled from the versioned process definition" + + def __init__(self) -> None: + super().__init__(builtin_bug_definition(), "BUILTIN") + + @property + def cache_key(self) -> str: + return f"builtin:{self.name}:{self.definition.metadata.revision}:{self.definition.digest}" + + def matches(self, ticket_type: TicketType, _labels: list[str], _event: dict[str, Any]) -> bool: + return ticket_type is TicketType.BUG + + +def builtin_task_takeover_definition() -> WorkflowDefinition: + """Return the immutable task-takeover golden-path definition.""" + return _load_builtin_definition("task_takeover") + + +class TaskTakeoverGoldenWorkflow(DeclarativeWorkflow): + name = "task_takeover" + description = "Task-takeover workflow compiled from the versioned process definition" + + def __init__(self) -> None: + super().__init__(builtin_task_takeover_definition(), "BUILTIN") + + @property + def cache_key(self) -> str: + return f"builtin:{self.name}:{self.definition.metadata.revision}:{self.definition.digest}" + + def matches(self, ticket_type: TicketType, labels: list[str], _event: dict[str, Any]) -> bool: + return ticket_type in {TicketType.TASK, TicketType.EPIC} and "forge:managed" in labels diff --git a/src/forge/workflow/declarative/capabilities.py b/src/forge/workflow/declarative/capabilities.py new file mode 100644 index 000000000..6c51162b8 --- /dev/null +++ b/src/forge/workflow/declarative/capabilities.py @@ -0,0 +1,88 @@ +"""Runtime capability scope for effects emitted by compiled process steps.""" + +from __future__ import annotations + +from collections.abc import Iterator +from contextlib import contextmanager +from contextvars import ContextVar + +_EFFECT_CAPABILITIES: ContextVar[tuple[str, ...] | None] = ContextVar( + "forge_workflow_effect_capabilities", default=None +) + +JIRA_EFFECT_CAPABILITIES = frozenset( + { + "jira.comment", + "jira.labels", + "jira.status", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.project_configuration", + } +) +SOURCE_CONTROL_EFFECT_CAPABILITIES = frozenset( + { + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review", + } +) +KNOWN_EFFECT_CAPABILITIES = JIRA_EFFECT_CAPABILITIES | SOURCE_CONTROL_EFFECT_CAPABILITIES + +_OPERATION_CAPABILITIES = { + "jira.comment.create": "jira.comment", + "jira.structured_comment.create": "jira.comment", + "jira.label.set": "jira.labels", + "jira.labels.add": "jira.labels", + "jira.labels.remove": "jira.labels", + "jira.issue.transition": "jira.status", + "jira.description.update": "jira.issue_content", + "jira.custom_field.update": "jira.issue_content", + "jira.attachment.replace": "jira.issue_content", + "jira.attachment.add": "jira.issue_content", + "jira.attachment.delete_by_name": "jira.issue_content", + "jira.error_comment.create": "jira.comment", + "jira.model_policy_error_comment.create": "jira.comment", + "jira.issue.archive": "jira.issue_lifecycle", + "jira.task.create": "jira.issue_structure", + "jira.epic.create": "jira.issue_structure", + "jira.issue_link.create": "jira.issue_structure", + "jira.remote_link.create": "jira.issue_structure", + "jira.project_property.set": "jira.project_configuration", + "jira.project_property.delete": "jira.project_configuration", + "source_control.branch.create": "source_control.branch", + "source_control.file.put": "source_control.commit", + "source_control.change_request.create": "source_control.pull_request", + "source_control.change_request.update": "source_control.pull_request", + "source_control.comment.create": "source_control.review", + "source_control.comment.reply": "source_control.review", + "repository.ref.push": "source_control.commit", +} + + +@contextmanager +def effect_capability_scope(capabilities: tuple[str, ...]) -> Iterator[None]: + token = _EFFECT_CAPABILITIES.set(capabilities) + try: + yield + finally: + _EFFECT_CAPABILITIES.reset(token) + + +def require_effect_capability(operation: str) -> None: + """Reject an effect not authorized by the currently executing process step. + + A missing scope denotes a direct/local or legacy invocation. Compiled workflows + always install a scope, including an empty one, before invoking a node. + """ + capabilities = _EFFECT_CAPABILITIES.get() + if capabilities is None: + return + required = _OPERATION_CAPABILITIES.get(operation) + if required is None: + raise PermissionError(f"effect operation '{operation}' has no governed capability") + if required in capabilities: + return + raise PermissionError(f"process step is not allowed to emit effect operation '{operation}'") diff --git a/src/forge/workflow/declarative/catalog.py b/src/forge/workflow/declarative/catalog.py index e7ec647c9..9e6c5a4b3 100644 --- a/src/forge/workflow/declarative/catalog.py +++ b/src/forge/workflow/declarative/catalog.py @@ -18,6 +18,12 @@ class StateProfile: routers: dict[str, Any] pause_nodes: frozenset[str] contracts: dict[str, NodeContract] = field(default_factory=dict) + station_bindings: dict[str, tuple[str, str]] = field(default_factory=dict) + mandatory_nodes: frozenset[str] = frozenset() + supported_policies: frozenset[str] = frozenset({"forge-contracts-v1"}) + supported_extensions: frozenset[str] = frozenset( + {"station-behavior", "optional-stations", "routing-branches"} + ) def _common_nodes() -> dict[str, Callable[..., Any]]: @@ -121,6 +127,7 @@ def get_state_profile(name: str) -> StateProfile: regenerate_epic_tasks, update_single_task, ) + from forge.workflow.nodes.task_router import route_tasks_parallel from forge.workflow.post_pr import route_after_pr_creation nodes: dict[str, Any] = { @@ -169,6 +176,7 @@ def get_state_profile(name: str) -> StateProfile: "route_prd_approval": route_prd_approval, "route_spec_approval": route_spec_approval, "route_task_approval": route_task_approval, + "route_tasks_parallel": route_tasks_parallel, } pauses = common_pauses | { "plan_approval_gate", @@ -183,6 +191,42 @@ def get_state_profile(name: str) -> StateProfile: routers, pauses, contracts_for(nodes), + { + "generate_prd": ("artifact-generation", "1.0"), + "regenerate_prd": ("artifact-generation", "1.0"), + "generate_spec": ("artifact-generation", "1.0"), + "regenerate_spec": ("artifact-generation", "1.0"), + "decompose_epics": ("artifact-generation", "1.0"), + "regenerate_all_epics": ("artifact-generation", "1.0"), + "update_single_epic": ("artifact-generation", "1.0"), + "update_single_task": ("artifact-generation", "1.0"), + "task_router": ("task-routing", "1.0"), + "prd_approval_gate": ("approval-policy", "1.0"), + "spec_approval_gate": ("approval-policy", "1.0"), + "plan_approval_gate": ("approval-policy", "1.0"), + "task_approval_gate": ("approval-policy", "1.0"), + "generate_tasks": ("agent-operation", "1.0"), + "regenerate_all_tasks": ("agent-operation", "1.0"), + "regenerate_epic_tasks": ("agent-operation", "1.0"), + "answer_question": ("agent-operation", "1.0"), + "implement_task": ("sandbox-execution", "1.0"), + "local_review": ("sandbox-execution", "1.0"), + "update_documentation": ("sandbox-execution", "1.0"), + "create_pr": ("agent-operation", "1.0"), + "ci_evaluator": ("sandbox-execution", "1.0"), + "attempt_ci_fix": ("sandbox-execution", "1.0"), + "human_review_gate": ("persistence-actions", "1.0"), + "implement_review": ("sandbox-execution", "1.0"), + }, + frozenset( + { + "prd_approval_gate", + "spec_approval_gate", + "plan_approval_gate", + "task_approval_gate", + "human_review_gate", + } + ), ) if name == "bug": @@ -198,6 +242,10 @@ def get_state_profile(name: str) -> StateProfile: _route_after_plan_bug_fix, _route_after_reflect_rca, _route_after_regenerate_plan, + _route_human_review_bug, + ) + from forge.workflow.bug.graph import ( + _route_after_workspace_setup as route_after_bug_workspace_setup, ) from forge.workflow.bug.state import BugState, create_initial_bug_state from forge.workflow.nodes import ( @@ -250,9 +298,11 @@ def get_state_profile(name: str) -> StateProfile: "route_after_pr_creation": route_after_pr_creation, "route_after_reflect_rca": _route_after_reflect_rca, "route_after_regenerate_plan": _route_after_regenerate_plan, + "route_after_workspace_setup": route_after_bug_workspace_setup, "route_plan_approval": route_bug_plan_approval, "route_rca_option": route_rca_option, "route_triage_gate": route_triage_gate, + "route_human_review_bug": _route_human_review_bug, } pauses = common_pauses | {"triage_gate", "rca_option_gate", "plan_approval_gate"} return StateProfile( @@ -262,6 +312,28 @@ def get_state_profile(name: str) -> StateProfile: routers, pauses, contracts_for(nodes), + { + "plan_approval_gate": ("approval-policy", "1.0"), + "triage_check": ("triage-evaluation", "1.0"), + "analyze_bug": ("sandbox-execution", "1.0"), + "reflect_rca": ("sandbox-execution", "1.0"), + "regenerate_rca": ("sandbox-execution", "1.0"), + "plan_bug_fix": ("sandbox-execution", "1.0"), + "regenerate_plan": ("sandbox-execution", "1.0"), + "answer_question": ("agent-operation", "1.0"), + "implement_bug_fix": ("sandbox-execution", "1.0"), + "local_review": ("sandbox-execution", "1.0"), + "update_documentation": ("sandbox-execution", "1.0"), + "create_pr": ("agent-operation", "1.0"), + "ci_evaluator": ("sandbox-execution", "1.0"), + "attempt_ci_fix": ("sandbox-execution", "1.0"), + "human_review_gate": ("persistence-actions", "1.0"), + "implement_review": ("sandbox-execution", "1.0"), + "post_merge_summary": ("persistence-actions", "1.0"), + }, + frozenset( + {"triage_gate", "rca_option_gate", "plan_approval_gate", "human_review_gate"} + ), ) if name == "task_takeover": @@ -283,6 +355,8 @@ def get_state_profile(name: str) -> StateProfile: _route_after_execution, _route_after_generate_plan, _route_after_qualitative_review, + _route_after_triage_check, + _route_human_review_task_takeover, complete_task_takeover, ) from forge.workflow.task_takeover.graph import ( @@ -311,9 +385,11 @@ def get_state_profile(name: str) -> StateProfile: "route_after_generate_plan": _route_after_generate_plan, "route_after_pr_creation": route_after_pr_creation, "route_after_qualitative_review": _route_after_qualitative_review, + "route_after_triage_check": _route_after_triage_check, "route_after_workspace_setup": route_after_task_workspace_setup, "route_task_plan_approval": route_task_plan_approval, "route_triage_gate": route_triage_gate, + "route_human_review_task_takeover": _route_human_review_task_takeover, } pauses = common_pauses | {"triage_gate", "task_plan_approval_gate"} return StateProfile( @@ -323,6 +399,20 @@ def get_state_profile(name: str) -> StateProfile: routers, pauses, contracts_for(nodes), + { + "task_plan_approval_gate": ("approval-policy", "1.0"), + "triage_check": ("triage-evaluation", "1.0"), + "generate_plan": ("agent-operation", "1.0"), + "answer_question": ("agent-operation", "1.0"), + "execute_task_changes": ("sandbox-execution", "1.0"), + "run_qualitative_review": ("sandbox-execution", "1.0"), + "create_pr": ("agent-operation", "1.0"), + "ci_evaluator": ("sandbox-execution", "1.0"), + "attempt_ci_fix": ("sandbox-execution", "1.0"), + "human_review_gate": ("persistence-actions", "1.0"), + "implement_review": ("sandbox-execution", "1.0"), + }, + frozenset({"triage_gate", "task_plan_approval_gate", "human_review_gate"}), ) raise ValueError(f"unknown state profile: {name}") diff --git a/src/forge/workflow/declarative/cli.py b/src/forge/workflow/declarative/cli.py index d817650e2..2b5e9bd10 100644 --- a/src/forge/workflow/declarative/cli.py +++ b/src/forge/workflow/declarative/cli.py @@ -8,10 +8,15 @@ import yaml # type: ignore[import-untyped] -from forge.integrations.jira.client import JiraClient from forge.workflow.declarative.compiler import DeclarativeWorkflowCompiler -from forge.workflow.declarative.loader import load_workflow_file, load_workflow_value -from forge.workflow.declarative.models import WORKFLOW_PROPERTY_PREFIX +from forge.workflow.declarative.loader import load_workflow_file +from forge.workflow.declarative.manifest import ( + build_process_manifest, + compare_process_definitions, + render_mermaid, + simulate_process_migration, +) +from forge.workflow.declarative.publication import DefinitionPublisher def _print_error(exc: Exception) -> int: @@ -35,57 +40,84 @@ async def cmd_workflow(args: Any) -> int: print(json.dumps(definition.canonical_dict(), indent=2)) return 0 - jira = JiraClient() + if action == "render": + try: + definition = load_workflow_file(args.file) + manifest = build_process_manifest(definition) + except Exception as exc: + return _print_error(exc) + if args.format == "json": + print(manifest.model_dump_json(indent=2)) + else: + print(render_mermaid(manifest)) + return 0 + + if action == "diff": + try: + previous = load_workflow_file(args.previous) + current = load_workflow_file(args.current) + impact = compare_process_definitions(previous, current) + except Exception as exc: + return _print_error(exc) + print(impact.model_dump_json(indent=2)) + return 0 if impact.compatible_for_in_flight else 2 + + if action == "simulate-migration": + try: + previous = load_workflow_file(args.previous) + current = load_workflow_file(args.current) + with open(args.instances, encoding="utf-8") as source: + instances = json.load(source) + if not isinstance(instances, list): + raise ValueError("active instance snapshot must be a JSON array") + simulation = simulate_process_migration(previous, current, instances) + except Exception as exc: + return _print_error(exc) + print(simulation.model_dump_json(indent=2)) + return 0 if simulation.compatible else 2 + try: project_key = args.project_key.upper() + publisher = DefinitionPublisher(project_key) + actor = getattr(args, "actor", None) or "forge-cli" + reason = getattr(args, "reason", None) or f"CLI {action} decision" if action == "publish": definition = load_workflow_file(args.file) - DeclarativeWorkflowCompiler(definition).validate() - existing = await jira.get_project_property(project_key, definition.property_key) - if existing is not None: - try: - previous = load_workflow_value(existing) - except Exception: - previous = None # A valid publication is allowed to repair a broken property. - if previous is not None: - if previous.metadata.name != definition.metadata.name: - raise ValueError("existing property has a different workflow name") - if previous.digest != definition.digest and ( - definition.metadata.revision <= previous.metadata.revision - ): - raise ValueError( - "changed workflow content must increment metadata.revision " - f"above {previous.metadata.revision}" - ) - await jira.set_project_property( - project_key, definition.property_key, definition.canonical_dict() + decision = await publisher.publish(definition, actor=actor, reason=reason) + print( + f"[OK] published {decision.workflow_name} revision {decision.revision} " + f"to {project_key} (digest {decision.digest})" ) + return 0 + + if action in {"activate", "rollback"}: + decision = await getattr(publisher, action)( + args.name, + args.revision, + actor=actor, + reason=reason, + expected_active_digest=getattr(args, "expected_active_digest", None), + ) + verb = "activated" if decision.action == "activate" else "rolled back" print( - f"[OK] published {definition.metadata.name} revision " - f"{definition.metadata.revision} to {project_key}" + f"[OK] {verb} {decision.workflow_name} revision " + f"{decision.revision} for {project_key}" ) return 0 if action == "show": - key = f"{WORKFLOW_PROPERTY_PREFIX}{args.name}" - value = await jira.get_project_property(project_key, key) - if value is None: + active_definition = await publisher.active(args.name) + if active_definition is None: raise ValueError(f"workflow '{args.name}' is not defined for {project_key}") - definition = load_workflow_value(value) - DeclarativeWorkflowCompiler(definition).validate() - if args.json: - print(json.dumps(definition.canonical_dict(), indent=2)) + DeclarativeWorkflowCompiler(active_definition).validate() + if getattr(args, "json", False): + print(json.dumps(active_definition.canonical_dict(), indent=2)) else: - print(yaml.safe_dump(definition.canonical_dict(), sort_keys=False).rstrip()) + print(yaml.safe_dump(active_definition.canonical_dict(), sort_keys=False).rstrip()) return 0 if action == "list": - keys = await jira.list_project_properties(project_key) - names = sorted( - key[len(WORKFLOW_PROPERTY_PREFIX) :] - for key in keys - if key.startswith(WORKFLOW_PROPERTY_PREFIX) - ) + names = await publisher.list_workflows() if not names: print(f"No custom workflows configured for {project_key}.") else: @@ -93,16 +125,22 @@ async def cmd_workflow(args: Any) -> int: print(name) return 0 + if action == "show-history": + decisions = await publisher.decisions(args.name) + if args.json: + print(json.dumps([item.model_dump(mode="json") for item in decisions], indent=2)) + else: + for item in decisions: + print( + f"{item.published_at.isoformat()} {item.action} " + f"revision {item.revision} actor={item.actor} reason={item.reason}" + ) + return 0 + if action == "delete": - if not args.yes: - raise ValueError("deleting a workflow requires --yes") - await jira.delete_project_property( - project_key, f"{WORKFLOW_PROPERTY_PREFIX}{args.name}" + raise ValueError( + "destructive workflow deletion is disabled; publish a replacement or use rollback" ) - print(f"[OK] deleted {args.name} from {project_key}") - return 0 except Exception as exc: return _print_error(exc) - finally: - await jira.close() return _print_error(ValueError(f"unknown workflow command: {action}")) diff --git a/src/forge/workflow/declarative/compiler.py b/src/forge/workflow/declarative/compiler.py index 1059a15fe..e7cc50e1b 100644 --- a/src/forge/workflow/declarative/compiler.py +++ b/src/forge/workflow/declarative/compiler.py @@ -7,7 +7,12 @@ from typing import Any from langgraph.graph import END, StateGraph +from langgraph.types import Send +from forge.workflow.declarative.capabilities import ( + KNOWN_EFFECT_CAPABILITIES, + effect_capability_scope, +) from forge.workflow.declarative.catalog import get_state_profile from forge.workflow.declarative.models import MAX_TRANSITIONS, WorkflowDefinition from forge.workflow.preconditions import NodeContract, with_preconditions @@ -28,6 +33,24 @@ def validate(self) -> None: if spec.entry not in steps: raise WorkflowValidationError(f"entry node '{spec.entry}' is not declared") + unknown_policies = set(spec.mandatory_policies) - set(self.profile.supported_policies) + if unknown_policies: + raise WorkflowValidationError( + f"unknown mandatory policy '{sorted(unknown_policies)[0]}'" + ) + missing_nodes = ( + set(self.profile.mandatory_nodes) - set(steps) if spec.mandatory_policies else set() + ) + if missing_nodes: + raise WorkflowValidationError( + f"workflow omits mandatory gate '{sorted(missing_nodes)[0]}'" + ) + unknown_extensions = set(spec.extension_points) - set(self.profile.supported_extensions) + if unknown_extensions: + raise WorkflowValidationError( + f"unsupported extension point '{sorted(unknown_extensions)[0]}'" + ) + unknown_nodes = set(steps) - set(self.profile.nodes) if unknown_nodes: raise WorkflowValidationError( @@ -42,7 +65,13 @@ def validate(self) -> None: f"router '{step.route}' on '{node_name}' is not registered for state " f"'{spec.state}'" ) - targets = [step.next] if step.next else list(step.branches.values()) + targets = ( + [step.next] + if step.next + else list(step.dynamic_targets) + if step.dynamic_route + else list(step.branches.values()) + ) for target in targets: if target == "__end__": has_terminal = True @@ -52,6 +81,31 @@ def validate(self) -> None: ) else: adjacency[node_name].add(target) + missing_policies = set(self.definition.spec.mandatory_policies) - set( + step.required_policies + ) + if missing_policies: + raise WorkflowValidationError( + f"step '{node_name}' omits mandatory policy '{sorted(missing_policies)[0]}'" + ) + unknown_effects = set(step.allowed_effects) - set(KNOWN_EFFECT_CAPABILITIES) + if unknown_effects: + raise WorkflowValidationError( + f"step '{node_name}' requests unknown effect capability " + f"'{sorted(unknown_effects)[0]}'" + ) + binding = self.profile.station_bindings.get(node_name) + if binding and (step.kind in {"station", "gate"} or step.station_contract): + declared = (step.station_contract, step.station_contract_version) + if binding != declared: + raise WorkflowValidationError( + f"station contract for '{node_name}' must be {binding}, got {declared}" + ) + elif step.station_contract: + raise WorkflowValidationError( + f"node '{node_name}' does not support station contract " + f"'{step.station_contract}'" + ) if not has_terminal: raise WorkflowValidationError("at least one path must target '__end__'") @@ -68,8 +122,22 @@ def validate(self) -> None: if unreachable: raise WorkflowValidationError(f"unreachable node '{sorted(unreachable)[0]}'") + incoming: dict[str, set[str]] = {name: set() for name in steps} + for source, incoming_targets in adjacency.items(): + for target in incoming_targets: + incoming[target].add(source) + for node_name, step in steps.items(): + if step.join and len(incoming[node_name]) < 2: + raise WorkflowValidationError( + f"join step '{node_name}' must have at least two incoming transitions" + ) + # A cycle is safe only if removing pause/bounded-boundary nodes breaks it. - unguarded = set(steps) - set(self.profile.pause_nodes) + unguarded = { + name + for name, step in steps.items() + if name not in self.profile.pause_nodes and step.kind != "gate" and not step.retry_bound + } colors: dict[str, int] = {} def visit(node: str) -> None: @@ -98,6 +166,68 @@ def visit(node: str) -> None: f"resume mapping targets undeclared node '{target}'" ) + def validate_for_publication(self) -> None: + """Apply organizational governance in addition to structural validity.""" + self.validate() + required_policies = {"forge-contracts-v1"} + missing_policies = required_policies - set(self.definition.spec.mandatory_policies) + if missing_policies: + raise WorkflowValidationError( + f"publication requires mandatory policy '{sorted(missing_policies)[0]}'" + ) + missing_nodes = set(self.profile.mandatory_nodes) - set(self.definition.spec.steps) + if missing_nodes: + raise WorkflowValidationError( + f"publication omits mandatory gate '{sorted(missing_nodes)[0]}'" + ) + for node_name, binding in self.profile.station_bindings.items(): + step = self.definition.spec.steps.get(node_name) + if step is None: + continue + declared = (step.station_contract, step.station_contract_version) + if declared != binding: + raise WorkflowValidationError( + f"published step '{node_name}' must declare station contract {binding}" + ) + self._validate_golden_route_contracts() + + def _validate_golden_route_contracts(self) -> None: + """Keep custom routing within the reviewed golden-path outcome contract.""" + from forge.workflow.declarative.builtins import ( + builtin_bug_definition, + builtin_feature_definition, + builtin_task_takeover_definition, + ) + + factories = { + "feature": builtin_feature_definition, + "bug": builtin_bug_definition, + "task_takeover": builtin_task_takeover_definition, + } + golden = factories[self.definition.spec.state]() + extensions = set(self.definition.spec.extension_points) + for node_name, step in self.definition.spec.steps.items(): + expected = golden.spec.steps.get(node_name) + if expected is None or not expected.route or step.route != expected.route: + continue + expected_outcomes = ( + set(expected.dynamic_targets) if expected.dynamic_route else set(expected.branches) + ) + declared_outcomes = ( + set(step.dynamic_targets) if step.dynamic_route else set(step.branches) + ) + missing = expected_outcomes - declared_outcomes + if missing: + raise WorkflowValidationError( + f"step '{node_name}' omits router outcome '{sorted(missing)[0]}'" + ) + extra = declared_outcomes - expected_outcomes + if extra and "routing-branches" not in extensions: + raise WorkflowValidationError( + f"step '{node_name}' adds outcome '{sorted(extra)[0]}' without the " + "routing-branches extension" + ) + def build_graph(self) -> StateGraph[Any]: self.validate() graph: StateGraph[Any] = StateGraph(self.profile.schema) @@ -110,6 +240,8 @@ def build_graph(self) -> StateGraph[Any]: node_name, terminal=step.next == "__end__", contract=self.profile.contracts.get(node_name), + retry_bound=step.retry_bound, + allowed_effects=step.allowed_effects, ), ) graph.set_entry_point("_forge_entry") @@ -130,6 +262,16 @@ def build_graph(self) -> StateGraph[Any]: continue assert step.route is not None + if step.dynamic_route: + graph.add_conditional_edges( + node_name, + self._guarded_dynamic_router( + self.profile.routers[step.route], + set(step.dynamic_targets), + step.max_concurrency, + ), + ) + continue branches: dict[Any, str] = { outcome: END if target == "__end__" else target for outcome, target in step.branches.items() @@ -158,6 +300,8 @@ def _guarded_node( *, terminal: bool, contract: NodeContract | None = None, + retry_bound: int | None = None, + allowed_effects: tuple[str, ...] = (), ) -> Callable[..., Awaitable[dict[str, Any]]]: guarded_func = with_preconditions(func, contract, node_name=node_name) @@ -171,14 +315,32 @@ async def run(state: dict[str, Any]) -> dict[str, Any]: "is_blocked": True, "last_error": f"Declarative workflow exceeded {MAX_TRANSITIONS} transitions", } - result = await guarded_func(state) + attempts = dict(state.get("workflow_node_attempts") or {}) + attempts[node_name] = int(attempts.get(node_name, 0)) + 1 + if retry_bound is not None and attempts[node_name] > retry_bound: + return { + **state, + "workflow_transition_count": transitions, + "workflow_node_attempts": attempts, + "current_node": node_name, + "is_blocked": True, + "last_error": ( + f"Declarative step '{node_name}' exceeded retry bound {retry_bound}" + ), + } + with effect_capability_scope(allowed_effects): + result = await guarded_func(state) if not isinstance(result, dict): raise TypeError(f"node '{node_name}' must return a state dictionary") if terminal and not any( (result.get("last_error"), result.get("is_paused"), result.get("is_blocked")) ): result = {**result, "current_node": "complete", "is_paused": False} - return {**result, "workflow_transition_count": transitions} + return { + **result, + "workflow_transition_count": transitions, + "workflow_node_attempts": attempts, + } run.__name__ = f"declarative_{node_name}" return run @@ -204,3 +366,32 @@ async def route(state: dict[str, Any]) -> str: return normalized return route + + @staticmethod + def _guarded_dynamic_router( + func: Callable[..., Any], targets: set[str], max_concurrency: int | None + ) -> Callable[..., Any]: + async def route(state: dict[str, Any]) -> str | Send | list[Send]: + if state.get("is_blocked"): + return "__end__" + result = func(state) + if inspect.isawaitable(result): + result = await result + values = result if isinstance(result, list) else [result] + if max_concurrency is not None and len(values) > max_concurrency: + raise WorkflowValidationError( + f"dynamic router emitted {len(values)} branches; maximum is {max_concurrency}" + ) + for value in values: + target = value.node if isinstance(value, Send) else value + if target not in targets: + raise WorkflowValidationError( + f"dynamic router returned undeclared target {target!r}" + ) + if isinstance(result, (str, Send)): + return result + if isinstance(result, list) and all(isinstance(value, Send) for value in result): + return result + raise WorkflowValidationError("dynamic router must return a target or Send values") + + return route diff --git a/src/forge/workflow/declarative/definitions/__init__.py b/src/forge/workflow/declarative/definitions/__init__.py new file mode 100644 index 000000000..3c7958af7 --- /dev/null +++ b/src/forge/workflow/declarative/definitions/__init__.py @@ -0,0 +1 @@ +"""Checked-in versioned built-in workflow definition artifacts.""" diff --git a/src/forge/workflow/declarative/definitions/bug.json b/src/forge/workflow/declarative/definitions/bug.json new file mode 100644 index 000000000..f5f2c84c4 --- /dev/null +++ b/src/forge/workflow/declarative/definitions/bug.json @@ -0,0 +1,487 @@ +{ + "apiVersion": "forge/v1", + "kind": "Workflow", + "metadata": { + "description": "Forge supported bug-fix golden path", + "name": "bug", + "revision": 1 + }, + "spec": { + "entry": "triage_check", + "extensionPoints": [ + "station-behavior" + ], + "mandatoryPolicies": [ + "forge-contracts-v1" + ], + "resume": { + "fromRevisions": {} + }, + "state": "bug", + "steps": { + "analyze_bug": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "escalate_blocked": "escalate_blocked", + "reflect_rca": "reflect_rca" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_analyze_bug", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "answer_question": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "plan_approval_gate": "plan_approval_gate", + "rca_option_gate": "rca_option_gate", + "triage_gate": "triage_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_answer", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "attempt_ci_fix": { + "allowedEffects": [], + "branches": { + "attempt_ci_fix": "escalate_blocked", + "ci_evaluator": "ci_evaluator", + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 5, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "ci_evaluator": { + "allowedEffects": [], + "branches": { + "attempt_ci_fix": "attempt_ci_fix", + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_ci_evaluation", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "create_pr": { + "allowedEffects": [ + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "escalate_blocked": "escalate_blocked", + "teardown_workspace": "teardown_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_pr_creation", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "decompose_plan": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "__end__": "__end__", + "escalate_blocked": "escalate_blocked", + "setup_workspace": "setup_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_decompose_plan" + }, + "escalate_blocked": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "__end__", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "human_review_gate": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "__end__": "__end__", + "ci_evaluator": "ci_evaluator", + "complete_tasks": "post_merge_summary", + "implement_review": "implement_review", + "post_merge_summary": "post_merge_summary" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_human_review_bug", + "stationContract": "persistence-actions", + "stationContractVersion": "1.0" + }, + "implement_bug_fix": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "implement_bug_fix": "implement_bug_fix", + "local_review": "local_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 100, + "route": "route_after_implementation", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "implement_review": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate", + "implement_review": "implement_review", + "review_response_gate": "review_response_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "local_review": { + "allowedEffects": [], + "branches": { + "create_pr": "create_pr", + "escalate_blocked": "escalate_blocked", + "implement_bug_fix": "implement_bug_fix", + "local_review": "local_review", + "update_documentation": "update_documentation" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 2, + "route": "route_after_local_review", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "plan_approval_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "decompose_plan": "decompose_plan", + "regenerate_plan": "regenerate_plan" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_plan_approval", + "stationContract": "approval-policy", + "stationContractVersion": "1.0" + }, + "plan_bug_fix": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "escalate_blocked": "escalate_blocked", + "plan_approval_gate": "plan_approval_gate", + "plan_bug_fix": "plan_bug_fix" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_after_plan_bug_fix", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "post_merge_summary": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "next": "__end__", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "stationContract": "persistence-actions", + "stationContractVersion": "1.0" + }, + "rca_option_gate": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "plan_bug_fix": "plan_bug_fix", + "regenerate_rca": "regenerate_rca" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_rca_option" + }, + "reflect_rca": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "analyze_bug": "analyze_bug", + "escalate_blocked": "escalate_blocked", + "rca_option_gate": "rca_option_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_after_reflect_rca", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "regenerate_plan": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "escalate_blocked": "escalate_blocked", + "plan_approval_gate": "plan_approval_gate", + "regenerate_plan": "regenerate_plan" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_after_regenerate_plan", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "regenerate_rca": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "next": "analyze_bug", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "review_response_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "human_review_gate": "human_review_gate", + "implement_review": "implement_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_review_response" + }, + "setup_workspace": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "implement_bug_fix": "implement_bug_fix" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_workspace_setup" + }, + "teardown_workspace": { + "allowedEffects": [], + "branches": { + "human_review_gate": "human_review_gate", + "setup_workspace": "setup_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_teardown" + }, + "triage_check": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "analyze_bug": "analyze_bug", + "escalate_blocked": "escalate_blocked", + "triage_check": "triage_check", + "triage_gate": "triage_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_current_node", + "stationContract": "triage-evaluation", + "stationContractVersion": "1.0" + }, + "triage_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "triage_check": "triage_check" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_triage_gate" + }, + "update_documentation": { + "allowedEffects": [], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "next": "create_pr", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + } + } + } +} diff --git a/src/forge/workflow/declarative/definitions/feature.json b/src/forge/workflow/declarative/definitions/feature.json new file mode 100644 index 000000000..75f8abe04 --- /dev/null +++ b/src/forge/workflow/declarative/definitions/feature.json @@ -0,0 +1,670 @@ +{ + "apiVersion": "forge/v1", + "kind": "Workflow", + "metadata": { + "description": "Forge supported feature golden path", + "name": "feature", + "revision": 1 + }, + "spec": { + "entry": "generate_prd", + "extensionPoints": [ + "station-behavior" + ], + "mandatoryPolicies": [ + "forge-contracts-v1" + ], + "resume": { + "fromRevisions": {} + }, + "state": "feature", + "steps": { + "aggregate_epic_status": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "aggregate_feature_status", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "aggregate_feature_status": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "__end__", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "answer_question": { + "allowedEffects": [], + "branches": { + "plan_approval_gate": "plan_approval_gate", + "prd_approval_gate": "prd_approval_gate", + "spec_approval_gate": "spec_approval_gate", + "task_approval_gate": "task_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_answer", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "attempt_ci_fix": { + "allowedEffects": [], + "branches": { + "attempt_ci_fix": "escalate_blocked", + "ci_evaluator": "ci_evaluator", + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 5, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "ci_evaluator": { + "allowedEffects": [], + "branches": { + "attempt_ci_fix": "attempt_ci_fix", + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_ci_evaluation", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "complete_tasks": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "aggregate_epic_status", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "create_pr": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "teardown_workspace": "teardown_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_pr_creation", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "decompose_epics": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "plan_approval_gate": "plan_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_epic_decomposition", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "escalate_blocked": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "__end__", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "generate_prd": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "prd_approval_gate": "prd_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_generation", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "generate_spec": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "spec_approval_gate": "spec_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_spec_generation", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "generate_tasks": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "task_approval_gate": "task_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_task_generation", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "human_review_gate": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "__end__": "__end__", + "ci_evaluator": "ci_evaluator", + "complete_tasks": "complete_tasks", + "implement_review": "implement_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_human_review", + "stationContract": "persistence-actions", + "stationContractVersion": "1.0" + }, + "implement_review": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate", + "implement_review": "implement_review", + "review_response_gate": "review_response_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "implement_task": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "implement_task": "implement_task", + "local_review": "local_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 100, + "route": "route_implementation", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "local_review": { + "allowedEffects": [], + "branches": { + "create_pr": "update_documentation", + "escalate_blocked": "escalate_blocked", + "local_review": "local_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 2, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "plan_approval_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "generate_tasks": "generate_tasks", + "regenerate_all_epics": "regenerate_all_epics", + "update_single_epic": "update_single_epic" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_plan_approval", + "stationContract": "approval-policy", + "stationContractVersion": "1.0" + }, + "prd_approval_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "generate_spec": "generate_spec", + "regenerate_prd": "regenerate_prd" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_prd_approval", + "stationContract": "approval-policy", + "stationContractVersion": "1.0" + }, + "regenerate_all_epics": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "plan_approval_gate": "plan_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_epic_regeneration", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "regenerate_all_tasks": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "task_approval_gate": "task_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_task_regeneration", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "regenerate_epic_tasks": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "task_approval_gate": "task_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_epic_task_regeneration", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "regenerate_prd": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "prd_approval_gate": "prd_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_prd_regeneration", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "regenerate_spec": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "spec_approval_gate": "spec_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_spec_regeneration", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "review_response_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "human_review_gate": "human_review_gate", + "implement_review": "implement_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_review_response" + }, + "setup_workspace": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "implement_task": "implement_task" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_workspace_setup" + }, + "spec_approval_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "decompose_epics": "decompose_epics", + "regenerate_spec": "regenerate_spec" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_spec_approval", + "stationContract": "approval-policy", + "stationContractVersion": "1.0" + }, + "task_approval_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "regenerate_all_tasks": "regenerate_all_tasks", + "regenerate_epic_tasks": "regenerate_epic_tasks", + "task_router": "task_router", + "update_single_task": "update_single_task" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_task_approval", + "stationContract": "approval-policy", + "stationContractVersion": "1.0" + }, + "task_router": { + "allowedEffects": [], + "branches": {}, + "dynamicRoute": true, + "dynamicTargets": [ + "setup_workspace" + ], + "kind": "station", + "maxConcurrency": 16, + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_tasks_parallel", + "stationContract": "task-routing", + "stationContractVersion": "1.0" + }, + "teardown_workspace": { + "allowedEffects": [], + "branches": { + "human_review_gate": "human_review_gate", + "setup_workspace": "setup_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_teardown" + }, + "update_documentation": { + "allowedEffects": [], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "next": "create_pr", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "update_single_epic": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "plan_approval_gate": "plan_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_single_epic_update", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + }, + "update_single_task": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status", + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "__end__": "__end__", + "task_approval_gate": "task_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_single_task_update", + "stationContract": "artifact-generation", + "stationContractVersion": "1.0" + } + } + } +} diff --git a/src/forge/workflow/declarative/definitions/task_takeover.json b/src/forge/workflow/declarative/definitions/task_takeover.json new file mode 100644 index 000000000..cb013470f --- /dev/null +++ b/src/forge/workflow/declarative/definitions/task_takeover.json @@ -0,0 +1,345 @@ +{ + "apiVersion": "forge/v1", + "kind": "Workflow", + "metadata": { + "description": "Forge supported task-takeover golden path", + "name": "task_takeover", + "revision": 1 + }, + "spec": { + "entry": "triage_check", + "extensionPoints": [ + "station-behavior" + ], + "mandatoryPolicies": [ + "forge-contracts-v1" + ], + "resume": { + "fromRevisions": {} + }, + "state": "task_takeover", + "steps": { + "answer_question": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "task_plan_approval_gate": "task_plan_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_answer", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "attempt_ci_fix": { + "allowedEffects": [], + "branches": { + "attempt_ci_fix": "escalate_blocked", + "ci_evaluator": "ci_evaluator", + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 5, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "ci_evaluator": { + "allowedEffects": [], + "branches": { + "attempt_ci_fix": "attempt_ci_fix", + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_ci_evaluation", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "complete_task_takeover": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "__end__", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "create_pr": { + "allowedEffects": [ + "source_control.branch", + "source_control.commit", + "source_control.pull_request", + "source_control.review" + ], + "branches": { + "escalate_blocked": "escalate_blocked", + "teardown_workspace": "teardown_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_pr_creation", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "escalate_blocked": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": {}, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "next": "__end__", + "requiredPolicies": [ + "forge-contracts-v1" + ] + }, + "execute_task_changes": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "execute_task_changes": "execute_task_changes", + "run_qualitative_review": "run_qualitative_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 100, + "route": "route_after_execution", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "generate_plan": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "generate_plan": "generate_plan", + "task_plan_approval_gate": "task_plan_approval_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_after_generate_plan", + "stationContract": "agent-operation", + "stationContractVersion": "1.0" + }, + "human_review_gate": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "__end__": "__end__", + "ci_evaluator": "ci_evaluator", + "complete_task_takeover": "complete_task_takeover", + "complete_tasks": "complete_task_takeover", + "implement_review": "implement_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_human_review_task_takeover", + "stationContract": "persistence-actions", + "stationContractVersion": "1.0" + }, + "implement_review": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "human_review_gate": "human_review_gate", + "implement_review": "implement_review", + "review_response_gate": "review_response_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_current_node", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "review_response_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "human_review_gate": "human_review_gate", + "implement_review": "implement_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_review_response" + }, + "run_qualitative_review": { + "allowedEffects": [], + "branches": { + "create_pr": "create_pr", + "escalate_blocked": "escalate_blocked", + "execute_task_changes": "execute_task_changes", + "run_qualitative_review": "run_qualitative_review" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_after_qualitative_review", + "stationContract": "sandbox-execution", + "stationContractVersion": "1.0" + }, + "setup_workspace": { + "allowedEffects": [], + "branches": { + "escalate_blocked": "escalate_blocked", + "execute_task_changes": "execute_task_changes" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_workspace_setup" + }, + "task_plan_approval_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "answer_question": "answer_question", + "regenerate_plan": "generate_plan", + "setup_workspace": "setup_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_task_plan_approval", + "stationContract": "approval-policy", + "stationContractVersion": "1.0" + }, + "teardown_workspace": { + "allowedEffects": [], + "branches": { + "human_review_gate": "human_review_gate", + "setup_workspace": "setup_workspace" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "operation", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_after_teardown" + }, + "triage_check": { + "allowedEffects": [ + "jira.comment", + "jira.issue_content", + "jira.issue_lifecycle", + "jira.issue_structure", + "jira.labels", + "jira.project_configuration", + "jira.status" + ], + "branches": { + "escalate_blocked": "escalate_blocked", + "generate_plan": "generate_plan", + "triage_check": "triage_check", + "triage_gate": "triage_gate" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "station", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "retryBound": 3, + "route": "route_after_triage_check", + "stationContract": "triage-evaluation", + "stationContractVersion": "1.0" + }, + "triage_gate": { + "allowedEffects": [], + "branches": { + "__end__": "__end__", + "triage_check": "triage_check" + }, + "dynamicRoute": false, + "dynamicTargets": [], + "kind": "gate", + "requiredPolicies": [ + "forge-contracts-v1" + ], + "route": "route_triage_gate" + } + } + } +} diff --git a/src/forge/workflow/declarative/manifest.py b/src/forge/workflow/declarative/manifest.py new file mode 100644 index 000000000..0c2512ead --- /dev/null +++ b/src/forge/workflow/declarative/manifest.py @@ -0,0 +1,797 @@ +"""Runtime-independent inspection and revision impact for workflow definitions.""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from enum import StrEnum +from typing import Any + +from pydantic import Field + +from forge.domain import DomainModel +from forge.workflow.declarative.catalog import get_state_profile +from forge.workflow.declarative.models import WorkflowDefinition + + +class ProcessNodeKind(StrEnum): + STATION = "station" + GATE = "gate" + OPERATION = "operation" + + +class ProcessTransition(DomainModel): + source: str + target: str + outcome: str | None = None + + +class ProcessNode(DomainModel): + name: str + kind: ProcessNodeKind + station_contract: str | None = None + station_contract_version: str | None = None + required_policies: tuple[str, ...] = () + allowed_effects: tuple[str, ...] = () + join: str | None = None + max_concurrency: int | None = None + retry_bound: int | None = None + + +class ProcessManifest(DomainModel): + workflow_name: str + revision: int + digest: str + state_profile: str + entry: str + nodes: tuple[ProcessNode, ...] + transitions: tuple[ProcessTransition, ...] + + +class ProcessChangeClassification(StrEnum): + """Compatibility class assigned to a process-definition revision change. + + The names intentionally mirror the governance vocabulary. In particular, + ``compatible`` does not mean that an active checkpoint may silently switch + topology: :func:`compare_process_definitions` still requires an explicit + mapping for changes that can affect a checkpoint. + """ + + PATCH = "patch" + COMPATIBLE = "compatible" + MIGRATABLE = "migratable" + BREAKING = "breaking" + + +# A few callers use the shorter terminology from the governance document. +ChangeClassification = ProcessChangeClassification +ProcessCompatibilityClass = ProcessChangeClassification + + +class ProcessChangeImpact(DomainModel): + workflow_name: str + from_revision: int + to_revision: int + added_nodes: tuple[str, ...] = () + removed_nodes: tuple[str, ...] = () + changed_nodes: tuple[str, ...] = () + missing_resume_mappings: tuple[str, ...] = () + compatible_for_in_flight: bool + notes: tuple[str, ...] = Field(default_factory=tuple) + classification: ProcessChangeClassification = ProcessChangeClassification.PATCH + # These fields make the impact report useful to release tooling without + # making consumers parse free-form notes. Names are node names unless + # otherwise stated, and all values are stable and sorted. + changed_transitions: tuple[str, ...] = () + routing_changes: tuple[str, ...] = () + outcome_changes: tuple[str, ...] = () + station_contract_changes: tuple[str, ...] = () + effect_capability_changes: tuple[str, ...] = () + policy_changes: tuple[str, ...] = () + join_changes: tuple[str, ...] = () + concurrency_changes: tuple[str, ...] = () + retry_changes: tuple[str, ...] = () + state_profile_changed: bool = False + entry_changed: bool = False + same_revision_mutation: bool = False + + @property + def compatibility_class(self) -> ProcessChangeClassification: + """Alias used by governance/reporting clients.""" + return self.classification + + +class ProcessMigrationClassification(StrEnum): + """Eligibility of an active instance for a definition revision change.""" + + STAYS_PINNED = "stays_pinned" + CAN_ADOPT_DIRECTLY = "can_adopt_directly" + REQUIRES_RESUME_MAPPING = "requires_resume_mapping" + BLOCKED = "blocked" + + +class ProcessInstanceSnapshot(DomainModel): + """The immutable metadata needed to dry-run one active checkpoint. + + Runtime checkpoints historically used ``workflow_revision`` and + ``workflow_digest``. The simulator deliberately calls these values + ``pinned_*`` to make it clear that they are instance-owned, not the + currently activated definition. + """ + + run_id: str | None = None + thread_id: str | None = None + instance_id: str | None = None + current_node: str | None = None + pinned_revision: int | None = None + pinned_digest: str | None = None + state_profile: str | None = None + + +class ProcessMigrationInstanceResult(DomainModel): + """Deterministic result for one active workflow instance.""" + + identity: str + run_id: str | None = None + thread_id: str | None = None + instance_id: str | None = None + current_node: str | None = None + pinned_revision: int | None = None + pinned_digest: str | None = None + classification: ProcessMigrationClassification + eligible: bool + target_revision: int | None = None + target_node: str | None = None + reason_code: str + reason: str + + @property + def status(self) -> ProcessMigrationClassification: + """Compatibility alias for consumers that call the class a status.""" + return self.classification + + +class ProcessMigrationSimulation(DomainModel): + """Aggregate, deterministic dry-run report for active instances.""" + + workflow_name: str + from_revision: int + to_revision: int + from_digest: str + to_digest: str + instances: tuple[ProcessMigrationInstanceResult, ...] + counts: dict[str, int] + compatible: bool + invalid_resume_mappings: tuple[str, ...] = () + + @property + def details(self) -> tuple[ProcessMigrationInstanceResult, ...]: + """Alias useful to callers that consume reports as ``details``.""" + return self.instances + + @property + def results(self) -> tuple[ProcessMigrationInstanceResult, ...]: + return self.instances + + @property + def by_classification(self) -> dict[str, int]: + return dict(self.counts) + + @property + def total_count(self) -> int: + return len(self.instances) + + @property + def blocked_count(self) -> int: + return self.counts[ProcessMigrationClassification.BLOCKED.value] + + @property + def stays_pinned_count(self) -> int: + return self.counts[ProcessMigrationClassification.STAYS_PINNED.value] + + @property + def can_adopt_directly_count(self) -> int: + return self.counts[ProcessMigrationClassification.CAN_ADOPT_DIRECTLY.value] + + @property + def requires_resume_mapping_count(self) -> int: + return self.counts[ProcessMigrationClassification.REQUIRES_RESUME_MAPPING.value] + + @property + def eligible_count(self) -> int: + return sum(value for key, value in self.counts.items() if key != "blocked") + + +# Short aliases keep the public API pleasant while retaining the explicit +# ``Process*`` names used by the manifest and change-impact models. +MigrationClassification = ProcessMigrationClassification +MigrationInstanceResult = ProcessMigrationInstanceResult +MigrationSimulationResult = ProcessMigrationSimulation +ActiveInstanceSnapshot = ProcessInstanceSnapshot +ProcessMigrationStatus = ProcessMigrationClassification +MigrationStatus = ProcessMigrationClassification + + +def build_process_manifest(definition: WorkflowDefinition) -> ProcessManifest: + """Build an inspectable view from the same definition used by the runtime compiler.""" + from forge.workflow.declarative.compiler import DeclarativeWorkflowCompiler + + DeclarativeWorkflowCompiler(definition).validate() + profile = get_state_profile(definition.spec.state) + nodes = [] + transitions = [] + for name, step in definition.spec.steps.items(): + binding = profile.station_bindings.get(name) + kind = ( + ProcessNodeKind(step.kind) + if step.kind + else ProcessNodeKind.GATE + if name in profile.pause_nodes + else ProcessNodeKind.STATION + if binding + else ProcessNodeKind.OPERATION + ) + nodes.append( + ProcessNode( + name=name, + kind=kind, + station_contract=binding[0] if binding else None, + station_contract_version=binding[1] if binding else None, + required_policies=tuple(sorted(step.required_policies)), + allowed_effects=tuple(sorted(step.allowed_effects)), + join=step.join, + max_concurrency=step.max_concurrency, + retry_bound=step.retry_bound, + ) + ) + if step.next: + transitions.append(ProcessTransition(source=name, target=step.next)) + elif step.dynamic_route: + transitions.extend( + ProcessTransition(source=name, target=target, outcome="dynamic") + for target in step.dynamic_targets + ) + else: + transitions.extend( + ProcessTransition(source=name, target=target, outcome=outcome) + for outcome, target in step.branches.items() + ) + # Mappings are semantically unordered. Keep inspection and rendering + # stable when equivalent definitions use a different source ordering. + nodes.sort(key=lambda node: node.name) + transitions.sort(key=lambda edge: (edge.source, edge.target, edge.outcome or "")) + return ProcessManifest( + workflow_name=definition.metadata.name, + revision=definition.metadata.revision, + digest=definition.digest, + state_profile=definition.spec.state, + entry=definition.spec.entry, + nodes=tuple(nodes), + transitions=tuple(transitions), + ) + + +def render_mermaid(manifest: ProcessManifest) -> str: + """Render a deterministic flowchart from the canonical manifest.""" + lines = ["flowchart TD", f" __start__([start]) --> {manifest.entry}"] + for node in sorted(manifest.nodes, key=lambda item: item.name): + if node.kind is ProcessNodeKind.GATE: + lines.append(f' {node.name}{{"{node.name}"}}') + elif node.kind is ProcessNodeKind.STATION: + lines.append(f' {node.name}["{node.name}\\n{node.station_contract}"]') + else: + lines.append(f' {node.name}["{node.name}"]') + lines.append(" __end__([end])") + for transition in sorted( + manifest.transitions, + key=lambda edge: (edge.source, edge.target, edge.outcome or ""), + ): + label = f"|{transition.outcome}|" if transition.outcome else "" + lines.append(f" {transition.source} -->{label} {transition.target}") + return "\n".join(lines) + + +def compare_process_definitions( + previous: WorkflowDefinition, current: WorkflowDefinition +) -> ProcessChangeImpact: + """Report structural and in-flight compatibility impact before publication.""" + if previous.metadata.name != current.metadata.name: + raise ValueError("Cannot compare definitions with different workflow names") + old = previous.spec.steps + new = current.spec.steps + old_names = set(old) + new_names = set(new) + added = tuple(sorted(new_names - old_names)) + removed = tuple(sorted(old_names - new_names)) + + def step_signature(step: Any) -> tuple[Any, ...]: + """Executable step fields, normalizing fields whose order is irrelevant.""" + return ( + step.next, + step.route, + tuple(sorted(step.branches.items())), + step.dynamic_route, + tuple(sorted(step.dynamic_targets)), + step.kind, + step.station_contract, + step.station_contract_version, + tuple(sorted(step.required_policies)), + tuple(sorted(step.allowed_effects)), + step.join, + step.max_concurrency, + step.retry_bound, + ) + + common = old_names & new_names + changed = tuple( + sorted(name for name in common if step_signature(old[name]) != step_signature(new[name])) + ) + + def transitions(steps: Mapping[str, Any]) -> dict[str, frozenset[tuple[str, str, str | None]]]: + result: dict[str, frozenset[tuple[str, str, str | None]]] = {} + for name, step in steps.items(): + edges: set[tuple[str, str, str | None]] + if step.next: + edges = {(name, step.next, None)} + elif step.dynamic_route: + edges = {(name, target, "dynamic") for target in step.dynamic_targets} + else: + edges = {(name, target, outcome) for outcome, target in step.branches.items()} + result[name] = frozenset(edges) + return result + + old_transitions = transitions(old) + new_transitions = transitions(new) + changed_transitions = tuple( + sorted(name for name in common if old_transitions[name] != new_transitions[name]) + ) + routing_changes: list[str] = [] + outcome_changes: list[str] = [] + for name in changed_transitions: + old_edges = old_transitions[name] + new_edges = new_transitions[name] + old_outcomes = {outcome for _source, _target, outcome in old_edges} + new_outcomes = {outcome for _source, _target, outcome in new_edges} + if old_outcomes != new_outcomes: + outcome_changes.append(name) + if old_edges != new_edges: + routing_changes.append(name) + + station_contract_changes = tuple( + sorted( + name + for name in common + if (old[name].station_contract, old[name].station_contract_version) + != (new[name].station_contract, new[name].station_contract_version) + ) + ) + effect_capability_changes = tuple( + sorted( + name + for name in common + if set(old[name].allowed_effects) != set(new[name].allowed_effects) + ) + ) + policy_changes = tuple( + sorted( + name + for name in common + if set(old[name].required_policies) != set(new[name].required_policies) + ) + ) + if set(previous.spec.mandatory_policies) != set(current.spec.mandatory_policies): + policy_changes = tuple(sorted(set(policy_changes) | {""})) + if set(previous.spec.extension_points) != set(current.spec.extension_points): + policy_changes = tuple(sorted(set(policy_changes) | {""})) + join_changes = tuple(sorted(name for name in common if old[name].join != new[name].join)) + concurrency_changes = tuple( + sorted(name for name in common if old[name].max_concurrency != new[name].max_concurrency) + ) + retry_changes = tuple( + sorted(name for name in common if old[name].retry_bound != new[name].retry_bound) + ) + + mappings = current.spec.resume.from_revisions.get(previous.metadata.revision, {}) + missing = tuple(sorted(name for name in removed if name not in mappings)) + notes: list[str] = [] + state_profile_changed = previous.spec.state != current.spec.state + entry_changed = current.spec.entry != previous.spec.entry + same_revision_mutation = ( + current.metadata.revision == previous.metadata.revision + and current.digest != previous.digest + ) + rollback = current.metadata.revision < previous.metadata.revision + if same_revision_mutation: + notes.append("changed content must increment metadata.revision") + notes.append("same revision has different content (immutable revision mutation)") + if rollback: + notes.append("target revision is older than the source revision") + if state_profile_changed: + notes.append("state profile changes cannot migrate in-flight instances") + if entry_changed: + notes.append("entry changed; this affects new instances only") + if added: + notes.append(f"added nodes: {', '.join(added)}") + if removed: + notes.append(f"removed nodes: {', '.join(removed)}") + if missing: + notes.append(f"missing resume mappings: {', '.join(missing)}") + if routing_changes: + notes.append(f"routing changed on: {', '.join(routing_changes)}") + if outcome_changes: + notes.append(f"outcomes changed on: {', '.join(outcome_changes)}") + if station_contract_changes: + notes.append(f"station contract/version changed on: {', '.join(station_contract_changes)}") + if effect_capability_changes: + notes.append(f"effect capabilities changed on: {', '.join(effect_capability_changes)}") + if policy_changes: + notes.append(f"policies changed on: {', '.join(policy_changes)}") + if join_changes: + notes.append(f"join semantics changed on: {', '.join(join_changes)}") + if concurrency_changes: + notes.append(f"concurrency changed on: {', '.join(concurrency_changes)}") + if retry_changes: + notes.append(f"retry policy changed on: {', '.join(retry_changes)}") + + # Fail closed for anything which can alter the meaning of a checkpoint. + severe = ( + same_revision_mutation + or rollback + or state_profile_changed + or bool(station_contract_changes) + or bool(effect_capability_changes) + or bool(policy_changes) + or bool(join_changes) + or bool(concurrency_changes) + or bool(retry_changes) + ) + removed_mapped = bool(removed) and not missing + if severe or missing: + classification = ProcessChangeClassification.BREAKING + elif removed_mapped: + classification = ProcessChangeClassification.MIGRATABLE + elif routing_changes: + # Retained outcomes are safe for newly-created instances, but there is + # no implicit checkpoint conversion for already-running instances. + old_outcome_removed = any( + {outcome for _s, _t, outcome in old_transitions[name]} + - {outcome for _s, _t, outcome in new_transitions[name]} + for name in changed_transitions + ) + only_additive_outcomes = all( + old_transitions[name] <= new_transitions[name] for name in changed_transitions + ) + classification = ( + ProcessChangeClassification.BREAKING + if old_outcome_removed + else ProcessChangeClassification.COMPATIBLE + if only_additive_outcomes + else ProcessChangeClassification.MIGRATABLE + ) + elif added or entry_changed: + classification = ProcessChangeClassification.COMPATIBLE + else: + classification = ProcessChangeClassification.PATCH + + compatible = classification in { + ProcessChangeClassification.PATCH, + ProcessChangeClassification.COMPATIBLE, + } + if removed_mapped and not severe and not routing_changes: + compatible = True + if routing_changes or station_contract_changes or effect_capability_changes: + compatible = False + if state_profile_changed or same_revision_mutation or rollback or missing: + compatible = False + return ProcessChangeImpact( + workflow_name=current.metadata.name, + from_revision=previous.metadata.revision, + to_revision=current.metadata.revision, + added_nodes=added, + removed_nodes=removed, + changed_nodes=changed, + missing_resume_mappings=missing, + compatible_for_in_flight=compatible, + notes=tuple(notes), + classification=classification, + changed_transitions=changed_transitions, + routing_changes=tuple(sorted(routing_changes)), + outcome_changes=tuple(sorted(outcome_changes)), + station_contract_changes=station_contract_changes, + effect_capability_changes=effect_capability_changes, + policy_changes=policy_changes, + join_changes=join_changes, + concurrency_changes=concurrency_changes, + retry_changes=retry_changes, + state_profile_changed=state_profile_changed, + entry_changed=entry_changed, + same_revision_mutation=same_revision_mutation, + ) + + +_CONTROL_NODES = frozenset({"", "start", "entry", "__end__", "complete"}) +_SNAPSHOT_KEYS: dict[str, tuple[str, ...]] = { + "run_id": ("run_id", "runId", "run"), + "thread_id": ("thread_id", "threadId", "thread"), + "instance_id": ("instance_id", "instanceId", "id"), + "current_node": ("current_node", "currentNode", "node"), + "pinned_revision": ( + "pinned_revision", + "pinnedRevision", + "workflow_revision", + "workflowRevision", + "revision", + ), + "pinned_digest": ( + "pinned_digest", + "pinnedDigest", + "workflow_digest", + "workflowDigest", + "digest", + ), + "state_profile": ( + "state_profile", + "stateProfile", + "workflow_state_profile", + "workflowStateProfile", + ), +} + + +def _snapshot_field(snapshot: Mapping[str, Any], name: str) -> Any: + for key in _SNAPSHOT_KEYS[name]: + if key in snapshot: + return snapshot[key] + return None + + +def _coerce_snapshot(value: ProcessInstanceSnapshot | Mapping[str, Any]) -> ProcessInstanceSnapshot: + if isinstance(value, ProcessInstanceSnapshot): + return value + if not isinstance(value, Mapping): + raise TypeError("active instances must be mappings or ProcessInstanceSnapshot values") + raw_revision = _snapshot_field(value, "pinned_revision") + try: + revision = int(raw_revision) if raw_revision is not None else None + except (TypeError, ValueError): + revision = None + + def as_text(field: str) -> str | None: + raw = _snapshot_field(value, field) + return str(raw) if raw is not None else None + + return ProcessInstanceSnapshot( + run_id=as_text("run_id"), + thread_id=as_text("thread_id"), + instance_id=as_text("instance_id"), + current_node=as_text("current_node"), + pinned_revision=revision, + pinned_digest=as_text("pinned_digest"), + state_profile=as_text("state_profile"), + ) + + +def _instance_identity(snapshot: ProcessInstanceSnapshot) -> str: + """Build an identity that remains stable when input order changes.""" + if snapshot.instance_id: + return snapshot.instance_id + parts = [] + if snapshot.run_id: + parts.append(f"run:{snapshot.run_id}") + if snapshot.thread_id: + parts.append(f"thread:{snapshot.thread_id}") + return "/".join(parts) or "anonymous" + + +def _invalid_mapping_entries(definition: WorkflowDefinition) -> tuple[str, ...]: + """Return invalid mapping entries in a stable, human-readable format.""" + invalid: list[str] = [] + for source_revision, mappings in definition.spec.resume.from_revisions.items(): + for source, target in mappings.items(): + if target not in definition.spec.steps: + invalid.append(f"{source_revision}:{source}->{target}") + return tuple(sorted(invalid)) + + +def _migration_result( + snapshot: ProcessInstanceSnapshot, + *, + classification: ProcessMigrationClassification, + eligible: bool, + reason_code: str, + reason: str, + target_revision: int | None = None, + target_node: str | None = None, +) -> ProcessMigrationInstanceResult: + return ProcessMigrationInstanceResult( + identity=_instance_identity(snapshot), + run_id=snapshot.run_id, + thread_id=snapshot.thread_id, + instance_id=snapshot.instance_id, + current_node=snapshot.current_node, + pinned_revision=snapshot.pinned_revision, + pinned_digest=snapshot.pinned_digest, + classification=classification, + eligible=eligible, + target_revision=target_revision, + target_node=target_node, + reason_code=reason_code, + reason=reason, + ) + + +def simulate_process_migration( + previous: WorkflowDefinition, + current: WorkflowDefinition, + active_instances: Iterable[ProcessInstanceSnapshot | Mapping[str, Any]], +) -> ProcessMigrationSimulation: + """Dry-run adoption of ``current`` by active instances pinned to ``previous``. + + A simulation never mutates checkpoints. An instance can be adopted directly + when its saved node still exists in the new definition; a removed node needs + an explicit mapping in the new immutable artifact. Every mismatch in pinned + identity is reported as blocked so operators can distinguish an unsafe source + snapshot from a merely unmapped node. + """ + impact = compare_process_definitions(previous, current) + invalid_mappings = _invalid_mapping_entries(current) + old_revision = previous.metadata.revision + new_revision = current.metadata.revision + mappings = current.spec.resume.from_revisions.get(old_revision, {}) + results: list[ProcessMigrationInstanceResult] = [] + + for item in active_instances: + snapshot = _coerce_snapshot(item) + revision = snapshot.pinned_revision + digest = snapshot.pinned_digest + node = snapshot.current_node + + if revision is None or digest is None or node is None: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="incomplete_snapshot", + reason="active instance is missing current_node, pinned revision, or pinned digest", + ) + ) + continue + if snapshot.state_profile and snapshot.state_profile != previous.spec.state: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="state_profile_incompatible", + reason="pinned state profile does not match the source definition", + ) + ) + continue + if previous.spec.state != current.spec.state: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="state_profile_incompatible", + reason="source and target definitions use incompatible state profiles", + ) + ) + continue + if revision == new_revision: + if digest != current.digest: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="same_revision_digest_mutation", + reason="pinned revision has a different digest than the target artifact", + ) + ) + else: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.STAYS_PINNED, + eligible=True, + reason_code="already_on_target", + reason="instance is already pinned to the target artifact", + target_revision=new_revision, + target_node=node, + ) + ) + continue + if revision != old_revision or digest != previous.digest: + code = "wrong_source_revision" if revision != old_revision else "wrong_source_digest" + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code=code, + reason="pinned artifact does not match the source definition being simulated", + ) + ) + continue + if new_revision <= old_revision: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="revision_rollback", + reason="target revision is not newer than the pinned source revision", + ) + ) + continue + + if node in _CONTROL_NODES or node in current.spec.steps: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.CAN_ADOPT_DIRECTLY, + eligible=True, + reason_code="node_preserved", + reason="saved node exists in the target definition", + target_revision=new_revision, + target_node=node, + ) + ) + continue + if node not in mappings: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="removed_node_without_mapping", + reason="saved node was removed and has no declared resume mapping", + ) + ) + continue + target = mappings[node] + if target not in current.spec.steps: + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.BLOCKED, + eligible=False, + reason_code="invalid_mapping_target", + reason="declared resume mapping targets an undeclared node", + ) + ) + continue + results.append( + _migration_result( + snapshot, + classification=ProcessMigrationClassification.REQUIRES_RESUME_MAPPING, + eligible=True, + reason_code="declared_resume_mapping", + reason="saved node requires the declared resume mapping", + target_revision=new_revision, + target_node=target, + ) + ) + + results.sort(key=lambda result: (result.identity, result.run_id or "", result.thread_id or "")) + counts = {classification.value: 0 for classification in ProcessMigrationClassification} + for result in results: + counts[result.classification.value] += 1 + return ProcessMigrationSimulation( + workflow_name=current.metadata.name, + from_revision=old_revision, + to_revision=new_revision, + from_digest=previous.digest, + to_digest=current.digest, + instances=tuple(results), + counts=counts, + compatible=not counts[ProcessMigrationClassification.BLOCKED.value] + and impact.compatible_for_in_flight, + invalid_resume_mappings=invalid_mappings, + ) + + +# Explicitly named alias for callers that use the report terminology. +simulate_process_definition_migration = simulate_process_migration +simulate_migration = simulate_process_migration diff --git a/src/forge/workflow/declarative/models.py b/src/forge/workflow/declarative/models.py index 735b2a505..b162d1393 100644 --- a/src/forge/workflow/declarative/models.py +++ b/src/forge/workflow/declarative/models.py @@ -40,6 +40,16 @@ class WorkflowStep(StrictModel): next: str | None = None route: str | None = None branches: dict[str, str] = Field(default_factory=dict) + dynamic_route: bool = Field(default=False, alias="dynamicRoute") + dynamic_targets: tuple[str, ...] = Field(default=(), alias="dynamicTargets") + kind: Literal["station", "gate", "operation"] | None = None + station_contract: str | None = Field(default=None, alias="stationContract") + station_contract_version: str | None = Field(default=None, alias="stationContractVersion") + required_policies: tuple[str, ...] = Field(default=(), alias="requiredPolicies") + allowed_effects: tuple[str, ...] = Field(default=(), alias="allowedEffects") + join: Literal["all", "any"] | None = None + max_concurrency: int | None = Field(default=None, alias="maxConcurrency", ge=1, le=64) + retry_bound: int | None = Field(default=None, alias="retryBound", ge=1, le=100) @model_validator(mode="after") def validate_transition(self) -> WorkflowStep: @@ -47,10 +57,28 @@ def validate_transition(self) -> WorkflowStep: raise ValueError("exactly one of 'next' or 'route' is required") if self.next and self.branches: raise ValueError("branches are only valid with 'route'") - if self.route and not self.branches: + if self.route and not self.branches and not self.dynamic_route: raise ValueError("a routed step requires non-empty branches") + if self.dynamic_route and (not self.route or self.branches or not self.dynamic_targets): + raise ValueError("dynamicRoute requires a route and cannot declare static branches") + if not self.dynamic_route and self.dynamic_targets: + raise ValueError("dynamicTargets are only valid with dynamicRoute") if len(self.branches) > MAX_BRANCHES: raise ValueError(f"a routed step may have at most {MAX_BRANCHES} branches") + if self.kind == "station" and not (self.station_contract and self.station_contract_version): + raise ValueError("station steps require stationContract and stationContractVersion") + if self.kind not in {None, "station", "gate"} and ( + self.station_contract or self.station_contract_version + ): + raise ValueError("station contract fields are only valid for station steps") + if bool(self.station_contract) != bool(self.station_contract_version): + raise ValueError("stationContract and stationContractVersion must be declared together") + if self.max_concurrency is not None and not self.dynamic_route: + raise ValueError("maxConcurrency is only valid for dynamic routing") + if self.dynamic_route and self.max_concurrency is None: + raise ValueError("dynamicRoute requires an explicit maxConcurrency") + if self.join is not None and self.dynamic_route: + raise ValueError("a fan-out step cannot also be a join") return self @@ -66,6 +94,8 @@ class WorkflowSpec(StrictModel): entry: str steps: dict[str, WorkflowStep] resume: WorkflowResume = Field(default_factory=WorkflowResume) + mandatory_policies: tuple[str, ...] = Field(default=(), alias="mandatoryPolicies") + extension_points: tuple[str, ...] = Field(default=(), alias="extensionPoints") @field_validator("entry") @classmethod diff --git a/src/forge/workflow/declarative/publication.py b/src/forge/workflow/declarative/publication.py new file mode 100644 index 000000000..b2c3d32f6 --- /dev/null +++ b/src/forge/workflow/declarative/publication.py @@ -0,0 +1,479 @@ +"""Immutable process-definition publication and explicit rollout decisions.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any, Literal + +from pydantic import Field + +from forge.domain import DomainModel +from forge.orchestrator.checkpointer import get_redis_client +from forge.workflow.declarative.compiler import DeclarativeWorkflowCompiler +from forge.workflow.declarative.manifest import ProcessChangeImpact, compare_process_definitions +from forge.workflow.declarative.models import WorkflowDefinition + +_DEFINITION_PREFIX = "forge:process:def:" +_ACTIVE_PREFIX = "forge:process:active:" +_DECISIONS_PREFIX = "forge:process:decisions:" +_LATEST_PREFIX = "forge:process:latest:" + +# Return values: 1 new publication, 0 idempotent publication, -1 immutable +# collision, -2 revision went backwards for changed content. +_PUBLISH_SCRIPT = """ +local existing = redis.call('GET', KEYS[1]) +if existing and existing ~= ARGV[1] then return -1 end +local latest = redis.call('GET', KEYS[2]) +if not existing and latest then + local separator = string.find(latest, ':') + local latest_revision = tonumber(string.sub(latest, 1, separator - 1)) + local latest_digest = string.sub(latest, separator + 1) + if ARGV[3] ~= latest_digest and tonumber(ARGV[2]) <= latest_revision then return -2 end +end +if not existing then + redis.call('SET', KEYS[1], ARGV[1]) + if not latest or tonumber(ARGV[2]) > tonumber(string.sub(latest, 1, string.find(latest, ':') - 1)) then + redis.call('SET', KEYS[2], ARGV[2] .. ':' .. ARGV[3]) + end +end +redis.call('RPUSH', KEYS[3], ARGV[4]) +return existing and 0 or 1 +""" + +_ACTIVATE_SCRIPT = """ +local target = redis.call('GET', KEYS[1]) +if not target then return -1 end +local active = redis.call('GET', KEYS[2]) +if ARGV[1] ~= '' and (not active or string.sub(active, string.find(active, ':') + 1) ~= ARGV[1]) then return -2 end +redis.call('SET', KEYS[2], ARGV[2]) +redis.call('RPUSH', KEYS[3], ARGV[3]) +return 1 +""" + + +class PublicationDecision(DomainModel): + """An immutable audit entry for publishing or changing activation.""" + + workflow_name: str + revision: int + digest: str + published_at: datetime + activated: bool = False + actor: str + impact: dict[str, Any] = Field(default_factory=dict) + project_key: str = "" + action: Literal["publish", "activate", "rollback"] = "publish" + reason: str = "" + + +def _compatible_impact(impact: ProcessChangeImpact, *, rollback: bool = False) -> bool: + """Fail closed for activation; rollback only relaxes revision direction.""" + if rollback: + return not impact.state_profile_changed and not impact.missing_resume_mappings + return impact.compatible_for_in_flight + + +class DefinitionPublisher: + """Project-scoped immutable definition store and rollout decision log.""" + + def __init__(self, project_key: str, redis_client: Any = None) -> None: + if not project_key or not project_key.strip(): + raise ValueError("project_key is required for governed publication") + self.project_key = project_key.upper() + self._redis = redis_client + + async def _client(self) -> Any: + if self._redis is None: + self._redis = await get_redis_client() + return self._redis + + async def publish( + self, definition: WorkflowDefinition, *, actor: str, reason: str, activate: bool = False + ) -> PublicationDecision: + """Validate and persist an immutable artifact, without activating it.""" + if activate: + raise ValueError("publication and activation are separate decisions; use activate()") + self._validate(definition) + decision = self._decision(definition, actor=actor, reason=reason, action="publish") + result = await (await self._client()).eval( + _PUBLISH_SCRIPT, + 3, + self._definition_key(definition.metadata.name, definition.metadata.revision), + self._latest_key(definition.metadata.name), + self._decisions_key(definition.metadata.name), + definition.canonical_json(), + str(definition.metadata.revision), + definition.digest, + decision.model_dump_json(), + ) + if result == -1: + raise ValueError("published revision is immutable and has different content") + if result == -2: + raise ValueError("changed workflow content must increment metadata.revision") + return decision + + async def activate( + self, + name: str | WorkflowDefinition, + revision: int | None = None, + *, + actor: str, + reason: str, + expected_active_digest: str | None = None, + ) -> PublicationDecision: + """Activate an existing artifact using compare-and-set semantics.""" + name, revision = self._target_identity(name, revision) + return await self._set_active( + name, + revision, + actor=actor, + reason=reason, + action="activate", + expected_active_digest=expected_active_digest, + ) + + async def rollback( + self, + name: str | WorkflowDefinition, + revision: int | None = None, + *, + actor: str, + reason: str, + expected_active_digest: str | None = None, + ) -> PublicationDecision: + """Move activation to an older compatible artifact; never mutate history.""" + name, revision = self._target_identity(name, revision) + current = await self.active(name) + if current is None or revision >= current.metadata.revision: + raise ValueError("rollback target must be an already-published older revision") + return await self._set_active( + name, + revision, + actor=actor, + reason=reason, + action="rollback", + expected_active_digest=expected_active_digest, + ) + + async def _set_active( + self, + name: str, + revision: int, + *, + actor: str, + reason: str, + action: Literal["activate", "rollback"], + expected_active_digest: str | None, + ) -> PublicationDecision: + target = await self.get(name, revision) + if target is None: + raise ValueError(f"published workflow '{name}' revision {revision} is unavailable") + if target.metadata.name != name: + raise ValueError("published workflow name does not match activation key") + self._validate(target) + previous = await self.active(name) + if previous is not None and expected_active_digest is None: + raise ValueError( + "expected_active_digest is required when replacing an active definition" + ) + if expected_active_digest and ( + previous is None or previous.digest != expected_active_digest + ): + raise ValueError("active definition changed concurrently") + impact = compare_process_definitions(previous, target) if previous else None + if impact is not None and not _compatible_impact(impact, rollback=action == "rollback"): + raise ValueError("definition is incompatible with active workflow instances") + decision = self._decision( + target, actor=actor, reason=reason, action=action, activated=True, impact=impact + ) + result = await (await self._client()).eval( + _ACTIVATE_SCRIPT, + 3, + self._definition_key(name, revision), + self._active_key(name), + self._decisions_key(name), + expected_active_digest or "", + self._pointer(target), + decision.model_dump_json(), + ) + if result == -1: + raise ValueError(f"published workflow '{name}' revision {revision} is unavailable") + if result == -2: + raise ValueError("active definition changed concurrently") + return decision + + async def get(self, name: str, revision: int) -> WorkflowDefinition | None: + value = await (await self._client()).get(self._definition_key(name, revision)) + return WorkflowDefinition.model_validate_json(value) if value else None + + async def active(self, name: str) -> WorkflowDefinition | None: + pointer = await (await self._client()).get(self._active_key(name)) + if not pointer: + return None + text = pointer.decode() if isinstance(pointer, bytes) else str(pointer) + revision, _, _digest = text.partition(":") + return await self.get(name, int(revision)) + + async def decisions(self, name: str) -> tuple[PublicationDecision, ...]: + values = await (await self._client()).lrange(self._decisions_key(name), 0, -1) + return tuple(PublicationDecision.model_validate_json(value) for value in values) + + async def history(self, name: str) -> tuple[WorkflowDefinition, ...]: + redis = await self._client() + keys: list[str] = [] + cursor = 0 + while True: + cursor, found = await redis.scan(cursor=cursor, match=self._definition_key(name, "*")) + keys.extend(found) + if cursor == 0: + break + definitions = [] + for key in keys: + value = await redis.get(key) + if value: + definitions.append(WorkflowDefinition.model_validate_json(value)) + return tuple(sorted(definitions, key=lambda item: item.metadata.revision)) + + async def list_workflows(self) -> tuple[str, ...]: + redis = await self._client() + names: set[str] = set() + cursor = 0 + prefix = f"{_DEFINITION_PREFIX}{self.project_key}:" + while True: + cursor, found = await redis.scan(cursor=cursor, match=f"{prefix}*") + for key in found: + text = key.decode() if isinstance(key, bytes) else str(key) + remainder = text[len(prefix) :] + if ":" in remainder: + names.add(remainder.rsplit(":", 1)[0]) + if cursor == 0: + break + return tuple(sorted(names)) + + def _validate(self, definition: WorkflowDefinition) -> None: + definition.validate_property_size() + DeclarativeWorkflowCompiler(definition).validate_for_publication() + + def _decision( + self, + definition: WorkflowDefinition, + *, + actor: str, + reason: str, + action: Literal["publish", "activate", "rollback"], + activated: bool = False, + impact: ProcessChangeImpact | None = None, + ) -> PublicationDecision: + if not actor.strip(): + raise ValueError("actor is required for governed decisions") + if not reason.strip(): + raise ValueError("reason is required for governed decisions") + return PublicationDecision( + project_key=self.project_key, + workflow_name=definition.metadata.name, + revision=definition.metadata.revision, + digest=definition.digest, + published_at=datetime.now(UTC), + activated=activated, + actor=actor, + reason=reason, + action=action, + impact=impact.model_dump(mode="json") if impact else {}, + ) + + @staticmethod + def _target_identity(name: str | WorkflowDefinition, revision: int | None) -> tuple[str, int]: + if isinstance(name, WorkflowDefinition): + if revision is not None and revision != name.metadata.revision: + raise ValueError("activation revision does not match definition") + return name.metadata.name, name.metadata.revision + if revision is None: + raise ValueError("activation revision is required") + return name, revision + + def _prefix(self, prefix: str, name: str) -> str: + return f"{prefix}{self.project_key}:{name}" + + def _definition_key(self, name: str, revision: int | str) -> str: + return f"{self._prefix(_DEFINITION_PREFIX, name)}:{revision}" + + def _latest_key(self, name: str) -> str: + return self._prefix(_LATEST_PREFIX, name) + + def _active_key(self, name: str) -> str: + return self._prefix(_ACTIVE_PREFIX, name) + + def _decisions_key(self, name: str) -> str: + return self._prefix(_DECISIONS_PREFIX, name) + + @staticmethod + def _pointer(definition: WorkflowDefinition) -> str: + return f"{definition.metadata.revision}:{definition.digest}" + + +class InMemoryDefinitionPublisher: + """Deterministic project-scoped publisher for local and contract tests.""" + + def __init__(self, project_key: str = "DEFAULT") -> None: + if not project_key or not project_key.strip(): + raise ValueError("project_key is required for governed publication") + self.project_key = project_key.upper() + self._definitions: dict[tuple[str, int], WorkflowDefinition] = {} + self._active: dict[str, WorkflowDefinition] = {} + self._decisions: dict[str, list[PublicationDecision]] = {} + + async def publish( + self, definition: WorkflowDefinition, *, actor: str, reason: str, activate: bool = False + ) -> PublicationDecision: + if activate: + raise ValueError("publication and activation are separate decisions; use activate()") + self._validate(definition) + key = (definition.metadata.name, definition.metadata.revision) + existing = self._definitions.get(key) + if existing is not None and existing.digest != definition.digest: + raise ValueError("published revision is immutable and has different content") + published = await self.history(definition.metadata.name) + if any( + item.digest != definition.digest + and item.metadata.revision >= definition.metadata.revision + for item in published + ): + raise ValueError("changed workflow content must increment metadata.revision") + self._definitions[key] = definition + decision = self._decision(definition, actor=actor, reason=reason, action="publish") + self._decisions.setdefault(definition.metadata.name, []).append(decision) + return decision + + async def activate( + self, + name: str | WorkflowDefinition, + revision: int | None = None, + *, + actor: str, + reason: str, + expected_active_digest: str | None = None, + ) -> PublicationDecision: + name, revision = self._target_identity(name, revision) + return await self._set_active( + name, + revision, + actor=actor, + reason=reason, + action="activate", + expected_active_digest=expected_active_digest, + ) + + async def rollback( + self, + name: str | WorkflowDefinition, + revision: int | None = None, + *, + actor: str, + reason: str, + expected_active_digest: str | None = None, + ) -> PublicationDecision: + name, revision = self._target_identity(name, revision) + current = self._active.get(name) + if current is None or revision >= current.metadata.revision: + raise ValueError("rollback target must be an already-published older revision") + return await self._set_active( + name, + revision, + actor=actor, + reason=reason, + action="rollback", + expected_active_digest=expected_active_digest, + ) + + async def _set_active( + self, + name: str, + revision: int, + *, + actor: str, + reason: str, + action: Literal["activate", "rollback"], + expected_active_digest: str | None, + ) -> PublicationDecision: + target = self._definitions.get((name, revision)) + if target is None: + raise ValueError(f"published workflow '{name}' revision {revision} is unavailable") + if target.metadata.name != name: + raise ValueError("published workflow name does not match activation key") + current = self._active.get(name) + if current is not None and expected_active_digest is None: + raise ValueError( + "expected_active_digest is required when replacing an active definition" + ) + if expected_active_digest and (current is None or current.digest != expected_active_digest): + raise ValueError("active definition changed concurrently") + impact = compare_process_definitions(current, target) if current else None + if impact and not _compatible_impact(impact, rollback=action == "rollback"): + raise ValueError("definition is incompatible with active workflow instances") + self._active[name] = target + decision = self._decision( + target, actor=actor, reason=reason, action=action, activated=True, impact=impact + ) + self._decisions.setdefault(name, []).append(decision) + return decision + + async def get(self, name: str, revision: int) -> WorkflowDefinition | None: + return self._definitions.get((name, revision)) + + async def active(self, name: str) -> WorkflowDefinition | None: + return self._active.get(name) + + async def decisions(self, name: str) -> tuple[PublicationDecision, ...]: + return tuple(self._decisions.get(name, ())) + + async def history(self, name: str) -> tuple[WorkflowDefinition, ...]: + return tuple( + sorted( + (definition for (item, _), definition in self._definitions.items() if item == name), + key=lambda item: item.metadata.revision, + ) + ) + + async def list_workflows(self) -> tuple[str, ...]: + return tuple(sorted({name for name, _ in self._definitions})) + + def _validate(self, definition: WorkflowDefinition) -> None: + definition.validate_property_size() + DeclarativeWorkflowCompiler(definition).validate_for_publication() + + def _decision( + self, + definition: WorkflowDefinition, + *, + actor: str, + reason: str, + action: Literal["publish", "activate", "rollback"], + activated: bool = False, + impact: ProcessChangeImpact | None = None, + ) -> PublicationDecision: + if not actor.strip(): + raise ValueError("actor is required for governed decisions") + if not reason.strip(): + raise ValueError("reason is required for governed decisions") + return PublicationDecision( + project_key=self.project_key, + workflow_name=definition.metadata.name, + revision=definition.metadata.revision, + digest=definition.digest, + published_at=datetime.now(UTC), + activated=activated, + actor=actor, + reason=reason, + action=action, + impact=impact.model_dump(mode="json") if impact else {}, + ) + + @staticmethod + def _target_identity(name: str | WorkflowDefinition, revision: int | None) -> tuple[str, int]: + if isinstance(name, WorkflowDefinition): + if revision is not None and revision != name.metadata.revision: + raise ValueError("activation revision does not match definition") + return name.metadata.name, name.metadata.revision + if revision is None: + raise ValueError("activation revision is required") + return name, revision diff --git a/src/forge/workflow/declarative/resolver.py b/src/forge/workflow/declarative/resolver.py index 47c2601c7..712e12848 100644 --- a/src/forge/workflow/declarative/resolver.py +++ b/src/forge/workflow/declarative/resolver.py @@ -13,6 +13,12 @@ from forge.workflow.declarative.workflow import DeclarativeWorkflow +class DefinitionReader(Protocol): + async def get(self, name: str, revision: int) -> Any | None: ... + + async def active(self, name: str) -> Any | None: ... + + class ProjectPropertyReader(Protocol): async def get_project_property(self, project_key: str, property_key: str) -> Any | None: ... @@ -35,18 +41,65 @@ def selected_workflow_name(labels: list[str]) -> str | None: async def load_project_workflow( - jira: ProjectPropertyReader, + jira: ProjectPropertyReader | None, project_key: str, workflow_name: str, + *, + pinned_revision: int | None = None, + pinned_digest: str | None = None, + pinned_definition: dict[str, Any] | None = None, + definition_reader: DefinitionReader | None = None, ) -> DeclarativeWorkflow: - value = await jira.get_project_property( - project_key.upper(), f"{WORKFLOW_PROPERTY_PREFIX}{workflow_name}" + """Resolve an active workflow, or an exact immutable pinned artifact. + + A checkpoint's canonical definition is preferred because it is the durable + source of truth for an in-flight instance. If only identity metadata was + persisted, ``definition_reader`` must provide the exact published revision; + this function deliberately never falls back to Jira's active property for a + pinned checkpoint. + """ + is_pinned = ( + pinned_revision is not None or pinned_digest is not None or pinned_definition is not None ) - if value is None: - raise ValueError( - f"project {project_key.upper()} does not define workflow '{workflow_name}'" - ) - definition = load_workflow_value(value) + if is_pinned: + if pinned_revision is None or not pinned_digest: + raise ValueError("pinned workflow identity requires both revision and digest") + if pinned_definition is not None: + definition = load_workflow_value(pinned_definition) + else: + if definition_reader is None: + raise ValueError( + f"published workflow '{workflow_name}' revision {pinned_revision} is unavailable" + ) + value = await definition_reader.get(workflow_name, int(pinned_revision)) + if value is None: + raise ValueError( + f"published workflow '{workflow_name}' revision {pinned_revision} is unavailable" + ) + definition = value if hasattr(value, "digest") else load_workflow_value(value) + if definition.metadata.name != workflow_name: + raise ValueError("pinned workflow definition name does not match checkpoint") + if definition.metadata.revision != int(pinned_revision): + raise ValueError("pinned workflow definition revision does not match checkpoint") + if definition.digest != pinned_digest: + raise ValueError("pinned workflow definition digest does not match checkpoint") + else: + published = await definition_reader.active(workflow_name) if definition_reader else None + if published is not None: + definition = ( + published if hasattr(published, "digest") else load_workflow_value(published) + ) + else: + if jira is None: + raise ValueError("no active governed workflow definition is available") + value = await jira.get_project_property( + project_key.upper(), f"{WORKFLOW_PROPERTY_PREFIX}{workflow_name}" + ) + if value is None: + raise ValueError( + f"project {project_key.upper()} does not define workflow '{workflow_name}'" + ) + definition = load_workflow_value(value) if definition.metadata.name != workflow_name: raise ValueError( f"workflow property name '{workflow_name}' does not match metadata name " diff --git a/src/forge/workflow/declarative/workflow.py b/src/forge/workflow/declarative/workflow.py index f29226ebc..5b6e8bbab 100644 --- a/src/forge/workflow/declarative/workflow.py +++ b/src/forge/workflow/declarative/workflow.py @@ -10,6 +10,7 @@ from forge.workflow.base import BaseWorkflow from forge.workflow.declarative.catalog import get_state_profile from forge.workflow.declarative.compiler import DeclarativeWorkflowCompiler, WorkflowValidationError +from forge.workflow.declarative.loader import load_workflow_value from forge.workflow.declarative.models import WorkflowDefinition @@ -65,22 +66,137 @@ def workflow_metadata(self) -> dict[str, Any]: "workflow_name": self.name, "workflow_revision": self.definition.metadata.revision, "workflow_digest": self.definition.digest, + "workflow_definition_revision": self.definition.metadata.revision, + "workflow_definition_digest": self.definition.digest, + "workflow_definition": self.definition.canonical_dict(), + "workflow_pin_status": "pinned", "workflow_state_profile": self.definition.spec.state, "workflow_project_key": self.project_key, "workflow_transition_count": 0, } + @staticmethod + def pin_status(state: dict[str, Any]) -> str: + """Classify checkpoint identity without changing the checkpoint. + + A checkpoint written before immutable definitions were introduced has a + workflow name (or no workflow identity at all), but no revision/digest. + Keeping this classification explicit lets callers choose a deliberate + legacy default instead of accidentally treating the active property as + an instance migration. + """ + if not state.get("workflow_name"): + return "unidentified" + revision = state.get("workflow_definition_revision", state.get("workflow_revision")) + digest = state.get("workflow_definition_digest", state.get("workflow_digest")) + if revision is None or digest is None: + return "legacy_unpinned" + return "pinned" + + def pin_legacy_state(self, state: dict[str, Any]) -> dict[str, Any]: + """Explicitly pin a legacy checkpoint to this active definition. + + This is intentionally separate from :meth:`migrate_state`: legacy + checkpoints have no source artifact to validate and therefore cannot be + migrated. Operators may use this one-time compatibility default when + accepting the currently active definition for an old checkpoint. + """ + if state.get("workflow_name") and state.get("workflow_name") != self.name: + raise WorkflowValidationError("a checkpoint cannot switch workflow identity") + return {**state, **self.workflow_metadata(), "workflow_pin_status": "legacy_active_default"} + + def validate_pinned_state(self, state: dict[str, Any]) -> None: + """Reject a checkpoint whose durable artifact identity is inconsistent.""" + if not state.get("workflow_name"): + return + if state.get("workflow_name") != self.name: + raise WorkflowValidationError("checkpoint workflow name does not match definition") + try: + revisions = { + int(value) + for value in ( + state.get("workflow_definition_revision"), + state.get("workflow_revision"), + ) + if value is not None + } + except (TypeError, ValueError) as exc: + raise WorkflowValidationError("checkpoint workflow revision is invalid") from exc + digests = { + str(value) + for value in ( + state.get("workflow_definition_digest"), + state.get("workflow_digest"), + ) + if value is not None + } + if len(revisions) > 1 or len(digests) > 1: + raise WorkflowValidationError("checkpoint contains conflicting workflow identities") + revision = next(iter(revisions), None) + digest = next(iter(digests), None) + if revision is None or digest is None: + return + if revision != self.definition.metadata.revision or digest != self.definition.digest: + raise WorkflowValidationError( + "checkpoint is pinned to an unavailable or different workflow definition" + ) + canonical = state.get("workflow_definition") + if canonical is not None: + try: + persisted = load_workflow_value(canonical) + except Exception as exc: + raise WorkflowValidationError( + "checkpoint contains an invalid workflow definition" + ) from exc + if persisted.digest != self.definition.digest: + raise WorkflowValidationError( + "checkpoint definition digest does not match its identity" + ) + def migrate_state(self, state: dict[str, Any]) -> dict[str, Any]: """Adopt this definition while refusing ambiguous or unsafe migration.""" if not state.get("workflow_name"): return state if state.get("workflow_name") != self.name: raise WorkflowValidationError("an active checkpoint cannot switch workflow identity") + + # A canonical artifact is authoritative. A caller must use this + # method explicitly to change revision; normal resume validates and + # resolves the pinned artifact instead. + canonical = state.get("workflow_definition") + if canonical is not None: + try: + persisted = load_workflow_value(canonical) + except Exception as exc: + raise WorkflowValidationError( + "checkpoint contains an invalid workflow definition" + ) from exc + old_digest = state.get("workflow_definition_digest", state.get("workflow_digest")) + if persisted.digest != old_digest: + raise WorkflowValidationError( + "checkpoint definition digest does not match its identity" + ) + if persisted.metadata.name != self.name: + raise WorkflowValidationError( + "checkpoint definition name does not match its identity" + ) + if persisted.metadata.revision != int( + state.get("workflow_definition_revision", state.get("workflow_revision", 0)) + ): + raise WorkflowValidationError( + "checkpoint definition revision does not match its identity" + ) if state.get("workflow_state_profile") != self.definition.spec.state: raise WorkflowValidationError("an active workflow cannot change state profile") - old_revision = int(state.get("workflow_revision", 0)) - old_digest = state.get("workflow_digest") + old_revision = int( + state.get("workflow_definition_revision", state.get("workflow_revision", 0)) + ) + old_digest = state.get("workflow_definition_digest", state.get("workflow_digest")) + if old_revision < 1 or not old_digest: + raise WorkflowValidationError( + "explicit migration requires a pinned source revision and digest" + ) new_revision = self.definition.metadata.revision if old_revision == new_revision and old_digest != self.definition.digest: raise WorkflowValidationError("workflow content changed without incrementing revision") diff --git a/src/forge/workflow/effect_runtime.py b/src/forge/workflow/effect_runtime.py index 1e2c1f2a8..3fb86a77b 100644 --- a/src/forge/workflow/effect_runtime.py +++ b/src/forge/workflow/effect_runtime.py @@ -61,6 +61,7 @@ ReviewComment, WriteTarget, ) +from forge.workflow.declarative.capabilities import require_effect_capability from forge.workspace.git_ops import GitOperations _service: ContextVar[EffectService | None] = ContextVar("workflow_effect_service", default=None) @@ -170,6 +171,7 @@ async def _write( "origin": location, } effect_id = stable_identity("effect", parts) + require_effect_capability(operation) record = await self._runtime().execute_required( EffectCommand( effect_id=effect_id, @@ -422,6 +424,7 @@ async def push_repository( "commit_sha": commit_sha, }, ) + require_effect_capability(REPOSITORY_PUSH_OPERATION) await service.execute_required( EffectCommand( effect_id=effect_id, @@ -522,6 +525,7 @@ async def _write( "origin": location, }, ) + require_effect_capability(operation) record = await self._runtime().execute_required( EffectCommand( effect_id=effect_id, diff --git a/src/forge/workflow/projections/common.py b/src/forge/workflow/projections/common.py index 6444660df..11b8e5e47 100644 --- a/src/forge/workflow/projections/common.py +++ b/src/forge/workflow/projections/common.py @@ -14,8 +14,10 @@ def project_workflow_identity(state: Mapping[str, Any]) -> WorkflowIdentity: return WorkflowIdentity( run_id=str(state.get("thread_id") or ticket_key), workflow_name=str(state.get("workflow_name") or state.get("ticket_type") or "legacy"), - definition_revision=int(state.get("workflow_revision") or 1), - definition_digest=state.get("workflow_digest"), + definition_revision=int( + state.get("workflow_definition_revision") or state.get("workflow_revision") or 1 + ), + definition_digest=state.get("workflow_definition_digest") or state.get("workflow_digest"), ) diff --git a/src/forge/workflow/registry.py b/src/forge/workflow/registry.py index 47db96cc9..dc4218fcd 100644 --- a/src/forge/workflow/registry.py +++ b/src/forge/workflow/registry.py @@ -1,15 +1,17 @@ """Default workflow registry.""" -from forge.workflow.bug import BugWorkflow -from forge.workflow.feature import FeatureWorkflow +from forge.workflow.declarative.builtins import ( + BugGoldenWorkflow, + FeatureGoldenWorkflow, + TaskTakeoverGoldenWorkflow, +) from forge.workflow.router import WorkflowRouter -from forge.workflow.task_takeover import TaskTakeoverWorkflow def create_default_router() -> WorkflowRouter: """Create router with built-in workflows.""" router = WorkflowRouter() - router.register(TaskTakeoverWorkflow) - router.register(FeatureWorkflow) - router.register(BugWorkflow) + router.register(TaskTakeoverGoldenWorkflow) + router.register(FeatureGoldenWorkflow) + router.register(BugGoldenWorkflow) return router diff --git a/src/forge/workflow/stations/runner.py b/src/forge/workflow/stations/runner.py index 5e4490e77..b088fe472 100644 --- a/src/forge/workflow/stations/runner.py +++ b/src/forge/workflow/stations/runner.py @@ -13,6 +13,7 @@ from forge.domain import DomainModel, StationOutcome, StationRequest from forge.effects import EffectRecord, EffectService +from forge.workflow.declarative.capabilities import require_effect_capability from forge.workflow.stations.agent_operation import ( AgentOperationInput, run_agent_operation_station, @@ -157,6 +158,7 @@ async def invoke_station( for effect in outcome.requested_effects: if effect.workflow != request.workflow: raise ValueError("Station effect does not belong to its workflow") + require_effect_capability(effect.operation) assert effect_service is not None record = await effect_service.execute_required(effect) if effect_records is not None: @@ -202,7 +204,7 @@ async def run_serialized_async( ) -> str: """Run a station from serialized input without the Forge control plane.""" definition = (registry or create_builtin_station_registry()).resolve(station_name) - request_type = StationRequest[definition.input_type] # type: ignore[valid-type] + request_type = StationRequest[definition.input_type] # type: ignore[name-defined] request = request_type.model_validate_json(request_json) outcome = await invoke_station(definition, request, effect_service=effect_service) return outcome.model_dump_json() diff --git a/tests/e2e/test_feature_workflow_e2e.py b/tests/e2e/test_feature_workflow_e2e.py index c68304aa9..7d8d1bdfd 100644 --- a/tests/e2e/test_feature_workflow_e2e.py +++ b/tests/e2e/test_feature_workflow_e2e.py @@ -1,9 +1,10 @@ """End-to-end smoke tests for the current pluggable workflow architecture.""" +import asyncio from unittest.mock import patch from forge.models.workflow import TicketType -from forge.workflow.feature import FeatureWorkflow +from forge.workflow.declarative.builtins import FeatureGoldenWorkflow from forge.workflow.registry import create_default_router @@ -22,12 +23,12 @@ def test_feature_workflow_routes_generates_and_pauses() -> None: router = create_default_router() workflow = router.resolve(TicketType.FEATURE, ["forge:managed"], {}) - assert isinstance(workflow, FeatureWorkflow) + assert isinstance(workflow, FeatureGoldenWorkflow) - with patch("forge.workflow.feature.graph.generate_prd", _generate_prd): + with patch("forge.workflow.nodes.generate_prd", _generate_prd): graph = workflow.build_graph().compile() state = workflow.create_initial_state("TEST-123") - result = graph.invoke(state) + result = asyncio.run(graph.ainvoke(state)) assert result["prd_content"].startswith("# PRD") assert result["current_node"] == "prd_approval_gate" diff --git a/tests/integration/workflow/test_definition_publication.py b/tests/integration/workflow/test_definition_publication.py new file mode 100644 index 000000000..c5c5d5c90 --- /dev/null +++ b/tests/integration/workflow/test_definition_publication.py @@ -0,0 +1,49 @@ +"""Redis contract for immutable process-definition governance.""" + +import pytest + +from forge.workflow.declarative.builtins import builtin_feature_definition +from forge.workflow.declarative.publication import DefinitionPublisher + + +@pytest.mark.asyncio +async def test_redis_publication_activation_and_cas(redis_client) -> None: + publisher = DefinitionPublisher("PROJ", redis_client=redis_client) + first = builtin_feature_definition() + second = first.model_copy( + update={ + "metadata": first.metadata.model_copy( + update={"revision": 2, "description": "compatible description update"} + ) + } + ) + + await publisher.publish(first, actor="platform", reason="initial publication") + await publisher.activate("feature", 1, actor="platform", reason="initial rollout") + await publisher.publish(second, actor="platform", reason="approved revision") + + with pytest.raises(ValueError, match="concurrently"): + await publisher.activate( + "feature", + 2, + actor="platform", + reason="stale rollout", + expected_active_digest="stale", + ) + + await publisher.activate( + "feature", + 2, + actor="platform", + reason="approved rollout", + expected_active_digest=first.digest, + ) + + assert (await publisher.active("feature")).digest == second.digest + assert [item.action for item in await publisher.decisions("feature")] == [ + "publish", + "activate", + "publish", + "activate", + ] + assert [item.metadata.revision for item in await publisher.history("feature")] == [1, 2] diff --git a/tests/unit/workflow/test_builtin_definition_artifacts.py b/tests/unit/workflow/test_builtin_definition_artifacts.py new file mode 100644 index 000000000..b39c91ec4 --- /dev/null +++ b/tests/unit/workflow/test_builtin_definition_artifacts.py @@ -0,0 +1,55 @@ +"""Tests for checked-in built-in workflow definition artifacts.""" + +from __future__ import annotations + +import json +from importlib import resources + +import pytest + +from forge.workflow.declarative.builtins import ( + builtin_bug_definition, + builtin_feature_definition, + builtin_task_takeover_definition, +) +from forge.workflow.declarative.compiler import DeclarativeWorkflowCompiler +from forge.workflow.declarative.loader import load_workflow_value + +_DEFINITIONS = { + "feature": builtin_feature_definition, + "bug": builtin_bug_definition, + "task_takeover": builtin_task_takeover_definition, +} + +# A changed digest is an intentional process revision and must update the +# checked-in artifact and this snapshot together. +_DIGESTS = { + "feature": "7bdc1d113890d69f49f5dce4e93c3fec7f293de8042cccf4ea9b8ca18a87b385", + "bug": "bf688cf548423e0577ceb93b7fc37f7771e5cf52c2a15cff892d54f0cda260d9", + "task_takeover": "eccdcaea925fdd61f938e1101b31f4b3be9e8d0e06394272fb6bba7f414885a6", +} + + +@pytest.mark.parametrize("name", tuple(_DEFINITIONS)) +def test_artifact_round_trip_preserves_canonical_definition(name: str) -> None: + resource = resources.files("forge.workflow.declarative.definitions").joinpath(f"{name}.json") + artifact = json.loads(resource.read_text(encoding="utf-8")) + + definition = load_workflow_value(artifact) + + assert definition.canonical_dict() == artifact + assert _DEFINITIONS[name]().canonical_dict() == artifact + + +@pytest.mark.parametrize("name", tuple(_DEFINITIONS)) +def test_builtin_digest_snapshot(name: str) -> None: + assert _DEFINITIONS[name]().digest == _DIGESTS[name] + + +@pytest.mark.parametrize("name", tuple(_DEFINITIONS)) +def test_default_compiler_consumes_checked_in_artifact(name: str) -> None: + definition = _DEFINITIONS[name]() + + compiler = DeclarativeWorkflowCompiler(definition) + compiler.validate() + assert compiler.build_graph() is not None diff --git a/tests/unit/workflow/test_declarative_workflows.py b/tests/unit/workflow/test_declarative_workflows.py index 9d49ecdac..5e49c939c 100644 --- a/tests/unit/workflow/test_declarative_workflows.py +++ b/tests/unit/workflow/test_declarative_workflows.py @@ -4,16 +4,25 @@ from unittest.mock import AsyncMock, MagicMock, patch import pytest +import yaml from pydantic import ValidationError from forge.orchestrator.worker import OrchestratorWorker +from forge.workflow.declarative.builtins import builtin_definitions, builtin_feature_definition from forge.workflow.declarative.cli import cmd_workflow from forge.workflow.declarative.compiler import ( DeclarativeWorkflowCompiler, WorkflowValidationError, ) from forge.workflow.declarative.loader import load_workflow_value +from forge.workflow.declarative.manifest import ( + ProcessNodeKind, + build_process_manifest, + compare_process_definitions, + render_mermaid, +) from forge.workflow.declarative.models import WORKFLOW_PROPERTY_PREFIX +from forge.workflow.declarative.publication import InMemoryDefinitionPublisher from forge.workflow.declarative.resolver import ( load_project_workflow, selected_workflow_name, @@ -25,6 +34,7 @@ PreconditionAction, Requirement, ) +from forge.workflow.registry import create_default_router def definition_value( @@ -50,6 +60,65 @@ def test_loads_strict_definition_and_computes_stable_digest() -> None: assert first.property_key == f"{WORKFLOW_PROPERTY_PREFIX}short-feature" +def test_builtin_feature_golden_path_is_valid_and_inspectable() -> None: + definition = builtin_feature_definition() + DeclarativeWorkflowCompiler(definition).validate() + manifest = build_process_manifest(definition) + + assert definition.metadata.name == "feature" + assert len(manifest.nodes) == 32 + assert any(node.name == "task_router" and node.station_contract for node in manifest.nodes) + assert any(node.name == "prd_approval_gate" and node.kind == "gate" for node in manifest.nodes) + + +def test_every_supported_golden_path_uses_the_versioned_definition_compiler() -> None: + definitions = builtin_definitions() + + assert {item.metadata.name for item in definitions} == {"feature", "bug", "task_takeover"} + for definition in definitions: + DeclarativeWorkflowCompiler(definition).validate() + graph = DeclarativeWorkflowCompiler(definition).build_graph() + assert graph is not None + assert definition.spec.mandatory_policies == ("forge-contracts-v1",) + assert all( + "forge-contracts-v1" in step.required_policies + for step in definition.spec.steps.values() + ) + + +def test_default_router_has_no_python_topology_workflow_runtime() -> None: + router = create_default_router() + + assert router._workflows # noqa: SLF001 - architecture assertion + assert all(issubclass(item, DeclarativeWorkflow) for item in router._workflows) # noqa: SLF001 + + +@pytest.mark.asyncio +async def test_publication_is_immutable_and_activation_is_explicit() -> None: + publisher = InMemoryDefinitionPublisher() + first = builtin_feature_definition() + + published = await publisher.publish(first, actor="platform", reason="initial publication") + assert published.activated is False + assert await publisher.active(first.metadata.name) is None + + activated = await publisher.activate( + first.metadata.name, + first.metadata.revision, + actor="platform", + reason="initial rollout", + ) + assert activated.activated is True + assert (await publisher.active(first.metadata.name)).digest == first.digest + + changed = first.canonical_dict() + changed["metadata"]["description"] = "changed without a revision" + with pytest.raises(ValueError, match="immutable"): + await publisher.publish( + load_workflow_value(changed), actor="platform", reason="invalid mutation" + ) + + def test_rejects_unknown_fields() -> None: value = definition_value() value["spec"]["execute"] = "os.system" @@ -76,6 +145,69 @@ def test_compiles_allowlisted_node() -> None: assert "_forge_entry" in graph.nodes +def test_process_manifest_exposes_stations_gates_and_transitions() -> None: + value = definition_value( + steps={ + "task_router": {"next": "prd_approval_gate"}, + "prd_approval_gate": { + "route": "route_prd_approval", + "branches": {"revise": "task_router", "approved": "__end__"}, + }, + } + ) + value["spec"]["entry"] = "task_router" + + manifest = build_process_manifest(load_workflow_value(value)) + + nodes = {node.name: node for node in manifest.nodes} + assert nodes["task_router"].kind is ProcessNodeKind.STATION + assert nodes["task_router"].station_contract == "task-routing" + assert nodes["prd_approval_gate"].kind is ProcessNodeKind.GATE + assert any( + edge.source == "prd_approval_gate" + and edge.outcome == "approved" + and edge.target == "__end__" + for edge in manifest.transitions + ) + assert manifest.digest == load_workflow_value(value).digest + + +def test_mermaid_uses_same_manifest_and_labels_routes() -> None: + manifest = build_process_manifest(load_workflow_value(definition_value())) + + rendered = render_mermaid(manifest) + + assert rendered.startswith("flowchart TD") + assert "__start__([start]) --> generate_prd" in rendered + assert "generate_prd --> __end__" in rendered + + +def test_revision_diff_reports_missing_resume_mapping() -> None: + previous = load_workflow_value(definition_value(revision=1)) + current_value = definition_value(revision=2, steps={"generate_spec": {"next": "__end__"}}) + current_value["spec"]["entry"] = "generate_spec" + current = load_workflow_value(current_value) + + impact = compare_process_definitions(previous, current) + + assert impact.removed_nodes == ("generate_prd",) + assert impact.added_nodes == ("generate_spec",) + assert impact.missing_resume_mappings == ("generate_prd",) + assert impact.compatible_for_in_flight is False + + +def test_revision_diff_accepts_explicit_resume_mapping() -> None: + previous = load_workflow_value(definition_value(revision=1)) + current_value = definition_value(revision=2, steps={"generate_spec": {"next": "__end__"}}) + current_value["spec"]["entry"] = "generate_spec" + current_value["spec"]["resume"] = {"fromRevisions": {"1": {"generate_prd": "generate_spec"}}} + + impact = compare_process_definitions(previous, load_workflow_value(current_value)) + + assert impact.missing_resume_mappings == () + assert impact.compatible_for_in_flight is True + + @pytest.mark.asyncio async def test_runtime_transition_budget_blocks_before_side_effect() -> None: value = definition_value( @@ -253,7 +385,15 @@ async def test_worker_resolves_label_selected_workflow() -> None: jira.get_project_property = AsyncMock(return_value=definition_value()) jira.close = AsyncMock() - with patch("forge.orchestrator.worker.JiraClient", return_value=jira): + publisher = AsyncMock() + publisher.active.return_value = None + with ( + patch("forge.orchestrator.worker.JiraClient", return_value=jira), + patch( + "forge.workflow.declarative.publication.DefinitionPublisher", + return_value=publisher, + ), + ): workflow = await worker._resolve_custom_workflow( "PROJ-1", ["forge:managed", "forge:workflow:short-feature"] ) @@ -279,7 +419,15 @@ async def test_worker_keeps_checkpoint_workflow_identity_when_label_is_removed() jira.get_project_property = AsyncMock(return_value=definition_value()) jira.close = AsyncMock() - with patch("forge.orchestrator.worker.JiraClient", return_value=jira): + publisher = AsyncMock() + publisher.active.return_value = None + with ( + patch("forge.orchestrator.worker.JiraClient", return_value=jira), + patch( + "forge.workflow.declarative.publication.DefinitionPublisher", + return_value=publisher, + ), + ): workflow = await worker._resolve_custom_workflow("PROJ-1", []) assert workflow is not None @@ -302,6 +450,33 @@ def test_worker_cache_key_separates_custom_revisions() -> None: @pytest.mark.asyncio async def test_cli_publish_validates_and_stores_canonical_json(tmp_path) -> None: + source = tmp_path / "workflow.yaml" + source.write_text( + yaml.safe_dump(builtin_feature_definition().canonical_dict(), sort_keys=False), + encoding="utf-8", + ) + publisher = InMemoryDefinitionPublisher("PROJ") + + with patch("forge.workflow.declarative.cli.DefinitionPublisher", return_value=publisher): + result = await cmd_workflow( + Namespace( + workflow_command="publish", + project_key="proj", + file=str(source), + actor="tester", + reason="contract test", + ) + ) + + assert result == 0 + history = await publisher.history("feature") + assert len(history) == 1 + assert history[0].canonical_dict()["apiVersion"] == "forge/v1" + assert history[0].metadata.revision == 1 + + +@pytest.mark.asyncio +async def test_cli_render_does_not_require_jira(tmp_path, capsys) -> None: source = tmp_path / "workflow.yaml" source.write_text( """apiVersion: forge/v1 @@ -318,19 +493,45 @@ async def test_cli_publish_validates_and_stores_canonical_json(tmp_path) -> None """, encoding="utf-8", ) - jira = MagicMock() - jira.get_project_property = AsyncMock(return_value=None) - jira.set_project_property = AsyncMock() - jira.close = AsyncMock() - with patch("forge.workflow.declarative.cli.JiraClient", return_value=jira): - result = await cmd_workflow( - Namespace(workflow_command="publish", project_key="proj", file=str(source)) - ) + result = await cmd_workflow( + Namespace(workflow_command="render", file=str(source), format="mermaid") + ) assert result == 0 - key, value = jira.set_project_property.await_args.args[1:] - assert key == "forge.workflow.short-feature" - assert value["apiVersion"] == "forge/v1" - assert value["metadata"]["revision"] == 1 - jira.close.assert_awaited_once() + assert "flowchart TD" in capsys.readouterr().out + + +@pytest.mark.asyncio +async def test_cli_diff_returns_nonzero_for_unsafe_in_flight_change(tmp_path, capsys) -> None: + previous = tmp_path / "previous.yaml" + current = tmp_path / "current.yaml" + previous.write_text( + """apiVersion: forge/v1 +kind: Workflow +metadata: {name: short-feature, revision: 1} +spec: + state: feature + entry: generate_prd + steps: {generate_prd: {next: __end__}} +""", + encoding="utf-8", + ) + current.write_text( + """apiVersion: forge/v1 +kind: Workflow +metadata: {name: short-feature, revision: 2} +spec: + state: feature + entry: generate_spec + steps: {generate_spec: {next: __end__}} +""", + encoding="utf-8", + ) + + result = await cmd_workflow( + Namespace(workflow_command="diff", previous=str(previous), current=str(current)) + ) + + assert result == 2 + assert '"missing_resume_mappings"' in capsys.readouterr().out diff --git a/tests/unit/workflow/test_definition_pinning.py b/tests/unit/workflow/test_definition_pinning.py new file mode 100644 index 000000000..e52e1efbd --- /dev/null +++ b/tests/unit/workflow/test_definition_pinning.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from forge.orchestrator.worker import OrchestratorWorker +from forge.workflow.declarative.builtins import builtin_feature_definition +from forge.workflow.declarative.loader import load_workflow_value +from forge.workflow.declarative.models import WorkflowMetadata +from forge.workflow.declarative.resolver import load_project_workflow +from forge.workflow.declarative.workflow import DeclarativeWorkflow + + +def definition(revision: int = 1, description: str = "") -> dict: + source = builtin_feature_definition() + metadata = WorkflowMetadata(name="pinned", revision=revision, description=description) + return source.model_copy(update={"metadata": metadata}).canonical_dict() + + +def test_new_state_contains_complete_immutable_identity() -> None: + workflow = DeclarativeWorkflow(load_workflow_value(definition()), "PROJ") + + state = workflow.create_initial_state("PROJ-1") + + assert state["workflow_name"] == "pinned" + assert state["workflow_revision"] == 1 + assert state["workflow_definition_revision"] == 1 + assert state["workflow_digest"] == state["workflow_definition_digest"] + assert state["workflow_definition"] == workflow.definition.canonical_dict() + assert state["workflow_pin_status"] == "pinned" + + +@pytest.mark.asyncio +async def test_pinned_resolution_uses_checkpoint_artifact_without_jira_property() -> None: + pinned = load_workflow_value(definition()) + jira = MagicMock() + jira.get_project_property = AsyncMock( + return_value=definition(revision=2, description="new active definition") + ) + + workflow = await load_project_workflow( + jira, + "PROJ", + "pinned", + pinned_revision=pinned.metadata.revision, + pinned_digest=pinned.digest, + pinned_definition=pinned.canonical_dict(), + ) + + assert workflow.definition.digest == pinned.digest + jira.get_project_property.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_worker_resolves_pinned_checkpoint_without_loading_active_property() -> None: + pinned = load_workflow_value(definition()) + worker = OrchestratorWorker.__new__(OrchestratorWorker) + worker._checkpointer = MagicMock() + worker._checkpointer.aget = AsyncMock( + return_value={"channel_values": {**DeclarativeWorkflow(pinned, "PROJ").workflow_metadata()}} + ) + + with patch("forge.orchestrator.worker.JiraClient") as jira_client: + workflow = await worker._resolve_custom_workflow("PROJ-1", []) + + assert workflow is not None + assert workflow.definition.digest == pinned.digest + jira_client.assert_not_called() + + +def test_pinned_state_rejects_digest_mismatch() -> None: + workflow = DeclarativeWorkflow(load_workflow_value(definition()), "PROJ") + state = {**workflow.workflow_metadata(), "workflow_definition_digest": "sha256:wrong"} + + with pytest.raises(Exception, match="conflicting workflow identities"): + workflow.validate_pinned_state(state) + + +def test_legacy_state_requires_explicit_compatibility_default() -> None: + workflow = DeclarativeWorkflow(load_workflow_value(definition()), "PROJ") + legacy = {"workflow_name": "pinned", "current_node": "generate_prd"} + + assert workflow.pin_status(legacy) == "legacy_unpinned" + pinned = workflow.pin_legacy_state(legacy) + assert pinned["workflow_pin_status"] == "legacy_active_default" + assert pinned["workflow_digest"] == workflow.definition.digest diff --git a/tests/unit/workflow/test_governed_publication.py b/tests/unit/workflow/test_governed_publication.py new file mode 100644 index 000000000..e479ae954 --- /dev/null +++ b/tests/unit/workflow/test_governed_publication.py @@ -0,0 +1,101 @@ +"""Contract tests for immutable, project-scoped workflow governance.""" + +from __future__ import annotations + +import pytest + +from forge.workflow.declarative.builtins import builtin_feature_definition +from forge.workflow.declarative.loader import load_workflow_value +from forge.workflow.declarative.publication import InMemoryDefinitionPublisher + + +def definition(revision: int, description: str = ""): + value = builtin_feature_definition() + return value.model_copy( + update={ + "metadata": value.metadata.model_copy( + update={"revision": revision, "description": description} + ) + } + ) + + +@pytest.mark.asyncio +async def test_publish_is_immutable_and_does_not_activate() -> None: + publisher = InMemoryDefinitionPublisher("proj") + first = definition(1) + decision = await publisher.publish(first, actor="alice", reason="initial approval") + + assert decision.action == "publish" + assert decision.activated is False + assert await publisher.active("feature") is None + + with pytest.raises(ValueError, match="immutable"): + await publisher.publish(definition(1, "changed"), actor="alice", reason="mistake") + + +@pytest.mark.asyncio +async def test_changed_content_must_use_strictly_increasing_revision() -> None: + publisher = InMemoryDefinitionPublisher("PROJ") + await publisher.publish(definition(2), actor="alice", reason="approved") + + with pytest.raises(ValueError, match="increment metadata.revision"): + await publisher.publish(definition(1, "changed"), actor="alice", reason="downgrade") + + +@pytest.mark.asyncio +async def test_activation_cas_and_rollback_are_audited_without_deleting_history() -> None: + publisher = InMemoryDefinitionPublisher("proj") + one = definition(1) + two = definition(2, "safe change") + await publisher.publish(one, actor="alice", reason="initial") + await publisher.publish(two, actor="alice", reason="change") + activated = await publisher.activate("feature", 2, actor="bob", reason="release") + + with pytest.raises(ValueError, match="concurrently"): + await publisher.activate("feature", 1, actor="bob", reason="stale", expected_active_digest="wrong") + + rollback = await publisher.rollback( + "feature", 1, actor="carol", reason="release recovery", expected_active_digest=activated.digest + ) + assert rollback.action == "rollback" + assert (await publisher.active("feature")).metadata.revision == 1 + assert [item.action for item in await publisher.decisions("feature")] == [ + "publish", + "publish", + "activate", + "rollback", + ] + assert len(await publisher.history("feature")) == 2 + + +@pytest.mark.asyncio +async def test_actor_and_reason_are_required() -> None: + publisher = InMemoryDefinitionPublisher("proj") + with pytest.raises(ValueError, match="actor"): + await publisher.publish(definition(1), actor="", reason="why") + with pytest.raises(ValueError, match="reason"): + await publisher.publish(definition(1), actor="alice", reason="") + + +@pytest.mark.asyncio +async def test_publication_rejects_ungoverned_definition() -> None: + ungoverned = load_workflow_value( + { + "apiVersion": "forge/v1", + "kind": "Workflow", + "metadata": {"name": "unsafe", "revision": 1}, + "spec": { + "state": "feature", + "entry": "generate_prd", + "steps": {"generate_prd": {"next": "__end__"}}, + }, + } + ) + + with pytest.raises(ValueError, match="mandatory policy"): + await InMemoryDefinitionPublisher("proj").publish( + ungoverned, + actor="alice", + reason="should fail", + ) diff --git a/tests/unit/workflow/test_process_change_classification.py b/tests/unit/workflow/test_process_change_classification.py new file mode 100644 index 000000000..e888342d7 --- /dev/null +++ b/tests/unit/workflow/test_process_change_classification.py @@ -0,0 +1,176 @@ +"""Focused tests for declarative process-definition change impact.""" + +from forge.workflow.declarative.builtins import builtin_feature_definition +from forge.workflow.declarative.loader import load_workflow_value +from forge.workflow.declarative.manifest import ( + ProcessChangeClassification, + build_process_manifest, + compare_process_definitions, + render_mermaid, +) + + +def definition( + *, + revision: int, + steps: dict, + state: str = "feature", + entry: str | None = None, + mandatory_policies: list[str] | None = None, +): + return load_workflow_value( + { + "apiVersion": "forge/v1", + "kind": "Workflow", + "metadata": {"name": "classification-test", "revision": revision}, + "spec": { + "state": state, + "entry": entry or next(iter(steps)), + "steps": steps, + **({"mandatoryPolicies": mandatory_policies} if mandatory_policies is not None else {}), + }, + } + ) + + +def test_patch_ignores_semantic_definition_order() -> None: + old = definition( + revision=1, + steps={ + "first": {"route": "router", "branches": {"b": "last", "a": "last"}}, + "last": {"next": "__end__"}, + }, + entry="first", + ) + new = definition( + revision=2, + steps={ + "last": {"next": "__end__"}, + "first": {"route": "router", "branches": {"a": "last", "b": "last"}}, + }, + entry="first", + ) + + impact = compare_process_definitions(old, new) + + assert impact.classification is ProcessChangeClassification.PATCH + assert impact.changed_nodes == () + assert impact.compatible_for_in_flight is True + + +def test_manifest_and_rendering_are_deterministically_ordered() -> None: + raw = builtin_feature_definition().canonical_dict() + raw["spec"]["steps"] = dict(reversed(list(raw["spec"]["steps"].items()))) + reordered = load_workflow_value(raw) + first = build_process_manifest(builtin_feature_definition()) + second = build_process_manifest(reordered) + + assert [node.name for node in second.nodes] == sorted(node.name for node in second.nodes) + assert [(edge.source, edge.target, edge.outcome or "") for edge in second.transitions] == sorted( + (edge.source, edge.target, edge.outcome or "") for edge in second.transitions + ) + assert first.nodes == second.nodes + assert first.transitions == second.transitions + assert render_mermaid(first) == render_mermaid(second) + + +def test_removed_nodes_need_mapping_and_mapped_removal_is_migratable() -> None: + old = definition(revision=1, steps={"old": {"next": "kept"}, "kept": {"next": "__end__"}}) + unmapped = definition(revision=2, steps={"kept": {"next": "__end__"}}, entry="kept") + mapped = load_workflow_value( + { + **unmapped.canonical_dict(), + "spec": { + **unmapped.canonical_dict()["spec"], + "resume": {"fromRevisions": {"1": {"old": "kept"}}}, + }, + } + ) + + blocked = compare_process_definitions(old, unmapped) + migrated = compare_process_definitions(old, mapped) + + assert blocked.classification is ProcessChangeClassification.BREAKING + assert blocked.compatible_for_in_flight is False + assert migrated.classification is ProcessChangeClassification.MIGRATABLE + assert migrated.compatible_for_in_flight is True + + +def test_routing_and_outcome_changes_are_explicit_and_not_silently_compatible() -> None: + old = definition( + revision=1, + steps={ + "route": {"route": "router", "branches": {"ok": "done"}}, + "done": {"next": "__end__"}, + "other": {"next": "__end__"}, + }, + ) + new = definition( + revision=2, + steps={ + "route": {"route": "router", "branches": {"ok": "other"}}, + "done": {"next": "__end__"}, + "other": {"next": "__end__"}, + }, + ) + + impact = compare_process_definitions(old, new) + + assert impact.routing_changes == ("route",) + assert impact.outcome_changes == () + assert impact.classification is ProcessChangeClassification.MIGRATABLE + assert impact.compatible_for_in_flight is False + + +def test_contract_effect_policy_and_execution_changes_are_breaking() -> None: + old = definition( + revision=1, + steps={ + "work": { + "next": "done", + "stationContract": "x", + "stationContractVersion": "1", + "allowedEffects": ["jira.*"], + "requiredPolicies": ["p"], + "retryBound": 2, + }, + "done": {"next": "__end__"}, + }, + ) + new = definition( + revision=2, + steps={ + "work": { + "next": "done", + "stationContract": "x", + "stationContractVersion": "2", + "allowedEffects": ["source_control.*"], + "requiredPolicies": ["q"], + "retryBound": 3, + }, + "done": {"next": "__end__"}, + }, + ) + + impact = compare_process_definitions(old, new) + + assert impact.classification is ProcessChangeClassification.BREAKING + assert impact.compatible_for_in_flight is False + assert impact.station_contract_changes == ("work",) + assert impact.effect_capability_changes == ("work",) + assert impact.policy_changes == ("work",) + assert impact.retry_changes == ("work",) + + +def test_state_profile_and_same_revision_mutation_are_breaking() -> None: + old = definition(revision=1, steps={"work": {"next": "__end__"}}) + profile = definition(revision=2, steps={"work": {"next": "__end__"}}, state="bug") + mutated = definition(revision=1, steps={"work": {"next": "__end__"}, "new": {"next": "__end__"}}) + + profile_impact = compare_process_definitions(old, profile) + mutation_impact = compare_process_definitions(old, mutated) + + assert profile_impact.state_profile_changed is True + assert profile_impact.classification is ProcessChangeClassification.BREAKING + assert mutation_impact.same_revision_mutation is True + assert mutation_impact.classification is ProcessChangeClassification.BREAKING diff --git a/tests/unit/workflow/test_process_governance_validation.py b/tests/unit/workflow/test_process_governance_validation.py new file mode 100644 index 000000000..aa318e023 --- /dev/null +++ b/tests/unit/workflow/test_process_governance_validation.py @@ -0,0 +1,179 @@ +from unittest.mock import AsyncMock + +import pytest +from langgraph.types import Send +from pydantic import ValidationError + +from forge.workflow.declarative.builtins import ( + builtin_bug_definition, + builtin_definitions, + builtin_feature_definition, + builtin_task_takeover_definition, +) +from forge.workflow.declarative.capabilities import require_effect_capability +from forge.workflow.declarative.catalog import get_state_profile +from forge.workflow.declarative.compiler import ( + DeclarativeWorkflowCompiler, + WorkflowValidationError, +) +from forge.workflow.declarative.models import WorkflowDefinition + + +def _replace(definition: WorkflowDefinition, **spec_updates) -> WorkflowDefinition: + value = definition.canonical_dict() + value["metadata"] = {**value["metadata"], "revision": value["metadata"]["revision"] + 1} + value["spec"] = {**value["spec"], **spec_updates} + return WorkflowDefinition.model_validate(value) + + +def test_every_builtin_station_step_declares_the_registered_contract() -> None: + for definition in builtin_definitions(): + profile = get_state_profile(definition.spec.state) + for node_name, binding in profile.station_bindings.items(): + if node_name not in definition.spec.steps: + continue + step = definition.spec.steps[node_name] + assert (step.station_contract, step.station_contract_version) == binding + + +@pytest.mark.parametrize( + ("factory", "gate"), + [ + (builtin_feature_definition, "spec_approval_gate"), + (builtin_bug_definition, "rca_option_gate"), + (builtin_task_takeover_definition, "task_plan_approval_gate"), + ], +) +def test_governed_definitions_cannot_remove_mandatory_gates(factory, gate: str) -> None: + definition = factory() + steps = definition.canonical_dict()["spec"]["steps"] + del steps[gate] + candidate = _replace(definition, steps=steps) + + with pytest.raises(WorkflowValidationError, match=f"mandatory gate '{gate}'"): + DeclarativeWorkflowCompiler(candidate).validate() + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("mandatoryPolicies", ["unknown-policy"], "unknown mandatory policy"), + ("extensionPoints", ["arbitrary-python"], "unsupported extension point"), + ], +) +def test_unknown_governance_capabilities_are_rejected(field, value, message) -> None: + definition = builtin_feature_definition() + candidate = _replace(definition, **{field: value}) + + with pytest.raises(WorkflowValidationError, match=message): + DeclarativeWorkflowCompiler(candidate).validate() + + +def test_unknown_effect_capability_is_rejected() -> None: + definition = builtin_feature_definition() + steps = definition.canonical_dict()["spec"]["steps"] + steps["generate_prd"]["allowedEffects"] = ["shell.execute"] + candidate = _replace(definition, steps=steps) + + with pytest.raises(WorkflowValidationError, match="unknown effect capability"): + DeclarativeWorkflowCompiler(candidate).validate() + + +def test_registered_station_contract_cannot_be_changed() -> None: + definition = builtin_feature_definition() + steps = definition.canonical_dict()["spec"]["steps"] + steps["generate_prd"]["stationContract"] = "sandbox-execution" + candidate = _replace(definition, steps=steps) + + with pytest.raises(WorkflowValidationError, match="must be"): + DeclarativeWorkflowCompiler(candidate).validate() + + +def test_join_requires_multiple_incoming_transitions() -> None: + definition = builtin_feature_definition() + steps = definition.canonical_dict()["spec"]["steps"] + steps["generate_prd"]["join"] = "all" + candidate = _replace(definition, steps=steps) + + with pytest.raises(WorkflowValidationError, match="at least two incoming"): + DeclarativeWorkflowCompiler(candidate).validate() + + +def test_dynamic_routes_require_an_explicit_concurrency_limit() -> None: + definition = builtin_feature_definition().canonical_dict() + del definition["spec"]["steps"]["task_router"]["maxConcurrency"] + + with pytest.raises(ValidationError, match="explicit maxConcurrency"): + WorkflowDefinition.model_validate(definition) + + +def test_publication_validates_complete_router_outcome_contract() -> None: + definition = builtin_feature_definition() + steps = definition.canonical_dict()["spec"]["steps"] + del steps["prd_approval_gate"]["branches"]["answer_question"] + candidate = _replace(definition, steps=steps) + + with pytest.raises(WorkflowValidationError, match="omits router outcome 'answer_question'"): + DeclarativeWorkflowCompiler(candidate).validate_for_publication() + + +@pytest.mark.asyncio +async def test_retry_bound_blocks_before_reinvoking_station() -> None: + operation = AsyncMock(return_value={"current_node": "work"}) + guarded = DeclarativeWorkflowCompiler._guarded_node( + operation, + "work", + terminal=False, + retry_bound=2, + ) + + state = await guarded({}) + state = await guarded(state) + blocked = await guarded(state) + + assert blocked["is_blocked"] is True + assert "retry bound 2" in blocked["last_error"] + assert operation.await_count == 2 + + +@pytest.mark.asyncio +async def test_compiled_step_enforces_effect_capabilities_at_runtime() -> None: + async def emit_jira_effect(_state): + require_effect_capability("jira.comment.create") + return {} + + allowed = DeclarativeWorkflowCompiler._guarded_node( + emit_jira_effect, + "allowed", + terminal=False, + allowed_effects=("jira.comment",), + ) + denied = DeclarativeWorkflowCompiler._guarded_node( + emit_jira_effect, + "denied", + terminal=False, + allowed_effects=("source_control.review",), + ) + + await allowed({}) + with pytest.raises(PermissionError, match="jira.comment.create"): + await denied({}) + + +@pytest.mark.asyncio +async def test_dynamic_router_enforces_targets_and_concurrency() -> None: + too_many = DeclarativeWorkflowCompiler._guarded_dynamic_router( + lambda _state: [Send("worker", {}), Send("worker", {})], + {"worker"}, + 1, + ) + undeclared = DeclarativeWorkflowCompiler._guarded_dynamic_router( + lambda _state: Send("arbitrary", {}), + {"worker"}, + 1, + ) + + with pytest.raises(WorkflowValidationError, match="maximum is 1"): + await too_many({}) + with pytest.raises(WorkflowValidationError, match="undeclared target"): + await undeclared({}) diff --git a/tests/unit/workflow/test_process_migration_simulation.py b/tests/unit/workflow/test_process_migration_simulation.py new file mode 100644 index 000000000..7011ab6a9 --- /dev/null +++ b/tests/unit/workflow/test_process_migration_simulation.py @@ -0,0 +1,152 @@ +import json +from argparse import Namespace + +import pytest +import yaml + +from forge.workflow.declarative.cli import cmd_workflow +from forge.workflow.declarative.loader import load_workflow_value +from forge.workflow.declarative.manifest import ( + ProcessMigrationClassification, + simulate_process_migration, +) + + +def definition(*, revision: int, steps: dict, state: str = "feature", resume: dict | None = None): + return load_workflow_value( + { + "apiVersion": "forge/v1", + "kind": "Workflow", + "metadata": {"name": "migration-test", "revision": revision}, + "spec": { + "state": state, + "entry": next(iter(steps)), + "steps": steps, + **( + {"resume": {"fromRevisions": {1: resume}}} + if resume is not None + else {} + ), + }, + } + ) + + +def test_simulation_classifies_direct_mapped_and_pinned_instances_deterministically(): + previous = definition(revision=1, steps={"old": {"next": "kept"}, "kept": {"next": "__end__"}}) + current = definition( + revision=2, + steps={"renamed": {"next": "kept"}, "kept": {"next": "__end__"}}, + resume={"old": "renamed"}, + ) + report = simulate_process_migration( + previous, + current, + [ + {"run_id": "run-mapped", "thread_id": "b", "current_node": "old", "workflow_revision": 1, "workflow_digest": previous.digest}, + {"run_id": "run-direct", "thread_id": "a", "current_node": "kept", "workflow_revision": 1, "workflow_digest": previous.digest}, + {"run_id": "run-pinned", "thread_id": "c", "current_node": "kept", "workflow_revision": 2, "workflow_digest": current.digest}, + ], + ) + + assert [item.run_id for item in report.instances] == ["run-direct", "run-mapped", "run-pinned"] + assert report.counts == {"stays_pinned": 1, "can_adopt_directly": 1, "requires_resume_mapping": 1, "blocked": 0} + assert report.instances[1].classification is ProcessMigrationClassification.REQUIRES_RESUME_MAPPING + assert report.instances[1].target_node == "renamed" + assert report.compatible is True + + +def test_simulation_reports_identity_and_state_safety_failures(): + previous = definition(revision=1, steps={"old": {"next": "__end__"}}) + current = definition(revision=2, steps={"new": {"next": "__end__"}}) + report = simulate_process_migration( + previous, + current, + [ + {"run_id": "mutated", "current_node": "old", "workflow_revision": 1, "workflow_digest": "wrong"}, + {"run_id": "profile", "current_node": "old", "workflow_revision": 1, "workflow_digest": previous.digest, "workflow_state_profile": "task_takeover"}, + {"run_id": "removed", "current_node": "old", "workflow_revision": 1, "workflow_digest": previous.digest}, + {"run_id": "future", "current_node": "new", "workflow_revision": 3, "workflow_digest": "future"}, + ], + ) + + assert report.compatible is False + assert report.blocked_count == 4 + assert {item.reason_code for item in report.instances} == { + "wrong_source_digest", + "state_profile_incompatible", + "removed_node_without_mapping", + "wrong_source_revision", + } + + profile = simulate_process_migration( + previous, + definition(revision=2, steps={"new": {"next": "__end__"}}, state="bug"), + [{"run_id": "profile-change", "current_node": "old", "workflow_revision": 1, "workflow_digest": previous.digest}], + ) + assert profile.instances[0].reason_code == "state_profile_incompatible" + + +def test_simulation_detects_invalid_mapping_targets(): + previous = definition(revision=1, steps={"old": {"next": "__end__"}}) + current = definition( + revision=2, + steps={"new": {"next": "__end__"}}, + resume={"old": "missing"}, + ) + report = simulate_process_migration( + previous, + current, + [{"thread_id": "t", "current_node": "old", "workflow_revision": 1, "workflow_digest": previous.digest}], + ) + + assert report.invalid_resume_mappings == ("1:old->missing",) + assert report.instances[0].reason_code == "invalid_mapping_target" + + +def test_simulation_detects_same_revision_digest_mutation(): + previous = definition(revision=1, steps={"old": {"next": "__end__"}}) + current = definition(revision=1, steps={"new": {"next": "__end__"}}) + report = simulate_process_migration( + previous, + current, + [{"run_id": "r", "current_node": "old", "workflow_revision": 1, "workflow_digest": previous.digest}], + ) + + assert report.instances[0].reason_code == "same_revision_digest_mutation" + + +@pytest.mark.asyncio +async def test_cli_simulation_returns_nonzero_for_blocked_instances(tmp_path, capsys): + previous = definition(revision=1, steps={"old": {"next": "__end__"}}) + current = definition(revision=2, steps={"new": {"next": "__end__"}}) + previous_path = tmp_path / "previous.yaml" + current_path = tmp_path / "current.yaml" + instances_path = tmp_path / "instances.json" + previous_path.write_text(yaml.safe_dump(previous.canonical_dict()), encoding="utf-8") + current_path.write_text(yaml.safe_dump(current.canonical_dict()), encoding="utf-8") + instances_path.write_text( + json.dumps( + [ + { + "run_id": "blocked", + "current_node": "old", + "workflow_revision": 1, + "workflow_digest": previous.digest, + } + ] + ), + encoding="utf-8", + ) + + result = await cmd_workflow( + Namespace( + workflow_command="simulate-migration", + previous=str(previous_path), + current=str(current_path), + instances=str(instances_path), + ) + ) + + assert result == 2 + assert '"blocked": 1' in capsys.readouterr().out