feat: advanced agentic capabilities

AGENTS.md

@@ -5,8 +5,10 @@
 Progi is an MCP-native workflow engine. Key terms:
 
 - **workflow**: a reusable template of ordered **steps**; defines a repeatable, reused across many tasks
-- **step**: one unit of work in a workflow; has a **playbook**, an input spec, and an output spec
-- **playbook**: markdown attached to a step; the agent reads it via `start_or_continue_task` and follows it
+- **step**: one unit of work in a workflow; has a **worker playbook** and **QA playbook**
+- **workflow playbook**: markdown on a workflow describing its purpose, input, and output; used for documentation and sub-workflow context
+- **worker playbook**: markdown on a step; the worker agent reads it via `start_or_continue_task` and follows it to produce the step's output
+- **QA playbook**: optional markdown on a step; the QA sub-agent reads it to evaluate the worker's output before the task advances
 - **task**: a single execution of a workflow; progresses through steps one at a time with lifecycle `todo` → `in_progress` → `done`
 
 Two interfaces over one SQLite DB:
@@ -27,7 +29,7 @@ what keeps LLM-driven and human-driven edits behaviorally identical.
 | `progi/db.py` | Schema (SQLAlchemy Core) + **all** queries, mutations, and state-transition logic |
 | `progi/mcp_server.py` | `@mcp.tool` wrappers (work loop + workflow authoring) |
 | `progi/web/app.py` | FastAPI routes → Jinja partials |
-| `progi/prompts/` | Pass 1 / Pass 2 authoring system prompts (served by tools) |
+| `progi/prompts/` | Pass 1 / Pass 2 authoring system prompts + `templates/` blank playbook templates |
 | `progi/seed.py` | "Blog Post" workflow + sample task (idempotent) |
 | `tests/test_db.py` | DB roundtrip, full work loop, authoring |
 
@@ -114,9 +116,13 @@ app.include_router(mypage.router)
 - **No SQL outside `db.py`.** Add a named function; wrap writes in `with engine.begin()`.
 - **Never `print()` / write to stdout** anywhere reachable from the MCP process — stdout is the MCP protocol channel. Log to stderr via `logging_setup`.
 - **Web returns HTML partials** for AJAX (not JSON); the swapped element's `id` must match the trigger's `x-target`.
-- `input_spec` / `output_spec` / `input_data` / `output` are `sa.JSON` columns — pass and receive plain dicts, no manual `json.dumps`.
+- `input_data` / `output` are `sa.JSON` columns — pass and receive plain dicts, no manual `json.dumps`.
 - Keep the web UI localhost-only (unauthenticated DB viewer).
 
+### Keeping `x-data` lean
+
+Never put non-trivial logic (multi-line functions, getters with loops) directly in `x-data` attributes in templates. Instead, add the state and methods to the relevant Alpine component function in `app.js` and reference them from the template. Inline `x-data` is fine for simple boolean flags (`x-data="{ open: false }"`), but anything complex belongs in `app.js`.
+
 ### Choosing between fetch + JS state vs. Alpine AJAX partials
 
 Use **plain `fetch` + Alpine reactive state** (in `app.js`) when:
@@ -165,11 +171,6 @@ uv run python -m pytest              # tests (do NOT use `uv run pytest` — a s
 uv run ruff check progi              # lint
 ```
 
-## Run modes
-
-`progi` (MCP + web), `progi --no-web` (MCP only), `progi-web` (web only).
-Config via env: `PROGI_DB_PATH`, `PROGI_WEB_HOST`, `PROGI_WEB_PORT`, `PROGI_NO_WEB`.
-
 ## Git commit convention
 Commit messages must follow Conventional Commits: `feat:`, `fix:`, `docs:`, `chore:`, `refactor:`, `test:`, `ci:`. Use `feat!:` for breaking changes. Release Please reads these to generate the changelog and determine the version bump.
 

README.md

@@ -76,7 +76,7 @@ Tweak playbooks in Progi Monitoring between runs. Because workflows live in a da
 |---|---|
 | `create_task` | Create a new task under a given workflow (status `todo`); returns a preview of its first step |
 | `list_tasks` | List tasks, optionally filtered by status and/or workflow |
-| `start_or_continue_task` | Main work-loop entry point — starts or resumes a task and returns the current step's playbook, input data, and output spec |
+| `start_or_continue_task` | Main work-loop entry point — starts or resumes a task and returns the current step's worker playbook and input data |
 | `update_progress_notes` | Overwrite a task's progress notes (mid-step save point) |
 | `finish_step` | Mark the current step complete, store its output, and advance to the next step (or mark done) |
 

docs/glossary.md

@@ -10,9 +10,11 @@ Terms used across the progi codebase, README, and documentation.
 
 **task** — A piece of work created from a workflow. Advances through steps one at a time and carries a `status` (`todo` / `in_progress` / `done`). Stored in the `tasks` table.
 
-**step** — A single unit of work within a workflow. Ordered, carries an `input_spec` and `output_spec`, and has exactly one **playbook**. Stored in the `steps` table.
+**step** — A single unit of work within a workflow. Ordered, has exactly one **worker playbook**, and optionally a **QA playbook**. Stored in the `steps` table.
 
-**playbook** — A markdown document attached to a step. The AI **agent** reads and follows it to perform the step, including when to involve the human and what output satisfies the `output_spec`. Authored in **Pass 2**. Stored in the `playbooks` table.
+**worker playbook** — A markdown document attached to a step describing what to do, what inputs are available, and what output is expected. The AI **agent** reads and follows it to perform the step. Stored in the `playbooks` table.
+
+**QA playbook** — Optional markdown on a step read by a QA sub-agent to evaluate the worker's output before the task advances.
 
 **edge** — A directed connection between two steps. Defines execution flow. Can be conditional (evaluated against the step's `output`) or unconditional. Supports branching. Stored in the `step_edges` table.
 
@@ -38,27 +40,19 @@ Terms used across the progi codebase, README, and documentation.
 
 ## Input / output
 
-**input_spec** — A JSON dict on a step describing what data the step needs to begin. Fields: `description`, `source` (`"static"` or `"previous_step_output"`), optional `from_step`. Used at authoring time to tell the agent what inputs are available.
-
-**output_spec** — A JSON dict on a step describing the deliverable that proves the step is done. Fields: `type` (`file` / `url` / `text`), `description`, `constraints`. The agent must produce output matching this spec.
-
-**input_data** — The resolved, concrete data passed to a step instance when it activates. Derived from `input_spec`; may pull from a prior step's `output`. Stored as JSON on `step_instances.input_data`.
+**input_data** — The resolved, concrete data passed to a step instance when it activates. May come from task creation (static) or a prior step's `output`. Stored as JSON on `step_instances.input_data`.
 
 **output** — The actual deliverable submitted by the agent via `finish_step`. Stored as JSON on `step_instances.output`. Used to resolve the next step's `input_data` and to evaluate edge `conditions`.
 
-**source** — Field inside `input_spec`. Either `"static"` (data comes from task creation) or `"previous_step_output"` (data comes from a prior step's `output`).
-
-**from_step** — Field inside `input_spec` when `source` is `"previous_step_output"`. Names the step whose output to pull.
-
 ---
 
 ## Workflow authoring
 
-**Pass 1** — First authoring phase. Converts a plain-language description into a structured **skeleton** (steps, specs, edges). Triggered by `get_process_skeleton_prompt`.
+**Pass 1** — First authoring phase. Converts a plain-language description into a structured **skeleton** (steps, edges). Triggered by `get_process_skeleton_prompt`.
 
 **Pass 2** — Second authoring phase. Authors the **playbook** for each step. Triggered by `get_playbook_authoring_prompt`, which injects full workflow context into the prompt.
 
-**skeleton** — Structured JSON produced in Pass 1: `{ name, description, process: [steps…], edges: [connections…] }`. Each step includes `order`, `name`, `input_spec`, `output_spec`. Reviewed and adjusted before saving.
+**skeleton** — Structured JSON produced in Pass 1: `{ name, description, process: [steps…], edges: [connections…] }`. Each step includes `order` and `name`. Reviewed and adjusted before saving.
 
 **condition** — A rule on an edge evaluated against a step's `output` to decide whether that edge is taken. Operators: `eq`, `neq`, `in`, `not_in`. A `null` condition means unconditional (always taken if no other condition matches).
 

progi/alembic/versions/0007_sub_workflow_support.py

@@ -0,0 +1,47 @@
+"""add sub-workflow support: workflow playbook, steps.sub_workflow_id, step_instances.sub_workflow_step_id
+
+Revision ID: 0007
+Revises: 0006
+Create Date: 2026-06-27
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0007"
+down_revision: Union[str, None] = "0006"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    conn = op.get_bind()
+
+    # workflows.playbook — skip if already exists (can happen from a partial prior run)
+    existing_wf_cols = {row[1] for row in conn.execute(sa.text("PRAGMA table_info(workflows)")).fetchall()}
+    if "playbook" not in existing_wf_cols:
+        with op.batch_alter_table("workflows") as batch_op:
+            batch_op.add_column(sa.Column("playbook", sa.Text(), nullable=True))
+
+    existing_step_cols = {row[1] for row in conn.execute(sa.text("PRAGMA table_info(steps)")).fetchall()}
+    if "sub_workflow_id" not in existing_step_cols:
+        with op.batch_alter_table("steps") as batch_op:
+            batch_op.add_column(sa.Column("sub_workflow_id", sa.Integer(), nullable=True))
+
+    existing_si_cols = {row[1] for row in conn.execute(sa.text("PRAGMA table_info(step_instances)")).fetchall()}
+    if "sub_workflow_step_id" not in existing_si_cols:
+        with op.batch_alter_table("step_instances") as batch_op:
+            batch_op.add_column(sa.Column("sub_workflow_step_id", sa.Integer(), nullable=True))
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("step_instances") as batch_op:
+        batch_op.drop_column("sub_workflow_step_id")
+
+    with op.batch_alter_table("steps") as batch_op:
+        batch_op.drop_column("sub_workflow_id")
+
+    with op.batch_alter_table("workflows") as batch_op:
+        batch_op.drop_column("playbook")

progi/alembic/versions/0008_qa_playbooks.py

@@ -0,0 +1,59 @@
+"""add QA playbook support: playbooks.type, steps.max_qa_retries, step_instances qa columns
+
+Revision ID: 0008
+Revises: 4643697c3984
+Create Date: 2026-06-27
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0008"
+down_revision: Union[str, None] = "0007"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # 1. Drop the old unique constraint on playbooks.step_id (was unique=True inline).
+    #    SQLite doesn't support DROP CONSTRAINT, so we recreate the table.
+    with op.batch_alter_table("playbooks") as batch_op:
+        batch_op.add_column(
+            sa.Column("type", sa.String(16), nullable=False, server_default="worker")
+        )
+        # Replace the per-step unique index with a (step_id, type) unique constraint.
+        # The old implicit unique index on step_id is dropped as part of the batch rebuild.
+        batch_op.create_unique_constraint("uq_playbooks_step_type", ["step_id", "type"])
+
+    # 2. steps: add max_qa_retries (nullable, NULL = use global default).
+    with op.batch_alter_table("steps") as batch_op:
+        batch_op.add_column(sa.Column("max_qa_retries", sa.Integer, nullable=True))
+
+    # 3. step_instances: add QA tracking columns.
+    with op.batch_alter_table("step_instances") as batch_op:
+        batch_op.add_column(sa.Column("qa_status", sa.String(16), nullable=True))
+        batch_op.add_column(sa.Column("qa_output", sa.JSON, nullable=True))
+        batch_op.add_column(
+            sa.Column(
+                "qa_retry_count",
+                sa.Integer,
+                nullable=False,
+                server_default="0",
+            )
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("step_instances") as batch_op:
+        batch_op.drop_column("qa_retry_count")
+        batch_op.drop_column("qa_output")
+        batch_op.drop_column("qa_status")
+
+    with op.batch_alter_table("steps") as batch_op:
+        batch_op.drop_column("max_qa_retries")
+
+    with op.batch_alter_table("playbooks") as batch_op:
+        batch_op.drop_constraint("uq_playbooks_step_type", type_="unique")
+        batch_op.drop_column("type")

progi/alembic/versions/0009_fix_playbooks_unique.py

@@ -0,0 +1,35 @@
+"""fix playbooks unique constraint: drop old step_id-only index, keep (step_id, type)
+
+The 0008 migration added the type column and (step_id, type) unique constraint
+but left the original step_id-only unique index in place (batch_alter_table used
+ADD COLUMN rather than rebuilding the table). This migration forces a full table
+rebuild to drop that stale index.
+
+Revision ID: 0009
+Revises: 0008
+Create Date: 2026-06-28
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+
+revision: str = "0009"
+down_revision: Union[str, None] = "0008"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # Force a full table rebuild so the stale unique index on step_id alone is
+    # dropped. recreate="always" tells Alembic to copy-rename-drop rather than
+    # issuing ADD/DROP statements (which SQLite doesn't support for constraints).
+    with op.batch_alter_table("playbooks", recreate="always") as batch_op:
+        # Re-declare the (step_id, type) constraint — the column-level unique is
+        # NOT carried over when we supply our own UniqueConstraint.
+        batch_op.create_unique_constraint("uq_playbooks_step_type", ["step_id", "type"])
+
+
+def downgrade() -> None:
+    # Nothing to do — downgrading 0008 already handles cleanup.
+    pass

progi/alembic/versions/0010_fix_playbooks_step_unique.py

@@ -0,0 +1,52 @@
+"""fix playbooks: drop stale step_id-only unique index via raw SQL table rebuild
+
+Alembic batch_alter_table reflects the existing table and reproduces the old
+column-level unique on step_id. This migration does the rebuild manually so we
+have full control over the resulting schema.
+
+Revision ID: 0010
+Revises: 0009
+Create Date: 2026-06-28
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+
+revision: str = "0010"
+down_revision: Union[str, None] = "0009"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.execute("""
+        CREATE TABLE playbooks_new (
+            id      INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
+            step_id INTEGER NOT NULL REFERENCES steps(id) ON DELETE CASCADE,
+            type    VARCHAR(16) NOT NULL DEFAULT 'worker',
+            content TEXT NOT NULL,
+            UNIQUE (step_id, type)
+        )
+    """)
+    op.execute("INSERT INTO playbooks_new SELECT id, step_id, type, content FROM playbooks")
+    op.execute("DROP TABLE playbooks")
+    op.execute("ALTER TABLE playbooks_new RENAME TO playbooks")
+
+
+def downgrade() -> None:
+    # Restore the original schema (single unique on step_id, no type column).
+    # Any qa-type rows are dropped because the old schema can't hold them.
+    op.execute("""
+        CREATE TABLE playbooks_old (
+            id      INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
+            step_id INTEGER NOT NULL UNIQUE REFERENCES steps(id) ON DELETE CASCADE,
+            content TEXT NOT NULL
+        )
+    """)
+    op.execute("""
+        INSERT INTO playbooks_old (id, step_id, content)
+        SELECT id, step_id, content FROM playbooks WHERE type = 'worker'
+    """)
+    op.execute("DROP TABLE playbooks")
+    op.execute("ALTER TABLE playbooks_old RENAME TO playbooks")

progi/alembic/versions/0011_qa_enabled.py

@@ -0,0 +1,70 @@
+"""Add qa_enabled to steps; insert default QA template for all steps without one
+
+qa_enabled=False means the QA playbook row exists (so it can be edited in the UI)
+but the gate is skipped at runtime. Set to True to activate QA for a step.
+
+Revision ID: 0011
+Revises: 0010
+Create Date: 2026-06-28
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+revision: str = "0011"
+down_revision: Union[str, None] = "0010"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+_QA_TEMPLATE = """\
+## What to evaluate
+
+<One sentence: what this step produces and what "good" looks like.>
+
+## Pass criteria
+
+- <Specific, checkable condition — e.g. "Output contains a valid file path">
+- <Specific, checkable condition>
+
+## Reject if
+
+- <Specific failure condition — e.g. "Output is vague or missing required field X">
+- <Specific failure condition>"""
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("steps") as batch_op:
+        batch_op.add_column(
+            sa.Column("qa_enabled", sa.Boolean, nullable=False, server_default="0")
+        )
+
+    # Insert default QA template for every step that doesn't already have one.
+    conn = op.get_bind()
+    step_ids = [r[0] for r in conn.execute(sa.text("SELECT id FROM steps")).fetchall()]
+    existing_qa = {
+        r[0]
+        for r in conn.execute(
+            sa.text("SELECT step_id FROM playbooks WHERE type = 'qa'")
+        ).fetchall()
+    }
+    missing = [sid for sid in step_ids if sid not in existing_qa]
+    if missing:
+        conn.execute(
+            sa.text(
+                "INSERT INTO playbooks (step_id, type, content) VALUES (:sid, 'qa', :content)"
+            ),
+            [{"sid": sid, "content": _QA_TEMPLATE} for sid in missing],
+        )
+
+
+def downgrade() -> None:
+    with op.batch_alter_table("steps") as batch_op:
+        batch_op.drop_column("qa_enabled")
+
+    # Remove template QA rows (those whose content matches the template exactly).
+    conn = op.get_bind()
+    conn.execute(
+        sa.text("DELETE FROM playbooks WHERE type = 'qa' AND content = :content"),
+        {"content": _QA_TEMPLATE},
+    )