feat(triage): effort-estimation skill and auto-promotion gate

FEATURES.md

@@ -17,7 +17,7 @@ A new configuration option often falls into one of three categories:
    the post-script needs to consume it. This usually also requires a
    schema update (`schemas/<agent>-result.schema.json`) to accommodate
    whatever new field the agent uses to expose its conclusions to the
-   post-script. Example: the `requires_workflow_changes` flag in triage.
+   post-script. Example: the `block_auto_promotion` field in triage.
 
 ## 1. Decide what changes and what stays the same
 

agents/triage.md

@@ -3,6 +3,7 @@ name: triage
 description: Inspect an issue, assess information sufficiency, and produce a structured triage decision.
 skills:
   - issue-labels
+  - effort-estimation
 # curl: required by GitLab and Jira forges. On GitHub, the network policy
 # binary allowlist (policies/github/triage.yaml) excludes **/curl,
 # preventing it from making network requests even though it is granted here.
@@ -130,7 +131,7 @@ Before forming any clarifying question, classify it:
 - Can you form a plausible root cause hypothesis from the available information?
 - Could a developer start investigating without contacting the reporter?
 - **Is progress blocked on other work?** Consider whether the fix depends on an unresolved issue or unmerged PR — in this repo or another. If a developer cannot meaningfully start work until some other issue is resolved, this issue has prerequisites regardless of how clear the problem description is. If the blocking work has no tracking issue yet, you can recommend creating one via the `prerequisites` action's `create` array.
-- **Would resolving this issue require modifying CI/workflow files?** Scan the issue title, body, referenced files, and labels for signals that the fix involves changes under CI/pipeline configuration (e.g., `.github/workflows/`, `.gitlab-ci.yml`, `.fullsend/.github/workflows/`, or enrolled-repo shim workflows). Prefer deterministic signals — explicit path references, CI/workflow-scoped labels, mentions of CI pipeline configuration — over vague mentions of "workflow" in non-CI contexts (e.g., "user onboarding workflow"). If the fix likely requires workflow file changes, set `requires_workflow_changes: true` in `triage_summary` and include a warning in the triage comment that the code agent cannot modify workflow files under current permissions and that manual intervention (human PR/MR or maintainer action) is required.
+- **Would resolving this issue require modifying CI/workflow files?** Scan the issue title, body, referenced files, and labels for signals that the fix involves changes under CI/pipeline configuration (e.g., `.github/workflows/`, `.gitlab-ci.yml`, `.fullsend/.github/workflows/`, or enrolled-repo shim workflows). Prefer deterministic signals — explicit path references, CI/workflow-scoped labels, mentions of CI pipeline configuration — over vague mentions of "workflow" in non-CI contexts (e.g., "user onboarding workflow"). If the fix likely requires workflow file changes, set `block_auto_promotion` with `blocked: true` and a reason explaining that the code agent cannot modify workflow files under current permissions and that manual intervention is required.
 - **Does this issue bundle multiple independent concerns?** An issue bundles independent concerns when it lists several distinct problems, tasks, or gaps that share no blocking relationship — each could be filed, triaged, and resolved independently. Use `action: "split"` to decompose the issue into separate sub-issues. Signs of a bundled issue:
   - A numbered or bulleted list of distinct items (e.g., "1. fix X, 2. add Y, 3. update Z")
   - Multiple unrelated components, files, or subsystems mentioned with no dependency between them
@@ -342,6 +343,8 @@ Information is sufficient for a developer to investigate and fix.
 
 **Choosing a category:** the `feature` category covers issues that describe desired new behavior rather than a defect in existing functionality — the reporter expects something that has never been implemented. Use `feature` only when the described behavior clearly never existed in the product. If there is _any_ possibility the behavior is a regression (it used to work, or the reporter references a specific version where it worked), use `insufficient` instead and ask for version or timeline information. When in doubt, ask — do not prematurely reclassify.
 
+**Estimating effort:** For bug, documentation, and performance categories, use the `effort-estimation` skill to score implementation effort and populate `block_auto_promotion`. Feature issues already route to human review and do not need effort estimation.
+
 ```json
 {
   "action": "sufficient",
@@ -364,9 +367,12 @@ Information is sufficient for a developer to investigate and fix.
     "impact": "Who is affected and how",
     "recommended_fix": "What a developer should investigate.",
     "proposed_test_case": "Conceptual description of a test that would verify the fix — what to test, expected vs actual behavior, and edge cases to cover. Do not assume a specific test framework or file layout.",
-    "requires_workflow_changes": false
+    "block_auto_promotion": {
+      "blocked": false,
+      "reason": "Low effort; single-file fix with existing test coverage"
+    }
   },
-  "comment": "A triage summary comment formatted in markdown. Focus on information not already present in the issue body — omit sections that merely restate what the reporter wrote. Include the proposed test case as a fenced code block.",
+  "comment": "A triage summary comment formatted in markdown. Focus on information not already present in the issue body — omit sections that merely restate what the reporter wrote. Do not include fenced code blocks; summarize test cases and fixes in prose. Use inline `backtick` references for identifiers.",
   "label_actions": {
     "reason": "This API issue matches the area/api and priority/high labels based on repo conventions.",
     "actions": [
@@ -377,7 +383,14 @@ Information is sufficient for a developer to investigate and fix.
 }
 ```
 
-**Workflow change detection (optional):** If the issue likely requires modifying CI/pipeline configuration files (`.github/workflows/`, `.gitlab-ci.yml`, `.fullsend/.github/workflows/`, or enrolled-repo shim workflows), set `requires_workflow_changes: true` in `triage_summary`. When set, the post-triage script skips auto-triggering the code agent because the code agent cannot modify workflow files under current permissions. The triage comment should warn about this limitation and note that manual intervention is required. When `requires_workflow_changes` is not set or is `false`, auto-triggering proceeds normally.
+**Blocking auto-promotion:** Use the `block_auto_promotion` field in `triage_summary` to prevent the post-triage script from auto-promoting the issue to the code agent. Set `blocked: true` with a `reason` when:
+- The `effort-estimation` skill determines the issue requires human review (effort >= 4, or any single dimension scores 5).
+- The fix requires modifying CI/pipeline configuration files (`.github/workflows/`, `.gitlab-ci.yml`, `.fullsend/.github/workflows/`, or enrolled-repo shim workflows) that the code agent cannot modify under current permissions.
+- Any other condition where auto-dispatch would be premature.
+
+These conditions are OR-combined: if any one applies, set `blocked: true`.
+
+The block only affects categories that would otherwise auto-promote (bug, documentation, performance). For those categories, when `blocked` is `true`, the post-script applies `triaged` instead of `ready-to-code` and appends the reason to the triage comment. Categories that already route to `triaged` (feature, security, other) are unaffected. When `blocked` is `false` (or omitted), auto-promotion proceeds normally for bug/documentation/performance categories.
 
 **Label recommendations (optional, all actions):** If the `issue-labels` skill identifies labels that should be applied or removed, include them in the `label_actions` field. This field is optional for all actions. If no labels clearly apply, omit it entirely.
 

docs/code.md

@@ -33,7 +33,7 @@ on issues (not PRs).
 
 | Label | Meaning |
 |-------|---------|
-| `ready-to-code` | Triggers the code agent. Applied by the [triage](triage.md) agent for low-risk categories (bug, documentation, performance), or manually by a human for feature work after prioritization. Not applied when the triage result sets `requires_workflow_changes`, since the code agent cannot modify workflow files. |
+| `ready-to-code` | Triggers the code agent. Applied by the [triage](triage.md) post-script for low-risk categories (bug, documentation, performance) when auto-promotion is not blocked, or manually by a human for feature work, high-effort issues, or workflow changes after review. |
 | `ready-for-review` | Applied by the code agent after pushing a PR. In per-repo installs, triggers the [review agent](review.md) when applied to a PR. Also marks workflow state for humans and the [retro agent](retro.md). |
 
 ## Configuration

docs/triage.md

@@ -40,14 +40,14 @@ These labels are managed by the triage agent based on its assessment of the issu
 | Label | Meaning |
 |-------|---------|
 | `needs-info` | The issue lacks sufficient information. The agent posted clarifying questions. |
-| `ready-to-code` | The issue is fully specified and low-risk (bug, documentation, performance). Bug and documentation categories also receive their eponymous labels (`bug`, `documentation`) automatically. Triggers the [code agent](code.md). This behavior is configurable via [Variables](#variables). Exception: when `requires_workflow_changes` is set in the triage result, `triaged` is applied instead because the code agent cannot modify workflow files. |
-| `triaged` | The issue is fully specified but is a feature or other category that requires human prioritization before coding. |
+| `ready-to-code` | The issue is fully specified and low-risk (bug, documentation, performance) with auto-promotion not blocked. Bug and documentation categories also receive their eponymous labels (`bug`, `documentation`) automatically. Triggers the [code agent](code.md). This behavior is configurable via [Variables](#variables). |
+| `triaged` | The issue requires human review before coding: feature work, other categories, or bug/docs/performance issues where `block_auto_promotion` is set (high effort, workflow changes, etc.). |
 | `duplicate` | The issue duplicates an existing one. The agent identified the original and the issue is closed automatically. |
 | `blocked` | The issue depends on another issue or external condition. The agent identified the blocker. |
 | `feature` | The issue is a feature request. Applied alongside `triaged` so humans can prioritize before coding begins. |
 | `question` | The issue is a question rather than a bug or feature request. |
-| `bug` | The issue is a confirmed bug. Applied alongside `ready-to-code` to categorize the issue. |
-| `documentation` | The issue concerns documentation improvements or additions. Applied alongside `ready-to-code` to categorize the issue. |
+| `bug` | The issue is a confirmed bug. Applied alongside `ready-to-code` or `triaged` to categorize the issue. |
+| `documentation` | The issue concerns documentation improvements or additions. Applied alongside `ready-to-code` or `triaged` to categorize the issue. |
 | `not-planned` | The issue is out of scope, invalid, or spam. The issue is closed with reason "not planned". |
 | `pr-open` | An open PR or merge request already addresses this issue. Applied either by the triage agent's `in-progress` action — used when a PR/MR *fixes* the issue, as opposed to `prerequisites`/`blocked` when a PR/MR must merely land first — or by the code agent's pre-check when it finds a human PR before dispatching. No automation clears this label when the linked PR/MR is closed without merging: nothing re-triages on PR/MR close, so the issue keeps `pr-open` — and the in-progress comment stays on the issue — until triage runs again, via an issue edit or a manual `/fs-triage`. |
 
@@ -145,6 +145,28 @@ This gives the triage agent the subtlety it needs to distinguish between
 controller-runtime code, without adding label documentation to `AGENTS.md`
 where every agent would pay the context cost.
 
+### Skill: `effort-estimation`
+
+The triage agent includes an `effort-estimation` skill that scores
+implementation effort and decides whether the issue should be held for human
+review before auto-promoting to the code agent. The skill populates the
+`block_auto_promotion` field in `triage_summary`:
+
+- `blocked: true` + `reason`: the post-script applies `triaged` instead of
+  `ready-to-code` and appends the reason to the triage comment.
+- `blocked: false` + `reason`: auto-promotion proceeds normally.
+
+By default, the skill scores effort on a 1 to 5 scale across four dimensions
+(scope, testing, domain knowledge, risk). Issues scoring >= 4 are blocked.
+
+The same `block_auto_promotion` field is used for workflow-change detection:
+if the fix requires modifying GitHub Actions workflow files, the agent sets
+`blocked: true` with a reason explaining that the code agent cannot modify
+workflow files.
+
+See [Fullsend's Customizing with Skills docs](https://fullsend.sh/docs/guides/user/customizing-with-skills.html)
+to know where to add the skill files.
+
 ### Variables
 
 | Variable | Description | Default | Valid values |

eval/triage/cases/009-effort-high-multi-component/annotations.yaml

@@ -0,0 +1,66 @@
+# This case tests the effort-estimation gate. The bug spans multiple
+# components (session, auth middleware, views, rate limiter) and requires
+# new test infrastructure. The agent should score effort >= 4 and set
+# block_auto_promotion.blocked = true, resulting in "triaged" instead
+# of "ready-to-code".
+state: open
+
+labels:
+  required:
+    - triaged
+    - bug
+  forbidden:
+    - ready-to-code
+
+triage_summary:
+  category: bug
+  block_auto_promotion:
+    blocked: true
+
+max_turns: 30
+max_cost_usd: 2.00
+
+triage_expectations: |
+  This issue reports a memory leak in the session store and a security
+  flaw where logout does not invalidate tokens. The fix touches at least
+  four files across two packages (auth, middleware) and requires new
+  test fixtures for session lifecycle. `src/api/users.py` is an unrelated
+  mass-assignment handler; do not count it toward scope.
+
+  The effort-estimation skill should score this high because:
+  - Scope: four files across two packages (session.py, views.py,
+    auth middleware, rate_limit.py) — 4.
+  - Testing: no existing session lifecycle tests; needs new test
+    infrastructure for time-dependent behavior (mocking time.time) — 4.
+  - Domain knowledge: requires understanding subsystem design across
+    the session store, auth middleware, and rate limiter — 3.
+  - Risk: changing session management is a breaking change affecting
+    every authenticated endpoint and downstream consumers; incorrect
+    eviction could log out active users or leave stale tokens valid — 5.
+
+  Overall effort should be >= 4 (average 4.0), triggering
+  block_auto_promotion. Risk=5 also independently triggers blocking
+  per the "any single dimension scores 5" rule.
+
+  A good triage should:
+
+  1. Verify the claims against the code: confirm SESSIONS only evicts
+     on lookup, confirm logout_handler is a no-op, confirm RATE_LIMITS
+     has the same accumulation pattern.
+  2. Identify this as a bug (category: bug), not security. The session
+     TTL is implemented but the eviction is broken, and logout is
+     documented but not functional. Do not split; the three symptoms
+     (session leak, no-op logout, rate-limiter accumulation) share the
+     same root cause (missing eviction/invalidation) and should be
+     fixed together as a single defect.
+  3. Set block_auto_promotion.blocked = true with a reason citing the
+     multi-component scope and testing requirements.
+  4. The post-script should apply "triaged" + "bug" (not "ready-to-code").
+
+  Scoring guide:
+  A score of 1 means the agent misidentified the action or category.
+  A score of 3 means correct action and category but did not flag the
+  effort level or missed the rate_limit.py parallel issue.
+  A score of 5 means correct triage, verified all claims against code,
+  identified the cross-cutting nature, and blocked auto-promotion with
+  a clear reason.

eval/triage/cases/009-effort-high-multi-component/input.yaml

@@ -0,0 +1,41 @@
+forge: github
+fixture:
+  type: issue
+  title: "Session tokens never expire in practice — memory leak and security risk"
+  body: |
+    ## Bug Report
+
+    **What happened:**
+    We noticed our production server's memory usage climbs steadily over time
+    and never drops. After profiling, we traced it to the in-memory `SESSIONS`
+    dict in `src/auth/session.py` — it grows indefinitely because expired
+    sessions are only removed on lookup (`get_session`), never proactively
+    cleaned. Tokens that are never looked up again stay in memory forever.
+
+    On top of the memory leak, `logout_handler` in `src/auth/views.py` does
+    not delete the session — the token remains valid for the full TTL even
+    after the user explicitly logs out. Combined with the lack of cleanup,
+    this means a stolen token can be used long after the user thought they
+    signed out.
+
+    **Steps to reproduce:**
+    1. Start the server and log in 10,000 times (scripted).
+    2. Never revisit any of those sessions.
+    3. Observe `SESSIONS` dict size — it holds all 10,000 entries.
+    4. Log out — the session token is still usable afterward.
+
+    **Expected behavior:**
+    - A background task or TTL-based eviction should remove expired sessions.
+    - `logout_handler` should invalidate the session token immediately.
+    - The auth middleware should stop accepting tokens after logout.
+
+    **Impact:**
+    - Production OOMs every ~3 days, requiring manual restarts.
+    - Security: logout does not actually invalidate access.
+    - Affects the rate limiter too — `RATE_LIMITS` in
+      `src/middleware/rate_limit.py` has the same pattern (entries accumulate
+      without proactive cleanup).
+
+    **Environment:**
+    - Python 3.12, single-process deployment
+    - ~2,000 active users, ~15,000 logins/day

eval/triage/cases/009-effort-high-multi-component/repo/README.md

@@ -0,0 +1,31 @@
+# User Service
+
+A Python web application with authentication, user management API,
+rate limiting, and session management.
+
+## Architecture
+
+```
+src/
+  auth/         Session management, validators, login/logout views
+  api/          REST endpoints for user CRUD
+  middleware/   Auth enforcement, rate limiting
+  db/           Data access layer
+tests/          Unit tests
+```
+
+## Running
+
+```bash
+pip install -r requirements.txt
+python -m src.main
+```
+
+## Configuration
+
+Environment variables:
+
+- `SESSION_TTL` — session timeout in seconds (default: 3600)
+- `RATE_LIMIT_WINDOW` — rate limit window in seconds (default: 60)
+- `RATE_LIMIT_MAX` — max requests per window (default: 100)
+- `LOG_LEVEL` — logging verbosity (default: INFO)