Conversation
chore: 0.9.0 kickoff — planning doc + backlog cleanup
…328) * fix(cost): forward --max-run-* flags from run_overnight.py to batch The cost/token limit flags (`--max-run-cost-usd`, `--max-run-input-tokens`, `--max-run-output-tokens`) are defined on solve_issues.py and consumed by workers/opencode_adapter.py. They were forwarded by solve_issues_batch.py to its worker processes, but were silently dropped by run_overnight.py before reaching the batch runner — so an overnight run with a cost or token cap set on the parent never actually enforced it. This was already partially fixed for the batch path; the gap was the overnight → batch hop. This commit closes that gap. Changes: - run_overnight.parse_args: 3 new flags (`--max-run-cost-usd`, `--max-run-input-tokens`, `--max-run-output-tokens`) with help text noting they are forwarded to the batch solver. - run_overnight.build_batch_command: forwards the 3 flags when set, using the same defensive `getattr(args, ..., None) is not None` pattern that solve_issues_batch.build_worker_command already uses, so unset values are not emitted as flags and 0.0 / 0 are preserved as real caps. Fixes #324 * test(cost): cover batch + overnight cost-limit forwarding Regression coverage for #324. Pins the forwarding contract so the fix in run_overnight.py cannot silently regress. Three test classes: * SolveIssuesBatchForwardingTests: solve_issues_batch.build_worker_command forwards each of the 3 limits when set, omits them when None, treats 0.0 as a real cap, and handles the case where only one limit is set. * RunOvernightForwardingTests: same contract, but for the overnight → batch hop (run_overnight.build_batch_command). * EndToEndForwardingTests: an integration test that simulates the full overnight → batch → worker forwarding path and asserts the limit value set on the overnight runner reaches the worker command intact. This complements the existing test_run_overnight.py suite (which covers bounded-solver, health, and verbosity flags) by adding cost/token-limit flag coverage. No existing tests are modified. Part of #324
* feat(reviewer): add review_pr.py — runtime for the 3 reviewer prompts The 3 reviewer prompt profiles introduced in #313 (under `./agents/reviewers/reviewer-{code,architecture,documentation}.md`) were loadable artifacts but nothing invoked them. This commit adds `scripts/review_pr.py`, a separate script that: - Takes `--pr N` and `--role {code,architecture,documentation}`. - Resolves the role via `role_routing_loader.get_role_config`, mapping the user-facing arg to the role_routing.yaml key (`code` -> `reviewer_code`, etc.). - Loads the prompt text from the role's `prompt_file` field (declared in role_routing.yaml); no prompt content is duplicated in the script. - Fetches the PR diff from GitHub via the `application/vnd.github.v3.diff` Accept header (unified diff in one call). - Calls OpenRouter's `chat/completions` endpoint with the system + user messages, using the model declared in role_routing.yaml for the resolved role — never a hardcoded model. - Parses the `**Verdict**: <value>` line the prompts declare and emits the structured Markdown to stdout. Architectural invariant: this is a SEPARATE script, not a flag on `solve_issues.py`. The solver stays dumb. The script is the infrastructure for the 0.9.0 validation run (issue #326). It does NOT, by itself, validate that ai-issue-solver works end-to-end — that is the role of #326's real Solver run with a real `reports/runs/.../summary.txt`. This PR explicitly does not claim a solver-validation success; it only claims the runtime exists and works for all three reviewer sub-roles. Exposes for testability: - `resolve_role` / `load_prompt` / `fetch_pull_request_diff` / `call_openrouter` / `parse_verdict` / `run_review` — each can be called independently with injected dependencies. Part of #325 * test(reviewer): cover code / architecture / documentation reviewer paths Regression coverage for #325. The reviewer runtime depends on the 3 prompt files in .agents/reviewers/ and on the 3 reviewer entries in config/role_routing.yaml. Both are loadable artifacts that can drift independently of the script, so the tests pin the contract. Coverage (8 test classes, 45 tests): * ConstantsTests (4) — ROLE_ALIASES has exactly the 3 documented reviewer paths, the alias-to-role_routing.yaml mapping is correct, REVIEWER_PROMPT_FILES matches the 3 known prompt files, and those files exist on disk. * ResolveRoleTests (5) — each of code/architecture/documentation resolves to the right role_routing.yaml entry; unknown role and missing-from-config cases raise ReviewerRoleError. * LoadPromptTests (5) — all 3 real prompt files load and contain the expected header + '**Verdict**' marker; role without 'prompt_file' raises; missing prompt file raises. * ParseVerdictTests (8) — the 3 documented verdict values parse correctly, including extra whitespace, case insensitivity, and negation of hallucinated / missing values. * FetchPullRequestDiffTests (6) — 200 returns the diff; 404 raises PullRequestNotFoundError; 5xx propagates HTTPError; the Accept header is application/vnd.github.v3.diff; Bearer token is sent when present and skipped when absent; oversized diffs are truncated with a clear marker. * CallOpenRouterTests (6) — success returns assistant content; the model in the request payload is the role's value, not a hardcoded string; missing token raises; HTTP error propagates; missing choices / missing message content raise ValueError. * RunReviewTests (6) — happy path for each of code / architecture / documentation (3 separate tests); role error and PR-not-found propagate end-to-end; the model passed to the LLM call is the role_routing.yaml value, not a test-local constant. * ParseArgsTests (4) — minimal args parse; --dry-run toggles; invalid role and missing --pr are rejected by argparse. * ReviewerVerdictDataclassTests (1) — sanity check on the output dataclass. Tests use real .agents/reviewers/ files and the real config/role_routing.yaml; only the network calls (GitHub API and OpenRouter) are mocked via a session-injection seam. This is the infrastructure for #326, not a Solver-validation run. The PR is a Mavis-as-dev contribution; #326 must produce a real `reports/runs/.../summary.txt` to count as system-under-test evidence. Part of #325
Pre-Work for #326 (first real Solver run with a real `reports/runs/.../summary.txt`). The 0.9.0 validation run needs a reliable issue pool to draw from, but `docs/BACKLOG/open.md` was no longer trustworthy: it listed 10 sections whose GitHub issues were already closed, plus 2 sections that were Priority-1 by label but should be parked until 0.9.0 results are in. Scope (per user instruction, 2026-06-18): 1. Move these 10 closed sections from open.md to done.md with provenance pointing at the closing issue number: | Section | Title | Closed via | |---------|--------------------------------------|------------| | §5 | Claude Code alternative | #191 | | §6 | Low-code / non-code repos | #188 | | §15 | Process retrospective | #218 | | §16 | GitHub repository intelligence | #213 | | §17 | Workflow control | #216 | | §18 | Codex sandbox hardening | #217 | | §19 | Structured rework workflow | #220 | | §21 | Solver process supervisor | #223 | | §24 | GitHub Actions solver trigger | #243 | | §25 | Decompose oversized issues | #244 | 2. Mark §37 (Free OpenCode models) and §39 (periodic documentation benchmark) as parked/future under a new `## Parked / Future` section at the end of open.md, with a `Parked because:` note attached to each: - §37: not 0.9.0-critical; the hard-coded free-models list is known stale (see agent memory 2026-06-14) and must be re-verified before any real run. - §39: depends on later validation/model-comparison data from 0.9.0; defer until the 0.9.0 validation report and free-model registry are stable. 3. Replace references to the legacy `NEXT_BACKLOG.md` filename inside the remaining open.md body with `open.md`. The single reference in the file header (line 14) is intentionally left alone: it documents the historical rename in 0.7.0 and is informative. After this commit: - `open.md` is 269 lines (was 648) and contains exactly 1 active section (§22) plus 2 parked sections (§37, §39). - `done.md` is 213 lines (was 142) with the 10 new entries appended in the existing '## Done — §N Title (#issue)' format. - The file is now a reliable input for #326's issue selection. Anything that still appears in open.md is either an open priority item (§22) or a parked/future entry explicitly marked as such; create_backlog_issues.py will not pick up parked entries. Out of scope (explicit, per user): - No Solver-validation runs (this is documentation hygiene, not a system-under-test event). - No main-branch changes. - No work on #326. Sibling: #326 is the first real Solver run, gated on this cleanup land-merging first.
Extends the existing post_merge_cleanup.py with a new `--local-branches` mode that classifies and optionally deletes stale local branches. The pre-existing remote-branch behavior is unchanged; this commit is purely additive. Safety rule (pinned by tests, per issue #331): protected -> never delete merged into base -> can delete (with --apply) gone upstream + unmerged -> manual review only, never delete unmerged -> manual review only, never delete `gone-upstream` is a hint, never a deletion approval on its own. A branch whose commits are already on `base` is safe to delete even if its upstream is gone (typical post-merge flow: upstream deleted by GitHub after squash-merge). A branch whose upstream is gone AND unmerged is treated as a manual-review candidate because the local commits might be lost work that the user has not pushed. Implementation: * `run_git(args, cwd=None)` — top-level runner using `subprocess.run(... capture_output=True, text=True, check=False)`. Injected as `git_runner` in cleanup_local_branches and apply_local_branch_cleanup. Tests use a `FakeGitRunner` that records every call and returns pre-configured responses per (args-substring) pattern — same injection style as `scripts/review_pr.py`. * `cleanup_local_branches(base, show_unmerged, dry_run, protected_names, git_runner)` — runs `git fetch --prune origin`, enumerates local branches, classifies each into protected / merged (deletable) / unmerged (manual review). Returns a `LocalBranchReport` with all categories and any errors. * `apply_local_branch_cleanup(report, dry_run, git_runner)` — invokes `git branch -d` (not `-D`; merged-not-protected guarantee makes `-d` safe) for each entry in `report.to_delete`. Returns a list of action strings including any `git branch -d` failures. * `print_local_branch_report(report, dry_run)` — German output in the same style as the existing `print_cleanup_result`. * `LocalBranchEntry` and `LocalBranchReport` frozen / mutable dataclasses for the structured result. CLI: python scripts/post_merge_cleanup.py --local-branches --base develop python scripts/post_merge_cleanup.py --local-branches --base develop --apply python scripts/post_merge_cleanup.py --local-branches --base develop --show-unmerged python scripts/post_merge_cleanup.py --local-branches --base develop --apply --show-unmerged `--base` defaults to `develop`. `--show-unmerged` is the visibility flag for the manual-review list — it does NOT enable deletion of unmerged branches. Deletion of unmerged branches is not supported by this script and would require a conscious `git branch -D` by the user. Tests (added to existing test_post_merge_cleanup.py): * `FakeGitRunner` — substring-based fake that records every call and returns pre-configured responses. * `LocalBranchSafetyTests` (3) — main, develop, and current branch are never deletable, even with --apply. * `LocalBranchDeletionTests` (5) — merged is deletable; unmerged is not (with or without --show-unmerged); gone-upstream-and-merged is still deletable; gone-upstream-and- unmerged is manual-review, never deletable; --show-unmerged controls visibility of the manual-review list. * `LocalBranchFetchAndErrorsTests` (5) — git fetch --prune is invoked; fetch failure does not block classification; list-branches failure returns a non-empty errors list; git branch -d failure is reported in the actions list, not raised; no-local-branches case is empty and sane. * `LocalBranchDryRunVsApplyTests` (2) — dry_run issues no `git branch -d`; --apply issues `git branch -d` once per deletable branch. * `LocalBranchDataclassTests` (2) — LocalBranchEntry is frozen; LocalBranchReport exposes the expected fields. Total tests: 8 existing + 17 new = 25 (all green). `-W` (Python 3.10 and 3.12 CI matrix) clean. Out of scope (per issue #331): - No Solver-validation runs. This is workflow hygiene, not a system-under-test event. - No main-branch changes. - No work on #326. #326 is gated on this PR landing only via the merge queue, not on its semantic content. - No interactive prompts. Safe to run in CI / cron. - No new dependencies. Stays stdlib + `requests`. - No changes to the existing remote-branch deletion behavior (`cleanup_stale_branches` is byte-identical to before). Fixes #331
Documents the state of the 0.9.0 prep as of 2026-06-18 02:30 UTC, just before the first real Solver run (#326). What's new in CURRENT_CONTEXT: * Each of the four 0.9.0 scope pieces is now annotated as **landed**, **in progress**, or **pre-work done** with the exact commit SHAs and PR numbers (#328, #329, #330, #332). This is the audit trail that separates Mavis-as-dev infrastructure work from the upcoming Solver-produced run. * A new section "Resolved Questions (closed by 0.9.0 prep, 2026-06-18)" records the four closed items: backlog health, cost-limit flags, reviewer runtime, local branch cleanup. * "Current Focus" explicitly distinguishes **Mavis-as-dev infrastructure** (the four landed pieces) from the **Solver-produced metric** that #326 will produce. This is the same distinction captured in agent memory as "Mavis-as-dev vs the system Mavis is being measured on" (2026-06-18) and in the validation report at reports/validation-0.9.0.md. * "Open Questions" updated: "Solver effectiveness" is the only real open question, and the validation-target issue (#333) plus the report skeleton are in place to receive the first run. * New open item: GITHUB_TOKEN validity — the next Solver run needs a fresh token (current one returns Bad credentials as of 2026-06-18 02:25 UTC). The first Solver run cannot push branches or open PRs until this is fixed. Out of scope (explicit): no Solver-validation run started in this commit. This is documentation, not a run-report. No source code outside docs/CURRENT_CONTEXT.md is touched. Validation report skeleton lives at reports/validation-0.9.0.md but is correctly gitignored (reports/* is in .gitignore, only .gitkeep is tracked). The report is a runtime artifact, not source. Sister issue: #326 (Validation Metrics & Run). Validation target: #333.
…ipts/review_pr.py Automatisch gelöst mit AI Issue Solver (Modell: opencode) Issue: #333
[AI] Fix: Add docstring examples to parse_verdict in scripts/review_pr.py
fix(solver): add pre-run hygiene gate
#347) * fix: Löse Issue #346 — Align OpenRouter Direct guardrails with OpenCode solver runs Automatisch gelöst mit AI Issue Solver (Modell: opencode) Issue: #346 * fix(openrouter): stabilize direct guardrail tests * fix(openrouter): wire direct guardrails into solver reports --------- Co-authored-by: AI Issue Solver <ai@github.com>
…355) * fix: Löse Issue #354 — Harden OpenRouter Direct patch output and reject artifacts Automatisch gelöst mit AI Issue Solver (Modell: opencode) Issue: #354 * fix(openrouter): stabilize reject artifact handling tests * test(openrouter): allow platform-specific patch failures --------- Co-authored-by: AI Issue Solver <ai@github.com>
Merge PR #368 after MiniMax/Mavis rework and Codex review.\n\nValidated:\n- GitHub CI green on Python 3.10 and 3.12\n- scripts/review_pr.py code review after rework: approve\n- Scope stayed limited to shared solver command specs and tests\n\nFollow-up: add an explicit solve_issues.py rework mode for open PR branches.
Merge PR #365 after updating it onto current develop and addressing documentation review feedback.\n\nValidated:\n- Local targeted tests: tests.test_reviewer_runtime, tests.test_role_routing, plus combined reviewer/solver command tests during branch update\n- GitHub CI green on Python 3.10 and Python 3.12\n- Code reviewer verdict: approve\n- Documentation reviewer final verdict: approve\n\nAlso documents the 1.0 app vision and first production target repository.
Merge PR #358 after updating onto current develop.\n\nValidated:\n- Local targeted tests: tests.test_solve_issues, tests.test_worker_adapters, tests.test_opencode_session_reader, tests.test_solver_commands\n- GitHub CI green on Python 3.10 and Python 3.12\n- Code reviewer verdict: approve\n\nFollow-up improvements for WAL state conflict handling continue in PR #360.
Merge PR #360 after retargeting onto develop and resolving the stack after #358.\n\nValidated:\n- Local targeted tests: tests.test_solve_issues, tests.test_solve_issues_batch, tests.test_run_overnight, tests.test_solver_commands, tests.test_worker_adapters, tests.test_opencode_session_reader\n- GitHub CI green on Python 3.10 and Python 3.12\n- Code reviewer verdict: approve\n- Follow-up added clean-state guard coverage and fixed run_overnight step numbering.
Closes #336.\n\nValidated:\n- python scripts/solve_issues_batch.py --help\n- python -m compileall scripts tests\n- python -m unittest tests.test_solve_issues_batch\n- GitHub CI green on Python 3.10 and Python 3.12
Closes #338.\n\nValidated:\n- python -m unittest tests.test_model_selection\n- python -m compileall scripts tests\n- GitHub CI green on Python 3.10 and Python 3.12\n- Code reviewer verdict: approve
…able token cap (§56) (#440) Three failure modes measured across 7 solver runs in this session (see BACKLOG §56 in docs/BACKLOG/open.md for the full data): - Mode A (OpenRouter 400 before any token output): stop forcing use_structured_output=True in the rework code path; emit a plain patch prompt via worker.build_patch_prompt(structured=False). - Mode B (output truncated by 4096-token cap, then 'no patches'): read OPENROUTER_REWORK_MAX_TOKENS (default 16384) instead of the hardcoded 8192. - Mode C (worker writes a full rewrite-from-scratch diff that git apply rejects against the current branch tip): rework prompt now carries the current head SHA + the existing PR commit list and instructs the model to produce an incremental patch only. Plus: ValidationGitHubClient.get_pull_request_commits() for the new prompt context (404 returns []). Tests added for prompt shape, commit formatter, max_tokens override, and worker invocation without response_format. Branch: codex/rework-pr-fix (uncommitted before this commit) Tests: 39+52+163 OK; git diff --check clean Full discover: skipped (known slow repo-profiling path, single relevant test green in isolation) Live 3-run rework: not executed (no open PRs available right now) stash@{0}: untouched Co-authored-by: MiniMax <MiniMax@local>
- open.md: §57 stubbed out as DONE-via-PR-#442 (squash 8d68b50); §58 (priority/3, anti-pattern-doc) stays open as the next follow-up after this fix. - done.md: full closure entry for §57 with bug description, repro (PR #441 / Issue #389 / gpt-4o / 1-of-2 patches), fix locations (workers/base.py, workers/openrouter_worker.py, workers/execution.py, scripts/solve_issues.py), acceptance test (total=2/applied=1/failed=1 → returncode 6, no commit, no push, no PR), verification matrix (53+21+95+163 tests OK + GitHub CI green + user live review 'mergebereit'), and out-of-scope notes for §58 + the still-relevant patch-mismatch hardening for the normal solve path (potential §59).
- open.md: §56 stubbed out as DONE-via-PR-#440 (squash 166f8b2). Cross-references §57 (done) and the still-open patch-mismatch hardening for the normal solve path. - done.md: full §56 closure entry with three failure modes (A: 400, B: 4096 cap, C: rewrite-from-scratch diff), fix locations (prompts/rework_pr.md, scripts/validation/rework.py, scripts/validation/github_client.py), verification matrix (39+52+163 OK + GitHub CI green + user live review 'Sieht gut aus'), live-validation gap (no open PRs at fix time), and out-of-scope notes for §57 (now also closed), §58 (open), and the still-relevant patch-mismatch hardening for the normal solve path (potential §59). Backlog is now: §58 priority/3 open, plus the implied §59 (Patch-Mismatch-Hardening for the normal Solve-Pfad) which can be filed explicitly if/when we want a separate item.
* docs(solver): guard against recently removed patterns (§58) * fix(solver): relativize recently-removed-patterns path in worker prompt Live-review finding by Guido (PR #443 review): `build_solve_prompt` included the local absolute path of docs/AGENTS.md in the worker prompt (`(Pattern list sourced from /Users/.../docs/AGENTS.md.)`), which leaked the operator's local filesystem layout into the LLM context. Cuts against the existing prompt-hygiene in this file. Fix: - In `build_solve_prompt`, display the patterns-file path as repo-relative (`docs/AGENTS.md`) when the resolved path lives under PROJECT_ROOT. - If the operator explicitly sets `AIS_RECENTLY_REMOVED_PATTERNS_FILE` to an absolute path outside PROJECT_ROOT, show that absolute path verbatim — it was an explicit choice, not a leak from default config. - Two new tests in tests/test_solve_issues.py: - `test_solve_prompt_does_not_leak_absolute_project_root_path`: default path is shown repo-relative, PROJECT_ROOT / /Users/ / tempdir are not in the prompt. - `test_solve_prompt_does_not_leak_absolute_path_from_env_var_override`: explicit env-var override outside PROJECT_ROOT is preserved verbatim. Tests: 168 OK (+2). `git diff --check` clean. --------- Co-authored-by: MiniMax <MiniMax@local>
- open.md: §58 stubbed out as DONE-via-PR-#443 (squash 11eafc1). Notes that the docs/AGENTS.md Recently-Removed-Patterns list is now maintainer-pflegbar; future PRs that intentionally remove a pattern should add a row in the same PR. Cross-references the still-open potential §59 (patch-mismatch hardening). - done.md: full §58 closure entry with problem (PR #441 episode), fix locations (docs/AGENTS.md, scripts/solve_issues.py, tests/test_solve_issues.py), the user-found path-leak fix from live review (now repo-relative when under PROJECT_ROOT, verbatim for explicit env-var override), verification matrix (168 OK + GitHub CI green + manual probe + user live review 'mergebereit'), and the maintainer obligation going forward. Backlog is now EMPTY: §51/§52/§53/§54 done-via-PRs, §55 done-via-PR-#439, §56/§57/§58 documented as done. The implicit §59 (patch-mismatch hardening for the normal solve path) remains a possible future item but is not filed.
§59 = 'Watchlist: Patch-mismatch hardening for the normal solve path'. Priority 4, labels kind/watchlist + theme/solver + area/prompt. Status: WATCHLIST ONLY — not an active backlog commitment. No code or architecture work until the activation trigger (≥3 Mode-C patch-mismatch runs on the normal solve path, ideally across different files) is met. Rationale: - The Mode-C failure (worker produces a patch that git apply rejects because it does not match the current working tree) is the same patch-mismatch symptom §56 addressed for --rework-pr. - Before §57, Mode-C on the normal solve path could produce regression-PRs (PR #441 vs. PR #439). After §57, Mode-C is a clean failure with no PR — acceptable behavior. - A §59 fix would turn these clean failures into clean successes, but the architectural cost (prompt anchoring, per-file git apply --check, optional re-prompt loop) is only worth it if the failure pattern is systematic. One data point (Issue #389 re-run, scripts/model_selection.py:52, 1 patch, 0 applied, no PR) is not enough to justify the work. - When a Mode-C failure appears on the normal solve path, log the file + issue + date here so we can see when the threshold is met. Non-goal while parked: no code changes. §57 + §58 are sufficient to keep the pipeline correct.
Tracked from PR #444 (Issue #390 validation run, 2026-06-26). Same failure mode as §57 (PR #442): worker reports nonzero exit + partial on-disk state, but classify_worker_outcome in workers/execution.py did not catch returncode 5 (reject artifacts) and let it fall through to the generic nonzero_with_changes branch with failure_class: success. The PR that slipped through: - worker_exit_code: 5 (Reject-Artefakte) - run_outcome_worker_status: failed - run_outcome_has_changes: True - run_outcome_failure_class: success (LIE classification) - run_outcome_delivery_status: pr_created - Diff: a 5-LOC trivial dummy function - Post-solve tests: not_run (timeout 300s) §57 closed returncode 6 (partial_patch_failure). §60 closes returncode 5 (reject_artifacts) and any future reject / partial- state class. General principle: 'Any nonzero worker-returncode that produces partial on-disk changes must be a hard stop. Commit + push + PR-create must not run.' Suggested scope: explicit allow-list (only returncode 0 may proceed with has_changes=True) OR explicit deny-list extending the §57 check; WorkerOutcome invariant test; unit tests for returncode 5 hard-stop; possibly split nonzero_with_changes into specific classes (reject_artifacts, partial_state, etc.). Touches: workers/execution.py, workers/base.py, workers/openrouter_worker.py, scripts/solve_issues.py, tests.
- open.md: §60 stubbed out as DONE-via-PR-#445 (squash 2549f0f). Cross-references §57 (also done) and the still-valid general principle for any future returncode class. Old §60 body (Measured/Suggested scope/Checks) trimmed from open.md. - done.md: full §60 closure entry with bug description, repro (PR #444 / Issue #390 / gpt-4o / returncode 5 / reject artifacts), fix locations (workers/base.py + workers/execution.py + workers/openrouter_worker.py + scripts/solve_issues.py), scope discipline (NO refactor of nonzero_with_changes, per User directive), verification matrix (22+96+53+168 OK + GitHub CI green + user live review 'mergebereit'), and the general principle that §57 (returncode 6) and §60 (returncode 5) both implement. Backlog is now: §59 watchlist (parked, Mode-C Threshold ≥3) + §59-related notes only. §60 closed. All §51-§58 done via earlier PRs.
* fix: Löse Issue #446 — Update README for current solver workflow Automatisch gelöst mit AI Issue Solver (Modell: openrouter_direct) Issue: #446 * docs(readme): extend #446 fix with dynamic-discovery + recently-removed-patterns guards Extends PR #447 (liquid/lfm-2.5-1.2b-instruct:free initial fix that hard-stopped cleanly after §57+§60, but covered only partial scope of Issue #446). User reviewed and asked for two missing points: 1. **OpenCode Free Models (dynamisch)**: new block in README that explains scripts/model_catalog.py as the live source of truth for free-model discovery (with cache + static fallback, 'missing' / 'stale' classification, --verify-opencode and --list-opencode-free-models CLI paths). 2. **Recently-Removed-Patterns-Guard**: new block that explains the docs/AGENTS.md 'Recently Removed Patterns' table the solve prompt now consults before re-introducing a recently-removed pattern. Lists the two current entries (static free_models removed by PR #439; $20/$20 cost-cap defaults replaced by $15/$50 + staggered warnings in PR #437). Notes that the guard is 'soft' (worker explains in the PR body if needed) and that maintainers update the table in every removal PR. PR #447 also kept its earlier additions (Hard-Stop-Sicherheit absatz + Backlog-Status-Update). Closes #446 (now full scope). * docs(readme): drop inaccurate 'Reviewer-Prompt uses model_catalog'-claim User live-review feedback on PR #447 (2026-06-26, second pass): the README block 'OpenCode Free Models (dynamisch)' claimed that the Reviewer-Prompt also uses the model_catalog.py mechanism. Code review shows scripts/review_pr.py does NOT consult model_catalog.py — it loads reviewer prompt files and does role-routing only. The claim was incorrect / overstated. Fix: scope the statement to the solver helpers that do use model_catalog.py ('--verify-opencode', '--list-opencode-free-models'). The reviewer path is unrelated and does not need a static model list, but also does not actively consult the dynamic discovery. --------- Co-authored-by: AI Issue Solver <ai@github.com> Co-authored-by: MiniMax <MiniMax@local>
…k-Sweep Adds a sequential benchmark runner that invokes solve_issues.py for each of the 31 free models available via OpenRouter (26 models, prompt=0/completion=0 per /models endpoint) and OpenCode (5 models, dynamically discovered via 'opencode models'). Each run targets Issue #446 (README-Update) and aggregates results into reports/benchmarks/free-models-2026-06-26.json plus a log file. Per-run classification (parsed from solver output): - success_pr_created / success_no_pr - patch_validation_failed_rc5 (reject artifacts; §60 territory) - patch_mismatch_mode_c (worker writes diff against wrong base) - no_patches (model produced no parseable diff) - openrouter_400 (model/slug invalid) - infrastructure_opencode_state_conflict (CLI/Serve version mismatch) - infrastructure_or_unknown_failure The benchmark sweep is invoked from Mavis after User directive ('Alle freien Modelle über OpenRouter und alle freien Modelle über OpenCode'); it is not part of the normal solver workflow.
- open.md: §61 stubbed out as DONE-via-PR-#447 (squash 9b85570). Notes that the three README areas (dynamic OpenCode free-model discovery, recently-removed-patterns guard, partial/reject-patch hard-stop) are now covered and that PR #447 also includes a correction removing an overstated 'Reviewer uses model_catalog' claim. - done.md: full §61 closure entry with the three-commit fix history (liquid/lfm-2.5-1.2b-instruct:free initial, Mavis extension, live-review correction), verification matrix (39+65+168 OK + GitHub CI green + user live review), and an out-of-scope note documenting the Free-Models-Benchmark-Sweep results (1/31 actually completed, 5 OpenCode fails on CLI/serve version conflict, methodology improvement tracked separately).
…' label Backlog: - §62 (priority/2, active) — Fix benchmark/open-PR workflow methodology. The current open-PR guard + missing GitHubClient.get_pull_requests() corrupted the 31-model Free-Models-Benchmark-Sweep (24 of 31 runs were silently aborted). Scope: --benchmark / --skip-pr combo + add get_pull_requests() method + 'foreign PR' distinction + update benchmark_free_models.py. §64 depends on this. - §63 (priority/3, active but not 0.9.0-blocking) — OpenCode app-state conflict resolution. CLI 1.15.13 vs MiniMax Code.app bundle 1.14.28 with app-launchd respawn. Scope: scripts/opencode_state_diagnostic.py + docs/OPENCODE_APP_STATE.md + option A/B/C for clean resolution. '--allow-opencode-state-conflict' stays as diagnostic tool. - §64 (priority/4, parked) — Free-model robustness study. Depends on §62. Goal: turn 'Free-Models-Produktivität ist Einzeldatenpunkt' into 'belastbare Statistik' across 5 models × 5 issues each. README update with recommendation table + 'experimental / supervised / docs-only' labels. NO production-ready claim until data is in. README: - After §61 PR #447 added the 'OpenCode Free Models (dynamisch)' block, this commit adds a sibling block: 'Free-Models Status (Stand 2026-06-26): experimentell und supervised-only'. Explains that gpt-4o (paid OpenRouter) stays the default for strategic issues, Free-Models are for smoke-tests / low-risk / docs-only, and every Free-Model PR needs Guido live review before merge.
* fix(solver): add benchmark mode for open PR sweeps * fix(benchmark): derive REPO from __file__ instead of hardcoded path PR #448 CI was failing with FileNotFoundError on /Users/Guido/... because scripts/benchmark_free_models.py hardcoded REPO to the developer's local absolute path. Local tests passed (path exists), CI failed (path does not exist in the GitHub runner). Fix: REPO = Path(__file__).resolve().parent.parent, same pattern as SCRIPT_DIR / PROJECT_ROOT in scripts/solve_issues.py. Verified locally: - test_benchmark_free_models: 1 OK - test_solve_issues: 173 OK - test_github_client: 25 OK --------- Co-authored-by: MiniMax <MiniMax@local>
- open.md: §62 stubbed out as DONE-via-PR-#448 (squash 0d08679). Notes the three-commit history (Codex fix + Mavis portability fix + merge), the scope discipline (§62 = methodology only, no §59/§64 mixed in), and that §64 is now unblocked. - done.md: full §62 closure entry with the bug description (hardcoded /Users/Guido path + missing get_open_pull_requests + open-PR-guard aborting 24 of 31 benchmark runs), fix locations (github_client.py alias, solve_issues.py --benchmark flag, benchmark_free_models.py flags + REPO-from-__file__ portability fix), verification matrix (1+173+25 OK + GitHub CI green after the portability fix + user live review), and the §64-unblock note. Backlog open.md is now: §59 (watchlist) + §63 (priority/3, active but not 0.9.0-blocking) + §64 (priority/4, parked, abhängig von §62). §64 is now unblocked for activation.
…sweeps Parametrize scripts/benchmark_free_models.py so it can run a small smoke-benchmark on any issue with any subset of free models, instead of the hardcoded 31-model sweep on Issue #446. New CLI: - --issue N (default 446) — issue to benchmark - --run-label LABEL — used in the aggregate/log filenames, so multiple benchmark runs can coexist without overwriting each other (smoke-free-models-2026-06-26.json vs the earlier free-models-2026-06-26.json for the 31-model sweep) - --models 'openrouter_direct:foo:free,openrouter_direct:bar:free' — comma-separated provider:model list. If omitted, falls back to the full OPENROUTER_FREE_MODELS + OPENCODE_FREE_MODELS list (preserves the original behavior) Test updated: - test_run_one_uses_benchmark_skip_pr_flags now also passes the issue_number argument and asserts that --issue and the issue number appear in the spawned solver command
§63 (OpenCode app-state conflict resolution, full app-side fix)
is parked. §65 (narrow repo-side scope) carries the diagnostic +
docs subset that is valuable immediately.
Adds:
- scripts/opencode_state_diagnostic.py — pure-stdlib script that
prints a clear report of:
- which opencode binaries are reachable (PATH, ~/.opencode/bin,
app bundles under /Applications/) with versions
- the running opencode-serve process (pid, binary path, version)
- which .app bundle owns the launchd respawn (matched from the
running serve binary)
- the configured $OPENCODE_BIN env-var
- a Verdict line that calls out the version conflict when the
serve binary does not match any discovered CLI version
Also supports --json for machine-readable output.
- docs/OPENCODE_APP_STATE.md — explains the conflict (CLI 1.15.13
vs Serve 1.14.28 in the MiniMax Code.app bundle), the
app-launchd respawn mechanism, and the three resolution
options (A: app update, B: rename app-bundled binary, C:
project-side always-use-configured-OPENCODE_BIN). Also
documents --allow-opencode-state-conflict as a diagnostic-
only override.
- README: cross-reference to docs/OPENCODE_APP_STATE.md in the
Free-Models section; also updates §64 reference to reflect its
closure-with-smoke-evidence status (instead of 'parked,
pending').
- tests/test_opencode_state_diagnostic.py — 8 tests covering the
human/JSON output, semver-with-suffix extraction, app-owner
detection, and OPENCODE_BIN env-var handling.
- docs/BACKLOG/open.md: §63 retitled to mark its diagnostic
subset as moved to §65, with §65 active and narrow. §64
closed with the smoke-benchmark evidence (4/4 attempted, 0/4
PR-relevant, no further 5x5 sweep planned).
Verified:
- 8 new tests OK
- ./venv/bin/python scripts/opencode_state_diagnostic.py prints
the expected report (2 binaries found, no running serve,
$OPENCODE_BIN pointing at the app bundle, Verdict OK since
no serve is currently running — conflict is latent)
PR #449 (squash e38c1f4) implemented dynamic OpenRouter free-model discovery: - new fetch_openrouter_free_models() in scripts/model_catalog.py, analog to fetch_opencode_free_models() (1h cache + XDG-aware path + OpenrouterModelCache dataclass with source='live|cache|fallback') - pricing-based free-filter via pricing.prompt == '0' AND pricing.completion == '0' (-suffix is intentionally NOT a fallback signal) - OPENROUTER_FALLBACK_FREE_MODELS tuple moved to model_catalog.py as single source of truth (replaces the old duplicated list in benchmark_free_models.py) - benchmark_free_models.py default path now calls both catalog functions; explicit --models still bypasses dynamic filtering - 5 new tests (3 in test_model_catalog.FetchOpenrouterFreeModelsTests, 2 in test_benchmark_free_models.BenchmarkFreeModelsTests) - README 'OpenRouter Free Models (dynamisch für Benchmarks)' block added between OpenCode and App-State sections §66 body in open.md retitled to DONE; full closure entry appended to done.md with repro + verification + scope-discipline note. Verified locally: 6 tests OK, live fetch returns source='live' with 26 models, git diff --check clean.
Triggered by 2026-06-26 Issue #450 benchmark sweep: all 4 worker runs were failures (rc=1 or rc=2), but benchmark_free_models.classify() returned 'success_no_pr' for all 4 because solve_issues.py exits rc=0 even on no_changes status. Fall-through `return 'success_no_pr'` is the inverse of the truth. Repro: 4 worker runs on Issue #450 — liquid (rc=2, empty response with 1387c Prosa), qwen (rc=1, 429), gemma (rc=1, 429), cohere (rc=2, 0-char response, 135s). Aggregate showed 4× success_no_pr. Fix-Plan: run-report summary.txt (worker_exit_code + has_changes + status) as ground truth, log-text heuristics as fallback. New classes: success_pr_skipped, no_changes, empty_response_rc2, model_failure_rc1, patch_validation_failed_rc5, partial_patch_failure_rc6, timeout_rc3, openrouter_429. ~140 LOC estimated, under 250 stop- criterion. Handover to Codex preferred (pipeline-correctness). User directive (2026-06-26 22:34): do §67 first (pipeline correctness), then Issue #450 with gpt-4o. Not parallel. Handover file at reports/handoffs/to-codex-20260626-2238-benchmark-classify-fix.md
…465) §67 priority/1 — pipeline-correctness blocker. The previous `classify()` had a fall-through `return "success_no_pr"` that masked real worker-failures (rc=1, rc=2, 429 rate-limits) as successful runs. The 2026-06-26 Issue #450 benchmark sweep confirmed the bug: all four Free-Models ran with rc=1 or rc=2, but the aggregate showed `success_no_pr` for all four. This change makes classify() consult the per-run `reports/runs/<ts>-<repo>-issue-<N>/summary.txt` (worker_exit_code + run_outcome.has_changes + status) as ground truth. New canonical classification classes replace the buggy fall-through: - success_pr_created - success_pr_skipped (--skip-pr with has_changes=True) - no_changes (rc=0 + has_changes=False; was success_no_pr) - empty_response_rc2 - model_failure_rc1 - patch_validation_failed_rc5 - partial_patch_failure_rc6 - openrouter_429 Log-text heuristics stay as a fallback for tests / mocked subprocess calls that do not produce a run-report. Inline summary.txt parsing keeps the module independent of the solver_reporting package, whose bare `from utils import` lines fail in the test harness. Adds 12 new tests covering each classification class, the run-report-discovery helper, and the log-text fallback path. Co-authored-by: mavis <Mavis@MiniMax.local>
Following the §66 / done-archive pattern from PR #449: - docs/BACKLOG/open.md: §67 header retitled to **DONE in PR #465 (squash 5fbc6f6)** (strikethrough). - docs/BACKLOG/done.md: full closure entry added. Documents: * root cause (classify() fall-through returning success_no_pr for every rc=0+no-PR run) * fix shape (_find_run_report, _read_run_report_summary, classify() with Run-Report-Primary classification) * new canonical classes table (success_pr_skipped, no_changes, empty_response_rc2, model_failure_rc1, etc.) * test coverage (15 tests: 3 existing + 12 new) * verification (15 OK, 37 OK cross-module, CI 3.10+3.12, AIS code-review ready to merge) * scope discipline (§66/§59/§63/§65/gpt-4o default untouched) * unblocked: Issue #450 ready to solve. No code changes in this commit; closure-only.
#466) Update the `README.md` Verzeichnisstruktur block to reflect the current repo layout (Issue #450). What changed: - Removed stale `docs/docs/BACKLOG/open.md` → correctly `docs/BACKLOG/open.md` (now under a BACKLOG/ subdirectory block with open.md and done.md). - Added missing directories: `workers/`, `prompts/`, `benchmarks/`. - Expanded `scripts/` to include current model/benchmark/runtime-relevant files (`model_catalog.py`, `benchmark_free_models.py`, `opencode_state_diagnostic.py`, `verify_openrouter_slugs.py`, `solver_reporting.py`, `watchdog.py`, `validation_run.py`, `workflow_congestion.py`, `review_pr.py`, plus the rest of the live scripts). - Expanded `docs/` to reference current core docs (`AGENTS.md`, `OPENCODE_APP_STATE.md`, `MODEL_OVERRIDE_POLICY.md`, `PLANNING_0.9.0.md`, `PRODUCT_VISION_1.0.md`, `ROADMAP.md`, `label_taxonomy.md`). - Updated `tests/` block to reference the representative current test modules (`test_benchmark_free_models.py`, `test_model_catalog.py`, `test_solve_issues*.py`, `test_opencode_state_diagnostic.py`, `test_reviewer_runtime.py`). - Added inline hints to `reports/` subpaths (`runs/`, `benchmarks/`, `preserved-worktrees/`) since those are produced by every solver run. - Removed the duplicate `---` separator before `## Verzeichnisstruktur`. Scope: README-only, no code/benchmark/backlog changes. Acceptance criteria (all met): - README no longer contains `docs/docs/BACKLOG/open.md`. - Tree includes all current major directories (`.github/`, `.agents/`, `config/`, `scripts/`, `workers/`, `prompts/`, `templates/`, `reports/`, `docs/`, `tests/`, `benchmarks/`). - `scripts/` includes the current model/benchmark/runtime-relevant files. - `workers/` listed with adapter purpose. - `docs/` references current core docs. - Redundant separator removed. - `git diff --check` passes. Closes #450.
Closure entry for Issue #450 (README Verzeichnisstruktur update). Documents: - root cause (README tree had drifted from the repo layout) - fix shape (README-only, +54/-31, single Mavis-as-dev commit) - new directories added (workers/, prompts/, benchmarks/) - expanded scripts/ and docs/ blocks - redundant --- separator removed - pre-merge live-review caught two path bugs the AIS documentation review missed (PLANNING_0.10.0_DESIGN_BRIEF.md untracked, test_review_pr.py → test_reviewer_runtime.py) - acceptance criteria met - scope discipline (README-only per User directive) Original labels: kind/feature, theme/documentation, area/readme, priority/2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release 0.9.0 close-out PR. This brings
mainup to the currentdevelopstate and includes the final release documentation update.Close-out commit on top of develop:
CHANGELOG.mdwith the late 0.9.0 solver-loop, model-catalog, benchmark-methodology, and README close-out itemsdocs/CURRENT_CONTEXT.mdfrom in-progress 0.9.0 validation to post-0.9.0 statusdocs/ROADMAP.mdfrom 0.9.0 planned to 0.9.0 donePLANNING_0.9.0.mdas completed in the README treeValidation requested before merge
scripts/review_pr.pyv0.9.0tag currently points to the earlier milestone-sync commit and is not moved by this PRNotes
developis fast-forwardable from currentmainbased on the local merge-base check before creating this PR.docs/PLANNING_0.10.0_DESIGN_BRIEF.mdwas intentionally not staged or committed.