Skip to content

refactor(optimize-sorting): restructure benchmark flow with catastrophic gate - #1430

Closed
colehurwitz wants to merge 5 commits into
mainfrom
factory/run-9967b585
Closed

colehurwitz wants to merge 5 commits into
mainfrom
factory/run-9967b585

Conversation

@colehurwitz

@colehurwitz colehurwitz commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Changes

  • Added gate_catastrophic_t{1,2,3} GateNodes: New first-line gates after run_benchmark that only catch catastrophic accuracy drops (>10% below baseline mean). On HALT, skip confirmation runs and go straight to archive. On PROCEED, continue to confirm_benchmark.

  • Moved confirm_benchmark_t{1,2,3} before accuracy gate: Confirmation runs (2 additional benchmarks for 3-run average) now always execute when not catastrophic, rather than only on accuracy pass.

  • Added new gate_accuracy_t{1,2,3} GateNodes after confirm_benchmark: These check the averaged 3-run result. Output pass (PROCEED) or fail (HALT) only — no reloop since 3-run average is definitive.

  • Updated _accuracy_gate_command(): Handles averaged format by checking if br['accuracy'] is a dict and using br['accuracy']['mean']. Removed catastrophic check (now handled by dedicated gate). Removed reloop output.

  • Updated _per_unit_gate_command(): Handles averaged per_unit_accuracy format — checks br['per_unit_accuracy'][uid]['mean'] when value is a dict.

  • Updated _archive_prompt(): Describes new flow semantics — catastrophic gate halt produces single-run format, accuracy gate produces averaged format.

  • Edge rewiring: New flow per tier:

    • run_benchmark → gate_catastrophic --PROCEED→ confirm_benchmark → gate_accuracy --PROCEED→ archive
    • gate_catastrophic --HALT→ archive (catastrophic failure, skip extra runs)
    • gate_accuracy --HALT→ archive (revert based on averaged result)
    • Tier 3: gate_accuracy --PROCEED→ gate_per_unit_accuracy → archive
  • Graph size: 31 nodes, 39 edges (was 28 nodes, 36 edges)


Test & prompt fixes (Closes #1432)

  • Test node sets: Added gate_catastrophic_t1/t2/t3 to tier node sets; total 28→31, per-tier counts +1 each
  • Test edge count: 36→39
  • Test edge topology: Updated all assertions to match new flow (run_benchmark → gate_catastrophic → confirm → gate_accuracy → archive)
  • Test reloop count: 5→2 (accuracy gate reloops removed; only config gate + per-unit gate remain)
  • New test: test_catastrophic_gates_are_gate_nodes
  • Builder T1 prompt: Removed stale gate_accuracy_t1 reloop reference
  • Builder T2 prompt: Removed stale gate_accuracy_t2 reloop handling section

Implement the three-tier optimize-sorting workflow for iterative speed
optimization of spike sorting pipelines with hard accuracy constraints.

New files:
- factory/workflow/contributed/optimize_sorting/__init__.py
- factory/workflow/contributed/optimize_sorting/workflow.py (25 nodes,
  33 edges, 10 unique prompt templates, 7 helper functions)
- factory/workflow/contributed/optimize_sorting/test_workflow.py (82 tests)

Modified files:
- factory/cli/_helpers.py: add optimize-sorting to CEO_MODES
- factory/cli/_task_builder.py: add mode suffix, tier-keyword focus suppression
- factory/cli/_ceo_helpers.py: add to focus allowlist, write focus.txt,
  skip add_backlog_item for tier-prefixed focus
- factory/workflow/skill_export.py: add WORKFLOW_META entry
- factory/workflow/definitions.py: register in _get_builtin_registry

File-Size Gate note: workflow.py is 835 lines, exceeding the 500-line
limit. This is a self-contained workflow definition with inline python3 -c
commands and 10+ prompt templates that must be co-located per the
contributed workflow pattern. Splitting would break the single-file
pattern used by all other contributed workflows.
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 629 total, 615 kept, 14 dropped (ext:14, meta:0, big:0)
[build_project_map] 615 files, 109 unique dirs, 101 cache misses, 5.7ms
[resolve] 1165 resolved, 1572 unresolved (of 2737 total specs)
[resolve_imports] project_map 5.8ms, suffix_idx 1.3ms, suffix_resolve 19.2ms, total 26.3ms
[build_graphs] 615 files | maps 2.8ms, imports 26.6ms, calls+inherit 7.2ms, total 36.5ms | 1164 import, 8401 call, 1 inherit edges
sentrux check — 3 rules checked

Quality: 4450

✗ [Error] max_cc: 6 function(s) exceed max cyclomatic complexity of 30
    factory/cli/_ceo_helpers.py:_execute_ceo (cc=46)
    factory/cli/_ceo_helpers.py:_validate_ceo_flags (cc=43)
    examples/chess_evolve.py:main (cc=33)
    factory/cli/_task_builder.py:_build_ceo_task (cc=32)
    factory/cli/run.py:cmd_run (cc=32)
    factory/outer_loop/mutations.py:validate_and_repair (cc=31)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 629 total, 615 kept, 14 dropped (ext:14, meta:0, big:0)
[build_project_map] 615 files, 109 unique dirs, 101 cache misses, 5.9ms
[resolve] 1165 resolved, 1572 unresolved (of 2737 total specs)
[resolve_imports] project_map 6.0ms, suffix_idx 1.3ms, suffix_resolve 19.5ms, total 26.7ms
[build_graphs] 615 files | maps 2.5ms, imports 27.0ms, calls+inherit 6.4ms, total 35.9ms | 1164 import, 8401 call, 1 inherit edges
sentrux gate — structural regression check

Quality:      4447 -> 4450
Coupling:     0.80 → 0.80
Cycles:       4 → 4
God files:    3 → 3

Distance from Main Sequence: 0.35

✗ DEGRADED
  ✗ Complex functions increased: 68 → 69

Add 'git checkout main' at the start of the select_tier FnNode command
so each experiment starts from a clean main branch state. This ensures
experiments are measured independently against the pinned baseline.

The baseline.json lock_baseline step remains idempotent (skips if
baseline already exists), preserving main branch performance as the
fixed reference point. Experiment branches are created downstream
by the builder agents.
@colehurwitz colehurwitz changed the title feat: add optimize-sorting contributed workflow feat: add git checkout main step to optimize-sorting select_tier Sep 1, 2026
@colehurwitz colehurwitz changed the title feat: add git checkout main step to optimize-sorting select_tier feat: add benchmark score PR comment and git checkout main to optimize-sorting workflow Sep 1, 2026
…cy gate passes

After the accuracy gate passes (verdict=keep), run 2 additional benchmark
runs (total 3) and compute mean/std for accuracy and speed_seconds before
archiving. This gives statistical confidence that the improvement is real.

Changes:
- Add _benchmark_3x_command() helper: reads existing benchmark-result.json
  (run 1), runs 2 more benchmarks, averages all 3, and overwrites
  benchmark-result.json with mean/std fields (same format as baseline.json)
- Add 3 new FnNodes: confirm_benchmark_t1, confirm_benchmark_t2,
  confirm_benchmark_t3
- Wire: accuracy_gate --PROCEED--> confirm_benchmark --> archive_result
  (HALT path unchanged — goes directly to archive for revert recording)
- For T3: gate_per_unit_accuracy --PROCEED--> confirm_benchmark_t3
- Update _archive_prompt() to handle averaged vs single-run result formats
- Node count: 25 → 28, Edge count: 33 → 36

File-size note: workflow.py (957 lines) exceeds 500-line gate but was
already 857 lines. Splitting a single workflow definition would fragment
the node/edge graph and harm readability.

Closes #1431
@colehurwitz colehurwitz changed the title feat: add benchmark score PR comment and git checkout main to optimize-sorting workflow optimize-sorting: add 3-run confirmation benchmark after accuracy gate passes Sep 1, 2026
…hic gate

Restructure the three-tier benchmark flow from:
  run_benchmark → accuracy_gate (1 run) → confirm_benchmark → archive
to:
  run_benchmark → catastrophic_gate → confirm_benchmark → accuracy_gate → archive

Changes:
- Add gate_catastrophic_t{1,2,3} GateNodes that only catch >10% accuracy
  drops (acc < baseline_mean * 0.9). HALT → archive, PROCEED → confirm.
- Move confirm_benchmark_t{1,2,3} BEFORE accuracy gate so 3 runs always
  execute when not catastrophic.
- Add new gate_accuracy_t{1,2,3} GateNodes AFTER confirm_benchmark that
  check the averaged 3-run result. Output pass/fail only (no reloop since
  3-run average is definitive).
- Update _accuracy_gate_command() to handle averaged format: check if
  br['accuracy'] is a dict and use br['accuracy']['mean'].
- Update _per_unit_gate_command() to handle averaged per_unit_accuracy
  format: check br['per_unit_accuracy'][uid]['mean'] vs baseline.
- Update _archive_prompt() to describe new flow semantics (catastrophic
  gate halt = single-run format, accuracy gate = averaged format).
- Rewire edges: 31 nodes, 39 edges (was 28 nodes, 36 edges).
@colehurwitz colehurwitz changed the title optimize-sorting: add 3-run confirmation benchmark after accuracy gate passes refactor(optimize-sorting): restructure benchmark flow with catastrophic gate Sep 1, 2026
… topology

- Update test node counts: 28→31 (add gate_catastrophic_t1/t2/t3)
- Update test edge count: 36→39
- Update test reloop count: 5→2 (accuracy gate reloops removed)
- Fix edge assertions: run_benchmark→gate_catastrophic→confirm→gate_accuracy→archive
- Add test for catastrophic gate node types
- Remove stale gate_accuracy_t1 reloop reference from builder T1 prompt
- Remove stale gate_accuracy_t2 reloop reference from builder T2 prompt

Closes #1432
@colehurwitz

Copy link
Copy Markdown
Collaborator Author

Superseded by unified PR — see #1443

@colehurwitz colehurwitz closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(optimize-sorting): update tests and remove stale prompts after catastrophic gate restructuring

1 participant