Skip to content

Add model-in-loop long-horizon benchmark - #127

Merged
SonAIengine merged 1 commit into
mainfrom
codex/model-long-horizon-0.43
Aug 16, 2026
Merged

Add model-in-loop long-horizon benchmark#127
SonAIengine merged 1 commit into
mainfrom
codex/model-long-horizon-0.43

Conversation

@SonAIengine

Copy link
Copy Markdown
Owner

Summary

  • add a real model-in-the-loop gate above the sealed 1,000-tool Arazzo 3/10/30-step benchmark
  • separate raw LLM target accuracy from selector-guard and end-to-end goal completion
  • add explicit DeepSeek provider options, credential-safe endpoint redaction, Make target, tests, and public result artifact

Actual model result

DeepSeek V4 Flash, 3 repeats across 3/10/30-step scenarios (9 executions):

  • retrieval target hit@8: 1.00
  • raw LLM target exact: 1.00
  • final selected target exact: 1.00
  • exact plan/execution order: 1.00 / 1.00
  • binding accuracy: 1.00
  • goal completion: 1.00

The model selects only the final target from the retrieved catalog. Arazzo supplies prerequisite order and bindings; the benchmark does not claim independent 30-call enumeration by the model.

Verification

  • poetry run ruff check .
  • poetry run ruff format --check .
  • poetry run pytest tests/ -q (1244 passed, 47 skipped)
  • DeepSeek V4 Flash 1,000-tool 3x model gate: pass
  • saved artifact secret scan: pass

@SonAIengine
SonAIengine merged commit 0a169a2 into main Aug 16, 2026
7 checks passed
@SonAIengine
SonAIengine deleted the codex/model-long-horizon-0.43 branch August 16, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant