Skip to content

Add preflight validation and cost estimation#64

Merged
gkamradt merged 1 commit into
mainfrom
feature/preflight-validation
Jan 15, 2026
Merged

Add preflight validation and cost estimation#64
gkamradt merged 1 commit into
mainfrom
feature/preflight-validation

Conversation

@ericc59
Copy link
Copy Markdown
Contributor

@ericc59 ericc59 commented Jan 15, 2026

Summary

  • Validates config, API keys, data files, and output directory before batch runs
  • Estimates costs upfront to prevent surprise bills
  • Integrates into run_all.py with --skip-preflight and --cost-limit flags

Usage

# Standalone
python -m arc_agi_benchmarking.utils.preflight --config gpt-4o-2024-11-20 --data_dir data/evaluation

# With cost limit
python cli/run_all.py --config o1-high --cost-limit 50.00

Test plan

  • 22 unit tests added and passing
  • Tested against real evaluation dataset (120 tasks)

- New preflight module validates config, API keys, data files, and output dir
- Estimates costs before running expensive batch operations
- Integrates into run_all.py with --skip-preflight and --cost-limit flags
- Standalone CLI: python -m arc_agi_benchmarking.utils.preflight
- Includes 22 unit tests for comprehensive coverage
@ericc59 ericc59 force-pushed the feature/preflight-validation branch from 5a5e8f6 to 41a7067 Compare January 15, 2026 17:37
@gkamradt gkamradt merged commit c6b9a72 into main Jan 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants