Skip to content

Add submission validation script#683

Open
mvanhorn wants to merge 1 commit intoopenai:mainfrom
mvanhorn:osc/feat-submission-validator
Open

Add submission validation script#683
mvanhorn wants to merge 1 commit intoopenai:mainfrom
mvanhorn:osc/feat-submission-validator

Conversation

@mvanhorn
Copy link

Adds validate_submission.py - a standalone pre-submission checker that catches the most common issues from #677 before participants open a PR.

Usage:

python3 validate_submission.py records/track_10min_16mb/2026-03-24_MySubmission/

What it checks:

  • Folder is under records/track_10min_16mb/ or records/track_non_record_16mb/
  • Required files present: README.md, submission.json, train_gpt*.py, and at least one train log
  • submission.json has required fields (val_bpb/mean_val_bpb, name/run_name, date) with correct types
  • Training script has valid Python syntax
  • Total folder size reported (warns near 16 MB cap)

Example output (current SOTA):

Validating: records/track_10min_16mb/2026-03-23_LeakyReLU_LegalTTT_ParallelMuon

  [PASS] Folder location: Track: track_10min_16mb
  [PASS] Required files: README.md, submission.json, train_gpt.py
  [PASS] Train log: Found: train_seed1337.log, train_seed2025.log, train_seed42.log
  [PASS] submission.json: OK
  [PASS] Training script syntax: OK (train_gpt.py)
  [PASS] Size check: Total size 0.14 MB

Result: PASS (6 passed)

Tested against all 25 existing records in the repo. 21 pass clean, 4 flag known issues (genuinely missing files or fields in those specific submissions).

Uses only Python stdlib - no extra dependencies. 263 lines.

This contribution was developed with AI assistance (Claude Code).

Pre-submission checker that catches common issues from openai#677:
missing required files, malformed submission.json, syntax errors
in training scripts, and artifact size violations.

Runs against all 25 existing records with zero false positives
on well-formed submissions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant