Skip to content

[codex] add isolated judge QA execution - #65

Draft
greenthree wants to merge 2 commits into
mainfrom
codex/judge-qa-execution
Draft

[codex] add isolated judge QA execution#65
greenthree wants to merge 2 commits into
mainfrom
codex/judge-qa-execution

Conversation

@greenthree

Copy link
Copy Markdown
Owner

What changed

  • add the lazy top-level judge_qa_problem(...) Core API and an isolated Judge QA runtime
  • run Validator, Checker/Interactor fixtures, controlled contestants, and automatic robustness probes from a byte-consistent snapshot under the existing per-problem Judge lock
  • enforce an overall deadline, default cancel-file and caller cancellation callbacks, live source/data/fixture hash fences, bounded structured results, and full process-tree cleanup
  • atomically publish judge-qa-evidence-v1.json only after a complete successful run; failures, cancellation, timeout, input drift, lock contention, cleanup failure, or publication failure preserve the previous evidence
  • share Validator policy and cancellation primitives with the formal Judge, and keep Interactor resource-failure attribution stable during simultaneous exits
  • require official Validator/Checker/Interactor programs to remain reproducible C++ sources, matching the formal Judge and DOMjudge package path; controlled Interactor contestants may use C++ or Python

Why

The P0 schema can describe adversarial Checker and Interactor fixtures, but it could not execute them. This P1 slice provides the isolated execution and evidence layer needed to actively test each problem's special Judge without mutating formal PDF, ZIP, metadata, Manifest, source, or data artifacts.

This PR intentionally does not add the judge-qa CLI, seal integration, lint/status/report evidence state, Skill/reference updates, WebUI, or Legacy support. Those remain the P2/P3 work.

Validation

  • targeted Python/unit/integration suite: 154 passed, 13 skipped
  • npm run check: 643 passed, 30 skipped
  • npm run pack:check: probhub=217, probhub-skill=5
  • Skill quick validation passed
  • git diff --check passed
  • local phtest nine-problem lint passed
  • local phtest L02 judge --no-cache passed: 33 inputs, 3 solutions, 99 case runs, all_expectations_met
  • final same-model read-only review found no blocking issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant