Skip to content

feat: custom-checker#13

Merged
rayokamoto merged 4 commits intomainfrom
custom-checker
Apr 28, 2026
Merged

feat: custom-checker#13
rayokamoto merged 4 commits intomainfrom
custom-checker

Conversation

@xxori
Copy link
Copy Markdown
Collaborator

@xxori xxori commented Apr 17, 2026

Add custom checker python support for aucpl problem test. It will check for checker.py files next to problem.md, and execute them instead of just doing string comparison if they exist. Here is an example of a valid checker.py:

def check(process_output, judge_output):
    return abs(float(process_output) - float(judge_output)) <= 0.5

Limitation: Often you will import dmoj things in the checkers, which doesn't work here. The functions can only be raw Python scripts. Also, you can only use the process output and judge output, although DMOJ passes in much more information to their custom judges. https://docs.dmoj.ca/#/problem_format/custom_checkers?id=custom-checkers-1

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for DMOJ-style custom Python checkers to aucpl problem test, allowing problems to define a checker.py that determines pass/fail instead of strict output byte comparison.

Changes:

  • Add embedded Python runner script and Rust glue code to execute checker.py per test case and parse a boolean result.
  • Detect checker.py in the problem directory and switch the test verdict mechanism accordingly.
  • Document checker.py usage and expected function signature in README.md.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
crates/cli/src/problem/test.rs Adds checker.py discovery and per-test execution via python3, wiring outputs into a check() call.
README.md Documents custom checker behavior and shows folder structure including checker.py.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/cli/src/problem/test.rs Outdated
Comment thread crates/cli/src/problem/test.rs Outdated
Comment thread crates/cli/src/problem/test.rs
Comment thread crates/cli/src/problem/test.rs Outdated
Comment thread crates/cli/src/problem/test.rs Outdated
Comment thread crates/cli/src/problem/test.rs Outdated
Comment thread crates/cli/src/problem/test.rs
@rayokamoto rayokamoto merged commit 7a0f27d into main Apr 28, 2026
1 check passed
@rayokamoto rayokamoto deleted the custom-checker branch April 28, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants