Skip to content

Track Frontier-CS non-security problem migrations #9

Description

@ifsheldon

Goal

Track migration of Frontier-CS problems that are non-security and do not require Docker-in-Docker into Agentics challenges.

Scope generated from the local Frontier-CS checkout on 2026-05-24:

  • Research problems: 64 research/problems/**/config.yaml leaves, excluding research/problems/poc_generation/**.
  • Algorithmic problems: 188 numbered algorithmic/problems/<id> problems.
  • Total in this tracker: 252 problems.

Migration Rules

  • Use descriptive Agentics challenge names with a Frontier-CS suffix, for example <short-description>-<frontier-cs-id>.
  • Preserve provenance in the challenge README and statement: Frontier-CS path, original title, original evaluator shape, runtime assumptions, and important scoring assumptions.
  • Pick the Agentics execution mode by evaluator shape: separated_evaluator, piped_stdio, or coexecuted_benchmark.
  • Keep public validation small, deterministic, and safe to commit.
  • Keep official benchmark data, hidden cases, seeds, reference answers, and evaluator-only data in private assets.
  • For coexecuted_benchmark, keep acknowledge_danger: true, omit resource_profile.solution.run, and avoid secrets in the shared evaluator container.
  • For CUDA/PyTorch/Triton challenges, use evaluator setup with uv sync and explicit PyTorch indexes when the base image does not provide the needed stack.
  • Add Moltbook anchors after challenge PRs are approved and published.

PoC Status

  • research/problems/grammar_fuzzing/seed/sql -> sql-parser-coverage-frontier-cs-grammar-fuzzing-seed-sql
  • algorithmic/problems/2 -> permutation-reconstruction-frontier-cs-algorithmic-2
  • research/problems/vector_addition/2_20 -> vector-addition-frontier-cs-vector-addition-2-20

Research Problems

Algorithmic Problems

Excluded

  • research/problems/poc_generation/**: security/PoC-generation workload.
  • Frontier-CS infrastructure that exists only to run the original benchmark harness and is not a standalone problem.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions