Migrated from spboyer/waza#390
Summary
Design how custom graders beyond config-driven types can be distributed via the registry.
Context
From waza-vs-openai-evals.md Row 15: OpenAI Evals allows Python class inheritance for custom eval logic. Waza is config-driven only (code, program graders). This is waza's #2 competitive gap.
Options
- WASM plugins — compile grader logic to WASM, distribute via registry. Sandboxed, cross-platform.
- External program protocol — any binary that accepts stdin JSON and outputs grader results. Already partially supported via
program grader.
- Go plugin interface —
plugin.Open() for native Go graders. Fast but platform-specific.
- Embedded scripting — extend
code grader beyond Python (JavaScript, Lua, Starlark).
Parent: #385
Summary
Design how custom graders beyond config-driven types can be distributed via the registry.
Context
From
waza-vs-openai-evals.mdRow 15: OpenAI Evals allows Python class inheritance for custom eval logic. Waza is config-driven only (code,programgraders). This is waza's #2 competitive gap.Options
programgrader.plugin.Open()for native Go graders. Fast but platform-specific.codegrader beyond Python (JavaScript, Lua, Starlark).Parent: #385