Standard evals#55
Merged
Merged
Conversation
…nd write objects. added eval pipeline schemas and implementation.
- Fix stage0: task_obj → task, capability.name → capability_name, domain.name → domain_name, ts.task → ts.task_statement - Fix stage1/stage2: Add proper generic type hints (Dict[str, Any]) - Fix eval_schemas.py: Correct stage numbers in docstrings (Stage 1/3 → Stage 0/2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Type
Fix and Feature
Short Description
Add evaluation pipeline with standardized schemas for running Inspect AI evaluations on generated tasks. Also moves legacy LBO code to a separate directory.
1. Evaluation Schemas
src/schemas/eval_schemas.py- Schema definitions for eval pipelinesrc/schemas/eval_io_utils.py- Save/load utilities for eval datasrc/schemas/__init__.py- Updated exportssrc/schemas/EVALUATION_PIPELINE_SCHEMAS.md- Documentation2. Evaluation Pipeline (New Feature)
Core implementation of the 3-stage evaluation pipeline:
src/eval_stages/__init__.py- Module exportssrc/eval_stages/stage0_setup_and_dataset.py- Setup & dataset preparation (no LLM calls)src/eval_stages/stage1_eval_execution.py- Run Inspect AI evaluationssrc/eval_stages/stage2_score_aggregation.py- Aggregate scores from resultssrc/eval_stages/prompts.py- Default prompt template for evaluation pipelinesrc/run_eval_pipeline.py- Entry point for running the pipeline3. Legacy Code Migration
Moved LBO code to
legacy/directory:src/lbo.py→legacy/src/lbo.pysrc/run_lbo.py→legacy/src/run_lbo.pysrc/utils/lbo_utils.py→legacy/src/utils/lbo_utils.pytests/src/test_lbo*.py→legacy/tests/4. Configuration Updates
src/cfg/run_cfg.yaml- Added eval config section, simplified generation configpyproject.toml- Exclude legacy tests from pytestTests Added
None