Autoresearch/scoring mar27#4
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (22)
📝 WalkthroughWalkthroughAdded comprehensive autoresearch framework including fixture-based evaluation for scoring and routing tracks, manifest configuration, evaluator scripts, a git-integrated runner for automated iteration loops, performance benchmarking with Puppeteer, and supporting documentation and tests. Changes
Sequence Diagram(s)sequenceDiagram
participant Runner as Autoresearch Runner
participant Git as Git Operations
participant Evaluator as Track Evaluator
participant Logger as Result Logger
participant TSV as results.tsv
Runner->>Runner: Parse --track argument
Runner->>Git: Fetch branch name & changed files
Git-->>Runner: Branch, changed files
Runner->>Runner: Validate files in track allowlist
Runner->>Git: Stage changed files
Runner->>Git: Commit with track-prefixed message
Git-->>Runner: Commit hash
Runner->>Logger: Read previous best metric
Logger->>TSV: Query best "keep" metric
TSV-->>Logger: Best metric (if exists)
Logger-->>Runner: Previous best
Runner->>Evaluator: Evaluate fixture set
Evaluator-->>Runner: Report with metric & failures
Runner->>Runner: Compare metric vs previous best
alt Metric improved or no previous best
Runner->>Logger: Append "keep" result row
Logger->>TSV: Write result
Runner->>Logger: Write JSON report
else Metric did not improve
Runner->>Git: Reset HEAD to parent
Runner->>Logger: Append "discard" result row
Logger->>TSV: Write result
Runner->>Logger: Write JSON report
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Release Notes
New Features
scoringandroutingtracks with constrained file edits and automated evaluation.Documentation
Chores