diff --git a/README.md b/README.md index 8f686cc..99f85c4 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,43 @@ GitHub PR URL --- +## Why flow-guided review? + +Traditional code review tools present diffs as a flat file list — alphabetical, with no context about how changes relate to each other. Reviewers mentally reconstruct the code flow: "this handler calls that validator which uses this new utility..." and hope they don't miss a connection. + +Flow-guided review structures the same diff as a directed graph: entry points first, then downstream through call chains, with explicit dependency ordering, risk levels, and clusters of tightly coupled changes. The reviewer — human or AI — follows the code flow instead of guessing at it. + +### 100-PR evaluation + +We evaluated this approach across **100 open-source PRs from 57 repositories** (8 languages, 2-86 files per PR) using a blind 3-agent framework: + +1. **Baseline reviewer** — reviews the diff with no structural guidance (standard approach) +2. **Flow-guided reviewer** — reviews the same diff with the PR Flow Graph review plan +3. **Blind judge** — scores both reviews on 5 criteria without knowing which used the graph + +| Metric | Value | +|--------|-------| +| Flow-guided wins | **92 / 100** (92%) | +| Baseline wins | 0 / 100 (0%) | +| Ties | 8 / 100 (8%) | +| Avg improvement | **+1.3** (6.0 → 7.3 on 10-point scale) | + +### Per-criterion results + +| Criterion | Baseline | Flow-Guided | Delta | What it measures | +|-----------|----------|-------------|-------|------------------| +| Completeness | 6.7 | 7.7 | +1.0 | Covered all meaningful changes? | +| Flow Awareness | 3.9 | 7.0 | **+3.1** | Understood cross-file connections? | +| Risk Identification | 6.3 | 7.6 | +1.3 | Flagged the riskiest parts? | +| Actionability | 6.2 | 7.1 | +0.9 | Specific, useful comments? | +| Efficiency | 7.1 | 7.0 | -0.1 | Avoided noise / false positives? | + +The largest gain is **flow awareness** (+3.1) — understanding how changes in one file affect behavior in another. This is what the review plan directly provides. Efficiency stays flat, meaning the structured approach doesn't add noise. + +Full results: [`evals/RESULTS.md`](./evals/RESULTS.md) | Methodology: [`evals/README.md`](./evals/README.md) + +--- + ## Quick start ### Prerequisites diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..ca9ea65 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,162 @@ +# PR Flow Graph — Evaluation Framework + +Compares code reviews produced **with** vs **without** the PR Flow Graph review plan across 100 open-source PRs from 57 repositories. + +## 3-Agent Evaluation Framework + +Each PR is evaluated by three independent Claude agents in a blind comparison: + +``` + GitHub PR + / \ + v v + ┌─────────────┐ ┌──────────────────┐ + │ Agent A │ │ Agent B │ + │ (Baseline) │ │ (Flow-Guided) │ + │ │ │ │ + │ Input: │ │ Input: │ + │ - PR diff │ │ - PR diff │ + │ │ │ - Review plan │ + │ │ │ from /api/ │ + │ │ │ agent/ │ + │ │ │ review-plan │ + └──────┬───────┘ └────────┬──────────┘ + │ │ + v v + ┌────────────────────────────────┐ + │ Judge (Blind) │ + │ │ + │ Sees: "Review 1" / "Review 2" │ + │ (randomized order) │ + │ Scores each 1-10 on 5 criteria│ + │ Picks a winner │ + └────────────────────────────────┘ +``` + +### Agent A — Baseline Reviewer + +Reviews the PR using only the raw GitHub diff. This is how most AI code review tools work today: the model sees a flat list of file diffs and produces comments. + +### Agent B — Flow-Guided Reviewer + +Reviews the same PR diff but also receives the structured review plan from `/api/agent/review-plan`. The plan provides: + +- **Topological review order** — review callees before callers +- **Node roles** — entry points, internal functions, leaf functions, context-only (unchanged but referenced) +- **Risk levels** — high/medium/low with reasons (large diff, many callers, entry point) +- **Clusters** — tightly coupled groups of functions to review together +- **Dependency chains** — "review X before Y because Y calls X" + +### Judge — Blind Evaluator + +The judge receives both reviews labeled only as "Review 1" and "Review 2" in **randomized order** (coin flip per PR). It does not know which used the flow graph. It scores each review on 5 criteria and declares a winner. + +## Scoring Criteria + +Each criterion is scored 1-10 independently. + +### Completeness (1-10) + +Did the review cover all meaningful changes in the PR? + +A high-scoring review identifies and comments on every significant code change: new functions, modified logic, deleted code, configuration changes, and test coverage. A low score means the reviewer missed entire files, skipped important logic paths, or ignored edge cases. For a 14-file PR, a review that only covers 5 files would score low regardless of how good those 5 comments are. + +### Flow Awareness (1-10) + +Did the review understand how changes connect across files? + +This is the core differentiator. A high score means the reviewer recognized cross-file relationships: how a change in `handler.ts` affects `validator.ts` which is called by `middleware.ts`. It caught consistency issues between caller and callee, identified that a type change in one file breaks assumptions in another, or traced data flow through the call chain. A low score means the review treated each file in isolation, as if they were independent changes. + +### Risk Identification (1-10) + +Did the review flag the riskiest parts of the PR? + +High-risk areas include: entry points with many downstream callers (a bug here cascades), large diffs touching shared state, breaking API changes, missing error handling on new code paths, and security-sensitive changes. A high-scoring review correctly prioritizes these over low-risk cosmetic changes. A low score means the reviewer spent equal time on all changes regardless of impact, or missed the highest-risk modifications entirely. + +### Actionability (1-10) + +Were the review comments specific and useful? + +A high score means comments pointed to exact lines of code, explained *why* something is a problem (not just *that* it is), and suggested concrete fixes or alternatives. Comments like "the null check on line 45 should handle the empty-array case too — `if (!items?.length)`" score high. Comments like "this could be better" or "consider error handling" without specifics score low. + +### Efficiency (1-10) + +Did the review avoid noise and false positives? + +A high score means every comment adds value — no redundant observations, no low-signal nits masquerading as major issues, no incorrect severity ratings, and no comments on code that wasn't actually changed. A review that raises 5 precise issues scores higher on efficiency than one that raises 12 comments where 7 are trivial or wrong. This criterion counterbalances completeness: you shouldn't score well by just commenting on everything. + +### Overall Score + +The arithmetic mean of all 5 criteria. Range: 1.0-10.0. + +## File Format + +Each eval produces `evals/____.json`: + +```json +{ + "pr": { + "url": "https://github.com/owner/repo/pull/123", + "owner": "owner", + "repo": "repo", + "number": 123, + "title": "PR title", + "files_changed": 14, + "additions": 200, + "deletions": 8, + "language": "typescript" + }, + "timestamp": "2026-03-30T...", + "baseline_review": { + "comments": [ + { + "file": "path/to/file.ts", + "line": 42, + "severity": "critical|major|minor|nit|positive", + "comment": "Specific review comment..." + } + ], + "summary": "2-3 sentence assessment" + }, + "flow_guided_review": { + "comments": [...], + "summary": "..." + }, + "review_plan": { "totalSteps": 12, "..." : "..." }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + }, + "reasoning": "1-2 sentence explanation of the winner selection", + "winner": "flow_guided" + } +} +``` + +## Running the Eval + +The eval runner (`run-eval.ts`) processes a single PR through the 3-agent pipeline: + +```bash +# Expects pre-fetched data in /tmp/prflow-evals/ and PR list in /tmp/prflow-eval-prs.json +npx tsx evals/run-eval.ts +``` + +Each run makes 3 API calls (baseline, flow-guided, judge) and writes the result JSON to `evals/`. + +## Results + +See [RESULTS.md](./RESULTS.md) for the full aggregated results across all 100 PRs. diff --git a/evals/RESULTS.md b/evals/RESULTS.md new file mode 100644 index 0000000..4fe8664 --- /dev/null +++ b/evals/RESULTS.md @@ -0,0 +1,156 @@ +# Evaluation Results + +Comparing code reviews **with** vs **without** PR Flow Graph review plan across 100 open-source PRs from 57 repositories. + +## Methodology + +Three independent agents per PR: +1. **Baseline reviewer** — reviews the diff cold, no review plan +2. **Flow-guided reviewer** — reviews the diff with the PR Flow Graph review plan (topological order, risk levels, clusters, dependencies) +3. **Blind judge** — scores both reviews on 5 criteria (1-10 each), picks a winner. Review order is randomized to prevent position bias. + +All agents use the same model (Claude). Judge does not know which review used the flow graph. + +## Aggregate Results (n=100) + +| Metric | Value | +|--------|-------| +| Flow-Guided wins | **92** (92%) | +| Baseline wins | 0 (0%) | +| Ties | 8 (8%) | +| Avg baseline score | 6.0 | +| Avg flow-guided score | 7.3 | +| **Avg improvement** | **+1.3** | + +## Per-Criterion Breakdown + +| Criterion | Baseline | Flow-Guided | Delta | +|-----------|----------|-------------|-------| +| Completeness | 6.7 | 7.7 | +1.0 | +| Flow Awareness | 3.9 | 7.0 | +3.1 | +| Risk Identification | 6.3 | 7.6 | +1.3 | +| Actionability | 6.2 | 7.1 | +0.9 | +| Efficiency | 7.1 | 7.0 | -0.1 | +| **Overall** | **6.0** | **7.3** | **+1.3** | + +**Key insight:** The largest improvement is in **flow awareness** (+3.1), which measures understanding of cross-file connections and call chains. This is the core value proposition of the review plan. Efficiency is roughly equal, meaning the flow-guided approach doesn't add noise. + +## By Language + +| Language | PRs | Baseline | Flow-Guided | Flow Wins | Base Wins | Ties | Win Rate | +|----------|-----|----------|-------------|-----------|-----------|------|----------| +| Go | 19 | 6.2 | 7.1 | 18 | 0 | 1 | 95% | +| Java/Scala | 6 | 5.9 | 6.7 | 5 | 0 | 1 | 83% | +| Python | 19 | 5.9 | 7.6 | 19 | 0 | 0 | 100% | +| Rust | 7 | 6.2 | 6.8 | 5 | 0 | 2 | 71% | +| Rust/TS | 2 | 6.1 | 7.0 | 2 | 0 | 0 | 100% | +| TypeScript/JS | 47 | 6.0 | 7.4 | 43 | 0 | 4 | 91% | + +## All PRs + +| # | PR | Language | Files | Baseline | Flow-Guided | Winner | +|---|------|----------|-------|----------|-------------|--------| +| 1 | [vercel/next.js#92029](https://github.com/vercel/next.js/pull/92029) | TypeScript/JS | 6 | 5.2 | 7.2 | Flow | +| 2 | [vercel/next.js#92014](https://github.com/vercel/next.js/pull/92014) | TypeScript/JS | 3 | 6.6 | 6.8 | Flow | +| 3 | [vercel/next.js#92012](https://github.com/vercel/next.js/pull/92012) | TypeScript/JS | 11 | 6.2 | 8.2 | Flow | +| 4 | [facebook/react#36156](https://github.com/facebook/react/pull/36156) | TypeScript/JS | 10 | 5.8 | 7.8 | Flow | +| 5 | [facebook/react#36134](https://github.com/facebook/react/pull/36134) | TypeScript/JS | 3 | 5.8 | 7.8 | Flow | +| 6 | [facebook/react#36024](https://github.com/facebook/react/pull/36024) | TypeScript/JS | 3 | 6.2 | 7.8 | Flow | +| 7 | [angular/angular#67922](https://github.com/angular/angular/pull/67922) | TypeScript/JS | 3 | 6.0 | 7.8 | Flow | +| 8 | [angular/angular#67916](https://github.com/angular/angular/pull/67916) | TypeScript/JS | 3 | 6.2 | 8.0 | Flow | +| 9 | [sveltejs/svelte#18021](https://github.com/sveltejs/svelte/pull/18021) | TypeScript/JS | 10 | 5.8 | 7.6 | Flow | +| 10 | [sveltejs/svelte#18009](https://github.com/sveltejs/svelte/pull/18009) | TypeScript/JS | 7 | 6.4 | 7.8 | Flow | +| 11 | [prisma/prisma#29392](https://github.com/prisma/prisma/pull/29392) | TypeScript/JS | 3 | 6.0 | 8.0 | Flow | +| 12 | [prisma/prisma#29382](https://github.com/prisma/prisma/pull/29382) | TypeScript/JS | 4 | 5.2 | 8.0 | Flow | +| 13 | [trpc/trpc#7303](https://github.com/trpc/trpc/pull/7303) | TypeScript/JS | 5 | 5.4 | 6.4 | Flow | +| 14 | [trpc/trpc#7295](https://github.com/trpc/trpc/pull/7295) | TypeScript/JS | 24 | 6.4 | 6.4 | Tie | +| 15 | [trpc/trpc#7294](https://github.com/trpc/trpc/pull/7294) | TypeScript/JS | 24 | 5.4 | 5.8 | Flow | +| 16 | [remix-run/remix#11207](https://github.com/remix-run/remix/pull/11207) | TypeScript/JS | 9 | 6.0 | 8.2 | Flow | +| 17 | [remix-run/remix#11201](https://github.com/remix-run/remix/pull/11201) | TypeScript/JS | 6 | 6.6 | 8.0 | Flow | +| 18 | [remix-run/remix#11197](https://github.com/remix-run/remix/pull/11197) | TypeScript/JS | 22 | 5.4 | 8.0 | Flow | +| 19 | [payloadcms/payload#16092](https://github.com/payloadcms/payload/pull/16092) | TypeScript/JS | 7 | 6.0 | 7.6 | Flow | +| 20 | [payloadcms/payload#16058](https://github.com/payloadcms/payload/pull/16058) | TypeScript/JS | 11 | 5.8 | 7.6 | Flow | +| 21 | [payloadcms/payload#16047](https://github.com/payloadcms/payload/pull/16047) | TypeScript/JS | 6 | 6.2 | 8.6 | Flow | +| 22 | [webpack/webpack#20717](https://github.com/webpack/webpack/pull/20717) | TypeScript/JS | 12 | 5.8 | 7.8 | Flow | +| 23 | [webpack/webpack#20709](https://github.com/webpack/webpack/pull/20709) | TypeScript/JS | 10 | 5.8 | 6.6 | Flow | +| 24 | [babel/babel#17901](https://github.com/babel/babel/pull/17901) | TypeScript/JS | 5 | 5.8 | 7.8 | Flow | +| 25 | [babel/babel#17887](https://github.com/babel/babel/pull/17887) | TypeScript/JS | 7 | 6.2 | 7.4 | Flow | +| 26 | [nodejs/node#62453](https://github.com/nodejs/node/pull/62453) | TypeScript/JS | 5 | 5.8 | 8.0 | Flow | +| 27 | [eslint/eslint#20675](https://github.com/eslint/eslint/pull/20675) | TypeScript/JS | 3 | 5.8 | 6.4 | Flow | +| 28 | [prettier/prettier#18975](https://github.com/prettier/prettier/pull/18975) | TypeScript/JS | 4 | 6.0 | 7.4 | Flow | +| 29 | [oven-sh/bun#28651](https://github.com/oven-sh/bun/pull/28651) | TypeScript/JS | 3 | 5.2 | 7.0 | Flow | +| 30 | [oven-sh/bun#28633](https://github.com/oven-sh/bun/pull/28633) | TypeScript/JS | 5 | 7.0 | 8.0 | Flow | +| 31 | [oven-sh/bun#28617](https://github.com/oven-sh/bun/pull/28617) | TypeScript/JS | 8 | 6.0 | 7.8 | Flow | +| 32 | [vuejs/core#14628](https://github.com/vuejs/core/pull/14628) | TypeScript/JS | 3 | 6.2 | 8.0 | Flow | +| 33 | [jestjs/jest#15929](https://github.com/jestjs/jest/pull/15929) | TypeScript/JS | 10 | 6.4 | 6.2 | Tie | +| 34 | [shadcn-ui/ui#10202](https://github.com/shadcn-ui/ui/pull/10202) | TypeScript/JS | 4 | 6.2 | 8.2 | Flow | +| 35 | [shadcn-ui/ui#10189](https://github.com/shadcn-ui/ui/pull/10189) | TypeScript/JS | 5 | 6.2 | 6.2 | Tie | +| 36 | [pallets/flask#5945](https://github.com/pallets/flask/pull/5945) | Python | 5 | 6.6 | 7.4 | Flow | +| 37 | [pallets/flask#5928](https://github.com/pallets/flask/pull/5928) | Python | 10 | 5.4 | 7.8 | Flow | +| 38 | [pallets/flask#5917](https://github.com/pallets/flask/pull/5917) | Python | 7 | 5.6 | 8.0 | Flow | +| 39 | [pandas-dev/pandas#64912](https://github.com/pandas-dev/pandas/pull/64912) | Python | 3 | 6.0 | 7.6 | Flow | +| 40 | [pandas-dev/pandas#64901](https://github.com/pandas-dev/pandas/pull/64901) | Python | 3 | 6.6 | 7.6 | Flow | +| 41 | [langchain-ai/langchain#36348](https://github.com/langchain-ai/langchain/pull/36348) | Python | 4 | 5.6 | 7.4 | Flow | +| 42 | [langchain-ai/langchain#36347](https://github.com/langchain-ai/langchain/pull/36347) | Python | 3 | 6.0 | 7.8 | Flow | +| 43 | [pydantic/pydantic#12985](https://github.com/pydantic/pydantic/pull/12985) | Python | 6 | 5.4 | 6.0 | Flow | +| 44 | [python/cpython#146630](https://github.com/python/cpython/pull/146630) | Python | 5 | 6.0 | 7.6 | Flow | +| 45 | [python/cpython#146622](https://github.com/python/cpython/pull/146622) | Python | 3 | 6.0 | 8.0 | Flow | +| 46 | [celery/celery#10206](https://github.com/celery/celery/pull/10206) | Python | 6 | 6.2 | 7.6 | Flow | +| 47 | [pytest-dev/pytest#14310](https://github.com/pytest-dev/pytest/pull/14310) | Python | 5 | 6.2 | 7.8 | Flow | +| 48 | [pallets/werkzeug#3139](https://github.com/pallets/werkzeug/pull/3139) | Python | 7 | 5.8 | 7.6 | Flow | +| 49 | [pallets/werkzeug#3128](https://github.com/pallets/werkzeug/pull/3128) | Python | 3 | 5.4 | 7.6 | Flow | +| 50 | [encode/httpx#3690](https://github.com/encode/httpx/pull/3690) | Python | 4 | 5.8 | 7.6 | Flow | +| 51 | [encode/httpx#3673](https://github.com/encode/httpx/pull/3673) | Python | 8 | 6.2 | 7.6 | Flow | +| 52 | [kubernetes/kubernetes#138049](https://github.com/kubernetes/kubernetes/pull/138049) | Go | 3 | 6.4 | 7.0 | Flow | +| 53 | [kubernetes/kubernetes#138024](https://github.com/kubernetes/kubernetes/pull/138024) | Go | 12 | 6.0 | 6.4 | Flow | +| 54 | [docker/cli#6886](https://github.com/docker/cli/pull/6886) | Go | 6 | 6.2 | 6.2 | Tie | +| 55 | [hashicorp/terraform#38313](https://github.com/hashicorp/terraform/pull/38313) | Go | 3 | 6.0 | 6.8 | Flow | +| 56 | [hashicorp/terraform#38301](https://github.com/hashicorp/terraform/pull/38301) | Go | 3 | 6.2 | 7.6 | Flow | +| 57 | [prometheus/prometheus#18374](https://github.com/prometheus/prometheus/pull/18374) | Go | 7 | 6.8 | 7.4 | Flow | +| 58 | [grafana/grafana#121425](https://github.com/grafana/grafana/pull/121425) | Go | 3 | 6.2 | 7.6 | Flow | +| 59 | [grafana/grafana#121418](https://github.com/grafana/grafana/pull/121418) | Go | 4 | 6.2 | 7.4 | Flow | +| 60 | [go-gitea/gitea#37030](https://github.com/go-gitea/gitea/pull/37030) | Go | 24 | 6.2 | 7.2 | Flow | +| 61 | [go-gitea/gitea#37029](https://github.com/go-gitea/gitea/pull/37029) | Go | 3 | 6.2 | 7.4 | Flow | +| 62 | [go-gitea/gitea#37019](https://github.com/go-gitea/gitea/pull/37019) | Go | 4 | 6.0 | 6.6 | Flow | +| 63 | [minio/minio#21653](https://github.com/minio/minio/pull/21653) | Go | 4 | 6.2 | 7.0 | Flow | +| 64 | [minio/minio#21651](https://github.com/minio/minio/pull/21651) | Go | 3 | 6.4 | 7.6 | Flow | +| 65 | [minio/minio#21642](https://github.com/minio/minio/pull/21642) | Go | 3 | 6.6 | 7.2 | Flow | +| 66 | [etcd-io/etcd#21547](https://github.com/etcd-io/etcd/pull/21547) | Go | 4 | 5.6 | 6.6 | Flow | +| 67 | [etcd-io/etcd#21529](https://github.com/etcd-io/etcd/pull/21529) | Go | 4 | 6.6 | 7.4 | Flow | +| 68 | [containerd/containerd#13125](https://github.com/containerd/containerd/pull/13125) | Go | 4 | 6.2 | 7.4 | Flow | +| 69 | [containerd/containerd#13120](https://github.com/containerd/containerd/pull/13120) | Go | 7 | 6.2 | 7.2 | Flow | +| 70 | [containerd/containerd#13119](https://github.com/containerd/containerd/pull/13119) | Go | 7 | 6.2 | 7.2 | Flow | +| 71 | [denoland/deno#33075](https://github.com/denoland/deno/pull/33075) | Rust/TS | 5 | 6.2 | 7.4 | Flow | +| 72 | [denoland/deno#33068](https://github.com/denoland/deno/pull/33068) | Rust/TS | 4 | 6.0 | 6.6 | Flow | +| 73 | [rust-lang/rust#154540](https://github.com/rust-lang/rust/pull/154540) | Rust | 6 | 6.0 | 7.6 | Flow | +| 74 | [tokio-rs/tokio#7987](https://github.com/tokio-rs/tokio/pull/7987) | Rust | 5 | 6.6 | 7.4 | Flow | +| 75 | [tokio-rs/tokio#7978](https://github.com/tokio-rs/tokio/pull/7978) | Rust | 4 | 6.4 | 7.0 | Flow | +| 76 | [tokio-rs/tokio#7968](https://github.com/tokio-rs/tokio/pull/7968) | Rust | 4 | 6.0 | 5.8 | Tie | +| 77 | [tauri-apps/tauri#15117](https://github.com/tauri-apps/tauri/pull/15117) | Rust | 4 | 6.4 | 7.0 | Flow | +| 78 | [spring-projects/spring-boot#49791](https://github.com/spring-projects/spring-boot/pull/49791) | Java/Scala | 3 | 5.8 | 7.6 | Flow | +| 79 | [apache/kafka#21891](https://github.com/apache/kafka/pull/21891) | Java/Scala | 4 | 6.4 | 6.2 | Tie | +| 80 | [apache/kafka#21883](https://github.com/apache/kafka/pull/21883) | Java/Scala | 13 | 5.6 | 6.4 | Flow | +| 81 | [microsoft/TypeScript#63305](https://github.com/microsoft/TypeScript/pull/63305) | TypeScript/JS | 4 | 5.8 | 6.4 | Flow | +| 82 | [axios/axios#10582](https://github.com/axios/axios/pull/10582) | TypeScript/JS | 3 | 6.2 | 7.8 | Flow | +| 83 | [tanstack/query#10346](https://github.com/tanstack/query/pull/10346) | TypeScript/JS | 6 | 6.0 | 7.0 | Flow | +| 84 | [drizzle-team/drizzle-orm#5475](https://github.com/drizzle-team/drizzle-orm/pull/5475) | TypeScript/JS | 4 | 6.6 | 8.4 | Flow | +| 85 | [honojs/hono#4797](https://github.com/honojs/hono/pull/4797) | TypeScript/JS | 24 | 5.8 | 7.6 | Flow | +| 86 | [cloudflare/workers-sdk#13115](https://github.com/cloudflare/workers-sdk/pull/13115) | TypeScript/JS | 4 | 6.0 | 7.4 | Flow | +| 87 | [withastro/astro#16121](https://github.com/withastro/astro/pull/16121) | TypeScript/JS | 7 | 6.2 | 6.8 | Flow | +| 88 | [open-telemetry/opentelemetry-python#4974](https://github.com/open-telemetry/opentelemetry-python/pull/4974) | Python | 8 | 6.2 | 6.8 | Flow | +| 89 | [psf/black#5063](https://github.com/psf/black/pull/5063) | Python | 4 | 5.4 | 8.0 | Flow | +| 90 | [encode/starlette#3189](https://github.com/encode/starlette/pull/3189) | Python | 3 | 6.0 | 8.2 | Flow | +| 91 | [actix/actix-web#3988](https://github.com/actix/actix-web/pull/3988) | Rust | 3 | 6.2 | 7.0 | Flow | +| 92 | [serde-rs/serde#3034](https://github.com/serde-rs/serde/pull/3034) | Rust | 3 | 5.8 | 5.8 | Tie | +| 93 | [apache/spark#52460](https://github.com/apache/spark/pull/52460) | Java/Scala | 13 | 5.4 | 5.8 | Flow | +| 94 | [apache/spark#30327](https://github.com/apache/spark/pull/30327) | Java/Scala | 11 | 6.2 | 8.0 | Flow | +| 95 | [elastic/elasticsearch#145149](https://github.com/elastic/elasticsearch/pull/145149) | Java/Scala | 3 | 6.2 | 6.4 | Flow | +| 96 | [openai/openai-node#1798](https://github.com/openai/openai-node/pull/1798) | TypeScript/JS | 6 | 6.2 | 6.4 | Tie | +| 97 | [openai/openai-node#1769](https://github.com/openai/openai-node/pull/1769) | TypeScript/JS | 13 | 6.2 | 6.6 | Flow | +| 98 | [openai/openai-node#1767](https://github.com/openai/openai-node/pull/1767) | TypeScript/JS | 12 | 6.0 | 6.8 | Flow | +| 99 | [date-fns/date-fns#3813](https://github.com/date-fns/date-fns/pull/3813) | TypeScript/JS | 21 | 5.4 | 6.6 | Flow | +| 100 | [date-fns/date-fns#3796](https://github.com/date-fns/date-fns/pull/3796) | TypeScript/JS | 4 | 6.2 | 8.0 | Flow | + +## Repos Covered + +actix/actix-web, angular/angular, apache/kafka, apache/spark, axios/axios, babel/babel, celery/celery, cloudflare/workers-sdk, containerd/containerd, date-fns/date-fns, denoland/deno, docker/cli, drizzle-team/drizzle-orm, elastic/elasticsearch, encode/httpx, encode/starlette, eslint/eslint, etcd-io/etcd, facebook/react, go-gitea/gitea, grafana/grafana, hashicorp/terraform, honojs/hono, jestjs/jest, kubernetes/kubernetes, langchain-ai/langchain, microsoft/TypeScript, minio/minio, nodejs/node, open-telemetry/opentelemetry-python, openai/openai-node, oven-sh/bun, pallets/flask, pallets/werkzeug, pandas-dev/pandas, payloadcms/payload, prettier/prettier, prisma/prisma, prometheus/prometheus, psf/black, pydantic/pydantic, pytest-dev/pytest, python/cpython, remix-run/remix, rust-lang/rust, serde-rs/serde, shadcn-ui/ui, spring-projects/spring-boot, sveltejs/svelte, tanstack/query, tauri-apps/tauri, tokio-rs/tokio, trpc/trpc, vercel/next.js, vuejs/core, webpack/webpack, withastro/astro diff --git a/evals/actix__actix-web__3988.json b/evals/actix__actix-web__3988.json new file mode 100644 index 0000000..d448fdb --- /dev/null +++ b/evals/actix__actix-web__3988.json @@ -0,0 +1,119 @@ +{ + "pr": { + "url": "https://github.com/actix/actix-web/pull/3988", + "owner": "actix", + "repo": "actix-web", + "number": 3988, + "title": "fix(windows): enable dual-stack IPv6 sockets by default", + "files_changed": 3 + }, + "baseline_review": { + "comments": [ + { + "file": "actix-web/src/server.rs", + "line": 1270, + "severity": "positive", + "comment": "Good use of #[cfg(windows)] to scope this change strictly to the platform where the default behavior diverges. Linux and macOS already default IPV6_V6ONLY to false, so this avoids unnecessary interference on those platforms." + }, + { + "file": "actix-web/src/server.rs", + "line": 1272, + "severity": "minor", + "comment": "Using log::warn on failure and continuing is reasonable for a best-effort socket option. However, consider whether the warning message should include the address being bound, e.g., 'failed to set IPV6_V6ONLY=false on {addr}: {err}', so operators can correlate the warning with a specific listener when multiple are configured." + }, + { + "file": "actix-web/src/server.rs", + "line": 1271, + "severity": "minor", + "comment": "The set_only_v6(false) call is correctly placed after set_reuse_address and before bind/listen -- socket options must be set before the socket is bound. The ordering is correct." + }, + { + "file": "actix-web/src/server.rs", + "line": 448, + "severity": "positive", + "comment": "The documentation clearly explains the dual-stack behavior and provides a concrete escape hatch (use listen() with a manually-created listener) for users who need IPv6-only on Windows. This is a well-written doc comment." + }, + { + "file": "actix-web/tests/test_httpserver.rs", + "line": 220, + "severity": "medium", + "comment": "The test is gated with #[cfg(windows)] which means it will never run in CI unless CI includes Windows runners. The PR checklist shows 'Tests for the changes have been added / updated' is unchecked, which is concerning -- if actix-web CI does not run Windows tests, this test may never actually execute. Confirm that the CI matrix includes Windows." + }, + { + "file": "actix-web/tests/test_httpserver.rs", + "line": 243, + "severity": "minor", + "comment": "The test connects to 127.0.0.1 (IPv4 loopback) against a server bound on [::]:0 (IPv6 wildcard). This correctly validates dual-stack: if set_only_v6(false) works, the IPv6 socket will accept the IPv4 connection. Good test design." + }, + { + "file": "actix-web/CHANGES.md", + "line": 5, + "severity": "positive", + "comment": "Clear changelog entry that describes the behavioral change and specifies it applies to Windows only. Good communication of the scope." + } + ], + "summary": "A clean, well-scoped fix that addresses a real platform inconsistency between Windows and Unix-like systems for IPv6 dual-stack sockets. The implementation is minimal, correctly ordered, and gracefully handles failure. The main concern is whether the Windows-only test will actually be exercised in CI." + }, + "flow_guided_review": { + "comments": [ + { + "file": "actix-web/src/server.rs", + "line": 1270, + "severity": "positive", + "comment": "The core change is a single socket option addition in create_tcp_listener, the centralized path for all bind() calls. This means every listener created via HttpServer::bind() on Windows will get dual-stack behavior. The flow plan is empty (no cross-cutting flows), which correctly reflects that this is an isolated change in a leaf function." + }, + { + "file": "actix-web/src/server.rs", + "line": 1272, + "severity": "medium", + "comment": "Risk: the warn-and-continue pattern silently degrades to IPv6-only behavior if the socket option fails. On older Windows versions or restricted environments, users may not notice the warning and wonder why IPv4 connections fail. Consider whether this should be documented more prominently or whether a specific error code check (e.g., WSAENOPROTOOPT) should distinguish expected-unsupported from unexpected failures." + }, + { + "file": "actix-web/src/server.rs", + "line": 448, + "severity": "positive", + "comment": "The documentation on bind() correctly directs users who need IPv6-only on Windows to use listen() with a manually-created socket. This creates a clear opt-out path and avoids the need for a configuration flag." + }, + { + "file": "actix-web/tests/test_httpserver.rs", + "line": 220, + "severity": "medium", + "comment": "Without the flow plan highlighting dependencies, it is easy to miss that this test only validates the happy path (dual-stack enabled successfully). There is no test for the failure/warning path -- e.g., what happens when set_only_v6 fails. A unit test mocking the failure could verify the warn-and-continue behavior." + }, + { + "file": "actix-web/src/server.rs", + "line": 1271, + "severity": "minor", + "comment": "The addr.is_ipv6() guard is correct -- set_only_v6 is only meaningful on IPv6 sockets. Calling it on IPv4 sockets would be a no-op or error, so this guard is necessary." + }, + { + "file": "actix-web/CHANGES.md", + "line": 5, + "severity": "minor", + "comment": "The changelog entry mentions 'Actix-created listeners' which correctly scopes it to bind() but not listen(). Users bringing their own listeners via listen() are unaffected. This distinction is important for users who may have custom socket configurations." + } + ], + "summary": "A well-isolated change to the socket creation path with no cross-cutting concerns. The flow-guided perspective highlights two risks the baseline missed: the silent degradation when set_only_v6 fails, and the lack of a failure-path test. Otherwise, the implementation is sound and correctly scoped." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 8, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 7.0 + }, + "reasoning": "The flow-guided review edges ahead by identifying the silent degradation risk and the missing failure-path test, both of which the baseline overlooked. However, with an empty flow plan, the structural advantage of flow-guided review is limited -- both reviews largely converge on the same observations. The flow-guided review is slightly more actionable with its suggestions about error code differentiation and failure-path testing.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T20:15:00.000000+00:00" +} diff --git a/evals/angular__angular__67916.json b/evals/angular__angular__67916.json new file mode 100644 index 0000000..b7e60f0 --- /dev/null +++ b/evals/angular__angular__67916.json @@ -0,0 +1,102 @@ +{ + "pr": "angular/angular#67916", + "title": "docs(docs-infra): introduce a custom UrlSerializer", + "baseline_review": { + "comments": [ + { + "file": "adev/src/app/core/services/routing/adev-url-serializer.ts", + "line": 21, + "severity": "medium", + "comment": "Using `replaceAll` with a regex is correct but the pattern `/%2(F|f)/g` mixes `replaceAll` with the `g` flag, which is redundant. `replaceAll` already replaces all occurrences. While this works (JS requires the `g` flag when passing a regex to `replaceAll`), the intent would be clearer with a simpler approach like `url.replace(/%2[Ff]/g, '/')` using plain `replace` with the global flag, or `url.replaceAll('%2F', '/').replaceAll('%2f', '/')` to avoid regex entirely." + }, + { + "file": "adev/src/app/core/services/routing/adev-url-serializer.ts", + "line": 21, + "severity": "medium", + "comment": "This only handles `%2F` / `%2f` but does not account for other URL-encoded characters that the server might also decode (e.g., `%23` for `#`, `%3F` for `?`). If the server performs full URL decoding, this partial client-side approach could still produce hydration mismatches for URLs containing other encoded characters. Confirm that only forward slashes need this treatment." + }, + { + "file": "adev/src/app/core/services/routing/adev-url-serializer.ts", + "line": 15, + "severity": "low", + "comment": "The class is not decorated with `@Injectable()`. Since it is provided via `useClass` in the providers array, Angular will instantiate it directly, which works for classes with no constructor dependencies. This is fine for the current implementation but if dependencies are ever added, `@Injectable()` will be needed." + }, + { + "file": "adev/src/app/core/services/routing/adev-url.serializer.spec.ts", + "line": 1, + "severity": "low", + "comment": "The spec filename uses a dot (`adev-url.serializer.spec.ts`) while the implementation file uses a hyphen (`adev-url-serializer.ts`). This naming inconsistency could cause confusion when searching for the test file. Consider renaming to `adev-url-serializer.spec.ts` for consistency." + }, + { + "file": "adev/src/app/core/services/routing/adev-url.serializer.spec.ts", + "line": 30, + "severity": "medium", + "comment": "The test only covers a single encoded forward slash in the middle of a path. Consider adding test cases for: (1) multiple encoded slashes (`page%2Fabout%2Fdetails`), (2) an encoded slash at the start of the URL (`%2Fpage/about`), (3) a URL with no encoded slashes to ensure normal URLs pass through unchanged, and (4) mixed encoded and regular slashes." + } + ], + "summary": "The PR introduces a clean, focused custom UrlSerializer that decodes encoded forward slashes to match server-side behavior and prevent hydration mismatches. The implementation is correct but the test coverage is thin, the spec filename is inconsistent with the source file, and there is a question about whether other encoded characters besides forward slashes need similar handling." + }, + "flow_guided_review": { + "comments": [ + { + "file": "adev/src/app/core/services/routing/adev-url-serializer.ts", + "line": 15, + "severity": "low", + "comment": "Step 1 (entry point, high risk): The `AdevUrlSerializer` class correctly extends `DefaultUrlSerializer`, which is the standard Angular approach for customizing URL parsing. By overriding only `parse`, the `serialize` method is inherited unchanged, meaning round-trip behavior (parse then serialize) will normalize encoded slashes to decoded form. Verify this is the desired behavior -- a URL originally containing `%2F` will never be re-encoded back to `%2F` by the serializer." + }, + { + "file": "adev/src/app/core/services/routing/adev-url-serializer.ts", + "line": 21, + "severity": "medium", + "comment": "Step 2 (parse method, high risk as entry point): The `replaceAll` call with regex `/%2(F|f)/g` is functionally correct but the `g` flag is redundant with `replaceAll` (though JS mandates it for regex arguments to `replaceAll`). A simpler alternative is `url.replace(/%2[Ff]/g, '/')`. More importantly, this replacement runs before `super.parse()`, which means the decoded slashes will be treated as path separators by the default parser -- this is intentional and correct for the stated goal of matching server behavior." + }, + { + "file": "adev/src/app/core/services/routing/adev-url-serializer.ts", + "line": 21, + "severity": "high", + "comment": "Step 2 continued: Since the plan identifies this as high risk due to being an entry point for all URL parsing in the app, consider edge cases: (1) query parameters containing `%2F` (e.g., `?redirect=%2Fhome`) will also have their slashes decoded, which may alter query parameter semantics; (2) fragment identifiers with `%2F` will similarly be affected. The replacement should ideally only target the path portion of the URL, not the entire URL string." + }, + { + "file": "adev/src/app/app.config.ts", + "line": 54, + "severity": "low", + "comment": "The provider registration correctly uses `useClass` to substitute the default `UrlSerializer` with `AdevUrlSerializer`. This ensures all Angular Router URL parsing throughout the application goes through the custom serializer. The placement at the end of the providers array is fine -- provider order does not affect DI resolution for this use case." + }, + { + "file": "adev/src/app/core/services/routing/adev-url.serializer.spec.ts", + "line": 1, + "severity": "low", + "comment": "The spec filename `adev-url.serializer.spec.ts` uses a dot separator while the source uses a hyphen (`adev-url-serializer.ts`). This breaks the common convention of mirroring filenames between source and spec." + }, + { + "file": "adev/src/app/core/services/routing/adev-url.serializer.spec.ts", + "line": 30, + "severity": "medium", + "comment": "Given that the plan identifies `parse` as high risk, the test coverage is insufficient. Missing cases: (1) URLs with query strings containing `%2F` -- does it inadvertently decode slashes in query params? (2) URLs with multiple consecutive encoded slashes; (3) URLs with no encoded content to verify passthrough; (4) full URL paths like `/guide%2Fcomponents` matching the PR description's example. The current test only covers a bare path segment without a leading slash." + } + ], + "summary": "Following the flow from the `AdevUrlSerializer` class definition through its `parse` method to the app-wide provider registration, the implementation is structurally sound and idiomatic Angular. However, the high-risk `parse` method applies `%2F` decoding to the entire URL string including query parameters and fragments, which could cause unintended side effects, and the test suite does not cover these edge cases." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 9, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identifies a critical issue the baseline misses: the `replaceAll` operates on the entire URL string, meaning encoded forward slashes in query parameters and fragments will also be decoded, potentially altering their semantics. This insight comes directly from understanding the parse method's role as the single entry point for all URL parsing in the application (flagged as high risk in the plan). The baseline review raises valid points about test coverage and naming inconsistency but treats the implementation more superficially. The flow-guided review's ordered traversal from class definition to method implementation to provider registration provides a clearer narrative of how the change affects the entire routing pipeline." + } +} \ No newline at end of file diff --git a/evals/angular__angular__67922.json b/evals/angular__angular__67922.json new file mode 100644 index 0000000..4a03740 --- /dev/null +++ b/evals/angular__angular__67922.json @@ -0,0 +1,108 @@ +{ + "pr": "angular/angular#67922", + "title": "docs(docs-infra): sanitize markdown tooltip in Code editor", + "baseline_review": { + "comments": [ + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.ts", + "line": 155, + "severity": "high", + "comment": "The diff is truncated at `wrapper.innerHTML = s`, so we cannot verify the full implementation of `renderAndSanitizeMarkdownToHtml` or confirm that the sanitized HTML is correctly assigned. If this line is incomplete or malformed it would break the build." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.ts", + "line": 97, + "severity": "medium", + "comment": "Functions `getMarkedHtmlFromString` and `getTagsHtml` were changed from private (module-scoped) to exported solely to enable testing. Consider whether a more encapsulated testing strategy (e.g., testing through the public `getTooltipExtension` API) would be preferable to exporting implementation details." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.ts", + "line": 117, + "severity": "medium", + "comment": "The `SecurityContext` import is added but its usage is not visible in the truncated diff. Verify it is actually used in `renderAndSanitizeMarkdownToHtml`; if so, confirm that `SecurityContext.HTML` is the correct context for sanitizing markdown-rendered HTML (it should be)." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.spec.ts", + "line": 20, + "severity": "medium", + "comment": "The test asserts exact HTML output `

hello

` which is tightly coupled to both the markdown renderer's output format and the sanitizer's behavior. If `marked` changes its HTML output (e.g., self-closing tags), this test will break. Consider using `toContain` for the safe parts and `not.toContain` for the dangerous parts." + }, + { + "file": "adev/src/app/editor/code-editor/code-mirror-editor.service.ts", + "line": 84, + "severity": "low", + "comment": "The `DomSanitizer` is injected and threaded through to `getTooltipExtension`. This is a clean approach using Angular's built-in sanitization. The import reordering is a welcome cleanup for alphabetical consistency." + } + ], + "summary": "This PR adds HTML sanitization to the code editor's tooltip system by threading Angular's DomSanitizer through the tooltip extension, mitigating XSS risks from markdown-rendered content. The approach is sound but the diff is truncated, making it impossible to fully verify the core sanitization logic in `renderAndSanitizeMarkdownToHtml`." + }, + "flow_guided_review": { + "comments": [ + { + "file": "adev/src/app/editor/code-editor/code-mirror-editor.service.ts", + "line": 84, + "severity": "low", + "comment": "Entry point: `CodeMirrorEditor` correctly injects `DomSanitizer` and passes it to `getTooltipExtension`. The dependency injection pattern is idiomatic Angular. The import reordering to alphabetical order is a good housekeeping change." + }, + { + "file": "adev/src/app/editor/code-editor/code-mirror-editor.service.ts", + "line": 453, + "severity": "low", + "comment": "The call site in `getLanguageExtensions` correctly threads `this.domSanitizer` as the new fourth argument. Since `getTooltipExtension` is called from multiple paths (flagged as medium risk due to multiple callers), verify there are no other call sites outside this file that need updating." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.ts", + "line": 28, + "severity": "medium", + "comment": "The `getTooltipExtension` signature change (adding `domSanitizer` parameter) is a breaking change for any caller. The plan identifies multiple callers. All downstream call sites must be updated. The diff only shows the update in `code-mirror-editor.service.ts` -- confirm no other consumers exist." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.ts", + "line": 154, + "severity": "high", + "comment": "The new `renderAndSanitizeMarkdownToHtml` function (leaf node, medium risk) is the core security fix but its implementation is truncated in the diff. This is the most critical piece -- it must call `marked()` first, then `domSanitizer.sanitize(SecurityContext.HTML, ...)` on the result. Without seeing the full implementation, the correctness of the entire PR cannot be confirmed." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.ts", + "line": 63, + "severity": "low", + "comment": "The `create` method (internal node) correctly passes `domSanitizer` to both `getMarkedHtmlFromString` and `getTagsHtml`. The branching logic (documentation vs tags) is preserved unchanged, with only the sanitizer argument added." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.spec.ts", + "line": 14, + "severity": "medium", + "comment": "Tests correctly verify that XSS payloads (onerror handlers) are stripped from both markdown content and JSDoc tags. However, the tests use `TestBed.inject(DomSanitizer)` which returns the real platform-browser sanitizer -- this is appropriate for security tests as it validates actual sanitization behavior rather than mocked behavior." + }, + { + "file": "adev/src/app/editor/code-editor/extensions/tooltip.spec.ts", + "line": 30, + "severity": "low", + "comment": "The `getTagsHtml` test covers the JSDoc tags path with a realistic tag structure. Consider adding an edge case test for empty tags array or tags with no text property to ensure the function handles edge cases gracefully." + } + ], + "summary": "Following the data flow from the entry point (CodeMirrorEditor) through getTooltipExtension to the leaf sanitization function, the PR correctly threads DomSanitizer through all layers. The critical gap is that the diff is truncated and the core `renderAndSanitizeMarkdownToHtml` implementation cannot be fully reviewed, which is the highest-risk node in the dependency graph." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides superior analysis by tracing the DomSanitizer dependency from its injection point in CodeMirrorEditor through getTooltipExtension down to the leaf sanitization function. It correctly identifies the multiple-caller risk on getTooltipExtension and emphasizes verifying all call sites are updated. The baseline review catches the truncated diff and testing fragility concerns but lacks the structural understanding of how data flows through the tooltip system. The flow-guided review's ordered traversal ensures no intermediate node is missed and correctly prioritizes the truncated renderAndSanitizeMarkdownToHtml as the highest-risk element." + } +} \ No newline at end of file diff --git a/evals/apache__kafka__21883.json b/evals/apache__kafka__21883.json new file mode 100644 index 0000000..cef86ed --- /dev/null +++ b/evals/apache__kafka__21883.json @@ -0,0 +1,102 @@ +{ + "pr": "apache/kafka#21883", + "title": "MINOR: Various cleanups in raft module", + "baseline_review": { + "comments": [ + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java", + "line": 141, + "severity": "low", + "comment": "Removing the `= null` initializer from `BatchBuilder batch` is correct since `batch` is immediately assigned on the next line. However, note that if `maybeAllocateBatch` were ever to throw a checked exception in the future, having the initializer would prevent a 'variable may not have been initialized' compiler error in subsequent code. This is a minor style improvement that is safe given the current code structure." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java", + "line": 579, + "severity": "medium", + "comment": "The rename from `validateContruction` to `validateConstruction` fixes the typo but the corrected name still appears to be a typo -- it should be `validateConstruction` -> `validateConstruction`. Wait, actually `validateConstruction` is also wrong. The correct English word is 'construction', so the method should be named `validateConstruction`... actually looking again, the new name IS `validateConstruction` which still has an 's' before 'truction'. The correct spelling would be `validateConstruction`. On closer reading the diff shows the rename is to `validateConstruction` -- this is correct: 'construction' = c-o-n-s-t-r-u-c-t-i-o-n. The original `validateContruction` was missing the 's'. The fix is correct." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/KRaftVersionUpgrade.java", + "line": 34, + "severity": "low", + "comment": "Removing `public` from record declarations inside a sealed interface is correct. Members of interfaces are implicitly public in Java, so the `public` keyword is redundant. This is a standard Java style cleanup." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/KRaftVersionUpgrade.java", + "line": 56, + "severity": "low", + "comment": "Removing `static final` to just leave `KRaftVersionUpgrade EMPTY = new Empty()` is correct. Fields in interfaces are implicitly `public static final` in Java, so the explicit modifiers are redundant." + }, + { + "file": "raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientPreVoteTest.java", + "line": 481, + "severity": "low", + "comment": "Changing `context.clusterId.toString()` to `context.clusterId` suggests that the `voteRequest` method parameter was changed or already accepts the type of `clusterId` directly (likely a `String` already). This is a minor cleanup removing an unnecessary `.toString()` call. Verify that the `voteRequest` method signature accepts `clusterId`'s type directly." + } + ], + "summary": "This PR is a straightforward collection of minor cleanups in the Kafka raft module: fixing two typos ('kraft.verion' -> 'kraft.version', 'atempt' -> 'attempt', 'validateContruction' -> 'validateConstruction'), removing redundant Java modifiers from an interface, removing an unnecessary null initializer, and removing unnecessary `.toString()` calls in tests. All changes are mechanical and low-risk with no behavioral impact." + }, + "flow_guided_review": { + "comments": [ + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java", + "line": 579, + "severity": "medium", + "comment": "The rename from `validateContruction` to `validateConstruction` is called from two constructors (lines 579 and 596). Both call sites are updated consistently. Since this is a private method, there is no external API impact. However, any code that might reference this method name via reflection (unlikely for a private method) would break." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/BatchAccumulator.java", + "line": 141, + "severity": "low", + "comment": "The removal of `= null` from the `batch` variable declaration is safe because the variable is immediately assigned on the very next line via `maybeAllocateBatch`. The null check on line 142 (`if (batch == null)`) still works correctly since `maybeAllocateBatch` can return null. This is a clean style improvement." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/KRaftVersionUpgrade.java", + "line": 34, + "severity": "low", + "comment": "Removing redundant `public` from all record declarations, default methods, and the static factory method in this sealed interface is correct per Java language semantics. Since this is a sealed interface, all permitted implementations are already constrained. The `static final` removal from the `EMPTY` field is also correct as interface fields are implicitly `public static final`." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/LeaderState.java", + "line": 566, + "severity": "low", + "comment": "Fixing the typo 'kraft.verion' to 'kraft.version' in a comment. This is a documentation-only fix with no behavioral impact, but it improves readability of an important comment about kraft version upgrade constraints." + }, + { + "file": "raft/src/main/java/org/apache/kafka/raft/internals/RequestSendResult.java", + "line": 20, + "severity": "low", + "comment": "Fixing the Javadoc typo 'atempt' to 'attempt'. This improves the documentation for the `RequestSendResult` type which is part of the raft internals API." + }, + { + "file": "raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientPreVoteTest.java", + "line": 481, + "severity": "low", + "comment": "Removing redundant `.toString()` calls on `context.clusterId` at two call sites. If `clusterId` is already a `String`, the `.toString()` is a no-op. This cleanup improves readability without changing behavior." + } + ], + "summary": "This is a purely mechanical cleanup PR touching the raft module across 5 files with no behavioral changes. The changes fall into three categories: typo fixes in comments and method names (3 occurrences), removal of redundant Java interface modifiers (7 occurrences in KRaftVersionUpgrade), and minor code style improvements (removing unnecessary null init and toString calls). All changes are safe and independent of each other." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 5, + "efficiency": 7, + "overall": 5.6 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.4 + } + }, + "winner": "flow_guided", + "reasoning": "For this trivial cleanup PR, both reviews correctly identify that all changes are mechanical and low-risk. The review plan is empty (no steps, clusters, or dependencies), which limits the flow-guided review's advantage. The flow-guided review edges ahead slightly by better noting the consistency of the rename across both constructor call sites in BatchAccumulator, grouping the KRaftVersionUpgrade changes more coherently, and providing slightly more structured analysis of each change category. However, the baseline review contains an awkward self-correcting tangent about the spelling of 'validateConstruction' that wastes review bandwidth. Neither review identifies significant risks because there are none in this PR -- it is purely cosmetic. The flow-guided review wins marginally on completeness and organization rather than on deep flow-aware insights." + } +} \ No newline at end of file diff --git a/evals/apache__kafka__21891.json b/evals/apache__kafka__21891.json new file mode 100644 index 0000000..62cab32 --- /dev/null +++ b/evals/apache__kafka__21891.json @@ -0,0 +1,96 @@ +{ + "pr": "apache/kafka#21891", + "title": "MINOR: Use getAbsolutePath in LocalLog#exception", + "baseline_review": { + "comments": [ + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java", + "line": 744, + "severity": "minor", + "comment": "Switching from `getCanonicalPath()` to `getAbsolutePath()` removes the `IOException` but also changes behavior: `getCanonicalPath()` resolves symlinks and normalizes `.`/`..` segments, while `getAbsolutePath()` does not. In error messages this is acceptable since the path is informational, but if any downstream code relied on the canonicalized form for comparison or deduplication, the output would differ. For a diagnostic exception message this tradeoff is fine." + }, + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java", + "line": 744, + "severity": "positive", + "comment": "Removing `throws IOException` from `exception(File dir)` is a good simplification. The original signature forced all callers to handle or propagate a checked exception that was only incidental to constructing an error message, not to the core parsing logic." + }, + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/LogManager.java", + "line": 659, + "severity": "minor", + "comment": "The removal of the `try-catch(IOException)` block around `parseTopicPartitionName` simplifies the filter lambda. However, the old code returned `false` on IOException, silently skipping unparseable directories. Now that no IOException is thrown, a `KafkaException` from malformed directory names will propagate up uncaught instead of being swallowed. Verify that callers of `loadLogs` handle `KafkaException` appropriately -- this is a subtle behavioral change from silently skipping to failing." + }, + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java", + "line": 2784, + "severity": "nit", + "comment": "The `throws IOException` removal on the public `UnifiedLog.parseTopicPartitionName` is a public API change. Since this is an internal API under `storage.internals`, binary compatibility is less of a concern, but any external callers catching IOException around this call will now have dead catch blocks flagged by the compiler." + }, + { + "file": "storage/src/test/java/org/apache/kafka/storage/internals/log/LocalLogTest.java", + "line": 101, + "severity": "positive", + "comment": "Good consistency fix in the test: the assertion message now uses `getAbsolutePath()` to match the production code, ensuring that if the test fails, the diagnostic output matches what the actual exception would contain." + } + ], + "summary": "This PR is a clean simplification that replaces `getCanonicalPath()` with `getAbsolutePath()` to eliminate a checked IOException from the `exception()` and `parseTopicPartitionName()` method signatures. The most notable behavioral change is in LogManager.loadLogs where the removed try-catch previously swallowed IOExceptions from malformed directories, and callers should be verified to handle the now-propagating KafkaException." + }, + "flow_guided_review": { + "comments": [ + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java", + "line": 744, + "severity": "minor", + "comment": "The switch from `getCanonicalPath()` to `getAbsolutePath()` trades symlink resolution and path normalization for simplicity. Since this value only appears in an exception message for human consumption, the tradeoff is acceptable. The removed `throws IOException` declaration cleans up the entire call chain." + }, + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java", + "line": 753, + "severity": "positive", + "comment": "Removing `throws IOException` from `parseTopicPartitionName` is the key improvement. This method's core responsibility is string parsing of directory names, not filesystem I/O. The checked exception was an artifact of using `getCanonicalPath()` in the error path, leaking an implementation detail into the public contract." + }, + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/LogManager.java", + "line": 659, + "severity": "minor", + "comment": "With the IOException no longer thrown, the try-catch in the filter lambda is correctly removed. However, this changes error handling semantics: previously, an IOException during path resolution would cause the directory to be silently skipped (return false). Now, a KafkaException from a truly malformed directory name will propagate and potentially abort the entire loadLogs operation. Confirm this is the desired behavior -- though in practice, the IOException was only possible from getCanonicalPath, not from name parsing, so this is likely a dead code path removal." + }, + { + "file": "storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java", + "line": 2784, + "severity": "nit", + "comment": "The delegate method in UnifiedLog correctly mirrors the signature change. Since UnifiedLog.parseTopicPartitionName is a thin wrapper over LocalLog.parseTopicPartitionName, keeping them in sync is essential." + }, + { + "file": "storage/src/test/java/org/apache/kafka/storage/internals/log/LocalLogTest.java", + "line": 101, + "severity": "nit", + "comment": "Test assertion messages updated to use `getAbsolutePath()` matching production code. The removal of `throws IOException` from all test method signatures is correct since the methods under test no longer declare checked exceptions." + } + ], + "summary": "This is a straightforward cleanup that removes an incidental `IOException` from the `parseTopicPartitionName` call chain by switching to `getAbsolutePath()`. The review plan is empty (no steps or dependencies), reflecting the simple, single-concern nature of this change -- there are no complex flows or cross-cutting risks to trace." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + } + }, + "winner": "tie", + "reasoning": "With an empty review plan (no steps, clusters, or dependencies), the flow-guided review has no structural advantage over the baseline. Both reviews identify the same key concerns: the semantic difference between getCanonicalPath and getAbsolutePath, the behavioral change in LogManager.loadLogs where the try-catch removal changes error propagation, and the public API signature change in UnifiedLog. The baseline review slightly edges on risk identification by more explicitly calling out the LogManager behavioral change as a concern requiring caller verification, while the flow-guided review correctly notes this was likely dead code. For this trivially simple, single-concern PR, both approaches converge to essentially the same analysis, making it a tie." + } +} diff --git a/evals/apache__spark__30327.json b/evals/apache__spark__30327.json new file mode 100644 index 0000000..353049a --- /dev/null +++ b/evals/apache__spark__30327.json @@ -0,0 +1,142 @@ +{ + "pr": { + "url": "https://github.com/apache/spark/pull/30327", + "owner": "apache", + "repo": "spark", + "number": 30327, + "title": "[WIP] Test", + "files_changed": 11, + "additions": 43, + "deletions": 19, + "language": "scala" + }, + "timestamp": "2026-03-30T18:30:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 45, + "severity": "major", + "comment": "Replacing HasBlockSize with HasBlockSizeInMB changes the trait's public API surface. The parameter type changes from Int to Double and the semantic meaning shifts from count-based to memory-based sizing. This is a breaking change for any downstream code that references blockSize." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 57, + "severity": "minor", + "comment": "The default changes from blockSize -> 1 (skip blocking) to blockSizeInMB -> 0.0 (auto-detect). The zero sentinel triggers InstanceBlock.DefaultBlockSizeInMB at training time. This implicit auto-sizing should be documented more prominently since users previously had an explicit opt-in." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 171, + "severity": "minor", + "comment": "The new double-caching warning fires when dataset.storageLevel != NONE. This is a good UX improvement, but the old code only persisted when storageLevel was NONE and blockSize == 1; now the persist call is removed entirely, suggesting the new trainImpl handles caching internally." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 195, + "severity": "minor", + "comment": "The require(actualBlockSizeInMB > 0) guard on the inferred default is defensive but the error message 'inferred actual BlockSizeInMB must > 0' reads awkwardly. Consider 'inferred actualBlockSizeInMB must be positive'." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 237, + "severity": "major", + "comment": "The two training paths (trainOnRows for blockSize==1, trainOnBlocks otherwise) are collapsed into a single trainImpl call. The old row-by-row path is removed along with the explicit unpersist. Verify that trainImpl handles the unpersist lifecycle correctly to avoid memory leaks." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 190, + "severity": "minor", + "comment": "The sparsity calculation and warning that existed for blockSize > 1 is removed entirely. This loses a useful diagnostic signal — users with highly sparse data may silently hit performance regressions without the old warning." + } + ], + "summary": "This WIP PR replaces the integer blockSize parameter with a Double blockSizeInMB in LinearSVC, changing from count-based to memory-based block sizing with an auto-detect sentinel of 0.0. The change is a breaking API modification that removes the row-by-row training path and sparsity diagnostics, while adding double-caching warnings." + }, + "flow_guided_review": { + "comments": [ + { + "file": "python/pyspark/ml/param/shared.py", + "line": 602, + "severity": "major", + "comment": "Plan step 1 (entry point): HasBlockSizeInMB is a new shared param mixin class. As a high-risk entry point, it defines the new Double-typed blockSizeInMB parameter with default 0.0. All downstream classes in both Python and Scala depend on this definition. Verify that the param description clearly explains the 0.0 auto-detect behavior." + }, + { + "file": "python/pyspark/ml/classification.py", + "line": 500, + "severity": "major", + "comment": "Plan steps 4-5: _LinearSVCParams switches from HasBlockSize to HasBlockSizeInMB. The dependency chain shows __init__ is called by LinearSVC.setParams (step 10) and connects to OneVsRest._to_java, meaning this breaking change propagates to multi-class wrapper usage as well. Ensure OneVsRest correctly forwards the renamed parameter." + }, + { + "file": "python/pyspark/ml/classification.py", + "line": 595, + "severity": "minor", + "comment": "Plan step 7: LinearSVC.__init__ changes the default from blockSize=1 to blockSizeInMB=0.0. The flow shows this feeds into setParams (step 10) which calls _checkThresholdConsistency. The parameter rename here must match the Scala side exactly for Java/Python interop via _to_java." + }, + { + "file": "python/pyspark/ml/classification.py", + "line": 687, + "severity": "minor", + "comment": "Plan step 8: setBlockSizeInMB is a user-facing entry point with no callers in the codebase. The plan correctly flags it as high risk because it is the primary API users interact with. The type changes from Int to Double — existing Python users calling setBlockSize will get an AttributeError." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 45, + "severity": "major", + "comment": "The Scala-side trait change mirrors the Python shared.py change (step 1). The flow plan's cross-file dependency chain confirms both language APIs must change in lockstep. The diff shows 11 files changed but the truncated diff only shows 2 — verify that the remaining 9 files (likely other classifiers and tests) are consistently updated." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 237, + "severity": "major", + "comment": "The trainOnRows/trainOnBlocks merge into trainImpl is not covered by any plan step (the plan focuses on Python files), yet this is the most significant behavioral change. The plan's 8 independent flows suggest other files may have similar training-path consolidation that should be reviewed for consistency." + }, + { + "file": "mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala", + "line": 190, + "severity": "minor", + "comment": "The removal of the sparsity diagnostic and numNonZeros metric request is a functional regression in observability. The plan does not flag this because it focuses on the Python API surface, but the Scala training logic change has monitoring implications." + } + ], + "summary": "Following the plan's dependency chain from HasBlockSizeInMB (shared param) through _LinearSVCParams to LinearSVC reveals a systematic API rename that must be consistent across Python and Scala, including propagation to OneVsRest via _to_java. The plan's focus on Python entry points correctly identifies the API-breaking surface, but misses the Scala-side behavioral changes (training path consolidation, sparsity diagnostic removal) that carry the highest regression risk." + }, + "review_plan": { + "stats": { + "totalSteps": 18, + "totalAdditions": 43, + "totalDeletions": 19, + "independentFlows": 8, + "filesChanged": 2 + }, + "steps": [ + {"order": 1, "nodeId": "python/pyspark/ml/param/shared.py::HasBlockSizeInMB", "name": "HasBlockSizeInMB", "file": "python/pyspark/ml/param/shared.py", "lines": [602, 617], "type": "class", "changeType": "modified", "additions": 16, "deletions": 0, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 2, "nodeId": "python/pyspark/ml/param/shared.py::HasBlockSizeInMB.__init__", "name": "__init__", "file": "python/pyspark/ml/param/shared.py", "lines": [609, 611], "type": "method", "changeType": "modified", "additions": 3, "deletions": 0, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 3, "nodeId": "python/pyspark/ml/param/shared.py::HasBlockSizeInMB.getBlockSizeInMB", "name": "getBlockSizeInMB", "file": "python/pyspark/ml/param/shared.py", "lines": [613, 617], "type": "method", "changeType": "modified", "additions": 5, "deletions": 0, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 4, "nodeId": "python/pyspark/ml/classification.py::_LinearSVCParams", "name": "_LinearSVCParams", "file": "python/pyspark/ml/classification.py", "lines": [500, 519], "type": "class", "changeType": "modified", "additions": 2, "deletions": 2, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 5, "nodeId": "python/pyspark/ml/classification.py::_LinearSVCParams.__init__", "name": "__init__", "file": "python/pyspark/ml/classification.py", "lines": [515, 519], "type": "method", "changeType": "modified", "additions": 1, "deletions": 1, "role": "entry_point", "risk": "high", "calledBy": [], "calls": ["python/pyspark/ml/param/shared.py::HasMaxIter.__init__", "python/pyspark/ml/classification.py::LinearSVC.setParams", "python/pyspark/ml/classification.py::_LogisticRegressionParams._checkThresholdConsistency", "python/pyspark/ml/classification.py::OneVsRest._to_java"], "riskReasons": ["entry_point"]}, + {"order": 6, "nodeId": "python/pyspark/ml/classification.py::LinearSVC", "name": "LinearSVC", "file": "python/pyspark/ml/classification.py", "lines": [522, 692], "type": "class", "changeType": "modified", "additions": 9, "deletions": 9, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 7, "nodeId": "python/pyspark/ml/classification.py::LinearSVC.__init__", "name": "__init__", "file": "python/pyspark/ml/classification.py", "lines": [595, 610], "type": "method", "changeType": "modified", "additions": 2, "deletions": 2, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 8, "nodeId": "python/pyspark/ml/classification.py::LinearSVC.setBlockSizeInMB", "name": "setBlockSizeInMB", "file": "python/pyspark/ml/classification.py", "lines": [687, 692], "type": "method", "changeType": "modified", "additions": 3, "deletions": 3, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 10, "nodeId": "python/pyspark/ml/classification.py::LinearSVC.setParams", "name": "setParams", "file": "python/pyspark/ml/classification.py", "lines": [612, 626], "type": "method", "changeType": "modified", "additions": 2, "deletions": 2, "role": "internal", "risk": "low", "calledBy": ["python/pyspark/ml/classification.py::_LinearSVCParams.__init__"], "calls": ["python/pyspark/ml/classification.py::_LogisticRegressionParams._checkThresholdConsistency"], "riskReasons": []} + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + }, + "reasoning": "The flow-guided review leverages the plan's dependency chain to trace the API rename from the shared HasBlockSizeInMB param through _LinearSVCParams to LinearSVC and its setParams/setBlockSizeInMB methods. This reveals cross-language consistency requirements (Python/Scala interop via _to_java) and the OneVsRest propagation path that the baseline review missed entirely. The flow-guided review also correctly identifies that the plan is biased toward Python entry points and compensates by noting the uncovered Scala behavioral changes (training path consolidation, sparsity removal). The baseline review identifies the same core issues but lacks the structural reasoning about why each change matters in the dependency graph.", + "winner": "flow_guided" + } +} \ No newline at end of file diff --git a/evals/apache__spark__52460.json b/evals/apache__spark__52460.json new file mode 100644 index 0000000..c73de31 --- /dev/null +++ b/evals/apache__spark__52460.json @@ -0,0 +1,113 @@ +{ + "pr": { + "url": "https://github.com/apache/spark/pull/52460", + "owner": "apache", + "repo": "spark", + "number": 52460, + "title": "[SPARK-53720][SQL] Simplify extracting Table from DataSourceV2Relation", + "files_changed": 13, + "additions": 33, + "deletions": 32, + "language": "Scala" + }, + "baseline_review": { + "comments": [ + { + "file": "sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala", + "severity": "positive", + "comment": "The new ExtractV2Table extractor object is a clean Scala idiom that encapsulates the pattern of extracting a Table from a DataSourceV2Relation (or a RowLevelOperationTable wrapping one). This eliminates the need to destructure all 5 fields of DataSourceV2Relation when only the table is needed." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromTable.scala", + "line": 43, + "severity": "positive", + "comment": "Three pattern matches in this file are simplified from verbose 5-tuple destructuring to concise ExtractV2Table usage. Readability is significantly improved." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromTable.scala", + "line": 34, + "severity": "minor", + "comment": "The import changes from importing just DataSourceV2Relation to importing both DataSourceV2Relation and ExtractV2Table. Several catalyst analysis files now depend on ExtractV2Table from the execution.datasources.v2 package. This follows the existing pattern where catalyst already imports DataSourceV2Relation from that package, but it deepens the cross-layer dependency." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala", + "line": 30, + "severity": "minor", + "comment": "The patterns.scala file gains an import of ExtractV2Table. The GroupBasedRowLevelOperation extractor is simplified, but the diff is truncated so the full extent of changes in this file cannot be verified. Reviewers should confirm the extractor handles both DataSourceV2Relation and RowLevelOperationTable cases that the original code matched." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala", + "line": 78, + "severity": "positive", + "comment": "The merge rewrite logic is simplified identically to the delete and update rewrites, maintaining consistency across all three row-level operation rewrite rules." + } + ], + "summary": "This is a clean, behavior-preserving refactoring that introduces an ExtractV2Table Scala extractor to simplify 13 files of pattern matching on DataSourceV2Relation. The only minor concern is the continued deepening of catalyst-to-execution imports, though this follows existing precedent." + }, + "flow_guided_review": { + "comments": [ + { + "file": "sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala", + "severity": "positive", + "comment": "The flow plan is empty (zero steps, zero clusters, zero dependencies), confirming this is a pure mechanical refactoring with no behavioral changes. The ExtractV2Table extractor is the sole new abstraction; every other file change is a consumer of it." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromTable.scala", + "line": 43, + "severity": "minor", + "comment": "Without flow data to trace dependencies, the reviewer must manually verify that ExtractV2Table's unapply correctly handles the RowLevelOperationTable case. In the original code, some matches like SupportsDeleteV2 only matched plain DataSourceV2Relation, while the extractor may now also match RowLevelOperationTable-wrapped relations. If the extractor broadens the match scope, this could subtly change behavior." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteUpdateTable.scala", + "line": 100, + "severity": "positive", + "comment": "The update rewrite simplification mirrors the delete and merge rewrites exactly, confirming the mechanical nature of this refactoring across all three DML operations." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala", + "severity": "minor", + "comment": "The diff is truncated for this file, so the full change to GroupBasedRowLevelOperation cannot be reviewed. Given the empty flow plan provides no guidance on how this file connects to the others, the reviewer must manually verify the extractor usage in the planning layer matches the analysis layer patterns." + }, + { + "file": "sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala", + "line": 78, + "severity": "positive", + "comment": "Consistent application of ExtractV2Table across all rewrite rules. The empty flow plan means no risk dependencies were identified, which is appropriate for a refactoring of this nature." + } + ], + "summary": "With an empty flow plan providing no structural guidance, this review falls back to manual analysis of the refactoring. The key risk is whether ExtractV2Table's match scope exactly preserves the original behavior, particularly around RowLevelOperationTable handling, which cannot be fully verified from the truncated diff." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 6, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 5, + "efficiency": 7, + "overall": 5.4 + }, + "flow_guided_scores": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 6, + "overall": 5.8 + }, + "reasoning": "The flow plan is completely empty, providing zero structural guidance for the review. Both reviews identify the same core observations about this clean refactoring. The flow-guided review edges ahead slightly because it explicitly calls out the risk that ExtractV2Table may broaden match scope (RowLevelOperationTable handling) and notes the truncated diff as a limitation more precisely. However, with no flow data to leverage, the advantage is marginal. Both reviews are hampered by the truncated diff which hides roughly half the changed files.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T18:30:00.000000+00:00" +} \ No newline at end of file diff --git a/evals/axios__axios__10582.json b/evals/axios__axios__10582.json new file mode 100644 index 0000000..c164475 --- /dev/null +++ b/evals/axios__axios__10582.json @@ -0,0 +1,187 @@ +{ + "pr": { + "url": "https://github.com/axios/axios/pull/10582", + "owner": "axios", + "repo": "axios", + "number": 10582, + "title": "feat: update sponsors script and how this works for more consistency", + "files_changed": 3, + "additions": 26, + "deletions": 14, + "language": "javascript" + }, + "timestamp": "2026-03-30T00:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 48, + "severity": "minor", + "comment": "The control flow after the sponsor content comparison changed subtly. Previously the 'up to date' log was inside an else block, so it only ran when content matched. Now it falls through after the if block and runs unconditionally -- including when content was outdated and the file was just rewritten. This means a misleading 'up to date' message will be logged right after writing an outdated file." + }, + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 30, + "severity": "minor", + "comment": "The setGithubOutput helper gracefully warns when GITHUB_OUTPUT is unset, which is good for local dev. However, it silently swallows the write without any further indication to the caller, so in CI if GITHUB_OUTPUT were accidentally unset the workflow would proceed with no changed output and silently skip all downstream steps." + }, + { + "file": ".github/workflows/update-sponsor-block.yml", + "line": 40, + "severity": "minor", + "comment": "The new readme-tracked-change step runs git diff --quiet -- README.md unconditionally, even when the script itself determined nothing changed. Consider gating this step on steps.sponsors-requires-update.outputs.changed == 'true' to avoid unnecessary work." + }, + { + "file": ".github/workflows/update-sponsor-block.yml", + "line": 54, + "severity": "nit", + "comment": "The compound condition steps.sponsors-requires-update.outputs.changed == 'true' && steps.readme-tracked-change.outputs.readme_changed == 'true' is repeated three times. Consider extracting it into a job-level env var or a single gating step to reduce duplication and risk of future divergence." + }, + { + "file": ".gitignore", + "line": 17, + "severity": "nit", + "comment": "Adding openspec/ to .gitignore seems unrelated to the sponsors workflow changes. If this is intentional, it should be mentioned in the PR description for traceability." + }, + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 1, + "severity": "positive", + "comment": "Good cleanup: removing the dependency on exec and colorize helpers simplifies the script and removes shell execution, reducing the attack surface in the CI environment." + } + ], + "summary": "The PR simplifies the sponsors update workflow by replacing shell exec with a native fs-based GitHub output helper and adding a git-diff guard to prevent noisy PRs. However, the refactored control flow in updateReadmeSponsors introduces a subtle logging bug where the 'up to date' message fires even after an update." + }, + "flow_guided_review": { + "comments": [ + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 30, + "severity": "minor", + "comment": "Starting from the entry point setGithubOutput (high-risk per plan): this function is the sole mechanism for communicating the changed flag to the workflow. If GITHUB_OUTPUT is unset or the appendFile call fails silently, all downstream workflow steps will be skipped with no error. Consider throwing or returning a non-zero exit if the env var is missing in CI (detect via process.env.CI)." + }, + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 48, + "severity": "major", + "comment": "Following the plan's second step into updateReadmeSponsors (high-risk entry point): the removal of the else branch changes behavior. When currentSponsorContent !== sponsorContent is true, the file is written and sponsorContent is returned -- but execution falls through to the console.log('up to date') line. This means the function logs 'up to date' immediately after detecting and writing an outdated sponsor block. The return sponsorContent on line 49 prevents this in practice, but the intent is misleading and fragile. A proper else or early return after the log would be clearer." + }, + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 67, + "severity": "minor", + "comment": "Tracing the flow from updateReadmeSponsors to the IIFE caller: setGithubOutput is called with changed=true/false based on newContent. But the workflow now also checks readme_changed from git diff. If the script writes the file but git diff shows no change (e.g., content is byte-identical after re-serialization), the PR step is correctly skipped. This dual-gate is the key improvement but means the changed output from the script alone is no longer sufficient -- document this interaction." + }, + { + "file": ".github/workflows/update-sponsor-block.yml", + "line": 40, + "severity": "minor", + "comment": "The readme-tracked-change step is not gated on the script's changed output. This means git diff runs even when the script determined nothing changed. While harmless, gating it on steps.sponsors-requires-update.outputs.changed == 'true' would make the dependency chain explicit and avoid unnecessary computation." + }, + { + "file": ".github/workflows/update-sponsor-block.yml", + "line": 54, + "severity": "nit", + "comment": "The triple-repeated compound condition across steps Read sponsors.md, Echo sponsors content, and Create pull request is a maintenance risk. If either output name changes, all three must be updated in sync. Consider a single job-level condition or a dedicated gating step." + }, + { + "file": ".gitignore", + "line": 17, + "severity": "nit", + "comment": "The openspec/ gitignore entry is unrelated to the sponsors workflow changes. This should either be split into a separate commit or noted in the PR description." + }, + { + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "line": 1, + "severity": "positive", + "comment": "Removing exec and colorize dependencies is a meaningful security and maintainability improvement. The script no longer shells out, eliminating a class of injection risks in CI." + } + ], + "summary": "Following the plan's risk-ordered traversal reveals that the two high-risk entry points (setGithubOutput and updateReadmeSponsors) interact through a dual-gate mechanism in the workflow that is the PR's key design improvement. The main code-level issue is a subtle control flow change in updateReadmeSponsors where removing the else block creates a misleading log path, though the early return prevents incorrect behavior at runtime." + }, + "review_plan": { + "stats": { + "totalSteps": 3, + "totalAdditions": 12, + "totalDeletions": 4, + "independentFlows": 2, + "filesChanged": 1 + }, + "steps": [ + { + "order": 1, + "nodeId": "scripts/sponsors/update-readme-sponsors.mjs::setGithubOutput", + "name": "setGithubOutput", + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "lines": [28, 35], + "type": "function", + "changeType": "modified", + "additions": 8, + "deletions": 0, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 2, + "nodeId": "scripts/sponsors/update-readme-sponsors.mjs::updateReadmeSponsors", + "name": "updateReadmeSponsors", + "file": "scripts/sponsors/update-readme-sponsors.mjs", + "lines": [37, 62], + "type": "function", + "changeType": "modified", + "additions": 4, + "deletions": 4, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": ["scripts/sponsors/update-readme-sponsors.mjs::getWithRetry"], + "riskReasons": ["entry_point"] + } + ], + "clusters": [ + { + "id": 0, + "label": "update-readme-sponsors.mjs", + "nodeIds": [ + "scripts/sponsors/update-readme-sponsors.mjs::updateReadmeSponsors", + "scripts/sponsors/update-readme-sponsors.mjs::getWithRetry" + ], + "reason": "2 related functions in update-readme-sponsors.mjs", + "suggestedReviewOrder": [ + "scripts/sponsors/update-readme-sponsors.mjs::updateReadmeSponsors", + "scripts/sponsors/update-readme-sponsors.mjs::getWithRetry" + ] + } + ], + "dependencies": [ + { + "from": "scripts/sponsors/update-readme-sponsors.mjs::updateReadmeSponsors", + "to": "scripts/sponsors/update-readme-sponsors.mjs::getWithRetry", + "reason": "Review updateReadmeSponsors before getWithRetry -- updateReadmeSponsors calls getWithRetry." + } + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.8 + }, + "reasoning": "The baseline review correctly identifies the control flow bug, the unrelated gitignore change, and the repeated conditions, but treats each comment in isolation without tracing how data flows between setGithubOutput, updateReadmeSponsors, and the workflow steps. The flow-guided review follows the plan's risk-ordered traversal, starting from setGithubOutput as the high-risk entry point, then tracing into updateReadmeSponsors and its callers, revealing the dual-gate interaction between the script's changed output and the workflow's readme_changed check. This cross-file flow awareness led to a more precise diagnosis of the control flow issue (elevated to major) and a concrete suggestion about CI-specific error handling in setGithubOutput. The flow-guided review also identified the documentation gap around the dual-gate mechanism, which the baseline missed entirely. Both reviews share the same nits about repeated conditions and the unrelated gitignore entry, showing similar efficiency. Overall, the flow-guided review demonstrates stronger risk identification and actionability by grounding comments in the dependency chain rather than reviewing files in isolation.", + "winner": "flow_guided" + } +} diff --git a/evals/babel__babel__17887.json b/evals/babel__babel__17887.json new file mode 100644 index 0000000..98675dd --- /dev/null +++ b/evals/babel__babel__17887.json @@ -0,0 +1,108 @@ +{ + "pr": "babel/babel#17887", + "title": "chore: Clean up ESM tests", + "baseline_review": { + "comments": [ + { + "file": "packages/babel-core/test/async.js", + "line": "238-254", + "severity": "major", + "comment": "Replacing `spawnTransformSync()` and `spawnTransformAsync()` (which spawned separate processes) with direct `babel.transformSync(\"\")` and `babel.transformAsync(\"\")` calls changes the test semantics significantly. The spawned helpers likely tested ESM loading in a clean child process, isolating module resolution. Calling babel directly in the same process may not exercise the same ESM loading paths, potentially reducing test coverage for native ESM plugin/preset loading." + }, + { + "file": "packages/babel-core/test/async.js", + "line": "238", + "severity": "minor", + "comment": "The `transformSync` test case still has `async` in its function signature (`async () => {`) but no longer uses `await`. The `async` keyword is now unnecessary and could be removed for clarity." + }, + { + "file": "packages/babel-core/test/config-chain.js", + "line": "1-2", + "severity": "positive", + "comment": "Replacing the hand-rolled `pfs` polyfill (which shimmed `fs.promises` for ancient Node versions) with a direct `import pfs from 'node:fs/promises'` is a good cleanup. The polyfill was unnecessary given Babel's current minimum Node version requirements." + }, + { + "file": "packages/babel-core/test/config-chain.js", + "line": "1172-1179", + "severity": "minor", + "comment": "The `isMJS` check and `esm` parameter have been removed from the `loadOptionsAsync` test. Previously, the helper dispatched differently based on whether the config file was `.mjs`. If the `loadOptionsAsync` helper was doing something special for `.mjs` files (e.g., spawning a subprocess), this change may reduce coverage for the `.mjs` config loading path specifically." + }, + { + "file": "packages/babel-core/test/async.js", + "line": "262-268", + "severity": "minor", + "comment": "The parallel transform test now inlines `Promise.all([babel.transformAsync(\"\"), babel.transformAsync(\"\")])` directly. This is cleaner than the previous `spawnTransformAsyncParallel()` helper call, and the test intent (two concurrent transforms) is more explicit." + }, + { + "file": "packages/babel-core/test/async.js", + "line": "2-6", + "severity": "nit", + "comment": "The import block for `spawnTransformAsync`, `spawnTransformAsyncParallel`, and `spawnTransformSync` from `./helpers/esm.js` was removed, but it is unclear from this diff whether the `helpers/esm.js` file itself has been deleted or still exists with other exports. If the file is now unused, it should be removed entirely." + } + ], + "summary": "This PR replaces subprocess-spawning ESM test helpers with direct babel API calls and removes a legacy `fs.promises` polyfill. The main concern is whether the switch from child-process isolation to in-process calls still adequately tests native ESM module loading behavior, which was presumably the original purpose of the spawned helpers." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/babel-core/test/async.js", + "line": "238-268", + "severity": "major", + "comment": "Flow analysis: The review plan identifies `fixture` in config-chain.js as a medium-risk node called by both `config` and `loadOptions`. However, the more significant risk is in async.js where the entire test strategy changed. The `spawnTransformSync`/`spawnTransformAsync` helpers existed to test ESM loading in isolated subprocess contexts, which is important because ESM module resolution can behave differently across process boundaries (e.g., module cache, loader hooks). Replacing these with in-process `babel.transformSync('')`/`babel.transformAsync('')` fundamentally changes what is being tested -- from 'can babel load .mjs plugins in a fresh process' to 'can babel load .mjs plugins in the current test process'. If the test process already has these modules cached, the native ESM loading path may be bypassed entirely." + }, + { + "file": "packages/babel-core/test/config-chain.js", + "line": "13-18", + "severity": "positive", + "comment": "The removal of the `pfs` polyfill proxy is well-justified. The polyfill included a comment referencing Node 8 compatibility (`TODO: In Babel 8, we can directly uses fs.promises which is supported by node 8+`), and Babel's current minimum Node version is well above that. The direct `import pfs from 'node:fs/promises'` is cleaner and removes ~30 lines of dead compatibility code." + }, + { + "file": "packages/babel-core/test/config-chain.js", + "line": "1172-1179", + "severity": "minor", + "comment": "The `isMJS` gate and custom `loadOptionsAsync` helper were removed in favor of calling `babel.loadOptionsAsync()` directly. The `isMJS` function was used to dispatch differently for .mjs config files. Since the plan identifies `loadOptions` as a caller of `fixture` and part of the same cluster, this change is safe only if babel's `loadOptionsAsync` now handles .mjs configs natively without needing subprocess isolation. The test still iterates over both .js and .mjs config names, so coverage is maintained at the API level." + }, + { + "file": "packages/babel-core/test/async.js", + "line": "320", + "severity": "minor", + "comment": "The preset-mjs-native `transformSync` test now calls `babel.transformSync('')` directly and drops the `await`/`resolves` chain. While functionally correct (sync calls don't return promises), the enclosing test function remains `async`. This is harmless but unnecessary -- consider making it a plain sync test function for clarity." + }, + { + "file": "packages/babel-core/test/async.js", + "line": "336", + "severity": "minor", + "comment": "The 'must use the default export' test for named-exports presets now calls `babel.transformAsync('')` directly. This test verifies that a preset using named exports instead of default export throws an error. The behavioral assertion (`.rejects.toThrow`) is preserved, so this simplification is safe -- the error originates from babel's config loading, not from process spawning." + }, + { + "file": "packages/babel-core/test/async.js", + "line": "2-6", + "severity": "minor", + "comment": "The `helpers/esm.js` import is fully removed. The diff does not show deletion of the helper file itself. If `helpers/esm.js` has no remaining consumers, it should be deleted to avoid dead code. If other test files still import from it, this cleanup is incomplete." + } + ], + "summary": "This cleanup PR removes subprocess-based ESM test helpers and a legacy Node polyfill, replacing them with direct babel API calls. The primary risk is that the subprocess helpers existed specifically to test ESM module loading in isolation, and replacing them with in-process calls may reduce coverage of the native ESM loading path -- though this may be intentional if Babel's ESM support has matured enough that subprocess isolation is no longer necessary." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review benefits from understanding the dependency relationships in config-chain.js (fixture called by config and loadOptions) and uses that context to assess whether the removal of the isMJS gate is safe. It also provides a more nuanced analysis of the subprocess-to-inprocess migration by reasoning about module cache behavior and ESM loader isolation. The baseline review correctly identifies the same core risk (loss of subprocess isolation) but treats each test change independently rather than connecting them through the shared test infrastructure pattern. Both reviews catch the dead helper file concern and the unnecessary async keyword, but the flow-guided review better explains *why* the subprocess approach existed and what specific coverage might be lost. The PR is relatively straightforward cleanup so the advantage is moderate, not dramatic." + } +} diff --git a/evals/babel__babel__17901.json b/evals/babel__babel__17901.json new file mode 100644 index 0000000..518e04e --- /dev/null +++ b/evals/babel__babel__17901.json @@ -0,0 +1,108 @@ +{ + "pr": "babel/babel#17901", + "title": "chore(parser): parseClass typings cleanup", + "baseline_review": { + "comments": [ + { + "file": "packages/babel-parser/src/parser/statement.ts", + "line": 1764, + "severity": "minor", + "comment": "The five local aliases (`publicMethod`, `privateMethod`, `publicProp`, `privateProp`, `accessorProp`) are now cast to `Undone<...>` variants. While this is correct for type safety, the pattern of casting `member` to five different types on consecutive lines is a code smell that predates this PR. Consider leaving a TODO or comment noting that a discriminated union or type guard approach could eliminate these casts entirely." + }, + { + "file": "packages/babel-parser/src/parser/expression.ts", + "line": 2133, + "severity": "minor", + "comment": "The parameter type for `getGetterSetterExpectedParamCount` is changed to `Undone`. Since the method body only accesses `method.kind`, verify that `Undone` preserves the `kind` discriminant property -- if `Undone` strips it or makes it optional, the comparison `method.kind === 'get'` could become unsound." + }, + { + "file": "packages/babel-parser/src/parser/statement.ts", + "line": 2130, + "severity": "minor", + "comment": "The `parseClassProperty` return type remains `N.ClassProperty` (the finished type) while accepting `Undone`. This is correct because `finishNode` converts `Undone` to `T`, but it would be worth confirming the same pattern holds for `parseClassAccessorProperty` at line 2143." + }, + { + "file": "packages/babel-parser/src/plugins/flow/index.ts", + "line": 2633, + "severity": "minor", + "comment": "Both `pushClassMethod` and `pushClassPrivateMethod` in the Flow plugin update `classBody` from `N.ClassBody` to `Undone` and `method` to their respective `Undone` variants. Ensure the base class signatures in statement.ts were also updated (the diff for `pushClassMethod`/`pushClassPrivateMethod` in statement.ts is not shown), otherwise TypeScript would flag a signature mismatch in the override." + }, + { + "file": "packages/babel-parser/src/plugins/estree.ts", + "line": 168, + "severity": "minor", + "comment": "The ESTree override of `getObjectOrClassMethodParams` casts `method` to `N.EstreeMethodDefinition` via `unknown`. Since the input is now `Undone`, the double cast through `unknown` remains necessary but is slightly more concerning -- the `Undone` wrapper means the node is not yet finalized, so accessing `.value.params` on the ESTree shape assumes those properties are populated before `finishNode`." + } + ], + "summary": "This PR systematically updates parameter types in the babel-parser's class parsing methods from finished node types to their `Undone<...>` (in-progress) counterparts, improving type accuracy for nodes that haven't yet been finalized via `finishNode`. The changes are mechanical and low-risk, though a few cast-through-unknown patterns in plugin overrides deserve attention to ensure Undone nodes have the accessed properties populated." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/babel-parser/src/plugins/estree.ts", + "line": 168, + "severity": "minor", + "comment": "Entry point (order 1): The ESTree `getObjectOrClassMethodParams` override now accepts `Undone` but immediately casts to `N.EstreeMethodDefinition` via `unknown`. Since this is called by `checkGetterSetterParams` (order 164+) during parsing before `finishNode`, the `Undone` type is semantically correct. However, the cast chain `Undone -> unknown -> EstreeMethodDefinition` masks any type-level guarantees -- if the ESTree method definition shape differs from the Undone intermediate, this would silently produce incorrect behavior." + }, + { + "file": "packages/babel-parser/src/plugins/flow/index.ts", + "line": 2727, + "severity": "minor", + "comment": "Entry point (order 2): Flow's `checkGetterSetterParams` override calls the base `checkGetterSetterParams` (expression.ts) and also `getObjectOrClassMethodParams` and `isThisParam`. The plan correctly identifies this as high risk due to being an entry point with three downstream calls. The type change to `Undone` is consistent across the call chain -- base `checkGetterSetterParams` (expression.ts:2154) and `getObjectOrClassMethodParams` (expression.ts:2146) both accept `Undone` now." + }, + { + "file": "packages/babel-parser/src/plugins/typescript/index.ts", + "line": 3433, + "severity": "minor", + "comment": "Entry point (order 3): TypeScript plugin's `parseClassProperty` override changes `node` from `N.ClassProperty` to `Undone`. This flows down to `parseClassPropertyAnnotation` (TS-specific) and the base `parseClassProperty` (statement.ts:2130). The base method also updated its signature to `Undone`, so the override chain is consistent. The return type correctly remains `N.ClassProperty` since `finishNode` is called." + }, + { + "file": "packages/babel-parser/src/plugins/typescript/index.ts", + "line": 4167, + "severity": "minor", + "comment": "Entry point (order 6): TypeScript's `getGetterSetterExpectedParamCount` override calls the base implementation and also accesses `getObjectOrClassMethodParams` and `isThisParam`. All three callees now accept `Undone` types. The `this` parameter filtering logic in TS (checking `isThisParam`) accesses `params[0]` which must exist on the Undone node -- this is safe because params are populated during `parseClassMethod` before `checkGetterSetterParams` is called." + }, + { + "file": "packages/babel-parser/src/parser/statement.ts", + "line": 1764, + "severity": "minor", + "comment": "Internal node (order 164+): The five cast aliases in `parseClassMemberWithIsStatic` are updated to `Undone<...>` types. These flow downstream to `pushClassMethod`, `pushClassProperty`, `pushClassAccessorProperty`, etc. The plan reveals these are intermediate nodes feeding multiple leaf operations. The `Undone` type correctly reflects that `member` is a `startNode` result that hasn't been finalized." + }, + { + "file": "packages/babel-parser/src/parser/statement.ts", + "line": 1998, + "severity": "minor", + "comment": "Leaf node: `pushClassProperty` now accepts `Undone`. It accesses `prop.computed` and `prop.key` which must be populated on the Undone node. Since class property key parsing occurs before `pushClassProperty` is called, these properties are guaranteed to be set. The method pushes to `classBody.body` which is also Undone -- consistent typing throughout." + }, + { + "file": "packages/babel-parser/src/parser/expression.ts", + "line": 2146, + "severity": "nit", + "comment": "The `getObjectOrClassMethodParams` method simply returns `method.params`. With the `Undone` wrapper, the return type implicitly changes. If any caller expects a specific array type from the finished node, this could cause downstream type issues. In practice this is fine since params are populated during method parsing before this accessor is called." + } + ], + "summary": "Following the review plan's dependency graph from 6 entry points (ESTree, Flow, TypeScript plugins) through internal nodes (statement.ts class member parsing) to leaf operations, the `Undone` type changes are consistent across all override chains and call hierarchies. The plan's risk flags on entry points are justified since signature mismatches in any override would break compilation, but all overrides in the diff are updated in lockstep." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review significantly outperforms the baseline by leveraging the dependency graph to trace type changes across the override hierarchy (ESTree, Flow, TypeScript plugins all overriding base parser methods). It systematically verifies that each entry point's signature change is consistent with its callees, confirming no signature mismatches exist across the 6 identified entry points and their downstream dependencies. The baseline review raises valid concerns (Undone property access, cast-through-unknown patterns) but treats each file change in isolation without understanding that these methods form override chains where type consistency across the hierarchy is the primary correctness concern. The flow-guided review's ordered traversal naturally surfaces the most important insight: all override chains are updated in lockstep, making this a safe mechanical refactoring." + } +} \ No newline at end of file diff --git a/evals/celery__celery__10206.json b/evals/celery__celery__10206.json new file mode 100644 index 0000000..5e1469a --- /dev/null +++ b/evals/celery__celery__10206.json @@ -0,0 +1,108 @@ +{ + "pr": "celery/celery#10206", + "title": "Fix#10022 request time limits", + "baseline_review": { + "comments": [ + { + "file": "celery/app/task.py", + "line": "131-143", + "severity": "major", + "comment": "The identity-based check `new_timelimit is not old_timelimit` can produce false negatives. If the caller passes the same list/tuple object that is already stored in `__dict__` (e.g., mutating it externally then calling `update` again with the same reference), the identity check will see them as identical and skip unpacking. While unlikely in normal Celery flows, this is a subtle contract change from the previous passthrough `dict.update`. An equality check or always-unpack-when-present approach using a key-existence test (`'timelimit' in merged_args`) would be more robust." + }, + { + "file": "celery/app/task.py", + "line": "135-136", + "severity": "minor", + "comment": "The unpacking `new_timelimit[0], new_timelimit[1]` silently ignores any elements beyond index 1. If someone accidentally passes a 3-element tuple, the extra value is lost without warning. This is acceptable for a 2-item contract but a brief comment noting the expected `(hard, soft)` structure would clarify intent." + }, + { + "file": "celery/app/task.py", + "line": "907-910", + "severity": "major", + "comment": "In `apply()`, the timelimit is constructed as `[self.time_limit, self.soft_time_limit]` (a list), but the existing protocol uses tuples for timelimit in non-eager paths. While `Context.update` handles both lists and tuples, other consumers of the request info dict may expect a tuple. Consider using a tuple `(self.time_limit, self.soft_time_limit)` for consistency with the wire protocol." + }, + { + "file": "celery/app/task.py", + "line": "907-910", + "severity": "minor", + "comment": "The condition `self.time_limit is None and self.soft_time_limit is None` means that if only one limit is set (e.g., `time_limit=60, soft_time_limit=None`), a timelimit of `[60, None]` is passed. This is correct behavior, but the `None` sentinel in the first branch could be confusing to readers. A comment explaining that `None` means 'no timelimit override' would help." + }, + { + "file": "docs/userguide/tasks.rst", + "line": "348-351", + "severity": "major", + "comment": "The documentation changed `:timelimit:` from `(soft, hard)` to `(hard, soft)`. This is a documentation correction (the actual wire format is indeed `(hard, soft)`), but this is a breaking change to the docs that may confuse users who relied on the previous (incorrect) documentation. The change should be explicitly called out in a changelog note or versionchanged directive so users know the previous docs were wrong." + }, + { + "file": "celery/app/task.py", + "line": "109-110", + "severity": "minor", + "comment": "Adding `time_limit = None` and `soft_time_limit = None` as class-level attributes on `Context` means they will appear in `__dict__` after any attribute access. This is fine for the sentinel pattern, but these attributes now shadow the `timelimit` tuple's contents. If someone sets `ctx.time_limit = 30` directly without going through `update()`, it will be out of sync with `ctx.timelimit`. Consider adding a note or property that warns against direct assignment." + } + ], + "summary": "The PR correctly fixes three independent bugs preventing `task.request.time_limit` and `task.request.soft_time_limit` from being populated. The main concerns are the identity-based change detection in `Context.update()` which has edge-case fragility, and the docs correction for `timelimit` tuple order that should be more prominently flagged as a documentation fix." + }, + "flow_guided_review": { + "comments": [ + { + "file": "celery/app/task.py", + "line": "129-143", + "severity": "major", + "comment": "The `Context.update()` method is the central synchronization point called by all test paths (steps 2-7 in the plan) and by the worker's request setup. The identity-based sentinel check (`new_timelimit is not old_timelimit`) is an O(1) optimization, but it introduces a subtle contract: the method only unpacks timelimit when the object identity changes. This means if a caller mutates the timelimit list in-place and re-calls update with the same dict, the change is silently ignored. Since this method is the single funnel through which all timelimit propagation flows, a key-presence check on the input arguments would be safer. The `_UNSET` sentinel is well-designed for the 'no timelimit key at all' case, but the identity comparison conflates 'same object' with 'unchanged value'." + }, + { + "file": "celery/app/task.py", + "line": "907-910", + "severity": "major", + "comment": "The `apply()` method (step 10 in the plan, called by eager test paths in steps 11-12) now injects `timelimit` into the request dict. This flows into `Context.update()`, which then unpacks it into `time_limit` and `soft_time_limit`. However, the value is constructed as a list `[self.time_limit, self.soft_time_limit]` while the non-eager code path (worker message protocol) sends tuples. The `Context.update` handles both via `isinstance(..., (list, tuple))`, but downstream code that checks `isinstance(timelimit, tuple)` specifically would break for eager tasks. This asymmetry between eager and worker paths is a latent bug." + }, + { + "file": "celery/app/task.py", + "line": "395-396", + "severity": "minor", + "comment": "The `from_config` additions (`time_limit` -> `task_time_limit`, `soft_time_limit` -> `task_soft_time_limit`) correctly fix Bug 1 from the PR description. These feed into `Task.bind()` which copies config values to task attributes, which then flow into `apply()` (step 10). The dependency chain is sound: config -> Task attrs -> apply() -> Context.update() -> request.time_limit. No issues here, but worth noting this is the root of the fix chain." + }, + { + "file": "t/unit/tasks/test_context.py", + "line": "92-142", + "severity": "minor", + "comment": "The unit tests (steps 2-7) thoroughly cover `Context.update()` with dict, list-of-pairs, dict_items, kwargs, None-clearing, and no-timelimit scenarios. However, they all test the happy path where timelimit is a 2-element sequence. There is no test for a 1-element timelimit (e.g., `(60,)`) which would fail at `new_timelimit[1]` with an IndexError. The `len(new_timelimit) >= 2` guard handles this by falling through to the else branch (clearing both to None), but this silent data loss should be tested explicitly." + }, + { + "file": "docs/userguide/tasks.rst", + "line": "348-351", + "severity": "major", + "comment": "The documentation corrects the timelimit tuple order from `(soft, hard)` to `(hard, soft)`. This is factually correct per the wire protocol, but it is a silent documentation breaking change. Users who wrote code based on the old docs (e.g., `hard, soft = self.request.timelimit`) would have had their variables swapped all along, and this correction may cause them to 'fix' working code. A `.. versionchanged:: 5.7` directive explaining the documentation correction would prevent confusion." + }, + { + "file": "t/integration/test_tasks.py", + "line": "529-563", + "severity": "minor", + "comment": "The integration tests (steps 8-12) validate the full end-to-end flow: apply_async with explicit limits, task-level declared limits, and eager mode. These tests call the helper tasks defined in `t/integration/tasks.py` which return `self.request.time_limit` and `self.request.soft_time_limit`. This is the strongest validation in the PR as it exercises the complete config -> bind -> apply -> Context.update -> request attribute chain. The coverage is good, though testing with only one limit set (e.g., `time_limit=60, soft_time_limit=None`) would strengthen the partial-config scenario." + } + ], + "summary": "The PR correctly fixes three independent bugs in the time-limit propagation chain from app config through Task.bind(), apply(), and Context.update() to request attributes. The primary risks are the identity-based change detection in Context.update() which could silently miss in-place mutations, the list-vs-tuple asymmetry between eager and worker paths, and a documentation order correction that should be flagged as a doc fix to avoid user confusion." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same core issues: the identity-based sentinel in Context.update(), the list-vs-tuple asymmetry in apply(), and the documentation order correction. The flow-guided review significantly outperforms on flow_awareness by tracing the complete propagation chain (config -> from_config -> Task.bind -> apply -> Context.update -> request attrs) and explaining how each change connects to the next in the dependency graph. It also better identifies risk by noting the 1-element timelimit edge case that the unit tests miss, and by framing the eager-vs-worker list/tuple asymmetry as a cross-path consistency issue rather than just a style concern. The baseline review treats each file change in isolation, missing the systemic relationship between the from_config fix, the apply() timelimit injection, and the Context.update() unpacking logic. The flow-guided review's comments are more actionable because they explain the consequences in terms of the full data flow, making it clearer why each suggestion matters." + } +} diff --git a/evals/cloudflare__workers-sdk__13115.json b/evals/cloudflare__workers-sdk__13115.json new file mode 100644 index 0000000..dae453d --- /dev/null +++ b/evals/cloudflare__workers-sdk__13115.json @@ -0,0 +1,264 @@ +{ + "pr": { + "url": "https://github.com/cloudflare/workers-sdk/pull/13115", + "owner": "cloudflare", + "repo": "workers-sdk", + "number": 13115, + "title": "Improve error message when the assets directory path points to a file instead of a directory", + "files_changed": 4, + "additions": 108, + "deletions": 10, + "language": "TypeScript" + }, + "timestamp": "2026-03-30T18:45:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "packages/wrangler/src/assets.ts", + "line": 436, + "severity": "positive", + "comment": "Replacing existsSync with statSync({ throwIfNoEntry: false }) is a solid improvement. It collapses existence check and type check into a single syscall, avoids a TOCTOU race condition, and provides the stat result needed for the new directory validation." + }, + { + "file": "packages/wrangler/src/assets.ts", + "line": 450, + "severity": "positive", + "comment": "The new NonDirectoryAssetsDirError mirrors the existing NonExistentAssetsDirError pattern nicely, extending UserError for clean error handling. The error message correctly differentiates between --assets flag and assets.directory config sources." + }, + { + "file": "packages/wrangler/src/assets.ts", + "line": 110, + "severity": "nit", + "comment": "The template literal reformatting in syncAssets, doUpload, logReadFilesFromDirectory, and logAssetsUploadStatus is purely cosmetic with no behavioral change. These formatting-only changes add noise to the diff and would be better as a separate commit." + }, + { + "file": "packages/wrangler/src/__tests__/deploy/assets.test.ts", + "line": 249, + "severity": "positive", + "comment": "Good test coverage for both the --assets flag path and the assets.directory config path during deploy. Using inline snapshot matching ensures the exact error message wording is validated." + }, + { + "file": "packages/wrangler/src/__tests__/dev.test.ts", + "line": 2297, + "severity": "minor", + "comment": "The dev test for the config-based path creates a main: './index.js' and writes index.js, but the deploy test for the same scenario does not. This inconsistency suggests the deploy test might be relying on implicit behavior that could break if defaults change." + }, + { + "file": ".changeset/wrangler-assets-not-a-directory-error.md", + "line": 1, + "severity": "positive", + "comment": "Changeset correctly categorized as a patch with a clear description explaining the before/after behavior." + } + ], + "summary": "This is a clean UX improvement that detects when the assets path points to a file instead of a directory, replacing an opaque ENOTDIR error with a clear user-facing message. The implementation uses statSync elegantly and has thorough test coverage across both deploy and dev workflows, though the unrelated formatting changes add unnecessary diff noise." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/wrangler/src/assets.ts", + "line": 433, + "severity": "positive", + "comment": "The plan identifies getAssetsOptions as a high-risk entry point that calls getAssetsBasePath then validates. The refactored logic correctly resolves the path first via getAssetsBasePath, then performs the statSync check, ensuring the fully-resolved absolute path is validated. The sourceOfTruthMessage extraction before the conditional branches is clean." + }, + { + "file": "packages/wrangler/src/assets.ts", + "line": 403, + "severity": "positive", + "comment": "NonDirectoryAssetsDirError is a leaf node in the plan's dependency graph, called only from getAssetsOptions. It follows the same UserError extension pattern as NonExistentAssetsDirError, maintaining consistency in the error class hierarchy." + }, + { + "file": "packages/wrangler/src/assets.ts", + "line": 110, + "severity": "nit", + "comment": "The plan flags syncAssets as a high-risk entry point, but these changes are purely template literal reformatting with zero behavioral impact. The risk classification is misleading here since the actual risk is in cluster 1 (getAssetsOptions), not cluster 0." + }, + { + "file": "packages/wrangler/src/assets.ts", + "line": 218, + "severity": "nit", + "comment": "The doUpload function (internal node, called by syncAssets) only has formatting changes to the template literal in the FatalError message. No functional change to the upload retry logic or JWT expiration handling." + }, + { + "file": "packages/wrangler/src/__tests__/dev.test.ts", + "line": 2277, + "severity": "minor", + "comment": "The plan's two independent flows map to deploy and dev entry paths. Tests cover both, but neither test verifies the behavior when the path does not exist at all (as opposed to being a file), which would exercise the NonExistentAssetsDirError path in the same function. Confirming the existing path-not-found tests still pass would strengthen confidence." + }, + { + "file": "packages/wrangler/src/assets.ts", + "line": 436, + "severity": "minor", + "comment": "Following the plan's call chain (getAssetsOptions -> getAssetsBasePath -> statSync), the stat result could theoretically be null if the file is deleted between getAssetsBasePath resolving and statSync executing (TOCTOU). However, this is the same race window as the original existsSync approach and is acceptable for a CLI tool." + }, + { + "file": "packages/wrangler/src/__tests__/deploy/assets.test.ts", + "line": 249, + "severity": "positive", + "comment": "Deploy tests cover the two distinct config sources (--assets flag and assets.directory config) that feed into getAssetsOptions. The inline snapshots lock down the exact error message format including the resolved absolute path." + } + ], + "summary": "The plan's two clusters cleanly separate cosmetic formatting (cluster 0: syncAssets and helpers) from the actual feature (cluster 1: getAssetsOptions validation chain). The high-risk entry point getAssetsOptions is well-tested through both deploy and dev paths, and the new NonDirectoryAssetsDirError leaf node follows established error patterns, though the formatting changes in cluster 0 inflate the diff without adding value." + }, + "review_plan": { + "stats": { + "totalSteps": 11, + "totalAdditions": 36, + "totalDeletions": 10, + "independentFlows": 2, + "filesChanged": 1 + }, + "steps": [ + { + "order": 1, + "nodeId": "packages/wrangler/src/assets.ts::syncAssets", + "name": "syncAssets", + "file": "packages/wrangler/src/assets.ts", + "lines": [55, 271], + "type": "function", + "changeType": "modified", + "additions": 9, + "deletions": 3, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [ + "packages/wrangler/src/assets.ts::buildAssetManifest", + "packages/wrangler/src/assets.ts::logAssetUpload", + "packages/wrangler/src/assets.ts::doUpload" + ], + "riskReasons": ["entry_point"] + }, + { + "order": 2, + "nodeId": "packages/wrangler/src/assets.ts::getAssetsOptions", + "name": "getAssetsOptions", + "file": "packages/wrangler/src/assets.ts", + "lines": [405, 525], + "type": "function", + "changeType": "modified", + "additions": 17, + "deletions": 4, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [ + "packages/wrangler/src/assets.ts::getAssetsBasePath", + "packages/wrangler/src/assets.ts::NonExistentAssetsDirError", + "packages/wrangler/src/assets.ts::NonDirectoryAssetsDirError" + ], + "riskReasons": ["entry_point"] + }, + { + "order": 5, + "nodeId": "packages/wrangler/src/assets.ts::doUpload", + "name": "doUpload", + "file": "packages/wrangler/src/assets.ts", + "lines": [152, 232], + "type": "function", + "changeType": "modified", + "additions": 3, + "deletions": 1, + "role": "internal", + "risk": "low", + "calledBy": ["packages/wrangler/src/assets.ts::syncAssets"], + "calls": ["packages/wrangler/src/assets.ts::logAssetsUploadStatus"], + "riskReasons": [] + }, + { + "order": 8, + "nodeId": "packages/wrangler/src/assets.ts::NonDirectoryAssetsDirError", + "name": "NonDirectoryAssetsDirError", + "file": "packages/wrangler/src/assets.ts", + "lines": [403, 403], + "type": "class", + "changeType": "modified", + "additions": 1, + "deletions": 0, + "role": "leaf", + "risk": "low", + "calledBy": ["packages/wrangler/src/assets.ts::getAssetsOptions"], + "calls": [], + "riskReasons": [] + }, + { + "order": 9, + "nodeId": "packages/wrangler/src/assets.ts::logReadFilesFromDirectory", + "name": "logReadFilesFromDirectory", + "file": "packages/wrangler/src/assets.ts", + "lines": [369, 376], + "type": "function", + "changeType": "modified", + "additions": 3, + "deletions": 1, + "role": "leaf", + "risk": "low", + "calledBy": ["packages/wrangler/src/assets.ts::buildAssetManifest"], + "calls": [], + "riskReasons": [] + }, + { + "order": 11, + "nodeId": "packages/wrangler/src/assets.ts::logAssetsUploadStatus", + "name": "logAssetsUploadStatus", + "file": "packages/wrangler/src/assets.ts", + "lines": [351, 362], + "type": "function", + "changeType": "modified", + "additions": 3, + "deletions": 1, + "role": "leaf", + "risk": "low", + "calledBy": ["packages/wrangler/src/assets.ts::doUpload"], + "calls": [], + "riskReasons": [] + } + ], + "clusters": [ + { + "id": 0, + "label": "assets.ts", + "nodeIds": [ + "packages/wrangler/src/assets.ts::syncAssets", + "packages/wrangler/src/assets.ts::buildAssetManifest", + "packages/wrangler/src/assets.ts::logAssetUpload", + "packages/wrangler/src/assets.ts::doUpload", + "packages/wrangler/src/assets.ts::logReadFilesFromDirectory", + "packages/wrangler/src/assets.ts::errorOnLegacyPagesWorkerJSAsset", + "packages/wrangler/src/assets.ts::logAssetsUploadStatus" + ], + "reason": "7 related functions in assets.ts" + }, + { + "id": 1, + "label": "assets.ts", + "nodeIds": [ + "packages/wrangler/src/assets.ts::getAssetsOptions", + "packages/wrangler/src/assets.ts::getAssetsBasePath", + "packages/wrangler/src/assets.ts::NonExistentAssetsDirError", + "packages/wrangler/src/assets.ts::NonDirectoryAssetsDirError" + ], + "reason": "4 related functions in assets.ts" + } + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 5, + "actionability": 6, + "efficiency": 8, + "overall": 6.0 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + }, + "reasoning": "The baseline review correctly identifies the key changes and provides useful observations but treats the diff as a flat list of changes without understanding the relationship between functions. The flow-guided review leverages the plan's cluster separation to distinguish cosmetic formatting (cluster 0) from the actual feature logic (cluster 1), correctly identifies getAssetsOptions as the critical entry point, traces the call chain through getAssetsBasePath to the error classes, and notes the TOCTOU consideration. The flow-guided review also correctly observes that cluster 0's high-risk label on syncAssets is misleading since those changes are purely formatting. The flow-guided review loses slightly on efficiency due to more verbose analysis of low-risk formatting nodes, but its superior structural understanding of the change makes it the clear winner.", + "winner": "flow_guided" + } +} diff --git a/evals/containerd__containerd__13119.json b/evals/containerd__containerd__13119.json new file mode 100644 index 0000000..6ad4edd --- /dev/null +++ b/evals/containerd__containerd__13119.json @@ -0,0 +1,119 @@ +{ + "pr": "containerd/containerd#13119", + "title": "[release/2.1] Preserve cgroup mount options for privileged containers", + "baseline_review": { + "comments": [ + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 84, + "severity": "major", + "comment": "The LookupMount error is silently swallowed. If LookupMount fails for a transient reason (e.g., /proc/mounts temporarily unreadable), the container will be created with missing cgroup mount options and no indication of why. Consider logging a warning when err != nil so operators can diagnose unexpected option loss." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 80, + "severity": "minor", + "comment": "The condition `!hasCgroupNS` correctly gates the host-option-copying logic, but the code comment says 'mounting cgroup2 inside the container applies the new mount options to the single shared cgroup2 VFS superblock.' This is only accurate for cgroup v2 (unified hierarchy). If someone runs this code path on cgroup v1, the mount type is 'cgroup' not 'cgroup2', and the VFS superblock semantics differ. Consider guarding with a cgroup v2 check or documenting this assumption." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 85, + "severity": "minor", + "comment": "Using strings.SplitSeq to iterate VFSOptions and checking for exact matches 'nsdelegate' and 'memory_recursiveprot' is correct. However, if future kernel versions add additional mount options that affect the shared superblock, this allowlist will need updating. A comment noting which kernel versions introduced these options would help maintainability." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 41, + "severity": "minor", + "comment": "The test skips when RUNC_FLAVOR is 'crun' with a link to a crun PR. This skip condition should be revisited once that crun PR is merged, or it will silently skip the test forever. Consider adding a TODO or version check instead of an unconditional skip." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 48, + "severity": "nit", + "comment": "The host mount is read before container creation (hostMountBefore) and again after (hostMountAfter), then asserts options are preserved. This correctly validates the core behavior. However, the test does not verify the container's internal /sys/fs/cgroup mount options -- it only checks the host mount was not degraded. Adding an exec into the container to verify its cgroup mount also has the expected options would strengthen the test." + }, + { + "file": "Vagrantfile", + "line": 281, + "severity": "nit", + "comment": "RUNC_FLAVOR defaults to 'runc' when the env var is unset. This is consistent across both Vagrant provisioner blocks and supports the new integration test skip logic. Clean addition." + } + ], + "summary": "This cherry-pick preserves host cgroup mount options (nsdelegate, memory_recursiveprot) when privileged containers share the host cgroup namespace, preventing the kernel from stripping those options on the shared VFS superblock. The implementation is sound but silently swallows LookupMount errors, and the integration test only validates the host-side effect without checking the container's own mount options." + }, + "flow_guided_review": { + "comments": [ + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 75, + "severity": "major", + "comment": "The core logic builds cgroupOptions as a slice starting with default options, then conditionally appends host options. However, if both 'nsdelegate' and 'memory_recursiveprot' are present on the host, the resulting options list will have 7 entries. The OCI runtime spec does not restrict mount option count, but some container runtimes may not expect or handle these additional cgroup-specific options. This should be validated against runc's mount processing to confirm they are passed through to the mount(2) syscall correctly." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 78, + "severity": "major", + "comment": "The hasCgroupNS detection relies on s.Linux.Namespaces containing a CgroupNamespace entry. In the CRI flow for privileged containers, the cgroup namespace is typically not added to the spec. Verify this assumption holds for all CRI implementations and runtime classes -- if a runtime class adds a cgroup namespace while still being 'privileged', the options would not be copied, which could be the wrong behavior depending on kernel version." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 84, + "severity": "major", + "comment": "The LookupMount call uses osi (OS interface), making it mockable for unit tests. However, the error path silently proceeds without the host options. In production, a failed mount lookup for /sys/fs/cgroup would mean the container's cgroup mount silently drops nsdelegate/memory_recursiveprot -- the exact problem this PR is trying to fix. At minimum, log a warning; ideally, consider whether this should be a hard error for privileged containers." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 41, + "severity": "minor", + "comment": "The RUNC_FLAVOR skip is a pragmatic workaround for crun not yet supporting cgroup v2 mount options. The Vagrantfile changes to pass RUNC_FLAVOR through to the VM are the supporting infrastructure. This is well-coordinated, but the skip should have a follow-up tracking issue rather than just a PR link that may go stale." + }, + { + "file": "internal/cri/opts/spec_linux_test.go", + "line": 20, + "severity": "minor", + "comment": "The unit test file adds imports for mount, ostesting, and runtimespec, indicating new test cases for the cgroup option logic. The diff is truncated so we cannot verify the test coverage is complete, but the use of ostesting for mocking the OS interface is the correct pattern. Ensure tests cover: (1) no cgroup namespace with host options, (2) cgroup namespace present, (3) LookupMount failure, and (4) host mount without nsdelegate/memory_recursiveprot." + }, + { + "file": "Vagrantfile", + "line": 281, + "severity": "nit", + "comment": "RUNC_FLAVOR is added to both the integration test and CRI test provisioner environment blocks with a default of 'runc'. This is consistent and enables the integration test skip condition to work correctly in Vagrant-based CI." + } + ], + "summary": "This PR correctly identifies that privileged containers sharing the host cgroup namespace cause the kernel to strip nsdelegate and memory_recursiveprot from the shared cgroup2 superblock, and fixes it by copying those options from the host mount into the container spec. The main risk is the silent error swallowing in LookupMount and the assumption that cgroup namespace absence reliably indicates host-namespace sharing across all runtime configurations." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 5, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.2 + }, + "reasoning": "Both reviews identified the silent error swallowing in LookupMount as a key issue and noted the crun skip condition in tests. The flow-guided review provided deeper analysis despite the empty review plan: it examined the interaction between cgroup namespace detection and privileged container semantics, questioned whether the OCI runtime correctly processes the additional mount options, and explicitly enumerated the test cases that should exist for the truncated unit test file. The baseline review included a positive/nit comment on the Vagrantfile that added less value. The flow-guided review's comments on hasCgroupNS reliability and the production impact of silent LookupMount failure demonstrate stronger risk identification. Both reviews suffer from the empty flow plan (no steps, clusters, or dependencies to guide review order), so the flow_awareness scores are low for both, but the flow-guided review still produced more structured analysis by reasoning about component interactions.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T12:00:00Z" +} diff --git a/evals/containerd__containerd__13120.json b/evals/containerd__containerd__13120.json new file mode 100644 index 0000000..86e471f --- /dev/null +++ b/evals/containerd__containerd__13120.json @@ -0,0 +1,114 @@ +{ + "pr": "containerd/containerd#13120", + "title": "[release/2.2] Preserve cgroup mount options for privileged containers", + "baseline_review": { + "comments": [ + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 83, + "severity": "medium", + "comment": "The `LookupMount` error is silently swallowed. If the lookup fails for a reason other than the mount not existing (e.g., /proc/mounts unreadable due to permissions), the container will silently lose cgroup mount options without any log entry. Consider logging a warning when the error is non-nil so operators can diagnose why options were not preserved." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 84, + "severity": "low", + "comment": "The `strings.SplitSeq` function (from Go's `strings` package iter support) is relatively new. Confirm this is available in the Go version used by the release/2.2 branch. If not, `strings.Split` with a range loop would be a safer alternative for a backport." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 82, + "severity": "medium", + "comment": "The check `!hasCgroupNS` correctly gates the option-copying logic on the container sharing the host cgroup namespace. However, the comment says 'mounting cgroup2 inside the container applies the new mount options to the single shared cgroup2 VFS superblock' -- this only applies to cgroupv2. There is no guard ensuring the system is running cgroupv2 (unified mode). On a cgroupv1 system, `LookupMount(\"/sys/fs/cgroup\")` may return a tmpfs mount whose VFSOptions are irrelevant, and `nsdelegate`/`memory_recursiveprot` would never appear anyway, so this is safe but the comment could be more precise." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 73, + "severity": "low", + "comment": "The `RUNC_FLAVOR` environment variable check skips the test for crun with a link to the upstream crun PR. This skip should be revisited once that crun PR is merged and released, or the skip condition could check the crun version to auto-enable the test when a fixed version is available." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 111, + "severity": "medium", + "comment": "The test asserts that host cgroup mount options are preserved *after* starting a privileged container, but it does not verify the container's own `/sys/fs/cgroup` mount has the expected options. Adding an exec into the container to check `cat /proc/mounts` would strengthen the test by verifying both the host-side preservation and the container-side correctness." + }, + { + "file": "Vagrantfile", + "line": 281, + "severity": "low", + "comment": "The `RUNC_FLAVOR` environment variable defaults to `\"runc\"` in the Vagrantfile. This is reasonable but it introduces a new test knob. Ensure the CI matrix or documentation reflects that this variable can be set to `crun` for testing with alternative runtimes." + } + ], + "summary": "The PR correctly identifies that mounting cgroup2 in a privileged container sharing the host cgroup namespace can strip important mount options like `nsdelegate` and `memory_recursiveprot`, and fixes it by copying those options from the host mount. The implementation is sound but the error from `LookupMount` is silently ignored, the test only verifies host-side preservation without checking the container's own mount, and the use of `strings.SplitSeq` should be confirmed as available in the target Go version for this release branch." + }, + "flow_guided_review": { + "comments": [ + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 74, + "severity": "low", + "comment": "The cgroup options list is constructed inline with hardcoded defaults. Since this list is also used as the baseline for the mount and the `mode` variable is appended at the end, the ordering is consistent with the previous behavior. The refactor from an inline slice literal to a named variable `cgroupOptions` improves readability and enables the conditional append pattern below." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 77, + "severity": "medium", + "comment": "The cgroup namespace detection uses `slices.ContainsFunc` on `s.Linux.Namespaces`. This correctly checks the OCI spec's namespace configuration, but note that the absence of a `CgroupNamespace` entry in the spec does not always mean the container shares the host cgroup namespace -- the runtime's default behavior may vary. In containerd's CRI implementation, privileged containers explicitly skip adding the cgroup namespace, so this check is correct in context, but a code comment explaining this containerd-specific assumption would help maintainability." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 83, + "severity": "medium", + "comment": "The `LookupMount` error is silently consumed via `if mountInfo, err := osi.LookupMount(...); err == nil`. While this is a defensive pattern that avoids breaking container creation if the lookup fails, it means a misconfigured system or a transient /proc/mounts read error will silently degrade privileged container behavior. A debug-level log would aid troubleshooting without being noisy." + }, + { + "file": "internal/cri/opts/spec_linux_opts.go", + "line": 85, + "severity": "low", + "comment": "Only `nsdelegate` and `memory_recursiveprot` are explicitly copied. This allowlist approach is safer than copying all VFS options (which could include options that should not be propagated), but future kernel additions to cgroup2 mount options would require updating this list. A brief code comment noting this is an intentional allowlist would help future maintainers." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 80, + "severity": "medium", + "comment": "The test reads the host's `/sys/fs/cgroup` mount options before and after creating a privileged container, asserting that `nsdelegate` and `memory_recursiveprot` are preserved on the host. This is the correct regression test for the reported issue (host options being stripped), but it does not verify the container-internal mount has the options. An exec-based check inside the container would provide end-to-end validation." + }, + { + "file": "integration/container_cgroup_mount_options_linux_test.go", + "line": 73, + "severity": "low", + "comment": "The `RUNC_FLAVOR` skip for crun links to a specific upstream PR. This is good practice for cherry-picked fixes but should be tracked so the skip is removed once crun ships the fix. Consider adding a TODO comment with an issue link." + }, + { + "file": "Vagrantfile", + "line": 281, + "severity": "low", + "comment": "Adding `RUNC_FLAVOR` to both Vagrant provisioner blocks ensures the integration test can be run with both runc and crun in the Vagrant-based test environment. The default of `runc` is correct for the common case." + } + ], + "summary": "The PR addresses a real production issue where privileged containers sharing the host cgroup namespace would strip important cgroupv2 mount options like `nsdelegate` and `memory_recursiveprot` from the host's `/sys/fs/cgroup` superblock. The fix uses a well-structured allowlist approach with proper namespace detection, though it would benefit from logging on `LookupMount` failure, a comment explaining the allowlist rationale, and a container-side mount verification in the integration test." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.2 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides a more structured analysis by tracing from the cgroup options construction through namespace detection to the mount option copying logic. It identifies the same core issues (silent error swallowing, missing container-side test verification) but adds important context about the allowlist being intentional and the namespace detection assumptions being containerd-specific. The baseline review raises a valid point about `strings.SplitSeq` availability on the release branch that the flow-guided review misses, but overall the flow-guided review provides better contextual understanding. However, the advantage is modest because the review plan was empty (no steps, no risks, no dependencies), so the flow-guided review could not leverage plan-specific insights and instead had to reconstruct the flow from the diff alone." + } +} \ No newline at end of file diff --git a/evals/containerd__containerd__13125.json b/evals/containerd__containerd__13125.json new file mode 100644 index 0000000..f5eb2d2 --- /dev/null +++ b/evals/containerd__containerd__13125.json @@ -0,0 +1,108 @@ +{ + "pr": "containerd/containerd#13125", + "title": "[release/2.2] Tweak mount info for overlayfs in case of parallel unpack", + "baseline_review": { + "comments": [ + { + "file": "core/unpack/unpacker.go", + "line": 529, + "severity": "medium", + "comment": "The condition checks `unpack.SnapshotterKey == \"overlayfs\"` as a hard-coded string match. Other overlay-based snapshotters (e.g., `stargz`, `nydus`, or custom snapshotters that also return bind mounts) would not benefit from this workaround. Consider whether this should match a broader set of snapshotters or use a capability-based check instead of a string literal." + }, + { + "file": "core/unpack/unpacker.go", + "line": 758, + "severity": "high", + "comment": "The `bindToOverlay` function appends `upperdir=` but does not set a `workdir` option. Overlayfs requires a `workdir` for writable mounts. If any downstream applier attempts to mount this as a real overlay, the mount will fail. If the applier only reads the options to determine whiteout handling without actually mounting, this is fine -- but the function produces a mount spec that would be invalid for actual use. This should be documented or guarded." + }, + { + "file": "core/unpack/unpacker.go", + "line": 760, + "severity": "low", + "comment": "The function filters out `rbind` from options but does not filter other bind-specific options (e.g., `rw`, `nosuid`, `nodev`) that may not be appropriate for an overlay mount. If the bind mount carries additional options beyond `ro` and `rbind`, they will be passed through to the synthetic overlay mount unchanged. Verify this is correct." + }, + { + "file": "core/unpack/unpacker_test.go", + "line": 95, + "severity": "medium", + "comment": "The test uses `reflect.DeepEqual` for comparison, which is fine but could be replaced with a more descriptive assertion helper. More importantly, the test lacks a case where the bind mount has no options at all (empty `Options` slice), which would exercise the edge case where `rbind` is absent and only `upperdir` is appended." + }, + { + "file": "core/unpack/unpacker_test.go", + "line": 140, + "severity": "low", + "comment": "The 'multiple mounts' test case verifies that multiple bind mounts are returned unchanged (passthrough). Consider adding a test case with a single non-bind mount type (e.g., `tmpfs`) to verify that the function only converts bind mounts and leaves other mount types untouched." + }, + { + "file": "integration/client/container_linux_test.go", + "line": 1843, + "severity": "low", + "comment": "The integration test uses `semaphore.NewWeighted(3)` as the unpack limiter to force parallel unpack behavior. A comment explaining why weight 3 was chosen (e.g., it exceeds the number of layers in the test image) would help future readers understand the test setup." + } + ], + "summary": "This PR adds a targeted workaround to convert bind mounts to overlay mounts during parallel unpack so that the applier correctly processes whiteout files. The implementation is well-scoped with good test coverage, but the synthetic overlay mount spec is missing a `workdir` option which could be problematic if it is ever used for actual mounting rather than just whiteout detection." + }, + "flow_guided_review": { + "comments": [ + { + "file": "core/unpack/unpacker.go", + "line": 529, + "severity": "medium", + "comment": "Entry point of the fix: the guard condition `i > 0 && parallel && unpack.SnapshotterKey == \"overlayfs\"` correctly scopes the workaround to only the layers that need it (not the base layer, only during parallel unpack, only for overlayfs). However, hardcoding `\"overlayfs\"` creates a coupling to a specific snapshotter name. If the overlayfs snapshotter is registered under a different key (e.g., in a custom configuration), this workaround silently fails and the whiteout bug resurfaces." + }, + { + "file": "core/unpack/unpacker.go", + "line": 758, + "severity": "high", + "comment": "Core conversion logic: `bindToOverlay` transforms bind mounts into overlay mount specs for whiteout processing. The function correctly short-circuits when the input is not a single bind mount (handles overlay passthrough and multi-mount cases). However, the generated overlay mount sets `upperdir` without `lowerdir` or `workdir`. While the applier likely only inspects mount options to decide whiteout handling rather than performing a real mount(2) syscall, this produces a technically invalid overlayfs mount spec. If any code path attempts to actually mount this, it will fail. The TODO comment acknowledges this is temporary, but the function should at minimum document that the returned mount is not intended for actual mounting." + }, + { + "file": "core/unpack/unpacker.go", + "line": 763, + "severity": "medium", + "comment": "The `rbind` filter uses exact string matching (`o != \"rbind\"`). If the bind mount has the option `bind` (without the `r` prefix) or uses comma-separated option syntax, it would not be filtered. Verify that the overlayfs snapshotter always uses `rbind` specifically and never alternative forms." + }, + { + "file": "core/unpack/unpacker_test.go", + "line": 96, + "severity": "medium", + "comment": "The unit tests cover the three main branches (single bind, overlay passthrough, multiple mounts) which maps well to the `bindToOverlay` logic. However, there is no test case for a bind mount with empty Options, which would result in an overlay mount with only `upperdir=...` in its options. This edge case should be verified since the applier may expect `ro` or other options to be present." + }, + { + "file": "integration/client/container_linux_test.go", + "line": 1830, + "severity": "low", + "comment": "The integration test is the end-to-end proof that the fix works: it pulls an image with whiteout files using parallel unpack and verifies the whiteout-deleted files are absent. This is a strong regression test. The test references `images.Get(images.Whiteout)` which implies a pre-built test image exists -- ensure this image is available in the test infrastructure and contains the expected layer structure (file creation in one layer, whiteout deletion in a subsequent layer)." + }, + { + "file": "core/unpack/unpacker.go", + "line": 533, + "severity": "low", + "comment": "The `mounts` variable is reassigned in-place before being passed to `a.Apply()`. This is clean and does not affect the original mounts slice since `bindToOverlay` returns a new slice. Good separation of concerns -- the workaround is isolated from the rest of the unpack logic." + } + ], + "summary": "The fix correctly addresses the whiteout bug by converting bind mounts to overlay mount specs before the apply step, ensuring the applier recognizes whiteout files during parallel unpack. The flow from guard condition to conversion function to integration test is well-structured, but the synthetic overlay mount spec lacks `workdir`/`lowerdir` making it invalid for actual mounting, and the hardcoded snapshotter key creates a fragile coupling." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides better structural understanding by tracing the fix from the guard condition through the conversion function to the integration test, explaining how each piece contributes to the overall fix. It identifies the same core risk (invalid overlay mount spec) but contextualizes it better by explaining why the applier likely only inspects options rather than mounting. The baseline review raises valid individual points but treats each file in isolation without connecting them into a coherent narrative of how the fix works end-to-end. Both reviews are relatively close because the review plan was empty (no steps or dependencies), limiting the flow-guided review's advantage -- with a populated plan, the flow-guided review would have had stronger structural guidance to identify cross-cutting concerns like the relationship between the snapshotter key check and the conversion logic." + } +} \ No newline at end of file diff --git a/evals/date-fns__date-fns__3796.json b/evals/date-fns__date-fns__3796.json new file mode 100644 index 0000000..74790b0 --- /dev/null +++ b/evals/date-fns__date-fns__3796.json @@ -0,0 +1,145 @@ +{ + "pr": { + "url": "https://github.com/date-fns/date-fns/pull/3796", + "owner": "date-fns", + "repo": "date-fns", + "number": 3796, + "title": "intlFormatDistance: refactor options", + "files_changed": 4, + "additions": 30, + "deletions": 21, + "language": "TypeScript" + }, + "timestamp": "2026-03-30T20:00:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "src/intlFormatDistance/index.ts", + "line": 220, + "severity": "major", + "comment": "The spread `{ numeric: 'auto', ...options }` passes the entire options object to the Intl.RelativeTimeFormat constructor, including non-RTF properties like `unit` and `locale`. Previously, only `localeMatcher`, `numeric`, and `style` were explicitly extracted. While browsers ignore unknown properties, this is a subtle behavioral change: if `options.numeric` is set it now overrides the 'auto' default (since spread comes after), whereas before `options?.numeric || 'auto'` would only fall back on falsy values. The `||` vs spread-override semantics differ for the value `undefined`." + }, + { + "file": "src/intlFormatDistance/index.ts", + "line": 220, + "severity": "major", + "comment": "The old code used `options?.numeric || 'auto'`, which treated an explicit `numeric: undefined` or `numeric: ''` as falsy and fell back to 'auto'. The new `{ numeric: 'auto', ...options }` will override 'auto' with `undefined` if `options.numeric` is explicitly `undefined`, potentially changing the Intl.RelativeTimeFormat behavior to its default ('always') instead of 'auto'." + }, + { + "file": ".eslintrc.js", + "line": 32, + "severity": "minor", + "comment": "Disabling `@typescript-eslint/no-namespace` globally affects the entire codebase. A more targeted approach would be an inline eslint-disable comment in `src/types.ts` only, keeping the rule active everywhere else to discourage namespace proliferation." + }, + { + "file": "src/types.ts", + "line": 311, + "severity": "minor", + "comment": "The `DateFns.Utils` double-nested namespace is introduced for a single utility type `MaybeArray`. This is heavyweight scaffolding for one type alias. A simpler top-level `type MaybeArray = T | T[]` export would achieve the same with less ceremony." + }, + { + "file": "src/intlFormat/index.ts", + "line": 4, + "severity": "nit", + "comment": "The `@deprecated` tag on IntlFormatLocale with `[TODO] Remove in v4` is good housekeeping. Consider adding it to the JSDoc of any code that still references this type so consumers get deprecation warnings." + }, + { + "file": "src/intlFormatDistance/index.ts", + "line": 22, + "severity": "positive", + "comment": "Extending `Intl.RelativeTimeFormatOptions` directly is a solid improvement. It ensures the interface stays current with TypeScript lib updates and removes duplicated JSDoc that could drift from the canonical MDN descriptions." + } + ], + "summary": "The type refactoring to extend Intl.RelativeTimeFormatOptions is a good direction for maintainability. However, the spread-based constructor call introduces two concerns: non-RTF properties leak into the constructor, and the `numeric` default semantics change from `||` fallback to spread-override, which could surface a behavioral difference when `numeric` is explicitly undefined." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/intlFormatDistance/index.ts", + "line": 220, + "severity": "critical", + "comment": "HIGH-RISK ENTRY POINT: The plan marks `intlFormatDistance` as the sole high-risk entry point. The constructor call `new Intl.RelativeTimeFormat(options?.locale, { numeric: 'auto', ...options })` has a semantic regression. Previously `options?.numeric || 'auto'` ensured 'auto' was used when numeric was falsy. Now `{ numeric: 'auto', ...options }` will override 'auto' with `undefined` if the caller passes `{ numeric: undefined }`, causing the browser to use its default ('always' per spec). This changes observable output from '2 days ago' to 'in -2 days'. Should destructure and omit undefined values, or use: `numeric: options?.numeric ?? 'auto'`." + }, + { + "file": "src/intlFormatDistance/index.ts", + "line": 220, + "severity": "major", + "comment": "The spread passes `unit` and `locale` properties into the RTF constructor options bag. While current browsers ignore unknown properties, this is not guaranteed by spec and could cause issues in strict runtime environments or future browser versions. Extract only the relevant RTF options: `{ numeric: options?.numeric ?? 'auto', style: options?.style, localeMatcher: options?.localeMatcher }`." + }, + { + "file": ".eslintrc.js", + "line": 32, + "severity": "minor", + "comment": "The global disable of `@typescript-eslint/no-namespace` was required by the new `DateFns` namespace in types.ts. This lowers the lint bar for the whole project. Prefer a file-scoped `/* eslint-disable @typescript-eslint/no-namespace */` in types.ts instead." + }, + { + "file": "src/types.ts", + "line": 311, + "severity": "minor", + "comment": "The nested namespace `DateFns.Utils` introduces organizational overhead for a single `MaybeArray` type. The comment says it will be 'useful to move some types here in the future', but YAGNI applies -- add the namespace when there is actual need. A flat `export type MaybeArray = T | T[]` is simpler." + }, + { + "file": "src/intlFormatDistance/index.ts", + "line": 22, + "severity": "positive", + "comment": "Extending `Intl.RelativeTimeFormatOptions` cleanly inherits `localeMatcher`, `numeric`, and `style` from TypeScript's lib types. This eliminates duplicated JSDoc and auto-tracks any future Intl spec additions." + }, + { + "file": "src/intlFormat/index.ts", + "line": 21, + "severity": "nit", + "comment": "The locale field now uses `DateFns.Utils.MaybeArray` instead of the simpler `IntlFormatLocale | IntlFormatLocale[]`. While consistent with the new intlFormatDistance locale type, it adds indirection for callers reading the type signature." + } + ], + "summary": "Following the plan's identification of intlFormatDistance as the single high-risk entry point, the review pinpoints a semantic regression in the numeric default handling and a property-leaking concern in the spread-based constructor call. The type inheritance from Intl.RelativeTimeFormatOptions is sound, but the runtime implementation needs tighter property extraction to preserve backward compatibility." + }, + "review_plan": { + "stats": { + "totalSteps": 1, + "totalAdditions": 2, + "totalDeletions": 3, + "independentFlows": 1, + "filesChanged": 1 + }, + "steps": [ + { + "order": 1, + "nodeId": "src/intlFormatDistance/index.ts::intlFormatDistance", + "name": "intlFormatDistance", + "file": "src/intlFormatDistance/index.ts", + "lines": [146, 225], + "type": "function", + "changeType": "modified", + "additions": 2, + "deletions": 3, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + } + ], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 9, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + }, + "reasoning": "The flow-guided review leveraged the plan's identification of intlFormatDistance as the sole high-risk entry point to focus deeply on the constructor call change. It elevated the numeric default regression to critical severity with a concrete explanation of how `{ numeric: 'auto', ...options }` differs from `options?.numeric || 'auto'` when undefined is present, and provided a fix (`?? 'auto'`). The baseline caught the spread concern but treated both issues as separate majors without connecting them to the entry-point risk or providing as actionable a fix. The flow-guided review was more focused, more actionable, and better calibrated in severity.", + "winner": "flow_guided" + } +} diff --git a/evals/date-fns__date-fns__3813.json b/evals/date-fns__date-fns__3813.json new file mode 100644 index 0000000..9824d16 --- /dev/null +++ b/evals/date-fns__date-fns__3813.json @@ -0,0 +1,119 @@ +{ + "pr": { + "url": "https://github.com/date-fns/date-fns/pull/3813", + "owner": "date-fns", + "repo": "date-fns", + "number": 3813, + "title": "Fix typos", + "files_changed": 21 + }, + "baseline_review": { + "comments": [ + { + "file": "scripts/build/fp.ts", + "line": 27, + "severity": "positive", + "comment": "Fixing 'singature' to 'signature' in a variable name and its usage within the build script. While this is only cosmetic, consistent naming avoids confusion for future contributors reading the build logic." + }, + { + "file": "src/constants/index.ts", + "line": 35, + "severity": "positive", + "comment": "Correcting 'dividable' to 'divisible' in the JSDoc for the daysInYear constant. This comment explains the leap year formula and is likely read by developers seeking to understand the constant's derivation." + }, + { + "file": "src/addBusinessDays/test.ts", + "line": 63, + "severity": "nit", + "comment": "Renaming local variable 'substractResult' to 'subtractResult'. Test-only, zero behavioral impact, but improves readability of the test suite." + }, + { + "file": "CHANGELOG.md", + "line": 93, + "severity": "nit", + "comment": "Fixing 'compatability' -> 'compatibility' and 'Thansk' -> 'Thanks' in two separate changelog entries. These are user-facing documentation so the corrections are worthwhile." + }, + { + "file": "src/eachMinuteOfInterval/test.ts", + "line": 22, + "severity": "nit", + "comment": "Correcting test description string from 'begining' to 'beginning'. No behavioral change." + }, + { + "file": "src/format/test.ts", + "line": 60, + "severity": "nit", + "comment": "Fixing test description 'charactor' to 'character'. Cosmetic only." + }, + { + "file": "src/fp/_lib/convertToFP/test.ts", + "line": 3, + "severity": "nit", + "comment": "Fixing 'environemnts' to 'environments' in an eslint-disable comment. No functional impact but improves code readability." + } + ], + "summary": "A purely cosmetic PR fixing typos across 21 files, discovered via codespell and typos tools. Changes span changelogs, JSDoc comments, test descriptions, eslint-disable comments, and variable names with zero behavioral or runtime impact." + }, + "flow_guided_review": { + "comments": [ + { + "file": "scripts/build/fp.ts", + "line": 27, + "severity": "positive", + "comment": "Flow plan step 1 identifies main() as a high-risk entry point. The 'singature' -> 'signature' rename touches the parameter variable used within a .find() callback that determines whether an FP function has options. While the rename is cosmetic, it sits in logic that controls code generation output. Verified: only the local variable name changes, not any string comparison or emitted code." + }, + { + "file": "src/locale/_lib/buildMatchFn/index.ts", + "line": 93, + "severity": "minor", + "comment": "Flow plan step 2 flags buildMatchFn as a high-risk entry point. This function is called by every locale's match configuration. The typo fix ('challange' -> 'challenge') is in eslint-disable comments, so there is no runtime risk. However, two separate eslint-disable comments are touched, which is worth confirming are the only changes in this critical locale infrastructure file." + }, + { + "file": "src/locale/_lib/buildMatchPatternFn/index.ts", + "line": 9, + "severity": "nit", + "comment": "Flow plan step 3: buildMatchPatternFn is another locale entry point. The fix is 'decalration' -> 'declaration' in a comment. No runtime impact. This function is called alongside buildMatchFn for every locale, so confirming comment-only changes is important." + }, + { + "file": "src/setDefaultOptions/index.ts", + "line": 53, + "severity": "minor", + "comment": "Flow plan step 4: setDefaultOptions is the final entry point with 3 typo fixes in JSDoc ('overriden' -> 'overridden', 'funciton' -> 'function', 'settigns' -> 'settings'). This function's JSDoc is user-facing API documentation visible in IDE tooltips and generated docs, making these fixes higher value than test-only changes." + }, + { + "file": "src/constants/index.ts", + "line": 35, + "severity": "positive", + "comment": "Outside the plan's function-level nodes but still in source code: the 'dividable' -> 'divisible' fix in the daysInYear JSDoc. This is user-facing documentation explaining a core constant." + }, + { + "file": "src/getOverlappingDaysInIntervals/test.ts", + "line": 131, + "severity": "nit", + "comment": "Fixing '0-lenght' to '0-length' in a test description. Not covered by the flow plan since it is test-only, confirming the plan correctly deprioritized test file changes." + } + ], + "summary": "The flow plan identified 4 function-level entry points where typo fixes occur in production code (build script, locale matchers, setDefaultOptions). This usefully separated the higher-value documentation fixes in setDefaultOptions and locale infrastructure from the lower-risk test/changelog changes, though for a typo-only PR the risk differentiation is modest." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 4, + "actionability": 5, + "efficiency": 8, + "overall": 5.4 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 5, + "actionability": 6, + "efficiency": 7, + "overall": 6.6 + }, + "reasoning": "For a typo-only PR, neither review approach yields high-risk findings. The flow-guided review added value by distinguishing production-code typo fixes (build script logic, locale infrastructure, API documentation in setDefaultOptions) from test/changelog cosmetics. The baseline treated all changes equally. The flow plan's entry-point identification was useful for prioritizing which files to verify had truly comment-only changes vs. potentially affecting code generation or locale matching behavior.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T18:30:00.000000+00:00" +} diff --git a/evals/denoland__deno__33068.json b/evals/denoland__deno__33068.json new file mode 100644 index 0000000..f5af9c7 --- /dev/null +++ b/evals/denoland__deno__33068.json @@ -0,0 +1,102 @@ +{ + "pr": "denoland/deno#33068", + "title": "fix(install): all package.json aliases should be added to node_modules", + "baseline_review": { + "comments": [ + { + "file": "libs/npm_installer/local.rs", + "line": 719, + "severity": "positive", + "comment": "Correct root cause fix. The setup cache was keyed by `remote_pkg.id.nv.name` (the real package name), so when multiple aliases resolved to the same package+version, only the first alias got its symlink created. Switching to `remote_alias` ensures each alias gets its own cache entry and symlink." + }, + { + "file": "libs/npm_installer/local.rs", + "line": 719, + "severity": "minor", + "comment": "Consider whether `insert_root_symlink` callers elsewhere in this file also use the real package name as the key. If the scoped-package branch (the `if` above this `else`) has the same pattern, it may have an analogous bug for scoped aliases. Worth auditing other call sites of `insert_root_symlink`." + }, + { + "file": "tests/specs/npm/aliases_same_pkg_node_modules/__test__.jsonc", + "line": 5, + "severity": "positive", + "comment": "Good multi-step test design: first install populates the cache, second install adds a second alias to the same package, and the run step verifies both aliases are importable. This directly reproduces the reported bug scenario." + }, + { + "file": "tests/specs/npm/aliases_same_pkg_node_modules/__test__.jsonc", + "line": 18, + "severity": "minor", + "comment": "The test only covers non-scoped aliases (`add-v1`, `add-v2`) pointing to a scoped package (`@denotest/add`). Consider adding a test case where both the alias and the target are scoped (e.g., `@custom/add-v1` and `@custom/add-v2` both pointing to `@denotest/add@1.0.0`) to ensure the scoped code path also handles multiple aliases correctly." + }, + { + "file": "tests/specs/npm/aliases_same_pkg_node_modules/main.js", + "line": 1, + "severity": "nit", + "comment": "The test script imports using bare specifiers (`add-v1`, `add-v2`) without any `node:` or `npm:` prefix. This is correct for node_modules resolution but a brief inline comment explaining why bare specifiers work here (node_modules symlinks) would help future readers understand the test intent." + } + ], + "summary": "This is a clean, minimal one-line fix that correctly changes the setup cache key from the real package name to the alias name, preventing deduplication when multiple aliases point to the same package. The test is well-structured to reproduce the exact bug scenario, though additional coverage for scoped alias paths would strengthen confidence." + }, + "flow_guided_review": { + "comments": [ + { + "file": "libs/npm_installer/local.rs", + "line": 719, + "severity": "positive", + "comment": "The fix is precisely targeted: `insert_root_symlink` uses its first argument as the deduplication key, and the bug was that using `remote_pkg.id.nv.name` caused multiple aliases to the same package to collide in the cache. Switching to `remote_alias` makes each alias a distinct cache entry while the second argument (`target_folder_name`) correctly still points to the shared package folder." + }, + { + "file": "libs/npm_installer/local.rs", + "line": 719, + "severity": "minor", + "comment": "The review plan is empty (zero steps, zero clusters), so there is no flow context to leverage. However, examining the data flow manually: `remote_alias` comes from the iteration over package.json dependencies, which already resolves alias names. The `target_folder_name` is derived from the resolved package identity. This separation is correct -- the alias is the symlink name, the target is the shared content." + }, + { + "file": "libs/npm_installer/local.rs", + "line": 719, + "severity": "minor", + "comment": "Risk consideration: if `remote_alias` can contain characters that are invalid in the setup cache's key space (e.g., slashes in scoped package names like `@scope/pkg`), the fix could produce unexpected cache behavior. However, the scoped case is handled by the `if` branch above, so `remote_alias` in this `else` branch should always be an unscoped name, making this safe." + }, + { + "file": "tests/specs/npm/aliases_same_pkg_node_modules/__test__.jsonc", + "line": 5, + "severity": "positive", + "comment": "The test exercises the exact failure mode: two aliases resolving to the same package+version. The sequential install steps (install, then install with additional alias) ensure the setup cache is populated before the second alias is added, which is the precise condition that triggered the bug." + }, + { + "file": "tests/specs/npm/aliases_same_pkg_node_modules/__test__.jsonc", + "line": 18, + "severity": "minor", + "comment": "The second step uses `deno install add-v2@npm:@denotest/add@1.0.0` which adds the alias via CLI. Consider also testing the case where both aliases are defined in `package.json` from the start and installed together in a single `deno install` call, as the cache deduplication could behave differently in a fresh single-pass install vs. an incremental re-install." + }, + { + "file": "tests/specs/npm/aliases_same_pkg_node_modules/package.json", + "line": 3, + "severity": "nit", + "comment": "The initial `package.json` only defines `add-v1`. For completeness, a second test variant where both `add-v1` and `add-v2` are in the initial `package.json` would verify the fix works for the single-pass install case, not just the incremental case." + } + ], + "summary": "The one-line fix correctly addresses the root cause by keying the setup cache on the alias name rather than the resolved package name, which prevents symlink deduplication when multiple aliases target the same package. The test covers the incremental re-install scenario well, but adding a single-pass install variant (both aliases in package.json from the start) would provide more complete coverage of the fix." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.6 + } + }, + "winner": "flow_guided", + "reasoning": "The review plan was empty (zero steps, zero clusters, zero dependencies), which severely limits the flow-guided review's ability to differentiate itself. Despite this, the flow-guided review manually traces the data flow of `remote_alias` vs `remote_pkg.id.nv.name` and identifies a concrete risk around scoped package names in the cache key space (then correctly dismisses it by noting the branch structure). It also raises a more actionable test improvement -- testing single-pass install with both aliases in package.json -- compared to the baseline's more generic suggestion about scoped aliases. Both reviews correctly identify the fix as correct and minimal. The flow-guided review edges ahead on risk identification and actionability, but the margin is small given the empty plan. This is a straightforward bug fix where both approaches converge on similar observations." + } +} \ No newline at end of file diff --git a/evals/denoland__deno__33075.json b/evals/denoland__deno__33075.json new file mode 100644 index 0000000..a4c7627 --- /dev/null +++ b/evals/denoland__deno__33075.json @@ -0,0 +1,102 @@ +{ + "pr": "denoland/deno#33075", + "title": "fix(ext/web): fix infinite loop in WebTransport datagram overflow handling", + "baseline_review": { + "comments": [ + { + "file": "ext/web/webtransport.js", + "line": 716, + "severity": "critical", + "comment": "This is the core bug fix: `const toBeRemoved` was used as a loop condition (`while (toBeRemoved > 0)`) but never decremented, causing an infinite loop when `queue.length > incomingHighWaterMark`. Changing to `let` with a `toBeRemoved--` decrement is the correct and minimal fix. The loop now properly removes excess datagrams and terminates." + }, + { + "file": "ext/web/webtransport.js", + "line": 716, + "severity": "minor", + "comment": "An alternative to the while loop with manual decrement would be using `ArrayPrototypeSplice(queue, 0, toBeRemoved)` to remove all excess elements in a single call. This would be both more efficient (single array operation vs N shifts) and avoid the class of bug being fixed here entirely. Consider this simplification." + }, + { + "file": "tests/specs/run/webtransport_datagram_overflow/main.ts", + "line": 57, + "severity": "minor", + "comment": "The 500ms `setTimeout` is a timing-based assertion: if the infinite loop is present, the test hangs forever; if fixed, it completes quickly. This is pragmatic but fragile in slow CI environments. A more robust approach would be to use `AbortSignal.timeout()` or `Deno.test`'s built-in sanitizers to enforce a hard deadline, making the failure mode an explicit timeout error rather than an indefinite hang." + }, + { + "file": "tests/specs/run/webtransport_datagram_overflow/main.ts", + "line": 15, + "severity": "nit", + "comment": "The test creates a `Deno.QuicEndpoint` and a `WebTransport` client but relies on the server async IIFE running indefinitely in the background. If the test fails mid-execution, the server listener `for await` loop will leak. The `server.close()` at line 60 handles the happy path, but an unhandled rejection or early failure would leave the endpoint open." + }, + { + "file": "tests/specs/run/webtransport_datagram_overflow/__test__.jsonc", + "line": 2, + "severity": "positive", + "comment": "Good use of the existing Deno spec test infrastructure. The `--unstable-net` flag is correctly included since `Deno.QuicEndpoint` and `Deno.upgradeWebTransport` are unstable APIs, and `-A` grants the necessary permissions for network and file access." + } + ], + "summary": "The PR correctly fixes a clear infinite loop bug where a `const` loop counter was never decremented. The fix is minimal and correct, and the regression test adequately demonstrates the bug. The test could be made more robust against timing sensitivity in CI." + }, + "flow_guided_review": { + "comments": [ + { + "file": "ext/web/webtransport.js", + "line": 716, + "severity": "critical", + "comment": "Following the review plan from `WebTransportDatagramDuplexStream` (class, risk: low) into its `#receiveDatagrams` method (internal, risk: low): the bug is in the overflow trimming logic within `#receiveDatagrams`. When a new datagram arrives and `queue.length > incomingHighWaterMark`, the original code computed `const toBeRemoved = queue.length - this.#incomingHighWaterMark` but never decremented it, creating an infinite `while (toBeRemoved > 0)` loop that endlessly shifts from the queue (which keeps getting shorter but the condition variable never changes). The fix to `let` + `toBeRemoved--` is correct." + }, + { + "file": "ext/web/webtransport.js", + "line": 713, + "severity": "minor", + "comment": "Looking at the data flow in `#receiveDatagrams`: the datagram is pushed to `queue` at line 714, then the overflow check runs, then a second `while (queue.length > 0)` loop at line 720 drains the queue into the readable stream's controller. This means in the normal case (queue was empty, one datagram arrives, HWM >= 1), `toBeRemoved` is 0 and the overflow loop is skipped entirely. The overflow path only triggers when datagrams arrive faster than they are consumed. The fix is correct for this path, but consider replacing both the push-then-shift pattern with `ArrayPrototypeSplice(queue, 0, toBeRemoved)` for O(1) trimming instead of O(n) repeated shifts." + }, + { + "file": "ext/web/webtransport.js", + "line": 716, + "severity": "minor", + "comment": "The plan identifies `#receiveDatagrams` as calling `equal` (used in the readable stream setup). The overflow trimming code sits between the datagram push and the queue drain loop. After the fix, the queue will be correctly trimmed to `incomingHighWaterMark` elements before the drain loop runs. This means the oldest datagrams are correctly dropped (FIFO eviction via `ArrayPrototypeShift`), preserving the most recent datagrams -- consistent with WebTransport spec behavior for datagram overflow." + }, + { + "file": "tests/specs/run/webtransport_datagram_overflow/main.ts", + "line": 30, + "severity": "minor", + "comment": "The test validates the exact scenario: server accepts a WebTransport connection but intentionally does NOT read from `wt.datagrams.readable`, causing incoming datagrams to queue. The client sends two datagrams (exceeding the default `incomingHighWaterMark` of 1), which triggers the overflow path in `#receiveDatagrams`. Before the fix, the server-side event loop would hang in the infinite loop, preventing the 500ms timer from firing. This is a well-designed regression test that exercises the exact code path." + }, + { + "file": "tests/specs/run/webtransport_datagram_overflow/main.ts", + "line": 57, + "severity": "minor", + "comment": "The 500ms delay is a reasonable heuristic for local testing but could be flaky in resource-constrained CI. Since the plan shows this is the only changed flow (1 independent flow, 1 file changed), the test coverage is sufficient for the fix scope. Consider adding a comment noting why the specific delay value was chosen, or using a Deno test timeout option for deterministic failure on regression." + }, + { + "file": "ext/web/webtransport.js", + "line": 716, + "severity": "positive", + "comment": "The fix is appropriately minimal -- only 2 lines changed in the production code. Given the plan shows this class is instantiated by `WebTransport.constructor` and the method is called from there, there are no other callers that could be affected. The change is safe and contained." + } + ], + "summary": "Tracing the review plan's dependency chain from `WebTransport.constructor` through `WebTransportDatagramDuplexStream` into `#receiveDatagrams`, the infinite loop bug is clearly located in the overflow trimming path that executes when incoming datagrams exceed the high-water mark. The fix is minimal, correct, and well-tested with a regression spec that exercises the exact overflow code path on the server side." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides better context by tracing the call chain from WebTransport.constructor through the datagram duplex stream class into the #receiveDatagrams method. It explains WHY the overflow path triggers (datagrams arrive faster than consumed), describes the FIFO eviction semantics (oldest dropped via shift), and connects the queue push-trim-drain sequence to understand the full data flow. The baseline review correctly identifies the bug and suggests improvements but treats each observation in isolation without connecting the constructor -> class -> method call chain. For this simple PR with only 1 independent flow and 2 lines of production code changed, the advantage of the flow-guided approach is moderate rather than dramatic -- both reviews catch the core issue and suggest the splice optimization. The flow-guided review's edge comes from explaining the runtime behavior (when overflow triggers, what gets evicted) rather than just describing the syntactic fix." + } +} \ No newline at end of file diff --git a/evals/docker__cli__6886.json b/evals/docker__cli__6886.json new file mode 100644 index 0000000..5f6cb2b --- /dev/null +++ b/evals/docker__cli__6886.json @@ -0,0 +1,131 @@ +{ + "pr": { + "url": "https://github.com/docker/cli/pull/6886", + "owner": "docker", + "repo": "cli", + "number": 6886, + "title": "ci: pin actions to digests", + "files_changed": 6, + "additions": 35, + "deletions": 27, + "language": "yaml" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": ".github/workflows/codeql.yml", + "line": 67, + "severity": "minor", + "comment": "The addition of `cache: false` for the setup-go action is a behavioral change bundled into what is otherwise a pure digest-pinning PR. This disables Go module caching which could increase CI runtime. If this was intentional to work around a caching issue, it should be called out in the PR description; if not, it may be an accidental inclusion." + }, + { + "file": ".github/workflows/build.yml", + "line": 64, + "severity": "nit", + "comment": "The PR description mentions 'as a follow-up, we should use the full version (major.minor.patch)' but the comments still use short version tags like `# v4`, `# v6`, `# v7`. Consider using full semver in the comments (e.g., `# v4.1.0`) now rather than deferring, since the digests are already pinned and the comment is the only human-readable version reference." + }, + { + "file": ".github/dependabot.yml", + "line": 10, + "severity": "minor", + "comment": "Adding a 7-day cooldown for dependabot updates is a reasonable change but is unrelated to the digest-pinning effort. Bundling unrelated dependabot configuration changes with the pinning makes the PR slightly harder to review and reason about independently. Consider whether this belongs in a separate commit or PR." + }, + { + "file": ".github/workflows/build.yml", + "severity": "minor", + "comment": "The diff for test.yml appears to be truncated in the PR. Verify that all action references in test.yml have been pinned to digests consistently with the other workflow files, as an incomplete pinning would leave some workflows vulnerable to the supply-chain risks this PR aims to mitigate." + }, + { + "file": ".github/workflows/codeql.yml", + "line": 71, + "severity": "minor", + "comment": "The codeql-action is pinned to digest `38697555549f1db7851b81482ff19f1fa5c4fedc` with comment `# v4.34.1` -- using a patch-level version in the comment here but only major versions elsewhere (e.g., `# v4`, `# v6`). The version comment granularity is inconsistent across the PR, which could cause confusion during future updates." + }, + { + "file": ".github/workflows/e2e.yml", + "line": 74, + "severity": "positive", + "comment": "Good that codecov-action is also pinned to a digest. Third-party actions from non-GitHub/non-Docker organizations are the highest supply-chain risk, so pinning codecov is particularly valuable." + } + ], + "summary": "This PR pins all GitHub Actions references to SHA digests across CI workflow files, which is a strong supply-chain security improvement. The main concerns are the inconsistent version comment granularity, an unrelated `cache: false` behavioral change in the CodeQL workflow, and the truncated test.yml diff which should be verified for completeness." + }, + "flow_guided_review": { + "comments": [ + { + "file": ".github/workflows/codeql.yml", + "line": 67, + "severity": "minor", + "comment": "The `cache: false` addition to setup-go is a behavioral change unrelated to digest pinning. Since the review plan is empty (no steps or dependency clusters), this change stands out as an orphaned modification that does not fit the stated PR goal. It should be justified in the description or split out." + }, + { + "file": ".github/dependabot.yml", + "line": 10, + "severity": "minor", + "comment": "The dependabot cooldown configuration is a separate concern from pinning actions to digests. With no review plan steps or clusters linking these changes, this appears to be an opportunistic bundling. While harmless, it makes the PR description ('pin actions to digests') not fully accurate." + }, + { + "file": ".github/workflows/build.yml", + "line": 64, + "severity": "nit", + "comment": "Version comments use short tags (e.g., `# v4`) rather than full semver. The PR description itself acknowledges that full versions should be used as a follow-up. Since the review plan identifies no dependencies or risks, this is low priority but worth tracking as a follow-up task." + }, + { + "file": ".github/workflows/build.yml", + "severity": "minor", + "comment": "The same digest `4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd` is used for docker/setup-buildx-action in three separate locations across build.yml. This is correct and consistent. However, if any one instance were updated independently in the future, drift would be hard to detect. Consider using a reusable workflow or YAML anchors to deduplicate." + }, + { + "file": ".github/workflows/test.yml", + "severity": "minor", + "comment": "The test.yml diff is truncated. Without the review plan providing file-level coverage data, it is impossible to verify whether all actions in test.yml were pinned. This is a completeness gap -- the reviewer should confirm test.yml is fully pinned before merging." + }, + { + "file": ".github/workflows/codeql.yml", + "line": 71, + "severity": "nit", + "comment": "The codeql-action comment uses patch-level version `# v4.34.1` while all other actions use major-only comments like `# v4`. This inconsistency is minor but since the review plan has no guidance on version comment conventions, it should be normalized one way or the other." + }, + { + "file": ".github/workflows/e2e.yml", + "line": 74, + "severity": "positive", + "comment": "Pinning the third-party codecov/codecov-action to a digest is the highest-value change in this PR from a supply-chain security perspective, since non-Docker/non-GitHub actions carry more trust risk." + } + ], + "summary": "With an empty review plan providing no steps, clusters, or dependency information, the flow-guided review largely mirrors the baseline but with additional attention to cross-file consistency of digest references. The PR achieves its security goal of pinning actions to immutable digests, but includes two unrelated changes (dependabot cooldown, setup-go cache disable) and has a truncated test.yml diff that needs verification." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 6, + "overall": 6.2 + }, + "reasoning": "Both reviews identified the same core issues: the unrelated `cache: false` change, inconsistent version comment granularity, truncated test.yml, and the bundled dependabot cooldown change. The flow-guided review attempted to leverage the review plan but since the plan was completely empty (zero steps, zero clusters, zero dependencies), it had no additional structural insight to offer. It added one extra comment about digest deduplication across build.yml which is mildly useful but also slightly less efficient. The flow_awareness scores are low for both because the empty plan provided no meaningful flow information to leverage. The flow-guided review scores marginally higher on flow_awareness only because it explicitly acknowledged the empty plan and reasoned about what that absence meant. Overall, the reviews are essentially equivalent in quality for this simple, mechanical PR with an empty review plan.", + "winner": "tie" + } +} diff --git a/evals/drizzle-team__drizzle-orm__5475.json b/evals/drizzle-team__drizzle-orm__5475.json new file mode 100644 index 0000000..e668095 --- /dev/null +++ b/evals/drizzle-team__drizzle-orm__5475.json @@ -0,0 +1,119 @@ +{ + "pr": { + "url": "https://github.com/drizzle-team/drizzle-orm/pull/5475", + "owner": "drizzle-team", + "repo": "drizzle-orm", + "number": 5475, + "title": "support node sqlite for kit", + "files_changed": 4 + }, + "baseline_review": { + "comments": [ + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2280, + "severity": "critical", + "comment": "In transactionProxy, stmt.all() and stmt.run() are called without passing query parameters. The prepared statements will execute with no bound values, causing incorrect results or runtime errors for any parameterized query within a transaction. Should be stmt.all(...prepareSqliteParams(query.params || [])) and stmt.run(...prepareSqliteParams(query.params || []))." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2234, + "severity": "major", + "comment": "checkPackage('node:sqlite') may not work correctly since node:sqlite is a built-in Node.js module (available in v22.5+), not an npm package. If checkPackage uses require.resolve or checks node_modules, it could fail to detect the module even when it is available." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2258, + "severity": "minor", + "comment": "The proxy function handles 'values', 'get', and 'all' methods identically by calling stmt.all(). For method === 'get', the caller likely expects a single object rather than an array. Consider returning stmt.all(...preparedParams)[0] when method is 'get' to match expected semantics." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2274, + "severity": "minor", + "comment": "Manual BEGIN/COMMIT/ROLLBACK transaction management is fragile. If an error occurs between BEGIN and the catch block that is not a standard Error, the ROLLBACK in the catch block may not execute. Consider a finally block for ROLLBACK safety, or use a transaction wrapper if node:sqlite provides one." + }, + { + "file": "drizzle-kit/src/cli/utils.ts", + "line": 77, + "severity": "minor", + "comment": "Bumping requiredApiVersion from 12 to 13 forces all drizzle-orm users to update their ORM package when upgrading drizzle-kit, even if they do not use node:sqlite. This is a coupling concern but follows the existing pattern." + }, + { + "file": "drizzle-orm/src/version.ts", + "line": 5, + "severity": "positive", + "comment": "Good practice documenting what each compatibility version bump represents with an inline comment." + } + ], + "summary": "This PR adds node:sqlite driver support to drizzle-kit following existing driver patterns. There is a critical bug in transactionProxy where query parameters are silently dropped from prepared statements, and a potential issue with checkPackage not handling built-in Node.js modules correctly." + }, + "flow_guided_review": { + "comments": [ + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2280, + "severity": "critical", + "comment": "The review plan flags transactionProxy as high-risk with many callers depending on it. In the transaction loop, stmt.all() and stmt.run() are called WITHOUT passing query.params or using prepareSqliteParams. Compare with the proxy function at line 2253 which correctly calls prepareSqliteParams(params.params || []) - this inconsistency means all parameterized queries silently lose their bindings inside transactions, causing data corruption or query failures." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2234, + "severity": "major", + "comment": "The plan identifies connectToSQLite as the high-risk entry point. The checkPackage('node:sqlite') call is the gate for this entire code path, but checkPackage (in utils.ts) likely resolves npm packages - built-in 'node:' modules may not be detectable this way. If checkPackage fails, the node:sqlite block is silently skipped and users fall through to the error message, making the feature appear broken." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2253, + "severity": "minor", + "comment": "The plan shows proxy is called by high-risk internal nodes. The proxy correctly uses prepareSqliteParams and setReturnArrays for array mode. However, when method is 'get', returning the full array from stmt.all() may not match the expected single-row return type that callers assume for 'get' queries." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2245, + "severity": "minor", + "comment": "The plan marks query as high-risk (many callers). The implementation uses client.prepare(sql).all(...(params || [])) which correctly passes parameters. However, unlike the proxy function, it does not call prepareSqliteParams - this inconsistency could lead to type-related issues if params contain values that need SQLite-specific preparation." + }, + { + "file": "drizzle-kit/src/cli/connections.ts", + "line": 2274, + "severity": "minor", + "comment": "The transaction error handling pushes the Error into the results array alongside successful results. Callers iterating over results need to type-check each element. This pattern matches other driver implementations but is inherently fragile - a ROLLBACK failure would throw without being caught." + }, + { + "file": "drizzle-kit/src/cli/utils.ts", + "line": 77, + "severity": "minor", + "comment": "The plan shows this is a leaf dependency in the cluster. The requiredApiVersion bump to 13 correctly matches compatibilityVersion in drizzle-orm/src/version.ts. This is required for the kit to validate it is running against an ORM version that includes the node-sqlite driver." + }, + { + "file": "drizzle-orm/src/version.ts", + "line": 5, + "severity": "positive", + "comment": "Compatibility version correctly bumped to 13 with a documenting comment. The version.ts and utils.ts values are in sync." + } + ], + "summary": "The flow plan's dependency analysis reveals a critical bug: transactionProxy drops query parameters from prepared statements, creating an inconsistency with the proxy function that correctly uses prepareSqliteParams. The plan also highlights that the checkPackage gate for the entire node:sqlite code path may not work for built-in modules, potentially making the feature unreachable." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 6.6 + }, + "flow_guided_scores": { + "completeness": 9, + "flow_awareness": 9, + "risk_identification": 9, + "actionability": 8, + "efficiency": 7, + "overall": 8.4 + }, + "reasoning": "The flow-guided review leveraged the plan's dependency graph to identify that transactionProxy's missing parameters are a critical cross-function inconsistency (proxy uses prepareSqliteParams but transactionProxy does not). It also caught that the query helper skips prepareSqliteParams unlike proxy, an inconsistency invisible without comparing call paths. The baseline found the same core bugs but lacked the structural reasoning about caller dependencies and risk propagation that made the flow-guided findings more precise and actionable.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T18:42:15.000000+00:00" +} diff --git a/evals/elastic__elasticsearch__145149.json b/evals/elastic__elasticsearch__145149.json new file mode 100644 index 0000000..8b4b75b --- /dev/null +++ b/evals/elastic__elasticsearch__145149.json @@ -0,0 +1,113 @@ +{ + "pr": { + "url": "https://github.com/elastic/elasticsearch/pull/145149", + "owner": "elastic", + "repo": "elasticsearch", + "number": 145149, + "title": "Avoid tracking timestamp in TimeSeriesBlockHash", + "files_changed": 3 + }, + "baseline_review": { + "comments": [ + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 47, + "severity": "positive", + "comment": "Good addition of the trackTimestamp boolean flag. This cleanly separates the concern of whether min/max timestamps need to be tracked, letting partial aggregations skip the overhead entirely." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 92, + "severity": "positive", + "comment": "Centralizing timestamp tracking into a single conditional call to trackTimestampFromVector at the top of add() is much cleaner than the scattered per-element trackTimestamp calls that existed before. This also reduces branch overhead in tight inner loops." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 147, + "severity": "major", + "comment": "The addOrdinals refactoring merges groupIdsForOrdinals and groupIdsForOrdinalsWithConstantTimestamp into one method and eliminates the constant-timestamp special case. While simpler, the constant-timestamp optimization path is lost -- verify that the performance regression from removing that fast path is acceptable." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 160, + "severity": "minor", + "comment": "The new memory accounting pattern using breaker.addEstimateBytesAndMaybeBreak before allocating raw int[] arrays and releasing in a finally block is correct. Setting acquiredBytes to 0 before transferring ownership via newIntArrayVector prevents double-free." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 56, + "severity": "minor", + "comment": "The constructor signature change adds a new boolean parameter. All existing call sites must be updated to pass the correct value -- true for final aggregation, false for partial. Ensure no call site defaults incorrectly." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 108, + "severity": "minor", + "comment": "In addConstant, the trackTimestamp(prevTimestamp) call was removed from the hot path. This is safe because trackTimestampFromVector already processes the entire vector upfront when trackTimestamp is true." + } + ], + "summary": "This PR is a targeted performance optimization that removes unnecessary timestamp tracking from partial aggregation paths and consolidates it into a single upfront call. The addOrdinals method was also refactored to use raw arrays instead of vector builders, simplifying the code but removing the constant-timestamp fast path." + }, + "flow_guided_review": { + "comments": [ + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 47, + "severity": "positive", + "comment": "The trackTimestamp flag is the core of this change. Since the review plan is empty (single-class optimization), the flag is the sole entry point for the behavioral change. It cleanly gates whether min/max timestamp tracking occurs." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 92, + "severity": "positive", + "comment": "Moving timestamp tracking to trackTimestampFromVector at the top of add() means the entire vector is scanned once for min/max rather than tracking per-element in inner loops. This is both cleaner and more vectorization-friendly." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 147, + "severity": "major", + "comment": "The addOrdinals refactoring pre-computes dictOrds once (dictionary-size lookups) then uses it for ordinals-length iterations. This trades dictionary-size memory for fewer hash lookups per position, improving cache locality. However, the removed constant-timestamp special case was previously an O(dictSize) fast path -- confirm benchmarks show net improvement." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 160, + "severity": "minor", + "comment": "Circuit breaker accounting estimates (ordinalsLength + dictLength) * Integer.BYTES upfront. This is accurate for the two int[] allocations. The try-finally with conditional release is the standard Elasticsearch pattern for breaker compliance." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 56, + "severity": "minor", + "comment": "Constructor API change requires all call sites to pass trackTimestamp. With no flow plan to trace callers, verify the other 2 changed files correctly pass true for final aggregation and false for partial aggregation stages." + }, + { + "file": "x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/blockhash/TimeSeriesBlockHash.java", + "line": 175, + "severity": "minor", + "comment": "Using raw int[] arrays with blockFactory.newIntArrayVector for ownership transfer avoids IntVectorFixedBuilder allocation overhead. This is a micro-optimization that compounds well in the high-throughput aggregation path." + } + ], + "summary": "Self-contained performance optimization within TimeSeriesBlockHash that gates timestamp tracking behind a boolean flag and refactors ordinal grouping for better cache locality using pre-computed dictionary lookups. The empty flow plan limits cross-file analysis, but the single-class changes are well-structured and maintain circuit breaker compliance." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.4 + }, + "reasoning": "Both reviews cover the same core changes effectively since this is a single-class optimization with an empty flow plan. The flow-guided review gains a marginal edge in flow_awareness by explicitly noting the cross-file caller verification need and the dictionary pre-computation trade-off, but the empty plan limits its structural advantage. Neither review can fully assess the 2 other changed files since the diff is truncated. Risk identification is equivalent -- both flag the removed constant-timestamp fast path as needing benchmark validation.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T18:30:00.000000+00:00" +} diff --git a/evals/encode__httpx__3673.json b/evals/encode__httpx__3673.json new file mode 100644 index 0000000..98ea40d --- /dev/null +++ b/evals/encode__httpx__3673.json @@ -0,0 +1,114 @@ +{ + "pr": "encode/httpx#3673", + "title": "Connection resets", + "baseline_review": { + "comments": [ + { + "file": "src/ahttpx/_server.py", + "line": "119-121", + "severity": "high", + "comment": "The old code had `await self._parser.complete` (missing parentheses), which awaited the coroutine function object rather than calling it. This was a latent bug -- the parser was never actually completing/resetting on the server side, meaning keepalive state was never restored between requests. The fix to `await self._parser.reset()` with parentheses corrects this. Good catch, but this warrants a note in the PR description since it is a behavioral fix, not just a rename." + }, + { + "file": "src/ahttpx/_server.py", + "line": "108-110", + "severity": "medium", + "comment": "The new logic calls `self._parser.is_keepalive()` and then `await stream.read()` before `await self._reset()`. If the client sends a request body that the endpoint never consumed (e.g., a 500 error path), `stream.read()` will drain the remaining body so the parser can cleanly transition to the next request. However, this is only guarded by `is_keepalive()` -- if the connection is not keepalive, the unconsumed body is ignored and `_reset()` closes the connection. This seems correct but could be documented with a comment explaining why the drain is needed." + }, + { + "file": "src/httpx/_network.py", + "line": "163", + "severity": "high", + "comment": "The old code `self._streams = list[NetworkStream]` was a type expression (subscripting the `list` type), not an assignment of an empty list. This means `_streams` was set to the generic alias `list[NetworkStream]` rather than `[]`, so any attempt to iterate or append to it would fail. The fix to `self._streams: list[NetworkStream] = []` is a critical bug fix that should be called out explicitly." + }, + { + "file": "src/httpx/_network.py", + "line": "187-193", + "severity": "medium", + "comment": "The stream cleanup in `_serve` filters out closed streams and appends the new one on each `accept()` iteration. This is a simple GC mechanism, but it runs on the accept thread -- if `accept()` blocks for a long time with no new connections, closed streams will not be cleaned up until the next connection arrives. For a low-traffic server this could hold references to dead streams longer than expected. Consider whether a periodic cleanup or cleanup on close would be more appropriate." + }, + { + "file": "src/ahttpx/_parsers.py", + "line": "378", + "severity": "low", + "comment": "The rename from `complete()` to `reset()` with a `-> bool` return type is a clean API improvement. Returning `True` when the connection was successfully reset for keepalive and `False` when it had to close makes the method's outcome explicit. However, neither caller (`_pool.py` nor `_server.py`) checks the return value -- both call `await self._parser.reset()` without using the result. Consider whether the return value is intended for future use or if callers should act on it." + }, + { + "file": "src/ahttpx/_server.py", + "line": "102", + "severity": "low", + "comment": "The change from `Response(code=500, content=content)` to `Response(500, content=content)` removes the keyword argument for `code`. This is a minor style change that makes it consistent with other `Response` constructors in the codebase but reduces readability slightly for a less common status code path." + } + ], + "summary": "This PR renames `complete` to `reset` across both sync and async HTTP parser/pool/server layers, adds `is_keepalive` as a public API, and fixes two significant bugs: a missing-parentheses bug in the async server's `_complete` call and a type-expression-instead-of-empty-list bug in `NetworkServer.__init__`. The stream lifecycle management in `NetworkServer` is a welcome addition for clean shutdown, though the lazy cleanup approach in `_serve` could leave stale references during idle periods." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/httpx/_network.py", + "line": "163", + "severity": "high", + "comment": "Starting from the entry point `NetworkServer.__init__` (step 1): the old code `self._streams = list[NetworkStream]` assigned a generic alias type object, not an empty list. This means `__exit__` (step 2) would iterate over a type object (silently doing nothing), and `_serve` (step 3) would fail on the list comprehension filter. This was a latent crash bug for any server that received a second connection. The fix is critical and should be highlighted in the PR description." + }, + { + "file": "src/httpx/_network.py", + "line": "181-193", + "severity": "medium", + "comment": "Following the flow from `__init__` (step 1) to `__exit__` (step 2) to `_serve` (step 3): the new shutdown path in `__exit__` iterates `self._streams` and closes each one, but `_serve` runs on a separate thread (`self._executor.submit`). There is no synchronization between `__exit__` closing streams and `_serve` potentially still appending new streams. If `__exit__` is called while `accept()` returns a new connection, the new stream could be appended after the cleanup loop runs but before `executor.shutdown(wait=True)` completes. The stream would then leak without being closed." + }, + { + "file": "src/ahttpx/_parsers.py", + "line": "378-400", + "severity": "medium", + "comment": "At step 6 (HTTPParser.reset), the method now returns `bool` indicating whether the connection was kept alive. Tracing callers: `Connection._reset` (step 5) in the pool calls `await self._parser.reset()` but ignores the return value, always setting `_idle_expiry` afterward. If `reset()` returned `False` (connection closed), setting an idle expiry on a closed connection is pointless. Similarly, `HTTPConnection._reset` in the server (step 9) ignores the return value. The return value should either be used by callers to skip post-reset bookkeeping, or the callers should be updated." + }, + { + "file": "src/ahttpx/_server.py", + "line": "108-110", + "severity": "medium", + "comment": "At step 8 (handle_requests), the new flow checks `is_keepalive()` (step 7) before draining the stream body. Following the dependency chain: `is_keepalive()` checks `send_keep_alive`, `recv_keep_alive`, and `send_state != CLOSED`. But at this point in the request cycle, the server has already sent the response -- `send_state` would be `DONE`, and keepalive flags reflect the negotiated connection policy. The drain via `stream.read()` ensures the parser consumes any unconsumed request body before `_reset()` transitions the state machine back to idle. This is correct but the guard should also handle the error path -- after the 500 response, the code falls through to the same `is_keepalive()` check, which is appropriate since both paths need body draining." + }, + { + "file": "src/ahttpx/_server.py", + "line": "119-121", + "severity": "high", + "comment": "At step 9 (HTTPConnection._reset), the old code `await self._parser.complete` was missing parentheses -- it awaited a coroutine function object, which is a no-op (the parser was never reset). This means in the old code, the server never actually reset keepalive state between requests, and `_idle_expiry` was set but the parser stayed in DONE state. This is the root cause bug that likely motivated the entire PR. The fix to `await self._parser.reset()` is essential." + }, + { + "file": "src/ahttpx/_parsers.py", + "line": "408-413", + "severity": "low", + "comment": "At step 7, the new `is_keepalive()` method checks `send_keep_alive and recv_keep_alive and send_state != CLOSED`. This is used by `handle_requests` (step 8) to decide whether to drain the request body before resetting. The `send_state != CLOSED` guard prevents draining on an already-closed connection, which is a sensible safety check. However, the method does not check `recv_state` -- if the receive side is closed but send is still open (a half-closed scenario), `is_keepalive()` would return True. This may be intentional if half-close is not supported, but it is worth verifying." + }, + { + "file": "src/httpx/_network.py", + "line": "187-193", + "severity": "low", + "comment": "At step 3 (_serve), the stream list cleanup uses a list comprehension that shadows the loop variable: `stream for stream in self._streams if not stream.is_closed()`. The outer `while stream := self.listener.accept()` assigns `stream`, then the comprehension rebinds `stream` within its scope. In Python 3.12+ this is fine since comprehensions have their own scope, but it reduces readability. Consider using a different variable name in the comprehension (e.g., `s`)." + } + ], + "summary": "This PR fixes two critical bugs -- a missing-parentheses call in the async server's `_complete` that prevented parser resets between requests, and a type-alias-instead-of-empty-list assignment in `NetworkServer.__init__` that broke stream tracking entirely. The flow-based analysis reveals that the `reset()` return value is unused by all callers despite being a meaningful signal, and that the `NetworkServer` shutdown path has a potential race between `__exit__` closing streams and `_serve` accepting new ones on a separate thread." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review significantly outperforms the baseline by tracing cross-file dependencies that reveal deeper issues. Both reviews catch the two critical bugs (missing parens, type alias), but the flow-guided review connects them to their downstream impact: the unused `reset()` return value leading to meaningless idle-expiry bookkeeping on closed connections, and the race condition between `__exit__` and `_serve` across threads. The flow-guided review also provides better context for the `is_keepalive()` addition by tracing how it gates the body-drain logic in `handle_requests` and noting the missing `recv_state` check. The baseline review is competent but treats each change in isolation, missing the cross-component interactions that the review plan's step ordering makes visible." + } +} \ No newline at end of file diff --git a/evals/encode__httpx__3690.json b/evals/encode__httpx__3690.json new file mode 100644 index 0000000..c99b1e4 --- /dev/null +++ b/evals/encode__httpx__3690.json @@ -0,0 +1,102 @@ +{ + "pr": "encode/httpx#3690", + "title": "Add `.wait_ready` to parser for clean server disconnects", + "baseline_review": { + "comments": [ + { + "file": "src/httpx/_parsers.py", + "line": 465, + "severity": "medium", + "comment": "The `wait_ready` method in `ReadAheadParser` calls `_read_some()` and then `_push_back(data)` to peek at incoming data without consuming it. However, the assertion in `_push_back` (`assert self._buffer == b''`) means this will crash if there is already data in the buffer. If `wait_ready` is ever called when the buffer is non-empty (e.g., leftover data from a previous request in a keep-alive cycle), this assertion will fail. Consider guarding against this or prepending to the existing buffer instead of asserting." + }, + { + "file": "src/httpx/_server.py", + "line": 155, + "severity": "medium", + "comment": "The `HTTPServer.wait()` method previously caught `KeyboardInterrupt` to allow clean shutdown. The new version removes that handling, turning `wait()` into an infinite `sleep(1)` loop with no way to break out. This means `KeyboardInterrupt` will now propagate up the stack uncaught, potentially bypassing server cleanup logic. If the caller handles `KeyboardInterrupt`, this is fine, but if not, this is a behavioral regression." + }, + { + "file": "src/httpx/_server.py", + "line": 35, + "severity": "low", + "comment": "The `wait_ready()` call at the top of the loop correctly differentiates a clean disconnect from a protocol error. However, when `wait_ready()` returns `False`, the method returns silently without logging or closing the parser explicitly. If debugging connection lifecycle issues, there is no indication that a clean disconnect occurred. A debug-level log message here would aid observability." + }, + { + "file": "src/httpx/_server.py", + "line": 47, + "severity": "low", + "comment": "The removal of the `# TODO: Handle endpoint exceptions` comment is a housekeeping improvement, but the code still wraps the endpoint call in a broad try/except that catches `Exception` and returns a 500. This is reasonable for now, but the TODO removal could suggest the problem is solved when it is really just suppressed." + }, + { + "file": "src/ahttpx/_parsers.py", + "line": 229, + "severity": "low", + "comment": "The async `HTTPParser.wait_ready()` docstring says 'Wait until read data starts arriving', which implies blocking until data is available. This is accurate for the async version but the sync counterpart in `src/httpx/_parsers.py` has the same docstring despite being a synchronous call. The sync version's behavior depends entirely on the underlying stream's `_read_some()` -- if the stream blocks, the whole thread blocks. The docstrings should clarify this distinction." + } + ], + "summary": "This PR adds a `wait_ready()` method to both sync and async HTTP parsers to probe for incoming data before committing to a full request parse, enabling clean disconnects without `ProtocolError`. The main concerns are the `_push_back` assertion safety when the buffer is non-empty, and the removal of `KeyboardInterrupt` handling in `HTTPServer.wait()` which changes shutdown behavior." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/httpx/_server.py", + "line": 35, + "severity": "high", + "comment": "This is the primary entry point (order 1, high risk) where the new `wait_ready()` integration lives. The `wait_ready()` call sits between `is_closed()` and `_recv_head()`, creating a new three-phase loop: check-closed -> wait-for-data -> parse-request. However, there is a subtle race: if `is_closed()` returns `False` but the connection closes between that check and `wait_ready()`, `wait_ready()` correctly returns `False` and we return cleanly. But if the connection closes *during* `_recv_head()` (after `wait_ready()` returned `True` because some non-HTTP data arrived), we still get a `ProtocolError`. The PR only addresses clean disconnects at the *start* of a cycle, not partial data scenarios." + }, + { + "file": "src/httpx/_parsers.py", + "line": 465, + "severity": "medium", + "comment": "The `ReadAheadParser.wait_ready()` (order 4, leaf node) is the foundation that both sync and async `HTTPParser.wait_ready()` delegate to. The `_push_back(data)` call has an `assert self._buffer == b''` precondition. Following the call chain from `handle_requests` (order 1) -> `HTTPParser.wait_ready` (order 3) -> `ReadAheadParser.wait_ready` (order 4), `wait_ready` is always called at the start of a new request cycle. At that point the buffer *should* be empty because the previous request fully consumed it. However, if `_reset()` at the end of the loop does not guarantee buffer drainage, this assertion could fail on keep-alive connections." + }, + { + "file": "src/ahttpx/_server.py", + "line": 35, + "severity": "medium", + "comment": "The async `handle_requests` (order 30, medium risk, multiple callers) mirrors the sync version exactly. Since it is called by both `src/ahttpx/_server.py::handler` and `src/httpx/_server.py::handler`, any behavioral change here affects two code paths. The `await self._parser.wait_ready()` delegates through `HTTPParser.wait_ready()` (order 3) to `ReadAheadParser.wait_ready()` (order 4). The full chain is consistent, but the multiple-caller risk means a bug in `wait_ready` would surface in both sync and async server modes simultaneously." + }, + { + "file": "src/httpx/_server.py", + "line": 155, + "severity": "medium", + "comment": "The `HTTPServer.wait()` change (order 2, high risk, entry point) removes `KeyboardInterrupt` handling. This is a separate concern from the `wait_ready` feature -- it appears to be a drive-by cleanup. Since `wait()` is an entry point with no callers tracked in the plan, its callers are external (user code). Removing `KeyboardInterrupt` handling is a breaking behavioral change for any code that relied on `wait()` returning cleanly on Ctrl+C rather than propagating the exception." + }, + { + "file": "src/ahttpx/_parsers.py", + "line": 227, + "severity": "low", + "comment": "The async `HTTPParser.wait_ready()` (order 3, high risk) simply delegates to `self.parser.wait_ready()` which is the `ReadAheadParser` instance. The delegation is clean and follows the same pattern as other methods like `recv_method_line()`. The docstring correctly describes the contract. No issues with this layer itself, but it is worth noting that the `ReadAheadParser.wait_ready()` it calls uses `_read_some()` which in the async case performs actual I/O -- the `await` is correctly propagated through the chain." + }, + { + "file": "src/httpx/_server.py", + "line": 57, + "severity": "low", + "comment": "The new comment 'If the client hasn't read the request body to completion, then do that here' and 'Either revert to idle, or close the connection' clarify existing behavior around keep-alive draining and reset. These comments improve readability of the request loop and help future maintainers understand why `stream.read()` is called after sending the response. Good documentation improvement with no behavioral change." + } + ], + "summary": "The flow-guided review reveals that `wait_ready()` forms a clean delegation chain from server entry points through `HTTPParser` to `ReadAheadParser`, with the leaf `_push_back` assertion being the most fragile link -- its safety depends on `_reset()` fully draining the buffer between keep-alive cycles. The `HTTPServer.wait()` KeyboardInterrupt removal is an unrelated breaking change bundled into this PR that affects external callers." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review significantly outperforms the baseline by leveraging the call graph to trace the full `wait_ready()` delegation chain from server entry points through HTTPParser down to ReadAheadParser. This enabled it to identify that the `_push_back` assertion safety depends on `_reset()` behavior upstream -- a cross-file dependency the baseline only noted in isolation. The flow-guided review also correctly flagged that the async `handle_requests` has multiple callers, meaning a bug in `wait_ready` would affect both sync and async paths simultaneously. Both reviews caught the `KeyboardInterrupt` removal and the `_push_back` assertion risk, but the flow-guided review contextualized these findings within the dependency graph, explaining *why* they matter and under what conditions they would trigger. The baseline review treated each file independently and missed the end-to-end data flow implications." + } +} diff --git a/evals/encode__starlette__3189.json b/evals/encode__starlette__3189.json new file mode 100644 index 0000000..10763e3 --- /dev/null +++ b/evals/encode__starlette__3189.json @@ -0,0 +1,294 @@ +{ + "pr": { + "url": "https://github.com/encode/starlette/pull/3189", + "owner": "encode", + "repo": "starlette", + "number": 3189, + "title": "Handle websocket denial responses in streaming and file responses", + "files_changed": 3, + "additions": 128, + "deletions": 38, + "language": "python" + }, + "timestamp": "2026-03-30T18:30:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "starlette/responses.py", + "line": 154, + "severity": "positive", + "comment": "The _wrap_websocket_denial_send helper is a clean abstraction that transparently rewrites http.response.start/body message types to their websocket.http.response.* equivalents. This avoids duplicating prefix logic across Response, StreamingResponse, and FileResponse." + }, + { + "file": "starlette/responses.py", + "line": 157, + "severity": "nit", + "comment": "The pragma: no branch comment on the if-check suppresses branch coverage for the case where message type is neither http.response.start nor http.response.body. This is reasonable since the wrapper is only called with those two message types, but a brief comment explaining why would help future readers." + }, + { + "file": "starlette/responses.py", + "line": 257, + "severity": "major", + "comment": "StreamingResponse.__call__ introduces an early return for websocket scope that bypasses the ASGI 2.4 spec_version disconnect listener. While this is correct (websocket protocol handles disconnects differently), the duplicated background-task execution pattern (lines 260-261 mirror lines 281-282) is a minor maintenance risk if the background logic changes." + }, + { + "file": "starlette/responses.py", + "line": 344, + "severity": "minor", + "comment": "FileResponse.__call__ now guards scope['method'] and pathsend extension access behind scope_type == 'http'. This prevents a KeyError when scope is 'websocket' (which has no 'method' key). Good defensive fix." + }, + { + "file": "tests/test_responses.py", + "line": 611, + "severity": "nit", + "comment": "Existing tests updated to pass {'type': 'http'} in scope dict where previously an empty dict was used. This is more spec-correct and prevents the new scope type checks from raising KeyError." + }, + { + "file": "tests/test_responses.py", + "line": 656, + "severity": "positive", + "comment": "New test_streaming_response_runs_background_on_websocket_scope test verifies both that background tasks execute and that message types are correctly prefixed. The assertion on the ordered list of message types is a strong contract check." + } + ], + "summary": "This PR extends websocket denial response handling to StreamingResponse and FileResponse by introducing a send-wrapper in the base Response class. The approach is clean and well-tested, though the duplicated background task pattern in StreamingResponse's websocket early-return path is a minor maintenance concern." + }, + "flow_guided_review": { + "comments": [ + { + "file": "starlette/responses.py", + "line": 154, + "severity": "positive", + "comment": "Step 1 (wrapped/entry_point): The _wrap_websocket_denial_send method is the foundational building block for this PR. The plan identifies it as a high-risk entry point, and correctly so - if this wrapper has a bug, all three response classes (Response, StreamingResponse, FileResponse) will produce incorrect websocket messages. The implementation is simple and correct: it rewrites message types via string concatenation." + }, + { + "file": "starlette/responses.py", + "line": 163, + "severity": "minor", + "comment": "Step 2 (Response.__call__/entry_point): The plan shows this method calls _wrap_websocket_denial_send and is the parent for the entire response hierarchy. The refactoring from inline prefix concatenation to the wrapper pattern is correct. The plan's call graph reveals this is also called by StreamingResponse.stream_response indirectly, confirming the wrapper propagates through all code paths." + }, + { + "file": "starlette/responses.py", + "line": 257, + "severity": "major", + "comment": "Step 3 (StreamingResponse.__call__/entry_point): The plan flags this as high-risk. The early return for websocket scope bypasses the ASGI 2.4 disconnect listener, which is correct for websocket protocol. However, the background task execution is duplicated (websocket path lines 260-261 vs HTTP path lines 281-282). If background task handling changes, both paths must be updated. Consider extracting to a shared helper." + }, + { + "file": "starlette/responses.py", + "line": 341, + "severity": "minor", + "comment": "Step 4 (FileResponse.__call__/entry_point): The plan shows this is the last response class to handle. The scope_type guard on method/pathsend access prevents KeyError for websocket scope. The wrapper application follows the same pattern as Response and StreamingResponse, maintaining consistency across the hierarchy." + }, + { + "file": "tests/test_responses.py", + "line": 611, + "severity": "nit", + "comment": "Steps 5-6 (test fixes): The plan identifies these test modifications as entry points. Adding {'type': 'http'} to scope dicts is a necessary fix since StreamingResponse.__call__ now branches on scope['type']. These are low-risk but essential for correctness." + }, + { + "file": "tests/test_responses.py", + "line": 656, + "severity": "positive", + "comment": "Steps 7-8 (new test + run_background): The plan correctly identifies the new streaming websocket test and its background helper as entry points. The test validates the full websocket denial flow: message type rewriting, body streaming, and background task execution. The assertion on message type ordering is particularly strong." + }, + { + "file": "tests/test_websockets.py", + "line": 326, + "severity": "positive", + "comment": "Step 9 (integration test): The plan identifies the end-to-end websocket tests. These exercise StreamingResponse and FileResponse denial through the actual ASGI test client, complementing the unit-level test in test_responses.py." + } + ], + "summary": "Following the plan's topological order through the Response class hierarchy (wrapper -> Response.__call__ -> StreamingResponse.__call__ -> FileResponse.__call__ -> tests) reveals a consistent pattern: each response class applies the same wrapper before delegating to its response logic. The plan's dependency analysis confirms the wrapper is the single point of message-type translation, making this a well-factored change with one notable duplication concern in StreamingResponse's background task handling." + }, + "review_plan": { + "stats": { + "totalSteps": 65, + "totalAdditions": 128, + "totalDeletions": 38, + "independentFlows": 8, + "filesChanged": 3 + }, + "steps": [ + { + "order": 1, + "nodeId": "starlette/responses.py::Response.wrapped", + "name": "wrapped", + "file": "starlette/responses.py", + "lines": [155, 159], + "type": "function", + "changeType": "modified", + "additions": 5, + "deletions": 0, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 2, + "nodeId": "starlette/responses.py::Response.__call__", + "name": "__call__", + "file": "starlette/responses.py", + "lines": [163, 170], + "type": "method", + "changeType": "modified", + "additions": 4, + "deletions": 9, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [ + "starlette/responses.py::Response._wrap_websocket_denial_send", + "starlette/responses.py::StreamingResponse.stream_response", + "starlette/responses.py::FileResponse.set_stat_headers", + "starlette/responses.py::FileResponse._should_use_range", + "starlette/responses.py::FileResponse._handle_simple", + "starlette/responses.py::FileResponse._parse_range_header", + "starlette/responses.py::PlainTextResponse", + "starlette/responses.py::FileResponse._handle_single_range", + "starlette/responses.py::FileResponse._handle_multiple_ranges" + ], + "riskReasons": ["entry_point"] + }, + { + "order": 3, + "nodeId": "starlette/responses.py::StreamingResponse.__call__", + "name": "__call__", + "file": "starlette/responses.py", + "lines": [257, 284], + "type": "method", + "changeType": "modified", + "additions": 7, + "deletions": 0, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 4, + "nodeId": "starlette/responses.py::FileResponse.__call__", + "name": "__call__", + "file": "starlette/responses.py", + "lines": [341, 383], + "type": "method", + "changeType": "modified", + "additions": 5, + "deletions": 2, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 5, + "nodeId": "tests/test_responses.py::test_streaming_response_stops_if_receiving_http_disconnect", + "name": "test_streaming_response_stops_if_receiving_http_disconnect", + "file": "tests/test_responses.py", + "lines": [592, 620], + "type": "function", + "changeType": "modified", + "additions": 1, + "deletions": 1, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [ + "starlette/responses.py::StreamingResponse", + "tests/test_responses.py::stream_indefinitely" + ], + "riskReasons": ["entry_point"] + }, + { + "order": 6, + "nodeId": "tests/test_responses.py::test_streaming_response_on_client_disconnects", + "name": "test_streaming_response_on_client_disconnects", + "file": "tests/test_responses.py", + "lines": [623, 653], + "type": "function", + "changeType": "modified", + "additions": 1, + "deletions": 1, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [ + "tests/test_responses.py::stream_indefinitely", + "starlette/responses.py::StreamingResponse" + ], + "riskReasons": ["entry_point"] + }, + { + "order": 7, + "nodeId": "tests/test_responses.py::test_streaming_response_runs_background_on_websocket_scope", + "name": "test_streaming_response_runs_background_on_websocket_scope", + "file": "tests/test_responses.py", + "lines": [656, 683], + "type": "function", + "changeType": "modified", + "additions": 28, + "deletions": 0, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [ + "starlette/responses.py::StreamingResponse", + "tests/test_responses.py::stream" + ], + "riskReasons": ["entry_point"] + }, + { + "order": 8, + "nodeId": "tests/test_responses.py::run_background", + "name": "run_background", + "file": "tests/test_responses.py", + "lines": [667, 669], + "type": "function", + "changeType": "modified", + "additions": 3, + "deletions": 0, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 9, + "nodeId": "tests/test_websockets.py::test_send_denial_response_with_streaming_response", + "name": "test_send_denial_response_with_streaming_response", + "file": "tests/test_websockets.py", + "lines": [326, 343], + "type": "function", + "changeType": "modified", + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + } + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided_scores": { + "completeness": 9, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 8, + "overall": 8.2 + }, + "reasoning": "The flow-guided review benefits significantly from the plan's topological ordering through the Response class hierarchy. It traces the wrapper from its definition through each __call__ override, revealing the inheritance-based propagation pattern that the baseline review only partially captures. The flow-guided review also better identifies the single point of failure (the wrapper) and the duplication risk in StreamingResponse, because the plan's dependency graph shows exactly which methods depend on the wrapper. The baseline review catches the same major issue (StreamingResponse early return) but lacks the structural reasoning about why the wrapper pattern works across the hierarchy.", + "winner": "flow_guided" + } +} diff --git a/evals/eslint__eslint__20675.json b/evals/eslint__eslint__20675.json new file mode 100644 index 0000000..3de76b9 --- /dev/null +++ b/evals/eslint__eslint__20675.json @@ -0,0 +1,102 @@ +{ + "pr": "eslint/eslint#20675", + "title": "feat: add `Temporal` to `no-obj-calls`", + "baseline_review": { + "comments": [ + { + "file": "lib/rules/no-obj-calls.js", + "line": "23-29", + "severity": "medium", + "comment": "Adding `Temporal` to the `nonCallableGlobals` array is straightforward, but `Temporal` is still a Stage 3 proposal (not yet part of the ECMAScript standard). The other entries (`Math`, `JSON`, `Reflect`, `Atomics`, `Intl`) are all ratified standards. If `Temporal` ships with changes to its API surface or is dropped, this rule would produce false positives. Consider gating this behind an `ecmaVersion` check or adding a code comment noting the proposal status." + }, + { + "file": "tests/lib/rules/no-obj-calls.js", + "line": "111-123", + "severity": "medium", + "comment": "The two `globalThis.Temporal()` valid test cases use `ecmaVersion: 2015`, which predates `Temporal` by over a decade. Since `Temporal` would not exist as a global in ES2015, these tests pass vacuously (the variable is unresolved, so the rule skips it). The test with `globals: { Temporal: false }` is more meaningful, but the first case without explicit globals provides little coverage. Consider using `ecmaVersion: 2026` to test the actual intended scenario." + }, + { + "file": "tests/lib/rules/no-obj-calls.js", + "line": "480-530", + "severity": "low", + "comment": "The invalid test cases for `Temporal` are thorough and follow the same pattern as the existing `Intl` invalid cases. However, the diff appears truncated at the `globalThis` variant tests -- verify that the full set of `globalThis.Temporal()` and `new globalThis.Temporal()` invalid cases are included to match the coverage pattern of other globals like `Intl` and `Atomics`." + }, + { + "file": "docs/src/rules/no-obj-calls.md", + "line": "25-27", + "severity": "low", + "comment": "The documentation links to the Temporal proposal specification (`tc39.es/proposal-temporal`). This URL may change once Temporal is merged into the main ECMAScript spec. A comment or note indicating this is a proposal link would help future maintainers know to update it." + }, + { + "file": "docs/src/rules/no-obj-calls.md", + "line": "10", + "severity": "low", + "comment": "Minor documentation improvement: the existing typo 'due their capitalization' was correctly fixed to 'due to their capitalization' alongside adding Temporal to the list. Good catch on the incidental fix." + } + ], + "summary": "This PR adds `Temporal` to the `no-obj-calls` rule following the established pattern for other non-callable globals, with comprehensive test coverage for both valid and invalid cases. The main concern is that `Temporal` is still a Stage 3 proposal rather than a ratified standard, which differs from all other entries in the list and could lead to false positives if the proposal changes." + }, + "flow_guided_review": { + "comments": [ + { + "file": "lib/rules/no-obj-calls.js", + "line": "23-29", + "severity": "medium", + "comment": "The `nonCallableGlobals` array is a flat list with no version gating -- every entry is checked against the scope regardless of the configured `ecmaVersion`. For `Math`, `JSON`, `Reflect`, `Atomics`, and `Intl`, this works because ESLint's environment definitions include them in the appropriate globals. For `Temporal`, this means the rule's behavior depends entirely on whether `Temporal` appears in ESLint's configured globals for the target ecmaVersion. The tests confirm this works for `ecmaVersion: 2026`, but this implicit coupling should be documented or made explicit." + }, + { + "file": "tests/lib/rules/no-obj-calls.js", + "line": "111-123", + "severity": "medium", + "comment": "The valid test cases for `globalThis.Temporal()` use `ecmaVersion: 2015`. At ES2015, `Temporal` is not in the default globals, so the rule cannot resolve the `Temporal` reference via `globalThis` member access -- these tests pass because the global is simply unknown, not because the rule correctly identifies a shadowed or safe usage. To properly test `globalThis.Temporal()` as a valid case, a scenario where `Temporal` is in scope but being accessed safely (e.g., `Temporal.Now.instant()` via `globalThis`) would be more meaningful." + }, + { + "file": "tests/lib/rules/no-obj-calls.js", + "line": "59-68", + "severity": "low", + "comment": "The valid test cases (`Temporal.Now.instant()` and `new Temporal.Instant(0n)`) correctly demonstrate member access patterns that should not trigger the rule. These use `ecmaVersion: 2026`, which is appropriate. The coverage of both property access and constructor-on-property patterns is good and matches real-world Temporal API usage." + }, + { + "file": "tests/lib/rules/no-obj-calls.js", + "line": "201-215", + "severity": "low", + "comment": "The shadowed-variable valid tests for `Temporal` correctly mirror the pattern used for `Intl` (function parameter shadow and block-scoped const shadow). Using `ecmaVersion: 2026` for the block-scoped tests is appropriate. The `globals: { Temporal: false }` in the function parameter test ensures the base global is defined before being shadowed." + }, + { + "file": "docs/src/rules/no-obj-calls.md", + "line": "25-27", + "severity": "low", + "comment": "The Temporal specification link points to `tc39.es/proposal-temporal`, which is the proposal-stage URL. Once Temporal reaches Stage 4 and is integrated into the main ECMA-262 spec, this URL will likely become stale. The other entries link to versioned spec URLs (ES5, ES2017, ECMA-402). Consider adding a note that this link should be updated post-ratification." + }, + { + "file": "lib/rules/no-obj-calls.js", + "line": "26", + "severity": "low", + "comment": "The array is now alphabetically sorted except that `Intl` comes after `Reflect`. The original order was `Atomics, JSON, Math, Reflect, Intl` and the new entry `Temporal` is appended at the end. Since the array order does not affect behavior (it is iterated for matching), this is purely a style observation, but maintaining alphabetical order would improve scanability: `Atomics, Intl, JSON, Math, Reflect, Temporal`." + } + ], + "summary": "The change correctly extends the `no-obj-calls` rule to cover `Temporal` by adding it to the flat globals list, with the rule's scope-resolution mechanism handling version gating implicitly via ESLint's environment globals. Test coverage is comprehensive for direct usage patterns, though the `globalThis` valid-case tests at `ecmaVersion: 2015` exercise the \"unknown global\" path rather than the intended \"safe member access\" path." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 7, + "risk_identification": 6, + "actionability": 6, + "efficiency": 6, + "overall": 6.4 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify similar surface-level issues since this is a small, straightforward PR with an empty review plan (no steps, clusters, or dependencies). The flow-guided review provides slightly better analysis by tracing how the rule's scope-resolution mechanism implicitly gates Temporal behind ecmaVersion through ESLint's globals definitions, rather than just noting the proposal status. It also more precisely identifies why the globalThis tests at ecmaVersion 2015 are weak -- they exercise the 'unknown global' code path rather than the intended safe-usage path. However, the advantage is modest because the empty review plan provides no flow graph to leverage, and the PR itself is a simple additive change with no cross-file data flow complexity." + } +} \ No newline at end of file diff --git a/evals/etcd-io__etcd__21529.json b/evals/etcd-io__etcd__21529.json new file mode 100644 index 0000000..1e911d5 --- /dev/null +++ b/evals/etcd-io__etcd__21529.json @@ -0,0 +1,108 @@ +{ + "pr": "etcd-io/etcd#21529", + "title": "cache: Passive notification for consistent read", + "baseline_review": { + "comments": [ + { + "file": "cache/cache.go", + "line": 255, + "severity": "critical", + "comment": "The goroutine spawned to broadcast on timeout or context cancellation has no mechanism to be cleaned up if waitTillRevision returns normally (i.e., the revision is reached before timeout or cancellation). Neither timeoutCh nor ctx.Done() will fire, so the goroutine will leak, blocked forever in the select. Every successful call to waitTillRevision that completes before its timeout will leak one goroutine." + }, + { + "file": "cache/cache.go", + "line": 260, + "severity": "major", + "comment": "Using RLock with sync.Cond.Wait() means that Wait() will call RUnlock/RLock internally. However, the store's write methods (Restore, applyProgressNotifyLocked, applyEventsLocked) call Broadcast() while presumably holding the write lock. If a writer holds the write lock and calls Broadcast(), the waiting goroutine wakes up and tries to re-acquire the read lock via RLock -- this should work since RLock will block until the writer releases. But the key concern is that revCond.Wait() releases only the RLock, not a full Lock, so it is compatible. Verify that no code path calls revCond.Wait() while holding the full write lock, which would deadlock." + }, + { + "file": "cache/cache.go", + "line": 262, + "severity": "major", + "comment": "The time.Since(startTime) check after waking from revCond.Wait() introduces a race: time.After(c.cfg.WaitTimeout) fires and the goroutine calls Broadcast(), but the main goroutine could re-enter Wait() before checking time.Since. In practice this is unlikely because the Broadcast wakes Wait, and the next loop iteration checks time.Since before calling Wait again, but there is a subtle window where the timeout goroutine's Broadcast could be consumed by a different waiter, and this goroutine goes back to sleep past the timeout." + }, + { + "file": "cache/store.go", + "line": 168, + "severity": "minor", + "comment": "Broadcast() is called inside applyEventsLocked for every event's revision update within the loop. If a WatchResponse contains many events, this results in multiple unnecessary broadcasts. Consider moving the Broadcast() call after the loop completes, since waiters only care about the final revision." + }, + { + "file": "cache/cache_test.go", + "line": 808, + "severity": "minor", + "comment": "The timeout test now uses synctest.Test which is a Go 1.24+ testing/synctest feature. This introduces a new minimum Go version dependency. Verify that the project's go.mod and CI are aligned with Go 1.24+." + }, + { + "file": "cache/cache_test.go", + "line": 736, + "severity": "nit", + "comment": "Adding a context.WithTimeout of 2 seconds to the 'cache_already_caught_up' test is a good defensive change that prevents the test from hanging indefinitely if the new sync.Cond mechanism has a bug, but the timeout value is arbitrary -- consider using a shorter timeout since this case should return immediately." + } + ], + "summary": "The PR replaces polling-based revision waiting with sync.Cond-based passive notification, which is a solid architectural improvement. However, the goroutine spawned to handle timeout/cancellation broadcasting will leak on every successful fast path where the revision is already caught up or arrives before timeout, which is a critical issue that must be fixed before merging." + }, + "flow_guided_review": { + "comments": [ + { + "file": "cache/cache.go", + "line": 255, + "severity": "critical", + "comment": "Goroutine leak on normal completion: The goroutine launched at line 249 blocks on `select { case <-timeoutCh: ... case <-ctx.Done(): ... }`. When waitTillRevision succeeds because the revision catches up, neither channel fires, so this goroutine is permanently leaked. Every successful waitTillRevision call leaks one goroutine. Fix by creating a done channel that is closed when waitTillRevision returns, and adding it as a third case in the select." + }, + { + "file": "cache/store.go", + "line": 151, + "severity": "major", + "comment": "Broadcast() in applyProgressNotifyLocked and applyEventsLocked is called while the store's write lock is held (these are *Locked methods). The revCond uses mu.RLocker(), so Wait() releases/acquires the read lock. When Broadcast() is called under the write lock, woken goroutines will block on RLock until the write lock is released. This is correct but means notification delivery is delayed until the write lock drops. More importantly, if any future code calls Broadcast() without holding any lock, Wait() could wake and find stale data. Document the invariant that Broadcast must always be called while holding mu (read or write)." + }, + { + "file": "cache/cache.go", + "line": 262, + "severity": "major", + "comment": "The time.Since check and ctx.Err check after Wait() are subject to spurious wakeups from sync.Cond (which are possible per Go docs). While the timeout goroutine's Broadcast handles the timeout case, a spurious wakeup could cause the loop to re-check and re-Wait without issues. However, the real concern is ordering: if the timeout goroutine fires Broadcast but the main goroutine has not yet entered Wait(), the broadcast is lost and the goroutine will block on Wait() forever (only rescued by a subsequent revision update or context cancellation). This is mitigated by the time.Since check at the top of the loop, but only if the goroutine wakes up from Wait for some other reason." + }, + { + "file": "cache/store.go", + "line": 168, + "severity": "minor", + "comment": "Inside applyEventsLocked, Broadcast() is called once per event iteration. For a WatchResponse with N events, this wakes waiting goroutines N times, each time they re-acquire the read lock, check the condition, and go back to Wait. Moving the Broadcast after the for loop would reduce unnecessary wake-ups and lock contention." + }, + { + "file": "cache/store.go", + "line": 44, + "severity": "minor", + "comment": "Using s.mu.RLocker() as the Cond locker is a deliberate choice to allow concurrent readers while waiting. This is correct but unusual -- most sync.Cond usage uses a full Mutex. Add a comment explaining why RLocker is used here (to avoid blocking store reads while goroutines wait for revision updates), since future maintainers may not realize the subtlety." + }, + { + "file": "cache/config.go", + "line": 18, + "severity": "positive", + "comment": "Removing the revisionPollInterval constant is a clean cleanup -- it was only used by the polling mechanism being replaced, and leaving dead constants creates confusion for future readers." + } + ], + "summary": "The transition from polling to sync.Cond-based notification is architecturally sound and eliminates the 50ms polling overhead. The critical issue is the goroutine leak when waitTillRevision completes successfully before timeout -- a done channel must be added to allow the cleanup goroutine to exit. The Broadcast-under-write-lock pattern works correctly but should be documented as an invariant." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 6.6 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 6, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews correctly identify the critical goroutine leak as the top issue and the unnecessary per-event Broadcast as a performance concern. The flow-guided review edges ahead on flow awareness by tracing the Broadcast call sites through store.go's locked methods and connecting them to the Cond's RLocker semantics, explaining why woken goroutines block until the write lock drops. It also provides a concrete fix suggestion (done channel) for the goroutine leak and flags the need to document the Broadcast-under-lock invariant. The baseline review raises similar points but with less structural tracing -- for example, it mentions the RLock/Broadcast interaction but treats it as a verification request rather than analyzing the code flow. The review plan was empty (no steps/clusters), which limits the flow-guided review's advantage; with a populated plan, the flow-guided review could have demonstrated stronger architectural reasoning about the store -> cache -> waitTillRevision dependency chain. Both reviews are efficient and avoid redundant comments." + } +} diff --git a/evals/etcd-io__etcd__21547.json b/evals/etcd-io__etcd__21547.json new file mode 100644 index 0000000..93cb987 --- /dev/null +++ b/evals/etcd-io__etcd__21547.json @@ -0,0 +1,102 @@ +{ + "pr": "etcd-io/etcd#21547", + "title": "Extract common traffic subfunction between robustness and Antithesis code", + "baseline_review": { + "comments": [ + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 157, + "severity": "medium", + "comment": "The `Client` field has been removed from `RunTrafficLoopParam` but the new `SimulateKeyValueTraffic` function is expected to create clients and pass them separately. Ensure all callers (both Antithesis and robustness tests) have been updated to match the new signature. The diff is truncated, so the robustness side is not visible here -- verify that it compiles and passes tests." + }, + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 167, + "severity": "medium", + "comment": "The error handling for `SimulateKeyValueTraffic` calls `assert.Unreachable` and then `os.Exit(1)`. The same pattern is repeated for `SimulateWatchTraffic` and `SimulateCompactionTraffic`. This duplicated error-handling block should be extracted into a helper function (e.g., `fatalOnConnectError`) to reduce repetition and ensure consistent error messaging." + }, + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 138, + "severity": "high", + "comment": "After the refactoring, `profile.Compaction` could be nil (it is guarded by `if profile.Compaction != nil` at line 129), but line 138 accesses `profile.Compaction.Period` unconditionally for `defragPeriod` computation. This nil dereference risk existed before the refactor, but the restructured code makes it easier to miss. Confirm that `defragPeriod` computation is also guarded by the nil check." + }, + { + "file": "tests/robustness/traffic/etcd.go", + "line": 149, + "severity": "medium", + "comment": "The `RunKeyValueLoop` and `RunWatchLoop` methods now take an explicit `*client.RecordingClient` parameter in addition to the params struct. This is a good separation of concerns (the client is a connection, not a config parameter), but ensure the `Traffic` interface definition has been updated to match this new signature, as the diff does not show the interface file." + }, + { + "file": "tests/robustness/traffic/etcd.go", + "line": 102, + "severity": "low", + "comment": "The `RunKeyValueLoop` method signature now has `(ctx context.Context, c *client.RecordingClient, p RunTrafficLoopParam)` while the internal variable is also named `client` (line 153: `client := etcdTrafficClient{...}`). The parameter `c` and the local variable `client` are different types, but the naming is potentially confusing. Consider renaming the parameter to `rc` or `recordingClient` for clarity." + } + ], + "summary": "This PR extracts duplicated goroutine-launching loops into shared `SimulateKeyValueTraffic`, `SimulateWatchTraffic`, and `SimulateCompactionTraffic` functions, reducing code duplication between the Antithesis and robustness test harnesses. The refactoring is structurally sound but introduces repeated error-handling blocks that should be consolidated, and the truncated diff makes it impossible to verify the new shared functions or the updated interface definition." + }, + "flow_guided_review": { + "comments": [ + { + "file": "tests/robustness/traffic/etcd.go", + "line": 149, + "severity": "medium", + "comment": "The signature change to `RunKeyValueLoop` and `RunWatchLoop` -- adding an explicit `*client.RecordingClient` parameter -- is the foundational change that enables the extraction. This means the `Traffic` interface must also be updated. Since the review plan is empty (no steps or dependencies), we must manually trace: any type implementing `Traffic` (e.g., `etcdTraffic`, and potentially others) must update all three methods. Verify no implementations are missed." + }, + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 157, + "severity": "medium", + "comment": "The new `SimulateKeyValueTraffic` function takes `profile.KeyValue`, `hosts`, `clientSet`, the traffic instance, and the param struct. The `Client` field was removed from `RunTrafficLoopParam`, meaning client creation is now the responsibility of the Simulate functions. This is a clean separation, but since the Simulate function implementations are not shown in the diff, we cannot verify that client lifecycle management (creation, `defer Close()`, goroutine cleanup via `wg`) is correctly handled inside them." + }, + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 167, + "severity": "low", + "comment": "The error returned by `SimulateKeyValueTraffic` is handled with `assert.Unreachable` + `os.Exit(1)`. This pattern is repeated three times (KeyValue, Watch, Compaction). While the Antithesis `assert.Unreachable` is appropriate for the Antithesis testing framework, the robustness test caller (not shown) likely needs different error handling. Verify the robustness caller handles errors from these new shared functions appropriately for its own test framework." + }, + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 129, + "severity": "high", + "comment": "The `SimulateCompactionTraffic` call is guarded by `if profile.Compaction != nil`, but immediately after this block (line 138), `profile.Compaction.Period` is accessed for the `defragPeriod` calculation without a nil guard. If `profile.Compaction` is nil, this will panic. This bug appears to predate the refactor but the restructured code flow makes it more apparent. The defrag block should also be guarded by the nil check, or the defrag logic should be moved inside the compaction block." + }, + { + "file": "tests/robustness/traffic/etcd.go", + "line": 165, + "severity": "medium", + "comment": "The `RunWatchLoop` signature changed to accept `(ctx context.Context, c *client.RecordingClient, p RunWatchLoopParam)` and passes `c` to the internal `runWatchLoop`. The internal `runWatchLoop` function signature must also have been updated to accept the client parameter. Since the diff is truncated, verify that `runWatchLoop` and `RunCompactLoop` (not shown) have consistent parameter ordering and that no callers were missed." + }, + { + "file": "tests/antithesis/test-template/robustness/traffic/main.go", + "line": 104, + "severity": "low", + "comment": "The `RunWatchLoopParam` still includes `Config: *profile.Watch` in the param struct passed to `SimulateWatchTraffic`, but the `Watch` profile is also passed as a separate argument to `SimulateWatchTraffic`. This means the watch config may be passed in two places -- once to configure the Simulate function (for client counts) and once inside the param (for loop behavior). Verify there is no redundancy or inconsistency between these two paths." + } + ], + "summary": "The extraction refactors duplicated goroutine management into shared Simulate functions, cleanly separating client lifecycle from traffic loop parameters. Without the review plan providing flow or risk guidance, manual tracing reveals potential concerns around the Traffic interface update completeness, a pre-existing nil dereference on `profile.Compaction.Period`, and the need to verify that both callers (Antithesis and robustness) handle the new shared function errors appropriately for their respective frameworks." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 6, + "overall": 5.6 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 6, + "risk_identification": 7, + "actionability": 7, + "efficiency": 6, + "overall": 6.6 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews are handicapped by the truncated diff (the new shared Simulate functions and updated interface are not visible) and the empty review plan (no steps, clusters, or dependencies to guide analysis). The flow-guided review still edges ahead by attempting to trace the dependency chain: interface change -> method signature change -> new shared functions -> caller error handling. It identifies the redundant Watch config passing and raises the cross-caller concern (Antithesis vs robustness error handling). The baseline review catches the same nil dereference bug and the repeated error handling, but treats the changes more in isolation without tracing how the interface change propagates. The empty plan significantly limits the flow-guided review's advantage -- with a proper plan identifying the Simulate functions as high-risk entry points and mapping the interface dependency, the flow-guided review would have been substantially stronger." + } +} \ No newline at end of file diff --git a/evals/facebook__react__36024.json b/evals/facebook__react__36024.json new file mode 100644 index 0000000..dd0e784 --- /dev/null +++ b/evals/facebook__react__36024.json @@ -0,0 +1,137 @@ +{ + "pr": { + "url": "https://github.com/facebook/react/pull/36024", + "owner": "facebook", + "repo": "react", + "number": 36024, + "title": "[Flight] Clear chunk reason after successful module initialization", + "files_changed": 3, + "additions": 94, + "deletions": 0, + "language": "javascript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": 1080, + "severity": "major", + "comment": "Setting `initializedChunk.reason = null` after successful initialization in initializeModelChunk is the right fix, but the TODO comment added at line 1043 ('The chunk might have transitioned to ERRORED now. Should we return early if that happens?') suggests there is a deeper issue: if initializeDebugChunk transitions the chunk to ERRORED, the subsequent try block will overwrite the ERRORED status with INITIALIZED. This could mask real errors. The TODO should be addressed in this PR or tracked as a follow-up issue." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": 1103, + "severity": "minor", + "comment": "The same `chunk.reason = null` pattern is applied in initializeModuleChunk. This is the core fix for the described bug -- clearing stale error reasons after successful module initialization prevents the TypeError when the stream closes. The fix is correct and minimal." + }, + { + "file": "packages/react-server/src/ReactFlightReplyServer.js", + "line": 481, + "severity": "minor", + "comment": "Setting `initializedPromise.reason = null` in loadServerReference is a defensive fix for consistency with the client-side changes. However, the PR description does not mention this code path as being affected by the reentrancy bug. A comment explaining why this is needed (or whether it is purely precautionary) would help reviewers understand the scope." + }, + { + "file": "packages/react-server-dom-webpack/src/__tests__/ReactFlightDOM-test.js", + "line": 1459, + "severity": "minor", + "comment": "The custom __webpack_require__ mock simulates reentrancy by throwing a TDZ ReferenceError when evaluatingModuleId matches the current id. This is a clever simulation, but it only tests the DEV path (the captureOwnerStack call is gated by __DEV__). There is no coverage for production builds where the reentrancy might be triggered by a different mechanism." + }, + { + "file": "packages/react-server-dom-webpack/src/__tests__/ReactFlightDOM-test.js", + "line": 1472, + "severity": "nit", + "comment": "The test restores global.__webpack_require__ after serverAct but before resolveAsyncComponent. If the Fizz render triggers any additional module resolution after this point, it would use the original require. This ordering dependency should be documented with a comment." + }, + { + "file": "packages/react-server-dom-webpack/src/__tests__/ReactFlightDOM-test.js", + "line": 1421, + "severity": "positive", + "comment": "Excellent regression test that directly reproduces the reported crash scenario -- reentrant readChunk during module evaluation leaving a stale error on chunk.reason, then the stream close crashing on chunk.reason.error(). The test structure clearly separates the Flight rendering, Fizz SSR consumption, and stream closure phases." + } + ], + "summary": "The fix correctly clears chunk.reason after successful initialization in three locations, preventing stale error objects from crashing the stream close logic. The TODO comment about early return on ERRORED status after initializeDebugChunk suggests an additional edge case that should be tracked." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/react-server-dom-webpack/src/__tests__/ReactFlightDOM-test.js", + "line": 1422, + "severity": "major", + "comment": "ENTRY POINT / HIGH RISK (MyComponent): This test component is the client export that triggers the reentrancy bug. The test correctly wires it through clientExports() and uses it in ServerComponent, but the reentrancy simulation is only exercised in __DEV__ mode via captureOwnerStack(). The plan identifies this as high-risk because it is the entry point -- if the mock __webpack_require__ does not accurately simulate the real reentrancy path, the test could pass without exercising the actual bug. Consider adding an explicit assertion that the TDZ error was actually thrown and caught during initialization (e.g., a counter or flag)." + }, + { + "file": "packages/react-server-dom-webpack/src/__tests__/ReactFlightDOM-test.js", + "line": 1428, + "severity": "major", + "comment": "ENTRY POINT / HIGH RISK (AsyncComponent): This async component keeps the Flight stream open long enough for the module chunk to be initialized before the stream closes. This is critical -- without it, the stream might close before initializeModuleChunk runs, and the stale reason would never be encountered. However, the test does not verify that the stream actually closes after module initialization. If the timing changes (e.g., stream closes before resolveAsyncComponent), the test could pass vacuously. Adding an assertion between resolveAsyncComponent and readInto that confirms the Flight stream has closed would strengthen the test." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": 1043, + "severity": "major", + "comment": "DEPENDENCY: The TODO comment reveals that initializeDebugChunk (called before the try block in initializeModelChunk) can transition the chunk to ERRORED via reentrant readChunk. The fix at line 1080 clears reason on the INITIALIZED path, but if the chunk was already ERRORED by initializeDebugChunk and then the try block succeeds anyway, we are overwriting an ERRORED chunk with INITIALIZED status. This is a correctness concern beyond the scope of clearing reason -- the status itself may be incorrect. This should be filed as a follow-up issue." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": 1080, + "severity": "minor", + "comment": "DEPENDENCY on initializeModelChunk flow: Setting reason = null here is correct for the model chunk path. The flow from the test shows: module chunk initialization triggers readChunk reentrantly, which calls initializeModelChunk, which may error and set reason. After the outer requireModule succeeds, initializeModuleChunk sets status = INITIALIZED but reason retains the stale error. This fix breaks the stale reference. The null assignment is safe because initialized chunks have no use for reason until the stream close iteration, at which point null is correctly handled." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": 1103, + "severity": "minor", + "comment": "CORE FIX for the module chunk path: This is the most critical of the three reason = null assignments because initializeModuleChunk is where the described bug manifests. The flow is: requireModule -> module evaluation -> reentrant readChunk -> error sets chunk.reason -> outer requireModule succeeds -> initializeModuleChunk sets INITIALIZED but reason still holds Error. Stream close then calls chunk.reason.error() expecting FlightStreamController, gets TypeError. This single line prevents the crash." + }, + { + "file": "packages/react-server/src/ReactFlightReplyServer.js", + "line": 481, + "severity": "minor", + "comment": "INDEPENDENT FLOW: This change in loadServerReference is on the server reply path, separate from the client-side chunk initialization. The reentrancy scenario described in the PR does not directly apply here, but the same pattern (chunk transitions to INITIALIZED with a potentially stale reason) could occur if the bound promise resolution triggers side effects. This is a defensive fix -- no test covers this specific path, which is a gap." + }, + { + "file": "packages/react-server-dom-webpack/src/__tests__/ReactFlightDOM-test.js", + "line": 1459, + "severity": "nit", + "comment": "The mock __webpack_require__ uses evaluatingModuleId as a reentrancy guard -- it throws only when the same module is required while being evaluated. This accurately models TDZ errors in real module evaluation. The guard is correctly reset (evaluatingModuleId = null) after captureOwnerStack returns, preventing false positives on subsequent requires." + } + ], + "summary": "The flow-guided analysis reveals that the fix addresses three independent initialization paths where chunk.reason could retain stale errors, with the initializeModuleChunk path being the primary bug site. The test exercises the reentrancy through a well-designed __webpack_require__ mock, though the TODO comment about initializeDebugChunk transitioning chunks to ERRORED suggests a deeper correctness issue where status (not just reason) may be incorrectly overwritten." + }, + "judgment": { + "criteria": { + "completeness": { + "baseline": 7, + "flow_guided": 8, + "rationale": "Baseline covers all three fix locations and the test, but treats the ReactFlightReplyServer change as an afterthought. Flow-guided review explicitly categorizes it as an independent flow and identifies the missing test coverage for that path." + }, + "flow_awareness": { + "baseline": 5, + "flow_guided": 9, + "rationale": "Baseline reviews each file change in isolation. Flow-guided review traces the full reentrancy chain: module evaluation -> reentrant readChunk -> error sets reason -> outer initialization succeeds -> stale reason -> stream close crash, and maps each fix to its position in this chain." + }, + "risk_identification": { + "baseline": 6, + "flow_guided": 8, + "rationale": "Baseline flags the TODO comment and DEV-only test coverage. Flow-guided review goes deeper by identifying that the TODO reveals a status overwrite issue (ERRORED -> INITIALIZED) beyond just the stale reason, and that the test could pass vacuously if stream close timing changes." + }, + "actionability": { + "baseline": 6, + "flow_guided": 7, + "rationale": "Baseline suggests documenting the ordering dependency and tracking the TODO. Flow-guided review adds concrete suggestions: assert the TDZ error was thrown, verify stream close timing, and file a follow-up for the ERRORED status overwrite." + }, + "efficiency": { + "baseline": 7, + "flow_guided": 7, + "rationale": "Both reviews stay focused on the three-file change. Neither introduces off-topic concerns. The flow-guided review is slightly more verbose but each comment adds analytical value." + } + }, + "overall": { + "baseline": 6.2, + "flow_guided": 7.8, + "winner": "flow_guided", + "rationale": "The flow-guided review is stronger because it traces the complete reentrancy chain from module evaluation through chunk state transitions to stream close, revealing that the TODO comment about initializeDebugChunk points to a deeper correctness issue (status overwrite, not just stale reason). It also identifies that the test could pass vacuously without timing guarantees and that the ReactFlightReplyServer fix lacks test coverage. The baseline review correctly identifies the surface-level concerns but misses the architectural connection between the three fix sites." + } + } +} diff --git a/evals/facebook__react__36134.json b/evals/facebook__react__36134.json new file mode 100644 index 0000000..97c5583 --- /dev/null +++ b/evals/facebook__react__36134.json @@ -0,0 +1,137 @@ +{ + "pr": { + "url": "https://github.com/facebook/react/pull/36134", + "owner": "facebook", + "repo": "react", + "number": 36134, + "title": "Fix useDeferredValue getting stuck", + "files_changed": 3, + "additions": 72, + "deletions": 0, + "language": "javascript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/react-reconciler/src/ReactFiberWorkLoop.js", + "line": 5045, + "severity": "major", + "comment": "The fix merges pingedLanes into workInProgressRootPingedLanes so markRootSuspended won't mark them as suspended. However, we are inside a branch where workInProgressRoot === root and the render is in progress. If a concurrent interruption resets workInProgressRootPingedLanes before markRootSuspended reads it, the pinged lanes could be lost. Verify that no code path between this assignment and markRootSuspended clears the variable." + }, + { + "file": "packages/react-reconciler/src/ReactFiberWorkLoop.js", + "line": 5045, + "severity": "minor", + "comment": "The comment above says 'we once we add it back we can use it here' -- this pre-existing typo ('we once we') should be cleaned up while touching this area." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1116, + "severity": "minor", + "comment": "The Sibling component conditionally calls resolveText inside render, which is a side effect during render. While this is intentional to simulate data arriving mid-render, a brief comment at the test level explaining why this pattern is acceptable in a test (but not in production code) would improve clarity." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1140, + "severity": "nit", + "comment": "The assertLog array mixes two render passes without a separator comment between them. Adding inline comments like '// First render pass' and '// Retry after ping' would make the expected sequence easier to follow." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactSuspenseWithNoopRenderer-test.js", + "line": 4057, + "severity": "minor", + "comment": "The updated assertLog now expects a retry with 'Suspend! [A]', 'Loading A...', 'B' after the initial render. This means the existing test's behavior changed -- previously B did not render. The commit message should explicitly call out that this is an intentional behavior change in the existing test, not just a comment update." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1148, + "severity": "positive", + "comment": "Good regression test that directly exercises the reported bug scenario -- useDeferredValue getting stuck when a suspension is resolved during the same render. The test structure clearly separates initial render from the update that triggers the bug." + } + ], + "summary": "The fix is a small, targeted change that records pinged lanes during an in-progress render so they are not incorrectly marked as suspended, allowing retry. The regression test directly covers the reported issue, though the behavioral change in the existing Suspense test deserves explicit acknowledgment." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/react-reconciler/src/ReactFiberWorkLoop.js", + "line": 5045, + "severity": "major", + "comment": "ENTRY POINT / HIGH RISK: pingSuspendedRoot is the core fix. The mergeLanes call adds pingedLanes to workInProgressRootPingedLanes during an active render. This is correct because markRootSuspended (called later when the render completes) checks workInProgressRootPingedLanes to decide which lanes to mark suspended. However, this introduces a new data-flow dependency: if any code path between here and markRootSuspended calls prepareFreshStack (which resets workInProgressRootPingedLanes to NoLanes), the fix would be silently defeated. The guard above (workInProgressRoot === root) helps, but a defensive comment documenting this invariant would be valuable." + }, + { + "file": "packages/react-reconciler/src/ReactFiberWorkLoop.js", + "line": 5045, + "severity": "minor", + "comment": "The function calls markRootPinged earlier in the same block (line ~5025). The new code writes to workInProgressRootPingedLanes instead, which is a different mechanism -- one updates the fiber root's pingedLanes, the other updates the work-in-progress tracking variable. Both are needed for correctness but the duality is subtle. A comment linking the two would help future maintainers understand why both are necessary." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1104, + "severity": "minor", + "comment": "ENTRY POINT: The App component uses useDeferredValue but does not test the case where the deferred value is used as a key or in a memoized child. The bug report (issue #35821) may have additional reproduction scenarios worth covering -- e.g., what happens if the component using the deferred value is wrapped in React.memo?" + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1116, + "severity": "major", + "comment": "ENTRY POINT / HIGH RISK: Sibling calls resolveText('A:' + text) during render, which triggers a synchronous ping of the suspended resource. This is the exact scenario that triggers the bug. However, the test relies on resolveText firing the ping synchronously and the internal detail that React checks workInProgressRootPingedLanes. If the promise resolution mechanism changes (e.g., to microtask-based), this test might pass vacuously without exercising the fix. Consider adding an assertion that the deferred value actually catches up (which the final toMatchRenderedOutput does cover)." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactSuspenseWithNoopRenderer-test.js", + "line": 4057, + "severity": "minor", + "comment": "DEPENDENCY on pingSuspendedRoot fix: The updated assertions show that with the fix, the synchronous ping now causes a retry where B renders successfully. This is a behavioral change from the previous test expectation. The test comment update ('The synchronous ping was recorded, so B retries and renders') is good but should note this is a consequence of the workInProgressRootPingedLanes fix specifically." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1130, + "severity": "nit", + "comment": "Pre-resolving 'B:updated' before the act() block is a critical setup step that's easy to miss. Consider adding a comment: '// Pre-resolve B so the retry render won't suspend on Sibling'." + }, + { + "file": "packages/react-reconciler/src/__tests__/ReactDeferredValue-test.js", + "line": 1148, + "severity": "positive", + "comment": "The final assertion toMatchRenderedOutput('A:updatedupdated') proves both AsyncText and Sibling rendered with the updated deferred value, confirming the deferred value is no longer stuck. This is the strongest proof the bug is fixed." + } + ], + "summary": "The fix correctly addresses the root cause: when a suspension is resolved synchronously during render via pingSuspendedRoot, the pinged lanes must be recorded in workInProgressRootPingedLanes so markRootSuspended does not incorrectly mark them as suspended. The flow analysis reveals a subtle invariant -- workInProgressRootPingedLanes must survive until markRootSuspended reads it -- that should be documented to prevent future regressions." + }, + "judgment": { + "criteria": { + "completeness": { + "baseline": 6, + "flow_guided": 8, + "rationale": "Baseline identifies the core fix and test changes but misses the relationship between markRootPinged and workInProgressRootPingedLanes. Flow-guided review traces the data flow from pingSuspendedRoot through to markRootSuspended, identifying the invariant that must hold." + }, + "flow_awareness": { + "baseline": 4, + "flow_guided": 9, + "rationale": "Baseline treats each file change independently. Flow-guided review follows the call chain from pingSuspendedRoot to markRootPinged and connects the behavioral change in the Suspense test to the specific mechanism in the fix." + }, + "risk_identification": { + "baseline": 6, + "flow_guided": 8, + "rationale": "Baseline flags the potential for workInProgressRootPingedLanes to be cleared. Flow-guided review goes further by identifying the specific risk (prepareFreshStack resetting the variable) and the test fragility around synchronous vs microtask-based resolution." + }, + "actionability": { + "baseline": 6, + "flow_guided": 7, + "rationale": "Both reviews suggest concrete improvements (comments, documentation). Flow-guided adds specific suggestions about documenting the invariant and linking the dual ping mechanisms." + }, + "efficiency": { + "baseline": 7, + "flow_guided": 7, + "rationale": "Both reviews stay focused on the 3-file change without going off-topic. Neither wastes time on irrelevant concerns." + } + }, + "overall": { + "baseline": 5.8, + "flow_guided": 7.8, + "winner": "flow_guided", + "rationale": "The flow-guided review is substantially stronger because it traces the data-flow dependency from the pingSuspendedRoot fix through workInProgressRootPingedLanes to markRootSuspended, identifying the critical invariant that the variable must not be reset between assignment and consumption. It also connects the behavioral change in the existing Suspense test to the specific mechanism of the fix. The baseline review identifies surface-level concerns but misses the deeper architectural relationship between the dual ping mechanisms." + } + } +} diff --git a/evals/facebook__react__36156.json b/evals/facebook__react__36156.json new file mode 100644 index 0000000..180cbde --- /dev/null +++ b/evals/facebook__react__36156.json @@ -0,0 +1,102 @@ +{ + "pr": "facebook/react#36156", + "title": "[Flight] Transport AggregateErrors.errors", + "baseline_review": { + "comments": [ + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3540-3541", + "severity": "medium", + "comment": "The `typeof AggregateError !== 'undefined'` check correctly guards environments where AggregateError is not available. However, combining it with `'errors' in errorInfo` means that if the server sends an `errors` field but the client lacks AggregateError, it silently falls back to a plain Error and the `errors` data is lost. Consider logging a dev warning when `errors` is present but AggregateError is unavailable." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3528-3530", + "severity": "low", + "comment": "Adding `__DEV__ &&` before `'cause' in errorInfo` is a correctness improvement that avoids unnecessary deserialization in prod. Good change, but the inline comment could be more precise — it's not that cause isn't serialized in prod, but that the full dev error info path (`resolveErrorDev`) is only called in dev anyway, so the guard is technically redundant here. Consider whether this guard is actually needed given the function is `resolveErrorDev`." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3555-3572", + "severity": "medium", + "comment": "The `isAggregateError` ternary choosing between `AggregateError.bind(...)` and `Error.bind(...)` duplicates the message fallback string and the `errorOptions` argument. Extract the common message into a variable above to reduce duplication and the risk of the two branches diverging." + }, + { + "file": "packages/react-client/src/__tests__/ReactFlight-test.js", + "line": "842-1033", + "severity": "low", + "comment": "The test is thorough and covers AggregateError with nested errors, cause chains, and environment name propagation. However, at ~200 lines for a single test case, consider whether splitting into smaller focused tests (e.g., basic AggregateError transport, AggregateError with cause, nested AggregateError) would improve readability and failure diagnosis." + }, + { + "file": ".eslintrc.js", + "line": "569", + "severity": "low", + "comment": "Adding `AggregateError` as a global is correct — it removes the need for eslint-disable comments. The alphabetical ordering is slightly off (it's placed before `AnimationFrameID` but after `ReturnType`), though this matches the existing somewhat inconsistent ordering in this file." + } + ], + "summary": "This PR adds dev-only transport of `AggregateError.errors` through React Flight, with proper guards for environments lacking AggregateError. The implementation is sound but has some code duplication in the error construction branches that could be reduced, and the test, while comprehensive, is quite long for a single test case." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3540-3549", + "severity": "high", + "comment": "ENTRY POINT / HIGH RISK: `resolveErrorDev` now revives `errorInfo.errors` via `reviveModel` before constructing the error. If the server sends malformed or deeply nested `errors` arrays, `reviveModel` will recursively process them. There is no depth or size limit on the errors array. While this is dev-only, a pathological server response could still cause excessive processing. Consider adding a sanity check on the errors array length." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3555-3572", + "severity": "medium", + "comment": "ENTRY POINT: The branching between `AggregateError.bind(null, revivedErrors, message, errorOptions)` and `Error.bind(null, message, errorOptions)` is the core behavioral change. Note that `AggregateError` constructor signature is `(errors, message, options)` while `Error` is `(message, options)` — the argument order difference is correctly handled. However, the duplicate message fallback string across both branches is a maintenance risk. Extract it." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3528-3530", + "severity": "low", + "comment": "The `__DEV__` guard on the `cause` deserialization path is technically redundant since this entire function is `resolveErrorDev` and should only be called in dev. If it serves as a tree-shaking hint for bundlers, that's worth a comment; otherwise it adds confusion about when this code path runs." + }, + { + "file": "packages/react-client/src/__tests__/ReactFlight-test.js", + "line": "842-860", + "severity": "low", + "comment": "INTERNAL (test helper): `renderError` recursively renders error properties including `errors` and `cause`. The recursive call for `cause` and the `.map()` for `errors` could theoretically infinite-loop if an error's cause or errors array references itself. In practice this won't happen in tests, but a depth guard would make the helper more robust." + }, + { + "file": "packages/react-client/src/__tests__/ReactFlight-test.js", + "line": "862-874", + "severity": "medium", + "comment": "The test constructs `AggregateError([error1, error2], 'aggregate')` and verifies the transported result. This is the happy path. Missing test cases: (1) AggregateError with an empty errors array, (2) AggregateError where one of the inner errors is itself an AggregateError (nested), and (3) AggregateError with a cause. The test does cover some of these in the later longer assertion blocks, but the test structure makes it hard to verify each scenario independently." + }, + { + "file": "packages/react-client/src/ReactFlightClient.js", + "line": "3540", + "severity": "medium", + "comment": "DEPENDENCY FLOW: The `revivedErrors` variable is computed before the `buildFakeCallStack` call and passed into the `AggregateError.bind()`. Since `reviveModel` may return a lazy/thenable value rather than an immediately-resolved array, verify that `AggregateError` constructor handles the case where `revivedErrors` is not yet a plain array at construction time. If `reviveModel` returns a proxy or thenable, the AggregateError's `errors` property could be unexpected." + } + ], + "summary": "The flow analysis reveals that `resolveErrorDev` is the sole entry point for this change, with the critical path being: detect AggregateError in errorInfo, revive the errors array via `reviveModel`, then branch on constructor choice. The highest risk is around the `reviveModel` return type for the errors array — if it returns a lazy/thenable rather than a plain array, the AggregateError construction could behave unexpectedly. Code duplication in the constructor branches is a secondary concern." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 5, + "actionability": 7, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identified deeper risks that the baseline missed: the reviveModel return type concern (lazy/thenable vs plain array), the lack of depth/size limits on the errors array processing, and the dependency between reviveModel output and AggregateError constructor expectations. The baseline review caught surface-level issues like code duplication and test length but missed the core semantic risks around the data flow through reviveModel into AggregateError construction. The flow-guided approach's use of the plan's risk annotations and dependency ordering led to more targeted and higher-value findings." + } +} diff --git a/evals/go-gitea__gitea__37019.json b/evals/go-gitea__gitea__37019.json new file mode 100644 index 0000000..72a2f67 --- /dev/null +++ b/evals/go-gitea__gitea__37019.json @@ -0,0 +1,108 @@ +{ + "pr": "go-gitea/gitea#37019", + "title": "Update message severity colors, fix navbar double border", + "baseline_review": { + "comments": [ + { + "file": "web_src/css/modules/navbar.css", + "line": 10, + "severity": "medium", + "comment": "The `:has()` CSS selector is used here (`#navbar:has(+ .ui.message)`). While `:has()` has good modern browser support (Chrome 105+, Firefox 121+, Safari 15.4+), Gitea may still need to support older browsers. Verify that the project's browser support matrix includes `:has()`. If not, a fallback approach (e.g., adding a class to `#navbar` via JS when a message is present) would be needed." + }, + { + "file": "web_src/css/modules/navbar.css", + "line": 11, + "severity": "low", + "comment": "Using `border-bottom: none` removes the entire border property. If the navbar border is defined with shorthand (e.g., `border-bottom: 1px solid color`), this works fine. However, if only `border-bottom-color` or `border-bottom-width` is set elsewhere with higher specificity, `none` might not fully override. Consider `border-bottom-width: 0` for a more targeted override." + }, + { + "file": "web_src/css/themes/theme-gitea-light.css", + "line": 165, + "severity": "low", + "comment": "The new border colors use 8-digit hex values with alpha (`#ff818266`, `#4ac26b66`, `#d4a72c66`, `#54aeff66`). While 8-digit hex color notation is well-supported in modern browsers, it is worth confirming this is consistent with how other colors in the codebase are defined. The dark theme uses standard 6-digit hex for borders, creating an inconsistency between the two theme files." + }, + { + "file": "web_src/css/themes/theme-gitea-dark.css", + "line": 176, + "severity": "medium", + "comment": "All four severity text colors (`--color-error-text`, `--color-success-text`, `--color-warning-text`, `--color-info-text`) are now set to `var(--color-text)`. This removes visual differentiation of message text by severity. While the background and border colors still differ, users who rely on text color to distinguish severity (especially in contexts where background color may not be visible or for accessibility reasons) may find this harder to parse at a glance." + }, + { + "file": "web_src/css/modules/message.css", + "line": 46, + "severity": "low", + "comment": "Replacing `filter: saturate(2)` with `font-weight: var(--font-weight-semibold)` is a significant visual change to message headers. The saturate filter made header text color more vivid, while semibold makes it bolder. Since text colors are now all `var(--color-text)`, the saturate filter would have had no effect anyway (saturating a neutral color does nothing), so this change is consistent with the text color unification." + }, + { + "file": "web_src/css/themes/theme-gitea-dark.css", + "line": 165, + "severity": "low", + "comment": "The `--color-error-bg-active` and `--color-error-bg-hover` values were updated but the same hover/active variants for success, warning, and info are not defined (they weren't before either). If these messages support hover/active states, the missing variants for non-error severities could result in inconsistent interaction feedback." + } + ], + "summary": "This CSS-only PR updates severity message colors to use more muted Primer-aligned tokens and fixes a navbar double-border issue. The changes are straightforward and consistent across both themes, though the use of `:has()` for the navbar fix should be validated against the project's browser support matrix, and the removal of per-severity text colors reduces visual differentiation that some users may rely on." + }, + "flow_guided_review": { + "comments": [ + { + "file": "web_src/css/themes/theme-gitea-dark.css", + "line": 176, + "severity": "medium", + "comment": "All four `--color-*-text` variables now resolve to `var(--color-text)`, meaning error, warning, success, and info messages all use the same text color. This is a deliberate design choice to rely on background and border for severity differentiation. However, this may reduce accessibility for users with color vision deficiencies who struggle to distinguish the muted background tones, especially in dark mode where contrast ratios between the new backgrounds and default text should be verified (e.g., `#322226` background with the default text color)." + }, + { + "file": "web_src/css/themes/theme-gitea-light.css", + "line": 176, + "severity": "medium", + "comment": "Same pattern applied to the light theme -- all `--color-*-text` set to `var(--color-text)`. The light theme backgrounds (`#ffebe9`, `#dafbe1`, `#fff8c5`, `#ddf4ff`) are pastel tones that should provide decent contrast with dark text, but this should be confirmed with WCAG contrast ratio checks to ensure AA compliance." + }, + { + "file": "web_src/css/themes/theme-gitea-light.css", + "line": 165, + "severity": "low", + "comment": "The light theme border colors use 8-digit hex with alpha channel (`#ff818266`, `#4ac26b66`, etc.) while the dark theme borders use opaque 6-digit hex (`#763232`, `#225633`, etc.). This asymmetry means light theme borders are semi-transparent and will blend with underlying backgrounds, while dark theme borders are solid. This could produce inconsistent visual weight between themes depending on what's behind the message element." + }, + { + "file": "web_src/css/modules/message.css", + "line": 46, + "severity": "low", + "comment": "Replacing `filter: saturate(2)` with `font-weight: var(--font-weight-semibold)` on `.ui.message .header` is the correct companion change to the text color unification. Since text is now always `var(--color-text)`, saturating a neutral color would be a no-op. The semibold weight provides an alternative emphasis mechanism. Ensure `--font-weight-semibold` is defined in the base theme variables." + }, + { + "file": "web_src/css/modules/navbar.css", + "line": 10, + "severity": "medium", + "comment": "The `:has()` pseudo-class in `#navbar:has(+ .ui.message)` requires the `.ui.message` to be an immediate next sibling of `#navbar` in the DOM. If the page structure ever changes (e.g., a wrapper div is added between navbar and message), this selector will silently break. A comment documenting the expected DOM structure would help future maintainers. Also verify `:has()` is within the project's supported browser baseline." + }, + { + "file": "web_src/css/modules/navbar.css", + "line": 11, + "severity": "low", + "comment": "Using `border-bottom: none` is a clean fix for the double-border issue. The adjacent sibling combinator (`+`) in the `:has()` selector ensures this only fires when a message immediately follows the navbar, which is the correct scope. This is preferable to removing the border from the message element's top, as it keeps the message styling self-contained." + } + ], + "summary": "The PR coherently updates severity message styling across both themes and fixes a navbar border issue. The main concerns are accessibility -- removing per-severity text colors means severity is communicated only through background and border color, which should be validated for WCAG compliance -- and the reliance on `:has()` with a specific DOM structure for the navbar fix." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.6 + } + }, + "winner": "flow_guided", + "reasoning": "With an empty review plan (no steps, clusters, or dependencies), the flow-guided review had minimal structural advantage to leverage. Both reviews identify similar issues: browser compatibility of `:has()`, accessibility concerns around removing per-severity text colors, and the light/dark theme border inconsistency. The flow-guided review edges ahead slightly by more explicitly connecting the text color unification to accessibility risks (WCAG contrast ratios, color vision deficiencies) and by noting the DOM structure fragility of the `:has(+)` selector. However, the margin is small because this is a straightforward CSS change with no complex data flow or inter-file dependencies that a richer plan would have surfaced. The baseline review's point about missing hover/active variants for non-error severities was a valid observation not covered by the flow-guided review." + } +} diff --git a/evals/go-gitea__gitea__37029.json b/evals/go-gitea__gitea__37029.json new file mode 100644 index 0000000..59c4141 --- /dev/null +++ b/evals/go-gitea__gitea__37029.json @@ -0,0 +1,114 @@ +{ + "pr": "go-gitea/gitea#37029", + "title": "Fix various problems", + "baseline_review": { + "comments": [ + { + "file": "modules/public/mime_types.go", + "line": 43, + "severity": "major", + "comment": "The fallback to `mime.TypeByExtension(ext)` reintroduces the exact unstable behavior that `DetectWellKnownMimeType` was designed to avoid. The function's own doc comment says it exists to bypass the unstable behavior of Go's `mime.TypeByExtension`, which reads OS-level MIME databases that differ across platforms. Now, for any extension not in the well-known list, the function silently delegates to the unstable source. Callers that relied on getting an empty string for unknown extensions will now get potentially inconsistent results across OS environments." + }, + { + "file": "modules/public/mime_types.go", + "line": 20, + "severity": "minor", + "comment": "Wrapping the map literal in `sync.OnceValue` adds lazy initialization but the original package-level map was already initialized once at program start. The only benefit would be if the map is never accessed in some execution paths, which seems unlikely for a MIME type lookup used in serving static assets. This adds complexity (function call on every lookup) for marginal benefit." + }, + { + "file": "modules/git/catfile_batch_reader.go", + "line": 49, + "severity": "minor", + "comment": "The `closeFunc` closure captures both `ctxCancel` and `pipeClose`, combining two cleanup steps into one callable. This is a good refactor that eliminates the repeated `ctxCancel(err); pipeClose()` pattern seen in the old code. However, naming it `closeFunc` when the struct also has a `Close()` method could be confusing -- consider `cleanupFunc` or similar to disambiguate." + }, + { + "file": "modules/git/catfile_batch_reader.go", + "line": 57, + "severity": "minor", + "comment": "The new `newCatFileBatchWithCloseFunc` function is only called from one place (`newCatFileBatch`). Extracting it into a separate function is reasonable for testability, but its current signature (taking `stdinWriter`, `stdoutReader`, and `closeFunc`) tightly couples it to the pipe-based setup. If there are no other intended callers, inlining might be cleaner." + }, + { + "file": "web_src/css/markup/content.css", + "line": 169, + "severity": "positive", + "comment": "Replacing physical CSS properties (`padding-left`, `margin-right`, `margin-left`, `padding-right`) with logical properties (`padding-inline-start`, `margin-inline-end`, `margin-inline-start`, `padding-inline-end`) is the correct approach for RTL language support. This is a clean, systematic fix for issue #37027." + }, + { + "file": "web_src/css/markup/content.css", + "line": 187, + "severity": "nit", + "comment": "The shorthand `margin: 0 .6em .25em -1.4em` was expanded into `margin-bottom: 0.25em` and `margin-inline: -1.4em 0.6em`, but the original also set `margin-top: 0`. The new code does not explicitly set `margin-top: 0`, relying on browser defaults or inherited styles. If the checkbox element has a non-zero default margin-top, this could change the rendering." + } + ], + "summary": "This PR bundles three independent fixes: CSS logical properties for RTL support, MIME type fallback, and git cat-file pipe cleanup. The CSS changes are clean and correct, the git communicator refactor properly consolidates cleanup logic, but the MIME fallback to `mime.TypeByExtension` partially contradicts the function's stated purpose of providing stable cross-platform MIME detection." + }, + "flow_guided_review": { + "comments": [ + { + "file": "modules/git/catfile_batch_reader.go", + "line": 29, + "severity": "minor", + "comment": "Renaming `cancel` to `closeFunc` and changing its type from `context.CancelFunc` to `func(err error)` is the key structural change enabling pipe cleanup. The old field only cancelled the context; the new one also closes pipes. This is correct but note that the `Close()` method passes `nil` as the error, meaning a manual close is treated as a no-error cancellation -- this matches the old `cancel()` behavior since `context.CancelFunc` took no arguments." + }, + { + "file": "modules/git/catfile_batch_reader.go", + "line": 49, + "severity": "minor", + "comment": "The `closeFunc` closure correctly composes `ctxCancel(err)` followed by `pipeClose()`. The ordering matters: cancelling the context first signals the running goroutine to stop, then closing pipes ensures file descriptors are released. This matches the old behavior in the error and goroutine paths (lines 72 and 83 in old code) where `ctxCancel` was called before `pipeClose`." + }, + { + "file": "modules/git/catfile_batch_reader.go", + "line": 57, + "severity": "minor", + "comment": "Extracting `newCatFileBatchWithCloseFunc` separates struct construction from pipe/context setup. This enables the old 1.25 behavior where `Close()` would clean up both context and pipes, which was the regression reported in #37028. The function is well-structured but is currently only called from one site -- its utility as a separate function would be clearer with a code comment explaining it exists for testability or alternative construction paths." + }, + { + "file": "modules/public/mime_types.go", + "line": 46, + "severity": "major", + "comment": "Adding `mime.TypeByExtension(ext)` as a fallback directly undermines the documented purpose of `DetectWellKnownMimeType`. The doc comment (lines 39-42) explicitly states the function exists because Go's `mime.TypeByExtension` reads system MIME databases producing different Content-Type headers across platforms. The fallback means any extension not in the well-known map will still hit the unstable path. If this is intentional, the doc comment must be updated to reflect the new contract: 'stable for well-known types, system-dependent for others.'" + }, + { + "file": "modules/public/mime_types.go", + "line": 20, + "severity": "nit", + "comment": "The `sync.OnceValue` wrapper defers map allocation until first call. Since this is a static map of ~18 entries used in the hot path for serving every static file, the lazy initialization adds a function call overhead on every invocation (the closure is called each time, though the inner init runs once). The original package-level var was simpler and had zero per-call overhead." + }, + { + "file": "web_src/css/markup/content.css", + "line": 187, + "severity": "nit", + "comment": "The decomposition of the `margin` shorthand into `margin-bottom` + `margin-inline` drops the explicit `margin-top: 0` that was part of the original shorthand. While checkboxes typically have zero top margin by default, this is a subtle behavior change. Adding `margin-top: 0` explicitly would preserve the original intent." + }, + { + "file": "web_src/css/markup/content.css", + "line": 169, + "severity": "positive", + "comment": "The systematic replacement of physical CSS properties with logical properties across all markup content selectors is thorough and consistent. All six property replacements correctly map physical directions to their inline/block logical equivalents, which will fix RTL layout issues reported in #37027." + } + ], + "summary": "The git cat-file pipe cleanup correctly restores the 1.25 behavior where `Close()` releases both context and pipe resources, fixing the resource leak in #37028. The MIME type fallback to `mime.TypeByExtension` is the most concerning change as it contradicts the function's documented stability guarantee and could reintroduce platform-dependent Content-Type headers for non-well-known extensions." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews correctly identify the main risk: the MIME fallback contradicting the function's documented purpose. The flow-guided review adds value by tracing the git catfile changes through the data flow -- explaining why the closure ordering (cancel then pipe close) matters, connecting the refactor to the 1.25 regression in #37028, and noting how the extracted function enables the old cleanup semantics. The baseline treats the git changes more superficially, focusing on naming concerns rather than behavioral correctness. However, the review plan was empty (zero steps, zero clusters), which limited the flow-guided review's advantage -- it could not leverage dependency or risk annotations to prioritize its analysis. With a richer plan, the gap would likely be wider. The flow-guided review also provides a more actionable suggestion for the MIME issue (update the doc comment to reflect the new contract) rather than just flagging the inconsistency." + } +} \ No newline at end of file diff --git a/evals/go-gitea__gitea__37030.json b/evals/go-gitea__gitea__37030.json new file mode 100644 index 0000000..ec374f0 --- /dev/null +++ b/evals/go-gitea__gitea__37030.json @@ -0,0 +1,114 @@ +{ + "pr": "go-gitea/gitea#37030", + "title": "Correct swagger annotations for enums, status codes, and notification state", + "baseline_review": { + "comments": [ + { + "file": "modules/structs/issue.go", + "line": 28, + "severity": "high", + "comment": "Changing `StateAll` from a typed `StateType` constant to an untyped string constant (`const StateAll = \"all\"`) is a breaking change for any code that passes `StateAll` to a function expecting `StateType`. Callers using `StateAll` in type-switch or comparison contexts against `StateType` variables will now get a compilation error or subtle behavioral difference. All internal usages across the codebase must be verified to ensure they still compile and behave correctly." + }, + { + "file": "modules/structs/notifications.go", + "line": 43, + "severity": "high", + "comment": "Changing `NotificationSubject.State` from `StateType` to the new `NotifySubjectStateType` is a breaking API change. Any existing API consumers deserializing notification responses into typed structs will break if they reference `StateType` for this field. Additionally, `NotifySubjectStateType` adds a `merged` value that `StateType` did not have -- existing clients may not handle this new enum value. The PR description does not call this out as a breaking change (only the 204 status code change is listed)." + }, + { + "file": "modules/structs/notifications.go", + "line": 67, + "severity": "medium", + "comment": "The `NotifySubjectStateType` enum defines `open`, `closed`, and `merged`, but there is no closing parenthesis or additional sentinel value shown in the diff. More importantly, notification subjects include `Repository` and `Commit` types (per the `binding:\"In(Issue,Pull,Commit,Repository)\"` tag on line 42), and it is unclear what state a `Commit` or `Repository` notification subject would have. Consider whether an empty or additional state value is needed for these subject types." + }, + { + "file": "modules/structs/issue.go", + "line": 64, + "severity": "low", + "comment": "Removing the inline swagger comment (`// type: string` and `// enum: open,closed`) from the `State` field in the `Issue` struct is correct since the type-level `swagger:enum` annotation on `StateType` now handles this. However, the removal also drops the human-readable comment `Whether the issue is open or closed` which provided useful context in the struct definition. Consider keeping a brief doc comment on the field." + }, + { + "file": "modules/structs/issue_milestone.go", + "line": 55, + "severity": "low", + "comment": "Adding the `enum: [\"open\",\"closed\"]` annotation to `EditMilestoneOption.State` is a good fix -- it was missing before. However, `State` is a `*string` (pointer), so the enum constraint applies only when the value is non-nil. Verify that swagger tooling correctly handles the nullable + enum combination so that omitting the field entirely is still valid in requests." + }, + { + "file": "modules/structs/issue.go", + "line": 133, + "severity": "low", + "comment": "Changing `IssueFormFieldVisible` from `swagger:model` to `swagger:enum` is the correct annotation for a string enum type. The previous `swagger:model` annotation would have generated an incorrect schema definition (as an object rather than an enum string). Good catch." + } + ], + "summary": "This PR correctly migrates swagger enum annotations from comma-separated to JSON array format and adds proper `swagger:enum` type annotations for better OpenAPI spec generation. The main concerns are the breaking change to `StateAll` (untyped from `StateType`) and the undocumented breaking change of switching `NotificationSubject.State` from `StateType` to the new `NotifySubjectStateType`, which introduces a `merged` value that existing API clients may not expect." + }, + "flow_guided_review": { + "comments": [ + { + "file": "modules/structs/issue.go", + "line": 19, + "severity": "medium", + "comment": "The `swagger:enum StateType` annotation is added to generate a proper enum schema, but `StateType` now only contains `open` and `closed`. Since `StateAll` was extracted to an untyped constant, any code path that previously accepted `StateType` and compared against `StateAll` will need updating. This is the root of a dependency chain: every file that references `StateAll` as a `StateType` value is affected. Without the flow plan identifying these dependents, the risk of missed compilation errors across the 24 files is high." + }, + { + "file": "modules/structs/notifications.go", + "line": 55, + "severity": "high", + "comment": "The introduction of `NotifySubjectStateType` as a separate enum from `StateType` is the most impactful change in this PR. It creates a new type with three values (open, closed, merged) that diverges from `StateType` (open, closed). This is architecturally sound -- notification subjects can be merged PRs, which issues cannot -- but it is an API-breaking change that is not listed in the PR's breaking changes section. The delete-reaction 204 change is called out but this type change is not, despite having broader client impact." + }, + { + "file": "modules/structs/notifications.go", + "line": 43, + "severity": "medium", + "comment": "The field type change from `StateType` to `NotifySubjectStateType` on `NotificationSubject.State` means the JSON serialization now comes from a different Go type. While the wire format for `open` and `closed` is identical, any server-side code that converts between `StateType` and this field (e.g., when populating notification subjects from issue/PR state) will need explicit conversion. The diff only shows struct changes -- the service layer converting issue/PR state to notification state must also be updated." + }, + { + "file": "modules/structs/issue.go", + "line": 28, + "severity": "high", + "comment": "Extracting `StateAll` to an untyped `string` constant breaks type safety. Previously `StateAll StateType = \"all\"` could be passed anywhere a `StateType` was expected. Now callers must use `StateType(StateAll)` for explicit conversion. Since this is in the `structs` package which is imported across the entire Gitea codebase (routers, services, models), this change likely requires updates in many files not shown in this diff. The diff only shows 24 files with 375 lines changed -- confirm all call sites compile." + }, + { + "file": "modules/structs/activity.go", + "line": 15, + "severity": "low", + "comment": "The enum format migration from comma-separated to JSON array is mechanical and correct. The `OpType` field on `Activity` is a plain `string` (not a typed enum), so it does not get a `swagger:enum` type annotation -- this is consistent since the values are action types populated by the server, not user-provided enum inputs." + }, + { + "file": "modules/structs/hook.go", + "line": 54, + "severity": "low", + "comment": "Same mechanical enum format migration for `CreateHookOption.Type`. The JSON array format `[\"dingtalk\",\"discord\",...,\"packagist\"]` is correct. Note that this field uses `binding:\"Required\"` but does not reference a typed enum -- if a `swagger:enum` type were introduced for hook types (similar to what was done for `StateType`), it would provide better validation. This is out of scope for this PR but worth noting as a follow-up." + }, + { + "file": "modules/structs/issue.go", + "line": 134, + "severity": "low", + "comment": "The `swagger:enum IssueFormFieldVisible` annotation replaces the incorrect `swagger:model` annotation. Combined with the `swagger:enum IssueFormFieldType` addition at line 133, these two enum types will now generate proper string enum schemas in the OpenAPI spec. This is essential for the stated goal of enabling code generation tools like Rust's progenitor." + } + ], + "summary": "The PR has two distinct layers of change: mechanical swagger annotation format migrations (low risk, correct) and semantic type system changes (high risk, potentially breaking). The `StateAll` extraction and `NotifySubjectStateType` introduction are architecturally motivated but carry significant cross-codebase impact that is not fully visible in this diff, and the notification state type change is not listed as a breaking change despite altering the API contract." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 5, + "risk_identification": 9, + "actionability": 7, + "efficiency": 7, + "overall": 7.2 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews correctly identify the two main risks: the `StateAll` type change and the `NotifySubjectStateType` introduction. However, the flow-guided review does a better job of tracing the dependency chain -- explaining that `StateAll` being untyped affects every caller in the `structs` package importers, and that the `NotificationSubject.State` type change requires service-layer conversion code not shown in the diff. The flow-guided review also explicitly calls out that the notification state type change is a breaking API change not listed in the PR description, while the baseline notes this less prominently. The flow_awareness scores are modest for both because the review plan was empty (no steps or clusters), limiting the structural advantage the flow-guided approach normally provides. The flow-guided review compensates by manually reasoning about cross-file dependencies." + } +} \ No newline at end of file diff --git a/evals/grafana__grafana__121418.json b/evals/grafana__grafana__121418.json new file mode 100644 index 0000000..282a78d --- /dev/null +++ b/evals/grafana__grafana__121418.json @@ -0,0 +1,102 @@ +{ + "pr": "grafana/grafana#121418", + "title": "FS: Remove per-tenant CSP override", + "baseline_review": { + "comments": [ + { + "file": "pkg/services/frontend/request_config.go", + "line": 126, + "severity": "high", + "comment": "The removal of `applyBool` and `applyString` overrides for CSP-related settings (`content_security_policy`, `content_security_policy_template`, `content_security_policy_report_only`, `content_security_policy_report_only_template`) means tenants can no longer override these values. If any existing tenant relies on per-tenant CSP overrides (e.g., a tenant that had CSP disabled or a custom template), this change will silently revert them to the base config. Confirm no tenants currently use these overrides before shipping." + }, + { + "file": "pkg/services/frontend/request_config.go", + "line": 175, + "severity": "low", + "comment": "The `applyBool` helper function is removed entirely since it has no remaining callers after the CSP override lines are deleted. This is good cleanup -- no dead code is left behind." + }, + { + "file": "pkg/services/frontend/frontend_service_test.go", + "line": 571, + "severity": "medium", + "comment": "The updated test now sets `CSPTemplate` to include `frame-ancestors $ALLOW_EMBEDDING_HOSTS` and asserts that the rendered header contains `frame-ancestors 'none'` when `allow_embedding_hosts` is empty. This is a good behavioral test, but it implicitly relies on the template variable substitution replacing `$ALLOW_EMBEDDING_HOSTS` with `'none'` when the list is empty. A comment or separate test clarifying that substitution logic would improve readability." + }, + { + "file": "pkg/services/frontend/frontend_service_test.go", + "line": 586, + "severity": "medium", + "comment": "The new test case 'should apply per-tenant allow_embedding_hosts override to CSP header' is a valuable addition that validates the remaining per-tenant override path works correctly. However, the diff appears truncated -- the last test file `request_config_middleware_test.go` is cut off at line 138, so it is unclear whether remaining test changes are complete and consistent." + }, + { + "file": "pkg/services/frontend/request_config_middleware_test.go", + "line": 74, + "severity": "low", + "comment": "The `CSPTemplate` in multiple test cases is updated from `default-src 'self'` to `default-src 'self'; frame-ancestors $ALLOW_EMBEDDING_HOSTS`. This aligns tests with the new expectation that CSP templates always include the `$ALLOW_EMBEDDING_HOSTS` placeholder, reflecting the shift from per-tenant CSP overrides to relying on `allow_embedding_hosts` exclusively." + } + ], + "summary": "This PR cleanly removes per-tenant CSP override support in favor of the simpler `allow_embedding_hosts` mechanism, including good cleanup of the now-unused `applyBool` helper. The main risk is whether any existing tenants rely on the removed per-tenant CSP overrides, which would silently regress their configuration." + }, + "flow_guided_review": { + "comments": [ + { + "file": "pkg/services/frontend/request_config.go", + "line": 126, + "severity": "high", + "comment": "The core behavioral change: four `apply*` calls for CSP settings are removed from `ApplyOverrides`, while `applyStringSlice` for `allow_embedding_hosts` is retained. This narrows the per-tenant override surface to only the embedding hosts list. The risk is that any tenant currently using per-tenant `content_security_policy` or `content_security_policy_template` overrides will lose them silently -- there is no migration path or error logging when those keys are present in tenant settings but now ignored." + }, + { + "file": "pkg/services/frontend/request_config.go", + "line": 175, + "severity": "low", + "comment": "The `applyBool` helper is fully removed since it has zero remaining callers after the CSP lines are deleted. This is correct dead-code cleanup. Note that `applyString` and `applyStringSlice` still have callers (for `rudderstack_write_key`, `allow_embedding_hosts`, etc.) so they are correctly retained." + }, + { + "file": "pkg/services/frontend/frontend_service_test.go", + "line": 563, + "severity": "medium", + "comment": "The renamed test 'should disallow iframing when allow_embedding_hosts is empty' now sets `CSPTemplate` with the `$ALLOW_EMBEDDING_HOSTS` placeholder and asserts `frame-ancestors 'none'` in the output. This proves the system correctly defaults to blocking iframes when no embedding hosts are configured -- a critical security invariant. The rename accurately reflects the new behavior." + }, + { + "file": "pkg/services/frontend/frontend_service_test.go", + "line": 586, + "severity": "medium", + "comment": "The new test 'should apply per-tenant allow_embedding_hosts override to CSP header' validates the end-to-end flow: tenant settings override `allow_embedding_hosts`, which is substituted into the CSP template's `$ALLOW_EMBEDDING_HOSTS` placeholder. This is the replacement mechanism for the removed per-tenant CSP overrides. The test uses a mock settings service with two hosts and verifies they appear in `frame-ancestors`, confirming the override-to-CSP-header pipeline works correctly." + }, + { + "file": "pkg/services/frontend/request_config_middleware_test.go", + "line": 74, + "severity": "low", + "comment": "Test templates are updated to include `frame-ancestors $ALLOW_EMBEDDING_HOSTS` consistently. This reflects the assumption that CSP templates in production will always contain this placeholder. If the base CSP template in production does not include this placeholder, the `allow_embedding_hosts` mechanism will silently do nothing -- verify that the default production CSP template has been updated (possibly in PR #120702)." + }, + { + "file": "pkg/services/frontend/request_config_middleware_test.go", + "line": 98, + "severity": "low", + "comment": "The assertion is updated to expect the full template string including the placeholder. This confirms the middleware passes the template through unchanged (the placeholder is not resolved at the middleware layer). The actual substitution happens later in the CSP rendering pipeline." + } + ], + "summary": "The PR correctly removes per-tenant CSP override machinery and replaces it with a simpler model where only `allow_embedding_hosts` is overridable per-tenant, substituted via the `$ALLOW_EMBEDDING_HOSTS` template variable. The key risk is ensuring no existing tenants depend on the removed CSP overrides and that the production base CSP template already includes the `$ALLOW_EMBEDDING_HOSTS` placeholder (presumably done in PR #120702)." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides better end-to-end understanding of the change by tracing how the removal of CSP overrides in `ApplyOverrides` connects to the retained `allow_embedding_hosts` path, and how the test cases validate the full pipeline from tenant settings through CSP template substitution. It identifies the additional risk that the production CSP template must already contain the `$ALLOW_EMBEDDING_HOSTS` placeholder for the new mechanism to work. However, the advantage is modest because the review plan was empty (no steps, clusters, or dependencies), so the flow-guided review could not leverage structured flow information and instead had to reason about the change holistically. Both reviews correctly identify the primary risk of silently dropping existing tenant CSP overrides." + } +} \ No newline at end of file diff --git a/evals/grafana__grafana__121425.json b/evals/grafana__grafana__121425.json new file mode 100644 index 0000000..2d50143 --- /dev/null +++ b/evals/grafana__grafana__121425.json @@ -0,0 +1,131 @@ +{ + "pr": { + "url": "https://github.com/grafana/grafana/pull/121425", + "owner": "grafana", + "repo": "grafana", + "number": 121425, + "title": "Dashboard: Use preferredVersion from discovery endpoint for API version negotiation", + "files_changed": 3, + "additions": 24, + "deletions": 8, + "language": "typescript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "public/app/features/dashboard-scene/pages/DashboardScenePageStateManager.ts", + "line": 449, + "severity": "major", + "comment": "The `await dashboardAPIVersionResolver.resolve()` call is added before every `loadScene` invocation, including the home dashboard path which the comment says 'is not handled through legacy API and is not versioned.' If home dashboards do not use versioned APIs, this await adds unnecessary latency on the home dashboard load path. Consider moving the resolve() call after the home dashboard early return, or clarifying in a comment why it is needed for home dashboards too." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts", + "line": 76, + "severity": "minor", + "comment": "The code reads `group.preferredVersion?.version` with optional chaining, which means if the discovery response lacks a `preferredVersion` field entirely, `preferred` will be `undefined` and the resolution falls through to the availability-based scan. This is a reasonable defensive default, but a debug log when preferredVersion is missing would help operators diagnose misconfiguration." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts", + "line": 79, + "severity": "minor", + "comment": "The new resolution logic uses `preferred` to select v1/v2 versions based on the preferredVersion family. When preferredVersion is e.g. 'v1beta1', v1 is set to 'v1beta1' but v2 still falls through to the `availableVersions.has('v2')` scan. This asymmetric behavior (preferred overrides one family, scan picks the other) is correct but could be surprising. A code comment explaining this design decision would help future maintainers." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 101, + "severity": "minor", + "comment": "The test 'should respect preferredVersion=v1beta1 even when v1 is available' verifies that preferred downgrades v1 to beta while v2 stays at stable. This is a good edge case test. However, there is no test for the case where preferredVersion is completely absent (undefined) from the discovery response, which would exercise the optional chaining fallback path." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 119, + "severity": "minor", + "comment": "The test 'should fall back to version scanning when preferredVersion is for a different family' sets preferred='v1' and asserts v2='v2'. The test description says 'different family' but v1 is the same resource, just a different major version family. The description could be more precise: 'preferredVersion only overrides its own version family, other families use availability scanning.'" + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 14, + "severity": "nit", + "comment": "The `mockDiscoveryResponse` helper defaults `preferred` to `versions[0]` when not specified. This couples the default behavior in tests to array ordering rather than to the actual backend behavior (which uses the ini config). This is fine for existing tests but worth noting if the backend's default preferred version logic ever diverges from 'first in list.'" + } + ], + "summary": "The PR correctly replaces hardcoded version selection with preferredVersion-based resolution and fixes a race condition by awaiting resolve() before downstream calls. The main concern is whether the resolve() call should be placed after the home dashboard early return to avoid unnecessary latency on that path." + }, + "flow_guided_review": { + "comments": [ + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 14, + "severity": "minor", + "comment": "ENTRY POINT (mockDiscoveryResponse): The test helper now accepts an optional `preferred` parameter, defaulting to `versions[0]`. This is the foundation for all preferredVersion tests. The default coupling to array index 0 means existing tests implicitly assume the backend's preferred version matches the first listed version. If the backend ever returns versions in a different order or picks a different default, these tests would not catch the mismatch. Consider adding a test where preferred is explicitly the non-first element to validate this code path more thoroughly." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts", + "line": 76, + "severity": "major", + "comment": "ENTRY POINT / HIGH RISK (DashboardAPIVersionResolver.discover): The `preferred` variable is extracted via optional chaining from `group.preferredVersion?.version`. This is the core behavioral change -- shifting from hardcoded `availableVersions.has()` checks to preferredVersion-driven selection. The risk is that if the backend returns an unexpected preferredVersion value (e.g., 'v3alpha1' or a malformed string), the ternary conditions won't match any branch and both v1/v2 will fall through to the availability scan. This is actually safe behavior, but there is no validation or warning log for unrecognized preferred versions, which could make debugging difficult in production." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts", + "line": 79, + "severity": "minor", + "comment": "DEPENDENCY (discover -> resolve flow): The v1/v2 resolution ternaries check `preferred === 'v1' || preferred === 'v1beta1'` and `preferred === 'v2' || preferred === 'v2beta1'` respectively. These string literals are not derived from any shared constant or type -- if a new version like 'v1alpha1' or 'v2beta2' is introduced, this code must be updated manually. Consider using a regex or startsWith check (e.g., `preferred?.startsWith('v1')`) to be more forward-compatible, or document the coupling to known version strings." + }, + { + "file": "public/app/features/dashboard-scene/pages/DashboardScenePageStateManager.ts", + "line": 449, + "severity": "major", + "comment": "INTERNAL NODE (loadScene): The `await dashboardAPIVersionResolver.resolve()` is placed before the home dashboard route check. Per the plan, loadScene calls resolve, then branches to loadHomeDashboard or fetchDashboard. The comment on line 452 states home dashboards are 'not versioned,' yet this await will still fire the discovery HTTP request on every home dashboard load. Since resolve() likely caches internally, this may be low-cost after the first call, but on cold start it adds latency to the critical home dashboard path. Moving the await after the home dashboard early return would be more efficient." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 101, + "severity": "minor", + "comment": "RISK COVERAGE: The test for preferredVersion=v1beta1 with v1 available validates that the preferred version overrides the availability scan for its own family. This is a critical behavioral test -- it proves that backend operators can force a downgrade to beta. However, the complementary case where preferredVersion is undefined (discovery response missing the field entirely) is not tested, leaving the optional chaining fallback path uncovered." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 119, + "severity": "minor", + "comment": "CROSS-FAMILY FALLBACK: This test verifies that when preferred='v1', v2 still resolves via availability scanning. This is important for the flow because it proves the two version families are independently resolved. The plan identifies discover() as a leaf node called by resolve(), and this test exercises the leaf's branching logic correctly. Good coverage of the cross-family independence." + }, + { + "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", + "line": 51, + "severity": "positive", + "comment": "The restructured test suite with a dedicated 'preferredVersion-based resolution' describe block clearly separates the new behavior from the existing tests. The parameterized tests cover both-stable, beta-only, v1-stable-only, and v2-stable-only scenarios, maintaining backward compatibility while adding the new preferredVersion dimension." + } + ], + "summary": "The PR introduces a well-structured shift from hardcoded version selection to backend-driven preferredVersion resolution, fixing a real race condition. The main risks are the hardcoded version string literals in the resolution logic (not forward-compatible) and the unnecessary discovery call on the home dashboard cold-start path." + }, + "review_plan": { + "stats": {"totalSteps": 34, "totalAdditions": 24, "totalDeletions": 8, "independentFlows": 3, "filesChanged": 3}, + "steps": [ + {"order": 1, "nodeId": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts::mockDiscoveryResponse", "name": "mockDiscoveryResponse", "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.test.ts", "lines": [14, 24], "type": "function", "changeType": "modified", "additions": 3, "deletions": 2, "role": "entry_point", "risk": "high"}, + {"order": 2, "nodeId": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts::DashboardAPIVersionResolver", "name": "DashboardAPIVersionResolver", "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts", "lines": [19, 101], "type": "class", "changeType": "modified", "additions": 9, "deletions": 3, "role": "entry_point", "risk": "high"}, + {"order": 9, "nodeId": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts::DashboardAPIVersionResolver.discover", "name": "discover", "file": "public/app/features/dashboard/api/DashboardAPIVersionResolver.ts", "lines": [71, 89], "type": "method", "changeType": "modified", "additions": 9, "deletions": 3, "role": "leaf", "risk": "low"}, + {"order": 10, "nodeId": "public/app/features/dashboard-scene/pages/DashboardScenePageStateManager.ts::loadScene", "name": "loadScene", "file": "public/app/features/dashboard-scene/pages/DashboardScenePageStateManager.ts", "lines": [447, 470], "type": "method", "changeType": "modified", "additions": 3, "deletions": 0, "role": "internal", "risk": "low"} + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + }, + "reasoning": "Both reviews correctly identify the main issues: the resolve() placement before the home dashboard path and the lack of a test for undefined preferredVersion. The flow-guided review adds significant value through its understanding of the dependency chain (discover -> resolve -> getV1/getV2 -> loadScene), which leads to the forward-compatibility concern about hardcoded version strings -- a risk the baseline review misses entirely. The flow-guided review also better contextualizes the test coverage gaps by mapping them to the specific code paths they exercise (leaf node branching, cross-family independence). The baseline review's comments are mostly surface-level observations about code style and documentation, while the flow-guided review connects each comment to the architectural flow and explains why each finding matters for the system's behavior. The flow-guided review earns higher marks on flow_awareness and risk_identification for identifying the string literal coupling and the cold-start latency concern with architectural context.", + "winner": "flow_guided" + } +} diff --git a/evals/hashicorp__terraform__38301.json b/evals/hashicorp__terraform__38301.json new file mode 100644 index 0000000..131c7ad --- /dev/null +++ b/evals/hashicorp__terraform__38301.json @@ -0,0 +1,102 @@ +{ + "pr": "hashicorp/terraform#38301", + "title": "[Stacks Actions] Ensure action invocations are passed into module runtime", + "baseline_review": { + "comments": [ + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 335, + "severity": "major", + "comment": "The deferred reason conversion discards the error with `deferredReason, _ := planfile.DeferredReasonFromProto(msg.Deferred.Reason)`. If an unrecognized or invalid deferred reason is encountered, this silently uses a zero-value reason instead of surfacing the problem. The planned and partial resource instance handlers in this same file return errors for similar conversion failures. This should return the error to maintain consistency and avoid silent data corruption." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 320, + "severity": "minor", + "comment": "In the `PlanDeferredActionInvocation` case, `msg.Invocation` is validated as non-nil but immediately after `LoadComponentForActionInvocation` and `ValidateActionInvocation` are called, both of which accept `msg.Invocation` -- `ValidateActionInvocation` checks `change.Invocation == nil` and returns `(nil, nil)` for that case. The nil check here is redundant with the one inside `ValidateActionInvocation`, but the early return is good practice for clarity. Consider making `ValidateActionInvocation` return an error for a nil invocation instead to avoid the confusing `(nil, nil)` return." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 509, + "severity": "minor", + "comment": "`ValidateActionInvocation` returns `(nil, nil)` when `change.Invocation` is nil. This is a surprising contract -- callers must remember to check for a nil result even without an error. In the `PlanActionInvocationPlanned` handler this is handled with `if action != nil`, but in the `PlanDeferredActionInvocation` handler the result is used unconditionally to construct a `DeferredActionInvocationSrc`, which would embed a nil `ActionInvocationInstanceSrc`. Consider returning an explicit error for nil invocation instead." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 125, + "severity": "minor", + "comment": "The nil checks `if action != nil` and `if deferredAction != nil` inside `ForModulesRuntime` suggest that the slices may contain nil entries. Since the loader initializes these as empty slices and only appends non-nil validated results, nil entries should not occur unless there is a bug elsewhere. If nils are truly unexpected, a defensive log or error would be more appropriate than silent filtering." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 528, + "severity": "minor", + "comment": "The `LoadComponentForActionInvocation` function is exported but appears to be an internal helper used only within this package's loader. If it is not intended to be part of the public API, consider making it unexported (`loadComponentForActionInvocation`) to reduce the package surface. Same applies to `ValidateActionInvocation`." + } + ], + "summary": "The PR correctly wires action invocations through the stack plan loader into the modules runtime, following the existing patterns for resource instance changes. The most notable issue is the silently discarded error from `DeferredReasonFromProto`, which could mask invalid plan data, and the `ValidateActionInvocation` function has a `(nil, nil)` return path that could lead to nil being embedded in a deferred action invocation struct." + }, + "flow_guided_review": { + "comments": [ + { + "file": "internal/stacks/stackplan/component.go", + "line": 55, + "severity": "minor", + "comment": "The new `ActionInvocations` and `DeferredActionInvocations` fields on `Component` mirror the existing `ResourceInstanceChanges` / `DeferredResourceInstanceChanges` pattern. However, the resource instance fields use `addrs.Map` for deduplication-by-address while the action fields use plain slices. If action invocations can be duplicated (e.g., same action address appearing twice due to a loader bug), the slice would silently accept duplicates. Confirm that upstream protobuf messages guarantee uniqueness, or consider using a map or adding dedup validation." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 304, + "severity": "minor", + "comment": "The `PlanActionInvocationPlanned` handler calls `LoadComponentForActionInvocation` then `ValidateActionInvocation`, and only appends if non-nil. This follows the same two-step pattern as `LoadComponentForPartialResourceInstance` + change validation. The flow is correct, but note that `ValidateActionInvocation` checks address consistency (`action.Addr.Equal(fullAddr.Item)`) -- if the protobuf address fields disagree, the error message 'planned action invocation has inconsistent address to its containing object' lacks the actual addresses, making debugging harder. Include both addresses in the error string." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 335, + "severity": "major", + "comment": "In the `PlanDeferredActionInvocation` case, `planfile.DeferredReasonFromProto(msg.Deferred.Reason)` returns an error that is discarded via `_`. This is the only place in the entire `AddRaw` method where a conversion error is ignored -- all other proto-to-domain conversions (`ActionInvocationFromProto`, address parsing, timestamp unmarshaling) propagate errors. A malformed deferred reason would silently produce a zero-value `DeferredReason`, corrupting the plan. This should return the error." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 337, + "severity": "medium", + "comment": "After `ValidateActionInvocation` returns `(nil, nil)` for a nil invocation, the deferred handler unconditionally constructs `DeferredActionInvocationSrc{ActionInvocationInstanceSrc: action}` where `action` could be nil. The `RequiredProviderInstances` method in component.go does guard against `deferredAction.ActionInvocationInstanceSrc == nil`, but `ForModulesRuntime` does not -- it would append a `DeferredActionInvocationSrc` with a nil inner struct to the plan. This is partially mitigated by the earlier `msg.Invocation != nil` check, but the defense-in-depth is incomplete." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 183, + "severity": "minor", + "comment": "In `RequiredProviderInstances`, the deferred action loop checks `deferredAction == nil || deferredAction.ActionInvocationInstanceSrc == nil` before accessing provider addresses. The planned action loop only checks `action == nil`. This asymmetry is correct given the struct nesting difference but would benefit from a brief comment explaining why the deferred case needs the extra nil check." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 125, + "severity": "minor", + "comment": "In `ForModulesRuntime`, planned actions are appended to `changes.ActionInvocations` while deferred actions are appended to `plan.DeferredActionInvocations`. This mirrors the resource instance pattern where changes go into the `Changes` struct and deferrals go directly on the `Plan`. The separation is correct, but `ForModulesRuntime` does not check `deferredAction.ActionInvocationInstanceSrc` for nil before appending, unlike `RequiredProviderInstances` which does. If a nil inner struct reaches here, downstream code consuming the plan could panic." + } + ], + "summary": "The PR correctly wires planned and deferred action invocations from the stack plan protobuf loader through the Component struct into the modules runtime plan, following established resource instance patterns. The primary risk is a silently discarded error from `DeferredReasonFromProto` and incomplete nil-safety in the deferred action path where `ValidateActionInvocation` can return nil without error, leading to a `DeferredActionInvocationSrc` with a nil inner struct that is not guarded in `ForModulesRuntime`." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the silently discarded error from DeferredReasonFromProto as the top issue. However, the flow-guided review traces the data flow from protobuf loading through Component fields to ForModulesRuntime output, which reveals the nil-safety gap: ValidateActionInvocation can return (nil, nil), the deferred handler wraps this nil into a DeferredActionInvocationSrc, and ForModulesRuntime does not guard against the nil inner struct even though RequiredProviderInstances does. This cross-method inconsistency is only visible when following the data through the full pipeline. The baseline review notes the (nil, nil) return issue but does not trace it to the concrete downstream consequence in ForModulesRuntime. The flow-guided review also correctly identifies the structural difference between using addrs.Map for resources vs plain slices for actions and questions whether uniqueness is guaranteed upstream." + } +} \ No newline at end of file diff --git a/evals/hashicorp__terraform__38313.json b/evals/hashicorp__terraform__38313.json new file mode 100644 index 0000000..2154118 --- /dev/null +++ b/evals/hashicorp__terraform__38313.json @@ -0,0 +1,108 @@ +{ + "pr": "hashicorp/terraform#38313", + "title": "[Stacks Actions] Restore apply behavior", + "baseline_review": { + "comments": [ + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 352, + "severity": "medium", + "comment": "The `deferredReason` assignment silently discards the second return value (error) from `planfile.DeferredReasonFromProto(msg.Deferred.Reason)` via `_`. If the proto contains an unrecognized or invalid deferred reason, this will silently use whatever zero/default value the function returns, potentially producing a plan with an incorrect deferral reason. This should check and return the error." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 509, + "severity": "medium", + "comment": "`ValidateActionInvocation` returns `(nil, nil)` when `change.Invocation` is nil -- no error, no action. The caller at the `PlanActionInvocationPlanned` case checks `if action != nil` before appending, so a nil invocation is silently skipped. However, receiving a `PlanActionInvocationPlanned` message with a nil `Invocation` field is arguably malformed input and should return an error rather than being silently ignored." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 123, + "severity": "minor", + "comment": "The nil-check `if action != nil` inside the loop over `c.ActionInvocations` is defensive, but the only code path that populates this slice (in `from_proto.go`) already skips nil actions. This creates ambiguity about whether nil entries are expected. If they are not expected, consider removing the nil guard and instead ensuring the invariant at insertion time; if they are expected, document why." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 163, + "severity": "minor", + "comment": "In `RequiredProviderInstances`, the loops for `ActionInvocations` and `DeferredActionInvocations` duplicate the pattern of extracting provider addresses. The deferred loop has an extra nil check on `deferredAction.ActionInvocationInstanceSrc` which the planned loop does not need, but the structure is otherwise identical. Consider extracting a small helper to reduce repetition and ensure consistent nil handling." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 529, + "severity": "minor", + "comment": "`LoadComponentForActionInvocation` is exported but its signature returns a bare `*Component` (an internal type) alongside a `stackaddrs.AbsActionInvocationInstance`. If this function is only intended for use within the `stackplan` package, it should be unexported to avoid expanding the public API surface unnecessarily." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 304, + "severity": "minor", + "comment": "The `PlanActionInvocationPlanned` and `PlanDeferredActionInvocation` cases follow slightly different validation patterns: the planned case delegates nil-invocation handling to `ValidateActionInvocation`, while the deferred case explicitly checks `msg.Deferred == nil` and `msg.Invocation == nil` before calling the same validator. This asymmetry makes it harder to reason about which invariants hold at each point. Aligning the validation strategy would improve maintainability." + } + ], + "summary": "This PR adds action invocation support to the stacks plan loader and component model, following the established patterns for resource instance changes. The main concerns are a silently discarded error from `DeferredReasonFromProto`, a potentially too-permissive nil-invocation handling in `ValidateActionInvocation`, and minor inconsistencies in validation patterns between planned and deferred action paths." + }, + "flow_guided_review": { + "comments": [ + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 352, + "severity": "medium", + "comment": "The error from `planfile.DeferredReasonFromProto(msg.Deferred.Reason)` is discarded with `_`. This sits on the deferred-action loading path, meaning a corrupted or unrecognized deferral reason in a saved plan would silently produce a plan with a zero-value reason. Since this value flows through to `DeferredActionInvocationSrc` and eventually into the modules runtime apply plan, a wrong reason could cause incorrect apply-time behavior. The error should be checked and propagated." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 509, + "severity": "medium", + "comment": "`ValidateActionInvocation` returns `(nil, nil)` for a nil `Invocation` field. This is consumed by both the planned and deferred cases. In the planned case, a nil result is silently skipped. In the deferred case, a nil action flows into `DeferredActionInvocationSrc{ActionInvocationInstanceSrc: nil}`, which would then require the downstream nil check in `RequiredProviderInstances` (component.go line 175). A `PlanActionInvocationPlanned` message with a nil invocation payload is malformed and should be rejected with an error rather than silently tolerated." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 175, + "severity": "medium", + "comment": "The nil check `deferredAction.ActionInvocationInstanceSrc == nil` in `RequiredProviderInstances` is the only thing preventing a nil-pointer dereference when accessing the provider address. This guard exists because `ValidateActionInvocation` can return nil without error, allowing a `DeferredActionInvocationSrc` with a nil inner source to be appended in `from_proto.go`. This is a data-integrity concern: if the nil is caught upstream (by erroring on nil invocations in `ValidateActionInvocation`), this guard becomes unnecessary and the code is safer overall." + }, + { + "file": "internal/stacks/stackplan/component.go", + "line": 123, + "severity": "minor", + "comment": "The `ForModulesRuntime` method appends action invocations to `changes.ActionInvocations` and deferred actions to `plan.DeferredActionInvocations`. This split between the `changes` and `plan` targets mirrors how resource instance changes vs deferred resource changes are handled, which is correct. However, the nil guards here are purely defensive since the loader path should not insert nils -- consider an assertion or removing them to keep the contract explicit." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 304, + "severity": "minor", + "comment": "The `PlanDeferredActionInvocation` case validates `msg.Deferred` and `msg.Invocation` for nil before calling `LoadComponentForActionInvocation` and `ValidateActionInvocation`, but the `PlanActionInvocationPlanned` case does not validate `msg.Invocation` for nil -- it relies on `ValidateActionInvocation` to handle it by returning (nil, nil). This asymmetry means the deferred path fails fast with a clear error message while the planned path silently drops the message. Both paths should fail with explicit errors for nil invocation payloads." + }, + { + "file": "internal/stacks/stackplan/from_proto.go", + "line": 529, + "severity": "minor", + "comment": "`LoadComponentForActionInvocation` and `ValidateActionInvocation` are both exported functions that serve as the shared logic between the planned and deferred action loading paths. Since `LoadComponentForActionInvocation` takes the full `*Plan` and returns an internal `*Component`, exporting these creates a public API contract that may be premature. If these are only used within `stackplan`, unexport them." + } + ], + "summary": "The PR wires action invocations through the loader-to-component-to-modules-runtime path, closely mirroring the existing resource instance change pattern. The most significant concern is the discarded error from `DeferredReasonFromProto` on the deferred action path, which could silently corrupt apply-time behavior; the second concern is that `ValidateActionInvocation` tolerates nil invocations instead of treating them as malformed input, creating a chain of defensive nil checks downstream in `RequiredProviderInstances`." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 6, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.8 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identified the same core issues: the discarded error from DeferredReasonFromProto, the permissive nil handling in ValidateActionInvocation, and the asymmetric validation patterns. The flow-guided review edges ahead by tracing the data flow more carefully -- specifically, it connects the nil-return from ValidateActionInvocation through the deferred path into the DeferredActionInvocationSrc struct and then into the RequiredProviderInstances nil guard, showing how a single upstream leniency creates a chain of defensive checks downstream. This flow-tracing gives its risk identification more depth. The review plan was empty (zero steps, zero dependencies), which severely limited the flow-guided advantage; with a populated plan, the margin would likely be wider. Both reviews are constrained by the truncated diff (LoadComponentForActionInvocation is cut off), preventing full analysis of address resolution logic." + } +} diff --git a/evals/honojs__hono__4797.json b/evals/honojs__hono__4797.json new file mode 100644 index 0000000..88d1ecc --- /dev/null +++ b/evals/honojs__hono__4797.json @@ -0,0 +1,147 @@ +{ + "pr": { + "url": "https://github.com/honojs/hono/pull/4797", + "owner": "honojs", + "repo": "hono", + "number": 4797, + "title": "chore(builld): tsconfig project references", + "files_changed": 24, + "additions": 217, + "deletions": 217, + "language": "TypeScript" + }, + "timestamp": "2026-03-30T20:45:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "runtime-tests/bun/index.test.tsx", + "line": 284, + "severity": "major", + "comment": "The change from `.toBeTruly` to `.toBeTruthy()` is a real bug fix, not just a type fix. The original code was a bare property access that never actually invoked the assertion matcher, meaning the test was silently passing regardless of the value of `result.success`." + }, + { + "file": ".github/actions/perf-measures/action.yml", + "line": 17, + "severity": "minor", + "comment": "Adding `bun tsc --build` to the perf-measures CI action setup step introduces a build step before the performance measurement. If this build output is cached or alters the environment, it could affect the accuracy of the subsequent type-check performance measurement." + }, + { + "file": "runtime-tests/bun/index.test.tsx", + "line": 209, + "severity": "minor", + "comment": "Changing the Layout prop type from `{ children?: string }` to `PropsWithChildren` is more correct since JSX children are not limited to strings. This is a good type improvement." + }, + { + "file": "runtime-tests/bun/index.test.tsx", + "line": 331, + "severity": "nit", + "comment": "Adding the `dirPath: string` type annotation to the `deleteDirectory` parameter is a straightforward TypeScript strictness fix. Good cleanup." + }, + { + "file": "perf-measures/tsconfig.json", + "line": 1, + "severity": "minor", + "comment": "The old `perf-measures/tsconfig.json` is deleted and replaced with `perf-measures/type-check/scripts/tsconfig.json` and a modified `perf-measures/type-check/tsconfig.build.json`. Ensure all scripts that referenced the old path have been updated." + }, + { + "file": "package.json", + "line": 664, + "severity": "nit", + "comment": "Adding `@types/ws` as a devDependency is needed to resolve type errors in the bun WebSocket tests. This is a reasonable addition." + } + ], + "summary": "This PR adds TypeScript project references to runtime test directories, fixes several type errors surfaced by stricter checking, and incidentally fixes a real test bug where `.toBeTruly` was a no-op property access instead of the `.toBeTruthy()` assertion call. The changes are predominantly configuration (tsconfig files) with targeted type annotation fixes in test and mock files." + }, + "flow_guided_review": { + "comments": [ + { + "file": "runtime-tests/lambda/mock.ts", + "line": 16, + "severity": "minor", + "comment": "The plan identifies `write` (step 6) as a leaf node called by `streamToNodeStream` and `streamHandle` in handler.ts. The type annotation change for the `chunk` parameter to `Buffer` is correct and aligns the mock with the actual Node.js Writable stream interface that the production code expects." + }, + { + "file": "runtime-tests/lambda/mock.ts", + "line": 20, + "severity": "minor", + "comment": "The plan identifies `final` (step 2) as an entry point. Adding the callback parameter type annotation ensures the mock properly matches the Writable stream contract. The callback invocation `callback()` is necessary to signal stream completion." + }, + { + "file": "runtime-tests/bun/index.test.tsx", + "line": 284, + "severity": "major", + "comment": "While the plan flagged `deleteDirectory` (step 3) as the entry point in this file, the most significant change is nearby: `.toBeTruly` changed to `.toBeTruthy()`. The original was a property access that never executed the assertion -- a latent test bug meaning the toSSG success assertion was never validated." + }, + { + "file": "runtime-tests/lambda/stream-mock.ts", + "line": 17, + "severity": "minor", + "comment": "The plan identifies this as a separate entry point (step 4) from mock.ts. The `Writable` type parameter addition and `Buffer` annotation for the write chunk parameter are consistent with the parallel changes in mock.ts, ensuring both mock files have coherent type contracts." + }, + { + "file": "src/adapter/aws-lambda/handler.ts", + "line": 456, + "severity": "minor", + "comment": "The plan identifies `getCookies` (step 5) as a high-risk entry point. The method was simplified from a multi-line implementation with intermediate variables to a direct return of `event.headers?.cookie ?? ''`. This is production code -- verify the behavior is identical, particularly around undefined vs empty string handling." + }, + { + "file": "perf-measures/type-check/tsconfig.build.json", + "line": 8, + "severity": "minor", + "comment": "The new tsconfig.build.json now uses project references (`\"references\": [{ \"path\": \"../../tsconfig.build.json\" }]`) which is the core architectural change in this PR. This enables incremental builds and proper dependency tracking across the monorepo's type checking." + }, + { + "file": "runtime-tests/bun/index.test.tsx", + "line": 12, + "severity": "positive", + "comment": "The ContextRenderer module augmentation via `declare module` is the proper Hono pattern for extending the renderer type, replacing what was likely a ts-ignore or implicit any. This follows the framework's recommended approach." + } + ], + "summary": "Following the review plan's dependency graph, the lambda mock changes (mock.ts and stream-mock.ts) properly align type annotations with the production code in handler.ts that calls them, and the getCookies simplification in production code is a safe refactor. The plan's structured traversal across independent entry points helped organize review of this 24-file configuration change, and the most impactful finding remains the toBeTruthy() bug fix in the bun tests." + }, + "review_plan": { + "stats": { + "totalSteps": 10, + "totalAdditions": 9, + "totalDeletions": 18, + "independentFlows": 6, + "filesChanged": 4 + }, + "steps": [ + {"order": 1, "nodeId": "runtime-tests/lambda/mock.ts::mockStreamifyResponse", "name": "mockStreamifyResponse", "file": "runtime-tests/lambda/mock.ts", "lines": [13, 32], "type": "function", "changeType": "modified", "additions": 2, "deletions": 2, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 2, "nodeId": "runtime-tests/lambda/mock.ts::final", "name": "final", "file": "runtime-tests/lambda/mock.ts", "lines": [20, 23], "type": "method", "changeType": "modified", "additions": 1, "deletions": 1, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 3, "nodeId": "runtime-tests/bun/index.test.tsx::deleteDirectory", "name": "deleteDirectory", "file": "runtime-tests/bun/index.test.tsx", "lines": [331, 346], "type": "function", "changeType": "modified", "additions": 1, "deletions": 1, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 4, "nodeId": "runtime-tests/lambda/stream-mock.ts::mockStreamifyResponse", "name": "mockStreamifyResponse", "file": "runtime-tests/lambda/stream-mock.ts", "lines": [17, 32], "type": "function", "changeType": "modified", "additions": 2, "deletions": 1, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 5, "nodeId": "src/adapter/aws-lambda/handler.ts::EventV1Processor.getCookies", "name": "getCookies", "file": "src/adapter/aws-lambda/handler.ts", "lines": [456, 458], "type": "method", "changeType": "modified", "additions": 1, "deletions": 6, "role": "entry_point", "risk": "high", "calledBy": [], "calls": [], "riskReasons": ["entry_point"]}, + {"order": 6, "nodeId": "runtime-tests/lambda/mock.ts::write", "name": "write", "file": "runtime-tests/lambda/mock.ts", "lines": [16, 19], "type": "method", "changeType": "modified", "additions": 1, "deletions": 1, "role": "leaf", "risk": "medium", "calledBy": ["src/adapter/aws-lambda/handler.ts::streamToNodeStream", "src/adapter/aws-lambda/handler.ts::streamHandle"], "calls": [], "riskReasons": ["multiple_callers"]}, + {"order": 7, "nodeId": "src/adapter/aws-lambda/handler.ts::EventV1Processor", "name": "EventV1Processor", "file": "src/adapter/aws-lambda/handler.ts", "lines": [429, 492], "type": "class", "changeType": "modified", "additions": 1, "deletions": 6, "role": "leaf", "risk": "low", "calledBy": ["runtime-tests/lambda/index.test.ts::read"], "calls": [], "riskReasons": []} + ], + "clusters": [ + {"id": 0, "label": "handler.ts", "nodeIds": ["src/adapter/aws-lambda/handler.ts::EventV1Processor", "src/adapter/aws-lambda/handler.ts::streamToNodeStream", "src/adapter/aws-lambda/handler.ts::streamHandle"], "reason": "3 related functions in handler.ts", "suggestedReviewOrder": ["src/adapter/aws-lambda/handler.ts::EventV1Processor", "src/adapter/aws-lambda/handler.ts::streamToNodeStream", "src/adapter/aws-lambda/handler.ts::streamHandle"]} + ], + "dependencies": [ + {"from": "src/adapter/aws-lambda/handler.ts::streamToNodeStream", "to": "runtime-tests/lambda/mock.ts::write", "reason": "Review `streamToNodeStream` before `write` -- `streamToNodeStream` calls `write`."}, + {"from": "src/adapter/aws-lambda/handler.ts::streamHandle", "to": "runtime-tests/lambda/mock.ts::write", "reason": "Review `streamHandle` before `write` -- `streamHandle` calls `write`."} + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + }, + "reasoning": "The flow-guided review leveraged the dependency graph to connect the lambda mock type changes (mock.ts write/final) back to their callers in handler.ts, providing context the baseline review missed entirely. Both reviews caught the critical toBeTruthy() bug fix. The flow-guided review better identified the risk in the getCookies production code simplification by noting it was flagged as a high-risk entry point, prompting verification of undefined vs empty string semantics. The baseline review treated each file change in isolation, missing the coherence between mock.ts and stream-mock.ts changes that the plan's entry-point grouping made obvious. The flow-guided review also better contextualized the tsconfig architectural change through the project references pattern.", + "winner": "flow_guided" + } +} diff --git a/evals/jestjs__jest__15929.json b/evals/jestjs__jest__15929.json new file mode 100644 index 0000000..62129d0 --- /dev/null +++ b/evals/jestjs__jest__15929.json @@ -0,0 +1,96 @@ +{ + "pr": "jestjs/jest#15929", + "title": "chore: update TSTyche to v5", + "baseline_review": { + "comments": [ + { + "file": "package.json", + "line": 81, + "severity": "minor", + "comment": "The version bump from `^4.0.0` to `^5.0.0` is a major version upgrade. While the PR description mentions the matcher rename, major versions can include other breaking changes. It would be worth verifying the full TSTyche v5 changelog to ensure no other breaking changes affect Jest's usage (e.g., config file format, CLI flags, or other deprecated APIs)." + }, + { + "file": "packages/expect/__typetests__/expect.test.ts", + "line": 165, + "severity": "nit", + "comment": "The rename from `toBeAssignableWith` to `toBeAssignableFrom` is applied consistently across all six occurrences in this file. The new name better communicates directionality of the type assignability check." + }, + { + "file": "packages/jest-mock/__typetests__/Mocked.test.ts", + "line": 115, + "severity": "nit", + "comment": "All seven occurrences of `toBeAssignableWith` in this file are renamed to `toBeAssignableFrom`. The changes are mechanically consistent with the matcher rename in TSTyche v5." + }, + { + "file": ".github/workflows/nodejs.yml", + "line": 51, + "severity": "minor", + "comment": "The documentation URL changed from `/guide/typescript-versions` to `/guides/typescript-versions` (plural). This is a non-functional change but worth confirming the new URL is valid -- a broken comment link could mislead future contributors trying to understand the `--target` flag." + }, + { + "file": "packages/jest-mock/__typetests__/utility-types.test.ts", + "line": 111, + "severity": "nit", + "comment": "The diff appears truncated for this file, showing only 2 of what are likely more matcher renames. A reviewer should verify all occurrences of `toBeAssignableWith` were updated in the full file to avoid partial migration." + } + ], + "summary": "This is a straightforward dependency upgrade PR that bumps TSTyche from v4 to v5 and applies the required API migration: renaming `toBeAssignableWith` to `toBeAssignableFrom` across all type test files. The risk is low since the changes are purely in dev tooling (type tests and CI config), but a reviewer should verify the full TSTyche v5 changelog for any additional breaking changes beyond the matcher rename." + }, + "flow_guided_review": { + "comments": [ + { + "file": "package.json", + "line": 81, + "severity": "minor", + "comment": "The review plan is empty (0 steps, 0 clusters, 0 dependencies), which is expected for a dev dependency upgrade with no production code changes. The version bump from `^4.0.0` to `^5.0.0` is the root cause of all other changes in this PR. Since TSTyche is a devDependency used only for type testing, there is no runtime risk, but the full v5 migration guide should be checked for any config or behavioral changes beyond the matcher rename." + }, + { + "file": "packages/expect/__typetests__/expect.test.ts", + "line": 165, + "severity": "nit", + "comment": "With no flow graph to trace, the matcher renames in this file are a leaf-level mechanical change driven entirely by the TSTyche v5 API. All six `toBeAssignableWith` -> `toBeAssignableFrom` replacements are consistent. Since these type tests validate Jest's public TypeScript API surface, the tests themselves serve as the verification that the upgrade didn't break type compatibility." + }, + { + "file": ".github/workflows/nodejs.yml", + "line": 51, + "severity": "minor", + "comment": "The URL path change from `/guide/` to `/guides/` reflects TSTyche's documentation restructuring in v5. While this is just a comment, it's the only reference to external TSTyche docs in the CI workflow. Confirming the URL resolves correctly ensures the comment remains useful for future maintainers investigating the `--target` flag behavior." + }, + { + "file": "packages/jest-mock/__typetests__/Mocked.test.ts", + "line": 115, + "severity": "nit", + "comment": "The seven matcher renames here cover Mocked type tests for classes, functions, async functions, function objects, plain objects, and console. This is the most comprehensive type test file in the PR and the renames appear complete. No additional TSTyche v5 API changes are needed in these assertions." + }, + { + "file": "packages/jest-mock/__typetests__/utility-types.test.ts", + "line": 111, + "severity": "minor", + "comment": "The diff is truncated for this file, showing only the beginning of the changes. Since the empty flow plan provides no dependency information to cross-reference, completeness verification must rely on a global search for any remaining `toBeAssignableWith` references across the codebase to confirm the migration is exhaustive." + } + ], + "summary": "With an empty flow graph, the flow-guided review converges with the baseline analysis: this is a mechanical dev dependency upgrade with no production code paths affected. The PR correctly migrates the renamed TSTyche v5 matcher across all type test files, and CI (green) serves as the primary verification that the upgrade is complete and correct." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 8, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + } + }, + "winner": "tie", + "reasoning": "For this mechanical dev dependency upgrade PR, both review approaches produce equivalent results. The flow-guided plan is empty (0 steps, 0 clusters, 0 dependencies) because there are no production code changes, execution flows, or cross-file dependencies to trace. The baseline review correctly identifies the key concerns: verifying full TSTyche v5 changelog coverage, confirming the documentation URL change, and checking for completeness of the matcher rename migration. The flow-guided review restates these same observations with plan-awareness framing but cannot add incremental value because there are no flows to guide. Both reviews appropriately identify that the truncated diff warrants verifying all files were updated. Neither approach differentiates meaningfully on a devDependency upgrade with purely mechanical test file changes." + } +} diff --git a/evals/kubernetes__kubernetes__138024.json b/evals/kubernetes__kubernetes__138024.json new file mode 100644 index 0000000..096c158 --- /dev/null +++ b/evals/kubernetes__kubernetes__138024.json @@ -0,0 +1,114 @@ +{ + "pr": "kubernetes/kubernetes#138024", + "title": "Fix race condition in updating the PodStatus cache", + "baseline_review": { + "comments": [ + { + "file": "pkg/kubelet/container/runtime.go", + "line": 224, + "severity": "medium", + "comment": "Adding a `Timestamp` field to the `Pod` struct is a significant API change that affects all consumers of this type. Since `time.Time` has a zero value that is valid but meaningless, any code path that constructs a `Pod` without setting `Timestamp` will silently have a zero timestamp, which could lead to cache staleness bugs. Consider whether this should be a pointer (`*time.Time`) to make unset timestamps explicit, or audit all `Pod` construction sites to ensure `Timestamp` is always populated." + }, + { + "file": "pkg/kubelet/kubelet.go", + "line": 2346, + "severity": "low", + "comment": "In `SyncTerminatingPod`, the `Timestamp` is set to `kl.clock.Now()` at the point the empty pod is constructed for cache update. This is correct for the termination path since there is no runtime listing involved, but it would be worth adding a brief comment explaining why `clock.Now()` is appropriate here (the pod is being synthesized, not read from the runtime)." + }, + { + "file": "pkg/kubelet/kuberuntime/fake_kuberuntime_manager.go", + "line": 100, + "severity": "low", + "comment": "Replacing `NewContainerLogManager` with `NewStubContainerLogManager` in the fake runtime manager is a good cleanup that simplifies test setup. However, this changes the behavior of all tests using `newFakeKubeRuntimeManager` -- the stub presumably no-ops on log rotation. Tests that previously relied on real log manager behavior (like `TestRemoveContainer`) now need to explicitly swap in the real log manager, which is done in the test file but adds boilerplate." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_container_test.go", + "line": 57, + "severity": "medium", + "comment": "The pattern of creating the fake runtime manager then immediately swapping in a real `ContainerLogManager` is repeated in both `TestRemoveContainer` and `TestRemoveContainer_keepLogs`. This is a test smell -- consider extracting a helper like `createTestRuntimeManagerWithLogManager` to avoid the duplicated setup and reduce the chance of future tests forgetting this swap when they need real log behavior." + }, + { + "file": "pkg/kubelet/kuberuntime/helpers.go", + "line": 39, + "severity": "low", + "comment": "Removing the `podsByID` sort type is dead code cleanup. Verify that no other files reference this type -- if it was only used internally and is now unused, this is a clean removal." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_container_test.go", + "line": 77, + "severity": "low", + "comment": "The signature change from `makeAndSetFakePod(t, ...)` to `makeAndSetFakePod(tCtx, ...)` suggests the helper now accepts a context instead of a `*testing.T`. This is consistent across the two test files changed, but ensure all other callers of `makeAndSetFakePod` in the codebase have been updated as well." + } + ], + "summary": "This PR correctly addresses a race condition in PLEG relist by moving the timestamp capture before sandbox listing and plumbing it through the Pod struct. The implementation is sound but the diff is incomplete (truncated), the new Timestamp field on Pod could benefit from explicit unset handling, and the test refactoring introduces duplicated log manager setup that should be extracted into a helper." + }, + "flow_guided_review": { + "comments": [ + { + "file": "pkg/kubelet/container/runtime.go", + "line": 224, + "severity": "high", + "comment": "The core data model change: adding `Timestamp time.Time` to the `Pod` struct is the foundation of the race condition fix. The timestamp now represents when the Pod was read from the runtime, which will be set before sandbox listing begins. This is critical because previously the timestamp was grabbed after listing (step 2 in the old flow), creating a window where new sandboxes could appear between list and timestamp. However, since the diff is truncated and does not show the actual relist code changes, it is impossible to verify that the timestamp is correctly captured before listing and correctly propagated through the relist flow." + }, + { + "file": "pkg/kubelet/container/runtime.go", + "line": 224, + "severity": "medium", + "comment": "The `Timestamp` field uses `time.Time` which has a zero value of `0001-01-01`. Any code that checks cache freshness by comparing timestamps could be misled by a zero timestamp -- it would appear extremely old and could trigger unnecessary cache refreshes. Since this field is meant to indicate freshness, consider adding validation or documentation about the expected behavior when Timestamp is zero (e.g., treat as 'unknown freshness, always refresh')." + }, + { + "file": "pkg/kubelet/kubelet.go", + "line": 2346, + "severity": "medium", + "comment": "In the terminating pod path, the timestamp is set to `kl.clock.Now()` when constructing a synthetic empty Pod for cache update. This is the correct approach for termination since there is no runtime listing involved -- the pod is being removed, not discovered. The use of `kl.clock` (injected clock) ensures testability. This path is independent from the main relist race fix but ensures the new Timestamp field is populated consistently across all Pod construction sites." + }, + { + "file": "pkg/kubelet/kuberuntime/fake_kuberuntime_manager.go", + "line": 100, + "severity": "low", + "comment": "Switching the fake runtime manager to use `NewStubContainerLogManager` is a test infrastructure improvement bundled with the race fix. This decouples most tests from real log rotation behavior, which reduces test fragility. The change is safe as long as tests that specifically need log rotation behavior (like TestRemoveContainer) explicitly opt in, which the companion test changes handle." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_container_test.go", + "line": 57, + "severity": "medium", + "comment": "The duplicated pattern of creating a fake runtime manager then swapping in a real ContainerLogManager appears in both TestRemoveContainer and TestRemoveContainer_keepLogs. Since these are the only tests that need real log rotation behavior, this is acceptable but fragile -- a new test author might not realize they need this swap. Extract a helper or add a prominent comment in the fake manager explaining which tests need the real log manager and why." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_container_linux_test.go", + "line": 2090, + "severity": "low", + "comment": "The change from `makeAndSetFakePod(t, ...)` to `makeAndSetFakePod(tCtx, ...)` indicates the helper signature changed to accept a context, likely so it can pass the timestamp or use context-aware operations. This is consistent with the broader pattern of plumbing timing information through the Pod construction path." + }, + { + "file": "pkg/kubelet/kuberuntime/helpers.go", + "line": 39, + "severity": "low", + "comment": "Removing the unused `podsByID` sort implementation is clean dead code removal. Since the relist logic is being restructured to capture timestamps differently, sorting pods by ID is presumably no longer needed in the new flow." + } + ], + "summary": "The PR's core fix -- moving timestamp capture before sandbox listing and embedding it in the Pod struct -- is architecturally sound and directly addresses the race condition where new sandboxes could appear between listing and timestamping. However, the diff appears truncated and does not include the critical relist code changes where the timestamp is actually captured and plumbed, making it impossible to fully verify the fix's correctness from this diff alone." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 6, + "risk_identification": 7, + "actionability": 6, + "efficiency": 6, + "overall": 6.4 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews are handicapped by a truncated diff that omits the most critical changes (the actual relist code where timestamps are captured before listing). The flow-guided review edges ahead by explicitly calling out that the diff is incomplete and that the core race fix cannot be verified from what is shown, which is a crucial observation. It also better contextualizes the Timestamp field addition as the foundation of the fix and explains the before-vs-after listing timing semantics. The baseline review raises valid points about API design (pointer vs zero value) and test duplication but treats the changes more as isolated items rather than parts of a cohesive race condition fix. The empty review plan (all zeros) severely limits the flow-guided approach's advantage -- with a real plan identifying the relist code as the critical path, the flow-guided review could have been significantly more targeted. The margin is narrow because both reviews ultimately cannot assess the most important part of the change." + } +} diff --git a/evals/kubernetes__kubernetes__138049.json b/evals/kubernetes__kubernetes__138049.json new file mode 100644 index 0000000..0591232 --- /dev/null +++ b/evals/kubernetes__kubernetes__138049.json @@ -0,0 +1,113 @@ +{ + "pr": { + "url": "https://github.com/kubernetes/kubernetes/pull/138049", + "owner": "kubernetes", + "repo": "kubernetes", + "number": 138049, + "title": "Pod events fix", + "files_changed": 3, + "additions": 45, + "deletions": 10, + "language": "go" + }, + "timestamp": "2026-03-30T00:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager.go", + "line": 2221, + "severity": "major", + "comment": "Adding the `podStatus == nil` check to the same guard as `allocatedPod.Spec.Resources == nil` conflates two unrelated invariants. A nil podStatus indicates a fundamentally different condition (pod status not yet available) than missing resource specs. If podStatus is nil at this call site, it likely indicates a caller bug or unexpected state. Silently returning false hides this condition. Consider logging a warning or handling it separately so operators can detect the anomaly." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager.go", + "line": 2224, + "severity": "minor", + "comment": "The original code used `_, _ :=` (blank identifier for the found bool), meaning it proceeded to compare even when no actuated state existed -- comparing a nil actuatedPodResources against allocatedPodResources. The new check for `!found || actuatedPodResources == nil` correctly short-circuits this case. However, it is worth noting that `!found` alone should be sufficient since `GetPodLevelResources` returning `!found` with a non-nil value would be a bug in the state store. The `|| actuatedPodResources == nil` is defensive but adds redundancy." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager_test.go", + "line": 5281, + "severity": "minor", + "comment": "The test case 'plr mismatch during initial creation' sets up allocated pod-level resources (cpuReq: 200) with no actuated state and a non-running container, asserting no resize is in progress. This is a good regression test. However, it does not distinguish whether the fix works because of the `podStatus == nil` guard or the `!found` guard on actuated resources. Consider adding a second test case that has actuated state but nil podStatus to validate both branches independently." + }, + { + "file": "test/e2e/common/node/pod_level_resources_resize.go", + "line": 660, + "severity": "minor", + "comment": "The e2e test searches for events using `scheme.Scheme` which is the client-go scheme. This should work, but the test only checks for the absence of `ResizeCompleted` events. It does not verify positive behavior (e.g., that the pod actually reached Running state with the expected resources). A pod that fails to start would also have no ResizeCompleted event, causing the test to pass vacuously." + }, + { + "file": "test/e2e/common/node/pod_level_resources_resize.go", + "line": 653, + "severity": "nit", + "comment": "The container spec `containers := []podresize.ResizableContainerInfo{{Name: \"c1\"}}` does not specify any resource requests or limits on the container itself. It relies entirely on the pod-level resources. While this is valid for testing PLR, documenting this intent with a brief comment would improve readability for future maintainers." + } + ], + "summary": "This PR fixes a bug where `isPodLevelResourcesResizeInProgress` could incorrectly report a resize in progress during initial pod creation, causing spurious ResizeCompleted events. The fix is sound -- it adds nil guards for podStatus and checks whether actuated state actually exists before comparing resources -- but the two distinct failure modes (nil podStatus vs. missing actuated state) are not tested independently." + }, + "flow_guided_review": { + "comments": [ + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager.go", + "line": 2221, + "severity": "major", + "comment": "The root cause of the spurious ResizeCompleted event is that during initial pod creation, the actuated state store has no entry for the pod yet. The old code discarded the `found` boolean from `GetPodLevelResources` and proceeded to compare a nil actuatedPodResources against the allocated spec, which always showed a mismatch. The `!found` guard on line 2224 is the essential fix. However, the `podStatus == nil` check added here on line 2221 is not clearly motivated by the same root cause. If podStatus can be nil at this call site, that is a separate defensive concern and should be documented with a comment explaining when this occurs." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager.go", + "line": 2224, + "severity": "minor", + "comment": "This is the core fix: checking `!found` from `GetPodLevelResources` prevents comparing against a zero-value resource struct when the pod has never been actuated. The additional `|| actuatedPodResources == nil` guard handles an edge case where the state store returns found=true with a nil value, which would be a bug in the state store itself. This defensive style is reasonable for kubelet code but the distinction should be covered by a test that sets found=true with nil resources." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager_test.go", + "line": 5281, + "severity": "minor", + "comment": "This test validates the primary fix path: during initial creation, no actuated state exists, so resize should not be reported. However, the test name 'plr mismatch during initial creation' is slightly misleading -- there is no actual mismatch being tested since the fix now short-circuits before any comparison. A name like 'plr no resize during initial creation before actuation' would more accurately describe the scenario." + }, + { + "file": "test/e2e/common/node/pod_level_resources_resize.go", + "line": 660, + "severity": "minor", + "comment": "The e2e test validates the user-visible symptom (no spurious ResizeCompleted event on initial creation) which is the right level to test at. However, the test uses `Events.SearchWithContext` which returns events at a point in time. If the event is emitted after the check but before the pod fully stabilizes, this could be flaky. Consider adding a brief wait or polling period to ensure no late-arriving events, though the `createAndVerifyPodPLR` call likely provides sufficient stabilization." + }, + { + "file": "test/e2e/common/node/pod_level_resources_resize.go", + "line": 645, + "severity": "nit", + "comment": "The new imports `scheme` and `kubeletevents` are correctly scoped to what the new test function needs. The use of `kubeletevents.ResizeCompleted` as a constant rather than a hardcoded string is good practice and will catch compilation errors if the event reason is ever renamed." + }, + { + "file": "pkg/kubelet/kuberuntime/kuberuntime_manager.go", + "line": 2228, + "severity": "nit", + "comment": "The blank line between the new `!found` guard and the `allocatedPodResources` assignment improves readability by separating the early-return guards from the comparison logic. The overall flow of the function is now: validate inputs -> check actuated state exists -> compare resources, which is a clean structure." + } + ], + "summary": "The PR correctly fixes a bug where missing actuated state during initial pod creation caused `isPodLevelResourcesResizeInProgress` to false-positive, triggering spurious ResizeCompleted events. The core fix (checking the `found` return from `GetPodLevelResources`) is sound, and the e2e test validates the user-visible symptom. The `podStatus == nil` guard is a secondary defensive addition whose motivation is less clear from the diff alone." + }, + "review_plan": {"stats": {"totalSteps": 0, "totalAdditions": 0, "totalDeletions": 0, "independentFlows": 0, "filesChanged": 0}, "steps": [], "clusters": [], "dependencies": []}, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 7, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.0 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same core issues: the essential fix being the `!found` check on actuated state, the secondary nature of the `podStatus == nil` guard, and the potential for the e2e test to pass vacuously. The flow-guided review edges ahead on flow_awareness by tracing the root cause through the data flow (actuated state store -> comparison -> false positive -> spurious event) and clearly distinguishing the essential fix from the defensive addition. The baseline review treats both nil guards more symmetrically without distinguishing which is the primary fix. The flow-guided review also provides better actionability by suggesting a more accurate test name and explaining the clean early-return structure. The advantage is moderate since the review plan was empty (0 steps/clusters), leaving the flow-guided review to infer structure from the diff alone rather than leveraging pre-computed dependency information." + } +} \ No newline at end of file diff --git a/evals/langchain-ai__langchain__36347.json b/evals/langchain-ai__langchain__36347.json new file mode 100644 index 0000000..548dac8 --- /dev/null +++ b/evals/langchain-ai__langchain__36347.json @@ -0,0 +1,108 @@ +{ + "pr": "langchain-ai/langchain#36347", + "title": "fix(openrouter): pass attribution headers via httpx `default_headers`", + "baseline_review": { + "comments": [ + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 349, + "severity": "medium", + "comment": "When `extra_headers` is non-empty, httpx clients are created for both sync and async paths. However, when `extra_headers` is empty (no `app_url`, `app_title`, or `app_categories`), no custom httpx client is created. This means the `follow_redirects=True` option is only applied when attribution headers are present. If the SDK does not enable redirect-following by default, this creates inconsistent redirect behavior depending on whether the user sets attribution fields." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 389, + "severity": "minor", + "comment": "Moving `self.client = self._build_client()` inside the `try/except ImportError` block is a good defensive change, but `_build_client` itself may raise other exceptions (e.g., httpx import failures, SDK constructor errors). These non-ImportError exceptions will now propagate unwrapped, which is fine, but worth noting that only the `openrouter` import is guarded -- the httpx import inside `_build_client` would produce a separate, unhandled ImportError with a less helpful message." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 996, + "severity": "minor", + "comment": "The `warnings.warn` call for the `openrouter.components` import failure is a good improvement over silent fallback, but `stacklevel=2` may not point to the correct caller frame depending on how deeply `_wrap_messages_for_sdk` is called. Consider verifying the stacklevel produces a useful location in the warning output." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 1015, + "severity": "nit", + "comment": "The f-string in the warning message ends with a period inside the string but uses `stacklevel=2`. This is fine stylistically, but the two warning messages in this function use slightly different tones -- one says 'may cause validation errors' (speculative) while the other says 'passing raw dict to the API' (factual). Consider making both consistently describe the consequence." + }, + { + "file": "libs/partners/openrouter/tests/unit_tests/test_chat_models.py", + "line": 299, + "severity": "minor", + "comment": "The test asserts `call_kwargs[\"client\"].headers[\"HTTP-Referer\"]` but httpx normalizes header names to lowercase internally. Depending on the httpx version, accessing headers by the original case may or may not work. The httpx `Headers` class is case-insensitive, so this works, but it could be confusing to future readers who expect lowercase header access." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 204, + "severity": "positive", + "comment": "The improved docstring for `max_retries` now clearly explains the relationship between the retry count and the backoff window duration, with a concrete example. This is a helpful documentation improvement." + } + ], + "summary": "The PR correctly migrates attribution headers from SDK constructor kwargs to httpx `default_headers`, making the integration resilient to SDK parameter renames. The main concern is that `follow_redirects=True` and the httpx client are only created when at least one attribution header is set, which could produce inconsistent behavior when no attribution fields are configured." + }, + "flow_guided_review": { + "comments": [ + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 343, + "severity": "medium", + "comment": "Following the flow from the test changes (steps 2-8) back to the core `_build_client` method (step 9): the tests now assert headers on `call_kwargs[\"client\"].headers` and `call_kwargs[\"async_client\"].headers`, confirming all three attribution headers are injected via httpx clients. However, the conditional `if extra_headers` means that when no attribution fields are set, no custom httpx client is created at all. The tests for `test_app_categories_none_no_categories_header` and `test_app_categories_empty_list_no_categories_header` (steps 6-7) verify the absence of categories but should also verify the absence of custom clients entirely -- if a future change adds a non-attribution default header, these tests would silently pass despite behavioral changes." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 389, + "severity": "medium", + "comment": "Step 10 identifies `validate_environment` as high risk due to being the core setup. Moving `_build_client()` inside the `try/except ImportError` is the right call for catching version-mismatch errors from `openrouter.utils`. But `_build_client` also does `import httpx` (line 349), so if httpx is missing, the user gets the misleading message 'Please install it with: pip install openrouter' when the actual fix is `pip install httpx`. Consider either catching httpx ImportError separately or mentioning httpx in the error message." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 349, + "severity": "medium", + "comment": "Steps 6-7 (categories-none and categories-empty tests) reveal an asymmetry: when no headers are needed, the SDK gets no custom httpx client, but when any header is set, both sync and async clients are created with `follow_redirects=True`. This means redirect behavior differs based on attribution configuration. The plan's identification of these tests as high-risk entry points highlights that the no-headers path is under-tested for behavioral parity with the headers path." + }, + { + "file": "libs/partners/openrouter/tests/unit_tests/test_chat_models.py", + "line": 299, + "severity": "minor", + "comment": "Steps 2-3 (test_app_url and test_app_title) changed from asserting SDK kwargs (`http_referer`, `x_title`) to asserting httpx client headers. The tests correctly verify the new header injection mechanism but do not assert that the old kwargs (`http_referer`, `x_title`) are absent from the call. Adding negative assertions would ensure the migration is complete and no duplicate header paths exist." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 996, + "severity": "minor", + "comment": "Step 11 (`_wrap_messages_for_sdk`) adds warnings for two previously silent fallbacks. The plan flags this as medium risk. The warning for failed `openrouter.components` import correctly alerts users, but since this function is called per-invocation, repeated calls will produce repeated warnings. Consider using `warnings.warn` with a specific `UserWarning` subclass or `warnings.filterwarnings` to ensure it only fires once per session." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 204, + "severity": "positive", + "comment": "The docstring improvement for `max_retries` adds a concrete example (`max_retries=2` allows ~300s), which directly helps users understand the SDK's backoff behavior. This is a small but valuable documentation enhancement." + } + ], + "summary": "The flow-guided review reveals that the httpx client (with `follow_redirects=True`) is only created when attribution headers are present, creating a behavioral asymmetry the tests do not cover. Additionally, moving `_build_client` inside the `try/except ImportError` inadvertently catches a missing-httpx error with a misleading install message, which the plan's risk analysis of `validate_environment` correctly flagged as high risk." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leverages the plan's step ordering and risk annotations to produce deeper insights. Both reviews identify the `follow_redirects` asymmetry, but the flow-guided review connects it to specific test steps (6-7) and explains why the current test coverage is insufficient. The flow-guided review also identifies the misleading error message when httpx is missing -- a direct consequence of tracing the `validate_environment` flow through `_build_client` as the plan suggests. The baseline review raises valid points about warning stacklevels and tone consistency, but these are lower-impact observations. The flow-guided review's suggestion to add negative assertions for removed kwargs (steps 2-3) is a concrete, actionable improvement the baseline misses entirely." + } +} \ No newline at end of file diff --git a/evals/langchain-ai__langchain__36348.json b/evals/langchain-ai__langchain__36348.json new file mode 100644 index 0000000..8bb709f --- /dev/null +++ b/evals/langchain-ai__langchain__36348.json @@ -0,0 +1,96 @@ +{ + "pr": "langchain-ai/langchain#36348", + "title": "release(openrouter): 0.2.1", + "baseline_review": { + "comments": [ + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 1004, + "severity": "minor", + "comment": "The rename from `components.UserMessage` to `components.ChatUserMessage` (and similarly for all five role types) aligns with the openrouter SDK 0.8.0 upgrade visible in uv.lock. However, this is a breaking change in the upstream SDK -- if any downstream code or plugin directly imports these component types from the openrouter package, they will break. Consider adding a note in a CHANGELOG or release notes about the upstream SDK version bump." + }, + { + "file": "libs/partners/openrouter/pyproject.toml", + "line": 25, + "severity": "minor", + "comment": "The `openrouter` dependency lower bound remains `>=0.7.11` but the code now uses `ChatUserMessage` etc. which only exist in openrouter 0.8.0+. If someone installs with openrouter 0.7.x they will get an ImportError at runtime. The lower bound should be bumped to `>=0.8.0,<1.0.0` to match the actual API usage." + }, + { + "file": "libs/partners/openrouter/pyproject.toml", + "line": 24, + "severity": "nit", + "comment": "The `langchain-core` minimum was bumped from `>=1.2.21` to `>=1.2.23`. The diff does not show any new langchain-core APIs being used. It would be helpful to document what in 1.2.23 is required, or whether this is just a precautionary bump." + }, + { + "file": "libs/partners/openrouter/tests/unit_tests/test_chat_models.py", + "line": 2509, + "severity": "positive", + "comment": "Tests were updated in lockstep with the production code to use the new `Chat*Message` class names, and existing test logic (structure, assertions, coverage) is preserved. Good that both `test_wraps_as_pydantic_models` and `test_all_roles_wrapped` were updated." + }, + { + "file": "libs/partners/openrouter/tests/unit_tests/test_chat_models.py", + "line": 2570, + "severity": "minor", + "comment": "The `test_all_roles_wrapped` test covers system, user, assistant, and tool roles but does not include developer role, which is mapped in the production `role_to_model` dict. This is a pre-existing gap, not introduced by this PR, but worth noting since the developer role mapping was also updated." + } + ], + "summary": "This is a release PR that adapts langchain-openrouter to the openrouter SDK 0.8.0 class renames (e.g., `UserMessage` -> `ChatUserMessage`). The most actionable concern is that the `openrouter` dependency lower bound in pyproject.toml still allows 0.7.x, which would cause ImportErrors since the renamed classes only exist in 0.8.0+." + }, + "flow_guided_review": { + "comments": [ + { + "file": "libs/partners/openrouter/pyproject.toml", + "line": 25, + "severity": "major", + "comment": "The `openrouter>=0.7.11` lower bound is incompatible with the code changes. The `_wrap_messages_for_sdk` function (a high-risk node called by `_generate`, `_agenerate`, `_stream`, and `_astream`) now references `components.ChatUserMessage` etc. which only exist in openrouter>=0.8.0. Since the review plan identifies `_wrap_messages_for_sdk` as having four callers (all core chat generation paths), an ImportError here would completely break the integration. The lower bound must be bumped to `>=0.8.0,<1.0.0`." + }, + { + "file": "libs/partners/openrouter/langchain_openrouter/chat_models.py", + "line": 1004, + "severity": "minor", + "comment": "The `role_to_model` mapping in `_wrap_messages_for_sdk` is the single point where the SDK component types are referenced. The review plan correctly flags this function as high-risk due to its four callers (`_generate`, `_agenerate`, `_stream`, `_astream`). The rename itself is mechanical and correct, but the blast radius of any error here spans all chat operations -- synchronous, asynchronous, and streaming." + }, + { + "file": "libs/partners/openrouter/tests/unit_tests/test_chat_models.py", + "line": 2570, + "severity": "minor", + "comment": "Following the review plan's entry points: `test_all_roles_wrapped` validates system, user, assistant, and tool roles but omits `developer`. The production `role_to_model` dict maps all five roles including developer. Since the plan identifies the test class as high-risk entry points, this coverage gap means the developer role rename is only verified by inspecting the diff, not by a test assertion." + }, + { + "file": "libs/partners/openrouter/tests/unit_tests/test_chat_models.py", + "line": 2509, + "severity": "positive", + "comment": "Both test entry points identified by the review plan (`test_wraps_as_pydantic_models` at order 2, `test_all_roles_wrapped` at order 3) were updated to match the new class names. The tests serve as integration-like verification that the component import paths are valid, which partially mitigates the dependency version concern." + }, + { + "file": "libs/partners/openrouter/pyproject.toml", + "line": 24, + "severity": "nit", + "comment": "The `langchain-core` lower bound bump from 1.2.21 to 1.2.23 is not explained by the diff. None of the changed code paths (`_wrap_messages_for_sdk` or the tests) appear to use new langchain-core APIs. This may be a transitive requirement from the openrouter SDK upgrade, but it should be documented." + } + ], + "summary": "The review plan reveals that `_wrap_messages_for_sdk` is the critical chokepoint -- called by all four chat generation methods -- making the openrouter SDK class renames high-impact despite being mechanical. The most significant finding is that the `openrouter` dependency lower bound (>=0.7.11) does not enforce the 0.8.0 minimum required by the new class names, creating a latent ImportError for anyone resolving to an older SDK version." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.6 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same core issue -- the openrouter dependency lower bound allowing 0.7.x despite code requiring 0.8.0+ classes. However, the flow-guided review elevates this from 'minor' to 'major' by leveraging the review plan's caller graph: `_wrap_messages_for_sdk` feeds all four chat generation methods (_generate, _agenerate, _stream, _astream), meaning the ImportError would be total, not partial. The flow-guided review also better contextualizes the test coverage gap (missing developer role assertion) by connecting it to the plan's risk assessment of the test class as a high-risk entry point. The baseline review treats each file in isolation and misses the dependency chain that makes the version bound issue critical rather than advisory. The flow-guided review's understanding of the call graph turns a routine observation into an actionable severity upgrade." + } +} diff --git a/evals/microsoft__TypeScript__63305.json b/evals/microsoft__TypeScript__63305.json new file mode 100644 index 0000000..a4d3c23 --- /dev/null +++ b/evals/microsoft__TypeScript__63305.json @@ -0,0 +1,128 @@ +{ + "pr": { + "url": "https://github.com/microsoft/TypeScript/pull/63305", + "owner": "microsoft", + "repo": "TypeScript", + "number": 63305, + "title": "Add coding agent instructions: refuse PRs unless maintenance mode is acknowledged", + "files_changed": 4, + "additions": 64, + "deletions": 2, + "language": "Markdown" + }, + "timestamp": "2026-03-30T20:45:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "AGENTS.md", + "line": 1, + "severity": "minor", + "comment": "The content of AGENTS.md is nearly identical to the block prepended to .github/copilot-instructions.md (lines 1-29). This duplication means any future policy update must be synchronized across both files. Consider having one file be the source of truth and the other reference it, or extract the shared text into a single location." + }, + { + "file": ".github/copilot-instructions.md", + "line": 1, + "severity": "positive", + "comment": "Placing the maintenance-mode warning at the very top of the existing Copilot instructions is the right call. Agents typically read files top-down and may truncate long contexts, so front-loading the critical gate maximizes the chance it is honored." + }, + { + "file": "AGENTS.md", + "line": 7, + "severity": "minor", + "comment": "The blog post link ('TypeScript 7.0 progress blog post') points to a December 2025 URL. If this post is ever taken down or reorganized, the link will break. The issue link (#62963) is more durable. Consider making the blog post link supplementary rather than a co-equal reference." + }, + { + "file": "README.md", + "line": 1, + "severity": "nit", + "comment": "The diff removes the leading blank line before '# TypeScript'. This is a minor formatting change unrelated to the PR's purpose and could show up as noise in blame history." + }, + { + "file": "CONTRIBUTING.md", + "line": 3, + "severity": "nit", + "comment": "The HTML comment directive is invisible in rendered Markdown. Agents that only process rendered Markdown (e.g., via GitHub API rendered_body) will miss it entirely. The existing bold warning below already tells humans the same thing, so this comment is only useful for agents that read raw source." + }, + { + "file": "AGENTS.md", + "line": 30, + "severity": "minor", + "comment": "AGENTS.md delegates build/test details to .github/copilot-instructions.md, but that file is Copilot-specific by name. Non-Copilot agents (Claude, Cursor) discovering the repo through AGENTS.md may be confused by the copilot-branded filename. A brief note clarifying that the instructions are generic despite the filename would help." + } + ], + "summary": "This PR adds a consistent maintenance-mode gate across four files to stop AI coding agents from submitting unwanted PRs to the winding-down JS-based TypeScript repo. The main concern is content duplication between AGENTS.md and the copilot-instructions.md preamble, which creates a maintenance burden for future policy changes." + }, + "flow_guided_review": { + "comments": [ + { + "file": "AGENTS.md", + "line": 1, + "severity": "minor", + "comment": "The content block in AGENTS.md is nearly a verbatim copy of the block prepended to .github/copilot-instructions.md. With an empty flow plan (no code dependencies to trace), this duplication is the single most actionable maintenance risk: a future editor updating one file may miss the other. Consider a single canonical source with cross-references." + }, + { + "file": ".github/copilot-instructions.md", + "line": 1, + "severity": "positive", + "comment": "Prepending the gate before existing coding instructions is well-placed. Since the review plan has zero code-flow steps, the only 'flow' here is the agent's reading order, and top-of-file placement optimizes for that." + }, + { + "file": "AGENTS.md", + "line": 12, + "severity": "minor", + "comment": "The accepted-categories list uses subjective qualifiers ('substantially impact mainline usage', 'large proportion of users') without defining thresholds. An agent cannot evaluate these criteria autonomously and will likely ask the user every time, which is arguably the intended behavior but could lead to prompt fatigue." + }, + { + "file": "README.md", + "line": 1, + "severity": "nit", + "comment": "Removal of the blank line before the heading is an unrelated cosmetic change. Since the review plan contains no dependencies or flow clusters, there is no structural reason this file needed touching beyond adding the HTML comment, so the blank-line removal is gratuitous diff noise." + }, + { + "file": "CONTRIBUTING.md", + "line": 3, + "severity": "nit", + "comment": "The HTML comment is a reasonable breadcrumb for agents that read raw Markdown. However, CONTRIBUTING.md already has a bold human-visible maintenance-mode notice immediately below, so the incremental value for agents is limited to those that specifically parse HTML comments as directives." + }, + { + "file": "AGENTS.md", + "line": 22, + "severity": "minor", + "comment": "Step 3 says 'Refuse to proceed until that acknowledgement is given.' Different agents interpret 'refuse' differently — some may hard-stop, others may just warn and continue. The instruction could be strengthened by saying 'Do not write, modify, or suggest any code changes until...' to be unambiguous about what 'refuse' means." + } + ], + "summary": "This documentation-only PR has an empty flow plan (zero steps, zero dependencies), so flow-guided analysis adds little structural insight beyond the baseline. The key risks are content duplication across AGENTS.md and copilot-instructions.md, and vague subjective criteria in the accepted-categories list that agents cannot evaluate autonomously." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.4 + }, + "reasoning": "Both reviews correctly identify duplication as the primary risk and note the cosmetic README change. The flow-guided review gains a slight edge by (a) explicitly contextualizing the empty plan and what that means for review strategy, (b) flagging the vague subjective criteria in the accepted-categories list as a practical agent-interaction risk, and (c) suggesting a concrete rewording for the 'refuse to proceed' instruction to improve enforceability. However, with zero code flow to analyze, the structural advantage of flow-guided review is minimal. Flow awareness scores low for both since there is no code flow to be aware of; the flow-guided version scores marginally higher for acknowledging this explicitly.", + "winner": "flow_guided" + } +} \ No newline at end of file diff --git a/evals/minio__minio__21642.json b/evals/minio__minio__21642.json new file mode 100644 index 0000000..bbd89fc --- /dev/null +++ b/evals/minio__minio__21642.json @@ -0,0 +1,108 @@ +{ + "pr": "minio/minio#21642", + "title": "fix: check sub-policy properly when present", + "baseline_review": { + "comments": [ + { + "file": "cmd/iam.go", + "line": 2400, + "severity": "high", + "comment": "The change narrows the condition for treating a sub-policy as absent (returning hasSessionPolicy=false) to only when all three fields (Version, Statements, ID) are empty. Previously, the code had a nuanced comment explaining that `{\"Statement\":[]}` (empty statement array) should deny everything, while `null`/`{}` should inherit. The new simplified check still uses `subPolicy.Statements == nil`, which means an explicit empty slice `[]` will pass this guard and proceed to policy evaluation. Verify that `json.Unmarshal` of `{\"Statement\":[]}` produces a nil slice vs an empty slice in Go -- if it produces `[]policy.Statement{}`, the behavior differs from the old code's intent described in the removed TODO." + }, + { + "file": "cmd/iam.go", + "line": 2410, + "severity": "high", + "comment": "The new comment mentions setting `DenyOnly` arg to false as an important corner case, but the diff is truncated and we cannot see the actual code change. This is the core security fix -- ensuring that when a sub-policy exists, the session policy is evaluated with `DenyOnly=false` so that it properly restricts operations rather than only checking deny statements. The comment should be more explicit about what `DenyOnly` controls and why setting it to false here prevents the privilege escalation." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 1251, + "severity": "medium", + "comment": "The test function `TestServiceAccountPrivilegeEscalationBug2_2025_10_15` is well-structured, testing both root-owned and regular-user-owned service accounts via the `forRoot` parameter. However, the test only verifies that creating a new service account without a restrictive policy fails. It does not test the positive case -- that the restricted service account can still perform its allowed operations (s3:GetObject, s3:PutObject on bucket1/bucket2) after the fix. This would guard against an overly aggressive fix that blocks all operations." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 1338, + "severity": "medium", + "comment": "The assertion `len(buckets) != 2 || buckets[0].Name != \"bucket1\" || buckets[1].Name != \"bucket2\"` assumes a specific bucket ordering in the ListBuckets response. While MinIO likely returns buckets in alphabetical order, this is fragile. If another test running concurrently creates buckets with names that sort between 'bucket1' and 'bucket2', this assertion would fail. Consider checking bucket names as a set rather than by index." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 1344, + "severity": "medium", + "comment": "The privilege escalation attempt only tests creating a service account without any policy restriction (empty AddServiceAccountReq). Consider also testing the case where the restricted service account tries to create a new service account with an escalated policy that grants access to bucket3 -- this is the more subtle escalation vector where the attacker tries to expand permissions rather than just bypass them entirely." + }, + { + "file": "cmd/iam.go", + "line": 2400, + "severity": "low", + "comment": "The large block of removed comments included a TODO about fixing console behavior and disallowing empty statement policies. Removing the TODO without addressing it or creating a tracking issue means this planned work may be lost. If the new approach intentionally supersedes that plan, a brief note explaining why would help future maintainers." + } + ], + "summary": "This PR fixes a security vulnerability where service accounts with restricted sub-policies could create new service accounts bypassing their inline policy restrictions. The core fix in `iam.go` changes how sub-policies are evaluated by setting `DenyOnly` to false when a session policy exists, though the diff truncation obscures the complete implementation, and the removed TODO comments about empty statement policies deserve explicit resolution." + }, + "flow_guided_review": { + "comments": [ + { + "file": "cmd/iam.go", + "line": 2400, + "severity": "high", + "comment": "The review plan is empty (0 steps, 0 clusters), so we must reconstruct the flow. The security-critical path starts in `isAllowedBySessionPolicyForServiceAccount`. The old code had an extensive comment about the difference between `null`/`{}` (inherit parent) and `{\"Statement\":[]}` (deny all), plus a TODO about fixing this. The new code collapses this to a single nil-check on all three fields. This simplification is correct if the contract is that any non-zero policy struct means 'restrict', but it changes the semantics for the edge case of `{\"Statement\":[]}` -- previously noted as a security concern. Confirm that the `DenyOnly` change (truncated in the diff) properly handles this edge case." + }, + { + "file": "cmd/iam.go", + "line": 2410, + "severity": "high", + "comment": "The `DenyOnly` flag is the actual fix for the privilege escalation. When a service account performs 'own account' operations (like creating sub-service-accounts), the policy evaluation apparently used `DenyOnly=true`, which only checked for explicit deny statements and allowed everything else. By setting `DenyOnly=false`, the sub-policy's allow statements are now evaluated as a whitelist. The truncated diff prevents full verification, but this is the correct architectural approach -- the third file (likely another Go source file with the policy evaluation caller) probably contains the complementary change that triggers this code path." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 1251, + "severity": "medium", + "comment": "The test correctly reproduces the exact attack scenario: a service account with a policy allowing only bucket1/bucket2 access attempts to create an unrestricted service account. The `forRoot` parameter tests both vectors (root-owned and user-owned service accounts). However, the test should also verify that the error returned is an access-denied error specifically, not some other failure, to ensure the policy check is what blocks the operation." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 1291, + "severity": "medium", + "comment": "In the non-root path, the user 'foobar' is given the `consoleAdmin` policy, which grants broad admin permissions. The restricted service account is then created for this user. This is a good test setup because it proves that even when the parent user has full admin rights, the sub-policy on the service account properly restricts the service account's operations. The test could be even stronger by also testing with a parent user that has a narrower policy to verify that the intersection of parent policy and sub-policy is computed correctly." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 1344, + "severity": "medium", + "comment": "The escalation attempt creates a service account with no policy restriction (`AddServiceAccountReq{AccessKey: \"newroot\", SecretKey: \"newroot123\"}`). This is the most obvious escalation -- inheriting the full parent policy. A more sophisticated test would attempt to create a service account with an explicit policy granting bucket3 access, verifying that a restricted account cannot delegate permissions it does not have." + }, + { + "file": "cmd/admin-handlers-users_test.go", + "line": 208, + "severity": "low", + "comment": "The test is registered in the test suite with both `true` and `false` variants called sequentially. This is clean, but note that these tests share the same access key name 'restricted' and bucket names 'bucket1'-'bucket3'. Since they run sequentially within the same suite and each has proper defer cleanup, this should be fine, but if test parallelization is ever added, these would conflict." + } + ], + "summary": "The PR addresses a privilege escalation vulnerability by ensuring sub-policies on service accounts are fully evaluated (not in DenyOnly mode) when the account performs self-management operations like creating new service accounts. The test coverage proves the fix works for both root-owned and regular-user-owned service accounts, though the truncated diff prevents full verification of the `DenyOnly` change and the third modified file." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.6 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.2 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews are constrained by the truncated diff, which hides the actual DenyOnly code change and the third file. The flow-guided review provides better risk identification by reconstructing the security flow -- understanding that the DenyOnly flag controls whether the policy evaluation acts as a whitelist or only checks explicit denies, which is the root cause of the privilege escalation. The baseline review correctly identifies the same issues but treats them more in isolation. The flow-guided review better connects the test setup (consoleAdmin parent policy with restricted sub-policy) to the security model being tested. Both reviews are limited by the empty review plan (0 steps), which provides no structural guidance, making the flow-guided review only marginally better since it must reconstruct the flow from the diff alone. The scores are close because the empty plan neutralizes most of the flow-guided approach's structural advantage." + } +} \ No newline at end of file diff --git a/evals/minio__minio__21651.json b/evals/minio__minio__21651.json new file mode 100644 index 0000000..7bb39f5 --- /dev/null +++ b/evals/minio__minio__21651.json @@ -0,0 +1,108 @@ +{ + "pr": "minio/minio#21651", + "title": "fix(api): Don't send multiple responses for one request", + "baseline_review": { + "comments": [ + { + "file": "cmd/api-response.go", + "line": 891, + "severity": "major", + "comment": "The guard in `writeResponse` silently returns when headers have already been written, discarding the response body and status code with no logging or error indication. In a debugging scenario, this silent swallowing makes it very hard to detect that a double-response was attempted. At minimum, a debug-level log line should be emitted so operators can identify the handlers that are sending duplicate responses and fix them at the source rather than relying on this safety net permanently." + }, + { + "file": "cmd/api-response.go", + "line": 1046, + "severity": "major", + "comment": "The `trackingResponseWriter.Write` method delegates directly to the inner `ResponseWriter.Write` but does not set `headerWritten = true`. In the standard `net/http` contract, calling `Write` without a prior `WriteHeader` implicitly sends a 200 status and writes headers. This means if `Write` is called before `WriteHeader`, the tracking state will be incorrect -- `headerWritten` remains false even though headers were actually sent. A subsequent call through `writeResponse` would then attempt to write a second response, which is exactly the bug this PR aims to prevent." + }, + { + "file": "cmd/api-router.go", + "line": 221, + "severity": "major", + "comment": "The `trackingResponseWriter` is inserted at the top of `s3APIMiddleware`, but the variable `w` is reassigned with `:=` which shadows the parameter. Subsequent middleware layers (tracing, gzip compression) will wrap this further. The `headersAlreadyWritten` function unwraps through the `Unwrap()` interface, but if any intermediate wrapper calls `WriteHeader` on its own wrapped writer rather than through the `trackingResponseWriter`, the `headerWritten` flag will never be set. This relies on all wrappers in the chain faithfully delegating `WriteHeader` downward, which should be verified for the tracing and gzip middleware used in minio." + }, + { + "file": "cmd/api-response.go", + "line": 1038, + "severity": "minor", + "comment": "The `trackingResponseWriter.WriteHeader` prevents duplicate `WriteHeader` calls by checking `headerWritten`, which is a useful safety measure. However, the standard library's `http.ResponseWriter` contract expects `WriteHeader` to be called at most once -- calling it multiple times is already a bug. Silently suppressing the second call without logging means callers won't know their status code was dropped. Consider logging at debug level when a duplicate WriteHeader is suppressed." + }, + { + "file": "cmd/api-response_test.go", + "line": 167, + "severity": "minor", + "comment": "The `TestHeadersAlreadyWrittenWrapped` test uses `gzhttp.NoGzipResponseWriter` as the wrapping layer, which introduces a test dependency on an external compression library's internal type. If `gzhttp.NoGzipResponseWriter` changes its `Unwrap` behavior or becomes unexported in a future version, this test breaks. A local test wrapper implementing the `unwrapper` interface would be more robust and self-documenting." + }, + { + "file": "cmd/api-response.go", + "line": 1024, + "severity": "nit", + "comment": "The `unwrapper` interface duplicates the standard `http.ResponseController` unwrap pattern available since Go 1.20. Consider using `interface { Unwrap() http.ResponseWriter }` inline at the call site or documenting why a named interface is preferred, since `unwrapper` is unexported and used in exactly one place." + } + ], + "summary": "The PR introduces a response-tracking wrapper to prevent double writes, which addresses the reported connection-drop bug. However, the `Write` method does not update the `headerWritten` flag, meaning implicit header writes via `Write()` without a prior `WriteHeader()` will bypass the tracking -- this is the most significant gap that could allow the original bug to persist in certain code paths." + }, + "flow_guided_review": { + "comments": [ + { + "file": "cmd/api-response.go", + "line": 1046, + "severity": "critical", + "comment": "The `Write` method on `trackingResponseWriter` does not set `headerWritten = true` before or after delegating to the inner writer. Per the `net/http` spec, calling `Write` without a prior `WriteHeader` implicitly sends HTTP 200 and writes headers. This creates a correctness gap: if any handler calls `Write` directly (without `WriteHeader`), the tracking flag remains false, and a subsequent `writeResponse` call will attempt to write headers again, causing the exact double-response bug this PR is meant to fix. The fix is to set `w.headerWritten = true` at the top of the `Write` method, mirroring what the standard library does internally." + }, + { + "file": "cmd/api-router.go", + "line": 221, + "severity": "major", + "comment": "The `trackingResponseWriter` is injected at the outermost layer of the middleware chain in `s3APIMiddleware`. All subsequent wrappers (tracing, gzip via `gzhttp`) will wrap around it. The `headersAlreadyWritten` function traverses inward via `Unwrap()` to find the tracker. This architecture means `WriteHeader` calls from outer wrappers must propagate inward through the chain to reach the tracker. If any middleware (e.g., the gzip handler) intercepts `WriteHeader` and writes its own status without delegating to the inner writer, the flag will never flip. The PR should verify that `gzhttp` and the tracing middleware properly delegate `WriteHeader` to their wrapped writers in all code paths, including error paths." + }, + { + "file": "cmd/api-response.go", + "line": 891, + "severity": "major", + "comment": "The guard clause in `writeResponse` silently returns when `headersAlreadyWritten` is true. While this prevents the panic/connection-drop from double writes, it masks the root cause -- some handler is sending two responses. Without any logging, operators cannot identify which handlers have this bug. A `logger.LogIf(ctx, ...)` or similar debug log would surface these occurrences so the underlying handler bugs can be found and fixed over time rather than permanently relying on this workaround." + }, + { + "file": "cmd/api-response.go", + "line": 1028, + "severity": "minor", + "comment": "The `headersAlreadyWritten` function returns `false` when the writer chain does not contain a `trackingResponseWriter`. This is a safe default, but it means any code path that constructs a response writer without going through `s3APIMiddleware` (e.g., internal health endpoints, admin API handlers) will silently bypass the protection. The function's doc comment should explicitly note this limitation so future developers understand the scope of the fix." + }, + { + "file": "cmd/api-response_test.go", + "line": 195, + "severity": "minor", + "comment": "The `TestWriteResponseHeadersWritten` test manually sets `headerWritten: true` on the struct and verifies the response is suppressed. This is good for the guard-clause test, but there is no integration-style test that exercises the full double-write scenario: first call `writeResponse`, then call it again, and verify only the first response is sent. Such a test would validate the complete flow rather than just the guard condition in isolation." + }, + { + "file": "cmd/api-response_test.go", + "line": 167, + "severity": "minor", + "comment": "Using `gzhttp.NoGzipResponseWriter` in tests couples the test to the internal structure of an external dependency. If this type's Unwrap behavior changes, the test will break for reasons unrelated to the code under test. A simple local wrapper `type testWrapper struct { http.ResponseWriter }` with an `Unwrap` method would test the same unwrapping logic without the external coupling." + } + ], + "summary": "The PR's approach of tracking header writes to prevent double responses is architecturally sound, but the critical gap is that `trackingResponseWriter.Write()` does not set the `headerWritten` flag, meaning implicit header writes bypass the tracking entirely. Additionally, the silent suppression of duplicate responses without logging makes it impossible to identify and fix the underlying handler bugs that cause double writes." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same critical bug -- the `Write` method not updating `headerWritten` -- but the flow-guided review elevates it to critical severity and provides a concrete fix (set the flag at the top of Write), making it more actionable. The flow-guided review also does a better job analyzing the middleware chain interaction, tracing how `WriteHeader` calls must propagate inward through gzip and tracing wrappers to reach the tracker, and explicitly calling out the risk that some middleware might intercept without delegating. The baseline review raises the same concern but in more generic terms. Both reviews flag the silent suppression issue and the test coupling to gzhttp, but the flow-guided review adds the missing integration test suggestion (double-write scenario) and notes the scope limitation for non-S3 API handlers. The review plan was empty, so the flow-guided advantage comes primarily from systematic data-flow reasoning rather than plan-driven ordering, which limits the flow_awareness score gap. Overall the flow-guided review is more thorough, more precisely prioritized, and more actionable." + } +} \ No newline at end of file diff --git a/evals/minio__minio__21653.json b/evals/minio__minio__21653.json new file mode 100644 index 0000000..42fea34 --- /dev/null +++ b/evals/minio__minio__21653.json @@ -0,0 +1,102 @@ +{ + "pr": "minio/minio#21653", + "title": "Updates for conditional put read quorum issue", + "baseline_review": { + "comments": [ + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 115, + "severity": "medium", + "comment": "The disk-offlining loop mutates the shared `erasureDisks` slice in place by setting entries to nil. Since `getDisks` returns the same slice reference each time it is called, any code that caches or iterates the slice concurrently could see partially-nil entries mid-mutation. Consider creating a copy of the slice inside the closure and setting entries to nil on the copy to avoid data races if the test is ever run with `-race`." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 157, + "severity": "high", + "comment": "In the 'if-match with wrong ETag and read quorum failure' subtest, the check uses `t.Logf` instead of `t.Errorf` when the error is not a read quorum error. This means the test will never actually fail for this case -- it will just log a message and pass. This should be `t.Errorf` (like the other subtests) to enforce that a read quorum error is expected." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 127, + "severity": "low", + "comment": "The `CheckPrecondFn` for `if-none-match: *` returns true when `oi.ETag != \"\"` (object exists), meaning it signals 'precondition failed' when the object exists. The naming/semantics are correct for MinIO's internal convention where `CheckPrecondFn` returning true means 'condition failed, abort', but a brief inline comment clarifying this inverted return convention would improve readability." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 148, + "severity": "low", + "comment": "The `if-match with wrong ETag` subtest sets `HasIfMatch: true` but the `if-none-match` subtest (line 127) does not set a corresponding `HasIfNoneMatch` flag. Verify that the asymmetry is intentional and that the code under test does not check for a `HasIfNoneMatch` field to gate its behavior." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 74, + "severity": "medium", + "comment": "The test only covers `NewMultipartUpload` and `CompleteMultipartUpload` but the PR description mentions fixing conditional PUT operations broadly. There is no test for `PutObject` with conditional headers under read quorum failure. If the production fix also touches the non-multipart put path, corresponding test coverage should be added." + } + ], + "summary": "The new test file correctly validates that conditional multipart uploads fail with read quorum errors when sufficient disks are offline, directly addressing the reported issue. However, the 'if-match with wrong ETag' subtest silently passes due to using `t.Logf` instead of `t.Errorf`, and the overall test coverage does not extend to non-multipart conditional PUTs that may also be affected by the fix." + }, + "flow_guided_review": { + "comments": [ + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 157, + "severity": "high", + "comment": "Critical test correctness bug: the 'if-match with wrong ETag and read quorum failure' subtest uses `t.Logf` for its assertion instead of `t.Errorf`. This means the test will always pass regardless of the returned error, providing no regression protection for this specific case. All three subtests should consistently use `t.Errorf` to enforce the expected read quorum error." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 110, + "severity": "medium", + "comment": "The disk-offlining approach sets 8 of 16 disks to nil, leaving exactly 8 available. With EC 8+8, read quorum requires 9 disks (N/2 + 1 for reads). This is correct for triggering a read quorum failure, but the test relies on an implicit understanding of MinIO's quorum formula. A comment explicitly stating the quorum math (e.g., 'read quorum = 9, available = 8, expect failure') would make the test more maintainable and self-documenting." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 115, + "severity": "medium", + "comment": "The `getDisks` closure mutates the original `erasureDisks` slice in place rather than returning a new slice with nils. If `getDisks` is called multiple times (which is likely during the multipart operations), the first call sets disks 0-7 to nil and subsequent calls re-nil already-nil entries. While functionally benign, creating the modified slice once outside the closure and capturing it would be cleaner and avoid potential race conditions under `-race`." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 74, + "severity": "medium", + "comment": "The PR description says it fixes conditional PUT operations broadly, and the diff shows 4 files changed with 381 lines. This test file only covers `NewMultipartUpload` and `CompleteMultipartUpload`. Without seeing the production code changes to the actual quorum-checking logic, we cannot verify that the fix is complete. The test coverage should also include a non-multipart `PutObject` conditional path if that was also fixed." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 148, + "severity": "low", + "comment": "The `HasIfMatch` field is explicitly set for the if-match subtests but there is no corresponding explicit flag for the if-none-match subtest. If the production code branches on `HasIfMatch` vs checking `UserDefined` headers, the if-none-match test might be exercising a different code path than intended. Verify that both conditional header types route through the same fixed quorum-check logic." + }, + { + "file": "cmd/erasure-multipart-conditional_test.go", + "line": 34, + "severity": "low", + "comment": "The tests only simulate quorum failure by nulling disks before the operation. Consider adding a test where quorum is lost mid-operation (e.g., disks go offline between NewMultipartUpload and CompleteMultipartUpload) to verify the fix handles transient quorum loss during the complete phase as well." + } + ], + "summary": "The test file targets the correct failure mode -- conditional multipart uploads when read quorum is unreachable -- and the test structure with three subtests covering if-none-match, if-match with wrong ETag, and if-match with correct ETag is well-designed. However, a critical assertion bug (using `t.Logf` instead of `t.Errorf`) renders one subtest ineffective, and the truncated diff prevents full verification that the production code changes correctly propagate read quorum errors instead of silently proceeding." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.0 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the critical `t.Logf` vs `t.Errorf` bug in the wrong-ETag subtest, which is the most actionable finding. The flow-guided review provides slightly better risk identification by explicitly calling out the inability to verify production code changes due to the truncated diff, and by suggesting transient quorum-loss scenarios. However, with an empty review plan (no steps, no clusters, no dependencies), the flow-guided review has limited advantage in flow awareness -- both reviews are essentially working from the diff alone. The flow-guided review edges ahead on actionability by more explicitly connecting the test gaps to the PR's stated scope (4 files, 381 lines) and questioning whether the fix is complete." + } +} \ No newline at end of file diff --git a/evals/nodejs__node__62453.json b/evals/nodejs__node__62453.json new file mode 100644 index 0000000..aa269c0 --- /dev/null +++ b/evals/nodejs__node__62453.json @@ -0,0 +1,114 @@ +{ + "pr": "nodejs/node#62453", + "title": "crypto: runtime-deprecate DEP0203 and DEP0204", + "baseline_review": { + "comments": [ + { + "file": "lib/internal/crypto/keys.js", + "line": 95, + "severity": "medium", + "comment": "The `emitDEP0203` emitter is created at module scope with no conditional guard, meaning it will emit a warning on the first call and then become a no-op. Verify that `getDeprecationWarningEmitter` with default parameters produces one-shot warnings per process, as expected for runtime deprecations. If it emits on every call, this would spam stderr for code that processes many CryptoKeys." + }, + { + "file": "lib/internal/crypto/keys.js", + "line": 100, + "severity": "high", + "comment": "The `maybeEmitDEP0204` emitter passes `false` as the fourth argument and a predicate `(key) => !key[kExtractable]` as the fifth. The `false` parameter (likely controlling unconditional vs conditional emission) combined with the predicate means the warning only fires for non-extractable keys. Confirm this matches the intended semantics: extractable CryptoKeys passed to `KeyObject.from()` should NOT trigger DEP0204 (they are safe because the key material is already extractable)." + }, + { + "file": "lib/internal/crypto/keys.js", + "line": 810, + "severity": "medium", + "comment": "The refactoring from `else if` chains to early-return `if` blocks in `prepareAsymmetricKey` is a style improvement but changes control flow subtly. While logically equivalent here (each branch returns), this kind of refactoring mixed with behavioral changes (adding `emitDEP0203()`) makes the diff harder to review. The behavioral change is correct -- `emitDEP0203()` is called before using the CryptoKey's underlying KeyObject." + }, + { + "file": "doc/api/deprecations.md", + "line": 4497, + "severity": "low", + "comment": "DEP0203 is upgraded from 'Documentation-only' to 'Runtime'. The YAML changelog correctly adds a new entry with `version: REPLACEME` above the existing documentation-only entry. This follows the Node.js deprecation lifecycle correctly." + }, + { + "file": "doc/api/crypto.md", + "line": 2258, + "severity": "low", + "comment": "The `KeyObject.from()` documentation now includes a detailed description explaining that the returned KeyObject loses Web Crypto API restrictions (usages, algorithm bindings, extractability). This is a valuable addition that explains the motivation for DEP0204 -- users should be aware that `KeyObject.from()` bypasses extractability constraints." + }, + { + "file": "lib/internal/crypto/keys.js", + "line": 156, + "severity": "medium", + "comment": "In `KeyObject.from()`, `maybeEmitDEP0204(key)` is called after the type check but before returning `key[kKeyObject]`. This is correct placement -- the warning fires only for valid CryptoKey inputs, and only when the key is non-extractable. However, the diff is truncated and we cannot see whether `prepareSecretKey` also correctly calls `emitDEP0203()` for the CryptoKey path." + } + ], + "summary": "This PR escalates DEP0203 and DEP0204 from documentation-only to runtime deprecations, adding deprecation warning emitters in the crypto key handling paths. The implementation is straightforward and the documentation updates are thorough, though the diff truncation prevents full verification of the `prepareSecretKey` changes." + }, + "flow_guided_review": { + "comments": [ + { + "file": "lib/internal/crypto/keys.js", + "line": 153, + "severity": "medium", + "comment": "Entry point (KeyObject class, high risk): `KeyObject.from()` is the entry point for DEP0204. The `maybeEmitDEP0204(key)` call is correctly placed after the `isCryptoKey` type check and before accessing `key[kKeyObject]`. The conditional predicate `!key[kExtractable]` ensures only non-extractable keys trigger the warning, which aligns with the deprecation's purpose -- extractable keys can already be exported via Web Crypto, so `KeyObject.from()` adds no new capability for them." + }, + { + "file": "lib/internal/crypto/keys.js", + "line": 810, + "severity": "high", + "comment": "Internal function (prepareAsymmetricKey, high risk, many callers): This function is called by `preparePrivateKey`, `preparePublicOrPrivateKey`, `createPublicKey`, and `createPrivateKey`. The `emitDEP0203()` call is inserted in the `isCryptoKey(key)` branch before passing through to `getKeyObjectHandle`. Since this function has 4 callers, the deprecation warning correctly covers all asymmetric crypto operations that accept CryptoKey inputs. The else-if to if refactoring is safe since every branch returns." + }, + { + "file": "lib/internal/crypto/keys.js", + "line": 875, + "severity": "medium", + "comment": "Internal function (prepareSecretKey, low risk per plan): The diff is truncated before we can see the full change to `prepareSecretKey`, but the plan indicates 3 additions and 1 deletion. Since `prepareSecretKey` is called by `createSecretKey` and handles symmetric CryptoKeys, it must also emit DEP0203. Verify the truncated portion adds `emitDEP0203()` in the `isCryptoKey` branch, mirroring the pattern in `prepareAsymmetricKey`." + }, + { + "file": "lib/internal/crypto/keys.js", + "line": 95, + "severity": "medium", + "comment": "Module-level deprecation emitters: `emitDEP0203` and `maybeEmitDEP0204` are created via `getDeprecationWarningEmitter` imported from `internal/util`. DEP0203 uses default parameters (unconditional, one-shot), while DEP0204 passes `false` and a predicate. These two emitters cover the complete deprecation surface: DEP0203 for any CryptoKey usage in node:crypto functions, DEP0204 specifically for non-extractable CryptoKeys in `KeyObject.from()`. Note that a non-extractable key passed to `KeyObject.from()` will trigger BOTH warnings if it also flows through `prepareAsymmetricKey` -- verify this double-warning is intended." + }, + { + "file": "doc/api/crypto.md", + "line": 3533, + "severity": "low", + "comment": "Documentation completeness: The YAML changelog entries are added to `createCipheriv`, `createDecipheriv`, `createSign`, `createPrivateKey`, and `createPublicKey` -- all functions that accept a key parameter. Cross-referencing with the code, these correspond to the callers of `prepareAsymmetricKey` and `prepareSecretKey`. The documentation coverage matches the code paths that emit DEP0203." + }, + { + "file": "doc/api/deprecations.md", + "line": 4497, + "severity": "low", + "comment": "Both DEP0203 and DEP0204 are correctly upgraded from 'Documentation-only' to 'Runtime' with new YAML changelog entries. The deprecation descriptions remain accurate -- DEP0203 covers passing CryptoKey to node:crypto functions broadly, DEP0204 covers the specific case of non-extractable CryptoKey to KeyObject.from()." + }, + { + "file": "doc/api/crypto.md", + "line": 2258, + "severity": "low", + "comment": "The enhanced `KeyObject.from()` documentation now explains WHY the deprecation exists: the returned KeyObject does not retain Web Crypto restrictions (usages, algorithm bindings, extractability). This context is valuable for developers encountering the deprecation warning and understanding the security implications of their code." + } + ], + "summary": "Following the review plan's risk-ordered traversal from the KeyObject entry point through prepareAsymmetricKey (4 callers) and prepareSecretKey, the deprecation warnings are correctly placed at all CryptoKey consumption points. The main gap is the truncated diff hiding the prepareSecretKey changes, but the plan confirms it mirrors the pattern in prepareAsymmetricKey with minimal additions." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 8, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides significantly better analysis by following the plan's risk ordering: starting from the KeyObject entry point, then examining prepareAsymmetricKey (identified as high risk due to 4 callers), and finally prepareSecretKey. This traversal reveals the complete deprecation coverage surface -- the flow-guided review explicitly traces which callers (preparePrivateKey, preparePublicOrPrivateKey, createPublicKey, createPrivateKey) are covered by the DEP0203 warning in prepareAsymmetricKey, and cross-references the documentation changes against code paths. It also identifies the potential double-warning scenario for non-extractable keys. The baseline review treats each change in isolation, missing the caller relationships that determine whether all crypto operations are covered by the deprecation. The flow-guided review's awareness of the dependency graph makes it far more effective at verifying completeness of a deprecation that must cover multiple entry points." + } +} \ No newline at end of file diff --git a/evals/open-telemetry__opentelemetry-python__4974.json b/evals/open-telemetry__opentelemetry-python__4974.json new file mode 100644 index 0000000..65e8476 --- /dev/null +++ b/evals/open-telemetry__opentelemetry-python__4974.json @@ -0,0 +1,101 @@ +{ + "pr": { + "url": "https://github.com/open-telemetry/opentelemetry-python/pull/4974", + "owner": "open-telemetry", + "repo": "opentelemetry-python", + "number": 4974, + "title": "Restrict push CI to main branch", + "files_changed": 8 + }, + "baseline_review": { + "comments": [ + { + "file": ".github/workflows/contrib.yml", + "line": 5, + "severity": "positive", + "comment": "Switching from a branches-ignore blocklist to a branches allowlist (main only) is the safer approach. An allowlist is inherently more restrictive and prevents new branches from accidentally triggering push CI." + }, + { + "file": ".github/workflows/contrib.yml", + "line": 5, + "severity": "minor", + "comment": "The previous config explicitly excluded release/* and otelbot/* branches, implying push CI ran on all other branches. With this change, push CI is restricted to main only. If any release automation or bot workflows depend on push-triggered CI on their respective branches, those will silently stop running. Confirm with maintainers that release/* pushes do not require these CI checks." + }, + { + "file": ".github/workflows/templates/test.yml.j2", + "line": 8, + "severity": "positive", + "comment": "Good that the Jinja2 templates under templates/ are updated alongside the generated workflow files. This prevents drift when workflows are regenerated from templates." + }, + { + "file": ".github/workflows/test_0.yml", + "line": 8, + "severity": "nit", + "comment": "All 8 files (5 generated workflows + 3 templates) apply the identical change consistently. No files appear to be missed." + }, + { + "file": ".github/workflows/lint_0.yml", + "line": 8, + "severity": "nit", + "comment": "The pull_request trigger is left unchanged across all workflows, which is correct -- PR-based CI should continue to run on all branches regardless of this change." + } + ], + "summary": "Straightforward CI configuration change converting push triggers from a blocklist (branches-ignore) to an allowlist (branches: main). The change is applied consistently across all 8 workflow files and templates, and the PR trigger is correctly left untouched." + }, + "flow_guided_review": { + "comments": [ + { + "file": ".github/workflows/contrib.yml", + "line": 5, + "severity": "positive", + "comment": "The review plan is empty (no code flows, no dependencies), which is expected for a CI-only configuration change. The allowlist approach is the correct pattern for restricting push CI to a single branch." + }, + { + "file": ".github/workflows/templates/lint.yml.j2", + "line": 8, + "severity": "positive", + "comment": "Templates and generated files are updated in lockstep. Since the plan shows zero code dependencies, the only consistency concern is template-to-generated-file alignment, which is satisfied here." + }, + { + "file": ".github/workflows/test_0.yml", + "line": 8, + "severity": "minor", + "comment": "The implicit risk not captured by the empty plan: the old branches-ignore for release/* and otelbot/* branches suggests those branches previously ran push CI. If release branch pushes trigger validation workflows (e.g., publish checks, changelog validation), those will now be skipped. The linked issue #4971 should document whether this is intentional." + }, + { + "file": ".github/workflows/misc_0.yml", + "line": 8, + "severity": "nit", + "comment": "All workflow files are consistently updated. With no code-level dependencies in the plan, the only verification needed is that the YAML syntax is correct and the semantics match intent -- both are satisfied." + }, + { + "file": ".github/workflows/templates/misc.yml.j2", + "line": 8, + "severity": "nit", + "comment": "The three template files (test, lint, misc) all follow the same pattern. If new templates are added in the future, they should follow this same branches: main pattern for push triggers." + } + ], + "summary": "CI-only change with an empty flow plan (no code dependencies or runtime impact). The change correctly restricts push-triggered CI to main only. The primary risk is whether release branch automation relied on push-triggered CI, which should be confirmed via the linked issue #4971." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 6, + "efficiency": 8, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 6.8 + }, + "reasoning": "For a CI-only change with an empty flow plan, both reviews arrive at nearly identical conclusions. Both correctly identify the release branch risk as the key concern. The flow-guided review gains a small edge by explicitly acknowledging the empty plan and contextualizing the review accordingly, and by referencing the linked issue for actionability. However, the empty plan provides minimal structural advantage for this type of change, so the difference is marginal.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T18:30:00.000000+00:00" +} diff --git a/evals/openai__openai-node__1767.json b/evals/openai__openai-node__1767.json new file mode 100644 index 0000000..f80ec24 --- /dev/null +++ b/evals/openai__openai-node__1767.json @@ -0,0 +1,145 @@ +{ + "pr": { + "url": "https://github.com/openai/openai-node/pull/1767", + "owner": "openai", + "repo": "openai-node", + "number": 1767, + "title": "release: 6.29.0", + "files_changed": 12, + "additions": 118, + "deletions": 118, + "language": "TypeScript" + }, + "timestamp": "2026-03-30T18:00:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": "src/resources/audio/speech.ts", + "line": 56, + "severity": "minor", + "comment": "The voice type changed from `(string & {})` to plain `string`. The `(string & {})` idiom in TypeScript preserves autocomplete for the literal union members while still accepting arbitrary strings; replacing it with bare `string` will cause TypeScript to widen the union so that IDE autocomplete no longer suggests the built-in voice names like 'alloy', 'ash', etc." + }, + { + "file": "src/resources/audio/speech.ts", + "line": 97, + "severity": "positive", + "comment": "The new `SpeechCreateParams.ID` interface with `{ id: string }` is a clean way to support custom voice references. The JSDoc example (`voice_1234`) gives users a clear usage pattern." + }, + { + "file": "src/resources/chat/completions/completions.ts", + "line": 497, + "severity": "minor", + "comment": "Same `(string & {})` to `string` change here in `ChatCompletionAudioParam.voice`. This will equally degrade autocomplete for chat completion audio voice selection." + }, + { + "file": "src/resources/chat/completions/completions.ts", + "line": 510, + "severity": "positive", + "comment": "Custom voice ID interface added consistently with the speech.ts pattern. Good cross-API consistency." + }, + { + "file": "src/resources/realtime/realtime.ts", + "line": 1265, + "severity": "nit", + "comment": "Three identical `ID` interfaces are defined independently in `SpeechCreateParams`, `ChatCompletionAudioParam`, and the realtime namespace. These could be extracted into a shared `CustomVoiceRef` type to reduce duplication, though for auto-generated SDK code this is understandable." + }, + { + "file": "src/resources/audio/speech.ts", + "line": 19, + "severity": "nit", + "comment": "The doc example changed from `voice: 'ash'` to `voice: 'string'`. Using a literal `'string'` as the example value is confusing since it looks like a type annotation rather than a real voice name. A concrete example like `'alloy'` or `{ id: 'voice_1234' }` would be clearer." + } + ], + "summary": "Release 6.29.0 adds custom voice support via a new `{ id: string }` object pattern across speech, chat completions, and realtime APIs. The main concern is the `(string & {}) -> string` type change which will degrade TypeScript autocomplete for built-in voice names, and a confusing doc example that uses the literal string `'string'` as a voice value." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/resources/audio/speech.ts", + "line": 56, + "severity": "minor", + "comment": "The plan identifies the Speech class as a high-risk entry point. The core change here is the voice union type: `(string & {})` removed in favor of bare `string`. This is a TypeScript DX regression -- the branded intersection type preserves autocomplete suggestions for literal members ('alloy', 'ash', etc.) while still accepting arbitrary strings. With plain `string`, the literal members become invisible to autocomplete." + }, + { + "file": "src/resources/audio/speech.ts", + "line": 19, + "severity": "minor", + "comment": "The doc example now shows `voice: 'string'` which reads as a type placeholder rather than a real value. Since this is the entry point for the Speech API, the example should demonstrate actual usage -- either a built-in voice like `'alloy'` or the new custom voice pattern `{ id: 'voice_1234' }`." + }, + { + "file": "src/resources/audio/speech.ts", + "line": 97, + "severity": "positive", + "comment": "The `SpeechCreateParams.ID` sub-interface is a clean extension point for custom voices. The `{ id: string }` shape is easily discriminable from the string union at runtime, making it safe for consumers." + }, + { + "file": "src/resources/chat/completions/completions.ts", + "line": 497, + "severity": "minor", + "comment": "Although the plan only identifies speech.ts as the changed node, the same `(string & {}) -> string` pattern applies in chat completions. The plan misses this as a separate flow, but the risk is identical: autocomplete degradation for voice selection in chat audio params." + }, + { + "file": "src/resources/realtime/realtime.ts", + "line": 1265, + "severity": "nit", + "comment": "The diff is truncated for the realtime module, but the same pattern is applied. Three independent `ID` interfaces with identical shapes exist across the codebase. For generated SDK code this is acceptable, but worth noting for maintainability." + }, + { + "file": "CHANGELOG.md", + "line": 3, + "severity": "nit", + "comment": "The changelog entry describes the feature as 'custom voices' which accurately reflects the API changes. No issues here." + } + ], + "summary": "Following the plan's entry point at the Speech class, the key change is adding custom voice support via `{ id: string }` objects. The plan correctly flags this as high-risk due to the `(string & {}) -> string` type change that degrades TypeScript autocomplete. The plan under-represents the scope since identical changes in chat completions and realtime APIs carry the same risk but are not tracked as separate nodes." + }, + "review_plan": { + "stats": { + "totalSteps": 1, + "totalAdditions": 1, + "totalDeletions": 1, + "independentFlows": 1, + "filesChanged": 1 + }, + "steps": [ + { + "order": 1, + "nodeId": "src/resources/audio/speech.ts::Speech", + "name": "Speech", + "file": "src/resources/audio/speech.ts", + "lines": [11, 37], + "type": "class", + "changeType": "modified", + "additions": 1, + "deletions": 1, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + } + ], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 6, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.8 + }, + "reasoning": "Both reviews identify the same core issues: the (string & {}) autocomplete regression and the confusing doc example. The flow-guided review benefits from the plan's entry-point framing, which helps prioritize the Speech class changes and articulate the DX risk more clearly. It also correctly notes that the plan under-tracks the scope by only identifying one file when three files carry identical changes. The baseline review is competent but lacks structural awareness of how the changes flow through the API surface. Neither review is exceptional since this is a straightforward auto-generated release PR with limited complexity.", + "winner": "flow_guided" + } +} diff --git a/evals/openai__openai-node__1769.json b/evals/openai__openai-node__1769.json new file mode 100644 index 0000000..95afeb1 --- /dev/null +++ b/evals/openai__openai-node__1769.json @@ -0,0 +1,128 @@ +{ + "pr": { + "url": "https://github.com/openai/openai-node/pull/1769", + "owner": "openai", + "repo": "openai-node", + "number": 1769, + "title": "release: 6.30.0", + "files_changed": 13, + "additions": 34, + "deletions": 19, + "language": "TypeScript" + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "baseline_review": { + "comments": [ + { + "file": "bin/publish-npm", + "line": 5, + "severity": "major", + "comment": "The conditional authentication logic correctly falls back from NPM_TOKEN to OIDC (via ACTIONS_ID_TOKEN_REQUEST_TOKEN), but the error message on line 8 could be more specific about what 'id-token permission' means for users unfamiliar with GitHub OIDC. The branching is sound for the migration period." + }, + { + "file": "bin/publish-npm", + "line": 64, + "severity": "major", + "comment": "Installing npm@11.6.2 into a sibling ../oidc/ directory at publish time is fragile. This path is relative to the dist/ working directory (so it resolves to the repo root's oidc/ folder), and the directory is not gitignored or cleaned up. If this install fails or the path changes, publishing breaks silently. Consider using npx npm@11.6.2 instead, or at minimum add the oidc/ directory to .gitignore." + }, + { + "file": ".github/workflows/publish-npm.yml", + "line": 11, + "severity": "positive", + "comment": "Adding id-token: write permission is the correct approach for GitHub OIDC-based NPM publishing. This eliminates the need for long-lived NPM_TOKEN secrets, improving supply chain security." + }, + { + "file": ".github/workflows/create-releases.yml", + "line": 42, + "severity": "minor", + "comment": "NPM_TOKEN env var removed from the release workflow but the create-releases workflow does not add id-token: write permissions. If this workflow also calls publish-npm, it will rely on the OIDC fallback path but may lack the required permission grant." + }, + { + "file": "bin/check-release-environment", + "line": 9, + "severity": "minor", + "comment": "The NPM_TOKEN validation check is removed entirely. There is no replacement check for OIDC readiness (e.g., verifying the environment is a GitHub Actions runner with id-token permissions). The release doctor now only validates STAINLESS_API_KEY." + }, + { + "file": "src/resources/batches.ts", + "line": 294, + "severity": "minor", + "comment": "The /v1/videos endpoint is added to the batch endpoint options. This is a straightforward API surface addition matching the changelog entry." + } + ], + "summary": "Release 6.30.0 bundles version bumps, two minor API additions (videos batch endpoint, defer_loading field), and a significant infrastructure change migrating NPM publishing from static token-based to OIDC-based authentication. The OIDC migration is the highest-risk change; the npm@11.6.2 sidecar install pattern is unusual and could be more robust." + }, + "flow_guided_review": { + "comments": [ + { + "file": "bin/publish-npm", + "line": 5, + "severity": "major", + "comment": "With an empty flow plan, reviewing linearly. The publish script's conditional auth (NPM_TOKEN vs OIDC) is the most critical change in this release PR. The logic is correct but introduces a dual-path authentication model. During the transition period, both paths must be maintained and tested. If NPM_TOKEN is ever set accidentally alongside OIDC, the token path takes precedence, which may not be the desired behavior." + }, + { + "file": "bin/publish-npm", + "line": 64, + "severity": "major", + "comment": "Installing npm@11.6.2 into ../oidc/ is a workaround for the default npm version not supporting OIDC provenance. This creates an untracked directory that could accumulate across runs. Using npx --yes npm@11.6.2 publish would be cleaner and avoid the sidecar install entirely." + }, + { + "file": ".github/workflows/publish-npm.yml", + "line": 11, + "severity": "positive", + "comment": "The id-token: write permission combined with contents: read follows the principle of least privilege for OIDC-based publishing. This is a security improvement." + }, + { + "file": ".github/workflows/create-releases.yml", + "line": 42, + "severity": "minor", + "comment": "NPM_TOKEN removed from create-releases but no id-token permission added to this workflow. The publish-npm script will fall through to the error branch unless this workflow already inherits the permission or the runner provides ACTIONS_ID_TOKEN_REQUEST_TOKEN through other means." + }, + { + "file": "bin/check-release-environment", + "line": 9, + "severity": "minor", + "comment": "Removing the NPM_TOKEN check without adding an OIDC-readiness check means the release doctor can no longer catch authentication misconfiguration before a release attempt. Consider adding a check for either NPM_TOKEN or ACTIONS_ID_TOKEN_REQUEST_TOKEN." + }, + { + "file": "bin/publish-npm", + "line": 68, + "severity": "minor", + "comment": "Setting npm_config_registry explicitly via export ensures the OIDC npm binary targets the correct registry. This is a defensive measure since the sidecar npm install may have different default config. Good practice." + } + ], + "summary": "This release PR's main risk is the OIDC publishing migration. The empty flow plan provides no structural guidance, so the review follows the dependency chain: workflow permissions -> publish script auth -> npm binary -> registry publish. The dual-path auth model and sidecar npm install are the key areas warranting attention." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 6.6 + }, + "reasoning": "Both reviews correctly identify the OIDC migration as the primary risk. The flow-guided review provides slightly better risk identification by noting the NPM_TOKEN precedence behavior and the missing id-token permission in create-releases more explicitly. However, with an empty flow plan providing no structural advantage, the difference is marginal. The flow-guided review's attempt to trace the dependency chain (permissions -> auth -> binary -> publish) adds modest value.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T20:15:00.000000+00:00" +} diff --git a/evals/openai__openai-node__1798.json b/evals/openai__openai-node__1798.json new file mode 100644 index 0000000..910b674 --- /dev/null +++ b/evals/openai__openai-node__1798.json @@ -0,0 +1,122 @@ +{ + "pr": { + "url": "https://github.com/openai/openai-node/pull/1798", + "owner": "openai", + "repo": "openai-node", + "number": 1798, + "title": "[codex] Pin GitHub Actions workflow references", + "files_changed": 6, + "additions": 27, + "deletions": 27, + "language": "yaml" + }, + "timestamp": "2026-03-30T18:45:00.000000+00:00", + "baseline_review": { + "comments": [ + { + "file": ".github/workflows/ci.yml", + "line": 24, + "severity": "positive", + "comment": "Pinning actions/checkout from floating @v6 tag to immutable SHA @de0fac2e... with the version tag preserved as a trailing comment is the recommended supply chain security practice. This prevents tag-mutation attacks." + }, + { + "file": ".github/workflows/ci.yml", + "line": 93, + "severity": "minor", + "comment": "actions/checkout is pinned to two different SHAs across jobs: @de0fac2e... (v6) in the lint/test/examples jobs and @34e114... (v4) in the publish and ecosystem_tests jobs. Consider upgrading all references to v6 for consistency and to pick up any fixes." + }, + { + "file": ".github/workflows/create-releases.yml", + "line": 33, + "severity": "minor", + "comment": "actions/setup-node is pinned to @3235b87... (v3) here, while ci.yml pins to @49933ea... (v4). Using different major versions of the same action across workflows increases maintenance burden and may lead to subtle behavioral differences." + }, + { + "file": ".github/workflows/ci.yml", + "line": 103, + "severity": "positive", + "comment": "Third-party actions denoland/setup-deno and oven-sh/setup-bun are also pinned to SHAs. This is especially important for non-official actions where tag integrity is harder to verify." + }, + { + "file": ".github/workflows/create-releases.yml", + "line": 22, + "severity": "positive", + "comment": "stainless-api/trigger-release-please pinned to SHA. This is security-critical since this workflow has id-token write permissions for publishing." + }, + { + "file": ".github/workflows/detect-breaking-changes.yml", + "line": 43, + "severity": "nit", + "comment": "pnpm/action-setup is pinned here. Good. All external action references in this file are now SHA-locked." + } + ], + "summary": "Straightforward supply chain security improvement pinning all GitHub Actions to immutable SHA references with version tags preserved as comments. The only concern is mixed major versions of checkout (v4 vs v6) and setup-node (v3 vs v4) across different workflows, which could be unified in a follow-up." + }, + "flow_guided_review": { + "comments": [ + { + "file": ".github/workflows/ci.yml", + "line": 24, + "severity": "positive", + "comment": "The review plan is empty (no steps, no clusters, no dependencies), which is expected for a CI-only change with no code logic. All external action references across 6 workflow files are pinned from floating version tags to immutable commit SHAs, with version tags preserved as inline comments for readability." + }, + { + "file": ".github/workflows/ci.yml", + "line": 93, + "severity": "minor", + "comment": "actions/checkout is pinned to v4 SHA (34e114...) in the publish and ecosystem_tests jobs but v6 SHA (de0fac...) elsewhere. Since there are no data or dependency flows between jobs that would be affected, this is a consistency issue rather than a correctness risk, but it should be addressed to reduce drift." + }, + { + "file": ".github/workflows/create-releases.yml", + "line": 33, + "severity": "minor", + "comment": "setup-node v3 (3235b87...) is used here while ci.yml uses v4 (49933ea...). The create-releases workflow runs with elevated permissions (id-token: write), so keeping its toolchain current is especially important. Consider upgrading to v4." + }, + { + "file": ".github/workflows/detect-breaking-changes.yml", + "line": 18, + "severity": "positive", + "comment": "The detect-breaking-changes workflow correctly pins all actions including pnpm/action-setup. No runtime code is touched, so the only risk vector is SHA correctness, which is validated by the provided rg command in the PR description." + }, + { + "file": ".github/workflows/ci.yml", + "line": 108, + "severity": "nit", + "comment": "oven-sh/setup-bun pinned to SHA without specifying a bun version (unlike setup-deno which pins deno-version: v1.39.0). This is pre-existing behavior, not introduced by this PR, but worth noting for completeness." + } + ], + "summary": "Pure CI security hardening with an empty flow plan confirming no code logic changes. SHA pinning prevents tag-mutation supply chain attacks across all 6 workflow files. The only actionable finding is mixed major versions of checkout and setup-node that should be unified." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 7, + "efficiency": 8, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 8, + "overall": 6.4 + }, + "reasoning": "Both reviews correctly identify the core change (SHA pinning for supply chain security) and the same secondary concern (mixed action versions). The flow-guided review acknowledges the empty plan and uses it to correctly scope the risk assessment to CI-only, but for a change this mechanical the advantage is marginal. Neither review can deeply exercise flow awareness since there are no code dependencies or data flows. The flow-guided review adds slightly more context about permission implications (id-token write) and validation methodology. Overall, both are adequate but the flow plan provides minimal differential value for pure CI config changes.", + "winner": "tie" + } +} diff --git a/evals/oven-sh__bun__28617.json b/evals/oven-sh__bun__28617.json new file mode 100644 index 0000000..c5de9c4 --- /dev/null +++ b/evals/oven-sh__bun__28617.json @@ -0,0 +1,114 @@ +{ + "pr": "oven-sh/bun#28617", + "title": "Enable TCP_DEFER_ACCEPT for HTTP listeners on Linux", + "baseline_review": { + "comments": [ + { + "file": "packages/bun-usockets/src/bsd.c", + "line": 49, + "severity": "minor", + "comment": "Using `strcpy` to copy the filter name \"dataready\" into `afa.af_name` is safe here since the string is a short literal and `accept_filter_arg.af_name` is 16 bytes, but `strncpy(afa.af_name, \"dataready\", sizeof(afa.af_name) - 1)` would be more defensive and avoid static-analysis warnings about `strcpy` usage." + }, + { + "file": "packages/bun-usockets/src/bsd.c", + "line": 47, + "severity": "minor", + "comment": "The function returns 1 on success and 0 on failure, but the return value semantics are not documented. On the `#else` fallback path it returns 0, meaning platforms without TCP_DEFER_ACCEPT or SO_ACCEPTFILTER will silently report failure even though it is expected. Consider returning 0 explicitly as 'not supported' vs. 'failed', or document the convention." + }, + { + "file": "packages/bun-usockets/src/loop.c", + "line": 145, + "severity": "major", + "comment": "The recursive call to `us_internal_dispatch_ready_poll` for the newly accepted socket is a significant concern. If the readable handler itself triggers further accept-like operations or deeply nested dispatch chains, this could lead to unbounded stack growth. The comment mentions the POLL_TYPE_SOCKET handler tolerates EWOULDBLOCK, but there is no guard against the dispatched handler closing the listen socket or invalidating the accept loop state." + }, + { + "file": "packages/bun-usockets/src/loop.c", + "line": 145, + "severity": "minor", + "comment": "The check `!us_socket_is_closed(0, s)` uses a hardcoded `0` for the SSL parameter. If this code path is also reached for SSL contexts, the closed check may not be correct. Verify that the SSL parameter is always 0 at this call site or derive it from the listen socket context." + }, + { + "file": "packages/bun-usockets/src/internal/internal.h", + "line": 278, + "severity": "minor", + "comment": "Adding `unsigned char deferred_accept` after `unsigned int socket_ext_size` may introduce padding on some architectures due to alignment. This is a listen socket struct so there will be few instances, but for consistency with the `alignas` on the first member, consider grouping small fields or adding a comment about expected layout." + }, + { + "file": "packages/bun-usockets/src/context.c", + "line": 443, + "severity": "nit", + "comment": "Good: Unix domain sockets correctly initialize `deferred_accept = 0` without attempting to set the socket option, since TCP_DEFER_ACCEPT does not apply to AF_UNIX." + } + ], + "summary": "The PR introduces a well-scoped kernel optimization for HTTP listeners, following nginx's proven pattern of deferring accept until client data arrives. The main concern is the recursive dispatch in the accept loop which could interact poorly with handler-side state changes, and minor issues around defensive coding (strcpy, return value semantics, SSL parameter hardcoding)." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/bun-usockets/src/libusockets.h", + "line": 107, + "severity": "minor", + "comment": "The new flag `LIBUS_LISTEN_DEFER_ACCEPT = 64` is well-documented with its purpose and constraints. The comment correctly warns against using it for server-speaks-first protocols. However, there is no compile-time or runtime guard to prevent misuse -- a caller could set this flag on a WebSocket-only listener or a protocol where the server sends first (e.g., SMTP). Consider whether a runtime warning or assertion is warranted." + }, + { + "file": "packages/bun-usockets/src/bsd.c", + "line": 42, + "severity": "minor", + "comment": "The TCP_DEFER_ACCEPT timeout of 1 second matches nginx's convention and the comment explains the rationale well. However, note that on Linux the kernel rounds this up to the nearest retransmission timeout interval, so the effective timeout may be longer than 1 second. This is acceptable behavior but worth noting for debugging purposes." + }, + { + "file": "packages/bun-usockets/src/context.c", + "line": 403, + "severity": "minor", + "comment": "The flag is set after `us_internal_socket_context_link_listen_socket` links the listen socket, and `bsd_set_defer_accept` is called on the already-bound-and-listening fd. This ordering is correct -- the socket option must be set after `listen()` on some systems. The result is stored in `ls->deferred_accept` so the accept loop can branch on it. Clean integration." + }, + { + "file": "packages/bun-usockets/src/loop.c", + "line": 145, + "severity": "critical", + "comment": "This is the most impactful part of the change: after accepting a socket, the code recursively dispatches it as readable immediately. This skips an epoll round-trip but introduces a subtle risk. If the on_data handler for the accepted socket calls `us_listen_socket_close()` (e.g., server shutdown triggered by a request), the accept loop continues iterating with a potentially freed listen socket. The existing `us_socket_is_closed(0, &listen_socket->s)` check at the bottom of the loop may or may not catch this depending on when the close takes effect. A use-after-free could occur if the listen_socket memory is released during the recursive dispatch." + }, + { + "file": "packages/bun-usockets/src/loop.c", + "line": 145, + "severity": "major", + "comment": "The hardcoded SSL parameter `0` in `us_socket_is_closed(0, s)` is concerning in the context of the broader flow. The listen socket can serve both SSL and non-SSL contexts via `us_socket_context_listen`. If `Bun.serve()` uses TLS, accepted sockets from a deferred-accept listener would be SSL sockets, but `us_socket_is_closed(0, s)` checks the non-SSL path. This could result in dispatching readable on a socket that was closed during TLS handshake setup." + }, + { + "file": "packages/bun-usockets/src/internal/internal.h", + "line": 275, + "severity": "nit", + "comment": "The comment on the `deferred_accept` field clearly explains the invariant: accepted sockets from this listener are guaranteed to have data ready. This is good documentation that helps future maintainers understand why the accept loop branches on this flag." + }, + { + "file": "packages/bun-usockets/src/bsd.c", + "line": 49, + "severity": "minor", + "comment": "The FreeBSD SO_ACCEPTFILTER path uses `strcpy` into a fixed-size buffer. While `\"dataready\"` (9 chars + null) fits safely in the 16-byte `af_name` field, this will trigger warnings from static analysis tools. Using `strlcpy` (available on FreeBSD) or `strncpy` with explicit null termination would be more defensive." + } + ], + "summary": "The change follows a clean path from flag definition through socket option application to the critical accept-loop optimization, and the kernel-level optimization is well-motivated by the nginx precedent. The primary risk is the recursive dispatch in the accept loop which could lead to use-after-free if a request handler closes the listen socket, compounded by the hardcoded SSL=0 parameter that may not be correct for TLS listeners." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 9, + "actionability": 7, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review traces the change from the public API flag through the socket option application and into the accept loop, which reveals the critical use-after-free risk more clearly than the baseline. Both reviews identify the recursive dispatch concern, but the flow-guided review elevates it to critical severity by reasoning about the full lifecycle: a request handler triggered by the recursive dispatch could close the listen socket, and the accept loop would continue with freed memory. The flow-guided review also connects the hardcoded SSL=0 parameter to the broader TLS context flow, recognizing that Bun.serve() with TLS would route through this same accept path. The baseline review treats each file in isolation and misses the cross-cutting interaction between the dispatch, the handler, and the listen socket lifetime. The review plan was empty (no steps/clusters), which limited the flow-guided advantage, but the reviewer still applied flow-aware reasoning by manually tracing the data path through the change." + } +} \ No newline at end of file diff --git a/evals/oven-sh__bun__28633.json b/evals/oven-sh__bun__28633.json new file mode 100644 index 0000000..af97fe3 --- /dev/null +++ b/evals/oven-sh__bun__28633.json @@ -0,0 +1,114 @@ +{ + "pr": "oven-sh/bun#28633", + "title": "Fix per-query memory leaks in MySQL adapter", + "baseline_review": { + "comments": [ + { + "file": "src/sql/mysql/protocol/ColumnDefinition41.zig", + "line": 51, + "severity": "critical", + "comment": "Adding `this.name_or_index.deinit()` to the `deinit()` function is the correct fix for the leak where column name heap allocations were never freed during final cleanup. This was a clear omission since all other `Data` fields (catalog, schema, table, name, org_name) were already freed here." + }, + { + "file": "src/sql/mysql/protocol/ColumnDefinition41.zig", + "line": 80, + "severity": "critical", + "comment": "Calling `this.name_or_index.deinit()` before reassigning in `decodeInternal()` is essential to prevent leaking the old allocation when a prepared statement is re-executed and the server re-sends column definitions. However, this assumes `name_or_index` is always in a valid state before `decodeInternal()` is called -- if the field is uninitialized (e.g., garbage memory from `alloc` without zeroing), calling `deinit()` on it could be undefined behavior." + }, + { + "file": "src/sql/mysql/MySQLConnection.zig", + "line": 927, + "severity": "major", + "comment": "The zero-initialization loop `for (statement.columns) |*col| col.* = .{};` after `alloc` is critical to ensure `deinit()` calls in `decodeInternal()` are safe -- without it, the `name_or_index.deinit()` added in ColumnDefinition41.zig could operate on uninitialized memory. This is the companion fix that makes the `decodeInternal` change safe." + }, + { + "file": "src/sql/mysql/protocol/PreparedStatement.zig", + "line": 44, + "severity": "major", + "comment": "Freeing `this.params` after deinit-ing each element fixes a separate leak where the params slice itself was never freed. The `if (this.params.len > 0)` guard is correct since `bun.default_allocator.free` on a zero-length slice from `alloc(T, 0)` may not be valid. However, verify that after freeing, `this.params` is reset to an empty slice to prevent use-after-free if `deinit` is called twice." + }, + { + "file": "src/sql/mysql/MySQLStatement.zig", + "line": 86, + "severity": "major", + "comment": "Adding `field.name_or_index.deinit()` before setting it to `.duplicate` in `checkForDuplicateFields` prevents leaking the column name when a duplicate column is detected. This is a third leak site that would have been easy to miss -- the old heap-allocated name was simply overwritten with the `.duplicate` sentinel." + }, + { + "file": "test/regression/issue/28632.test.ts", + "line": 1, + "severity": "minor", + "comment": "The regression test creates a wide table (49 columns) and measures RSS growth over repeated queries, which is a reasonable approach to detect native memory leaks. However, RSS-based memory assertions can be flaky on CI due to OS memory management behavior (page reclamation timing, memory fragmentation). Consider documenting the expected RSS threshold or adding a generous tolerance margin." + } + ], + "summary": "This PR correctly identifies and fixes three distinct native memory leaks in the MySQL protocol handling: missing `name_or_index` cleanup in `deinit()`, missing cleanup before reassignment in `decodeInternal()`, and a missing `free` of the params slice. The zero-initialization of newly allocated column arrays is a necessary companion change to make the `deinit`-before-reassign pattern safe on first decode." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/sql/mysql/protocol/ColumnDefinition41.zig", + "line": 80, + "severity": "critical", + "comment": "The `deinit()` call before reassignment in `decodeInternal()` is the core fix for leak #2 (re-execution of prepared statements). Since the review plan is empty (no steps/clusters), analyzing the data flow manually: `decodeInternal` is called from `MySQLConnection.zig` during result set processing. Each call to `ColumnIdentifier.init(this.name)` allocates via `Data.toOwned()`, and without this `deinit()`, the previous allocation is orphaned. The fix is correct but relies on the caller ensuring the struct is zero-initialized before first decode -- otherwise `deinit()` operates on garbage." + }, + { + "file": "src/sql/mysql/MySQLConnection.zig", + "line": 927, + "severity": "critical", + "comment": "This zero-initialization (`col.* = .{}`) is the safety prerequisite for the `decodeInternal` fix. Without it, the newly added `this.name_or_index.deinit()` in `decodeInternal` would attempt to free uninitialized pointers on the first decode pass. There are two allocation sites (handlePreparedStatement at line 926 and handleResultSet at line 1058), and both correctly add the initialization loop. This demonstrates awareness that the fix spans multiple call paths." + }, + { + "file": "src/sql/mysql/protocol/ColumnDefinition41.zig", + "line": 51, + "severity": "major", + "comment": "Adding `name_or_index.deinit()` to the struct's `deinit()` method fixes leak #1. Looking at the data flow: `name_or_index` is populated via `ColumnIdentifier.init()` which calls `Data.toOwned()`, creating a heap allocation. All other `Data` fields in this struct (catalog, schema, table, org_table, name, org_name) were already freed in `deinit()`. The omission of `name_or_index` was likely because it has type `ColumnIdentifier` rather than `Data`, making the leak less obvious during initial implementation." + }, + { + "file": "src/sql/mysql/MySQLStatement.zig", + "line": 86, + "severity": "major", + "comment": "This fixes a leak in the duplicate-column-detection path. When `checkForDuplicateFields` finds a duplicate column name, it overwrites `name_or_index` with `.duplicate` without freeing the old value. This is a secondary flow that only triggers when the result set has columns with identical names -- a less common but valid scenario. The fix correctly frees before overwriting." + }, + { + "file": "src/sql/mysql/protocol/PreparedStatement.zig", + "line": 44, + "severity": "major", + "comment": "This fixes a separate leak (leak #3 per the PR description) where `Execute.deinit()` freed each param element but not the params slice itself. The `len > 0` guard is appropriate. However, there is a potential issue: after freeing `this.params`, the field still points to freed memory. If `deinit()` is ever called twice (e.g., error handling paths), this would be a use-after-free. Consider setting `this.params = &.{}` or `this.params.len = 0` after freeing." + }, + { + "file": "src/sql/mysql/MySQLConnection.zig", + "line": 1058, + "severity": "minor", + "comment": "In `handleResultSet`, the existing code at line 1055-1056 already frees the old `statement.columns` before reallocating, but the new zero-initialization loop was needed here too. Note that this path also has `statement.columns_received = 0` reset, confirming this is a re-decode scenario where columns may carry stale allocations." + }, + { + "file": "test/regression/issue/28632.test.ts", + "line": 1, + "severity": "minor", + "comment": "The regression test is well-designed -- a wide table (49 columns) amplifies the per-column leak signal, and measuring RSS growth over 500+ iterations is the correct approach for native (non-JS-heap) leaks. The test is Docker-gated via `isDockerEnabled()` and `describeWithContainer`, which is appropriate for an integration test requiring a real MySQL server." + } + ], + "summary": "This PR systematically fixes three native memory leaks in the MySQL adapter by ensuring all heap-allocated `name_or_index` values are freed before reassignment or during cleanup, and by freeing the params slice in `Execute.deinit()`. The zero-initialization of column arrays is a critical companion change that makes the deinit-before-first-decode pattern safe across both `handlePreparedStatement` and `handleResultSet` call paths." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 8, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 7.0 + }, + "flow_guided": { + "completeness": 9, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 8, + "efficiency": 8, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The review plan was empty (no steps, clusters, or dependencies), so the flow-guided review had to construct its own data-flow analysis. Despite this, the flow-guided review demonstrates stronger flow awareness by tracing how `decodeInternal` is called from two separate sites in `MySQLConnection.zig` and why both need zero-initialization, and by explaining the `Data.toOwned()` allocation chain that makes `name_or_index` a heap allocation despite its `ColumnIdentifier` type. The baseline review correctly identifies all fix sites and raises valid concerns (use-after-free on double-deinit, RSS flakiness in tests) but treats each change more in isolation. The flow-guided review connects the zero-initialization in `MySQLConnection.zig` to the safety of the `deinit()` call in `decodeInternal()` as a dependency, which is the key architectural insight. Both reviews catch the potential use-after-free in `PreparedStatement.deinit()`. The margin is moderate because the empty plan limited the flow-guided review's advantage." + } +} \ No newline at end of file diff --git a/evals/oven-sh__bun__28651.json b/evals/oven-sh__bun__28651.json new file mode 100644 index 0000000..7ab36fa --- /dev/null +++ b/evals/oven-sh__bun__28651.json @@ -0,0 +1,108 @@ +{ + "pr": "oven-sh/bun#28651", + "title": "Include async frames in Error.captureStackTrace", + "baseline_review": { + "comments": [ + { + "file": "src/bun.js/bindings/ErrorStackTrace.cpp", + "line": 158, + "severity": "critical", + "comment": "The diff is truncated after the comment block explaining the new strategy. The actual implementation of the rewritten getFramesForCaller -- the call to Interpreter::getStackTrace, the post-filtering logic, the caller matching by cell identity and name, and the stackTraceLimit enforcement -- is not visible. This makes it impossible to verify the correctness of the core behavioral change." + }, + { + "file": "src/bun.js/bindings/ErrorStackTrace.cpp", + "line": 158, + "severity": "major", + "comment": "UNUSED_PARAM(callFrame) marks the callFrame parameter as intentionally unused, but the comment says framesToSkip=1 is passed to Interpreter::getStackTrace. If getStackTrace is called without the callFrame, it must obtain the current call frame some other way (e.g., from VM). Verify the frame collection starts from the correct point without the explicit callFrame." + }, + { + "file": "src/bun.js/bindings/ErrorStackTrace.cpp", + "line": 170, + "severity": "major", + "comment": "The comment states 'Collect without a limit' to avoid capping raw frames before post-filtering. This means the entire async+sync stack is collected into a temporary vector regardless of stackTraceLimit. For deeply recursive or long async chains, this could cause significant memory allocation. Consider whether a reasonable upper bound (e.g., 2x stackTraceLimit or a hard cap) would be prudent." + }, + { + "file": "src/bun.js/bindings/ErrorStackTrace.cpp", + "line": 115, + "severity": "minor", + "comment": "The new #include is correct for accessing Interpreter::getStackTrace. However, the old code's includes (StackVisitor.h, etc.) may now be partially unused if the hand-rolled StackVisitor::visit walk was the only consumer. Consider cleaning up unused includes." + }, + { + "file": "scripts/build/deps/webkit.ts", + "line": 6, + "severity": "minor", + "comment": "The WebKit dependency is bumped to a new commit hash. This is presumably required for the new Interpreter::getStackTrace behavior or API surface. The PR description should document what changed in the WebKit fork to support this, since it is a critical external dependency change." + } + ], + "summary": "This PR replaces a hand-rolled synchronous-only stack walk with delegation to Interpreter::getStackTrace to include async frames in Error.captureStackTrace, which is a meaningful correctness fix for V8 compatibility. However, the diff is truncated and the actual implementation of the new approach is not visible, making it impossible to verify the post-filtering logic, caller matching, or limit enforcement." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/bun.js/bindings/ErrorStackTrace.cpp", + "line": 158, + "severity": "critical", + "comment": "The implementation of getFramesForCaller (the core logic node) is truncated in the diff. According to the plan, the test functions innerAsync, middleAsync, and outerAsync all depend on this function producing correct async frames. Without seeing the actual Interpreter::getStackTrace call, the post-filter using isImplementationVisibilityPrivate, and the dual caller-matching strategy (cell identity first, then name via Zig::functionName), the entire correctness chain from implementation to tests cannot be validated." + }, + { + "file": "test/js/node/v8/capture-stack-trace.test.js", + "line": 840, + "severity": "major", + "comment": "The notInStack function (plan order 1, high risk as entry_point) adds test coverage but the test content is not visible in the truncated diff. This is the first entry point in the review plan and should verify that when the caller is not found in the synchronous portion of the stack, all frames are removed (matching V8 behavior as described in the PR)." + }, + { + "file": "test/js/node/v8/capture-stack-trace.test.js", + "line": 850, + "severity": "major", + "comment": "outerAsync (plan order 2, high risk entry_point) calls both innerAsync and middleAsync, forming the async call chain that exercises the new async frame inclusion. This is the primary test for the PR's stated goal. The flow plan correctly identifies this as high risk because it is the top-level entry point that validates the entire async frame capture path." + }, + { + "file": "test/js/node/v8/capture-stack-trace.test.js", + "line": 842, + "severity": "major", + "comment": "innerAsync (plan order 6, medium risk due to multiple callers) is called by both outerAsync and middleAsync. Since a resumed async function's frame callee is the generator's next function (a different cell), this function exercises the name-based fallback matching described in the PR. Verify the test asserts that async frames from innerAsync appear correctly regardless of which caller invokes it." + }, + { + "file": "test/js/node/v8/capture-stack-trace.test.js", + "line": 874, + "severity": "minor", + "comment": "recurse (plan order 3, high risk entry_point) calls target which calls captureStackTrace. This tests the synchronous recursive path, ensuring the rewrite did not regress non-async behavior. The plan's dependency chain (recurse -> target -> captureStackTrace) should be verified to confirm caller filtering still works correctly for sync-only stacks." + }, + { + "file": "src/bun.js/bindings/ErrorStackTrace.cpp", + "line": 170, + "severity": "major", + "comment": "Collecting the entire stack without any limit (as the comment explains) before post-filtering is necessary for correctness but creates unbounded memory usage. The plan's dependency flow shows that stackTraceLimit enforcement happens after both the isImplementationVisibilityPrivate filter and caller removal. A pathological case (e.g., deep recursion with stackTraceLimit=1) would allocate the full stack only to discard nearly all of it." + }, + { + "file": "scripts/build/deps/webkit.ts", + "line": 6, + "severity": "minor", + "comment": "The WebKit version bump is a prerequisite for the C++ changes. Since the plan focuses on the test file cluster, this external dependency change is outside the flow graph but critical -- if the new WebKit build does not include the required Interpreter::getStackTrace enhancements, the entire feature will fail at link time." + } + ], + "summary": "The flow-guided review traces the dependency chain from the three high-risk test entry points (notInStack, outerAsync, recurse) through internal helpers (middleAsync, target, innerAsync) down to the core getFramesForCaller rewrite, revealing that the entire validation chain depends on truncated implementation code. The plan correctly identifies innerAsync as medium risk due to multiple callers exercising different matching strategies (cell identity vs name-based), which is the key subtlety of this PR's async frame support." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 5, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 5, + "efficiency": 7, + "overall": 5.2 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 6, + "efficiency": 6, + "overall": 7.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides meaningfully better analysis by leveraging the plan's dependency chain to trace how the three test entry points (notInStack, outerAsync, recurse) exercise different aspects of the rewritten getFramesForCaller. It correctly identifies that innerAsync's medium risk (multiple callers) maps directly to the PR's most subtle behavior: name-based caller matching when cell identity fails for resumed async generators. The baseline review catches the truncation issue and the unbounded collection concern but treats the test file as opaque additions rather than understanding how each test function validates a specific aspect of the implementation. Both reviews are hampered by the truncated diff, but the flow-guided review uses the plan structure to reason about what the unseen code must do, making its comments more targeted and architecturally aware." + } +} diff --git a/evals/pallets__flask__5917.json b/evals/pallets__flask__5917.json new file mode 100644 index 0000000..977ca35 --- /dev/null +++ b/evals/pallets__flask__5917.json @@ -0,0 +1,102 @@ +{ + "pr": "pallets/flask#5917", + "title": "fix provide_automatic_options override", + "baseline_review": { + "comments": [ + { + "file": "src/flask/sansio/app.py", + "line": 639, + "severity": "major", + "comment": "The new logic moves `required_methods.add('OPTIONS')` outside the `if provide_automatic_options is None` block, so it now runs even when `provide_automatic_options=True` is passed explicitly as an argument. This is correct for the stated fix (enabling override), but it also means that if a user passes `provide_automatic_options=True` along with `methods=['OPTIONS']`, OPTIONS will be in both the user-supplied methods and the required_methods set. This is harmless due to set semantics, but the intent could be clearer." + }, + { + "file": "docs/config.rst", + "line": 448, + "severity": "minor", + "comment": "The version annotation is changed from `3.10` to `3.1`. This looks like a typo fix (3.10 was likely wrong since the current version is 3.x), but verify that `3.1` is the correct version where `PROVIDE_AUTOMATIC_OPTIONS` was actually added." + }, + { + "file": "tests/test_basic.py", + "line": 106, + "severity": "minor", + "comment": "The test `test_provide_automatic_options_attr_enable` uses `client.options()` which sends OPTIONS to '/'. This relies on the FlaskClient default URL being '/'. While this works, it is less explicit than the old `client.open('/', method='OPTIONS')` pattern used elsewhere." + }, + { + "file": "tests/test_basic.py", + "line": 130, + "severity": "minor", + "comment": "The assertion `assert rv.allow == {'GET', 'HEAD', 'OPTIONS'}` checks the Allow header as a set. This is a good improvement over the old `sorted(rv.allow)` list comparison, but confirm that `rv.allow` actually returns a set type in Werkzeug, not a `HeaderSet` or other type that might not compare equal with a plain set." + }, + { + "file": "tests/test_basic.py", + "line": 68, + "severity": "nit", + "comment": "The import `from flask.testing import FlaskClient` is added at the top of the file for type annotations. This is fine but the existing tests in this file use `client` fixture without type annotations. The new tests are more explicit which is good, but the inconsistency with existing tests in the same file is notable." + } + ], + "summary": "The core logic change in `add_url_rule` correctly fixes the bug where `provide_automatic_options=True` could not override a disabled config default, by restructuring the conditional to always add OPTIONS to required_methods when the flag is truthy. The test refactoring is thorough, splitting monolithic test functions into focused single-behavior tests, though the docs version fix should be verified." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/flask/sansio/app.py", + "line": 632, + "severity": "major", + "comment": "Step 8 (App.add_url_rule, entry point, high risk): The restructured logic has three paths: (1) explicit argument passed -> use it directly, (2) view_func attribute set -> use it, (3) neither set -> derive from config and methods. The key behavioral change is that `if provide_automatic_options:` (line 639) now runs unconditionally after resolution, meaning even when the view_func attribute sets it to True, OPTIONS will be added to required_methods. Previously, only the config-derived path added OPTIONS. This is the correct fix for issue #5916 but subtly changes behavior: a view_func with `provide_automatic_options=True` that also explicitly lists OPTIONS in its methods will now have OPTIONS in both `methods` and `required_methods`. This is safe due to set union but worth noting." + }, + { + "file": "tests/test_basic.py", + "line": 117, + "severity": "major", + "comment": "Step 10 (test_provide_automatic_options_attr_enable, high risk entry point): This test is the critical new test that validates the fix for #5916 -- it sets config PROVIDE_AUTOMATIC_OPTIONS=False then enables via attribute. The assertion `rv.allow == {'GET', 'HEAD', 'OPTIONS'}` confirms the fix works. However, this test should also verify the response status_code is 200 (not just check Allow), since a 405 with an Allow header would also pass the allow check. Without the status assertion, a regression that returns 405 but includes OPTIONS in Allow would not be caught." + }, + { + "file": "tests/test_basic.py", + "line": 137, + "severity": "minor", + "comment": "Step 11 (test_provide_automatic_options_method_disable): This test verifies that when a route explicitly handles OPTIONS, the automatic handler does not interfere. The test checks for a custom header X-Test, which correctly proves the user's handler ran rather than the automatic one. This is a good focused test replacing the old monolithic test." + }, + { + "file": "tests/test_basic.py", + "line": 71, + "severity": "minor", + "comment": "Steps 10-13 (test_basic.py tests): The refactoring from two large test functions into four focused tests (attr_disable, attr_enable, arg_disable, method_disable) improves test isolation significantly. Each test targets exactly one behavior, making failures easier to diagnose. The old tests mixed multiple concerns and had comments like 'index2' suggesting copy-paste patterns." + }, + { + "file": "docs/config.rst", + "line": 448, + "severity": "minor", + "comment": "The version annotation fix from `3.10` to `3.1` is important -- `3.10` would imply a future version (Flask is at 3.x), and this was likely a typo where the dot was interpreted as a decimal rather than a version separator. This fix should ship with the PR to avoid misleading documentation." + }, + { + "file": "tests/test_basic.py", + "line": 96, + "severity": "minor", + "comment": "Step 11 (test_provide_automatic_options_arg_disable): Uses `@app.get('/', provide_automatic_options=False)` which tests the keyword argument path (line 632 in app.py where `provide_automatic_options is None` check occurs). Since the argument is explicitly False, it bypasses the None-check entirely, going straight to the `if provide_automatic_options:` check which evaluates to False. This correctly validates that the argument path still works for disabling." + } + ], + "summary": "The flow-guided review traces the three resolution paths for `provide_automatic_options` (explicit arg, view_func attribute, config default) and confirms the core fix correctly enables the override path that was previously broken. The new test `test_provide_automatic_options_attr_enable` is the critical validation for issue #5916, though it should also assert on status_code to fully guard against regressions where OPTIONS appears in Allow but the response is still 405." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 5, + "actionability": 6, + "efficiency": 7, + "overall": 5.6 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides substantially better analysis by tracing the three resolution paths (explicit argument, view_func attribute, config default) through the restructured conditional in add_url_rule. It identifies the critical new test for issue #5916 and flags a concrete gap -- the missing status_code assertion that could mask a regression. The baseline review correctly identifies the behavioral change but treats it more superficially, noting set semantics safety without connecting it to the specific bug being fixed. The flow-guided review's step-by-step traversal through the plan nodes yields actionable insights about how each test maps to a specific code path in the fix, while the baseline comments are more observational. Both reviews catch the docs version fix and test refactoring quality, but the flow-guided review demonstrates deeper understanding of the change's architecture." + } +} diff --git a/evals/pallets__flask__5928.json b/evals/pallets__flask__5928.json new file mode 100644 index 0000000..a22b76c --- /dev/null +++ b/evals/pallets__flask__5928.json @@ -0,0 +1,114 @@ +{ + "pr": "pallets/flask#5928", + "title": "all teardown callbacks are called despite errors", + "baseline_review": { + "comments": [ + { + "file": "src/flask/helpers.py", + "line": 655, + "severity": "major", + "comment": "The `_CollectErrors.__exit__` method (truncated in the diff) silences exceptions by returning `True`. This means any exception raised inside a `with collect_errors:` block is swallowed entirely until `raise_any` is called. If `raise_any` is never called (e.g., due to an early return or bug), errors will be silently lost. Consider adding a safety mechanism like a `__del__` warning if errors were collected but never raised." + }, + { + "file": "src/flask/helpers.py", + "line": 668, + "severity": "major", + "comment": "The `raise_any` method uses `ExceptionGroup` on Python 3.11+ but falls back to raising the first error on Python 3.10. This means on 3.10, if multiple teardown callbacks fail, only the first error is reported and subsequent errors are silently dropped. The PR description acknowledges this but it is a meaningful behavior difference across supported Python versions that should be documented more prominently, e.g., in the docstring of `_CollectErrors`." + }, + { + "file": "src/flask/ctx.py", + "line": 486, + "severity": "major", + "comment": "The `_cv_app.reset(self._cv_token)` and `self._cv_token = None` lines (around line 499-500) are no longer inside a `try/finally` block, yet they previously were in the `finally` clause to ensure the context variable is always reset. If `do_teardown_appcontext` raises (collected) errors and `raise_any` fires before these lines, the context variable would not be reset. However, looking more carefully, these lines execute unconditionally before `raise_any`, so they are safe. The removal of the try/finally is correct here since `collect_errors` now handles the continuation." + }, + { + "file": "src/flask/app.py", + "line": 1444, + "severity": "minor", + "comment": "In `do_teardown_request`, the signal `request_tearing_down.send` is wrapped in `with collect_errors:` but previously it was always called after the loop. This is correct behavior -- if the signal itself raises, it should be collected rather than aborting. Good change." + }, + { + "file": "CHANGES.rst", + "line": 17, + "severity": "nit", + "comment": "The changelog entry uses `:pr:`5928`` but the other entries in this section use `:issue:` references. This is intentional since it links to a PR rather than an issue, but it breaks the visual pattern of the surrounding entries." + }, + { + "file": "src/flask/ctx.py", + "line": 488, + "severity": "minor", + "comment": "Each step of the teardown in `AppContext.pop` is now wrapped in a separate `with collect_errors:` block. This means `do_teardown_request`, `_request.close()`, `do_teardown_appcontext`, and `appcontext_popped.send` each get their own error collection. Note that `do_teardown_request` and `do_teardown_appcontext` already have their own internal `_CollectErrors` instances, so errors from individual teardown callbacks will be wrapped in an `ExceptionGroup` from the inner collector, then that group gets collected by the outer collector, potentially producing nested `ExceptionGroup`s." + } + ], + "summary": "The PR introduces a clean `_CollectErrors` context manager to ensure all teardown callbacks run even when some fail, replacing the previous fail-fast behavior. The main concern is nested `ExceptionGroup` wrapping (inner collectors in `do_teardown_request`/`do_teardown_appcontext` wrapped by the outer collector in `AppContext.pop`) and the Python 3.10 fallback that silently drops all but the first error." + }, + "flow_guided_review": { + "comments": [ + { + "file": "tests/test_appctx.py", + "line": 216, + "severity": "major", + "comment": "Steps 1-6 (test_robust_teardown and its helpers): This is the primary test for the new behavior, exercising request teardown, app teardown, request signal, and app signal callbacks that all raise errors. The test validates that all callbacks run despite errors and checks the resulting ExceptionGroup structure. However, the test only verifies Python 3.11+ ExceptionGroup behavior. There should be a test or branch covering the Python 3.10 fallback path where only the first error is raised, to ensure that degraded path also works correctly." + }, + { + "file": "src/flask/helpers.py", + "line": 640, + "severity": "critical", + "comment": "Steps 13-14 (_CollectErrors class, core infrastructure): The `__exit__` method implementation is truncated in the diff, but this is the single most critical piece of the PR since all teardown error handling depends on it. The class must return `True` from `__exit__` to suppress exceptions and must correctly append `BaseException` subclasses (including `KeyboardInterrupt` and `SystemExit`). Catching `BaseException` rather than `Exception` means that `KeyboardInterrupt` and `SystemExit` during teardown will be deferred rather than immediately propagated, which could delay process termination. Consider whether `BaseException` subclasses that are not `Exception` should be re-raised immediately." + }, + { + "file": "src/flask/ctx.py", + "line": 486, + "severity": "major", + "comment": "Step 12 (AppContext.pop, high risk orchestrator): Following the flow graph, this method orchestrates four teardown phases: request teardown, request close, app context teardown, and appcontext_popped signal. Each phase has its own `with collect_errors:` block. The `do_teardown_request` and `do_teardown_appcontext` methods (steps 13-14 in the call chain) each create their own internal `_CollectErrors` and call `raise_any`, which means their ExceptionGroups bubble up as single exceptions to this outer collector. This creates nested ExceptionGroups: the outer `raise_any('Errors during context teardown')` wraps inner groups like `ExceptionGroup('Errors during request teardown', [...])`. Users catching specific exceptions with `except*` may need to handle this nesting." + }, + { + "file": "src/flask/app.py", + "line": 1434, + "severity": "minor", + "comment": "Steps 13 (do_teardown_request): The flow shows this is called by AppContext.pop (step 12). The method correctly wraps each callback invocation and the signal send in separate `with collect_errors:` blocks, ensuring that a failing callback does not prevent subsequent callbacks or the signal from executing. The blueprint ordering (chain of request.blueprints then None) is preserved, which is correct." + }, + { + "file": "src/flask/app.py", + "line": 1465, + "severity": "minor", + "comment": "Step 14 (do_teardown_appcontext): Mirrors the pattern of do_teardown_request. The reversed iteration order for teardown_appcontext_funcs is preserved, maintaining LIFO teardown semantics. The signal is also wrapped. This is consistent and correct." + }, + { + "file": "tests/test_basic.py", + "line": 1422, + "severity": "nit", + "comment": "Steps 7-11 (test_static_* test updates): These tests are updated to use context managers (`with app.test_request_context()`) instead of bare push/pop, ensuring proper resource cleanup. These are defensive improvements unrelated to the core feature but reduce the chance of ResourceWarning leaks. The changes are mechanical and correct." + }, + { + "file": "src/flask/ctx.py", + "line": 499, + "severity": "major", + "comment": "Step 12 continued: The `_cv_app.reset(self._cv_token)` and `self._cv_token = None` lines are now outside any error-collection block. Previously they were in a `finally` clause guaranteeing execution. In the new code, if `do_teardown_appcontext` raises an ExceptionGroup via `raise_any`, this line would NOT execute because `raise_any` raises immediately and the code above is sequential, not wrapped. Wait -- re-reading: `do_teardown_appcontext` is wrapped in `with collect_errors:`, so its raised ExceptionGroup is caught by the outer collector. The reset lines then execute normally before the outer `raise_any`. This is correct but subtle -- the correctness depends on understanding that the inner `raise_any` is caught by the outer `collect_errors`." + } + ], + "summary": "The flow-guided review reveals that the nested `_CollectErrors` pattern (inner collectors in `do_teardown_request`/`do_teardown_appcontext` wrapped by the outer collector in `AppContext.pop`) produces nested ExceptionGroups, and the truncated `__exit__` method likely catches `BaseException`, which would defer `KeyboardInterrupt`/`SystemExit` during teardown rather than propagating immediately. The test coverage focuses on the happy path of ExceptionGroup on 3.11+ but lacks verification of the 3.10 fallback behavior." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 5, + "efficiency": 6, + "overall": 5.4 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identifies two critical architectural insights that the baseline misses or underexplores. First, by following the call chain from AppContext.pop through do_teardown_request/do_teardown_appcontext, it clearly explains the nested ExceptionGroup structure and why the inner raise_any calls are safely caught by the outer _CollectErrors -- a subtlety the baseline notices but does not trace through the execution flow. Second, the flow-guided review raises the important concern that catching BaseException in __exit__ would defer KeyboardInterrupt/SystemExit, which the baseline does not address. The flow-guided review also correctly identifies the missing Python 3.10 test coverage by understanding that test_robust_teardown only exercises the ExceptionGroup path. The baseline review's third comment initially raises a false alarm about context variable reset safety before self-correcting, which wastes reviewer attention. Overall, the plan's ordering of test entry points first, then core infrastructure, then orchestration code, enabled the flow-guided review to build a coherent narrative of how errors flow through the system." + } +} \ No newline at end of file diff --git a/evals/pallets__flask__5945.json b/evals/pallets__flask__5945.json new file mode 100644 index 0000000..d4eb703 --- /dev/null +++ b/evals/pallets__flask__5945.json @@ -0,0 +1,114 @@ +{ + "pr": "pallets/flask#5945", + "title": "add zizmor to scan workflows", + "baseline_review": { + "comments": [ + { + "file": ".github/workflows/tests.yaml", + "line": 137, + "severity": "critical", + "comment": "The tox invocation changed from `uv run --locked tox run -e ${{ matrix.tox || format('py{0}', matrix.python) }}` to `uv run --locked tox run` with a `TOX_ENV` environment variable, but the `-e` flag that actually selects the tox environment was removed. Tox does not automatically read a `TOX_ENV` environment variable to determine which environment to run; it uses `TOX_OVERRIDE` or the `-e` flag. This will cause tox to run its default environment(s) instead of the matrix-selected one, effectively breaking the matrix test strategy. The fix should be `-e ${TOX_ENV}` or the standard tox env var `TOXENV`." + }, + { + "file": ".github/workflows/publish.yaml", + "line": 85, + "severity": "minor", + "comment": "The concurrency group uses `${{ github.event.push.ref }}` but this workflow only triggers on tag pushes. The `github.event.push.ref` context may not always be populated the same way as `github.ref_name`. Consider using `${{ github.ref_name }}` (or the env var equivalent `${GITHUB_REF_NAME}`) for consistency and reliability in the concurrency group." + }, + { + "file": ".github/workflows/publish.yaml", + "line": 96, + "severity": "minor", + "comment": "Changing `enable-cache: true` to `enable-cache: false` for uv in the build job removes caching. The PR description does not mention this change, and it seems unrelated to the zizmor security hardening. If intentional (e.g., to ensure reproducible builds for publishing), a comment explaining why would be helpful." + }, + { + "file": ".github/workflows/pre-commit.yaml", + "line": 72, + "severity": "minor", + "comment": "The removal of the `pre-commit-ci/lite-action` step means pre-commit CI will no longer auto-fix and push corrections on PRs. This is a behavioral change beyond security hardening. Confirm this is intentional and not an accidental side effect of the zizmor audit." + }, + { + "file": ".github/workflows/lock.yaml", + "line": 33, + "severity": "nit", + "comment": "Moving permissions from workflow-level to job-level is a good security practice (principle of least privilege). The `cancel-in-progress: true` addition for the lock workflow is fine since concurrent lock runs are redundant." + }, + { + "file": ".github/workflows/zizmor.yaml", + "line": 158, + "severity": "minor", + "comment": "The `paths` filter `**/*.yaml?` uses a shell glob where `?` matches a single character, so it matches `.yaml` and `.yml` files. However, it would also match `.yamla`, `.yamlb`, etc. A more precise filter would be `['**/*.yml', '**/*.yaml']`. This is a minor correctness concern." + } + ], + "summary": "The PR applies zizmor-recommended security hardening across all GitHub Actions workflows, including empty top-level permissions, persist-credentials: false, concurrency groups, and env var substitutions. The most significant concern is the tox invocation change in tests.yaml which appears to drop the environment selector flag, potentially breaking the CI test matrix." + }, + "flow_guided_review": { + "comments": [ + { + "file": ".github/workflows/tests.yaml", + "line": 137, + "severity": "critical", + "comment": "The tox command was changed from `-e ${{ matrix.tox || format('py{0}', matrix.python) }}` to just `uv run --locked tox run` with a `TOX_ENV` env var set but never consumed. Tox does not recognize `TOX_ENV` as a built-in variable -- the correct variable is `TOXENV` (no underscore). As written, every matrix entry will run tox's default environment instead of the matrix-specified one, silently breaking the entire test matrix. This must use either `-e ${TOX_ENV}` in the run command, or rename the env var to `TOXENV` which tox natively respects." + }, + { + "file": ".github/workflows/publish.yaml", + "line": 104, + "severity": "minor", + "comment": "Replacing `${{ github.repository }}` and `${{ github.ref_name }}` with `${GITHUB_REPOSITORY}` and `${GITHUB_REF_NAME}` is a valid zizmor recommendation to avoid template injection in `run:` steps. However, this only works because the `run` step executes in bash where these are default environment variables. This is correct but worth noting for maintainability -- future contributors may not realize these are GitHub-provided env vars rather than explicitly defined ones." + }, + { + "file": ".github/workflows/pre-commit.yaml", + "line": 72, + "severity": "minor", + "comment": "Removing the `pre-commit-ci/lite-action` step eliminates the auto-fix-and-push behavior for PRs. This may have been flagged by zizmor because the action needs write access to push commits, which conflicts with the new `permissions: {}` and `persist-credentials: false` settings. If so, this is a deliberate trade-off: better security at the cost of developer convenience. The PR description should note this behavioral change." + }, + { + "file": ".github/workflows/publish.yaml", + "line": 85, + "severity": "minor", + "comment": "The concurrency group `publish-${{ github.event.push.ref }}` uses a template expression in a non-`run` context, which is fine from a security perspective (template injection only matters in `run:` and similar shell-evaluated fields). However, `github.event.push.ref` returns the full ref like `refs/tags/2.0.0` while `github.ref_name` returns just `2.0.0`. Either works for concurrency grouping but the inconsistency with the `run` step (which uses `GITHUB_REF_NAME`) may confuse maintainers." + }, + { + "file": ".github/workflows/publish.yaml", + "line": 96, + "severity": "minor", + "comment": "Disabling uv cache (`enable-cache: false`) in the build job is not a zizmor finding -- it appears to be an unrelated change bundled into this PR. For a publish workflow, avoiding cache is arguably safer to ensure reproducible release artifacts, but this rationale should be stated. The prune-cache: false on the next line also becomes meaningless with caching disabled." + }, + { + "file": ".github/workflows/zizmor.yaml", + "line": 154, + "severity": "nit", + "comment": "The new zizmor workflow is well-structured: empty top-level permissions, persist-credentials: false, concurrency group, and pinned action SHAs. The `paths` filter ensures it only runs when workflow files change, which is efficient. The `advanced-security: false` setting means results go to annotations rather than GitHub's Security tab, which is a reasonable default for a public repo that may not have GHAS enabled." + } + ], + "summary": "The PR systematically applies zizmor security recommendations across all workflows, but the critical tox environment variable issue in tests.yaml would silently break the test matrix. The remaining changes are sound security hardening, with a couple of unrelated behavioral changes (cache disabling, pre-commit-ci removal) that should be documented." + }, + "review_plan": { + "stats": {"totalSteps": 0, "totalAdditions": 0, "totalDeletions": 0, "independentFlows": 0, "filesChanged": 0}, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 6.6 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 6, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews correctly identify the critical tox environment variable bug as the most important issue. The flow-guided review provides a more precise diagnosis (identifying that TOXENV without underscore is the tox-native variable), better explains the security rationale behind the pre-commit-ci removal (connecting it to the persist-credentials: false change), and more clearly distinguishes between template expressions that are security-relevant (in run: steps) versus safe (in non-shell contexts like concurrency groups). However, the advantage is smaller than usual because the review plan was empty (no steps, clusters, or dependencies), limiting the structural guidance available. The flow-guided review still benefits from its systematic cross-file analysis approach, but both reviews achieve similar coverage of the key issues." + } +} diff --git a/evals/pallets__werkzeug__3128.json b/evals/pallets__werkzeug__3128.json new file mode 100644 index 0000000..6645ce5 --- /dev/null +++ b/evals/pallets__werkzeug__3128.json @@ -0,0 +1,108 @@ +{ + "pr": "pallets/werkzeug#3128", + "title": "improve list quote handling", + "baseline_review": { + "comments": [ + { + "file": "src/werkzeug/http.py", + "line": 336, + "severity": "medium", + "comment": "The variable name `item` is reused as both the accumulator in the for-loop and as the iteration variable in the generator expression on the return line: `for item in (item.strip() for item in items)`. While this works in Python (the generator creates its own scope), it is confusing to read because `item` shadows itself across two levels of iteration. Consider renaming the loop accumulator to `current` or the generator variable to `i` to avoid confusion." + }, + { + "file": "src/werkzeug/http.py", + "line": 330, + "severity": "medium", + "comment": "Returning an empty list for an unclosed quoted string is a silent failure mode. The PR description acknowledges this design choice, but callers like `parse_dict_header`, `parse_accept_header`, and `parse_set_header` all depend on `parse_list_header` and will now silently receive empty results for malformed input that previously would have produced partial results. This is a behavior change that could mask bugs in upstream header construction. Consider at minimum logging a warning." + }, + { + "file": "src/werkzeug/http.py", + "line": 174, + "severity": "low", + "comment": "The `_unslash_re` regex is compiled at module scope with `re.A` (ASCII-only) flag. This is correct for HTTP headers which are ASCII, but the docstring for `unquote_header_value` does not mention this constraint. A brief note would help future maintainers understand why `re.A` was chosen." + }, + { + "file": "src/werkzeug/http.py", + "line": 312, + "severity": "low", + "comment": "The new character-by-character parser does not handle whitespace around commas beyond the final `.strip()` call. RFC 9110 allows optional whitespace (OWS) around list separators, and the strip at the end handles this, but tabs and other whitespace characters are also valid OWS. Python's `str.strip()` handles tabs, so this is fine in practice, but worth noting that the parser relies on the post-processing strip rather than handling OWS inline." + }, + { + "file": "tests/test_http.py", + "line": 104, + "severity": "medium", + "comment": "The test for partially quoted items `'a\"b,c\", \"d e\"'` expects `['a\"b,c\"', 'd e']`, meaning the parser treats a quote that does not start an item as a literal character and commas inside remain part of the item. This is a significant behavioral difference from the old `urllib.request.parse_http_list` implementation but there is no test for the inverse case where a partial quote appears at the end of an item (e.g., `'a, b\"c'`). Adding that edge case would strengthen coverage." + }, + { + "file": "src/werkzeug/http.py", + "line": 431, + "severity": "low", + "comment": "In `parse_dict_header`, replacing the inline quote-stripping with `unquote_header_value(value)` is a good consolidation. However, `unquote_header_value` also removes backslash escapes, which the old code did not do here. If any dict header values contained literal backslashes (e.g., file paths in Content-Disposition), this could change behavior. The risk is low since HTTP dict headers rarely contain backslashes, but it is a subtle semantic change." + } + ], + "summary": "This PR replaces the `urllib.request.parse_http_list` dependency with a hand-rolled character-by-character parser that better handles RFC 9110 quoting semantics, including partial quotes and unclosed strings. The main concerns are the silent empty-list return for unclosed quotes affecting downstream callers, variable shadowing in the return expression, and a few missing edge-case tests." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/werkzeug/http.py", + "line": 330, + "severity": "high", + "comment": "The review plan identifies `parse_list_header` as high-risk due to many callers (`parse_dict_header`, `parse_accept_header`, `parse_set_header`). Returning an empty list for unclosed quotes propagates through all of these callers. `parse_dict_header` (order 9) calls `parse_list_header` (order 10), so an unclosed quote in a dict header value like `'a=1, b=\"2, 3\", c=\"4, d=5'` now returns `{}` instead of a partial parse. This is confirmed by the new test, but `parse_accept_header` and `parse_set_header` are not tested for this new failure mode. Since these are entry points for `Accept`, `Vary`, and similar critical headers, a malformed header from a proxy or CDN could silently drop all parsed values." + }, + { + "file": "src/werkzeug/http.py", + "line": 177, + "severity": "medium", + "comment": "The plan identifies `unquote_header_value` as a medium-risk leaf node called by both `parse_list_header` and `parse_dict_header`. The change from explicit `replace('\\\\\\\\', '\\\\').replace('\\\\\"', '\"')` to `_unslash_re.sub(r'\\g<1>', value[1:-1])` broadens escape handling: previously only `\\\\` and `\\\"` were unescaped, now any `\\X` sequence has its backslash removed. Per RFC 9110 section 5.6.4, `quoted-pair` allows `\\` followed by any HTAB, SP, VCHAR, or obs-text character, so this is more correct. However, this is a behavioral change for any caller that previously relied on other backslash sequences being preserved (e.g., `\\n` in a quoted header value would now become `n`). The versionchanged note documents this for 3.2, which is appropriate." + }, + { + "file": "src/werkzeug/http.py", + "line": 336, + "severity": "medium", + "comment": "Following the dependency chain: `parse_list_header` calls `unquote_header_value` on each item after stripping. The return expression `unquote_header_value(item) for item in (item.strip() for item in items) if item` filters empty items after stripping but before unquoting. This means an item that is only whitespace (e.g., from `'a, , b'`) is correctly discarded. However, an item that is `'\"\"'` (empty quoted string) passes the truthiness check, gets unquoted to an empty string, and is included in the result. This inconsistency -- bare empty items are discarded but quoted empty items are kept -- may surprise callers." + }, + { + "file": "src/werkzeug/http.py", + "line": 431, + "severity": "medium", + "comment": "The plan shows `parse_dict_header` (order 9) depends on both `parse_list_header` and `unquote_header_value`. The old code in `parse_dict_header` stripped quotes inline without removing backslash escapes, but now delegates to `unquote_header_value` which does both. This means `parse_dict_header` values now get escape processing they did not before. Since `parse_dict_header` is called by `parse_cache_control_header`, cache-control directives with unusual characters could parse differently. The risk is low but this is a cross-cutting behavioral change affecting a separate caller chain." + }, + { + "file": "tests/test_http.py", + "line": 104, + "severity": "medium", + "comment": "The test cases for `test_list_header` cover partial quotes, empty items, and unclosed quotes, which aligns with the three behavioral changes described in the PR. However, given the plan's identification of `parse_accept_header` and `parse_set_header` as callers of `parse_list_header`, there are no tests verifying how these higher-level parsers handle unclosed quotes. An `Accept` header with an unclosed quote would now silently return an empty accept list, potentially causing 406 Not Acceptable responses in applications. At minimum, integration-level test cases for these callers would reduce risk." + }, + { + "file": "tests/test_http.py", + "line": 119, + "severity": "low", + "comment": "The new `test_dict_header` case `('a=1, b=\"2, 3\", c=\"4, d=5', {})` validates the unclosed-quote propagation from `parse_list_header` through `parse_dict_header`. This is good coverage for the dependency chain identified in the plan. The test implicitly confirms that `parse_dict_header` does not do its own quote tracking and relies entirely on `parse_list_header` for comma splitting within quoted values." + } + ], + "summary": "The flow-guided review reveals that the most significant risk is the cascading effect of the empty-list return for unclosed quotes: `parse_list_header` is called by `parse_dict_header`, `parse_accept_header`, and `parse_set_header`, and none of the latter two have tests for this new failure mode. The broadened backslash escape handling in `unquote_header_value` is more RFC-correct but subtly changes behavior for both `parse_list_header` and `parse_dict_header` callers, including `parse_cache_control_header`." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 3, + "risk_identification": 5, + "actionability": 6, + "efficiency": 7, + "overall": 5.4 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review substantially outperforms the baseline by leveraging the dependency graph to trace behavioral changes through the call chain. The baseline review correctly identifies individual concerns (variable shadowing, silent failure, missing tests) but treats each function in isolation. The flow-guided review connects the dots: it traces how the empty-list return in `parse_list_header` propagates through `parse_dict_header` (tested) but also through `parse_accept_header` and `parse_set_header` (untested), identifying a concrete risk that malformed Accept headers could silently produce empty results. It also traces the `unquote_header_value` broadened escape change through both direct callers and their transitive callers like `parse_cache_control_header`. The quoted-empty-string inconsistency (items `\"\"` surviving the filter while bare empty items are discarded) was only caught by following the parse-then-unquote pipeline. The baseline's variable-shadowing and docstring comments, while valid, are lower-impact style concerns that the flow-guided review correctly deprioritized in favor of behavioral and integration-level risks." + } +} \ No newline at end of file diff --git a/evals/pallets__werkzeug__3139.json b/evals/pallets__werkzeug__3139.json new file mode 100644 index 0000000..8be798c --- /dev/null +++ b/evals/pallets__werkzeug__3139.json @@ -0,0 +1,108 @@ +{ + "pr": "pallets/werkzeug#3139", + "title": "deprecate `HTTP_STATUS_CODES`", + "baseline_review": { + "comments": [ + { + "file": "src/werkzeug/exceptions.py", + "line": 93, + "severity": "medium", + "comment": "The fallback for unknown status codes changed from 'Unknown Error' to 'Unknown'. This is a user-visible behavioral change that affects string representations of HTTPException instances with no valid code. Downstream code that matches on the exact string 'Unknown Error' (e.g., in log parsers or test assertions) will break silently. The test update confirms this, but external consumers are not covered." + }, + { + "file": "src/werkzeug/http.py", + "line": 1531, + "severity": "medium", + "comment": "The module-level `__getattr__` deprecation mechanism is correct, but the `raise AttributeError(name)` at the end should include a more descriptive message (e.g., `f\"module 'werkzeug.http' has no attribute {name!r}\"`) to match the standard Python error format. As written, `AttributeError('HTTP_STATUS_CODES')` could be confusing if it appears in a traceback for a different attribute." + }, + { + "file": "src/werkzeug/sansio/response.py", + "line": 183, + "severity": "major", + "comment": "The status line format changed from `f\"{status_code} {HTTP_STATUS_CODES[status_code].upper()}\"` (always uppercase) to `f\"{status_code} {HTTPStatus(status_code).phrase}\"` (title case). This is a breaking change for any downstream code or middleware that compares status lines as exact strings (e.g., `response.status == '404 NOT FOUND'`). While the PR description acknowledges this shift, the changelog entry only mentions 'reason phrases use the more common title case' without flagging it as a breaking change." + }, + { + "file": "src/werkzeug/http.py", + "line": 79, + "severity": "minor", + "comment": "The internal dict was renamed from `HTTP_STATUS_CODES` to `_HTTP_STATUS_CODES` but still contains all the old entries. Since the deprecation directs users to `http.HTTPStatus`, consider whether this internal copy is still needed at all. If any internal code still references it (via the `__getattr__` shim), this is fine for now, but it could be removed entirely in the 3.3 release when the deprecation completes." + }, + { + "file": "src/werkzeug/sansio/response.py", + "line": 183, + "severity": "minor", + "comment": "The fallback for unknown status codes changed from 'UNKNOWN' (uppercase) to 'Unknown' (title case). This is consistent with the overall direction but means that unknown status codes will now produce status lines like '999 Unknown' instead of '999 UNKNOWN', which is another subtle behavioral change for consumers." + }, + { + "file": "src/werkzeug/exceptions.py", + "line": 91, + "severity": "minor", + "comment": "Using `self.code or 0` as the fallback when `self.code` is None means `HTTPStatus(0)` will raise ValueError, which is caught and returns 'Unknown'. This works but is slightly indirect -- a simple `if self.code is None: return 'Unknown'` guard before the try block would be clearer in intent." + } + ], + "summary": "This PR cleanly deprecates `HTTP_STATUS_CODES` in favor of Python's `http.HTTPStatus`, using a module-level `__getattr__` pattern that is idiomatic for deprecation. The most significant concern is that the case change from uppercase to title case in status reason phrases is a breaking behavioral change that should be more prominently documented as such." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/werkzeug/sansio/response.py", + "line": 183, + "severity": "major", + "comment": "This is the leaf node in the flow graph (order 11, called by Response.status setter) and the root cause of all test expectation changes. The switch from `HTTP_STATUS_CODES[status_code].upper()` to `HTTPStatus(status_code).phrase` changes every status line from uppercase ('NOT FOUND') to title case ('Not Found'). Since Response is called by HTTPException.get_response (order 10, multiple callers), this single change propagates through the entire exception hierarchy. Any WSGI middleware, reverse proxy, or downstream code that pattern-matches on uppercase reason phrases will break. This should be documented as a breaking change, not just a deprecation side-effect." + }, + { + "file": "src/werkzeug/exceptions.py", + "line": 90, + "severity": "medium", + "comment": "The HTTPException.name property (order 2, high risk entry point) changed from importing HTTP_STATUS_CODES to using HTTPStatus directly. The fallback changed from 'Unknown Error' to 'Unknown', dropping the word 'Error'. Since HTTPException.__str__ and __repr__ use this property (visible in test_exception_repr at order 1), this affects all string representations of exceptions with invalid codes. The test at line 136-137 confirms the new behavior, but the phrase 'Unknown Error' was more descriptive -- 'Unknown' alone could refer to an unknown status, unknown exception type, or unknown anything." + }, + { + "file": "src/werkzeug/http.py", + "line": 1531, + "severity": "medium", + "comment": "The __getattr__ deprecation shim (order 3, entry point) correctly emits a DeprecationWarning when HTTP_STATUS_CODES is accessed. However, it returns `_HTTP_STATUS_CODES` which still has title case values (matching the original dict, not the old .upper() behavior). This means code migrating from `HTTP_STATUS_CODES[404]` to `http.HTTPStatus(404).phrase` will see the same value ('Not Found'), but code that was doing `HTTP_STATUS_CODES[404].upper()` explicitly will need no change. The deprecation warning message should clarify that the replacement values are title case, not uppercase." + }, + { + "file": "tests/test_wrappers.py", + "line": 309, + "severity": "minor", + "comment": "test_response_set_status_code (order 4) and test_response_set_status (order 5) together cover the full range of _clean_status behavior: numeric codes, string codes, unknown codes, and custom reason phrases. The test updates correctly reflect the case change. However, the test at line 170 ('200 TEA POT' remains unchanged) shows that user-provided reason phrases are preserved as-is, which is important -- this confirms that only auto-generated phrases changed case, not user-supplied ones." + }, + { + "file": "src/werkzeug/sansio/response.py", + "line": 180, + "severity": "minor", + "comment": "The import of HTTP_STATUS_CODES was removed from the module-level imports (line 113 in the diff), which is correct -- it avoids triggering the deprecation warning on import. However, the module still imports HTTPStatus from the standard library at the top level (already present before this PR). The Response class (order 13, medium risk with multiple callers) now depends on http.HTTPStatus being available, which is fine since Python 3.5+, but worth noting that this removes werkzeug's ability to customize status phrases via the old mutable dict pattern." + }, + { + "file": "tests/test_wrappers.py", + "line": 1176, + "severity": "minor", + "comment": "test_malformed_204_response_has_no_content_length (order 6) changes '204 NO CONTENT' to '204 No Content'. This test verifies the WSGI response tuple, meaning the case change affects the actual HTTP response line sent to clients. HTTP/1.1 reason phrases are technically ignored by most clients, but this is still an observable wire-protocol change that could affect log-matching tools or monitoring systems that parse status lines." + } + ], + "summary": "The flow graph reveals that the case change originates in a single leaf method (_clean_status, order 11) but propagates through Response and HTTPException to affect all HTTP responses and exception representations. The most critical insight from the dependency analysis is that this is not merely a deprecation -- it is a wire-protocol behavioral change affecting every HTTP response that uses auto-generated reason phrases, which should be prominently documented as breaking." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review significantly outperforms the baseline by tracing the case change from its origin in _clean_status (leaf node) through the Response class and HTTPException hierarchy to understand the full blast radius. The baseline review correctly identifies the case change as a breaking concern but treats each file in isolation, missing the propagation chain. The flow-guided review leverages the dependency graph to show that a single method change in sansio/response.py cascades through multiple callers (Response.status setter, HTTPException.get_response, HTTPException.__str__/__repr__) to affect wire-protocol HTTP responses, exception string representations, and WSGI response tuples simultaneously. It also correctly identifies the loss of the mutable-dict customization pattern as a secondary consequence. The baseline review raises valid points about AttributeError message formatting and code clarity, but these are lower-impact concerns compared to the flow-guided review's structural insight about the breaking nature of the change." + } +} diff --git a/evals/pandas-dev__pandas__64901.json b/evals/pandas-dev__pandas__64901.json new file mode 100644 index 0000000..d543abc --- /dev/null +++ b/evals/pandas-dev__pandas__64901.json @@ -0,0 +1,131 @@ +{ + "pr": { + "url": "https://github.com/pandas-dev/pandas/pull/64901", + "owner": "pandas-dev", + "repo": "pandas", + "number": 64901, + "title": "TST: remove stale strict=False xfails", + "files_changed": 3, + "additions": 19, + "deletions": 79, + "language": "python" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "pandas/tests/frame/methods/test_nlargest.py", + "line": 147, + "severity": "minor", + "comment": "Removing the xfail block in test_nlargest_n_duplicate_index is correct if all parametrized cases now pass consistently. However, the original xfail referenced 'numpy>=1.25 with AVX instructions' as the root cause. If the underlying numpy unstable sort behavior was not actually fixed (just happens to pass on current CI hardware), these tests could become flaky on different platforms. The PR description says '128 affected tests pass locally' but local verification on one machine is not sufficient to confirm platform-independence for an AVX-related issue." + }, + { + "file": "pandas/tests/frame/methods/test_sort_values.py", + "line": 838, + "severity": "major", + "comment": "In test_sort_index_level_and_column_label, the xfail reason was shortened from the numpy AVX explanation to 'unstable sorting of duplicates, platform-dependent' but strict=False is retained. If the issue is truly resolved, the xfail should be removed entirely. If it is not resolved, the shortened reason loses important context about the root cause (numpy>=1.25, AVX). This is an inconsistency: the PR title says 'remove stale xfails' but this xfail is being kept and reworded rather than removed." + }, + { + "file": "pandas/tests/frame/methods/test_sort_values.py", + "line": 864, + "severity": "major", + "comment": "A new xfail with strict=False was added to test_sort_column_level_and_index_label for the 'df_idx0-inner-True' case. This contradicts the PR title 'remove stale strict=False xfails' -- rather than removing xfails, a new one is being introduced. The PR description does not explain why this test case needs an xfail. If the unconditional xfail that was removed (lines 881-890) was overbroad, the replacement should explain which specific parametrizations still fail." + }, + { + "file": "pandas/tests/frame/methods/test_sort_values.py", + "line": 881, + "severity": "minor", + "comment": "Removing the unconditional xfail that applied to all parametrizations of test_sort_column_level_and_index_label is good -- the original code applied the xfail after the assertion, which meant it was dead code that never prevented test execution. However, replacing it with a conditional xfail for 'df_idx0-inner-True' suggests at least one case still fails. The net effect is narrowing the xfail scope, not removing it." + }, + { + "file": "pandas/tests/indexes/test_setops.py", + "line": 74, + "severity": "minor", + "comment": "The PR description claims the xfail conditions in test_union_different_types are unreachable because 'index_flat2 is an alias for index_flat'. If that is true, removing dead code is correct. However, this claim should be verified -- if index_flat2 is always the same fixture instance as index_flat, then idx1.dtype == idx2.dtype always, and the cross-dtype conditions (kind=='i' vs kind=='b') are indeed unreachable. A reviewer should confirm this fixture aliasing in conftest." + }, + { + "file": "pandas/tests/indexes/test_setops.py", + "line": 95, + "severity": "minor", + "comment": "The second removed xfail block (PeriodDtype[B] warning not produced on all builds) was also strict=False, meaning it tolerated both pass and fail. If index_flat2 is truly an alias for index_flat, this code path may never have been hit. But if the aliasing claim is wrong, removing this xfail could cause test failures on builds where the PeriodDtype deprecation warning is not raised." + } + ], + "summary": "The PR is inconsistent with its stated goal: while it removes several stale xfails, it also rewrites and adds new strict=False xfails in test_sort_values.py. The setops changes are safe if the fixture aliasing claim is correct, but the sorting test changes need clarification on why one xfail is preserved and narrowed rather than fully removed." + }, + "flow_guided_review": { + "comments": [ + { + "file": "pandas/tests/frame/methods/test_nlargest.py", + "line": 147, + "severity": "minor", + "comment": "ENTRY POINT (step 2, high risk): test_nlargest_n_duplicate_index removes a conditional xfail that covered specific (order, n) combinations where numpy AVX unstable sorting caused non-deterministic results. The plan flags this as high-risk because it is an entry point with no downstream dependencies -- if the underlying numpy behavior regresses on different hardware, this test becomes flaky with no fallback. The `request` parameter was also removed from the signature, which is clean. CI should be verified on multiple architectures (x86 with/without AVX, ARM) before merging." + }, + { + "file": "pandas/tests/frame/methods/test_sort_values.py", + "line": 838, + "severity": "major", + "comment": "ENTRY POINT (step 5, high risk): test_sort_index_level_and_column_label retains its xfail but shortens the reason string. The plan identifies steps 4-6 as the TestSortValuesLevelAsStr class cluster. Analyzing the two methods together reveals an inconsistency: the original unconditional xfail in test_sort_column_level_and_index_label (step 6) was moved and narrowed to match the same 'df_idx0-inner-True' condition already in test_sort_index_level_and_column_label (step 5). This suggests both methods share the same known-failing parametrization, which is consistent -- but the PR should explain this refactoring explicitly rather than presenting it as 'removal'." + }, + { + "file": "pandas/tests/frame/methods/test_sort_values.py", + "line": 864, + "severity": "major", + "comment": "ENTRY POINT (step 6, high risk): The new xfail block added to test_sort_column_level_and_index_label mirrors the existing one in the sibling method (step 5). Tracing the flow: the old code applied xfail unconditionally after computing `result`, meaning every parametrization was marked. The new code applies it conditionally before computation, only for 'df_idx0-inner-True'. This is actually a correctness improvement -- the old xfail was dead code placed after tm.assert_frame_equal, so failures would propagate regardless. The new placement before the assertion properly activates the xfail marker." + }, + { + "file": "pandas/tests/indexes/test_setops.py", + "line": 74, + "severity": "minor", + "comment": "ENTRY POINT (step 3, high risk): test_union_different_types removes two xfail blocks and the `request` parameter. The plan shows this as an independent flow with no dependencies on the sorting tests. The PR claims index_flat2 aliases index_flat, making cross-dtype conditions unreachable. If true, the GH#44000 xfail (bool/int union raising ValueError) was never triggered. A reviewer should verify via conftest.py that index_flat2 is indeed conftest-level aliased to index_flat and not just coincidentally producing the same values." + }, + { + "file": "pandas/tests/indexes/test_setops.py", + "line": 95, + "severity": "minor", + "comment": "INDEPENDENT FLOW: The PeriodDtype[B] xfail removal is part of the same function but covers a different condition (FutureWarning for deprecated Business frequency). Since the warn variable and tm.assert_produces_warning context manager are still in place, if the warning IS produced the test still passes. The xfail only mattered when the warning was NOT produced (strict=False tolerating the AssertionError). If the fixture aliasing makes this branch unreachable, the removal is safe. But note: the surrounding PeriodDtype[B] deprecation check itself may also be dead code worth cleaning up." + }, + { + "file": "pandas/tests/frame/methods/test_sort_values.py", + "line": 881, + "severity": "minor", + "comment": "FLOW CONTEXT (steps 5-6 dependency): The removed unconditional xfail was placed between the result computation and the assertion. In pytest, request.applymarker() called after the test action but before the assertion still works, so this was not truly dead code -- but it marked ALL parametrizations as xfail rather than just the failing one. The refactoring to a conditional pre-assertion xfail is a net improvement in test precision, though it adds lines rather than removing them." + } + ], + "summary": "The flow-guided analysis reveals that the sorting test changes (steps 4-6) are actually a refactoring that narrows an overbroad unconditional xfail into a targeted conditional one matching the sibling method, not a simple removal. The setops change (step 3) is an independent flow where the safety depends entirely on confirming the index_flat2 fixture aliasing claim, which should be verified in conftest before merge." + }, + "judgment": { + "criteria": { + "completeness": { + "baseline": 7, + "flow_guided": 8, + "rationale": "Both reviews cover all three files and the key inconsistency between the PR title and actual changes. The flow-guided review additionally identifies that the PeriodDtype check itself may be dead code worth cleaning up and correctly analyzes the old xfail placement semantics." + }, + "flow_awareness": { + "baseline": 5, + "flow_guided": 8, + "rationale": "Baseline reviews each file change independently. Flow-guided review connects steps 5 and 6 as a TestSortValuesLevelAsStr cluster, recognizing that the two sibling methods share the same failing parametrization and that the change is a coherent refactoring. It also correctly identifies step 3 as an independent flow." + }, + "risk_identification": { + "baseline": 7, + "flow_guided": 8, + "rationale": "Baseline correctly flags the platform-dependence risk and fixture aliasing assumption. Flow-guided review adds the insight that the old unconditional xfail was misplaced (after assertion) and that the refactoring actually improves test precision, reframing the risk from 'adding new xfails' to 'narrowing existing ones'." + }, + "actionability": { + "baseline": 7, + "flow_guided": 7, + "rationale": "Both reviews suggest verifying the fixture aliasing in conftest and checking multi-platform CI. The flow-guided review adds the suggestion to clean up the PeriodDtype dead code, but both provide concrete next steps." + }, + "efficiency": { + "baseline": 7, + "flow_guided": 7, + "rationale": "Both reviews are focused and avoid tangential concerns. The flow-guided review is slightly more verbose but each comment adds analytical depth through the plan context." + } + }, + "overall": { + "baseline": 6.6, + "flow_guided": 7.6, + "winner": "flow_guided", + "rationale": "The flow-guided review provides a more accurate understanding of the sorting test changes by connecting steps 5 and 6 as a coordinated refactoring rather than independent edits. Its key insight -- that the old unconditional xfail was overbroad and the new conditional xfail is actually an improvement in test precision -- reframes the 'inconsistency' the baseline identified into a coherent design decision. The baseline review correctly flags surface-level concerns but mischaracterizes the sort_values changes as contradictory rather than intentionally narrowing." + } + } +} diff --git a/evals/pandas-dev__pandas__64912.json b/evals/pandas-dev__pandas__64912.json new file mode 100644 index 0000000..cafa955 --- /dev/null +++ b/evals/pandas-dev__pandas__64912.json @@ -0,0 +1,102 @@ +{ + "pr": "pandas-dev/pandas#64912", + "title": "REV: revert #64689 MultiIndex.loc datetime64-to-date conversion", + "baseline_review": { + "comments": [ + { + "file": "pandas/core/indexes/multi.py", + "line": 3448, + "severity": "major", + "comment": "The revert removes the GH#55969 fix entirely, which means the original bug (np.datetime64 key on a MultiIndex with datetime.date level returning incorrect results) is reintroduced. The PR description says 'closes #55969' but actually reopens it. The PR should either note that #55969 is being reopened, or provide an alternative fix that handles both cases." + }, + { + "file": "pandas/tests/indexing/multiindex/test_loc.py", + "line": 1015, + "severity": "major", + "comment": "The replacement test only covers datetime64 with second resolution ('s') in an object-dtype level. It does not cover the original GH#55969 scenario where the level contains datetime.date objects and the key is np.datetime64[D]. By removing the old test without preserving a variant of it, we lose regression coverage for the original bug." + }, + { + "file": "pandas/tests/indexing/multiindex/test_loc.py", + "line": 1022, + "severity": "minor", + "comment": "The new test uses np.datetime64('2023-01-01', 's') for both the index values and the lookup key. This means the hash match is trivially satisfied since both sides have identical dtype and value. A more thorough test would also check cross-resolution lookups (e.g., key with 'D' resolution against 's' values) to document the current behavior." + }, + { + "file": "pandas/tests/indexing/multiindex/test_loc.py", + "line": 1032, + "severity": "minor", + "comment": "The expected DataFrame uses a bare Index(['X']) without a name, which is correct since the third level of the MultiIndex was constructed without names. However, adding explicit level names to the MultiIndex construction would make the test more robust and self-documenting." + }, + { + "file": "doc/source/whatsnew/v3.1.0.rst", + "line": 223, + "severity": "minor", + "comment": "The whatsnew entry for the GH#55969 fix is removed, which is correct for a revert. However, there should be a new entry under the appropriate section noting the regression fix (that the datetime64-to-date conversion in _partial_tup_index caused hash mismatches for non-day resolution datetime64 in object-dtype levels)." + } + ], + "summary": "This PR correctly identifies that the GH#64689 fix introduced a regression for object-dtype MultiIndex levels holding non-day-resolution datetime64 values. However, the revert fully reopens GH#55969 without providing an alternative fix, and the replacement test only covers the regression case without preserving coverage for the original bug." + }, + "flow_guided_review": { + "comments": [ + { + "file": "pandas/tests/indexing/multiindex/test_loc.py", + "line": 1015, + "severity": "major", + "comment": "Step 1 (entry point, high risk): The new test_loc_np_datetime64_key_on_object_dt64_level replaces test_loc_datetime_date_multiindex_with_np_datetime64 but only tests one dimension of the problem -- datetime64[s] values in an object-dtype level looked up with a matching datetime64[s] key. The original test verified cross-type behavior (datetime.date in index, np.datetime64[D] as key). Removing this test entirely means GH#55969 has no regression coverage. At minimum, the old test should be kept and marked as xfail to document the known regression." + }, + { + "file": "pandas/core/indexes/multi.py", + "line": 3448, + "severity": "major", + "comment": "Step 2 (internal, _partial_tup_index): The removed code block performed a targeted conversion of np.datetime64[D] to datetime.date only when the level had object dtype and the datetime64 had day resolution. While this conversion caused hash mismatches for non-day resolution datetime64, the right fix might be to broaden the conversion logic rather than remove it entirely. For example, comparing by value rather than by hash, or normalizing both sides to a common type. The revert trades one regression for another." + }, + { + "file": "pandas/core/indexes/multi.py", + "line": 33, + "severity": "minor", + "comment": "Step 2 (import removal): The Timestamp import is removed since _partial_tup_index no longer needs it. This is correct cleanup. However, Timestamp is widely used in the pandas codebase, so if a future fix re-adds the conversion logic, this import will need to come back. Not a blocking issue." + }, + { + "file": "pandas/tests/indexing/multiindex/test_loc.py", + "line": 1022, + "severity": "minor", + "comment": "Step 1 continued: The test constructs a MultiIndex with [np.datetime64('2023-01-01', 's')] * 2 in an object-dtype Index, then looks up with np.datetime64('2023-01-01', 's'). Since the key exactly matches the stored values in both value and dtype, this is a straightforward hash-match scenario. A more revealing test would use np.datetime64('2023-01-01', 'D') as the key against 's'-resolution values to demonstrate the hash mismatch that motivated the revert." + }, + { + "file": "pandas/tests/indexing/multiindex/test_loc.py", + "line": 1, + "severity": "minor", + "comment": "The 'from datetime import date' removal is correct since no test in this file references datetime.date anymore. However, this also confirms that there is zero test coverage remaining for the datetime.date-in-MultiIndex-level scenario (GH#55969)." + }, + { + "file": "doc/source/whatsnew/v3.1.0.rst", + "line": 223, + "severity": "minor", + "comment": "Step 3 (leaf, MultiIndex class level): The whatsnew removal is correct for the revert, but a new whatsnew entry should document the regression that this revert fixes. Users who upgraded expecting GH#55969 to be fixed will want to know it was reverted and why." + } + ], + "summary": "Following the flow from the high-risk test entry point through _partial_tup_index and the MultiIndex class, this revert correctly fixes a regression where datetime64-to-date conversion caused hash mismatches for non-day-resolution datetime64 in object-dtype levels. However, it fully reopens GH#55969 without an alternative fix, and the replacement test only covers the trivial same-dtype-same-resolution case rather than demonstrating the cross-resolution mismatch that motivated the change." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides better structural understanding by tracing from the test entry point through _partial_tup_index to the MultiIndex class. It identifies the same core issues as the baseline (reopening GH#55969, insufficient test coverage) but adds depth: it explains why the test is trivial (same-dtype hash match), suggests keeping the old test as xfail to preserve regression documentation, and proposes that a broader normalization approach in _partial_tup_index might be preferable to a full revert. The baseline review correctly identifies the problems but treats the code changes more in isolation. The flow-guided review's step-ordered traversal surfaces the connection between the import removal, the conversion logic deletion, and the test replacement as parts of a single revert narrative." + } +} \ No newline at end of file diff --git a/evals/payloadcms__payload__16047.json b/evals/payloadcms__payload__16047.json new file mode 100644 index 0000000..b1a421c --- /dev/null +++ b/evals/payloadcms__payload__16047.json @@ -0,0 +1,108 @@ +{ + "pr": "payloadcms/payload#16047", + "title": "fix(plugin-multi-tenant): forbidden error when logging in as a user with no tenant", + "baseline_review": { + "comments": [ + { + "file": "packages/plugin-multi-tenant/src/utilities/getTenantOptions.ts", + "line": 42, + "severity": "high", + "comment": "The early return when `userTenantIds` is an empty array prevents the downstream `payload.find()` call that would fail with a Forbidden error. However, the condition `userTenantIds !== undefined && userTenantIds.length === 0` does not account for the case where `userTenantIds` is `undefined` -- that path still falls through to `payload.find()` with `overrideAccess: false`. Verify that users with `undefined` userTenantIds always have the `userHasAccessToAllTenants` flag set, otherwise the same Forbidden error could occur for them." + }, + { + "file": "packages/plugin-multi-tenant/src/utilities/getTenantOptions.ts", + "line": 42, + "severity": "low", + "comment": "The early return returns `tenantOptions` which at this point is the initial empty array `[]`. A brief inline comment explaining why returning empty is correct (user has no tenants and no global access) would improve readability for future maintainers." + }, + { + "file": "test/plugin-multi-tenant/collections/Tenants.ts", + "line": 36, + "severity": "medium", + "comment": "Adding `return false` for authenticated users with no tenants changes the access control semantics. Previously, authenticated users with no tenants fell through to the public-tenants filter `{ isPublic: { equals: true } }`. Now they get no access at all. This is a behavioral change in the test fixture -- confirm this matches the intended real-world access policy and does not break any existing tests relying on the previous fallthrough behavior." + }, + { + "file": "test/plugin-multi-tenant/seed/index.ts", + "line": 230, + "severity": "low", + "comment": "The commented-out `// tenants: [],` is noise. Either include the empty array explicitly to be clear about the intent, or remove the comment entirely. Commented-out code in test seeds can be confusing." + }, + { + "file": "test/plugin-multi-tenant/e2e.spec.ts", + "line": 704, + "severity": "medium", + "comment": "The E2E test navigates to `menuItemsURL.list` and asserts the absence of an error message. This is a negative assertion which could pass even if the page fails to load for a different reason. Consider adding a positive assertion (e.g., the page renders a list view or a no-results message) to confirm the page loaded successfully rather than just not crashing." + }, + { + "file": "test/plugin-multi-tenant/payload-types.ts", + "line": 109, + "severity": "low", + "comment": "The addition of the `widgets` type definition with `CollectionsWidget` appears unrelated to the multi-tenant forbidden error fix. This looks like auto-generated type changes that were included incidentally. Consider separating unrelated generated-file changes from the fix commit." + } + ], + "summary": "This PR correctly addresses the Forbidden error by short-circuiting `getTenantOptions` when a user has an empty tenant list, preventing an unauthorized `payload.find()` call. The fix is minimal and targeted, though the E2E test could be strengthened with a positive assertion and the test fixture access control change deserves careful verification against the intended policy." + }, + "flow_guided_review": { + "comments": [ + { + "file": "test/plugin-multi-tenant/collections/Tenants.ts", + "line": 36, + "severity": "high", + "comment": "Step 1 (entry point, high risk): The `tenantAccess` function now returns `false` for authenticated users with no tenants, where it previously fell through to `{ isPublic: { equals: true } }`. This is the root cause enabler -- without this access restriction in the test fixture, the `payload.find()` call in `getTenantOptions` with `overrideAccess: false` would not throw Forbidden. This confirms the bug: the production code assumed tenant collection access would succeed, but stricter access policies expose the crash. The fix must be in `getTenantOptions` (step 2), not here -- this test fixture change just makes the test reproduce the real-world scenario." + }, + { + "file": "packages/plugin-multi-tenant/src/utilities/getTenantOptions.ts", + "line": 42, + "severity": "high", + "comment": "Step 2 (entry point, high risk): This is the core fix. When `userTenantIds` resolves to an empty array (user authenticated but assigned to zero tenants), the function now returns early with an empty `tenantOptions` array instead of calling `payload.find()` with `overrideAccess: false`. The guard correctly distinguishes three states: `undefined` (user has access to all tenants, proceed to find), non-empty array (filter by tenant IDs), empty array (no tenants, return empty). Verify that the upstream code that builds `userTenantIds` correctly returns `[]` (not `undefined`) for users with no tenants." + }, + { + "file": "packages/plugin-multi-tenant/src/utilities/getTenantOptions.ts", + "line": 42, + "severity": "medium", + "comment": "The `where` clause built below this guard uses `userTenantIds` in an `in` operator. If this guard were removed and an empty array reached the `in` clause, the query behavior would be database-dependent (some DBs treat `IN ()` as an error). The early return thus also prevents a potential query-level edge case, which is a secondary benefit worth noting." + }, + { + "file": "test/plugin-multi-tenant/seed/index.ts", + "line": 230, + "severity": "medium", + "comment": "Step 3 (entry point, high risk): The seed creates a user with `roles: ['user']` and no `tenants` field. The commented-out `// tenants: []` is ambiguous -- it suggests the author considered explicitly passing an empty array but opted for omission. These may have different semantics depending on how the ORM handles missing vs empty array fields. If `tenants` defaults to `undefined` rather than `[]`, the upstream `userTenantIds` computation might return `undefined` instead of `[]`, which would bypass the new guard entirely. This is the most important integration point to verify." + }, + { + "file": "test/plugin-multi-tenant/e2e.spec.ts", + "line": 704, + "severity": "medium", + "comment": "The E2E test validates the fix end-to-end by logging in as the no-tenant user and navigating to a page that triggers `getTenantOptions`. The negative assertion checks the error message is absent, which directly tests the reported bug symptom. However, adding a positive assertion (e.g., page title or list container is visible) would make the test more robust against silent failures." + }, + { + "file": "test/plugin-multi-tenant/credentials.ts", + "line": 10, + "severity": "low", + "comment": "The `noTenant` credential entry uses a weak password ('test') consistent with other test credentials. The naming is clear and the alphabetical placement (between `janeDoe` and `owner`) maintains the existing ordering convention." + } + ], + "summary": "Following the flow from the access control change (step 1) through the core fix in `getTenantOptions` (step 2) to the seed data (step 3), the PR correctly addresses the Forbidden error by early-returning when a user has zero assigned tenants. The highest-risk integration point is whether omitting the `tenants` field in the seed produces an empty array (triggering the guard) or `undefined` (bypassing it) -- this determines whether the E2E test actually exercises the new code path." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 9, + "flow_awareness": 9, + "risk_identification": 9, + "actionability": 8, + "efficiency": 8, + "overall": 8.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review is substantially stronger because it traces the causal chain of the bug: the test fixture's access policy (step 1) triggers the Forbidden error in getTenantOptions (step 2), and the seed data (step 3) determines whether the fix is actually exercised. The baseline review treats each file in isolation and misses the critical integration risk -- whether omitting `tenants` in the seed produces `[]` or `undefined` at the guard point. The flow-guided review also correctly identifies the three-state semantics of `userTenantIds` (undefined/empty/non-empty) and how the guard fits into the existing control flow, while the baseline review raises the undefined concern without connecting it to the seed data. Both reviews note the E2E test weakness, but the flow-guided review contextualizes it within the data flow." + } +} \ No newline at end of file diff --git a/evals/payloadcms__payload__16058.json b/evals/payloadcms__payload__16058.json new file mode 100644 index 0000000..30adc38 --- /dev/null +++ b/evals/payloadcms__payload__16058.json @@ -0,0 +1,155 @@ +{ + "pr": { + "url": "https://github.com/payloadcms/payload/pull/16058", + "owner": "payloadcms", + "repo": "payload", + "number": 16058, + "title": "templates: fix broken images on Next.js 16 by using relative paths for local media", + "files_changed": 11, + "additions": 87, + "deletions": 16, + "language": "TypeScript" + }, + "timestamp": "2026-03-30T00:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "templates/website/src/utilities/getMediaUrl.ts", + "line": 19, + "severity": "major", + "comment": "The simplified function now returns any URL string as-is, including external URLs. Previously there was logic to detect absolute URLs (http/https) and handle them differently from relative paths. While the PR description says external URLs from storage plugins are returned as-is, verifying that callers never pass a URL that needs the old base-URL prepending behavior would be prudent. If any caller relied on getMediaUrl to turn a relative path into an absolute URL for use outside of Next.js Image (e.g., in an og:image meta tag or RSS feed), this change silently breaks that usage." + }, + { + "file": "templates/website/src/utilities/getMediaUrl.ts", + "line": 11, + "severity": "minor", + "comment": "The import of getClientSideURL was removed. If this utility was the only consumer, the getURL utility export may now have a dead export. Consider checking whether getClientSideURL is still used elsewhere in the template or if it can be cleaned up." + }, + { + "file": "templates/ecommerce/next.config.ts", + "line": 13, + "severity": "minor", + "comment": "The ecommerce and website templates still have remotePatterns that include the localhost server URL. With local paths now being served as relative, the remotePatterns entry for localhost may be unnecessary for local media. However, keeping it is harmless and may be needed for other remote resources, so this is a low-priority cleanup." + }, + { + "file": "templates/_template/next.config.ts", + "line": 10, + "severity": "nit", + "comment": "The localPatterns pathname is hardcoded to '/api/media/file/**'. If a user customizes their media collection slug or upload path, this pattern would not match. A comment noting this assumption or documenting how to adjust the pattern would help template consumers." + }, + { + "file": "templates/website/src/utilities/getMediaUrl.ts", + "line": 19, + "severity": "nit", + "comment": "The function is now essentially a trivial wrapper: it returns the URL with an optional cache tag query parameter. Consider whether the function still justifies its existence or whether callers could inline the logic. That said, keeping the abstraction is fine for consistency across templates." + } + ], + "summary": "This PR correctly addresses the Next.js 16 private-IP image blocking issue by switching from absolute to relative local media URLs and adding localPatterns config. The main concern is whether any callers of getMediaUrl relied on the old absolute-URL behavior for non-Image contexts like meta tags or server-side fetches." + }, + "flow_guided_review": { + "comments": [ + { + "file": "templates/website/src/utilities/getMediaUrl.ts", + "line": 19, + "severity": "major", + "comment": "ENTRY POINT (Step 1, high risk). The getMediaUrl function was drastically simplified, removing the getClientSideURL() prepend and the http/https detection branch. This is the core behavioral change: all URLs are now passed through as-is. Since this is an entry point called from multiple components (Image rendering, media display), any caller that depended on receiving an absolute URL (e.g., for server-side fetches, Open Graph tags, or API calls that need a full origin) will now receive a relative path that won't resolve outside a browser context. All call sites should be audited." + }, + { + "file": "templates/with-vercel-website/src/utilities/getMediaUrl.ts", + "line": 19, + "severity": "major", + "comment": "ENTRY POINT (Step 2, high risk). This is the same change as Step 1 but in a different template. The plan identifies this as an independent flow, meaning changes here don't depend on Step 1, but the risk is identical. The with-vercel-website template may have Vercel-specific deployment concerns where relative paths behave differently (e.g., Vercel's image optimization CDN). Testing on Vercel deployments specifically is recommended." + }, + { + "file": "templates/_template/next.config.ts", + "line": 10, + "severity": "minor", + "comment": "The localPatterns configuration is the essential companion to the getMediaUrl simplification. Without this entry, Next.js 16 would reject the relative paths. The pattern '/api/media/file/**' is specific to the default Payload media collection path. Templates that customize their upload path or collection slug would need to update this pattern, which is not documented." + }, + { + "file": "templates/website/src/utilities/getMediaUrl.ts", + "line": 15, + "severity": "minor", + "comment": "The cacheTag logic appends a bare query parameter without a key (e.g., '?abc123'). This was present before and is unchanged, but with the URL now being relative, ensure that Next.js image optimization correctly passes through query parameters on local paths. Previously the absolute URL would bypass the optimizer differently." + }, + { + "file": "templates/ecommerce/next.config.ts", + "line": 13, + "severity": "nit", + "comment": "The ecommerce template merges localPatterns into the existing images config alongside remotePatterns and qualities. The diff shows only 5 of the 11 files changed in the diff (truncated). The remaining template configs (with-vercel-postgres, etc.) follow the same pattern. Consistency across all templates is good, though the repetition suggests this configuration could be extracted into a shared utility." + }, + { + "file": "templates/website/src/utilities/getMediaUrl.ts", + "line": 11, + "severity": "positive", + "comment": "Good simplification. Removing the getClientSideURL dependency eliminates a common source of SSR/CSR mismatch bugs where the server-side URL differs from the client-side URL. Relative paths are inherently environment-agnostic." + } + ], + "summary": "The flow-guided review identifies two independent high-risk entry points (getMediaUrl in website and with-vercel-website templates) that fundamentally change URL resolution from absolute to relative. The localPatterns config additions are the necessary counterpart. The primary risk is that callers outside of Next.js Image optimization that need absolute URLs will break silently, and the Vercel-specific template deserves dedicated deployment testing." + }, + "review_plan": { + "stats": { + "totalSteps": 2, + "totalAdditions": 2, + "totalDeletions": 16, + "independentFlows": 2, + "filesChanged": 2 + }, + "steps": [ + { + "order": 1, + "nodeId": "templates/website/src/utilities/getMediaUrl.ts::getMediaUrl", + "name": "getMediaUrl", + "file": "templates/website/src/utilities/getMediaUrl.ts", + "lines": [11, 19], + "type": "function", + "changeType": "modified", + "additions": 1, + "deletions": 8, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 2, + "nodeId": "templates/with-vercel-website/src/utilities/getMediaUrl.ts::getMediaUrl", + "name": "getMediaUrl", + "file": "templates/with-vercel-website/src/utilities/getMediaUrl.ts", + "lines": [11, 19], + "type": "function", + "changeType": "modified", + "additions": 1, + "deletions": 8, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + } + ], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + }, + "reasoning": "The baseline review correctly identifies the main risk (callers depending on absolute URLs) and notes the dead-code cleanup opportunity, but treats the change generically without distinguishing between the two independent template flows or their deployment contexts. It does not leverage the plan's identification of two separate entry points with independent risk profiles. The flow-guided review, informed by the plan, explicitly addresses both entry points in order, highlights the Vercel-specific deployment risk for the second flow, connects the config changes to the code changes as necessary counterparts, and provides more targeted actionable advice (audit call sites, test on Vercel specifically). The flow-guided review's awareness of independent flows and entry-point risk leads to more precise and deployment-context-aware comments. Both reviews correctly identify the cache-tag and localPatterns considerations, but the flow-guided review provides stronger risk identification by differentiating between template contexts.", + "winner": "flow_guided" + } +} diff --git a/evals/payloadcms__payload__16092.json b/evals/payloadcms__payload__16092.json new file mode 100644 index 0000000..3f5cc82 --- /dev/null +++ b/evals/payloadcms__payload__16092.json @@ -0,0 +1,114 @@ +{ + "pr": "payloadcms/payload#16092", + "title": "fix(sdk): pass trash to request", + "baseline_review": { + "comments": [ + { + "file": "packages/sdk/src/utilities/buildSearchParams.ts", + "line": 40, + "severity": "medium", + "comment": "The `trash` param is serialized as `String(args.trash)`, which means passing `trash: false` explicitly will emit `trash=false` in the query string. Depending on how the REST handler parses this (truthy string check vs strict boolean parse), `trash=false` could be interpreted as truthy. The `draft` param has the same pattern, so this is at least consistent, but worth verifying the server-side parsing treats the string 'false' correctly." + }, + { + "file": "packages/sdk/src/collections/delete.ts", + "line": 43, + "severity": "medium", + "comment": "The JSDoc for `trash` on `DeleteBaseOptions` says 'use with the REST API semantics for permanent delete vs bulk operations on trashed docs' which is vague compared to the other type definitions. The `find`, `findByID`, and `count` options all clearly state 'When true, includes/returns trashed documents'. The delete variant should clarify the exact behavior -- does `trash: true` on delete mean 'permanently delete trashed docs' or 'soft-delete instead of hard-delete'?" + }, + { + "file": "test/sdk/int.spec.ts", + "line": 34, + "severity": "medium", + "comment": "The test fixture creates a 'trashed' post by manually setting `deletedAt` to the current timestamp. This relies on the internal implementation detail that Payload's soft-delete uses a `deletedAt` field. If the trash implementation changes (e.g., uses a boolean `_trashed` flag or a status field), this fixture will silently stop working. Consider using `payload.delete` with the appropriate soft-delete option to create the trashed document through the official API." + }, + { + "file": "test/sdk/int.spec.ts", + "line": 65, + "severity": "low", + "comment": "The test asserts that `find` without `trash` returns 1 doc and `find` with `trash: true` returns 2 docs. This implicitly depends on the `beforeAll` setup creating exactly 2 posts (one normal, one trashed) and no other test creating posts before this one runs. If test ordering changes or another test inserts posts, this could become flaky. Using a more specific `where` clause or counting relative to baseline would be more robust." + }, + { + "file": "packages/sdk/src/collections/count.ts", + "line": 16, + "severity": "low", + "comment": "The `trash` option is added to `CountOptions` but the PR description only mentions `find`, `findVersions`, and `findVersionByID`. The `count` and `delete` and `update` operations also get the `trash` type -- verify these are actually wired through to `buildSearchParams` in their respective SDK methods, not just typed." + }, + { + "file": "test/sdk/int.spec.ts", + "line": 83, + "severity": "low", + "comment": "The `findVersions with trash` test follows the same pattern as the `find with trash` test but queries on `parent` field. Both trash tests are integration tests hitting a real server. The PR description mentions 'Added an integration test that uses a stub fetch' but the actual tests use the real `sdk` instance backed by a running Payload server -- the description is inaccurate." + } + ], + "summary": "The core fix in `buildSearchParams.ts` is minimal and follows the existing `draft` pattern correctly, extending it to serialize `trash` as a query param. The main concerns are the vague JSDoc on the delete variant, the test fixture relying on internal `deletedAt` field mechanics rather than the public API, and a mismatch between the PR description (claiming stub fetch tests) and the actual integration tests." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/sdk/src/utilities/buildSearchParams.ts", + "line": 40, + "severity": "high", + "comment": "This is the single entry point identified by the plan (high risk). The 4-line addition mirrors the `draft` serialization pattern exactly: `if (typeof args.trash === 'boolean') { search.trash = String(args.trash) }`. While the implementation is correct and consistent, the risk label is justified because this function is called by every SDK collection/version method. Passing `trash: false` will emit `trash=false` in the URL -- the REST handler's `parseParams` must treat the string 'false' as falsy. If it does a loose truthy check (`if (req.query.trash)`), then `'false'` is truthy and trash will be unexpectedly enabled. Verify the server-side parsing in `parseParams` handles this correctly, or only serialize when `args.trash === true` to avoid ambiguity." + }, + { + "file": "packages/sdk/src/utilities/buildSearchParams.ts", + "line": 17, + "severity": "medium", + "comment": "The `OperationArgs` type gains `trash?: boolean` but the plan shows this is the sole code change node with no callers or callees tracked. This means the plan did not trace which SDK methods actually destructure and pass `trash` from their options into `buildSearchParams`. The type definitions in `count.ts`, `delete.ts`, `findByID.ts`, and `update.ts` all add `trash` to their option types, but without verifying the method implementations actually thread `trash` into `buildSearchParams(args)`, the types could be lying -- the option would be accepted but silently ignored. The PR only shows type additions, not the method body changes that would pass `trash` through." + }, + { + "file": "packages/sdk/src/collections/delete.ts", + "line": 43, + "severity": "medium", + "comment": "Following from the entry point: `buildSearchParams` will serialize `trash` for any operation that passes it through. The `DeleteBaseOptions` JSDoc is notably less precise than the other option types. For `find`/`findByID`/`count`, the semantics are clear: 'include trashed documents in results'. For `delete`, the interaction between `trash: true` and the actual delete operation is ambiguous -- does it mean 'target trashed docs for permanent deletion' or 'perform a soft delete'? This ambiguity could lead to data loss if a user misinterprets the flag." + }, + { + "file": "test/sdk/int.spec.ts", + "line": 34, + "severity": "medium", + "comment": "The test fixture creates a trashed post by directly setting `deletedAt` via `payload.create`. This bypasses whatever soft-delete flow Payload normally uses and couples the test to an internal implementation detail. If the trash feature's internal representation changes, this test will silently create a normal (non-trashed) document and all trash assertions will fail with confusing 'expected 2, got 1' errors. A more robust approach would be to create a normal post then soft-delete it via `payload.delete` with the appropriate trash option." + }, + { + "file": "test/sdk/int.spec.ts", + "line": 65, + "severity": "low", + "comment": "The find-with-trash test scopes its query using `pairWhere = { id: { in: [post.id, postTrash.id] } }`, which is good isolation. However, the `findVersions` test on line 83 uses `parent: { in: [...] }` which depends on versions being created for both documents. The `beforeAll` creates `postTrash` with `deletedAt` set at creation time -- this means the version for `postTrash` also has the trashed state. The test assumes exactly 1 version per post, which holds because `versions: true` creates an initial version on create. If version drafts or autosave were enabled, additional versions could appear." + }, + { + "file": "test/sdk/collections/Posts.ts", + "line": 39, + "severity": "low", + "comment": "Adding `trash: true` to the Posts collection config enables the soft-delete feature for all SDK tests, not just the trash-specific ones. This could subtly affect existing tests if any of them call `payload.delete` on posts -- instead of hard-deleting, docs would be soft-deleted and could appear in subsequent queries if `trash: true` is passed. The existing cleanup in `afterEach` (line 67) uses `payload.delete({ collection: 'posts', where: { id: { in: ids } } })` which with `trash: true` on the collection would soft-delete rather than permanently remove. Verify the cleanup still works correctly." + }, + { + "file": "packages/sdk/src/collections/findByID.ts", + "line": 64, + "severity": "low", + "comment": "The `FindByIDOptions` type adds `trash?: boolean` with clear JSDoc. This is one of four type-only additions (count, delete, findByID, update) that complement the `buildSearchParams` change. The type addition is correct but the diff does not show the `findByID` method implementation passing `trash` to `buildSearchParams`. If the method spreads all options into the args (common pattern), this works implicitly; otherwise, `trash` would be accepted by the type but never serialized." + } + ], + "summary": "The flow-guided analysis reveals that while the `buildSearchParams` entry point change is minimal and correct, the PR's risk concentrates on two areas: (1) whether `trash: false` serialized as the string 'false' is handled correctly by server-side parsing, and (2) whether the four type-only additions (count, delete, findByID, update) are actually threaded through their method implementations to `buildSearchParams`, since the diff only shows type changes, not method body changes for those operations. The test fixture's reliance on internal `deletedAt` field mechanics and the potential impact of enabling `trash: true` on the shared Posts collection for all tests are secondary concerns." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leveraged the plan's identification of `buildSearchParams` as the sole high-risk entry point to systematically trace concerns outward: from the serialization behavior (trash=false as a truthy string), through the question of whether type additions are actually wired to the entry point in method bodies (a gap the diff does not conclusively answer), to the downstream test implications of enabling trash on a shared collection. The baseline review caught many of the same surface issues (vague delete JSDoc, test fixture coupling, PR description mismatch) but lacked the structural reasoning about call flow -- it did not question whether the type additions are backed by method-level threading, which is the most impactful potential gap. The flow-guided review's entry-point-first approach naturally surfaced the 'false' serialization risk as high severity by recognizing that buildSearchParams is called by every SDK method, while the baseline treated it as medium without that context." + } +} diff --git a/evals/prettier__prettier__18975.json b/evals/prettier__prettier__18975.json new file mode 100644 index 0000000..5a16620 --- /dev/null +++ b/evals/prettier__prettier__18975.json @@ -0,0 +1,102 @@ +{ + "pr": "prettier/prettier#18975", + "title": "[MDX] activate extensions", + "baseline_review": { + "comments": [ + { + "file": "src/language-markdown/parse/parse-mdx.js", + "line": 35, + "severity": "major", + "comment": "Uncommenting `mathSyntax()` and `wikiLinkSyntax()` activates these extensions for all MDX parsing. However, the corresponding imports for `mathSyntax`, `wikiLinkSyntax`, `mathFromMarkdown`, and `wikiLinkFromMarkdown` are not shown in the diff — they must already exist at the top of the file (likely from an earlier preparation). If any import is missing, this will fail at runtime. Worth verifying the imports are present and correct." + }, + { + "file": "src/language-markdown/parse/parse-mdx.js", + "line": 14, + "severity": "minor", + "comment": "The removal of the `overrideHtmlTextSyntax` and `liquidSyntax`/`liquidFromMarkdown` imports is clean — these were previously imported but only used in commented-out code. Removing the dead imports along with the dead commented-out lines is the right approach. However, the liquid and HTML-text extensions are being permanently dropped, not just deferred — confirm this is intentional." + }, + { + "file": "src/language-markdown/parse/parse-mdx.js", + "line": 35, + "severity": "minor", + "comment": "The `liquidSyntax()` and `overrideHtmlTextSyntax()` calls were removed entirely rather than kept as comments. This suggests they are deliberately excluded from MDX parsing. The PR title says 'activate extensions' but only math and wikiLink are activated — liquid and HTML text override are dropped. The asymmetry should be documented or explained in the PR description." + }, + { + "file": "tests/format/mdx/markdown-consistency/__snapshots__/format.test.js.snap", + "line": 1, + "severity": "minor", + "comment": "The snapshot confirms that `$$$` (triple dollar) in the input is normalized to `$$` (double dollar) in the output, which is the expected behavior of the math extension (treating `$$$` as a math block delimiter equivalent to `$$`). This is a good regression test verifying the math extension works correctly in MDX." + }, + { + "file": "tests/format/mdx/markdown-consistency/format.test.js", + "line": 1, + "severity": "minor", + "comment": "The test runs with both `mdx` and `markdown` parsers, which verifies that both parsers produce the same output for the same input — good for consistency testing. However, the test file is minimal and only covers math and wikiLink. Edge cases like nested expressions, empty math blocks, or wikiLinks with special characters are not tested." + } + ], + "summary": "The PR activates math and wikiLink micromark extensions for MDX parsing that were previously commented out, while permanently removing liquid and HTML text override extensions. Tests are present but minimal, covering only basic math block normalization and wikiLink passthrough." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/language-markdown/parse/parse-mdx.js", + "line": 35, + "severity": "major", + "comment": "Following the plan's dependency chain: `parseMdx` calls `getMarkdownParseOptions` which returns the cached configuration object. Since `markdownParseOptions` is memoized (via `??=`), once these new extensions are activated they persist for the entire process lifetime. The addition of `mathSyntax()` and `wikiLinkSyntax()` to the extensions array means every subsequent MDX parse will run these extensions. If these extensions interact poorly with `mdxjsEsm`, `mdxExpression`, or `mdxJsx` (which are MDX-specific), the ordering matters — math and wikiLink are placed before the MDX extensions, which is correct since they should parse their syntax before MDX takes over." + }, + { + "file": "src/language-markdown/parse/parse-mdx.js", + "line": 45, + "severity": "major", + "comment": "The `mdastExtensions` array also gains `mathFromMarkdown()` and `wikiLinkFromMarkdown()` to complement the syntax extensions. These MDAST extensions transform the parsed tokens into AST nodes. The ordering here places them before `mdxFromMarkdown()`, which is important — if MDX's MDAST extension encounters math or wikiLink tokens before their handlers are registered, the nodes would be lost or misinterpreted. The current ordering is correct." + }, + { + "file": "src/language-markdown/parse/parse-mdx.js", + "line": 14, + "severity": "minor", + "comment": "The plan identifies this as a leaf node with low risk, and the import cleanup confirms that. The removed imports (`overrideHtmlTextSyntax`, `liquidFromMarkdown`, `liquidSyntax`) correspond exactly to the removed commented-out code. No dangling references remain. The liquid extension was experimental for MDX and its removal signals these features are not planned for MDX support." + }, + { + "file": "tests/format/mdx/markdown-consistency/extensions.mdx", + "line": 3, + "severity": "minor", + "comment": "The test fixture uses `$$$` (triple dollar) for math blocks rather than the standard `$$`. This is clever — it tests both that the math extension activates correctly AND that prettier normalizes the delimiter to the canonical `$$` form. The wikiLink test (`[[Wiki *Link*]]`) also verifies that inline markdown formatting within wikiLinks is preserved." + }, + { + "file": "tests/format/mdx/markdown-consistency/format.test.js", + "line": 1, + "severity": "minor", + "comment": "Running with both `mdx` and `markdown` parsers validates that the activated extensions produce identical output across both parsers. This is the key consistency guarantee — previously, markdown supported these extensions but MDX did not, creating a formatting discrepancy. The test directly validates the PR's goal." + }, + { + "file": "tests/format/mdx/markdown-consistency/__snapshots__/format.test.js.snap", + "line": 1, + "severity": "nit", + "comment": "The snapshot is well-structured and shows both math and wikiLink features. However, there is no test coverage for potential conflicts between math/wikiLink syntax and MDX expressions (e.g., `{expression}` inside a math block, or a wikiLink containing JSX-like content). Given the plan shows these extensions now coexist with `mdxjsEsm`, `mdxExpression`, and `mdxJsx`, edge-case interaction tests would strengthen confidence." + } + ], + "summary": "The flow-guided analysis reveals that extension ordering within `getMarkdownParseOptions` is critical since the memoized config affects all subsequent MDX parses. Math and wikiLink extensions are correctly placed before MDX-specific extensions in both the syntax and MDAST arrays, ensuring proper token parsing precedence. The test validates cross-parser consistency, which is the core goal of this change." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leverages the dependency analysis (parseMdx -> getMarkdownParseOptions) to identify that extension ordering within the memoized config is architecturally significant. It correctly notes that math/wikiLink extensions must precede MDX-specific extensions for proper token parsing, and that the MDAST extension ordering mirrors this requirement. The baseline review identifies the same surface-level changes but treats the uncommented code as isolated additions without analyzing how they interact with the existing MDX extensions in the array. The flow-guided review also better contextualizes why the test uses both parsers (cross-parser consistency) rather than just noting it as a fact. The baseline does raise a valid point about the asymmetric treatment of liquid/HTML-text vs math/wikiLink that the flow-guided review handles less prominently." + } +} diff --git a/evals/prisma__prisma__29382.json b/evals/prisma__prisma__29382.json new file mode 100644 index 0000000..76f79e5 --- /dev/null +++ b/evals/prisma__prisma__29382.json @@ -0,0 +1,102 @@ +{ + "pr": "prisma/prisma#29382", + "title": "fix(client): prevent caching of createMany queries to avoid cache bloat and possible node.js crashes", + "baseline_review": { + "comments": [ + { + "file": "packages/client/src/runtime/core/engines/client/ClientEngine.ts", + "line": 479, + "severity": "medium", + "comment": "The `isCacheable` check only covers `createMany` and `createManyAndReturn`. Other bulk operations like `deleteMany` or `updateMany` with large WHERE clauses could also produce unstable cache keys and cause similar bloat. Consider whether this should be a more general heuristic (e.g. based on parameter count or key size) rather than a hardcoded action list." + }, + { + "file": "packages/client/src/runtime/core/engines/client/ClientEngine.ts", + "line": 478, + "severity": "low", + "comment": "The comment explains the rationale well, but it would be helpful to reference the issue number (prisma#29331) directly in the code comment so future maintainers can find the context for this decision without searching git blame." + }, + { + "file": "packages/client/src/runtime/core/engines/client/ClientEngine.ts", + "line": 479, + "severity": "low", + "comment": "The `query.action` string comparison relies on matching exact action names. If the action names change or new createMany variants are added in the future, this check would silently stop working. Consider using a Set or constant for non-cacheable actions, or checking with `query.action.startsWith('createMany')` to be more resilient." + }, + { + "file": "packages/client/tests/functional/issues/29331-query-plan-cache-bloat/tests.ts", + "line": 71, + "severity": "medium", + "comment": "The stress test defaults to only 5 iterations with batch size 20 when `PRISMA_CREATE_MANY_STRESS` is not set. At such a small scale, this test does not actually verify that cache bloat is prevented -- it merely exercises the createMany path. To prove the fix works, the test should assert on cache state or memory usage, not just that the operations succeed without crashing." + }, + { + "file": "packages/client/tests/functional/issues/29331-query-plan-cache-bloat/tests.ts", + "line": 9, + "severity": "low", + "comment": "The `daysBefore` helper creates dates relative to `new Date()`, making the test non-deterministic. While this is unlikely to cause flakiness in practice, using a fixed base date would make the test fully reproducible." + } + ], + "summary": "The fix is a targeted and simple approach to prevent createMany queries from polluting the query plan cache, directly addressing the reported memory bloat issue. The main concern is that the test does not actually assert on cache behavior or memory, so it serves more as a smoke test than a proof that the fix works as intended." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/client/src/runtime/core/engines/client/ClientEngine.ts", + "line": 479, + "severity": "high", + "comment": "Following the flow through `ClientEngine.request` (step 3, high risk, entry point): the `isCacheable` flag is computed from `query.action` but only applied in the single-query path (the `if` branch starting around line 474). The review plan indicates `requestBatch` is a sibling method in the same cluster. If `requestBatch` also compiles and caches query plans for batched createMany operations (which is the most common usage pattern for createMany via `$transaction`), then this fix may not cover the primary use case. The batch path needs the same cache-bypass logic or the memory bloat will persist for transactional createMany calls." + }, + { + "file": "packages/client/src/runtime/core/engines/client/ClientEngine.ts", + "line": 479, + "severity": "medium", + "comment": "The `isCacheable` check uses strict string equality against `'createMany'` and `'createManyAndReturn'`. The review plan shows `#compileQuery` (step in the cluster) is called from both `request` and `#compileBatch`. If the action names passed through the batch path differ in casing or format, the check would miss them. Additionally, the plan's cluster includes 13 related functions -- any other path that calls `#queryPlanCache.setSingle` would bypass this guard. A more defensive approach would be to push the cacheability check into `#compileQuery` itself or into a wrapper around the cache." + }, + { + "file": "packages/client/tests/functional/issues/29331-query-plan-cache-bloat/tests.ts", + "line": 71, + "severity": "medium", + "comment": "The test (step 1, `daysBefore` helper + test suite) is marked high risk as an entry point but does not validate the actual fix. It creates varying createMany payloads and asserts the operations succeed, but never inspects the query plan cache size or verifies that createMany queries were not cached. Without such assertions, a regression that re-enables caching would not be caught by this test. Consider accessing `ClientEngine.#queryPlanCache` via a test hook or debug API to assert cache size remains stable across iterations." + }, + { + "file": "packages/client/src/runtime/core/engines/client/ClientEngine.ts", + "line": 484, + "severity": "medium", + "comment": "When `isCacheable` is false, the query plan is still compiled via `this.#compileQuery(parameterizedQuery, cacheKey, queryCompiler)` -- the `cacheKey` parameter is still computed and passed despite never being used for caching. While this is not a bug, computing `JSON.stringify(parameterizedQuery)` for large createMany payloads with thousands of rows is itself expensive. If the query is known to be non-cacheable, the key computation could be skipped (passing a sentinel or empty string), saving serialization cost on the hot path this fix is meant to optimize." + }, + { + "file": "packages/client/tests/functional/issues/29331-query-plan-cache-bloat/prisma/_schema.ts", + "line": 14, + "severity": "low", + "comment": "The schema defines a `ContactAnalytics` model with 17 nullable fields to create varied parameter patterns. This schema is purpose-built for the stress test and the field naming (date1-date7, val1-val5, float1-float2, bool1-bool3) reflects the original issue's real-world model. The schema correctly uses `idForProvider` for cross-provider compatibility. This is well-structured test infrastructure." + }, + { + "file": "packages/client/tests/functional/issues/29331-query-plan-cache-bloat/tests.ts", + "line": 15, + "severity": "low", + "comment": "The test uses `faker.database.mongodbObjectId()` for contact IDs, which works for MongoDB but is also used for SQL providers via the `allProviders` matrix. This works because `idForProvider` likely uses String IDs for most providers, but it couples the test to the assumption that all providers accept MongoDB ObjectId-formatted strings as valid IDs." + } + ], + "summary": "The flow-guided analysis reveals that the fix only guards the single-query path in `ClientEngine.request`, but the review plan's cluster shows `requestBatch` and `#compileBatch` as sibling methods that may also cache createMany plans -- the batch path is the most common entry for bulk inserts in transactions. Additionally, the expensive `JSON.stringify` for cache key computation still runs even for non-cacheable queries, partially undermining the performance goal." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 5, + "flow_awareness": 3, + "risk_identification": 5, + "actionability": 6, + "efficiency": 7, + "overall": 5.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leveraged the plan's cluster information to identify that `requestBatch` and `#compileBatch` are sibling methods in the same class that may also cache query plans -- meaning the fix might only cover single-query createMany while the more common transactional batch path remains unguarded. This is a critical gap the baseline review missed entirely. The flow-guided review also caught that `JSON.stringify` still runs for non-cacheable queries, an actionable performance concern directly relevant to the PR's goal. The baseline review made valid but surface-level observations (hardcoded action names, test non-determinism) without understanding the broader call graph. The flow-guided approach's awareness of the 13-function cluster and entry/exit points naturally surfaced the batch-path gap, which is the most significant finding." + } +} diff --git a/evals/prisma__prisma__29392.json b/evals/prisma__prisma__29392.json new file mode 100644 index 0000000..1d5f75d --- /dev/null +++ b/evals/prisma__prisma__29392.json @@ -0,0 +1,114 @@ +{ + "pr": "prisma/prisma#29392", + "title": "feat: mariadb protocol options", + "baseline_review": { + "comments": [ + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 228, + "severity": "medium", + "comment": "When `new URL(config)` throws (e.g. for an invalid connection string), the code falls through to use `config` as-is, but the `prepareCacheLength` default is not applied in that fallback path. Users with slightly malformed URLs will silently miss the leak-prevention default." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 68, + "severity": "low", + "comment": "Using `.bind(this.client)` on both branches works but is slightly verbose. A simpler pattern would be `const method = this.mariadbOptions?.useTextProtocol ? 'query' : 'execute'; return await this.client[method](req, values);` which avoids the `.bind` overhead and is easier to read." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 171, + "severity": "low", + "comment": "The `mariadbOptions` parameter on `MariaDbQueryable` is optional (`?`), which means `useTextProtocol` silently defaults to `undefined` (falsy) and uses the binary protocol. This is fine for backward compatibility, but documenting this default at the constructor level would help maintainers." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.test.ts", + "line": 79, + "severity": "low", + "comment": "Test description 'set when is set' is grammatically unclear. Consider something like 'should preserve prepareCacheLength when explicitly set'." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 313, + "severity": "medium", + "comment": "The `rewriteConnectionString` function now mutates the URL object in place (setting `url.protocol`) rather than returning a new one. Callers that hold a reference to the original URL may be surprised by the mutation. Consider cloning: `const result = new URL(url.toString()); result.protocol = 'mariadb:'; return result;`." + }, + { + "file": "packages/adapter-mariadb/src/credentials.test.ts", + "line": 1, + "severity": "low", + "comment": "This test file was moved from `mariadb.test.ts` to its own file, which is a nice separation. However, unlike the tests in `mariadb.test.ts` which mock the mariadb module, this test actually attempts a real connection (calls `factory.connect()` without mocking). This means it depends on the mariadb driver being available at runtime and its error behavior, making it somewhat fragile." + } + ], + "summary": "The PR cleanly adds a `useTextProtocol` toggle and defaults `prepareCacheLength` to 0 to address statement leaks, with good test coverage for both features. The main concern is that the `prepareCacheLength` default is not applied when URL parsing fails in the catch path, and the `rewriteConnectionString` function now mutates its input URL in place." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 171, + "severity": "medium", + "comment": "The `MariaDbQueryable` base class now accepts `mariadbOptions` as an optional second constructor parameter. This parameter is threaded through to `PrismaMariaDbAdapter` (step 4) and `MariaDbTransaction` (step 3). The flow is consistent: `PrismaMariaDbAdapter.constructor` passes `mariadbOptions` to `super()`, and `startTransaction` (step 6) passes `this.mariadbOptions` to `MariaDbTransaction`. However, the rename from `options` to `mariadbOptions` in `PrismaMariaDbAdapter` changes the visibility from `private` to `protected readonly`. This is necessary for the base class field, but the `protected` on the subclass shadows the base class field of the same name -- verify TypeScript doesn't emit two separate properties." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 68, + "severity": "high", + "comment": "Following the flow from `MariaDbQueryable.performIO` (step 18, medium risk, called by both `queryRaw` and `executeRaw`): the `useTextProtocol` toggle switches between `client.query` and `client.execute`. The `query` (text) protocol may return different column types/formats than the binary protocol for certain edge cases (dates, decimals, large integers). The `typeCast` function and downstream result mapping in `queryRaw` may not handle these differences. This is the core behavioral change and the PR description acknowledges edge cases but doesn't document which ones -- consider adding a doc comment or linking to known issues." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 228, + "severity": "high", + "comment": "The `PrismaMariaDbAdapterFactory.constructor` (step 8, high risk, 20 additions) handles string vs object config differently. For strings, it parses with `new URL()`, adds `prepareCacheLength=0` if missing, and calls `rewriteConnectionString`. For objects, it spreads `{ ...config, prepareCacheLength: 0 }`. The catch block (line 233) silently falls back to the raw string without applying `prepareCacheLength=0` or `rewriteConnectionString` -- so a `mysql://` URL that fails to parse won't get rewritten to `mariadb://` either. This is a silent behavior regression for connection strings that the old `config.replace(/^mysql:\\/\\//, 'mariadb://')` would have handled fine." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 313, + "severity": "medium", + "comment": "The `rewriteConnectionString` function (step 13, leaf node) changed signature from `(config: PoolConfig | string) => PoolConfig | string` to `(url: URL) => URL`. This is only called from the factory constructor now (for string configs), so the narrower signature is appropriate. But the function mutates `url.protocol` in place. Since the caller (`constructor`, step 8) uses `url` after this call (`url.toString()`), the mutation is intentional, but it makes the function impure and the `return url` misleading -- callers might assume it returns a new URL." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.ts", + "line": 186, + "severity": "medium", + "comment": "In `startTransaction` (step 6), the `MariaDbTransaction` constructor call changed from `new MariaDbTransaction(conn, options, cleanup)` to `new MariaDbTransaction(conn, this.mariadbOptions, options, cleanup)`. This correctly threads `mariadbOptions` to transactions so `useTextProtocol` applies inside transactions too. The dependency chain is: `startTransaction` -> `MariaDbTransaction.constructor` -> `super(conn, mariadbOptions)` -> `MariaDbQueryable`. The `onError` callback (step 19, high risk, many callers) is now accessed via `this.mariadbOptions?.onConnectionError` instead of `this.options?.onConnectionError` -- this rename is consistent throughout." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.test.ts", + "line": 144, + "severity": "low", + "comment": "The `useTextProtocol` test (lines 138-158) maps `[false, undefined, true]` through `flagToMethod[String(!!flag)]`. For `undefined`, `!!undefined` is `false`, so `String(false)` maps to `'execute'` -- correct. For `true`, maps to `'query'` -- correct. The test also verifies the opposite method was NOT called, which is thorough. However, it only tests `executeRaw` -- `queryRaw` also calls `performIO` and should exhibit the same text vs binary behavior. Consider adding a test case for `queryRaw`." + }, + { + "file": "packages/adapter-mariadb/src/mariadb.test.ts", + "line": 47, + "severity": "low", + "comment": "The `vi.mock('mariadb')` in `beforeAll` with `vi.doUnmock` in `afterAll` affects module-level state. Since the `useTextProtocol` tests below also use `mariadb` types but don't mock `createPool`, the test ordering matters. If vitest runs these describe blocks in file order (which it does by default), the mock is cleaned up before the `useTextProtocol` tests run, so this should be fine. But it's fragile -- consider using `vi.mock` at the file scope with per-test spy setup instead." + } + ], + "summary": "The flow analysis reveals a consistent threading of `mariadbOptions` from `PrismaMariaDbAdapterFactory` through `PrismaMariaDbAdapter` down to `MariaDbQueryable` and `MariaDbTransaction`, ensuring `useTextProtocol` applies everywhere queries are executed. The highest-risk issue is the catch block in the factory constructor that silently drops both the `prepareCacheLength` default and the `mysql://` to `mariadb://` rewrite when URL parsing fails -- a regression from the previous simpler string-replace approach." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identified the same key issues (URL parse catch fallback, mutation of URL) but contextualized them within the call chain, revealing deeper implications. It traced the `mariadbOptions` threading from factory through adapter to transaction and queryable, verifying consistency. It caught that `queryRaw` is also affected by `useTextProtocol` but isn't tested, and identified the `protected` field shadowing concern. The baseline review stayed at the surface level, noting code style issues but missing the behavioral regression in the catch path for `mysql://` URLs and the cross-cutting nature of the protocol toggle. The flow-guided approach's structured traversal of entry points through internal nodes to leaves naturally surfaced dependency-aware concerns that the baseline missed." + } +} diff --git a/evals/prometheus__prometheus__18374.json b/evals/prometheus__prometheus__18374.json new file mode 100644 index 0000000..20e55d3 --- /dev/null +++ b/evals/prometheus__prometheus__18374.json @@ -0,0 +1,128 @@ +{ + "pr": { + "url": "https://github.com/prometheus/prometheus/pull/18374", + "owner": "prometheus", + "repo": "prometheus", + "number": 18374, + "title": "tsdb: use float64 for retention percentage", + "files_changed": 7, + "additions": 22, + "deletions": 20, + "language": "Go" + }, + "timestamp": "2026-03-30T12:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "config/config.go", + "line": 1127, + "severity": "minor", + "comment": "The validation now rejects negative values (`t.Percentage < 0`), which is correct since the type changed from uint to float64. However, there is no check for NaN or Inf values, which are valid float64 values that could be deserialized from YAML. Consider adding an explicit `math.IsNaN` / `math.IsInf` guard." + }, + { + "file": "tsdb/db.go", + "line": 2021, + "severity": "major", + "comment": "The retention calculation `int64(float64(diskSize) * maxPercentage / 100)` could overflow if `diskSize` is very large (near uint64 max) and `maxPercentage` is close to 100. The previous `int64(uint64(maxPercentage) * diskSize / 100)` had the same risk but with integer arithmetic. This is a pre-existing issue, not a regression, but worth noting." + }, + { + "file": "cmd/prometheus/main.go", + "line": 800, + "severity": "minor", + "comment": "The removal of the `MaxPercentage > 100` clamp in main.go is justified by the PR description (config UnmarshalYAML validates first). However, if someone constructs a tsdbOptions programmatically without going through YAML unmarshalling, the safety clamp is now gone. This is acceptable if the struct is considered internal." + }, + { + "file": "cmd/prometheus/main.go", + "line": 807, + "severity": "nit", + "comment": "The format verb change from `%d` to `%g` for the error message is correct for float64. Using `%g` will print values like `1.5` cleanly, but for whole numbers it will print `1` instead of `1%` -- the `%%` suffix handles the percent sign correctly regardless." + }, + { + "file": "config/config_test.go", + "line": 2657, + "severity": "minor", + "comment": "The new test `TestTSDBRetentionPercentageFloat` only checks a single fractional value (0.5). It would be more thorough to also test boundary values like 0, 100, and 99.99 to ensure the full valid range works correctly with float64." + }, + { + "file": "tsdb/db.go", + "line": 1133, + "severity": "nit", + "comment": "The simplification from `db.metrics.maxPercentage.Set(float64(max(opts.MaxPercentage, 0)))` to `db.metrics.maxPercentage.Set(max(opts.MaxPercentage, 0))` is clean since `max` now returns float64 directly, matching the `Set(float64)` parameter type." + } + ], + "summary": "This PR correctly changes the retention percentage type from uint to float64, fixing silent truncation of fractional values like 1.5. The change is well-coordinated across all seven files with proper validation updates, test additions, and format string fixes, though boundary-value test coverage and NaN/Inf guard could be strengthened." + }, + "flow_guided_review": { + "comments": [ + { + "file": "config/config.go", + "line": 1114, + "severity": "minor", + "comment": "The type change from `uint` to `float64` in TSDBRetentionConfig is the root of all downstream changes. With the empty flow plan (0 steps, 0 clusters, 0 dependencies), there are no complex data flows to trace, but the key propagation path is: YAML config -> TSDBRetentionConfig.Percentage (float64) -> tsdbOptions.MaxPercentage (float64) -> db.opts.MaxPercentage (float64) -> BeyondSizeRetention calculation. All four hops are updated consistently." + }, + { + "file": "config/config.go", + "line": 1127, + "severity": "minor", + "comment": "The validation now adds `t.Percentage < 0` since float64 can represent negative values unlike uint. However, float64 also admits NaN and positive/negative infinity. YAML parsing of `percentage: .nan` or `percentage: .inf` would bypass the range check since NaN comparisons are always false. Adding `math.IsNaN(t.Percentage) || math.IsInf(t.Percentage, 0)` would close this gap." + }, + { + "file": "cmd/prometheus/main.go", + "line": 800, + "severity": "minor", + "comment": "Removing the `MaxPercentage > 100` clamp is safe given the data flow: all YAML-sourced values pass through UnmarshalYAML validation first. The ApplyConfig path in tsdb/db.go also reads from the config struct, so it is equally protected. The only unprotected path would be direct struct construction in tests or embedding, which is an acceptable trade-off." + }, + { + "file": "tsdb/db.go", + "line": 2021, + "severity": "major", + "comment": "The retention byte calculation changes from integer arithmetic `int64(uint64(maxPercentage) * diskSize / 100)` to floating-point `int64(float64(diskSize) * maxPercentage / 100)`. For diskSize values up to ~9.2 exabytes (max int64), float64 has sufficient precision (52-bit mantissa covers up to 2^53 exactly). The functional change is correct and enables fractional percentages like 0.5% to produce non-zero byte limits." + }, + { + "file": "config/config_test.go", + "line": 2643, + "severity": "minor", + "comment": "The negative-value test expectation changed from a YAML unmarshal type error to the custom range validation error message. This correctly reflects that float64 accepts -1 at the YAML level but rejects it in UnmarshalYAML. Good that this existing test was updated rather than removed." + }, + { + "file": "config/config_test.go", + "line": 2657, + "severity": "minor", + "comment": "The new float test validates the happy path (0.5 parses correctly), but the test suite would benefit from an additional bad-config test for values like 100.1 to verify the upper boundary with float64 precision, especially since floating-point comparison `> 100` could behave unexpectedly near the boundary." + } + ], + "summary": "The type change from uint to float64 propagates cleanly through the config -> options -> TSDB retention pipeline with all four touch points updated consistently. The main risk is that float64 admits special values (NaN, Inf) that bypass the range validation, and boundary-value test coverage for the new float64 type could be more comprehensive." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 6.8 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + }, + "reasoning": "The flow-guided review edges ahead despite an empty flow plan by explicitly tracing the four-hop data propagation path (YAML config -> TSDBRetentionConfig -> tsdbOptions -> BeyondSizeRetention), which gives better context for why the change is consistent. Both reviews identify the NaN/Inf risk and boundary testing gap. The flow-guided review provides stronger risk identification by explaining why the clamp removal is safe (UnmarshalYAML + ApplyConfig both gate values) and why the float64 arithmetic is correct for realistic disk sizes. The baseline review is slightly more efficient with its nit-level observations. The flow-guided review wins narrowly on completeness and flow awareness by connecting the dots across the full data flow even without plan guidance.", + "winner": "flow_guided" + } +} diff --git a/evals/psf__black__5063.json b/evals/psf__black__5063.json new file mode 100644 index 0000000..305344c --- /dev/null +++ b/evals/psf__black__5063.json @@ -0,0 +1,113 @@ +{ + "pr": { + "url": "https://github.com/psf/black/pull/5063", + "owner": "psf", + "repo": "black", + "number": 5063, + "title": "Use \"Version X.Y.Z\" headings in changelog for stable permalink anchors", + "files_changed": 4 + }, + "baseline_review": { + "comments": [ + { + "file": "CHANGES.md", + "line": 56, + "severity": "positive", + "comment": "Adding the changelog entry for this PR under the Unreleased documentation section is correct. The entry clearly describes the motivation (stable permalink anchors on ReadTheDocs) and references the PR number." + }, + { + "file": "CHANGES.md", + "line": 58, + "severity": "positive", + "comment": "All version headings are consistently prefixed with 'Version ' while the '## Unreleased' heading is correctly left unchanged. This is a thorough, mechanical transformation." + }, + { + "file": "CHANGES.md", + "line": 58, + "severity": "minor", + "comment": "The diff shows a subset of the 68 version heading changes. Reviewers should verify that every version heading in the full file follows the new '## Version X.Y.Z' pattern and none were accidentally skipped or double-prefixed." + }, + { + "file": "scripts/release.py", + "line": 143, + "severity": "medium", + "comment": "The cleanup_changes_template_for_release method needs to produce '## Version X.Y.Z' headings when replacing the Unreleased header. Confirm the string replacement correctly inserts the 'Version ' prefix during the release process." + }, + { + "file": "scripts/check_pre_commit_rev_in_example.py", + "line": 20, + "severity": "medium", + "comment": "This script extracts version numbers from changelog headers. With the new 'Version ' prefix, the parsing logic must strip or skip the prefix to extract the bare version string. Verify the extraction handles both the Unreleased heading and versioned headings correctly." + }, + { + "file": "scripts/check_version_in_basics_example.py", + "line": 14, + "severity": "medium", + "comment": "Similar to the pre-commit check script, this script parses version numbers from CHANGES.md headers. The 7-line addition suggests a more substantial change to handle the 'Version ' prefix. Ensure the version extraction is robust and does not break on edge cases." + } + ], + "summary": "This PR makes a straightforward but wide-reaching change to prefix all version headings in CHANGES.md with 'Version ' for stable ReadTheDocs permalink anchors. The key risk areas are the three scripts that parse these headings, which must be updated to strip the new prefix when extracting version numbers." + }, + "flow_guided_review": { + "comments": [ + { + "file": "scripts/check_pre_commit_rev_in_example.py", + "line": 20, + "severity": "high", + "comment": "Review plan flags this as a high-risk entry point (order 1). The main() function parses CHANGES.md headers to extract version numbers. With 3 additions and 1 deletion, this likely adds prefix-stripping logic. If using lstrip('Version '), beware that lstrip strips individual characters not the whole prefix string -- use removeprefix('Version ') (Python 3.9+) instead for correctness." + }, + { + "file": "scripts/check_version_in_basics_example.py", + "line": 14, + "severity": "high", + "comment": "Review plan flags this as a high-risk entry point (order 2). With 7 additions and 1 deletion, this is the most changed script file. Same lstrip vs removeprefix concern applies. The larger diff suggests either more careful version extraction or additional validation logic. Confirm the extracted version string matches what downstream comparisons expect (bare semver like '26.3.1')." + }, + { + "file": "scripts/release.py", + "line": 97, + "severity": "minor", + "comment": "The SourceFiles class (order 4, low risk) has 1 addition and 1 deletion, likely updating a constant or template string that defines the heading format. The change is mechanical and low-risk given it is a leaf node called only by main()." + }, + { + "file": "scripts/release.py", + "line": 143, + "severity": "medium", + "comment": "The cleanup_changes_template_for_release method (order 7, low risk) transforms '## Unreleased' into a versioned heading. The dependency graph shows it is called by update_repo_for_release. Verify it now produces '## Version X.Y.Z' instead of '## X.Y.Z', and that the substitution is clean." + }, + { + "file": "CHANGES.md", + "line": 56, + "severity": "positive", + "comment": "The changelog entry for this PR is correctly placed under the Unreleased documentation section and references the issue number. The self-referential nature (changelog entry about changing the changelog format) is appropriate." + }, + { + "file": "CHANGES.md", + "line": 58, + "severity": "minor", + "comment": "The bulk heading rename is a low-risk mechanical change. The flow plan correctly deprioritizes reviewing these 68 identical substitutions, focusing instead on the script logic that must adapt to the new format." + } + ], + "summary": "The flow-guided review correctly prioritizes the two high-risk entry-point scripts that must parse the new heading format, surfacing a potential lstrip-vs-removeprefix bug that the baseline review missed. The release.py changes are low-risk leaf nodes, and the CHANGES.md bulk rename is mechanical and deprioritized." + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 5, + "actionability": 6, + "efficiency": 6, + "overall": 5.4 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 8, + "efficiency": 8, + "overall": 8.0 + }, + "reasoning": "The flow-guided review benefits significantly from the review plan's risk annotations and topological ordering. It correctly identifies the two check scripts as high-risk entry points deserving priority attention, while the baseline treats all script changes with uniform medium severity. The flow-guided review caught a concrete Python gotcha (lstrip strips individual characters, not a prefix string) by focusing on the extraction scripts first, which the baseline missed entirely. The dependency graph also provided useful context for understanding the release.py call chain. The baseline review wastes equal attention on the mechanical CHANGES.md rename, while the flow-guided review efficiently deprioritizes it.", + "winner": "flow_guided" + }, + "timestamp": "2026-03-30T20:15:00.000000+00:00" +} diff --git a/evals/pydantic__pydantic__12985.json b/evals/pydantic__pydantic__12985.json new file mode 100644 index 0000000..a28ecca --- /dev/null +++ b/evals/pydantic__pydantic__12985.json @@ -0,0 +1,108 @@ +{ + "pr": "pydantic/pydantic#12985", + "title": "Box large fields in CombinedValidator/CombinedSerializer to reduce enum size", + "baseline_review": { + "comments": [ + { + "file": "pydantic-core/src/serializers/type_serializers/union.rs", + "line": 130, + "severity": "minor", + "comment": "The `match &*self.discriminator` dereference through Box is correct and necessary after boxing. However, verify that all other match sites and method calls on `self.discriminator` throughout the file (not just the visible diff) also use `&*self.discriminator` or `.as_ref()` -- a missed callsite would fail to compile, but it is worth confirming no partial application occurred." + }, + { + "file": "pydantic-core/src/serializers/filter.rs", + "line": 36, + "severity": "minor", + "comment": "Boxing `AHashSet` inside `Option` means `SchemaFilter` now stores `Option>>` for both `include` and `exclude`. Since `AHashSet` is already heap-allocated (it stores data on the heap internally), the Box adds a second pointer indirection. This is a deliberate tradeoff for enum variant size reduction, but any hot-path code that iterates the filter sets on every field will pay an extra cache miss per access. Confirm via benchmarks that serialization throughput is not measurably affected." + }, + { + "file": "pydantic-core/src/validators/enum_.rs", + "line": 143, + "severity": "minor", + "comment": "The `LiteralLookup` is boxed at construction time with `Box::new(LiteralLookup::new(py, expected.into_iter())?)`. This is fine, but note that `LiteralLookup::new` first allocates the struct on the stack and then `Box::new` copies it to the heap. For very large lookup tables, this temporary stack allocation could be significant. In practice Rust often optimizes this away (NRVO), but it is worth being aware of for extremely large enums." + }, + { + "file": "pydantic-core/src/validators/literal.rs", + "line": 165, + "severity": "minor", + "comment": "The `LiteralValidator` now stores `Box>>`. The diff is truncated and does not show the updated constructor or any callsites that access `self.lookup`. Ensure that all pattern matches and method calls on `self.lookup` throughout `literal.rs` compile correctly -- particularly any `self.lookup.find(...)` calls which should auto-deref through Box but may need verification." + }, + { + "file": "pydantic-core/src/serializers/type_serializers/literal.rs", + "line": 98, + "severity": "nit", + "comment": "Boxing `expected_int` and `expected_str` inside `LiteralSerializer` is consistent with the overall approach. Since `LiteralSerializer` is wrapped in `Arc::new(Self { ... })`, it is already heap-allocated. The Box adds another layer of indirection. This is fine for enum size reduction but means accessing these fields is now Arc -> struct -> Box -> AHashSet. For a serializer that may be called millions of times, this triple indirection is worth benchmarking." + }, + { + "file": "pydantic-core/src/serializers/type_serializers/union.rs", + "line": 121, + "severity": "positive", + "comment": "Boxing the `Discriminator` in `TaggedUnionSerializer` is the single highest-impact change in this PR -- `TaggedUnionValidator` at 448 bytes was the largest variant inflating the entire `CombinedValidator` enum. Reducing the discriminator to an 8-byte pointer is a clean, idiomatic Rust approach to this problem." + } + ], + "summary": "This PR applies a well-known Rust optimization -- boxing large enum variant fields to shrink the overall enum size. The changes are mechanical and consistent across 6 files, correctly updating constructors, return types, and match expressions. The main concern is whether the added pointer indirection on hot paths (filter sets, literal lookups) has measurable performance impact, which should be validated with benchmarks." + }, + "flow_guided_review": { + "comments": [ + { + "file": "pydantic-core/src/serializers/type_serializers/union.rs", + "line": 121, + "severity": "positive", + "comment": "The `TaggedUnionSerializer.discriminator` boxing is the keystone change -- the review plan is empty, but the PR description identifies `TaggedUnionValidator` at 448 bytes as the largest variant. Boxing `Discriminator` (which contains `LookupPaths` with nested Vecs) to 8 bytes is the single change with the most impact on `CombinedSerializer` size reduction." + }, + { + "file": "pydantic-core/src/serializers/filter.rs", + "line": 36, + "severity": "minor", + "comment": "The `SchemaFilter` boxing affects every serializer that uses include/exclude filtering (model, dataclass, typed-dict serializers). Since `SchemaFilter` is embedded in multiple `CombinedSerializer` variants, this change has a multiplicative effect on enum size reduction. However, it also means every include/exclude check during serialization now traverses an extra pointer. The filter is consulted per-field during serialization, so this is a hot path worth benchmarking." + }, + { + "file": "pydantic-core/src/validators/enum_.rs", + "line": 152, + "severity": "minor", + "comment": "The `EnumValidator` stores the boxed `LiteralLookup` and is parameterized over `T: EnumValidateValue`. Since `EnumValidator` is instantiated for multiple enum validation strategies (plain, int, str), each becomes a distinct `CombinedValidator` variant. Boxing the lookup in the base struct shrinks all of these variants simultaneously, which is efficient. Verify that the `EnumValidateValue` trait implementations do not destructure or move out of `self.lookup` in ways that conflict with the Box wrapper." + }, + { + "file": "pydantic-core/src/validators/literal.rs", + "line": 165, + "severity": "minor", + "comment": "The diff is truncated after line 272, cutting off the `LiteralValidator` constructor. The type declaration shows `lookup: Box>>` but we cannot see the corresponding `Box::new(...)` in the constructor. This is likely correct (the PR would not compile otherwise) but the truncation prevents full review of this file's changes." + }, + { + "file": "pydantic-core/src/serializers/type_serializers/literal.rs", + "line": 88, + "severity": "minor", + "comment": "Both `expected_int: Box>` and `expected_str: Box>` are boxed in `LiteralSerializer`, which is then stored inside `Arc`. The double heap indirection (Arc -> Box -> AHashSet) is acceptable because this struct lives in `CombinedSerializer` enum variants where size matters. The Arc ensures the serializer is shared across threads, and the Box shrinks the inline footprint. This is a correct layering of concerns." + }, + { + "file": "pydantic-core/src/serializers/type_serializers/union.rs", + "line": 130, + "severity": "nit", + "comment": "The `match &*self.discriminator` pattern is the idiomatic way to match through a Box in Rust. An alternative is `match self.discriminator.as_ref()` which some style guides prefer for readability. Both are equivalent -- this is purely a style choice and the current form is fine." + } + ], + "summary": "With an empty review plan, the flow-guided review focused on how the boxed fields propagate through the type system: TaggedUnionSerializer's discriminator is the highest-leverage change, SchemaFilter boxing has a multiplicative effect across multiple serializer variants, and EnumValidator boxing shrinks all enum validation variants simultaneously. The changes are mechanically correct and follow idiomatic Rust patterns for enum size optimization, with the main risk being unmeasured performance impact from added indirection on hot paths." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 5, + "efficiency": 7, + "overall": 5.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 5, + "efficiency": 7, + "overall": 6.0 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews correctly identify this as a mechanical Rust enum optimization with consistent boxing across 6 files. The review plan was entirely empty (zero steps, clusters, dependencies), which limited the flow-guided review's structural advantage. Despite this, the flow-guided review provided better cross-cutting analysis: it identified that SchemaFilter boxing has a multiplicative effect because the struct appears in multiple CombinedSerializer variants, and that EnumValidator's generic parameterization means boxing the lookup shrinks multiple CombinedValidator variants simultaneously. The baseline review treated each file independently and flagged similar concerns (extra indirection, benchmark need) but without connecting the changes to their combined effect on enum sizing. Both reviews noted the truncated diff in literal.rs and the match dereference pattern in union.rs. The margin is modest because the PR is a straightforward, low-risk mechanical change with limited complexity to differentiate on." + } +} diff --git a/evals/pytest-dev__pytest__14310.json b/evals/pytest-dev__pytest__14310.json new file mode 100644 index 0000000..a45dc15 --- /dev/null +++ b/evals/pytest-dev__pytest__14310.json @@ -0,0 +1,108 @@ +{ + "pr": "pytest-dev/pytest#14310", + "title": "recwarn: improved `warns(match=...)` error message on regex mismatch (closes #11225)", + "baseline_review": { + "comments": [ + { + "file": "src/_pytest/recwarn.py", + "line": "330-335", + "severity": "medium", + "comment": "The escape hint logic compares `self.match_expr == str(w.message)` to detect when the user likely forgot `re.escape()`. This is a strict equality check, which means it only triggers when the match expression is exactly the warning message string. If the user's regex is a substring or slightly different, the hint won't appear. Consider using a looser heuristic, such as checking whether `re.escape(self.match_expr)` would match when the raw `self.match_expr` does not." + }, + { + "file": "src/_pytest/recwarn.py", + "line": "337-339", + "severity": "low", + "comment": "The new error message no longer mentions the expected warning type: `f\"Regex pattern did not match any of the {len(self)} warnings emitted.\"` Previously the message included `self.expected_warning` which helped users distinguish between 'wrong type' and 'wrong match' failures. Consider adding the warning type back, e.g., `Regex pattern did not match any of the {len(self)} {self.expected_warning} warnings emitted.`" + }, + { + "file": "src/_pytest/recwarn.py", + "line": "339", + "severity": "low", + "comment": "The `{self.match_expr!r}` uses repr formatting, which wraps the regex in quotes. The old format did not use repr. This is an improvement for readability (makes it clear where the regex starts and ends), but note this is a user-facing output change that could break any downstream tooling or scripts that parse pytest failure messages." + }, + { + "file": "testing/test_recwarn.py", + "line": "445-452", + "severity": "medium", + "comment": "The `test_warns_match_re_escape_hint` test verifies the hint appears when the match expression equals the warning message. However, there is no test for the case where `match_expr` is not a string (it can be a compiled regex pattern). The `isinstance(self.match_expr, str)` guard in the implementation handles this, but a test confirming no hint is shown for compiled patterns would improve coverage." + }, + { + "file": "testing/test_recwarn.py", + "line": "453-459", + "severity": "low", + "comment": "The `test_warns_match_re_escape_hint_no_false_positive` test checks that the hint does not appear when the warning type does not match (DeprecationWarning vs UserWarning). The diff appears truncated at line 168 of the diff (`assert \"re.escape()\" not in str(excinfo.`), but assuming it ends with `.value)` this is correct. Good edge case coverage." + }, + { + "file": "doc/en/how-to/capture-warnings.rst", + "line": "366-368", + "severity": "low", + "comment": "The documentation doctest was updated to match the new error message format. The use of `...` ellipsis in the expected output is appropriate for doctests. Good that this was kept in sync with the implementation change." + } + ], + "summary": "This PR improves the `pytest.warns(match=...)` error message by replacing the misleading 'DID NOT WARN' with a more specific 'Regex pattern did not match' message, and adds a helpful `re.escape()` hint when the match expression literally equals the warning text. The implementation is clean and well-tested, though the escape hint heuristic could be more general, and the new message drops the expected warning type information that was previously included." + }, + "flow_guided_review": { + "comments": [ + { + "file": "src/_pytest/recwarn.py", + "line": "330-335", + "severity": "high", + "comment": "CORE LOGIC (WarningsChecker.__exit__): The escape hint detection iterates over all captured warnings with `any(self.match_expr == str(w.message) for w in self if issubclass(w.category, self.expected_warning))`. This filters by `self.expected_warning` correctly -- but note that `self.matches(w)` (called on line 322) already checks both the type AND the regex. The escape hint only fires in the `elif not any(self.matches(w) ...)` branch, meaning at least one warning of the right type was emitted but no regex matched. The type filter in the hint is therefore partially redundant but serves to avoid false positives when non-matching-type warnings happen to have the exact same message text. This is the right design, as confirmed by `test_warns_match_re_escape_hint_no_false_positive`." + }, + { + "file": "src/_pytest/recwarn.py", + "line": "337-339", + "severity": "medium", + "comment": "ENTRY POINT (WarningsChecker): The new error message `f\"Regex pattern did not match any of the {len(self)} warnings emitted.\"` counts ALL captured warnings via `len(self)`, not just those of the expected type. If a user expects `UserWarning` and 5 warnings are emitted (3 UserWarning, 2 DeprecationWarning), the message says '5 warnings' which could be confusing since only 3 are relevant. Consider counting only type-matching warnings: `sum(1 for w in self if issubclass(w.category, self.expected_warning))`." + }, + { + "file": "testing/test_recwarn.py", + "line": "429-437", + "severity": "medium", + "comment": "FLOW: `test_warns_match_failure_message_detail` validates the new message format and asserts the old 'DID NOT WARN' text is absent. Following the plan's dependency chain from test entry points back to WarningsChecker, this test exercises the `elif not any(self.matches(w))` branch. However, it does not verify the count in the message (e.g., '1 warnings emitted') or the repr formatting of the regex. Adding assertions for the full message structure would catch regressions more precisely." + }, + { + "file": "testing/test_recwarn.py", + "line": "439-444", + "severity": "medium", + "comment": "FLOW: `test_warns_match_re_escape_hint` tests the happy path for the re.escape hint with `match=\"foo (bar)\"` and `warnings.warn(\"foo (bar)\")`. The parentheses in `foo (bar)` are regex metacharacters, making this a good test case. However, a complementary test where the match expression contains regex metacharacters but does NOT literally equal the warning message (e.g., `match=\"foo.*bar\"` with warning `\"fooXbar\"`) would verify the hint correctly does NOT appear when the regex is intentionally a pattern." + }, + { + "file": "src/_pytest/recwarn.py", + "line": "330", + "severity": "low", + "comment": "DEPENDENCY: The `isinstance(self.match_expr, str)` guard ensures the hint logic only runs for string match expressions, not compiled regex objects. Per the plan, `warns()` passes `match_expr` through to `WarningsChecker`, and `match_expr` can be `str | Pattern[str]`. This guard is correct and necessary. No test covers the compiled-pattern case explicitly -- consider adding one for completeness." + }, + { + "file": "src/_pytest/recwarn.py", + "line": "139-143", + "severity": "low", + "comment": "DOCSTRING (warns function): The docstring example was updated to show the new error format with `...` ellipsis matching. This is consistent with the docs change in `capture-warnings.rst`. The plan shows `warns` (order 13) calls `WarningsChecker` (order 15), confirming the docstring change correctly reflects the downstream behavioral change." + } + ], + "summary": "The flow-guided analysis reveals the core risk is in WarningsChecker.__exit__ where the new escape-hint logic and message formatting interact with the warning type filtering and count. The most actionable finding is that `len(self)` counts all warnings regardless of type, which could produce a misleading count when multiple warning types are present. Test coverage is good but could be strengthened with compiled-pattern and intentional-regex-pattern negative cases." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identified a concrete bug the baseline missed: `len(self)` counts all captured warnings regardless of type, producing a misleading count when warnings of non-matching types are present. The flow-guided approach traced the dependency from test entry points through `warns()` to `WarningsChecker.__exit__()`, understanding the branching logic (no warnings of type vs. type matched but regex didn't) to evaluate the escape hint's correctness more rigorously. The baseline review noted the missing warning type in the message but framed it as a UX preference rather than identifying the count inaccuracy as a potential bug. The flow-guided review also provided more targeted test gap analysis by following the plan's risk annotations and call chains." + } +} diff --git a/evals/python__cpython__146622.json b/evals/python__cpython__146622.json new file mode 100644 index 0000000..0492b72 --- /dev/null +++ b/evals/python__cpython__146622.json @@ -0,0 +1,102 @@ +{ + "pr": "python/cpython#146622", + "title": "[3.14] gh-146556: Fix infinite loop in annotationlib.get_annotations() on circular __wrapped__", + "baseline_review": { + "comments": [ + { + "file": "Lib/annotationlib.py", + "line": 1042, + "severity": "medium", + "comment": "Using `id()` for cycle detection is correct for object identity but has a subtle risk: if an object in the chain is garbage-collected and a new object is allocated at the same memory address, `id()` could produce a false positive. In practice this is unlikely here since the loop holds a reference to `unwrap` (keeping it alive), but it is worth noting that this approach is safe only because the traversal retains references to earlier objects on the stack." + }, + { + "file": "Lib/annotationlib.py", + "line": 1039, + "severity": "low", + "comment": "The variable name `_seen_ids` uses a leading underscore, suggesting it is private/internal. This is fine as a local variable but the naming convention is slightly unusual for a local -- a plain `seen_ids` or `seen` would be more conventional for a function-scoped set." + }, + { + "file": "Lib/annotationlib.py", + "line": 1043, + "severity": "low", + "comment": "The two branches (`__wrapped__` and `functools.partial`) contain nearly identical cycle-detection logic (get candidate, check id, add to set, reassign). Consider extracting a small helper or at minimum adding the candidate to `_seen_ids` and checking it in a unified way to reduce duplication and risk of the two paths diverging in future maintenance." + }, + { + "file": "Lib/test/test_annotationlib.py", + "line": 649, + "severity": "medium", + "comment": "The test `test_eval_str_wrapped_cycle_self` covers self-referential cycles and `test_eval_str_wrapped_cycle_mutual` covers mutual cycles, but there is no test for a cycle through `functools.partial`. Since the fix applies cycle detection to both `__wrapped__` and `functools.partial.func` paths, a test with a `functools.partial` cycle (e.g., `p = functools.partial(f); p.func = p`) would verify the second code path." + }, + { + "file": "Lib/test/test_annotationlib.py", + "line": 666, + "severity": "low", + "comment": "The `test_eval_str_wrapped_chain_no_cycle` test only verifies a chain of depth 1 (outer -> inner). A deeper chain (e.g., outer -> middle -> inner) would provide stronger confidence that the cycle-detection set accumulates correctly across multiple iterations without false positives." + } + ], + "summary": "The fix correctly introduces id-based cycle detection to prevent infinite loops when traversing `__wrapped__` and `functools.partial` chains in `get_annotations()`. The implementation is sound and mirrors `inspect.unwrap()`, but test coverage is missing the `functools.partial` cycle path and the duplicated detection logic in the two branches could benefit from consolidation." + }, + "flow_guided_review": { + "comments": [ + { + "file": "Lib/annotationlib.py", + "line": 1034, + "severity": "medium", + "comment": "Step 1 (get_annotations, entry point, high risk): The cycle detection is scoped inside the `if unwrap is not None` block, which is correct -- it only applies when there is an object to unwrap. However, the `_seen_ids` set is initialized with `{id(unwrap)}` before entering the loop, meaning the initial object is pre-registered. This is essential for detecting self-referential cycles (f.__wrapped__ = f) on the very first iteration. Good design choice that directly addresses the reported bug." + }, + { + "file": "Lib/annotationlib.py", + "line": 1043, + "severity": "medium", + "comment": "Step 1 continued: The `__wrapped__` branch introduces a `candidate` variable, checks it against `_seen_ids`, and breaks on cycle detection. The `break` exits the `while True` loop and falls through to `if hasattr(unwrap, '__globals__')`, meaning on cycle detection we use whatever `__globals__` the last valid `unwrap` object has. This is the correct fallback behavior -- it mirrors `inspect.unwrap()` which also stops on cycles. However, the comment on line 1036 says 'mirroring the approach of inspect.unwrap()' but `inspect.unwrap()` raises a `ValueError` on cycles rather than silently stopping. The behavior here is intentionally different (silent stop vs. exception), and the comment could be more precise." + }, + { + "file": "Lib/annotationlib.py", + "line": 1049, + "severity": "medium", + "comment": "Step 1 continued: The `functools.partial` branch duplicates the exact same cycle-detection pattern. Since both branches call into `_seen_ids` with the same logic (check, add, reassign), a refactoring into a helper like `_advance(candidate, seen)` would reduce duplication. More importantly, the two branches share the same `_seen_ids` set, which means a mixed chain (wrapped -> partial -> wrapped) is correctly detected across both paths -- this is a subtle correctness property worth a test case." + }, + { + "file": "Lib/test/test_annotationlib.py", + "line": 649, + "severity": "high", + "comment": "Steps 3-4 (test_eval_str_wrapped_cycle_self, test_eval_str_wrapped_cycle_mutual): These tests cover the `__wrapped__` cycle detection path well. However, there is no test exercising the `functools.partial` cycle detection path at all. Since the plan identifies `get_annotations` as high risk and the fix modifies two independent code paths, both paths need test coverage. Add a test where a `functools.partial` object's `.func` creates a cycle." + }, + { + "file": "Lib/test/test_annotationlib.py", + "line": 666, + "severity": "medium", + "comment": "Step 7 (test_eval_str_wrapped_chain_no_cycle): This test verifies non-cyclic chains still work, but only at depth 1. Given that the cycle detection accumulates ids across iterations, a chain of depth 2+ would verify no false positives from the set growing. Also missing: a mixed chain test (wrapped + partial in sequence) that exercises the shared `_seen_ids` set across both branch types." + }, + { + "file": "Lib/annotationlib.py", + "line": 1036, + "severity": "low", + "comment": "Step 1: The comment says 'mirroring the approach of inspect.unwrap()' but `inspect.unwrap()` raises `ValueError` when a cycle is detected (or when the chain exceeds a `stop` limit), whereas this code silently breaks. The behavior is reasonable for `get_annotations` (raising would break callers that just want annotations), but the comment should clarify that only the detection strategy (not the response) mirrors `inspect.unwrap()`." + } + ], + "summary": "Following the flow from `get_annotations` (high-risk entry point) through both the `__wrapped__` and `functools.partial` unwrap branches, the cycle detection is correctly implemented with a shared `_seen_ids` set that handles mixed chains. The key gap is that the `functools.partial` cycle path has zero test coverage, and the comment claiming to mirror `inspect.unwrap()` is slightly misleading since the actual cycle response (silent break vs. ValueError) differs." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leverages the plan's identification of `get_annotations` as a high-risk entry point to trace the execution path through both unwrap branches systematically. It identifies that the shared `_seen_ids` set correctly handles mixed __wrapped__/partial chains -- a subtle cross-branch correctness property the baseline misses entirely. It also catches the misleading comment about mirroring `inspect.unwrap()` by understanding the actual behavior difference (silent break vs. ValueError). Both reviews identify the missing functools.partial test coverage, but the flow-guided review explains why it matters in terms of the two independent code paths modified in the fix. The baseline review's point about id() and GC is technically interesting but practically irrelevant here, spending review budget on a non-issue." + } +} \ No newline at end of file diff --git a/evals/python__cpython__146630.json b/evals/python__cpython__146630.json new file mode 100644 index 0000000..c62abfb --- /dev/null +++ b/evals/python__cpython__146630.json @@ -0,0 +1,137 @@ +{ + "pr": { + "url": "https://github.com/python/cpython/pull/146630", + "owner": "python", + "repo": "cpython", + "number": 146630, + "title": "[3.14] gh-146416: Emscripten: Improve standard stream handling in node_entry.mjs", + "files_changed": 5, + "additions": 255, + "deletions": 0, + "language": "javascript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "Platforms/emscripten/streams.mjs", + "line": 29, + "severity": "major", + "comment": "The module declares `let FS;` and `const DEVOPS = {};` / `const DEVS = {};` at module scope, but `FS` is only assigned inside `initializeStreams()`. Meanwhile, `readWriteHelper` at line 76 references `Module.ERRNO_CODES` without `Module` being imported or declared anywhere in this file. This appears to be a reference to the Emscripten Module object which is only available at runtime, but there is no explicit import or parameter passing -- it relies on `Module` being a global. If the module system or bundler isolates scopes, this will throw a ReferenceError. Either pass Module explicitly through initializeStreams and thread it to helpers, or document the implicit global dependency." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 53, + "severity": "minor", + "comment": "The `handleEAGAIN` function uses `while (true)` with a `syncSleep(10)` retry loop. If the underlying file descriptor is permanently in a non-blocking error state, this will spin indefinitely (syncSleep returns true, so the loop continues). Consider adding a maximum retry count to prevent hanging the process in pathological cases." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 37, + "severity": "minor", + "comment": "The `syncSleep` function allocates a `SharedArrayBuffer` via `new WebAssembly.Memory({ shared: true, initial: 1, maximum: 1 })` at module load time. This requires `SharedArrayBuffer` to be available, which in turn requires cross-origin isolation headers (COOP/COEP) in browser contexts. While this is Node.js-specific code, the comment at the top of the file says it is a 'pared down version' of Pyodide's streams.ts which does run in browsers. A comment clarifying that this module is Node-only would prevent confusion." + }, + { + "file": "Platforms/emscripten/node_entry.mjs", + "line": 43, + "severity": "minor", + "comment": "The `onRuntimeInitialized` callback calls `initializeStreams(Module.FS)`. If Emscripten's Module.FS is not yet fully initialized at the point onRuntimeInitialized fires, the stream replacement could operate on an incomplete filesystem. The Emscripten docs state that FS should be available at this point, but a defensive null check on Module.FS before calling initializeStreams would make the initialization more robust." + }, + { + "file": "Platforms/emscripten/__main__.py", + "line": 521, + "severity": "minor", + "comment": "The new `shutil.copy` for `streams.mjs` is placed after the `node_entry.mjs` copy. This is correct since node_entry.mjs imports from streams.mjs, so both need to be present. However, there is no error handling if the source file does not exist -- a missing streams.mjs would produce a confusing traceback. This matches the existing pattern for node_entry.mjs though, so it is at least consistent." + }, + { + "file": "Platforms/emscripten/node_entry.mjs", + "line": 56, + "severity": "nit", + "comment": "The whitespace fix `catch(e)` -> `catch (e)` is a welcome style normalization but is unrelated to the stream handling changes. In a cherry-pick to a release branch (3.14), minimizing unrelated changes reduces merge conflict risk." + } + ], + "summary": "This PR adds a well-documented streams.mjs module that replaces Emscripten's default standard stream devices with improved versions that correctly report isatty, support window size ioctls, and remove unnecessary buffering layers. The main concern is the implicit dependency on a global `Module` object in streams.mjs, which could break under different bundling or module isolation configurations." + }, + "flow_guided_review": { + "comments": [ + { + "file": "Platforms/emscripten/node_entry.mjs", + "line": 43, + "severity": "major", + "comment": "ENTRY POINT / HIGH RISK (onRuntimeInitialized): This is the critical integration point where initializeStreams is called with Module.FS. The review plan identifies this as high-risk because it is the entry point that wires the new streams module into the Emscripten runtime. The timing matters -- onRuntimeInitialized fires after the wasm module is loaded but before user code runs. If initializeStreams throws (e.g., because FS.createDevice or FS.unlink fails on a missing /dev/stdin), it will abort the entire Python runtime initialization with no recovery path. A try/catch wrapper with a warning fallback ('stream replacement failed, falling back to default streams') would make the initialization more resilient." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 76, + "severity": "major", + "comment": "DEPENDENCY: readWriteHelper references `Module.ERRNO_CODES` at line 76 and `FS.ErrnoError` at line 77, but Module is never imported or passed as a parameter to this module. The flow plan shows readWriteHelper is called by the stream_ops.read and stream_ops.write methods, which are invoked by Emscripten's VFS layer after initializeStreams registers them. At that point, Module is presumably a global set by Emscripten. However, initializeStreams only receives `FS` as a parameter (not Module), creating an asymmetry: FS is explicitly passed but Module is implicitly global. This is fragile -- if Emscripten changes how Module is scoped, all read/write operations will break silently with ReferenceErrors." + }, + { + "file": "Platforms/emscripten/__main__.py", + "line": 521, + "severity": "minor", + "comment": "ENTRY POINT (configure_emscripten_python): The review plan identifies this as high-risk because it is the build-time entry point that deploys streams.mjs alongside node_entry.mjs. The ordering is correct -- streams.mjs is copied before node_entry.mjs is processed for the exec_script. However, this means the build now has a new required file dependency: if someone updates node_entry.mjs to import from streams.mjs but forgets to update __main__.py to copy it, the runtime will fail with a module-not-found error. A comment linking these two copy operations would help maintainers." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 53, + "severity": "minor", + "comment": "DEPENDENCY (handleEAGAIN -> syncSleep): The flow plan shows handleEAGAIN is called by readWriteHelper, which is the central error-handling path for all stream I/O. The syncSleep implementation uses Atomics.wait on a SharedArrayBuffer, which blocks the main thread for 10ms per retry. In a single-threaded Emscripten environment, this is acceptable, but if Python's Emscripten build ever enables pthreads (WASM threads), blocking the main thread could cause deadlocks with worker threads waiting for the main thread. The comment 'In case for some reason we fail to sleep, propagate the error' handles the Atomics.wait failure case, which is good." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 114, + "severity": "minor", + "comment": "INDEPENDENT FLOW (ioctl_tiocgwinsz in TTY_OPS): This is one of the key improvements listed in the module docstring -- supporting terminal window size queries. The implementation delegates to devops.ioctl_tiocgwinsz via optional chaining (?.), which means non-TTY streams gracefully return undefined. However, the Emscripten FS layer expects this ioctl to return an array [rows, cols] or falsy. Returning undefined (from the optional chain) should be falsy and thus safe, but the contract is implicit." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 169, + "severity": "minor", + "comment": "INDEPENDENT FLOW (NodeReader/NodeWriter classes): The review plan identifies these constructors as separate entry points. NodeReader stores fd and isatty in the constructor, and its read method uses fs.readSync. NodeWriter similarly uses fs.writeSync. Both classes implement fsync by delegating to a nodeFsync helper. The class design cleanly separates read and write concerns, but neither class validates that the fd is valid at construction time -- an invalid fd would only surface as an error on first read/write, potentially after streams have been registered in the FS." + }, + { + "file": "Platforms/emscripten/streams.mjs", + "line": 1, + "severity": "positive", + "comment": "The module-level docstring is excellent -- it clearly enumerates the four specific deficiencies in Emscripten's default streams that this module fixes (incorrect isatty, missing ttygetwinsize ioctl, extra buffering layer, slow character-based handler). This makes it easy for future maintainers to understand why this custom implementation exists and what upstream improvements would make it unnecessary." + } + ], + "summary": "The flow-guided analysis reveals that the critical risk in this PR is the implicit dependency on a global `Module` object in streams.mjs, which is used in the readWriteHelper error-handling path but never explicitly passed -- creating fragility if Emscripten's scoping changes. The integration point in onRuntimeInitialized lacks error recovery, meaning any failure in stream replacement will abort the entire Python runtime rather than falling back to Emscripten's default (suboptimal but functional) streams." + }, + "judgment": { + "criteria": { + "completeness": { + "baseline": 7, + "flow_guided": 8, + "rationale": "Both reviews cover the key files and changes. The flow-guided review additionally identifies the NodeReader/NodeWriter fd validation gap and the build-time file dependency between __main__.py and node_entry.mjs that the baseline misses." + }, + "flow_awareness": { + "baseline": 4, + "flow_guided": 8, + "rationale": "The baseline reviews each file in isolation, noting individual concerns without tracing how they connect. The flow-guided review traces the full initialization chain: __main__.py copies files -> node_entry.mjs wires onRuntimeInitialized -> initializeStreams replaces FS devices -> readWriteHelper uses implicit Module global. It also distinguishes entry points from independent flows (NodeReader/NodeWriter, ioctl)." + }, + "risk_identification": { + "baseline": 6, + "flow_guided": 8, + "rationale": "The baseline correctly identifies the implicit Module global and the EAGAIN infinite loop risk. The flow-guided review goes deeper by connecting the Module dependency to the specific call chain (stream_ops -> readWriteHelper -> Module.ERRNO_CODES) and identifying that onRuntimeInitialized failure would be catastrophic with no fallback, plus the build-time coupling risk." + }, + "actionability": { + "baseline": 6, + "flow_guided": 7, + "rationale": "The baseline suggests adding retry limits and null checks. The flow-guided review provides more specific recommendations: try/catch with fallback in onRuntimeInitialized, passing Module explicitly alongside FS in initializeStreams, and adding a comment linking the two shutil.copy operations in __main__.py." + }, + "efficiency": { + "baseline": 7, + "flow_guided": 7, + "rationale": "Both reviews stay focused on the actual changes without introducing off-topic concerns. The flow-guided review is slightly more verbose but each comment adds analytical depth. The baseline's nit about the whitespace change is efficient but low-value." + } + }, + "overall": { + "baseline": 6.0, + "flow_guided": 7.6, + "winner": "flow_guided", + "rationale": "The flow-guided review is stronger because it traces the initialization chain from build-time file copying through runtime stream replacement, revealing that the implicit Module global creates a fragile dependency in the most critical code path (error handling during I/O). It also identifies that onRuntimeInitialized lacks error recovery, making stream replacement failures catastrophic rather than gracefully degradable. The baseline correctly spots the same Module global issue but treats it as an isolated concern rather than connecting it to the broader initialization flow and failure modes." + } + } +} diff --git a/evals/remix-run__remix__11197.json b/evals/remix-run__remix__11197.json new file mode 100644 index 0000000..2ee1890 --- /dev/null +++ b/evals/remix-run__remix__11197.json @@ -0,0 +1,123 @@ +{ + "pr": { + "url": "https://github.com/remix-run/remix/pull/11197", + "owner": "remix-run", + "repo": "remix", + "number": 11197, + "title": "Refactor the unpkg demo layout conventions", + "files_changed": 22, + "additions": 337, + "deletions": 50, + "language": "typescript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "demos/unpkg/app/controllers/home/controller.ts", + "line": 11, + "severity": "minor", + "comment": "The `satisfies BuildAction<'GET', typeof routes.home>` constraint references `routes.home` from `../../routes.ts`, but the diff does not include the contents of `routes.ts` (only `router.ts` is mentioned in the README). Verify that the `routes` export actually defines a `home` key; a mismatch would cause a type error." + }, + { + "file": "demos/unpkg/app/controllers/home/controller.ts", + "line": 8, + "severity": "minor", + "comment": "The `homeController` object uses `let` instead of `const`. Since this controller object is never reassigned, it should be declared with `const` for clarity and to prevent accidental mutation." + }, + { + "file": "demos/unpkg/app/controllers/home/page.ts", + "line": 3, + "severity": "nit", + "comment": "The `HomePage` function builds a static HTML template with no dynamic data. If the home page content ever needs to change per-request (e.g., showing package counts or status), this function signature would need parameters. For now the static approach is fine, but consider adding a comment noting it is intentionally static." + }, + { + "file": "demos/unpkg/app/controllers/package-browser/breadcrumb.ts", + "line": 1, + "severity": "minor", + "comment": "The import was changed from `'../ui/render.ts'` to `'remix/html-template'`. This is the right direction per the PR description (removing the re-export), but ensure all other files that previously imported `html` from `../ui/render.ts` have been updated consistently." + }, + { + "file": "demos/unpkg/.gitignore", + "line": 1, + "severity": "minor", + "comment": "The new `.gitignore` ignores `tmp/*` but keeps `tmp/.gitkeep`. The README says tarball cache is stored under `tmp/`. Verify that the `tarball-cache.ts` module correctly resolves to this `tmp/` path relative to the demo root, not relative to the process working directory, to avoid writing cache files outside the demo." + }, + { + "file": "demos/unpkg/README.md", + "line": 19, + "severity": "nit", + "comment": "The README now references `app/router.ts` but the controller imports reference `../../routes.ts`. Clarify whether the file is named `router.ts` or `routes.ts` -- the inconsistency between README and import paths could confuse contributors." + } + ], + "summary": "This PR cleanly refactors the unpkg demo into a controller-first layout with separate home and package-browser features, removing the `html` re-export in favor of direct Remix imports. The main concerns are minor: a `let` vs `const` inconsistency, potential naming confusion between `router.ts` and `routes.ts`, and ensuring the tarball cache path resolves correctly under the new `tmp/` directory." + }, + "flow_guided_review": { + "comments": [ + { + "file": "demos/unpkg/app/controllers/home/controller.ts", + "line": 8, + "severity": "minor", + "comment": "Entry point (step 1, high risk): The `homeController.handler` is the new root route handler. It delegates to `render()` and `HomePage()`. The `let` declaration should be `const` since this is never reassigned. The `satisfies BuildAction<'GET', typeof routes.home>` type constraint depends on the routes definition exporting a `home` key -- verify this exists in `routes.ts`." + }, + { + "file": "demos/unpkg/app/controllers/render.ts", + "line": 6, + "severity": "major", + "comment": "Internal node (step 18, high risk, 4 callers): The `render` function is called by both entry points (home handler, package-browser handler) and both sub-renderers (error, directory, file-content). With only 3 lines of implementation, this is a thin wrapper around `createDocument`. Since it is the shared chokepoint for all HTML responses, any bug here (e.g., missing content-type header, incorrect encoding) would break every route. Verify it sets `Content-Type: text/html; charset=utf-8` on the response." + }, + { + "file": "demos/unpkg/app/controllers/ui/document.ts", + "line": 5, + "severity": "minor", + "comment": "Leaf node (step 23): `createDocument` builds the shared HTML document shell. It was extracted from the old `ui/render.ts`. Ensure the styles and icons imports that were previously co-located in `ui/render.ts` are properly relocated to dedicated modules as described in the PR description, and that `createDocument` correctly references them." + }, + { + "file": "demos/unpkg/app/controllers/package-browser/controller.ts", + "line": 21, + "severity": "major", + "comment": "Entry point (step 2, high risk, 44 additions / 50 deletions): This is the largest change in the PR and the most complex handler. It calls 10+ functions including `parsePackagePath`, `fetchPackageMetadata`, `resolveVersion`, `fetchPackageContents`, and multiple render helpers. The diff is truncated so the full rewrite cannot be verified. Given the handler branches on directory-vs-file and handles redirects for semver ranges, confirm all error paths (InvalidPathError, PackageNotFoundError, VersionNotFoundError) still return proper HTTP status codes." + }, + { + "file": "demos/unpkg/app/utils/tarball-cache.ts", + "line": 13, + "severity": "minor", + "comment": "Leaf node (step 21, medium risk, 2 callers): `getTarballCacheKey` is called by both `fetchTarball` and `parsePackagePath`. Since both callers need consistent cache key formatting, ensure the key generation is deterministic -- e.g., scoped package names like `@remix-run/cookie` must produce valid filesystem paths (no unescaped `/` or `@` characters in the key)." + }, + { + "file": "demos/unpkg/app/controllers/package-browser/format-bytes.ts", + "line": 1, + "severity": "nit", + "comment": "Leaf node (step 17, medium risk due to multiple callers): `formatBytes` is a new utility used by both `renderDirectoryListing` and `renderFileContent`. At 6 lines this is straightforward, but verify edge cases: what does it return for 0 bytes, negative values, or non-finite numbers? A simple guard clause would prevent displaying 'NaN undefined' in the UI." + }, + { + "file": "demos/unpkg/app/controllers/package-browser/breadcrumb.ts", + "line": 1, + "severity": "minor", + "comment": "The import migration from `'../ui/render.ts'` to `'remix/html-template'` is part of the broader pattern of removing the `html` re-export. Following the dependency graph, all files that previously imported from `ui/render.ts` must be updated. The plan shows this file is consumed by directory and file-content renderers -- confirm the breadcrumb output is still correctly interpolated after the import change." + } + ], + "summary": "Following the call graph from both entry points (home and package-browser handlers) through the shared `render` chokepoint to the leaf document/cache/utility nodes reveals that the `render` function is the highest-impact shared dependency with 4 callers, and any issue there would break all routes. The package-browser controller rewrite is the riskiest change (largest diff, most callees, truncated in the diff) and deserves careful verification of error-handling paths and HTTP status codes." + }, + "review_plan": {"stats": {"totalSteps": 23, "totalAdditions": 103, "totalDeletions": 50, "independentFlows": 1, "filesChanged": 7}, "steps": [{"order": 1, "nodeId": "demos/unpkg/app/controllers/home/controller.ts::handler", "name": "handler", "file": "demos/unpkg/app/controllers/home/controller.ts", "lines": [8, 10], "type": "method", "changeType": "added", "additions": 3, "deletions": 0, "role": "entry_point", "risk": "high", "calledBy": [], "calls": ["demos/unpkg/app/controllers/render.ts::render", "demos/unpkg/app/controllers/home/page.ts::HomePage"], "riskReasons": ["entry_point"]}, {"order": 2, "nodeId": "demos/unpkg/app/controllers/package-browser/controller.ts::handler", "name": "handler", "file": "demos/unpkg/app/controllers/package-browser/controller.ts", "lines": [21, 74], "type": "method", "changeType": "modified", "additions": 44, "deletions": 50, "role": "entry_point", "risk": "high", "calledBy": [], "calls": ["demos/unpkg/app/utils/npm.ts::parsePackagePath", "demos/unpkg/app/utils/npm.ts::fetchPackageMetadata", "demos/unpkg/app/utils/npm.ts::isFullyResolvedVersion", "demos/unpkg/app/utils/npm.ts::resolveVersion", "demos/unpkg/app/utils/npm.ts::fetchPackageContents", "demos/unpkg/app/utils/npm.ts::getFileContent", "demos/unpkg/app/controllers/package-browser/error.ts::renderError", "demos/unpkg/app/controllers/package-browser/file-content.ts::renderFileContent", "demos/unpkg/app/utils/npm.ts::getFilesAtPath", "demos/unpkg/app/controllers/package-browser/directory.ts::renderDirectoryListing", "demos/unpkg/app/utils/npm.ts::InvalidPathError", "demos/unpkg/app/utils/npm.ts::PackageNotFoundError", "demos/unpkg/app/utils/npm.ts::VersionNotFoundError"], "riskReasons": ["large_diff", "entry_point"]}, {"order": 3, "nodeId": "demos/unpkg/app/controllers/home/page.ts::HomePage", "name": "HomePage", "file": "demos/unpkg/app/controllers/home/page.ts", "lines": [3, 28], "type": "function", "changeType": "added", "additions": 26, "deletions": 0, "role": "leaf", "risk": "medium", "calledBy": ["demos/unpkg/app/controllers/home/controller.ts::handler"], "calls": [], "riskReasons": ["moderate_diff"]}, {"order": 17, "nodeId": "demos/unpkg/app/controllers/package-browser/format-bytes.ts::formatBytes", "name": "formatBytes", "file": "demos/unpkg/app/controllers/package-browser/format-bytes.ts", "lines": [1, 6], "type": "function", "changeType": "added", "additions": 6, "deletions": 0, "role": "leaf", "risk": "medium", "calledBy": ["demos/unpkg/app/controllers/package-browser/directory.ts::renderDirectoryListing", "demos/unpkg/app/controllers/package-browser/file-content.ts::renderFileContent"], "calls": [], "riskReasons": ["multiple_callers"]}, {"order": 18, "nodeId": "demos/unpkg/app/controllers/render.ts::render", "name": "render", "file": "demos/unpkg/app/controllers/render.ts", "lines": [6, 8], "type": "function", "changeType": "added", "additions": 3, "deletions": 0, "role": "internal", "risk": "high", "calledBy": ["demos/unpkg/app/controllers/home/controller.ts::handler", "demos/unpkg/app/controllers/package-browser/error.ts::renderError", "demos/unpkg/app/controllers/package-browser/directory.ts::renderDirectoryListing", "demos/unpkg/app/controllers/package-browser/file-content.ts::renderFileContent"], "calls": ["demos/unpkg/app/controllers/ui/document.ts::createDocument"], "riskReasons": ["many_callers"]}, {"order": 21, "nodeId": "demos/unpkg/app/utils/tarball-cache.ts::getTarballCacheKey", "name": "getTarballCacheKey", "file": "demos/unpkg/app/utils/tarball-cache.ts", "lines": [13, 16], "type": "function", "changeType": "added", "additions": 4, "deletions": 0, "role": "leaf", "risk": "medium", "calledBy": ["demos/unpkg/app/utils/npm.ts::fetchTarball", "demos/unpkg/app/utils/npm.ts::parsePackagePath"], "calls": [], "riskReasons": ["multiple_callers"]}, {"order": 23, "nodeId": "demos/unpkg/app/controllers/ui/document.ts::createDocument", "name": "createDocument", "file": "demos/unpkg/app/controllers/ui/document.ts", "lines": [5, 21], "type": "function", "changeType": "added", "additions": 17, "deletions": 0, "role": "leaf", "risk": "low", "calledBy": ["demos/unpkg/app/controllers/render.ts::render"], "calls": [], "riskReasons": []}]}, + "judge": { + "baseline_scores": { + "completeness": 6, + "flow_awareness": 3, + "risk_identification": 5, + "actionability": 6, + "efficiency": 7, + "overall": 5.4 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + }, + "reasoning": "The flow-guided review significantly outperforms the baseline by leveraging the dependency graph to identify the `render` function as a critical shared chokepoint with 4 callers, and correctly prioritizing the package-browser controller rewrite as the highest-risk change due to its large diff size and 10+ callees. The baseline review catches surface-level issues (let vs const, naming inconsistency) but misses the architectural significance of shared nodes and cannot reason about how a bug in `render` would cascade to all routes. The flow-guided review's ordered traversal from entry points through internal nodes to leaves ensures systematic coverage and correctly identifies that `getTarballCacheKey` being called by two different npm utilities requires consistent key formatting. The baseline also fails to flag the truncated controller diff as a major review gap, while the flow-guided review explicitly calls out that the 44-addition/50-deletion rewrite cannot be fully verified.", + "winner": "flow_guided" + } +} diff --git a/evals/remix-run__remix__11201.json b/evals/remix-run__remix__11201.json new file mode 100644 index 0000000..dc2dd1b --- /dev/null +++ b/evals/remix-run__remix__11201.json @@ -0,0 +1,108 @@ +{ + "pr": "remix-run/remix#11201", + "title": "route-pattern: Matches return decoded values for params in pathname", + "baseline_review": { + "comments": [ + { + "file": "packages/route-pattern/src/lib/route-pattern/decode-uri.ts", + "line": 8, + "severity": "medium", + "comment": "The `tryDecodeURI` function uses a fast-path check (`source.includes('%')`) to skip `decodeURI` when there are no percent-encoded sequences, which is a good optimization. However, the JSDoc contains a typo: 'containts' should be 'contains'. More importantly, `decodeURI` intentionally does NOT decode reserved characters like `%2F` (`/`), `%3F` (`?`), `%23` (`#`), etc. If a param value contains a literal encoded slash (`%2F`), it will remain encoded after `decodeURI` — this is likely desirable for path safety but should be documented." + }, + { + "file": "packages/route-pattern/src/lib/route-pattern.ts", + "line": 249, + "severity": "high", + "comment": "Decoding is applied to the entire pathname (`url.pathname.slice(1)`) before matching, rather than decoding individual param values after extraction. This means the pattern matcher sees decoded text, which could cause mismatches if a route pattern contains literal characters that are also valid decoded output from percent-encoded sequences. For example, a pathname with `%2F` would be preserved by `decodeURI` (since `/` is reserved), but other edge cases involving non-reserved characters might alter matching behavior." + }, + { + "file": "packages/route-pattern/src/lib/matcher.test.ts", + "line": 330, + "severity": "medium", + "comment": "The old test 'preserves URL encoding in variable values' was removed entirely and replaced with the non-ASCII test. This loses coverage for the ASCII percent-encoding case (e.g., `hello%20world` becoming `hello world`). It would be valuable to keep a test confirming that `%20` in a pathname param is decoded to a space, since that is a common real-world scenario affected by this change." + }, + { + "file": "packages/route-pattern/src/lib/matcher.test.ts", + "line": 340, + "severity": "low", + "comment": "The test constructs a URL via `new URL(template-literal)` which causes the browser/Node URL parser to percent-encode the non-ASCII characters automatically. The test then asserts the decoded values match the original strings. This is a solid approach — it mirrors real-world usage where URLs arrive percent-encoded from HTTP and the library returns decoded params." + }, + { + "file": "packages/route-pattern/src/lib/route-pattern.test.ts", + "line": 893, + "severity": "low", + "comment": "The RoutePattern test mirrors the matcher test with the same set of non-ASCII param categories (accented, CJK, RTL, combining, emoji, ZWJ, NBSP, fullwidth). This is good coverage across Unicode categories, and the `assertMatch` helper keeps it concise." + }, + { + "file": "packages/route-pattern/src/lib/route-pattern/decode-uri.ts", + "line": 11, + "severity": "medium", + "comment": "The function catches exceptions from `decodeURI` and returns the original string unchanged. This is safe for malformed percent sequences (e.g., `%ZZ`), but callers should be aware that partial decoding is not possible — the function either fully decodes or returns the original. If a string has a mix of valid and invalid sequences, the entire string stays encoded, which could be surprising." + } + ], + "summary": "The PR aligns route-pattern with the common convention of returning decoded param values, using `tryDecodeURI` as a safe wrapper around `decodeURI`. The approach of decoding the entire pathname before matching is simple and effective, though the removal of the ASCII percent-encoding test (`%20`) leaves a small coverage gap for the most common encoding scenario." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/route-pattern/src/lib/route-pattern.ts", + "line": 249, + "severity": "high", + "comment": "Following the plan's call graph, `RoutePattern.match` (order 11) is the primary integration point where decoding is applied. It is called by `testSuite` (order 1), `RoutePattern.test`, `assertMatch`, and `Trie.search` (order 2). Decoding the full pathname before passing to `this.ast.pathname.match()` means all downstream matchers — including the trie-based matcher — receive decoded text. This is architecturally clean (single decode point) but means the pattern AST must handle decoded Unicode characters in segment matching. Any pattern that uses literal percent-encoded text would break." + }, + { + "file": "packages/route-pattern/src/lib/route-pattern/decode-uri.ts", + "line": 8, + "severity": "high", + "comment": "As a leaf node (order 19) called by both `RoutePattern.match` and `Trie.search`, `tryDecodeURI` is the foundational building block of this change. The plan marks it high-risk because it is called from multiple entry points. The fast-path `includes('%')` check is a good optimization since most pathnames with non-ASCII characters will be percent-encoded by the URL parser. The try/catch fallback for malformed sequences ensures robustness. However, the typo in the JSDoc ('containts') should be fixed." + }, + { + "file": "packages/route-pattern/src/lib/matcher.test.ts", + "line": 330, + "severity": "medium", + "comment": "The `testSuite` function (order 1, entry point, high risk) exercises the `TrieMatcher.add` and matcher flow. The replacement of the 'preserves URL encoding' test with the non-ASCII test changes the behavioral contract: previously, encoded params were returned as-is; now they are decoded. This is a breaking change for consumers relying on encoded values. The changeset file correctly marks this as a minor version bump, and the migration path (use `encodeURI()`) is documented, but the deleted test case for `hello%20world` should be retained and updated to assert the decoded value `hello world`." + }, + { + "file": "packages/route-pattern/src/lib/route-pattern.test.ts", + "line": 893, + "severity": "medium", + "comment": "This test validates the `RoutePattern.match` path (order 11) through the `assertMatch` helper. The test covers 8 Unicode categories which is thorough. However, it does not test the error fallback path in `tryDecodeURI` — a test with a malformed percent sequence (e.g., `/%ZZfoo`) would verify the graceful degradation behavior and exercise the catch branch of the leaf function." + }, + { + "file": "packages/route-pattern/src/lib/route-pattern.ts", + "line": 7, + "severity": "low", + "comment": "The import of `tryDecodeURI` from `./route-pattern/decode-uri.ts` into the `RoutePattern` class (order 16) cleanly separates the decoding concern into its own module. This follows good module design — the decode logic is reusable by both `RoutePattern.match` and `Trie.search` without duplication." + }, + { + "file": "packages/route-pattern/.changes/minor.non-ascii-pathname-params.md", + "line": 1, + "severity": "low", + "comment": "The changeset clearly documents the before/after behavior with concrete examples across multiple Unicode scripts, plus the migration path via `encodeURI()`. This is well-written and will serve as good release notes." + } + ], + "summary": "The flow analysis reveals that `tryDecodeURI` is a shared leaf called from two distinct entry points (`RoutePattern.match` and `Trie.search`), making its correctness and error handling critical. The single-point-of-decode strategy in `RoutePattern.match` is architecturally clean but constitutes a breaking behavioral change that the tests should more thoroughly cover, including the error fallback path and common ASCII encodings like `%20`." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 6.6 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 8.0 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leverages the plan's call graph to identify that `tryDecodeURI` is a shared leaf node called from two entry points (RoutePattern.match and Trie.search), raising its risk profile beyond what a flat diff reading would suggest. It correctly traces the decode-before-match strategy through the caller chain and identifies the architectural implication that all downstream matchers receive decoded text. The baseline review catches the same key issues (decode-before-match tradeoffs, lost test coverage for %20, typo) but lacks the structural awareness of how the change propagates through the codebase's call graph. Both reviews identify the missing %20 test as a gap, but the flow-guided review additionally flags the untested error fallback path in tryDecodeURI, which is a meaningful coverage gap for a high-risk leaf node." + } +} diff --git a/evals/remix-run__remix__11207.json b/evals/remix-run__remix__11207.json new file mode 100644 index 0000000..03fb569 --- /dev/null +++ b/evals/remix-run__remix__11207.json @@ -0,0 +1,108 @@ +{ + "pr": "remix-run/remix#11207", + "title": "route-pattern: decode hostname params", + "baseline_review": { + "comments": [ + { + "file": "packages/route-pattern/src/lib/punycode.ts", + "line": 1, + "severity": "major", + "comment": "Vendoring a full punycode decoder (~205 lines) is a significant maintenance burden. Node.js has a built-in `punycode` module (deprecated but still available) and the `url` module's WHATWG URL parser already handles IDN decoding. If this package targets browsers too, consider depending on the well-maintained `punycode.js` npm package rather than forking it — a vendored copy will silently miss upstream security or correctness fixes." + }, + { + "file": "packages/route-pattern/src/lib/decode.ts", + "line": 1, + "severity": "minor", + "comment": "Re-exporting `toUnicode` as `decodeHostname` is a clean abstraction boundary. However, the module mixes a re-export with a function definition, making it easy to miss that `decodeHostname` is backed by the punycode vendored code. A brief JSDoc on the re-export (similar to `decodePathname`) would improve discoverability." + }, + { + "file": "packages/route-pattern/src/lib/decode.ts", + "line": 10, + "severity": "nit", + "comment": "The `decodePathname` function was presumably extracted from its previous inline location as part of this PR. The coarse `%` check is a nice fast-path optimization. The try/catch around `decodeURI` correctly handles malformed percent-escape sequences by returning the original string — good defensive coding." + }, + { + "file": "packages/route-pattern/src/lib/matcher.test.ts", + "line": 95, + "severity": "minor", + "comment": "The test covers accented, CJK, RTL, and combining characters — good breadth. However, there is no negative test for hostnames that are already in punycode form (e.g., `xn--caf-dma.example.com`). It would be valuable to verify that a punycode-encoded hostname is still decoded to its Unicode equivalent, confirming idempotent behavior." + }, + { + "file": "packages/route-pattern/src/lib/matcher.test.ts", + "line": 107, + "severity": "minor", + "comment": "The test constructs the URL via template literal with the Unicode params. Browsers and the URL constructor will automatically encode these to punycode (e.g., `cafe` becomes `xn--caf-dma`), so the test implicitly validates the decode round-trip. This is correct but worth a comment in the test to clarify why the assertion expects Unicode despite the URL constructor encoding." + }, + { + "file": "packages/route-pattern/src/lib/punycode.ts", + "line": 126, + "severity": "nit", + "comment": "The `decode` function adapted from punycode.js uses `error()` which throws a `RangeError` on invalid input. In the context of route matching, an invalid punycode label in a hostname would cause an unhandled exception rather than a graceful fallback (like `decodePathname` does with its try/catch). Consider wrapping calls to `decode` with a try/catch in `toUnicode` or `mapDomain` to return the raw label on failure." + } + ], + "summary": "The PR cleanly extends the existing pathname decoding approach to hostnames by vendoring a punycode decoder and introducing a `decodeHostname` helper. The main concern is the maintenance cost of vendoring 205 lines of punycode logic rather than depending on the established npm package, plus the lack of error-resilient fallback for malformed punycode labels." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/route-pattern/src/lib/trie-matcher.ts", + "line": 236, + "severity": "critical", + "comment": "The `Trie.search` method is the integration point where `decodeHostname` is wired into the matching pipeline. With 5 additions and 4 deletions, the change swaps the previous raw param extraction to decoded params. This is the highest-risk modification because `search` is called by both `match` and `matchAll` on the TrieMatcher, meaning any regression here affects all route resolution. The plan confirms this is an entry point with multiple downstream consumers. Verify that the decoding is applied only to hostname param segments and not to static hostname labels." + }, + { + "file": "packages/route-pattern/src/lib/punycode.ts", + "line": 106, + "severity": "major", + "comment": "The `decode` function (82 additions) is the largest single change and flagged as high-risk in the plan due to its size. It is adapted from punycode.js and implements RFC 3492 bootstring decoding. The function calls `error()` which throws `RangeError` on invalid input — but the call chain from `Trie.search` -> `decodeHostname` -> `toUnicode` -> `mapDomain` -> `decode` has no try/catch boundary. A malformed punycode label (e.g., `xn--` with no valid payload) would crash route matching rather than gracefully falling back to the encoded form. This is a real risk since hostnames come from user-controlled URLs." + }, + { + "file": "packages/route-pattern/src/lib/punycode.ts", + "line": 47, + "severity": "minor", + "comment": "The `mapDomain` function splits on `@` for email support and on RFC 3490 separators. In the route-pattern context, hostnames will never contain `@`, so the email-handling branch is dead code. This is inherited from the upstream punycode.js library. While harmless, stripping unused email logic would reduce the vendored surface area and make intent clearer." + }, + { + "file": "packages/route-pattern/src/lib/decode.ts", + "line": 1, + "severity": "minor", + "comment": "This module creates a clean separation between hostname and pathname decoding. The plan shows `decodePathname` was previously inline in `trie-matcher.ts` (the Trie.search node shows it as a dependency). Extracting both decoders into a shared module is good for maintainability. The asymmetry in error handling is notable though — `decodePathname` has try/catch, `decodeHostname` (via punycode) does not." + }, + { + "file": "packages/route-pattern/src/lib/matcher.test.ts", + "line": 95, + "severity": "major", + "comment": "The test is the entry point (order 1) in the review plan and covers the happy path well with accented, CJK, RTL, and combining character hostnames. However, there is no test for error/edge cases: (1) a hostname with an invalid punycode label like `xn--.example.com`, (2) a hostname mixing punycode and non-punycode labels, (3) a hostname with the maximum IDN label length (63 bytes). Given that the punycode `decode` function throws on invalid input and there is no catch boundary, test case (1) would likely expose a crash." + }, + { + "file": "packages/route-pattern/src/lib/specificity.ts", + "line": 63, + "severity": "minor", + "comment": "The `compare` function has a small 2-addition/1-deletion change. The plan flags it as high-risk due to its many callers (lessThan, greaterThan, equal, ascending, descending). The diff was truncated, but since this function underpins all specificity ordering — and thus route priority — even a minor logic change here could silently reorder route matches. Ensure existing specificity tests cover the modified comparison logic." + } + ], + "summary": "The flow-guided review reveals that the most significant risk is the missing error boundary between `Trie.search` and the punycode `decode` function — user-controlled hostnames with malformed punycode labels will throw unhandled `RangeError` exceptions rather than falling back gracefully. The review plan also highlights that the `compare` change in specificity.ts, despite being small, has five direct callers that govern route ordering, warranting careful verification of existing test coverage." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 9, + "actionability": 8, + "efficiency": 7, + "overall": 8.2 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identifies the critical missing error boundary in the call chain from Trie.search through toUnicode/mapDomain/decode — a concrete crash path for malformed punycode hostnames that the baseline review only hints at as a nit. The plan's risk annotations correctly flagged decode() as high-risk due to its size and entry-point role, leading the flow-guided review to trace the full call chain and discover the asymmetric error handling between decodePathname (has try/catch) and decodeHostname (throws). The baseline review raises valid concerns about vendoring but lacks the caller-aware analysis needed to assess real production risk. The flow-guided review also correctly identifies the specificity.ts change as high-impact due to its five callers, whereas the baseline review does not mention it at all since it was in a truncated portion of the diff." + } +} \ No newline at end of file diff --git a/evals/run-eval.ts b/evals/run-eval.ts new file mode 100644 index 0000000..7769e51 --- /dev/null +++ b/evals/run-eval.ts @@ -0,0 +1,257 @@ +/** + * Eval runner - processes a single PR through the 3-agent eval pipeline. + * Called by the orchestrator with PR index as argument. + * + * Usage: npx tsx evals/run-eval.ts + * where is the 0-based index into /tmp/prflow-eval-prs.json + * + * Outputs: /Users/fbenitez/Personal/prflow/evals/____.json + */ + +import Anthropic from "@anthropic-ai/sdk"; +import * as fs from "fs"; +import * as path from "path"; + +const EVALS_DIR = "/Users/fbenitez/Personal/prflow/evals"; +const PREFETCH_DIR = "/tmp/prflow-evals"; + +interface PR { + repo: string; + number: number; + title: string; + files: number; + changes: number; + url: string; +} + +async function main() { + const index = parseInt(process.argv[2], 10); + const prs: PR[] = JSON.parse( + fs.readFileSync("/tmp/prflow-eval-prs.json", "utf8") + ); + const pr = prs[index]; + if (!pr) { + console.error(`No PR at index ${index}`); + process.exit(1); + } + + const [owner, repo] = pr.repo.split("/"); + const slug = `${owner}__${repo}__${pr.number}`; + const outPath = path.join(EVALS_DIR, `${slug}.json`); + + // Skip if already done + if (fs.existsSync(outPath)) { + console.log(`[${index}] ${slug} already done, skipping`); + return; + } + + const diff = fs.readFileSync( + path.join(PREFETCH_DIR, `${slug}.diff`), + "utf8" + ); + const desc = fs.existsSync(path.join(PREFETCH_DIR, `${slug}.desc.txt`)) + ? fs.readFileSync(path.join(PREFETCH_DIR, `${slug}.desc.txt`), "utf8") + : pr.title; + const plan = fs.readFileSync( + path.join(PREFETCH_DIR, `${slug}.plan.json`), + "utf8" + ); + + // Trim diff and plan to avoid token limits + const trimmedDiff = diff.slice(0, 12000); + const planObj = JSON.parse(plan); + const trimmedPlan = JSON.stringify({ + stats: planObj.stats, + steps: (planObj.steps || []) + .filter((s: any) => s.role !== "context_only") + .slice(0, 30), + clusters: (planObj.clusters || []).slice(0, 10), + dependencies: (planObj.dependencies || []).slice(0, 15), + }); + + const client = new Anthropic(); + + console.log(`[${index}] ${slug} — running baseline review...`); + + // Agent A: Baseline review + const baselineResp = await client.messages.create({ + model: "claude-sonnet-4-20250514", + max_tokens: 2000, + messages: [ + { + role: "user", + content: `You are a senior code reviewer. Review this PR and produce structured feedback. + +**PR:** ${pr.repo}#${pr.number} — "${pr.title}" +**Description:** ${desc.slice(0, 1500)} + +**Diff:** +\`\`\` +${trimmedDiff} +\`\`\` + +Output ONLY a JSON object (no markdown fences): +{"comments":[{"file":"path","line":0,"severity":"critical|major|minor|nit|positive","comment":"..."}],"summary":"2-3 sentence assessment"} + +Be specific. Reference exact code. Cover correctness, edge cases, test coverage.`, + }, + ], + }); + + console.log(`[${index}] ${slug} — running flow-guided review...`); + + // Agent B: Flow-guided review + const flowResp = await client.messages.create({ + model: "claude-sonnet-4-20250514", + max_tokens: 2000, + messages: [ + { + role: "user", + content: `You are a senior code reviewer using a flow-guided approach. You have a structured review plan showing code flow, dependencies, risk levels, and optimal review order. + +**PR:** ${pr.repo}#${pr.number} — "${pr.title}" +**Description:** ${desc.slice(0, 1500)} + +**Diff:** +\`\`\` +${trimmedDiff} +\`\`\` + +**Review Plan (from PR Flow Graph):** +${trimmedPlan} + +The plan tells you: review ORDER, each function's ROLE (entry_point/internal/leaf/context_only), RISK levels, DEPENDENCIES (review order matters), CLUSTERS (tightly coupled functions), calledBy/calls (the call chain). + +Output ONLY a JSON object (no markdown fences): +{"comments":[{"file":"path","line":0,"severity":"critical|major|minor|nit|positive","comment":"..."}],"summary":"2-3 sentence assessment"} + +Follow the review order. Call out cross-file consistency issues. Pay extra attention to high-risk nodes. Be specific.`, + }, + ], + }); + + // Extract text from responses + const baselineText = + baselineResp.content[0].type === "text" ? baselineResp.content[0].text : ""; + const flowText = + flowResp.content[0].type === "text" ? flowResp.content[0].text : ""; + + // Parse JSON (handle markdown fences if present) + function parseReview(text: string) { + const cleaned = text + .replace(/```json\n?/g, "") + .replace(/```\n?/g, "") + .trim(); + try { + return JSON.parse(cleaned); + } catch { + return { comments: [], summary: text.slice(0, 500) }; + } + } + + const baselineReview = parseReview(baselineText); + const flowReview = parseReview(flowText); + + console.log(`[${index}] ${slug} — running judge...`); + + // Randomize order for judge + const flipOrder = Math.random() > 0.5; + const review1 = flipOrder ? flowReview : baselineReview; + const review2 = flipOrder ? baselineReview : flowReview; + const label1 = flipOrder ? "flow_guided" : "baseline"; + const label2 = flipOrder ? "baseline" : "flow_guided"; + + // Judge + const judgeResp = await client.messages.create({ + model: "claude-sonnet-4-20250514", + max_tokens: 1500, + messages: [ + { + role: "user", + content: `You evaluate code reviews. Score two reviews of the same PR on 5 criteria (1-10 each). + +**PR:** ${pr.repo}#${pr.number} — "${pr.title}" (${pr.files} files, ${pr.changes} lines changed) + +**Review 1:** ${JSON.stringify(review1).slice(0, 3000)} + +**Review 2:** ${JSON.stringify(review2).slice(0, 3000)} + +Score each on: +1. Completeness: covered all meaningful changes? +2. Flow awareness: understood cross-file connections? +3. Risk identification: flagged riskiest parts? +4. Actionability: specific and useful comments? +5. Efficiency: avoided noise/false positives? + +Output ONLY JSON: +{"review_1_scores":{"completeness":N,"flow_awareness":N,"risk_identification":N,"actionability":N,"efficiency":N},"review_2_scores":{"completeness":N,"flow_awareness":N,"risk_identification":N,"actionability":N,"efficiency":N},"reasoning":"1-2 sentences","winner":"review_1|review_2|tie"}`, + }, + ], + }); + + const judgeText = + judgeResp.content[0].type === "text" ? judgeResp.content[0].text : ""; + const judgeResult = parseReview(judgeText); + + // Un-flip the scores + function avg(scores: any) { + const vals = [ + scores.completeness, + scores.flow_awareness, + scores.risk_identification, + scores.actionability, + scores.efficiency, + ].filter((v: any) => typeof v === "number"); + return vals.length > 0 + ? Math.round((vals.reduce((a: number, b: number) => a + b, 0) / vals.length) * 10) / 10 + : 0; + } + + const baselineScores = + label1 === "baseline" + ? judgeResult.review_1_scores + : judgeResult.review_2_scores; + const flowScores = + label1 === "flow_guided" + ? judgeResult.review_1_scores + : judgeResult.review_2_scores; + + let winner: string; + if (judgeResult.winner === "review_1") winner = label1; + else if (judgeResult.winner === "review_2") winner = label2; + else winner = "tie"; + + const evalResult = { + pr: { + url: pr.url, + owner, + repo, + number: pr.number, + title: pr.title, + files_changed: pr.files, + additions: 0, + deletions: 0, + language: "mixed", + }, + timestamp: new Date().toISOString(), + baseline_review: baselineReview, + flow_guided_review: flowReview, + review_plan: planObj.stats, + judge: { + baseline_scores: { ...baselineScores, overall: avg(baselineScores) }, + flow_guided_scores: { ...flowScores, overall: avg(flowScores) }, + reasoning: judgeResult.reasoning || "", + winner, + }, + }; + + fs.writeFileSync(outPath, JSON.stringify(evalResult, null, 2)); + console.log( + `[${index}] ${slug} — done! baseline=${avg(baselineScores)} flow=${avg(flowScores)} winner=${winner}` + ); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/evals/rust-lang__rust__154540.json b/evals/rust-lang__rust__154540.json new file mode 100644 index 0000000..ae87dd8 --- /dev/null +++ b/evals/rust-lang__rust__154540.json @@ -0,0 +1,102 @@ +{ + "pr": "rust-lang/rust#154540", + "title": "Fix invalid type suggestion for item nested in function", + "baseline_review": { + "comments": [ + { + "file": "compiler/rustc_middle/src/ty/print/mod.rs", + "line": "136-141", + "severity": "major", + "comment": "Two new trait methods `reset_path` and `should_omit_parent_def_path` are added to the `Printer` trait with default implementations. Adding methods to a public trait is a semver-compatible change only because `Printer` is not object-safe and has a `Sized` bound. However, any downstream or out-of-tree implementors of `Printer` that override all methods will get these new defaults silently, which could produce incorrect output if their printer has state that `reset_path` should clear. Consider documenting the contract: `reset_path` must reset the printer to a state as if no path segment has been printed yet." + }, + { + "file": "compiler/rustc_middle/src/ty/print/mod.rs", + "line": "220-227", + "severity": "major", + "comment": "The `omit_parent` logic is placed inside `default_print_def_path`, which the comment says 'should not be overridden.' This means all printers get this behavior. However, `should_omit_parent_def_path` only returns true in `FmtPrinter` when `RtnMode::ForSuggestion` is active. The coupling between a generic printing method and a mode flag specific to one printer implementation is fragile. If another printer implements `should_omit_parent_def_path` without properly implementing `reset_path`, the `p.reset_path()?; Ok(())` call would succeed but leave stale path segments in the output." + }, + { + "file": "compiler/rustc_middle/src/ty/print/pretty.rs", + "line": "2235-2241", + "severity": "minor", + "comment": "The `should_omit_parent_def_path` implementation checks for `RtnMode::ForSuggestion` using `RTN_MODE.with(|mode| mode.get())`. This means the parent-omission behavior is only active in suggestion context, which is correct for the fix. However, the check for `DefPathData::ValueNs | DefPathData::Closure | DefPathData::AnonConst` covers closures and anonymous constants in addition to functions. While closures and anon consts can contain items, it would be good to have test cases for those scenarios too, not just the function-nested case." + }, + { + "file": "compiler/rustc_hir_analysis/src/collect/type_of.rs", + "line": "454", + "severity": "minor", + "comment": "The `with_types_for_suggestion!` macro wrapping is the trigger that activates `RtnMode::ForSuggestion`, which in turn makes `should_omit_parent_def_path` return true. This is the correct fix point -- the suggestion was previously printing the fully qualified path including the parent function, producing invalid syntax like `main::Error`. Wrapping only the format call keeps the suggestion-mode scope tight." + }, + { + "file": "tests/ui/suggestions/function-local-item-type-suggestion-issue-146786.rs", + "line": "1-11", + "severity": "minor", + "comment": "The test covers the basic case of a struct defined inside `main()` with a `const` missing its type annotation. The `.fixed` file confirms the suggestion produces `: Error` instead of the previous `: main::Error`. Good minimal reproduction. However, this only tests the `ValueNs` (function) case -- there are no tests for items nested inside closures or anonymous constants, which are also handled by `should_omit_parent_def_path`." + } + ], + "summary": "This PR fixes a compiler suggestion bug where type annotations for items nested inside functions would include the parent function name (e.g., `main::Error` instead of `Error`), producing invalid Rust syntax. The fix adds parent-path omission logic to the printer trait gated on suggestion mode, which is correct but introduces coupling between the generic printer trait and a mode flag specific to `FmtPrinter`, and lacks test coverage for closure/anon-const nested items." + }, + "flow_guided_review": { + "comments": [ + { + "file": "compiler/rustc_hir_analysis/src/collect/type_of.rs", + "line": "454", + "severity": "minor", + "comment": "ENTRY POINT: The `with_types_for_suggestion!` macro sets `RtnMode::ForSuggestion` via a thread-local. This is the activation point for the entire fix. The scope is minimal -- only the format string for the suggestion span. This is correct because `with_forced_trimmed_paths` was already in scope (line 450 in the original), but it did not trigger parent omission. The `with_types_for_suggestion!` macro is the precise additional context needed to distinguish 'printing for diagnostics' from 'printing for code suggestions.'" + }, + { + "file": "compiler/rustc_middle/src/ty/print/pretty.rs", + "line": "2229-2234", + "severity": "minor", + "comment": "PRINTER STATE: `reset_path` sets `self.empty_path = true`, which tells the printer that no path segments have been written yet. This is called when omitting the parent, effectively making the printer 'forget' it was in the middle of printing a path. The `empty_path` flag controls whether `::` separators are emitted between path segments. Setting it to true means the next segment (the type name) will be printed without a leading `::`, which is the desired behavior for suggestion output." + }, + { + "file": "compiler/rustc_middle/src/ty/print/pretty.rs", + "line": "2235-2241", + "severity": "major", + "comment": "CORE LOGIC: `should_omit_parent_def_path` checks `RtnMode::ForSuggestion` AND that the parent is a `ValueNs`, `Closure`, or `AnonConst`. The `RtnMode` check is critical -- without it, normal diagnostic printing (error messages, not suggestions) would also omit parent paths, producing confusing messages like 'expected Error' when two different `Error` types from different functions are in play. However, the `RtnMode::ForSuggestion` flag is also used by RTN (return-type notation) printing logic elsewhere in pretty.rs. There is a risk that if RTN mode is activated for non-suggestion purposes in the future, this parent-omission logic would incorrectly fire. The semantic overloading of `ForSuggestion` for two distinct purposes (RTN suggestions and type-annotation suggestions) could cause subtle bugs." + }, + { + "file": "compiler/rustc_middle/src/ty/print/mod.rs", + "line": "220-227", + "severity": "major", + "comment": "DISPATCH: The omit logic in `default_print_def_path` checks `DefPathData::TypeNs(..)` before calling `should_omit_parent_def_path`. This means only type-namespace items (structs, enums, type aliases, traits) get parent omission. Value-namespace items like associated constants or functions nested in functions would not be affected. This is the correct narrowing for the reported bug (suggesting `Error` instead of `main::Error`), but consider: if a user defines a function-local `const` type alias, would that also need omission? The `TypeNs` guard should cover type aliases, so this appears safe." + }, + { + "file": "compiler/rustc_middle/src/ty/print/mod.rs", + "line": "136-141", + "severity": "minor", + "comment": "TRAIT EXTENSION: The two new default trait methods are placed above the 'Defaults (should not be overridden)' comment, which is appropriate since these ARE meant to be overridden by specific printers. The `reset_path` default is a no-op and `should_omit_parent_def_path` defaults to false, ensuring zero behavioral change for all existing printers except `FmtPrinter`. This is a clean extension point." + }, + { + "file": "tests/ui/suggestions/function-local-item-type-suggestion-issue-146786.rs", + "line": "7", + "severity": "minor", + "comment": "TEST COVERAGE: The test exercises the exact bug from issue #146786: `const ERROR = Error;` inside `main()` where `Error` is a function-local struct. The `.fixed` file shows `const ERROR: Error = Error;` confirming the suggestion no longer includes `main::`. The `.stderr` file confirms `MachineApplicable` applicability. Missing: tests for nested closures (`|| { struct Foo; const X = Foo; }`), nested functions (`fn outer() { fn inner() { struct Foo; const X = Foo; } }`), and anon consts, all of which are handled by the `should_omit_parent_def_path` match arms." + } + ], + "summary": "The fix threads suggestion-mode context from the type-of suggestion site through the printer trait to the path-printing logic, correctly omitting function/closure/anon-const parent paths that would produce invalid syntax in code suggestions. The main risk is semantic overloading of `RtnMode::ForSuggestion` for both RTN and type-annotation suggestions, which could cause unintended parent-path omission if RTN mode is activated in new contexts; test coverage should be expanded to closures and anon consts." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review traced the activation chain from `with_types_for_suggestion!` through `RtnMode::ForSuggestion` to `should_omit_parent_def_path` to `reset_path`, understanding how the `empty_path` flag prevents `::` separators. This allowed it to identify the key risk: `RtnMode::ForSuggestion` is semantically overloaded for both RTN printing and type-annotation suggestions, creating a potential for unintended parent-path omission in future RTN contexts. The baseline review noted the trait coupling concern but could not trace why it matters because it lacked understanding of how `RtnMode` is used elsewhere. The flow-guided review also correctly identified that the `TypeNs` guard in `default_print_def_path` is the key narrowing that prevents value-namespace items from being affected, while the baseline treated this as part of a general observation about the trait design. Both reviews identified the missing test coverage for closures and anon consts, but the flow-guided version connected this gap directly to the match arms in `should_omit_parent_def_path`." + } +} diff --git a/evals/serde-rs__serde__3034.json b/evals/serde-rs__serde__3034.json new file mode 100644 index 0000000..aae1c08 --- /dev/null +++ b/evals/serde-rs__serde__3034.json @@ -0,0 +1,116 @@ +{ + "pr": { + "url": "https://github.com/serde-rs/serde/pull/3034", + "owner": "serde-rs", + "repo": "serde", + "number": 3034, + "title": "serde_derive: rewrite .iter().filter_map() with less llvm-lines", + "files_changed": 3, + "additions": 75, + "deletions": 75, + "language": "Rust" + }, + "baseline_review": { + "comments": [ + { + "file": "serde_derive/src/bound.rs", + "line": 34, + "severity": "positive", + "comment": "Replacing .extend(predicates.iter().cloned()) with an explicit for loop and individual push(predicate.clone()) reduces monomorphization overhead from the Extend trait, which is a well-known technique for reducing LLVM IR in Rust derive macros." + }, + { + "file": "serde_derive/src/bound.rs", + "line": 53, + "severity": "positive", + "comment": "Eliminating the .filter_map().flat_map(<[syn::WherePredicate]>::to_vec) chain avoids both iterator adapter monomorphization and the intermediate Vec allocation from to_vec(). The let-else with continue is a clean equivalent." + }, + { + "file": "serde_derive/src/bound.rs", + "line": 263, + "severity": "nit", + "comment": "Replacing .iter().filter(|field| ...) with for + if is straightforward and semantically identical. While less idiomatic, the consistency with the rest of the PR is good." + }, + { + "file": "serde_derive/src/de/enum_adjacently.rs", + "line": 34, + "severity": "minor", + "comment": "Converting the .iter().enumerate().filter().map().collect() chain to a mutable Vec with push is the highest-impact change in terms of monomorphization savings, since the closure here captures params, variant, and cattrs. Verify the ordering of variant_arms remains identical (it does, since enumerate preserves order)." + }, + { + "file": "serde_derive/src/de/enum_adjacently.rs", + "line": 66, + "severity": "minor", + "comment": "The missing_content_arms conversion correctly preserves the side effect of assigning to missing_content_fallthrough inside the loop body. In the original filter_map chain, this side effect happened in the None-returning branch; the new continue-based approach is actually clearer about the mutation." + } + ], + "summary": "Systematic, behavior-preserving replacement of iterator combinator chains with explicit for loops across serde_derive to reduce LLVM IR by ~2200 lines. All transformations are correct and follow a consistent pattern, trading slightly less idiomatic Rust for meaningful compile-time improvement in a widely-depended-upon crate." + }, + "flow_guided_review": { + "comments": [ + { + "file": "serde_derive/src/bound.rs", + "line": 34, + "severity": "positive", + "comment": "The flow plan is empty (zero steps, zero clusters), confirming these are purely mechanical, isolated transformations with no cross-cutting dependencies. The with_where_predicates change is a self-contained refactor." + }, + { + "file": "serde_derive/src/bound.rs", + "line": 53, + "severity": "positive", + "comment": "with_where_predicates_from_fields and with_where_predicates_from_variants follow the same pattern. Without any flow dependencies to trace, each function can be reviewed independently. Both correctly replace flat_map(to_vec) with nested for loops, eliminating the intermediate allocation." + }, + { + "file": "serde_derive/src/de/enum_adjacently.rs", + "line": 34, + "severity": "minor", + "comment": "The variant_arms Vec construction replaces a &Vec<_> binding (from collect) with a mutable Vec built via push. The old code took a reference to the collected Vec; the new code owns it directly. Both are equivalent since variant_arms is only used by reference in the quote! macro below." + }, + { + "file": "serde_derive/src/de/enum_adjacently.rs", + "line": 66, + "severity": "minor", + "comment": "The missing_content_arms conversion has a subtle correctness aspect: the original filter_map returned None for the fallthrough case (which also mutated missing_content_fallthrough), and the new code uses continue. Both skip appending to the arms list while performing the side effect. Semantically equivalent." + }, + { + "file": "serde_derive/src/bound.rs", + "line": 263, + "severity": "nit", + "comment": "The Data::Enum and Data::Struct match arms now use simple if-guards instead of .filter(). With no flow plan dependencies to consider, this is a straightforward, low-risk transformation." + } + ], + "summary": "With an empty flow plan (no steps, clusters, or dependencies), the flow-guided review confirms each change site is independent and behavior-preserving. The refactoring consistently replaces iterator adapters with explicit loops to reduce LLVM IR, with no cross-file or cross-function dependencies to track." + }, + "review_plan": { + "stats": { + "totalSteps": 0, + "totalAdditions": 0, + "totalDeletions": 0, + "independentFlows": 0, + "filesChanged": 0 + }, + "steps": [], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 6, + "actionability": 5, + "efficiency": 8, + "overall": 5.8 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 5, + "efficiency": 7, + "overall": 5.8 + }, + "reasoning": "Both reviews handle this mechanical refactoring adequately. The empty flow plan provides minimal additional signal for purely isolated, behavior-preserving changes. The baseline review slightly edges on efficiency by not over-referencing the empty plan, while the flow-guided review adds marginally more context about the side-effect preservation in missing_content_arms. Neither review identifies novel risks the other misses, since the changes are straightforward loop-for-iterator swaps with no cross-cutting concerns.", + "winner": "tie" + }, + "timestamp": "2026-03-30T13:50:33.809940+00:00" +} diff --git a/evals/shadcn-ui__ui__10189.json b/evals/shadcn-ui__ui__10189.json new file mode 100644 index 0000000..0c44a49 --- /dev/null +++ b/evals/shadcn-ui__ui__10189.json @@ -0,0 +1,84 @@ +{ + "pr": "shadcn-ui/ui#10189", + "title": "Revert \"deps: update next to 16.2.1 (#10180)\"", + "baseline_review": { + "comments": [ + { + "file": "apps/v4/package.json", + "line": 64, + "severity": "minor", + "comment": "The PR description says this reverts the Next.js 16.2.1 update but does not explain why the revert is needed. Adding a brief note about the issue encountered with 16.2.1 (build failure, runtime bug, etc.) would help future maintainers understand the reasoning." + }, + { + "file": "apps/v4/package.json", + "line": 100, + "severity": "minor", + "comment": "eslint-config-next is being reverted to 16.0.0 rather than 16.1.6 (the version next itself is reverting to). This version mismatch between next (16.1.6) and eslint-config-next (16.0.0) is presumably what was in place before #10180, but it is worth confirming that these versions are compatible and that lint rules match the runtime behavior." + }, + { + "file": "pnpm-lock.yaml", + "line": 40, + "severity": "nit", + "comment": "The lockfile changes are consistent with the package.json version downgrades. All transitive dependency references correctly reflect the 16.2.1 -> 16.1.6 change for next and related @next/* packages." + }, + { + "file": "apps/v4/package.json", + "line": 64, + "severity": "minor", + "comment": "Since this is a revert, consider whether a follow-up issue or task should be tracked for re-upgrading to Next.js 16.2.1 (or a later patch) once the underlying problem is resolved, so this downgrade does not become permanent unintentionally." + } + ], + "summary": "This PR cleanly reverts the Next.js 16.2.1 upgrade back to 16.1.6 across package.json and the lockfile, but provides no context on why the revert is needed. The version mismatch between next (16.1.6) and eslint-config-next (16.0.0) is the only notable asymmetry worth verifying." + }, + "flow_guided_review": { + "comments": [ + { + "file": "apps/v4/package.json", + "line": 64, + "severity": "minor", + "comment": "The revert commit message provides no context on what issue was encountered with Next.js 16.2.1. For a dependency revert, it is especially important to document the failure mode (build error, runtime regression, etc.) so the team knows what to watch for when re-attempting the upgrade." + }, + { + "file": "apps/v4/package.json", + "line": 100, + "severity": "minor", + "comment": "eslint-config-next reverts to 16.0.0 while next reverts to 16.1.6, creating a version gap. This was presumably the state before the original upgrade PR, but mismatched versions between the runtime and linting config could cause false positives or missed lint warnings for Next.js-specific rules." + }, + { + "file": "pnpm-lock.yaml", + "line": 168, + "severity": "nit", + "comment": "The lockfile correctly propagates the version change through all transitive peer dependency resolution strings (@vercel/analytics, fumadocs-core, fumadocs-mdx, fumadocs-ui, nuqs). The revert appears to be a clean git-revert with no manual edits, which is the safest approach." + }, + { + "file": "apps/v4/package.json", + "line": 64, + "severity": "minor", + "comment": "Consider creating a tracking issue for re-upgrading to Next.js >= 16.2.1 once the blocking issue is resolved. Without this, the revert may persist indefinitely and the project will miss security patches and features in newer Next.js releases." + } + ], + "summary": "This is a straightforward git-revert of a dependency upgrade with no code logic changes, only version pins and lockfile updates. The review plan is empty (0 steps, 0 clusters) because there are no code flow dependencies to analyze -- this is purely a version change with no behavioral impact beyond whatever Next.js regression triggered the revert." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 6.2 + } + }, + "winner": "tie", + "reasoning": "For this PR, both reviews produce nearly identical analysis because the review plan is empty (0 steps, 0 dependencies, 0 clusters). There are no code flows, data dependencies, or risk-ordered traversals to leverage -- the change is purely a version pin revert in package.json and its lockfile. The flow-guided approach has no structural advantage when there is no code logic to trace. Both reviews correctly identify the missing revert rationale, the eslint-config-next version mismatch, and the need for a follow-up upgrade tracking issue. Neither review can add meaningful flow-awareness commentary because the diff contains no executable code changes." + } +} \ No newline at end of file diff --git a/evals/shadcn-ui__ui__10202.json b/evals/shadcn-ui__ui__10202.json new file mode 100644 index 0000000..58defad --- /dev/null +++ b/evals/shadcn-ui__ui__10202.json @@ -0,0 +1,191 @@ +{ + "pr": { + "url": "https://github.com/shadcn-ui/ui/pull/10202", + "owner": "shadcn-ui", + "repo": "ui", + "number": 10202, + "title": "fix: packageManager in package.json", + "files_changed": 4, + "additions": 199, + "deletions": 3, + "language": "typescript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 135, + "severity": "major", + "comment": "The `isMonorepo` check is based solely on the existence of `pnpm-workspace.yaml`. If a template ships without this file (e.g. a yarn-only monorepo using `workspaces` in package.json), the monorepo path is never taken and `packageManager` is deleted even though turbo still requires it. Consider also checking for `turbo.json` or a `workspaces` field in package.json." + }, + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 171, + "severity": "minor", + "comment": "The `getPackageManagerVersion` function shells out to ` --version` at scaffold time. If the user has a different version on PATH than what their project expects (e.g. via Corepack), this embeds an incorrect version string. A comment noting this limitation would help future maintainers." + }, + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 176, + "severity": "minor", + "comment": "The fallback `${packageManager}@*` when `execa` fails is not a valid Corepack version specifier. Corepack requires an exact semver version; `@*` will cause Corepack to error when it tries to resolve the package manager. Consider falling back to a known minimum version or re-throwing the error." + }, + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 134, + "severity": "nit", + "comment": "The two `if (isMonorepo)` blocks (one for `packageManager`, one for `workspaces`) could be consolidated into a single block to reduce nesting and improve readability." + }, + { + "file": "packages/shadcn/src/utils/scaffold.test.ts", + "line": 279, + "severity": "minor", + "comment": "The existing test for 'strip packageManager field from package.json for non-pnpm non-monorepo' now mocks `execa` for `bun --version`, but this test path should never call `getPackageManagerVersion` since it is not a monorepo. Adding an unnecessary mock obscures what the test is actually exercising and could hide regressions if the non-monorepo path incorrectly starts calling `execa`." + } + ], + "summary": "This PR fixes monorepo scaffolding by preserving and updating the `packageManager` field (required by Turbo) instead of unconditionally deleting it. The approach is sound but the fallback version string `@*` is not a valid Corepack specifier, and the monorepo detection relies solely on `pnpm-workspace.yaml` which may miss other workspace configurations." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 135, + "severity": "major", + "comment": "Following the call chain `defaultScaffold -> adaptWorkspaceConfig -> getPackageManagerVersion`: the branching on `isMonorepo` determines whether `getPackageManagerVersion` is ever called. Since `isMonorepo` is derived only from `pnpm-workspace.yaml` existence, any monorepo template that does not ship this file (e.g. after conversion to npm/yarn workspaces) will silently delete `packageManager`, breaking Turbo. The detection should be broadened or documented as a known constraint." + }, + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 176, + "severity": "major", + "comment": "As a leaf function called by `adaptWorkspaceConfig`, `getPackageManagerVersion` has a silent catch that returns `${packageManager}@*`. This value flows back into `packageJson.packageManager` and gets written to disk. Since `@*` is not a valid semver range for Corepack, any project using Corepack will fail on `npm install` / `pnpm install` after scaffolding. The fallback should either throw (letting the caller handle it) or use a documented minimum version." + }, + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 171, + "severity": "minor", + "comment": "The function runs `execa(packageManager, ['--version'])` without validating the `packageManager` argument. While `adaptWorkspaceConfig` receives it from higher up the call chain (`defaultScaffold`), passing an unexpected string (e.g. from a corrupted config) would shell out an arbitrary command. Adding a guard that `packageManager` is one of the known values (pnpm, npm, yarn, bun) would be a low-cost safety improvement." + }, + { + "file": "packages/shadcn/src/templates/create-template.ts", + "line": 134, + "severity": "minor", + "comment": "The dependency graph shows `adaptWorkspaceConfig` calls both `getPackageManagerVersion` and `rewriteWorkspaceProtocol`. The new `packageManager` assignment and the existing `workspaces` assignment both gate on `isMonorepo` in separate if-blocks. Merging them into one block would make the monorepo adaptation logic easier to follow and reduce the risk of future changes updating one block but not the other." + }, + { + "file": "packages/shadcn/src/utils/scaffold.test.ts", + "line": 279, + "severity": "minor", + "comment": "The test 'strip packageManager for non-pnpm non-monorepo' now mocks `execa` even though the non-monorepo code path should never invoke `getPackageManagerVersion`. This mock silently masks any regression where the non-monorepo path accidentally calls `execa`. Adding `expect(execa).not.toHaveBeenCalled()` would turn this into a meaningful assertion." + }, + { + "file": "packages/shadcn/src/utils/scaffold.test.ts", + "line": 411, + "severity": "positive", + "comment": "Good coverage: the new tests exercise both bun and npm monorepo paths with version detection, and the updated existing test correctly adjusts assertions to expect the new behavior. The test descriptions are also clear about the monorepo vs non-monorepo distinction." + } + ], + "summary": "The flow-guided review traces the call chain from `defaultScaffold` through `adaptWorkspaceConfig` to the new `getPackageManagerVersion` leaf, revealing two key risks: the `@*` fallback is invalid for Corepack and the monorepo detection is fragile. Test coverage is solid but the non-monorepo test should assert that `execa` is not called to prevent silent regressions." + }, + "review_plan": { + "stats": { + "totalSteps": 5, + "totalAdditions": 19, + "totalDeletions": 3, + "independentFlows": 1, + "filesChanged": 1 + }, + "steps": [ + { + "order": 2, + "nodeId": "packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig", + "name": "adaptWorkspaceConfig", + "file": "packages/shadcn/src/templates/create-template.ts", + "lines": [108, 169], + "type": "function", + "changeType": "modified", + "additions": 11, + "deletions": 3, + "role": "internal", + "risk": "low", + "calledBy": ["packages/shadcn/src/templates/create-template.ts::defaultScaffold"], + "calls": ["packages/shadcn/src/templates/create-template.ts::getPackageManagerVersion", "packages/shadcn/src/templates/create-template.ts::rewriteWorkspaceProtocol"], + "riskReasons": [] + }, + { + "order": 4, + "nodeId": "packages/shadcn/src/templates/create-template.ts::getPackageManagerVersion", + "name": "getPackageManagerVersion", + "file": "packages/shadcn/src/templates/create-template.ts", + "lines": [172, 179], + "type": "function", + "changeType": "modified", + "additions": 8, + "deletions": 0, + "role": "leaf", + "risk": "low", + "calledBy": ["packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig"], + "calls": [], + "riskReasons": [] + } + ], + "clusters": [ + { + "id": 0, + "label": "create-template.ts", + "nodeIds": [ + "packages/shadcn/src/templates/create-template.ts::defaultScaffold", + "packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig", + "packages/shadcn/src/templates/create-template.ts::getInstallArgs", + "packages/shadcn/src/templates/create-template.ts::getPackageManagerVersion", + "packages/shadcn/src/templates/create-template.ts::rewriteWorkspaceProtocol" + ], + "reason": "5 related functions in create-template.ts", + "suggestedReviewOrder": [ + "packages/shadcn/src/templates/create-template.ts::defaultScaffold", + "packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig", + "packages/shadcn/src/templates/create-template.ts::getInstallArgs", + "packages/shadcn/src/templates/create-template.ts::getPackageManagerVersion", + "packages/shadcn/src/templates/create-template.ts::rewriteWorkspaceProtocol" + ] + } + ], + "dependencies": [ + { + "from": "packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig", + "to": "packages/shadcn/src/templates/create-template.ts::getPackageManagerVersion", + "reason": "Review `adaptWorkspaceConfig` before `getPackageManagerVersion` -- `adaptWorkspaceConfig` calls `getPackageManagerVersion`." + }, + { + "from": "packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig", + "to": "packages/shadcn/src/templates/create-template.ts::rewriteWorkspaceProtocol", + "reason": "Review `adaptWorkspaceConfig` before `rewriteWorkspaceProtocol` -- `adaptWorkspaceConfig` calls `rewriteWorkspaceProtocol`." + }, + { + "from": "packages/shadcn/src/templates/create-template.ts::defaultScaffold", + "to": "packages/shadcn/src/templates/create-template.ts::adaptWorkspaceConfig", + "reason": "Review `defaultScaffold` before `adaptWorkspaceConfig` -- `defaultScaffold` calls `adaptWorkspaceConfig`." + } + ] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 8, + "overall": 8.2 + }, + "reasoning": "Both reviews identify the same core issues: the fragile monorepo detection based on pnpm-workspace.yaml and the invalid @* fallback in getPackageManagerVersion. However, the flow-guided review provides significantly better context by tracing the call chain (defaultScaffold -> adaptWorkspaceConfig -> getPackageManagerVersion) and explaining how the invalid fallback value flows through the system to be written to disk. The flow-guided review also makes a stronger actionable suggestion about the non-monorepo test (asserting execa is not called rather than just noting the unnecessary mock). The baseline review lacks structural awareness -- it treats each comment in isolation without connecting how the functions interact. The flow-guided review's dependency-aware framing makes the risk of the @* fallback much clearer (it is not just an incorrect string, it propagates through the write path to corrupt the output file). The baseline gets partial credit for noticing the consolidation opportunity and the test mock issue, but misses the input validation concern that the flow-guided review raises about arbitrary packageManager strings.", + "winner": "flow_guided" + } +} \ No newline at end of file diff --git a/evals/spring-projects__spring-boot__49791.json b/evals/spring-projects__spring-boot__49791.json new file mode 100644 index 0000000..d5efd66 --- /dev/null +++ b/evals/spring-projects__spring-boot__49791.json @@ -0,0 +1,102 @@ +{ + "pr": "spring-projects/spring-boot#49791", + "title": "Support spring.webflux.default-html-escape property for application-wide HTML escaping configuration", + "baseline_review": { + "comments": [ + { + "file": "module/spring-boot-webflux/src/main/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfiguration.java", + "line": 70, + "severity": "major", + "comment": "The `defaultHtmlEscape` property is `@Nullable Boolean`, so `properties.getDefaultHtmlEscape()` can return `null`. Calling `handlerBuilder.defaultHtmlEscape(null)` may throw a NullPointerException or have unintended behavior depending on the Spring Framework method signature. The null check should also verify `properties.getDefaultHtmlEscape() != null` before calling `defaultHtmlEscape()`, or use a pattern like `Optional.ofNullable(properties.getDefaultHtmlEscape()).ifPresent(handlerBuilder::defaultHtmlEscape)`." + }, + { + "file": "module/spring-boot-webflux/src/main/java/org/springframework/boot/webflux/autoconfigure/WebFluxProperties.java", + "line": 63, + "severity": "minor", + "comment": "The Javadoc says 'Whether default HTML escaping is enabled for the web application' but the type is `@Nullable Boolean` (tri-state: true/false/null). The documentation should clarify that null means 'use the framework default' or 'not configured', to distinguish from an explicit false which disables escaping." + }, + { + "file": "module/spring-boot-webflux/src/test/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfigurationTests.java", + "line": 109, + "severity": "minor", + "comment": "The test `shouldConfigureDefaultHtmlEscape` validates both true and false via `@ValueSource(booleans)`, which is good. However, there is no test for the default case where the property is not set at all, verifying that `getDefaultHtmlEscape()` returns null and the handler is not configured with an explicit value. This would validate the null/unset path." + }, + { + "file": "module/spring-boot-webflux/src/test/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfigurationTests.java", + "line": 117, + "severity": "minor", + "comment": "The test method `shouldNotConfigureDefaultHtmlEscaperWithoutWebFluxAutoConfiguration` has a typo: 'Escaper' instead of 'Escape'. This is inconsistent with the property name and the other test method naming." + }, + { + "file": "module/spring-boot-webflux/src/main/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfiguration.java", + "line": 67, + "severity": "nit", + "comment": "Moving `propsProvider.getIfAvailable()` to before the handler builder construction is a good change -- it enables the HTML escape configuration to be applied before customizers run. This means customizers can observe or override the default HTML escape setting, which is the correct ordering." + } + ], + "summary": "The PR adds a straightforward `spring.webflux.default-html-escape` property mirroring MVC's existing capability for WebFlux applications. The main concern is that the nullable Boolean property is passed directly to `defaultHtmlEscape()` without a null guard, which could cause issues when the property is not set; additionally the test method has a minor typo in its name." + }, + "flow_guided_review": { + "comments": [ + { + "file": "module/spring-boot-webflux/src/main/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfiguration.java", + "line": 70, + "severity": "major", + "comment": "The `defaultHtmlEscape` property is `@Nullable Boolean`. When the property is not configured, `properties.getDefaultHtmlEscape()` returns null. The code only checks `properties != null` but not whether `getDefaultHtmlEscape()` is null before passing it to `handlerBuilder.defaultHtmlEscape()`. If the Spring Framework method `defaultHtmlEscape(Boolean)` does not accept null, this will NPE at runtime for any WebFlux app that has WebFluxProperties available but hasn't set the property. The fix should be: `if (properties != null && properties.getDefaultHtmlEscape() != null)`." + }, + { + "file": "module/spring-boot-webflux/src/main/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfiguration.java", + "line": 67, + "severity": "minor", + "comment": "The ordering change -- fetching properties before building the handler, applying defaultHtmlEscape before customizers, then building -- is significant. Customizers that call `handlerBuilderCustomizers.orderedStream()` will now see the default HTML escape already set on the builder. This is likely intentional to allow customizers to override the property-based default, but it should be documented or tested that customizer ordering interacts correctly with this setting." + }, + { + "file": "module/spring-boot-webflux/src/main/java/org/springframework/boot/webflux/autoconfigure/WebFluxProperties.java", + "line": 63, + "severity": "minor", + "comment": "The property uses `@Nullable Boolean` which is the correct tri-state design (null = not set, true = enable, false = disable), matching how Spring MVC's `defaultHtmlEscape` context-param works. However, the Javadoc should clarify the tri-state semantics -- 'Whether default HTML escaping is enabled. When not set (null), the framework default applies.'" + }, + { + "file": "module/spring-boot-webflux/src/test/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfigurationTests.java", + "line": 103, + "severity": "minor", + "comment": "The parameterized test covers true/false but not the unset case. Since the null path through `getDefaultHtmlEscape()` is the most common case (most apps won't set this property), there should be a test that verifies when no property is set, the HttpHandler is created without calling `defaultHtmlEscape()` at all -- confirming the framework default is preserved." + }, + { + "file": "module/spring-boot-webflux/src/test/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfigurationTests.java", + "line": 117, + "severity": "nit", + "comment": "Method name `shouldNotConfigureDefaultHtmlEscaperWithoutWebFluxAutoConfiguration` contains 'Escaper' instead of 'Escape'. Should be `shouldNotConfigureDefaultHtmlEscapeWithoutWebFluxAutoConfiguration` for consistency." + }, + { + "file": "module/spring-boot-webflux/src/test/java/org/springframework/boot/webflux/autoconfigure/HttpHandlerAutoConfigurationTests.java", + "line": 113, + "severity": "positive", + "comment": "The test for the custom WebHandler path is a good defensive test -- it verifies that when WebFluxAutoConfiguration is not present (e.g., a manual WebHandler setup), setting the property has no effect. This confirms the feature is scoped correctly to the auto-configuration path." + } + ], + "summary": "The PR correctly exposes Spring Framework 7.0.6's programmatic HTML escape support as a Boot property, following the established pattern of MVC's property-based configuration. The primary risk is a potential NPE when `defaultHtmlEscape` is null (unset) but `WebFluxProperties` is available -- the null guard should check the property value itself, not just the properties object; the ordering of property application before customizers is correct but should be tested." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 8, + "actionability": 8, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the core nullable Boolean / null guard issue, which is the most important finding. However, the flow-guided review provides deeper analysis in several ways: (1) it explicitly traces the ordering significance -- properties fetched before builder, HTML escape set before customizers, then build -- and explains why this ordering matters for customizer interaction; (2) it connects the tri-state Boolean design to the MVC precedent, providing architectural context; (3) it more precisely articulates the NPE risk by noting it affects 'any WebFlux app that has WebFluxProperties available but hasn't set the property', which is the majority case. The baseline review catches the same bugs but treats them more as isolated observations without connecting them to the broader flow. The flow-guided review's score is limited because the review plan was empty (0 steps, 0 clusters), so the 'flow guidance' was minimal -- the reviewer had to construct the flow analysis independently. With a richer plan, the gap would likely be larger." + } +} diff --git a/evals/sveltejs__svelte__18009.json b/evals/sveltejs__svelte__18009.json new file mode 100644 index 0000000..d3a71f2 --- /dev/null +++ b/evals/sveltejs__svelte__18009.json @@ -0,0 +1,102 @@ +{ + "pr": "sveltejs/svelte#18009", + "title": "fix: correctly handle bindings on the server", + "baseline_review": { + "comments": [ + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 471, + "severity": "critical", + "comment": "The original code had `item.subsume(item)` which is a self-referential no-op — an item subsuming itself does nothing. The fix correctly maps each `other.#out[i]` item against the corresponding `this.#out[i]` (named `current`), calling `current.subsume(item)` so the current renderer absorbs the other's state. This is the core bug fix and looks correct." + }, + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 473, + "severity": "medium", + "comment": "The new code assumes `this.#out` and `other.#out` have the same length and positional correspondence (using index `i` to look up `this.#out[i]` while mapping over `other.#out`). If `other.#out` is longer than `this.#out`, `current` will be `undefined` and the fallback `return item` is fine. But if `this.#out` is longer, those trailing entries are silently dropped since the result is assigned from `other.#out.map(...)`. This may be intentional but deserves a comment." + }, + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 474, + "severity": "low", + "comment": "The guard `current instanceof Renderer && item instanceof Renderer` is a good defensive check. If only one side is a Renderer, the code falls through to `return item`, which replaces the current entry. This seems correct — if the other side has a plain string where we had a Renderer, we should take the other's value." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-hydration-binding/_config.js", + "line": 6, + "severity": "low", + "comment": "The test only runs in `hydrate` mode and checks the resolved HTML plus empty warnings. This is a reasonable regression test for the specific scenario (async + bindings on the server) that triggered the bug." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-hydration-binding/Bound.svelte", + "line": 1, + "severity": "low", + "comment": "Bound.svelte is an empty file. It serves as a minimal component that accepts a `bind:open` prop. This is fine for a test fixture — the binding existence is what matters, not the component's content." + } + ], + "summary": "The PR fixes a clear bug where `item.subsume(item)` was a no-op self-reference instead of correctly subsuming the other renderer's output entries into the current renderer's entries. The fix is small, targeted, and includes a well-structured regression test covering async hydration with bindings." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 471, + "severity": "critical", + "comment": "This is the entry point of the change and the highest-risk node. The `subsume` method is called from multiple paths (via `copy`, `#collect_content_async`, `#open_render`, etc.), so getting the index-based correspondence right is essential. The fix correctly pairs `this.#out[i]` with `other.#out[i]` and calls `current.subsume(item)` — the current renderer absorbs the other's state rather than the previous self-referential no-op." + }, + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 473, + "severity": "medium", + "comment": "Given the many callers identified in the plan (head, child, boundary, option, title, copy, #collect_content_async, #open_render), the assumption that `this.#out` and `other.#out` are positionally aligned needs to hold across all call sites. Since `subsume` is meant to transfer state from `other` to `this` when they represent the same logical rendering position, this positional mapping is structurally sound. However, an out-of-bounds access on `this.#out[i]` (returning `undefined`) is handled gracefully by the instanceof guard." + }, + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 475, + "severity": "medium", + "comment": "After `current.subsume(item)`, the code returns `current` rather than `item`. This is important — the current renderer has been updated in-place with the other's state, and any existing references to `current` elsewhere in the tree remain valid. Returning `item` instead would break those references. This is a subtle but correct design choice." + }, + { + "file": "packages/svelte/src/internal/server/renderer.js", + "line": 470, + "severity": "low", + "comment": "The `this.local = other.local` assignment on the line before the fix is unchanged but worth noting in context: the subsume method transfers both the local state and the output array. The fix ensures the output array transfer correctly recurses into nested Renderers rather than performing a no-op." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-hydration-binding/_config.js", + "line": 4, + "severity": "medium", + "comment": "The test exercises the specific flow that exposed the bug: async resolution combined with bindings triggers the subsume path on the server. The test validates both correct HTML output and absence of warnings. Given the high number of callers to the Renderer class, additional test coverage for other subsume paths (e.g., boundary, head) could help prevent regressions, but this test adequately covers the reported issue." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-hydration-binding/Binding.svelte", + "line": 1, + "severity": "low", + "comment": "The Binding component creates a `bind:open` relationship with the empty Bound component. This is the minimal reproduction needed to trigger the subsume code path during SSR — bindings cause the renderer to create a copy and later subsume it back." + } + ], + "summary": "The flow analysis reveals that `subsume` is called from 8+ sites across the Renderer class, making the index-correspondence fix high-impact despite its small size. The fix correctly preserves `current` (the existing renderer reference) after subsuming `item` (the other's renderer), maintaining referential integrity across the rendering tree while transferring the updated state." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 8, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.8 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review leverages knowledge of the 8+ call sites to the Renderer class to assess the broader impact of the index-correspondence fix. It identifies the critical detail that returning `current` (not `item`) preserves referential integrity, and contextualizes the positional mapping assumption against the structural design of subsume. The baseline review correctly identifies the self-referential no-op bug but lacks the caller-aware risk analysis that makes the flow-guided review more thorough in assessing whether the fix is safe across all code paths." + } +} diff --git a/evals/sveltejs__svelte__18021.json b/evals/sveltejs__svelte__18021.json new file mode 100644 index 0000000..13749d7 --- /dev/null +++ b/evals/sveltejs__svelte__18021.json @@ -0,0 +1,114 @@ +{ + "pr": "sveltejs/svelte#18021", + "title": "fix: reschedule new effects in prior batches", + "baseline_review": { + "comments": [ + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "148-152", + "severity": "medium", + "comment": "The `#new_effects` array grows unboundedly as effects are created during the batch lifetime but is never cleared. If a batch lives long (e.g. due to prolonged async work), this could retain references to destroyed effects. Consider clearing the array after `#commit` finishes or filtering out DESTROYED effects before iterating." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "540-541", + "severity": "medium", + "comment": "The `current_unequal` filter logic `this.current.has(c) ? this.current.get(c)[0] !== c : true` compares the stored value tuple's first element against `c` (the source key itself). This looks like it should be comparing values rather than a source against its own stored value. If `current` maps `Source -> [value, boolean]`, then `[0]` is the previous/current value and `c` is the Source -- comparing a Source to a value seems like a type mismatch or semantic error. Needs careful verification." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "543-555", + "severity": "low", + "comment": "The new effects loop duplicates the pattern of checking `(effect.f & (ASYNC | BLOCK_EFFECT)) !== 0` and branching between `schedule` and `#dirty_effects.add`. This same branching logic likely exists elsewhere in the batch system. Consider extracting a shared helper to reduce duplication and ensure consistent handling." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/effects.js", + "line": "123", + "severity": "low", + "comment": "The `current_batch?.register_created_effect(effect)` call registers every newly created effect unconditionally (all types). However, the commit loop later filters out DESTROYED, INERT, and EAGER_EFFECT. Registering effects that will always be filtered out adds unnecessary overhead -- consider filtering at registration time." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "538", + "severity": "low", + "comment": "`checked` is reassigned with `checked = new Map()` using `var`-scoped variable. This shadows the earlier `checked` Map used in the `mark_effects` loop above. While functionally fine due to sequential execution, using `let` or a distinct variable name would improve clarity." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-state-new-branch-1/_config.js", + "line": "19-32", + "severity": "low", + "comment": "The test assertions were updated with specific intermediate values ('world' appearing before 'universe'), and commented-out alternative assertions remain (line 37). These commented-out assertions should either be removed or converted to a documented rationale, as they create ambiguity about which behavior is actually correct." + } + ], + "summary": "This patch adds a mechanism to track newly created effects during a batch and reschedule them onto prior batches when their dependencies change. The core logic in `#commit` has a potentially confusing value comparison in `current_unequal` that warrants verification, and the `#new_effects` array lacks lifecycle management which could cause memory retention issues during long-lived batches." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "540-541", + "severity": "high", + "comment": "Critical data flow concern: `current_unequal` filters `batch.current.keys()` (which are Source objects) by checking `this.current.get(c)[0] !== c`. The `current` map stores `Source -> [value, boolean]` tuples. Comparing `[0]` (a value) against `c` (a Source) is a cross-type comparison that will almost always be true (since a Source object !== a primitive value). This means `current_unequal` would include nearly all sources, potentially over-scheduling effects. This needs verification against the actual `current` map semantics -- if `[0]` is meant to be the source's committed value, the comparison should be against the source's current value, not the source itself." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/effects.js", + "line": "123", + "severity": "medium", + "comment": "Flow analysis: `create_effect` is called for ALL effect types (render, block, async, eager, user). The `current_batch` export is imported from batch.js, meaning every single effect creation now has a call-site dependency on the batch module. Since `#commit` only processes effects that are NOT (DESTROYED | INERT | EAGER_EFFECT), registering eager effects is wasted work. For a hot path like effect creation, this overhead could be meaningful -- consider gating registration on the effect type." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "543-555", + "severity": "medium", + "comment": "Cross-batch scheduling risk: When iterating `this.#new_effects` and scheduling onto `batch` (a different batch), there's no guard against an effect being scheduled onto multiple prior batches. If multiple prior batches exist and an effect depends on sources changed in several of them, the effect could be added to multiple batches' `#dirty_effects` sets or scheduled multiple times. The `Set` for `#dirty_effects` prevents duplicates within one batch, but cross-batch double-scheduling could cause redundant re-execution." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "148-152", + "severity": "medium", + "comment": "Lifecycle gap: `#new_effects` is append-only with no cleanup. The `Batch` class has a `remove()` method that cleans up other state, and `#commit()` processes the array, but neither clears it. After commit, these references keep destroyed effects alive, preventing GC. Since batches can persist across async boundaries, this is a potential memory leak in long-running applications." + }, + { + "file": "packages/svelte/src/internal/client/reactivity/batch.js", + "line": "538", + "severity": "low", + "comment": "The `checked` variable is reused (reassigned to a new Map) from the outer scope where it was first used for `mark_effects`. Both the `mark_effects` pass and the `depends_on` pass use `checked` as a memoization cache with `Map` semantics. Reusing the same variable name obscures that these are independent caching scopes. A distinct name like `depends_checked` would improve readability." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-state-new-branch-1/_config.js", + "line": "19-37", + "severity": "low", + "comment": "Test expectations now assert that intermediate 'world' values are visible during batch resolution, which is the core behavioral change. However, the commented-out alternative assertion on line 37 (`// this would also be ok`) suggests the expected output is non-deterministic or implementation-dependent. If both orderings are valid, the test should accommodate both rather than asserting one and commenting out the other." + }, + { + "file": "packages/svelte/tests/runtime-runes/samples/async-state-new-branch-2/_config.js", + "line": "20-28", + "severity": "low", + "comment": "Similar to test-1, the expected HTML now includes intermediate 'world' values that were previously absent. The comment '// if this does not show world ... then this would also be ok' weakens confidence in the test's value as a regression guard. Consider making the assertion either strict or flexible, not ambiguously both." + } + ], + "summary": "The fix correctly identifies the problem (new branches created during batch processing are invisible to prior batches) and introduces a tracking mechanism via `#new_effects`. However, the `current_unequal` comparison logic appears to have a type mismatch (Source vs value comparison) that could cause over-eager rescheduling, and the lack of lifecycle management for the `#new_effects` array introduces a memory retention risk for long-running async batches." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 6, + "efficiency": 7, + "overall": 5.8 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.6 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review benefits significantly from understanding the data flow between `create_effect` (registration site) and `#commit` (consumption site). It correctly identifies the cross-batch double-scheduling risk that the baseline missed, and provides a more precise analysis of the `current_unequal` comparison issue by tracing the types through the `current` Map's actual structure. The baseline review caught many of the same surface-level issues but lacked the deeper understanding of how data flows across batch boundaries and effect lifecycles. The flow-guided review's elevation of the type mismatch concern from medium to high severity is well-justified given the potential for over-scheduling." + } +} \ No newline at end of file diff --git a/evals/tanstack__query__10346.json b/evals/tanstack__query__10346.json new file mode 100644 index 0000000..5b73417 --- /dev/null +++ b/evals/tanstack__query__10346.json @@ -0,0 +1,149 @@ +{ + "pr": { + "url": "https://github.com/tanstack/query/pull/10346", + "owner": "tanstack", + "repo": "query", + "number": 10346, + "title": "test(preact-query): apply 'prefer-template', 'no-else-return', 'object-shorthand', 'radix', and 'require-await' eslint rules", + "files_changed": 6, + "additions": 73, + "deletions": 73, + "language": "typescript" + }, + "timestamp": "2026-03-30T18:30:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/preact-query/src/__tests__/useQueries.test.tsx", + "line": 193, + "severity": "positive", + "comment": "Adding explicit radix parameter (10) to all eight parseInt calls is correct and prevents potential issues with non-decimal interpretation. Consistent application across all instances in this file." + }, + { + "file": "packages/preact-query/src/__tests__/useMutationState.test.tsx", + "line": 209, + "severity": "nit", + "comment": "Template literal conversion from 'data' + input to `data${input}` is semantically identical. The original had no space before the variable, and the template literal correctly preserves that." + }, + { + "file": "packages/preact-query/src/__tests__/usePrefetchInfiniteQuery.test.tsx", + "line": 28, + "severity": "positive", + "comment": "Template literal in the error message preserves readability while satisfying the prefer-template rule. The error message semantics are unchanged." + }, + { + "file": "packages/preact-query/src/__tests__/useQuery.test.tsx", + "line": 476, + "severity": "nit", + "comment": "Template literal `data: ${value}` correctly preserves the space from the original 'data: ' + value concatenation. All template literal conversions in this file are semantically equivalent." + }, + { + "file": "packages/preact-query/src/__tests__/useQuery.test.tsx", + "line": 3689, + "severity": "positive", + "comment": "Removing the unnecessary else block after a return statement follows the no-else-return rule and reduces nesting, improving readability without changing behavior." + } + ], + "summary": "This PR applies purely mechanical ESLint rule compliance changes across preact-query test files: template literals replace string concatenation, explicit radix added to parseInt, unnecessary else-after-return removed, and unused eslint-disable directives cleaned up. All changes are behavior-preserving and confined to test code, making this extremely low risk." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/preact-query/src/__tests__/usePrefetchInfiniteQuery.test.tsx", + "line": 28, + "severity": "positive", + "comment": "Step 1 of the plan targets generateInfiniteQueryOptions (lines 17-42), flagged as an entry point. The single change here converts a string concatenation in an error throw to a template literal. Since this is a test helper function and the change is purely cosmetic, the 'high' risk rating from the plan is a false positive driven by the entry_point heuristic rather than actual risk." + }, + { + "file": "packages/preact-query/src/__tests__/useQuery.test.tsx", + "line": 3689, + "severity": "nit", + "comment": "Step 2 targets the Component function (lines 5785-5797) in useQuery.test.tsx, but the diff also shows changes at lines 476, 772, and 3689 in the same file. The plan's focus on only one function misses the broader scope of changes in this file, though all are equally trivial lint fixes." + }, + { + "file": "packages/preact-query/src/__tests__/useQueries.test.tsx", + "line": 193, + "severity": "positive", + "comment": "The plan has zero steps covering useQueries.test.tsx despite it having the most changes (8 parseInt radix fixes plus 4 template literal conversions). This is the file most deserving of review attention, yet the plan missed it entirely because it only tracked 2 additions/2 deletions total, far below the actual 73-line change scope." + }, + { + "file": "packages/preact-query/src/__tests__/useQueries.test.tsx", + "line": 559, + "severity": "nit", + "comment": "All parseInt radix additions are in select callbacks within test query configurations. These are type-level tests (using expectTypeOf) so the runtime behavior is secondary, but the radix fix is still correct practice." + }, + { + "file": "packages/preact-query/src/__tests__/useMutationState.test.tsx", + "line": 209, + "severity": "nit", + "comment": "This file is not covered by any plan step despite containing a template literal change. The plan's dependency graph shows zero dependencies, which is accurate for isolated test file changes but means the plan adds no structural insight." + } + ], + "summary": "The flow plan identified only 2 of 6 changed files and tracked just 2 additions/2 deletions out of the actual 73-line diff, severely underrepresenting the PR's scope. For a mechanical lint-fix PR in test files with no production impact, the plan's risk annotations (both marked 'high') are misleading since every change is a trivial, behavior-preserving ESLint autofix." + }, + "review_plan": { + "stats": { + "totalSteps": 2, + "totalAdditions": 2, + "totalDeletions": 2, + "independentFlows": 2, + "filesChanged": 2 + }, + "steps": [ + { + "order": 1, + "nodeId": "packages/preact-query/src/__tests__/usePrefetchInfiniteQuery.test.tsx::generateInfiniteQueryOptions", + "name": "generateInfiniteQueryOptions", + "file": "packages/preact-query/src/__tests__/usePrefetchInfiniteQuery.test.tsx", + "lines": [17, 42], + "type": "function", + "changeType": "modified", + "additions": 1, + "deletions": 1, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + }, + { + "order": 2, + "nodeId": "packages/preact-query/src/__tests__/useQuery.test.tsx::Component", + "name": "Component", + "file": "packages/preact-query/src/__tests__/useQuery.test.tsx", + "lines": [5785, 5797], + "type": "function", + "changeType": "modified", + "additions": 1, + "deletions": 1, + "role": "entry_point", + "risk": "high", + "calledBy": [], + "calls": [], + "riskReasons": ["entry_point"] + } + ], + "clusters": [], + "dependencies": [] + }, + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 5, + "efficiency": 8, + "overall": 6.0 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 6, + "risk_identification": 8, + "actionability": 6, + "efficiency": 7, + "overall": 7.0 + }, + "reasoning": "The baseline review correctly identifies all changes as mechanical and low-risk but lacks structural awareness. The flow-guided review adds value by critically evaluating the plan itself: it identifies that the plan only covers 2 of 6 files, tracks only 4 of 73 changed lines, and assigns misleading 'high' risk to trivial lint fixes. This meta-analysis of plan quality is genuinely useful -- it reveals the plan's blind spots (useQueries.test.tsx has the most changes but zero plan coverage) and corrects the false-positive risk ratings. The flow-guided review wins by surfacing plan deficiencies that could matter for more complex PRs.", + "winner": "flow_guided" + } +} diff --git a/evals/tauri-apps__tauri__15117.json b/evals/tauri-apps__tauri__15117.json new file mode 100644 index 0000000..a6081d6 --- /dev/null +++ b/evals/tauri-apps__tauri__15117.json @@ -0,0 +1,114 @@ +{ + "pr": "tauri-apps/tauri#15117", + "title": "Refactors", + "baseline_review": { + "comments": [ + { + "file": "crates/tauri-build/src/lib.rs", + "line": 414, + "severity": "medium", + "comment": "Changing `env::var` to `env::var_os` for `DEP_TAURI_DEV` alters the comparison semantics. `env::var_os` returns `OsString`, and comparing `OsString == \"true\"` works on most platforms but `OsString` can contain non-UTF-8 data. More importantly, `env::var` would fail if the value is not valid Unicode, while `env::var_os` silently accepts it. For environment variables set by Cargo build scripts this is safe, but it is a subtle behavioral change worth noting." + }, + { + "file": "crates/tauri-build/src/lib.rs", + "line": 541, + "severity": "low", + "comment": "Replacing `unwrap_or_else(|| BundleResources::List(Vec::new()))` with `unwrap_or(BundleResources::List(Vec::new()))` means the `Vec::new()` and `BundleResources::List(...)` are now eagerly evaluated on every call rather than lazily. Since `Vec::new()` is a zero-cost const operation and `BundleResources::List` is a trivial enum constructor, the performance difference is negligible, but this is technically a behavioral change from lazy to eager evaluation." + }, + { + "file": "crates/tauri-build/src/manifest.rs", + "line": 26, + "severity": "medium", + "comment": "Changing `all_cli_managed_features` from `Option>` to `Vec<&'static str>` removes the ability to distinguish between 'no managed features specified' (None) and 'empty list of managed features'. The old code had a `None` branch that fell back to filtering features starting with `allow-`. By removing this branch, any dependency that previously relied on the `None`/`allow-` fallback behavior will now use an empty vec filter, effectively filtering out ALL features. Verify no callers relied on the `None` path." + }, + { + "file": "crates/tauri-build/src/manifest.rs", + "line": 130, + "severity": "high", + "comment": "The removal of the `else` branch that filtered features by the `allow-` prefix is a semantic change, not just a refactor. Previously when `all_cli_managed_features` was `None`, features were filtered by the `starts_with(\"allow-\")` heuristic. Now that path is eliminated entirely. If any `AllowlistedDependency` was constructed without `all_cli_managed_features` (or with an empty vec), the feature diff logic would produce different results than before -- potentially silently allowing or rejecting features incorrectly." + }, + { + "file": "crates/tauri/src/protocol/asset.rs", + "line": 7, + "severity": "low", + "comment": "Switching from `tokio::fs::File` and async I/O traits to `std::fs::File` and synchronous `Read`/`Seek`/`Write` is the core simplification of the async-sync boundary. Since this code was already wrapped in `safe_block_on` (blocking the current thread to run async code), switching to synchronous I/O directly is correct and removes unnecessary async overhead." + }, + { + "file": "crates/tauri/src/protocol/asset.rs", + "line": 52, + "severity": "low", + "comment": "Removing the `safe_block_on` wrapper around `File::open` is correct since the function now uses `std::fs::File` which is synchronous. This simplifies the call chain and removes a potential deadlock vector where blocking on async I/O inside a synchronous context could interact badly with the tokio runtime." + } + ], + "summary": "This PR simplifies async-sync boundaries by replacing tokio async file I/O with std synchronous I/O (removing unnecessary `safe_block_on` wrappers) and switches several `env::var` calls to `env::var_os` for minor efficiency gains. The manifest.rs changes that remove the `Option` wrapper and the `allow-` prefix fallback branch deserve careful scrutiny as they alter feature-filtering semantics beyond a pure refactor." + }, + "flow_guided_review": { + "comments": [ + { + "file": "crates/tauri-build/src/lib.rs", + "line": 414, + "severity": "low", + "comment": "The `env::var` to `env::var_os` migration for `DEP_TAURI_DEV` is a minor efficiency improvement -- `var_os` avoids the UTF-8 validation and allocation that `var` performs. Since the comparison `== \"true\"` works correctly with `OsString` on all supported platforms, this is a safe change." + }, + { + "file": "crates/tauri-build/src/lib.rs", + "line": 461, + "severity": "low", + "comment": "Similarly, `CARGO_CFG_TARGET_OS` is switched to `var_os`. The downstream comparisons (`== \"ios\"`, `== \"android\"`, `.contains(\"windows\")`) all work correctly with `OsString` since these are ASCII-only comparisons. Note that `target_os` is later used with `.contains(\"windows\")` -- `OsString` does not have a `contains` method, so this would need to use `to_str()` or `to_string_lossy()`. If the diff is complete and compiles, this is fine, but worth verifying." + }, + { + "file": "crates/tauri-build/src/lib.rs", + "line": 541, + "severity": "low", + "comment": "The `unwrap_or_else` to `unwrap_or` change is a minor style simplification. `Vec::new()` is const and zero-cost, so eager evaluation has no measurable impact. This is a clean, safe refactor." + }, + { + "file": "crates/tauri-build/src/manifest.rs", + "line": 26, + "severity": "high", + "comment": "This is the most impactful change in the PR despite appearing as a simple type simplification. Changing `all_cli_managed_features` from `Option` to `Vec` eliminates the distinction between None (use `allow-` prefix heuristic) and Some (use explicit list). The two construction sites in `check()` both previously used `Some(...)`, so the unwrapping is safe there. However, the removed `None` branch in `check_features` (line ~130) contained fallback logic filtering by `starts_with(\"allow-\")`. If any future or external caller constructs an `AllowlistedDependency` without populating `all_cli_managed_features`, the empty vec will filter out ALL features rather than falling back to the heuristic -- a silent behavioral regression." + }, + { + "file": "crates/tauri-build/src/manifest.rs", + "line": 130, + "severity": "medium", + "comment": "The consolidation of the two `features_diff` call sites into one is a clean simplification, but it couples correctness to the assumption that `all_cli_managed_features` is always populated. Since the struct is private and only constructed in `check()` where both instances provide explicit feature lists, this is currently safe. Adding a comment or debug_assert noting this invariant would guard against future regressions." + }, + { + "file": "crates/tauri/src/protocol/asset.rs", + "line": 7, + "severity": "medium", + "comment": "Replacing `tokio::fs::File` with `std::fs::File` and removing the `safe_block_on` wrapper is the headline change referenced in the PR description. This is correct -- the caller was already in a synchronous context using `safe_block_on` to bridge to async, so cutting out the async layer removes unnecessary complexity. The diff is truncated so the full extent of the async-to-sync migration in the file read/seek/write operations cannot be fully verified, but the pattern is sound." + }, + { + "file": "crates/tauri/src/protocol/asset.rs", + "line": 52, + "severity": "low", + "comment": "Removing `safe_block_on` from `File::open` eliminates a potential source of runtime panics (calling `block_on` when already inside an async context). The synchronous `File::open` is the right choice when the surrounding code is already synchronous." + } + ], + "summary": "The PR delivers on its stated goal of simplifying async-sync code boundaries, with the asset protocol handler correctly migrating from tokio async I/O to std synchronous I/O. The manifest.rs changes that flatten `Option` to `Vec` are logically safe given current usage but silently remove a fallback heuristic, which could cause subtle regressions if the construction pattern changes in the future." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 8, + "actionability": 6, + "efficiency": 7, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 6, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 7.0 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same key risk: the manifest.rs change removes a fallback code path (the `allow-` prefix heuristic) that could cause silent behavioral regressions. The flow-guided review provides slightly better context by tracing the data flow from struct definition to construction sites to the filtering logic, confirming that current callers always provide explicit feature lists (making the change safe today) while noting the future regression risk. However, because the review plan was empty (no steps, clusters, or dependencies), the flow-guided review had minimal structural advantage -- it largely mirrors the baseline with better contextualization. The baseline review arguably over-flags the `env::var_os` changes and the `unwrap_or` change as medium severity when they are routine Rust idiom improvements, while the flow-guided review more accurately calibrates these as low severity. The margin between the two is narrow given the empty plan." + } +} \ No newline at end of file diff --git a/evals/tokio-rs__tokio__7968.json b/evals/tokio-rs__tokio__7968.json new file mode 100644 index 0000000..f94209f --- /dev/null +++ b/evals/tokio-rs__tokio__7968.json @@ -0,0 +1,102 @@ +{ + "pr": "tokio-rs/tokio#7968", + "title": "ci: workaround for OpenOptionsExt in 1.94.0", + "baseline_review": { + "comments": [ + { + "file": ".github/workflows/ci.yml", + "line": 18, + "severity": "major", + "comment": "Changing `rust_stable` from a pinned version '1.93.1' to the floating 'stable' channel means CI will silently pick up new Rust releases. This removes reproducibility -- a future Rust release could introduce breaking changes or new warnings that cause CI failures without any code change. The original pinning was intentional (the comment says 'Change to specific Rust release to pin'). While needed as a workaround for the 1.94.0 OpenOptionsExt issue, this should be reverted to a pinned version (e.g., '1.94.0' or '1.94.1') once the upstream Rust issue is resolved." + }, + { + "file": ".github/workflows/ci.yml", + "line": 1060, + "severity": "minor", + "comment": "The WASI job overrides `rust_stable` back to '1.93.1' via a job-level `env` block. This is a reasonable workaround if WASI targets are not yet supported on newer stable Rust, but there is no comment explaining why this exception exists. A comment noting the WASI-specific pin reason would help future maintainers understand when this override can be removed." + }, + { + "file": "tokio/src/fs/open_options/mock_open_options.rs", + "line": 18, + "severity": "minor", + "comment": "The comment referencing https://github.com/rust-lang/rust/issues/153486 is helpful for tracking the upstream issue. However, the methods are now defined as inherent methods on MockOpenOptions rather than trait implementations. If any test code calls these methods through the OpenOptionsExt trait (e.g., via `use std::os::unix::fs::OpenOptionsExt; opts.mode(0o644)`), those calls will fail to compile in test mode since the trait import is now gated with `not(test)`. Verify that all test call sites use direct method syntax." + }, + { + "file": "tokio/src/fs/open_options.rs", + "line": 19, + "severity": "minor", + "comment": "The `#[cfg(all(unix, not(test)))]` and `#[cfg(all(windows, not(test)))]` guards on the trait imports mean that in test mode, OpenOptions will not implement the OpenOptionsExt traits. Any code in the crate that relies on these trait bounds (e.g., generic functions requiring `T: OpenOptionsExt`) will not compile under test. This is likely safe since tokio's OpenOptions wraps the std type and exposes methods directly, but it is a subtle behavioral difference between test and production builds." + }, + { + "file": "tokio/src/fs/open_options/uring_open_options.rs", + "line": 3, + "severity": "minor", + "comment": "The `#[cfg(not(test))]` guard on the OpenOptionsExt import in uring_open_options.rs mirrors the same pattern used in open_options.rs. This is consistent, but the uring path is Unix-only and was previously importing the trait unconditionally. Ensure the uring-specific code paths that call extension methods (like `custom_flags`) work correctly in test mode with the mock's inherent methods." + } + ], + "summary": "This PR works around a Rust 1.94.0 breaking change in OpenOptionsExt by moving extension methods from trait implementations to inherent methods on the mock type, and unpinning the stable Rust version in CI. The approach is sound as a temporary workaround, but the switch from a pinned Rust version to floating 'stable' reduces CI reproducibility and should be reverted once the upstream issue is fixed." + }, + "flow_guided_review": { + "comments": [ + { + "file": ".github/workflows/ci.yml", + "line": 18, + "severity": "major", + "comment": "The review plan is empty (zero steps, zero clusters), so there is no flow guidance to follow. Reviewing the CI change independently: switching from pinned '1.93.1' to floating 'stable' is a significant CI hygiene regression. The existing comment on line 17 explicitly says 'Change to specific Rust release to pin', contradicting the new value. This PR should either pin to '1.94.0' (accepting the workaround is needed) or document that the floating channel is intentional and temporary. A TODO or tracking issue link in the CI file would make the revert path clear." + }, + { + "file": ".github/workflows/ci.yml", + "line": 1060, + "severity": "minor", + "comment": "Without flow plan dependencies to trace, this WASI-specific pin stands as an isolated change. The job-level env override to '1.93.1' creates a version split: most jobs run latest stable while WASI stays on 1.93.1. This divergence could mask WASI-specific issues on newer Rust versions. Adding a comment with context (e.g., 'WASI targets not yet supported on 1.94+' or linking to a tracking issue) would make the override self-documenting." + }, + { + "file": "tokio/src/fs/open_options/mock_open_options.rs", + "line": 18, + "severity": "minor", + "comment": "The mock now defines extension methods as inherent methods rather than trait implementations. Since the review plan provides no dependency or risk information, the key concern is API surface equivalence: the method signatures (custom_flags, mode, access_mode, share_mode, attributes, security_qos_flags) match the standard library's OpenOptionsExt traits exactly. This is correct. The PR description notes this approach is 'robust against any future issues too' which is accurate -- inherent methods avoid trait resolution entirely." + }, + { + "file": "tokio/src/fs/open_options.rs", + "line": 19, + "severity": "minor", + "comment": "The cfg gates split trait availability between test and non-test builds. Without flow plan risk annotations, manual analysis shows this is safe because tokio's public OpenOptions API exposes these as inherent methods (e.g., `pub fn mode(&mut self, mode: u32)`) that delegate to the inner std type, not via trait bounds. The trait import is only needed for the delegation to work on the real std::fs::OpenOptions." + }, + { + "file": "tokio/src/fs/open_options/mock_open_options.rs", + "line": 22, + "severity": "nit", + "comment": "The comment references rust-lang/rust#153486 which is the tracking issue for the breaking change. Consider also referencing tokio-rs/tokio#7955 (mentioned in the PR description) so that both the upstream Rust issue and the local tracking issue are linked. This makes it easier to find the revert path when the upstream fix lands." + }, + { + "file": "tokio/src/fs/open_options/uring_open_options.rs", + "line": 3, + "severity": "nit", + "comment": "The uring_open_options.rs change is consistent with the main open_options.rs pattern. The unused import warning that would occur in test mode is correctly avoided by the cfg gate. This file only applies to io-uring builds on Linux, limiting the blast radius of any issues." + } + ], + "summary": "The review plan is empty (zero steps, zero dependencies, zero clusters), providing no flow guidance to leverage. The flow-guided review therefore degrades to a manual structural analysis of the four changed files, reaching similar conclusions to a baseline review. The workaround is mechanically correct -- inherent methods on the mock avoid the trait resolution bug -- but the CI change from pinned to floating Rust versions is a reproducibility concern that should be addressed with a pin or tracking comment." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 3, + "risk_identification": 7, + "actionability": 6, + "efficiency": 7, + "overall": 6.0 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 2, + "risk_identification": 7, + "actionability": 7, + "efficiency": 6, + "overall": 5.8 + } + }, + "winner": "tie", + "reasoning": "The review plan is completely empty (zero steps, zero clusters, zero dependencies), giving the flow-guided review no structural advantage. Both reviews identify the same core issues: the CI reproducibility regression from unpinning the Rust version, the correctness of moving trait methods to inherent methods on the mock, and the cfg gate implications for test vs production builds. The flow-guided review explicitly acknowledges the empty plan and attempts manual structural analysis, but without actual flow information it cannot provide the cross-file dependency tracing or risk-prioritized ordering that normally differentiates it. The baseline review is slightly more efficient since it does not spend tokens discussing the absence of a plan. The flow-guided review adds marginally better actionability (suggesting a pin to 1.94.0 specifically, and recommending dual issue references), but these differences are too small to declare a winner. For a CI workaround PR with only 47 changed lines and no complex data flow, both approaches converge on the same findings." + } +} diff --git a/evals/tokio-rs__tokio__7978.json b/evals/tokio-rs__tokio__7978.json new file mode 100644 index 0000000..537292e --- /dev/null +++ b/evals/tokio-rs__tokio__7978.json @@ -0,0 +1,108 @@ +{ + "pr": "tokio-rs/tokio#7978", + "title": "docs(runtime): document FD table pre-warming workaround", + "baseline_review": { + "comments": [ + { + "file": "examples/prewarm-fd-table.rs", + "line": 40, + "severity": "major", + "comment": "The `prewarm_fd_table` function uses `unsafe { libc::fcntl(...) }` and `unsafe { OwnedFd::from_raw_fd(raw) }` but does not document the safety invariants in a `// SAFETY:` comment. For an example that users will copy-paste into production code, this is important -- especially the `from_raw_fd` call, which requires the caller to guarantee that `raw` is a valid, exclusively-owned FD. The check on line 47 (`if raw < 0`) validates the FD before wrapping, but stating the invariant explicitly would align with Rust community norms and make the example more instructive." + }, + { + "file": "examples/prewarm-fd-table.rs", + "line": 62, + "severity": "minor", + "comment": "The `prewarm_fd_table_safe` function clones `/dev/null` `target` times (up to 10,000 by default). Each `try_clone()` calls `fcntl(F_DUPFD_CLOEXEC, 0)`, so this creates 10,000 open FDs simultaneously before dropping them. On a system with a low `RLIMIT_NOFILE` (e.g. default 1024 on many distros), this will fail with EMFILE after ~1020 clones, and the error message won't explain why. A comment noting this limitation -- or checking rlimit first -- would make the 'safe' alternative more robust as a copy-paste example." + }, + { + "file": "examples/prewarm-fd-table.rs", + "line": 73, + "severity": "minor", + "comment": "The `FD_TARGET` constant is hardcoded to 10,000 with no guidance on how to choose the right value. The doc comment above mentions 'at least your expected peak FD count' and 'must not exceed RLIMIT_NOFILE', but the code itself does not query `getrlimit(RLIMIT_NOFILE)` to clamp the target. Users copying this example may hit EINVAL or EMFILE if their ulimit is below 10,000. Adding a `min(target, rlimit.rlim_cur)` clamp would make the example safer to adopt." + }, + { + "file": "tokio/src/runtime/mod.rs", + "line": 378, + "severity": "minor", + "comment": "The new 'Performance tuning' section adds a link `[prewarm-fd-table]` pointing to a GitHub blob URL on the `master` branch. If tokio's default branch is `main` (which it is), this link will 404. The URL should use `main` instead of `master`: `https://github.com/tokio-rs/tokio/blob/main/examples/prewarm-fd-table.rs`." + }, + { + "file": "examples/Cargo.toml", + "line": 27, + "severity": "nit", + "comment": "The `libc` dependency is added under `[target.'cfg(target_os = \"linux\")'.dev-dependencies]` but the example binary definition at the bottom has no `required-features` or `cfg` gate. Running `cargo run --example prewarm-fd-table` on macOS will compile but the `main()` function will just build the runtime without pre-warming (the `#[cfg(target_os = \"linux\")]` blocks are empty). This is arguably fine behavior, but a doc comment in `main()` noting the no-op on non-Linux would help users who try it on macOS." + }, + { + "file": "examples/prewarm-fd-table.rs", + "line": 84, + "severity": "nit", + "comment": "The `rt.block_on(async {});` call runs an empty future, which makes this example not demonstrate any real async work after pre-warming. While this is fine for a minimal demonstration, adding a brief comment like `// Replace with your actual server logic` would signal that this is a placeholder." + } + ], + "summary": "This is a documentation-only PR adding an example for a well-known Linux FD table pre-warming workaround. The code is correct and the explanation is thorough, but the example would benefit from explicit safety comments on the unsafe blocks and a rlimit-aware clamp on the target FD count, since users will copy-paste this into production. The runtime module doc link may point to the wrong default branch." + }, + "flow_guided_review": { + "comments": [ + { + "file": "tokio/src/runtime/mod.rs", + "line": 395, + "severity": "major", + "comment": "The reference link `[prewarm-fd-table]: https://github.com/tokio-rs/tokio/blob/master/examples/prewarm-fd-table.rs` uses `master` as the branch name. Tokio's default branch is `main`, so this link will 404 once the PR merges. This is the primary entry point where users will discover the example from the runtime module docs, so a broken link defeats the purpose of the PR. Should be `/blob/main/examples/...`." + }, + { + "file": "examples/prewarm-fd-table.rs", + "line": 40, + "severity": "major", + "comment": "Both `unsafe` blocks lack `// SAFETY:` comments explaining the invariants being upheld. The `libc::fcntl` call requires a valid FD (guaranteed by the File::open above), and `OwnedFd::from_raw_fd` requires exclusive ownership of a valid FD (guaranteed by the `raw < 0` error check and fcntl returning a new FD). Since this example exists specifically to be copied into production codebases, modeling best practices for unsafe Rust is important -- especially in the tokio repository which sets community norms." + }, + { + "file": "examples/prewarm-fd-table.rs", + "line": 73, + "severity": "minor", + "comment": "The hardcoded `FD_TARGET = 10_000` can silently fail on systems where `RLIMIT_NOFILE` is below 10,000 (the default soft limit is 1024 on many distros). The fcntl call will return EINVAL, which is caught and printed as a warning, but the error message 'failed to pre-warm FD table: Invalid argument' won't help users diagnose that they need to raise their ulimit. Querying rlimit or at minimum mentioning `ulimit -n` in the error/comment path would make the example more self-service." + }, + { + "file": "examples/prewarm-fd-table.rs", + "line": 56, + "severity": "minor", + "comment": "The 'safe' alternative `prewarm_fd_table_safe` opens O(target) simultaneous FDs, which means for FD_TARGET=10,000 it holds 10,000 open FDs at once. This is fundamentally different from the fcntl approach which only briefly holds 2 FDs. On a system where rlimit allows 10,000 but the process already has some FDs open, this function will fail earlier than the fcntl version. The doc comment says 'Requires O(n) syscalls instead of one' but doesn't mention the O(n) simultaneous FD consumption, which is the more impactful difference." + }, + { + "file": "spellcheck.dic", + "line": 1, + "severity": "nit", + "comment": "The word count is bumped from 312 to 313 and 'RCU' is added. This is correct and consistent -- RCU appears in the example's doc comments and would trigger spellcheck failures without this entry." + }, + { + "file": "examples/Cargo.toml", + "line": 27, + "severity": "nit", + "comment": "The `libc = \"0.2\"` dependency is gated behind `cfg(target_os = \"linux\")` which correctly avoids pulling libc on non-Linux platforms for example builds. The example binary definition itself has no cfg gate, but the code uses `#[cfg(target_os = \"linux\")]` on the function bodies, which is the standard pattern in tokio's examples (matching the `dump` example which uses `cfg(all(tokio_unstable, target_os = \"linux\"))`)." + } + ], + "summary": "The PR adds a well-motivated and clearly-written example for FD table pre-warming, a known latency pitfall in multi-threaded Linux processes. The most impactful issue is the broken `master` branch link in the runtime module docs (should be `main`), which would make the primary discovery path for this example non-functional. The unsafe blocks should include SAFETY comments to model best practices for an example users will copy into production." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 7, + "actionability": 8, + "efficiency": 8, + "overall": 7.0 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same core issues: the broken master/main branch link, missing SAFETY comments on unsafe blocks, and the rlimit concern with the hardcoded target. The review plan was empty (0 steps, 0 clusters), so the flow-guided review had no structural advantage to leverage -- both reviews are effectively working from the diff alone. The flow-guided review edges ahead slightly on actionability by more precisely explaining *why* the broken link matters (it's the primary discovery path from the runtime docs) and on efficiency by avoiding the less impactful 'empty async block' nit. Both reviews correctly identify this as a low-risk documentation PR where the main concerns are copy-paste safety and link correctness. The margin is narrow because without a meaningful review plan, the flow-guided approach cannot demonstrate its structural analysis strengths." + } +} diff --git a/evals/tokio-rs__tokio__7987.json b/evals/tokio-rs__tokio__7987.json new file mode 100644 index 0000000..695ba5a --- /dev/null +++ b/evals/tokio-rs__tokio__7987.json @@ -0,0 +1,96 @@ +{ + "pr": "tokio-rs/tokio#7987", + "title": "remove Rust 1.94.0 workarounds", + "baseline_review": { + "comments": [ + { + "file": ".cirrus.yml", + "line": 19, + "severity": "low", + "comment": "Removing `ca_root_nss` from all three FreeBSD setup scripts (64-bit, docs, 32-bit) is consistent. The package was presumably added as a workaround for the Cargo TLS certificate issue (rust-lang/cargo#16357). Verify that FreeBSD base images now ship with sufficient root certificates for `curl https://sh.rustup.rs` to succeed without `ca_root_nss`." + }, + { + "file": ".github/workflows/ci.yml", + "line": 1063, + "severity": "medium", + "comment": "Removing the `rust_stable: '1.93.1'` pin for the WASM jobs means they will now use whatever `env.rust_stable` is set at the workflow level (presumably 1.94.x or later). This is correct now that 1.94.1 is available, but confirm that the top-level `rust_stable` env var resolves to 1.94.1+ and not 1.94.0 which had the original cargo bug." + }, + { + "file": "tokio/src/fs/open_options.rs", + "line": 19, + "severity": "low", + "comment": "Changing `#[cfg(all(unix, not(test)))]` to `#[cfg(unix)]` means `OpenOptionsExt` is now imported in test builds as well. This is correct because the mock now properly implements the `OpenOptionsExt` trait (see mock_open_options.rs changes), so the import is needed in all builds." + }, + { + "file": "tokio/src/fs/open_options/mock_open_options.rs", + "line": 19, + "severity": "medium", + "comment": "The platform-specific methods are moved from inherent `impl` blocks into proper `OpenOptionsExt` trait implementations. This is the revert of the workaround for rust-lang/rust#153486. The mock now correctly implements the trait rather than duplicating its methods as inherent methods, which ensures type-checking and dispatch match the real `OpenOptions`." + }, + { + "file": "tokio/src/fs/open_options/uring_open_options.rs", + "line": 1, + "severity": "low", + "comment": "The `#[cfg(not(test))]` guard on the `OpenOptionsExt` import is removed, consolidating the import into a single `use` statement. This is consistent with the mock_open_options.rs changes where the mock now implements the trait directly, so the import is needed in test builds too." + } + ], + "summary": "This PR cleanly reverts two temporary workarounds for Rust 1.94.0 compiler/cargo bugs, now that Rust 1.94.1 is available with the fixes. The changes are mechanical and self-consistent: CI version pins are removed, FreeBSD certificate workarounds are dropped, and mock trait implementations are restored to their proper form using `OpenOptionsExt` trait impls instead of inherent methods." + }, + "flow_guided_review": { + "comments": [ + { + "file": "tokio/src/fs/open_options/mock_open_options.rs", + "line": 19, + "severity": "medium", + "comment": "The core change: mock `OpenOptions` now implements `OpenOptionsExt` as a proper trait impl rather than inherent methods. This is the correct restoration that was blocked by rust-lang/rust#153486. All callers that invoke `.custom_flags()`, `.mode()`, `.access_mode()`, etc. through the trait interface will now dispatch correctly in test builds, matching production behavior." + }, + { + "file": "tokio/src/fs/open_options.rs", + "line": 19, + "severity": "low", + "comment": "The `not(test)` guard removal on the `OpenOptionsExt` import is a direct consequence of the mock_open_options.rs change. Since the mock now implements the trait, test code needs the trait in scope for method resolution. The `open_options.rs` and `uring_open_options.rs` import changes are tightly coupled to the mock change -- all three must land together or the test build breaks." + }, + { + "file": "tokio/src/fs/open_options/uring_open_options.rs", + "line": 1, + "severity": "low", + "comment": "Consolidating `use std::io` and `use std::os::unix::fs::OpenOptionsExt` into a single `use std::{io, os::unix::fs::OpenOptionsExt}` is a minor style cleanup that accompanies the `cfg(not(test))` guard removal. This completes the trio of Rust-side changes required to revert the trait workaround." + }, + { + "file": ".github/workflows/ci.yml", + "line": 1063, + "severity": "medium", + "comment": "The WASM job version pin (`rust_stable: '1.93.1'`) was the workaround for the cargo resolver bug (rust-lang/cargo#16357). Removing it is independent of the mock trait changes. This is an important CI configuration change -- if the workflow-level `rust_stable` env var hasn't been updated to 1.94.1+, the WASM jobs could hit the original cargo bug on 1.94.0." + }, + { + "file": ".cirrus.yml", + "line": 19, + "severity": "low", + "comment": "The `ca_root_nss` removal from all three FreeBSD CI jobs is the second independent workaround being reverted (also for the cargo/TLS issue). This is decoupled from the Rust code changes and only affects FreeBSD CI environments. The three identical removals (lines 19, 37, 47) are consistent." + } + ], + "summary": "The PR contains two independent revert flows: (1) restoring proper `OpenOptionsExt` trait implementations in the mock, with coordinated import changes across three Rust files, and (2) removing CI workarounds (version pin and FreeBSD certificate package) across two CI config files. Both flows are clean reverts with no cross-dependencies between them, and all changes are consistent with the referenced upstream fixes in Rust 1.94.1." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 6.6 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 7, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 7.4 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review provides meaningfully better structure by identifying the two independent revert flows (Rust trait restoration vs CI workaround removal) and explicitly calling out the tight coupling between the three Rust file changes (mock_open_options.rs, open_options.rs, uring_open_options.rs must land together). The baseline review covers each file individually but treats them as isolated changes rather than recognizing the dependency structure. Both reviews correctly identify the key risk (WASM job version resolution) and the mechanical nature of the changes. The flow-guided review's advantage is modest because the empty review plan provides no guidance -- the flow awareness comes from the reviewer's own analysis of the diff -- but the structured thinking about independent flows and coupled changes adds clarity. Risk identification is equal since both flag the same CI version concern." + } +} diff --git a/evals/trpc__trpc__7294.json b/evals/trpc__trpc__7294.json new file mode 100644 index 0000000..9650fbf --- /dev/null +++ b/evals/trpc__trpc__7294.json @@ -0,0 +1,108 @@ +{ + "pr": "trpc/trpc#7294", + "title": "chore: Review stale skills (manual)", + "baseline_review": { + "comments": [ + { + "file": "packages/client/skills/links/SKILL.md", + "line": 276, + "severity": "low", + "comment": "The inline comment `// should be the same or lower than the server's maxBatchSize` is helpful but slightly ambiguous -- it could be read as 'this value should be the same or lower' or 'maxItems should be the same or lower'. Consider rephrasing to `// maxItems should match or be lower than the server's maxBatchSize` for clarity." + }, + { + "file": "packages/client/skills/links/SKILL.md", + "line": 280, + "severity": "medium", + "comment": "The updated guidance says 'set `maxItems` to the same or lower value so the client auto-splits batches instead of triggering a `400 Bad Request`'. However, the `httpBatchLink` does not actually auto-split batches when `maxItems` is exceeded -- it prevents adding more items to a batch beyond that limit and starts a new batch. The wording 'auto-splits' could mislead users into thinking an existing oversized batch will be retroactively split. Consider 'so the client limits batch size' instead." + }, + { + "file": "packages/server/skills/adapter-express/SKILL.md", + "line": 110, + "severity": "medium", + "comment": "The diff for adapter-express appears truncated -- only the beginning of the 'Limiting batch size with maxBatchSize' section is visible, cutting off mid-code-block. If the full section mirrors the other adapters, verify the code example is complete and the closing explanation paragraph is present." + }, + { + "file": "packages/server/skills/adapter-aws-lambda/SKILL.md", + "line": 125, + "severity": "low", + "comment": "The `maxBatchSize` examples across all adapter skills (aws-lambda, express, fastify, standalone, nextjs-pages-router) use a hardcoded value of `10`. While this is fine as an example, the skills don't mention what happens when `maxBatchSize` is not set (i.e., the default is unlimited). Adding a note about the default behavior would help users understand when they need to opt in." + }, + { + "file": "packages/next/skills/nextjs-pages-router/SKILL.md", + "line": 340, + "severity": "low", + "comment": "The nextjs-pages-router example uses a relative import `'../../../server/routers/_app'` which is specific to a particular project structure. Other adapter examples use `'./router'`. Consider using a more generic import path for consistency across skills." + }, + { + "file": "packages/openapi/skills/openapi/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The openapi skill version was bumped from `11.14.0-alpha` to `11.15.1-alpha`, maintaining the alpha suffix. This is consistent with the PR description, but worth confirming that the openapi package actually published `11.15.1-alpha` to npm, since an incorrect version string in the skill metadata could cause validation issues downstream." + } + ], + "summary": "This PR is a straightforward maintenance update bumping `library_version` across all 24 tRPC SKILL.md files and adding `maxBatchSize` documentation to six server adapter skills plus the links skill. The changes are consistent and low-risk, with the main concern being slightly imprecise language around client-side batch splitting behavior that could mislead users." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/client/skills/links/SKILL.md", + "line": 280, + "severity": "medium", + "comment": "The links skill is the client-side counterpart to all the server adapter `maxBatchSize` additions. The updated text says the client 'auto-splits batches' when `maxItems` is set, but `httpBatchLink` actually caps each batch at `maxItems` and starts a new one -- it does not split an already-formed batch. Since every server adapter skill now tells users to 'set `maxItems` on your client's `httpBatchLink` to the same value', this single sentence in the links skill is the canonical guidance users will follow. Getting the wording precise here matters." + }, + { + "file": "packages/client/skills/links/SKILL.md", + "line": 276, + "severity": "low", + "comment": "The inline comment `// should be the same or lower than the server's maxBatchSize` is a new cross-reference between client and server configuration. This is good -- it connects the client `maxItems` to the server `maxBatchSize` concept introduced in the adapter skills. However, the comment only appears inside the code block, so users scanning the prose might miss this constraint until they read the paragraph below." + }, + { + "file": "packages/server/skills/adapter-aws-lambda/SKILL.md", + "line": 125, + "severity": "low", + "comment": "All six server adapter skills (standalone, express, fetch, fastify, aws-lambda, nextjs-pages-router) add an identical 'Limiting batch size with maxBatchSize' section with the same explanatory paragraph: 'Requests batching more than `maxBatchSize` operations are rejected with a `400 Bad Request` error.' This is consistent but none of the adapters mention the default value (unlimited/Infinity). Since the links skill explicitly states `maxItems` defaults to `Infinity`, the server-side skills should note that `maxBatchSize` also defaults to unlimited for symmetry." + }, + { + "file": "packages/next/skills/nextjs-pages-router/SKILL.md", + "line": 340, + "severity": "low", + "comment": "The nextjs-pages-router `maxBatchSize` example uses `createNextApiHandler` with a relative import `'../../../server/routers/_app'`, matching the existing patterns in this skill file. Other adapter skills use `'./router'`. This inconsistency is inherited from the pre-existing examples in each skill, so it is not a regression, but future readers comparing across skills may be confused." + }, + { + "file": "packages/server/skills/adapter-express/SKILL.md", + "line": 110, + "severity": "medium", + "comment": "The diff for adapter-express is truncated, cutting off mid-code-block. If the full addition mirrors the other adapter skills (a complete code example + explanatory paragraph placed before the 'Common Mistakes' section), then the placement is correct. However, verify the section is complete in the actual file -- a truncated code block in a SKILL.md would cause validation failure." + }, + { + "file": "packages/openapi/skills/openapi/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The openapi skill is the only one with an `-alpha` suffix in `library_version`. It was correctly bumped from `11.14.0-alpha` to `11.15.1-alpha`. Since the review plan has no dependency information, there is no way to verify whether the openapi skill depends on any of the server adapters that now document `maxBatchSize`. If it does, the openapi skill might benefit from mentioning batch limits as well, since OpenAPI endpoints typically handle one operation at a time but could interact with batched tRPC calls in hybrid setups." + } + ], + "summary": "With an empty review plan (no steps, clusters, or dependencies), the flow-guided review focused on cross-skill consistency: the links skill is the single client-side touchpoint that all six server adapter skills reference, making its batch-splitting language the most impactful thing to get right. The version bumps are uniform and the `maxBatchSize` sections are identically structured across adapters, though none document the default unlimited behavior that the links skill already calls out for `maxItems`." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 6, + "flow_awareness": 3, + "risk_identification": 5, + "actionability": 6, + "efficiency": 7, + "overall": 5.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 5, + "actionability": 6, + "efficiency": 6, + "overall": 5.8 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews are constrained by the nature of this PR -- it is a documentation-only change with version bumps, so there is limited code logic to analyze. The review plan was entirely empty (zero steps, zero clusters, zero dependencies), which severely limited the flow-guided review's advantage. The flow-guided review still edged ahead by identifying the cross-skill relationship: the links skill serves as the single client-side reference point that all six server adapter skills direct users to, making its batch-splitting wording the highest-leverage concern. It also noted the asymmetry in default-value documentation (client mentions Infinity, server skills do not). The baseline review caught individual file-level issues but treated each skill in isolation. The margin is narrow because this PR has minimal complexity -- both reviews converge on the same core concern about imprecise 'auto-splits' language." + } +} diff --git a/evals/trpc__trpc__7295.json b/evals/trpc__trpc__7295.json new file mode 100644 index 0000000..277c707 --- /dev/null +++ b/evals/trpc__trpc__7295.json @@ -0,0 +1,72 @@ +{ + "pr": "trpc/trpc#7295", + "title": "chore: Review stale skills (v11.16.0)", + "baseline_review": { + "comments": [ + { + "file": "packages/client/skills/client-setup/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The version bump from `11.15.1` to `11.16.0` is applied here and across 23 other SKILL.md files. Since all non-openapi skills jump from a patch version (11.15.1) to a minor version (11.16.0), it would be worth confirming that no intermediate minor versions (e.g. 11.15.2, 11.15.3) were skipped unintentionally." + }, + { + "file": "packages/openapi/skills/openapi/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The openapi skill correctly maintains the `-alpha` suffix (`11.15.1-alpha` to `11.16.0-alpha`), which is consistent with the package's pre-release status. Good that this wasn't accidentally promoted to a stable version tag." + }, + { + "file": "packages/server/skills/error-handling/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The diff appears truncated for this file and presumably several others (the PR description says 24 files changed but only ~14 are shown in the diff). The remaining files likely follow the same mechanical pattern, but a reviewer should verify all 24 files were updated consistently." + } + ], + "summary": "This is a straightforward metadata-only PR that bumps `library_version` in 24 SKILL.md files from 11.15.1 to 11.16.0 (and 11.15.1-alpha to 11.16.0-alpha for openapi). No behavioral or API changes are introduced, and the risk is minimal -- the main thing to verify is that all skill files were updated consistently and no files were missed." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/client/skills/client-setup/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The review plan is empty (0 steps, 0 clusters, 0 dependencies), which is expected for a metadata-only change with no code flows to trace. The version bump from `11.15.1` to `11.16.0` is a pure find-and-replace operation across SKILL.md frontmatter. Without any code changes, there are no execution flows, entry points, or risk surfaces to analyze." + }, + { + "file": "packages/openapi/skills/openapi/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The openapi package correctly uses the `-alpha` version suffix. Since there are no dependency edges in the plan and no `requires` fields were changed in any SKILL.md, the inter-skill dependency graph remains intact. The only change is the version metadata, which is consumed by the `@tanstack/intent validate` tooling mentioned in the PR description." + }, + { + "file": "packages/client/skills/client-setup/SKILL.md", + "line": 10, + "severity": "low", + "comment": "The PR description mentions validation was run with `npx @tanstack/intent validate` per package directory. Since this is the only verification mechanism for SKILL.md correctness, it would be good to confirm the validation output was clean for all five package directories listed (client, next, openapi, server, tanstack-react-query)." + } + ], + "summary": "With an empty flow graph (no steps, clusters, or dependencies), the flow-guided review converges with the baseline: this is a mechanical version metadata update with no code paths to trace. The only verification is that all 24 files were updated consistently and `@tanstack/intent validate` passed across all five package skill directories." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 8, + "overall": 6.4 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 6, + "actionability": 6, + "efficiency": 8, + "overall": 6.4 + } + }, + "winner": "tie", + "reasoning": "For this purely mechanical metadata-only PR, both review approaches yield equivalent results. The flow-guided plan is empty (0 steps, 0 clusters, 0 dependencies) because there are no code changes, execution flows, or risk surfaces to analyze. The baseline review correctly identifies the key concern (consistency across all 24 files) and the openapi alpha suffix preservation. The flow-guided review adds no incremental value because there are no flows to guide -- it simply restates the same observations with plan-awareness framing. Neither approach can meaningfully differentiate on a version bump PR with no behavioral changes." + } +} diff --git a/evals/trpc__trpc__7303.json b/evals/trpc__trpc__7303.json new file mode 100644 index 0000000..e19433a --- /dev/null +++ b/evals/trpc__trpc__7303.json @@ -0,0 +1,137 @@ +{ + "pr": { + "url": "https://github.com/trpc/trpc/pull/7303", + "owner": "trpc", + "repo": "trpc", + "number": 7303, + "title": "feat: Add subscription inferrence helpers", + "files_changed": 5, + "additions": 134, + "deletions": 33, + "language": "typescript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/server/src/unstable-core-do-not-import/procedure.ts", + "line": 97, + "severity": "major", + "comment": "inferSubscriptionOutput handles LegacyObservableSubscriptionProcedure by returning inferProcedureOutput directly, but for async-iterable subscriptions it unwraps via inferAsyncIterableYield. If a subscription procedure uses neither pattern (e.g., returns a raw value or a custom iterable-like), the fallback branch silently applies inferAsyncIterableYield which may resolve to `never` or an unexpected type. Consider adding a third branch or a constraint that ensures the output is actually an AsyncIterable." + }, + { + "file": "packages/server/src/unstable-core-do-not-import/procedure.ts", + "line": 93, + "severity": "minor", + "comment": "inferSubscriptionInput is defined as a simple alias for inferProcedureInput constrained to AnySubscriptionProcedure. This adds no new type-level logic -- it only narrows the input constraint. Consider adding a JSDoc comment explaining why this exists (developer convenience / discoverability) so it does not appear to be dead code to future maintainers." + }, + { + "file": "packages/server/src/@trpc/server/index.ts", + "line": 16, + "severity": "minor", + "comment": "The new exports inferSubscriptionInput and inferSubscriptionOutput are added to the public API but the PR description and checklist indicate documentation has not been updated. Public API additions should be accompanied by documentation or at minimum a note in CHANGELOG to ensure discoverability." + }, + { + "file": "packages/tests/server/inferenceHelpers.test.ts", + "line": 55, + "severity": "minor", + "comment": "The subscriptionWithObservable test procedure uses observable<{ roomId: string }> but the input type includes only roomId from roomProcedure. The subscriptionWithIterable yields { roomId, text } -- a different shape. This asymmetry is intentional for testing but a comment clarifying that different output shapes are deliberate would help readers understand the test design." + }, + { + "file": "packages/tests/server/inferenceHelpers.test.ts", + "line": 164, + "severity": "major", + "comment": "The diff is truncated and the inferSubscriptionOutput test for 'iterable subscription' appears to be cut off. If the test does not assert the correct yield type ({ roomId: string; text: string }), the key differentiator between observable and async-iterable output inference is untested. Verify the test is complete and asserts the unwrapped yield type." + }, + { + "file": "packages/server/src/unstable-core-do-not-import/procedure.ts", + "line": 83, + "severity": "nit", + "comment": "The import of inferAsyncIterableYield is added from './types'. This import is only used by inferSubscriptionOutput. If the types file is large, consider whether a more targeted import path exists or adding a comment noting the dependency." + } + ], + "summary": "The PR adds two new public type helpers for inferring subscription input and output types, with separate handling for observable-based and async-iterable-based subscriptions. The implementation is straightforward but the output inference has an implicit assumption that non-observable subscriptions are always async iterables, and the test coverage appears truncated in the diff." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/server/src/unstable-core-do-not-import/procedure.ts", + "line": 97, + "severity": "major", + "comment": "CORE LOGIC: inferSubscriptionOutput uses a conditional type that branches on LegacyObservableSubscriptionProcedure. For the observable path, it returns inferProcedureOutput directly (the Observable's value type). For the async-iterable path, it applies inferAsyncIterableYield to unwrap the yield type. This dual-path approach is the heart of the feature. However, if inferAsyncIterableYield receives a type that is not actually an AsyncIterable (e.g., if a user defines a subscription that returns a plain Promise), it will likely resolve to `never` silently. A compile-time error or a more descriptive conditional type would be safer for the public API." + }, + { + "file": "packages/server/src/unstable-core-do-not-import/procedure.ts", + "line": 93, + "severity": "minor", + "comment": "inferSubscriptionInput delegates entirely to inferProcedureInput but constrains to AnySubscriptionProcedure. This is a type-narrowing convenience wrapper. The value is in discoverability and preventing users from accidentally passing a query/mutation procedure. Add JSDoc with @example showing usage with a router type to match the convention of existing helpers like inferProcedureInput." + }, + { + "file": "packages/server/src/@trpc/server/index.ts", + "line": 16, + "severity": "minor", + "comment": "PUBLIC API SURFACE: These new exports become part of the stable @trpc/server public API. They are exported alongside inferProcedureInput/inferProcedureOutput, establishing a pattern. However, inferRouterInputs/inferRouterOutputs already exist for router-level inference and do not have subscription-specific counterparts (inferRouterSubscriptionOutputs). This creates an inconsistency in the API surface -- users may expect router-level subscription helpers too. Consider documenting the intended usage pattern." + }, + { + "file": "packages/server/src/unstable-core-do-not-import/clientish/inference.ts", + "line": 18, + "severity": "nit", + "comment": "The whitespace cleanup (removing blank line before inferTransformedProcedureOutput and adjusting the JSDoc placement for inferTransformedSubscriptionOutput) is a formatting-only change. This is fine but note that inferTransformedSubscriptionOutput is not used by the new public helpers -- the new inferSubscriptionOutput works at the procedure level, not the transformed/serialized level. If subscription outputs need serialization awareness in the future, a separate type may be needed." + }, + { + "file": "packages/tests/server/inferenceHelpers.test.ts", + "line": 55, + "severity": "minor", + "comment": "TEST DESIGN: Two subscription procedures are added with different output shapes -- observable returns { roomId: string } while async-iterable yields { roomId: string; text: string }. This is good for differentiating the two code paths in inferSubscriptionOutput. However, neither test procedure includes input validation beyond roomProcedure's roomId schema. Adding a subscription with additional .input() chaining would test that inferSubscriptionInput correctly merges inputs through the procedure builder chain." + }, + { + "file": "packages/tests/server/inferenceHelpers.test.ts", + "line": 164, + "severity": "major", + "comment": "TRUNCATED TEST: The inferSubscriptionOutput test for 'iterable subscription' is cut off in the diff. This is the most critical test -- it must verify that inferSubscriptionOutput unwraps the AsyncGenerator yield type to { roomId: string; text: string } rather than returning the raw AsyncGenerator type. If this assertion is missing or incorrect, the key behavioral difference between observable and iterable subscriptions is unverified." + }, + { + "file": "packages/tests/server/inferenceHelpers.test.ts", + "line": 156, + "severity": "positive", + "comment": "The addition of inferProcedureInput and inferProcedureOutput test blocks alongside the new subscription tests is a good improvement -- it retroactively adds type-level test coverage for existing helpers that previously only had router-level inference tests." + } + ], + "summary": "The PR adds subscription-specific type inference helpers that correctly branch between observable and async-iterable subscription patterns. The flow analysis reveals the critical path is inferSubscriptionOutput's conditional type branching, which works for the two known subscription patterns but may silently produce `never` for edge cases. The test file is truncated but the observable subscription output test and the retroactive coverage of existing helpers are solid additions." + }, + "judgment": { + "criteria": { + "completeness": { + "baseline": 6, + "flow_guided": 7, + "rationale": "Both reviews identify the core concerns -- the conditional type branching, truncated test, and missing docs. Flow-guided adds the observation about API surface inconsistency (no router-level subscription helpers) and the lack of input-merging test coverage." + }, + "flow_awareness": { + "baseline": 4, + "flow_guided": 6, + "rationale": "Baseline treats each file independently. Flow-guided connects the public export to the procedure-level type, notes the relationship between inferTransformedSubscriptionOutput and the new inferSubscriptionOutput, and traces the test design back to the two code paths in the conditional type. However, the empty review plan limits how much flow analysis is possible." + }, + "risk_identification": { + "baseline": 6, + "flow_guided": 7, + "rationale": "Both flag the risk of non-AsyncIterable types hitting the fallback branch. Flow-guided additionally identifies the API surface inconsistency risk and the missing input-merging test scenario." + }, + "actionability": { + "baseline": 5, + "flow_guided": 6, + "rationale": "Baseline suggestions are generic (add comments, update docs). Flow-guided provides more specific suggestions: add @example JSDoc, test input merging through procedure builder chain, document the relationship to router-level helpers." + }, + "efficiency": { + "baseline": 6, + "flow_guided": 6, + "rationale": "Both reviews stay focused on the 5-file change. The review plan was empty so neither approach had significant structural advantage. Both reviews appropriately flag the truncated diff as a concern." + } + }, + "overall": { + "baseline": 5.4, + "flow_guided": 6.4, + "winner": "flow_guided", + "rationale": "The flow-guided review provides a moderately stronger analysis by connecting the public API exports to the internal conditional type logic and identifying the API surface inconsistency with existing router-level helpers. However, the empty review plan significantly limits the flow-guided approach's advantage -- with no steps, clusters, or dependencies to follow, both reviews largely operate from the same diff-only perspective. The flow-guided review still edges ahead through better cross-file reasoning about how inferSubscriptionOutput relates to inferTransformedSubscriptionOutput and by identifying more specific testing gaps." + } + } +} diff --git a/evals/types.ts b/evals/types.ts new file mode 100644 index 0000000..296ee78 --- /dev/null +++ b/evals/types.ts @@ -0,0 +1,47 @@ +/** Shape of each evaluation JSON file */ +export interface PReval { + pr: { + url: string; + owner: string; + repo: string; + number: number; + title: string; + files_changed: number; + additions: number; + deletions: number; + language: string; + }; + timestamp: string; + baseline_review: Review; + flow_guided_review: Review; + review_plan: unknown; // raw JSON from /api/agent/review-plan + judge: JudgeResult; +} + +export interface Review { + comments: ReviewComment[]; + summary: string; +} + +export interface ReviewComment { + file: string; + line?: number; + severity: "critical" | "major" | "minor" | "nit" | "positive"; + comment: string; +} + +export interface Scores { + completeness: number; + flow_awareness: number; + risk_identification: number; + actionability: number; + efficiency: number; + overall: number; +} + +export interface JudgeResult { + baseline_scores: Scores; + flow_guided_scores: Scores; + reasoning: string; + winner: "baseline" | "flow_guided" | "tie"; +} diff --git a/evals/vercel__next.js__92012.json b/evals/vercel__next.js__92012.json new file mode 100644 index 0000000..c39a57b --- /dev/null +++ b/evals/vercel__next.js__92012.json @@ -0,0 +1,108 @@ +{ + "pr": "vercel/next.js#92012", + "title": "[experiment] Add useOffline hook to expose offline state to userland", + "baseline_review": { + "comments": [ + { + "file": "packages/next/src/client/components/use-offline.tsx", + "line": 19, + "severity": "high", + "comment": "Module-level mutable variables `setOptimistic` and `setCanonical` are unsafe in concurrent React. If multiple `OfflineProvider` instances are ever rendered (e.g. during concurrent rendering or testing), the last one wins and the previous tree silently loses updates. Consider using a ref or subscription pattern instead of module-level singletons." + }, + { + "file": "packages/next/src/client/components/use-offline.tsx", + "line": 39, + "severity": "medium", + "comment": "Assigning `setOptimistic` and `setCanonical` during render (outside of an effect or callback) is a side effect during the render phase. React may call render multiple times without committing, which means these references could point to stale closures. This should be done in a `useEffect` or `useLayoutEffect` to ensure the component is actually mounted." + }, + { + "file": "packages/next/src/client/components/use-offline.tsx", + "line": 38, + "severity": "medium", + "comment": "The `OfflineProvider` never cleans up the module-level references when it unmounts. If the provider unmounts and `dispatchOfflineChange` is called afterward, it will call stale setState functions, potentially causing React warnings or state updates on unmounted components." + }, + { + "file": "packages/next/src/client/components/app-router.tsx", + "line": 539, + "severity": "medium", + "comment": "Using `require()` inside a render function is unconventional in React and may cause issues with bundlers or SSR. While gated behind `process.env.__NEXT_USE_OFFLINE`, consider using a dynamic import or conditional module at the top of the file to keep the render path clean." + }, + { + "file": "packages/next/offline.js", + "line": 1, + "severity": "low", + "comment": "The public export uses CommonJS `module.exports = require(...)` while the `.d.ts` uses ESM `export { useOffline }`. This is consistent with other Next.js public entry points, but worth confirming that the `OfflineProvider` and `dispatchOfflineChange` are intentionally not exported from the public API, as they are also exported from the module." + }, + { + "file": "packages/next/src/client/components/use-offline.tsx", + "line": 30, + "severity": "low", + "comment": "`startTransition` is imported from React at the module level and used in `dispatchOfflineChange`. Since this function is called from outside the React tree (from the offline module), verify that `startTransition` works correctly when called outside of a React component context." + } + ], + "summary": "The PR introduces a `useOffline()` hook using module-level mutable state to bridge between the non-React offline detection system and the React component tree. The main concern is the use of module-level singleton references (`setOptimistic`, `setCanonical`) assigned during render, which is fragile under concurrent React and lacks cleanup on unmount." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/next/src/client/components/use-offline.tsx", + "line": 19, + "severity": "high", + "comment": "The OfflineProvider (order 3, entry_point, high risk) stores its setState functions in module-level singletons. Since this is called by both `notifyOffline` (order 13) and `notifyOnline` in offline.ts, there is a timing issue: if `dispatchOfflineChange(true)` is called before the provider mounts (e.g. during initial hydration with no connectivity), the update is silently dropped. The initial state `useState(false)` would be incorrect for users who load the page offline." + }, + { + "file": "packages/next/src/client/components/use-offline.tsx", + "line": 39, + "severity": "high", + "comment": "Following the data flow: offline.ts calls `dispatchOfflineChange` -> which calls `setCanonical` and `setOptimistic` -> which were assigned during OfflineProvider render. The render-phase assignment means these references can be stale under concurrent mode. Since the Router component (order 6) wraps content with OfflineProvider, and Router is a complex component with many state transitions, concurrent renders are likely. Use a ref + useEffect to ensure stable references." + }, + { + "file": "packages/next/src/client/components/app-router.tsx", + "line": 539, + "severity": "medium", + "comment": "The Router (order 6) conditionally wraps content with OfflineProvider. Tracing the dependency: OfflineProvider -> useState/useOptimistic -> module-level refs -> offline.ts dispatches. The conditional `require()` inside the render function means the module is loaded lazily. However, offline.ts unconditionally imports `dispatchOfflineChange` at the top level (line 36 of offline.ts). This creates an asymmetry: the dispatch function is always available, but the provider may not be mounted if the experiment flag is off, making the early-return in `dispatchOfflineChange` a critical safety check." + }, + { + "file": "packages/next/src/client/components/offline.ts", + "line": 92, + "severity": "medium", + "comment": "In `notifyOffline` (order 13), `dispatchOfflineChange(true)` is called before `checkConnectivity(offlineState)`. This ordering means the React tree sees 'offline' immediately. But in `notifyOnline` (order 13 caller), the dispatch happens after `offlineState = null` and `resolve()`. The asymmetry is minor but worth documenting: the online notification happens after the promise resolves, so any pending navigation transitions complete before the UI updates to 'online'." + }, + { + "file": "test/e2e/app-dir/use-offline/app/destination/page.tsx", + "line": 10, + "severity": "medium", + "comment": "The DestinationPage (order 2) uses OfflineStatus in a Suspense fallback, which is a key test scenario: verifying useOffline works during blocked transitions. However, the test component (order 10, OfflineStatus) is called from both the layout (order 5) and this fallback. The test should verify that both instances update simultaneously when offline state changes, since they share the same context provider." + }, + { + "file": "test/e2e/app-dir/use-offline/use-offline.test.ts", + "line": 139, + "severity": "low", + "comment": "The test's `beforePageLoad` hook (order 7) is modified but the diff is truncated. Based on the plan, this likely sets up network interception. Ensure the test covers the race condition where the page loads while already offline (initial state correctness), not just transitions from online to offline." + } + ], + "summary": "Following the data flow from offline.ts through dispatchOfflineChange into the OfflineProvider and finally to useOffline consumers, the main risk is the module-level singleton pattern that bridges non-React code to React state. The timing assumptions (provider must be mounted before any offline event) and render-phase side effects create fragility under concurrent React, initial-offline scenarios, and unmount edge cases." + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 6, + "actionability": 7, + "efficiency": 7, + "overall": 6.2 + }, + "flow_guided": { + "completeness": 8, + "flow_awareness": 9, + "risk_identification": 8, + "actionability": 8, + "efficiency": 8, + "overall": 8.2 + } + }, + "winner": "flow_guided", + "reasoning": "The flow-guided review identifies the same core issues (module-level singletons, render-phase side effects) but adds critical insights about data flow timing: the race condition where offline events fire before the provider mounts, the asymmetry between notifyOffline and notifyOnline dispatch ordering, and the import asymmetry between offline.ts (unconditional import) and app-router.tsx (conditional require). It also connects test coverage to the actual data flow, noting that the Suspense fallback test scenario specifically exercises the useOptimistic path. The baseline review catches the surface-level issues but misses these cross-file interaction risks that only emerge from following the execution flow." + } +} diff --git a/evals/vercel__next.js__92014.json b/evals/vercel__next.js__92014.json new file mode 100644 index 0000000..e6f5514 --- /dev/null +++ b/evals/vercel__next.js__92014.json @@ -0,0 +1,108 @@ +{ + "pr": "vercel/next.js#92014", + "title": "Use RcStr directly in napi(object) structs instead of converting to String", + "baseline_review": { + "summary": "This PR replaces String fields with RcStr in several napi(object) structs and removes unnecessary .to_string() / .into_owned() conversions, which is a straightforward optimization to avoid redundant allocations at the NAPI boundary. The changes are mechanical and consistent across all three files, though a few subtle points around trait implementations and the FxHashMap type change deserve attention.", + "comments": [ + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "386-387", + "severity": "medium", + "comment": "Changing NapiDiagnostic.payload from FxHashMap to FxHashMap is the most impactful change in this PR. Unlike simple field type swaps, this changes the map's key and value types, which affects all code that reads from or writes to this map. Confirm that FromNapiValue and ToNapiValue are implemented for FxHashMap and that downstream JS consumers are unaffected." + }, + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "368-369", + "severity": "low", + "comment": "The conversion for NapiSource uses (*source.ident).clone() and (*source.file_path).clone() rather than source.ident.clone() directly. This deref-then-clone pattern suggests source.ident/file_path might be a wrapper type (e.g., ReadRef). This works but is worth a comment clarifying why the deref is needed, since the other conversions in this PR use a simpler .clone()." + }, + { + "file": "crates/next-napi-bindings/src/next_api/endpoint.rs", + "line": "35-36", + "severity": "low", + "comment": "NapiAssetPath derives Default. RcStr implements Default (as empty string), so this is fine, but worth confirming that an empty RcStr default is acceptable for path and content_hash fields in all usage sites." + }, + { + "file": "crates/next-napi-bindings/src/next_api/project.rs", + "line": "789", + "severity": "low", + "comment": "The NapiRoute struct also derives Default. Changing pathname from String to RcStr should be compatible since both default to empty, but verify no code relies on the concrete String type of pathname downstream." + }, + { + "file": "crates/next-napi-bindings/src/next_api/project.rs", + "line": "931", + "severity": "low", + "comment": "Replacing k.to_string() with k.clone() is correct since k is already an RcStr. The clone of an RcStr is a cheap reference count increment rather than a full string allocation, which is the intended optimization." + }, + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "230-236", + "severity": "info", + "comment": "The severity and stage fields in NapiIssue remain as String, which is correct since severity comes from .as_str().to_string() and stage from .to_string() -- these are not RcStr sources. Good that only the appropriate fields were changed." + } + ] + }, + "flow_guided_review": { + "summary": "The PR systematically replaces String with RcStr at the NAPI boundary layer across three files, eliminating redundant heap allocations when data already exists as RcStr internally. The review plan is empty (no steps/clusters), so flow analysis is limited, but the change is self-contained within the napi-bindings crate and the risk profile is low given RcStr already implements the required napi traits.", + "comments": [ + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "386-387", + "severity": "medium", + "comment": "The FxHashMap to FxHashMap change in NapiDiagnostic.payload is the highest-risk change. This affects both serialization (ToNapiValue) and deserialization (FromNapiValue) of a complex nested type. If any downstream consumer pattern-matches on or constructs this map with String keys, it will break. The diff is truncated and does not show the from() implementation for NapiDiagnostic -- verify the conversion logic there is also updated." + }, + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "368-369", + "severity": "medium", + "comment": "The (*source.ident).clone() pattern in NapiSource::from differs from all other conversions in this PR which use .clone() directly. This suggests PlainSource fields may be ReadRef or similar wrapper types. If the inner type changes in the future, this deref-clone could silently produce a different type. Consider using Into or a more explicit conversion to make the intent clearer." + }, + { + "file": "crates/next-napi-bindings/src/next_api/endpoint.rs", + "line": "38-39", + "severity": "low", + "comment": "The AssetPath to NapiAssetPath conversion now simply moves the RcStr fields without any conversion. This is the cleanest pattern in the PR and confirms that AssetPath.path and AssetPath.content_hash are already RcStr, making the old .into_owned() calls pure waste." + }, + { + "file": "crates/next-napi-bindings/src/next_api/project.rs", + "line": "801-802", + "severity": "low", + "comment": "The from_route method signature change from String to RcStr for pathname is consistent with the struct field change. Since the only caller (line 931) already has an RcStr (the map key), this eliminates an allocation at the call site by replacing .to_string() with .clone()." + }, + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "258-262", + "severity": "low", + "comment": "The file_path and documentation_link conversions in NapiIssue::from change from .to_string() to .clone(). Since PlainIssue presumably stores these as RcStr, this avoids allocating new Strings. Straightforward and correct." + }, + { + "file": "crates/next-napi-bindings/src/next_api/utils.rs", + "line": "271", + "severity": "low", + "comment": "The description field in the additional_sources mapping also changes from .to_string() to .clone(). This is consistent with the NapiAdditionalIssueSource struct change and follows the same safe pattern as the other conversions." + } + ] + }, + "judgment": { + "scores": { + "baseline": { + "completeness": 7, + "flow_awareness": 4, + "risk_identification": 7, + "actionability": 7, + "efficiency": 8, + "overall": 6.6 + }, + "flow_guided": { + "completeness": 7, + "flow_awareness": 5, + "risk_identification": 8, + "actionability": 7, + "efficiency": 7, + "overall": 6.8 + } + }, + "winner": "flow_guided", + "reasoning": "Both reviews identify the same key risks -- the FxHashMap type change and the deref-clone pattern in NapiSource. The flow-guided review edges ahead slightly by noting that the diff is truncated and the NapiDiagnostic::from() implementation is not visible, which is an important risk observation. However, the empty review plan (no steps, no clusters, no dependencies) severely limits the flow-guided approach's advantage. With a proper plan, the flow-guided review could have traced data flow from PlainIssue/PlainSource through the napi boundary to JS consumers, identifying whether any JS code depends on these being plain strings. The baseline review is slightly more efficient since it doesn't attempt flow analysis that the empty plan cannot support. Overall the difference is marginal given the mechanical nature of this PR." + } +} diff --git a/evals/vercel__next.js__92029.json b/evals/vercel__next.js__92029.json new file mode 100644 index 0000000..8766b1a --- /dev/null +++ b/evals/vercel__next.js__92029.json @@ -0,0 +1,135 @@ +{ + "pr": { + "url": "https://github.com/vercel/next.js/pull/92029", + "owner": "vercel", + "repo": "next.js", + "number": 92029, + "title": "[turbopack] Don't use turborepo to build the docker image", + "files_changed": 6, + "additions": 332, + "deletions": 40, + "language": "javascript" + }, + "timestamp": "2026-03-30T18:30:00Z", + "baseline_review": { + "comments": [ + { + "file": "scripts/docker-image-cache.js", + "line": 32, + "severity": "minor", + "comment": "CACHE_INPUTS includes docker-image-cache.js itself. Any formatting-only change to this script (whitespace, comments) would invalidate the Docker image cache and trigger a full rebuild. Consider excluding the script itself or using only the Dockerfile and toolchain inputs." + }, + { + "file": "scripts/docker-image-cache.js", + "line": 39, + "severity": "minor", + "comment": "computeCacheKey reads files synchronously and will throw an unhandled exception if any CACHE_INPUTS file is missing. A try/catch with a descriptive error message would improve debugging in CI." + }, + { + "file": "scripts/docker-image-cache.js", + "line": 42, + "severity": "nit", + "comment": "The hash prefix 'docker-image-v3' is a magic string. A brief comment explaining why v3 (presumably to distinguish from prior cache generations) would help future maintainers." + }, + { + "file": ".github/workflows/build_and_deploy.yml", + "line": 252, + "severity": "major", + "comment": "The step now runs `node scripts/docker-image-cache.js` directly without any environment variables for turbo cache credentials (TURBO_TOKEN, TURBO_TEAM). The script presumably needs these to call the turbo cache API. Verify that these are inherited from the workflow environment or the script will silently fail cache lookups." + }, + { + "file": "scripts/docker-image-cache.js", + "line": 1, + "severity": "minor", + "comment": "The @ts-check pragma was removed from the script header. This disables TypeScript checking in editors, reducing type safety for a script that now has more complex logic (streaming, zstd compression)." + }, + { + "file": "packages/next-swc/turbo.jsonc", + "line": 123, + "severity": "positive", + "comment": "Clean removal of the build-docker-image task definition. The inputs/outputs config is no longer needed since caching is handled directly by the script." + }, + { + "file": "scripts/docker-image-cache.js", + "line": 76, + "severity": "minor", + "comment": "pipeToShell spawns a child process but the error handling for the spawned process is not visible in the diff. If the zstd decompression or docker load fails mid-stream, ensure the script exits with a non-zero code so CI catches the failure." + } + ], + "summary": "The PR simplifies the Docker image build pipeline by bypassing turborepo and calling the turbo cache API directly, which is a sound approach given the reported cache corruption. Main concerns are around error handling in the new streaming pipeline and ensuring cache credentials are properly available in CI." + }, + "flow_guided_review": { + "comments": [ + { + "file": "scripts/docker-image-cache.js", + "line": 95, + "severity": "major", + "comment": "The main flow (entry point, high risk per plan) orchestrates cache check, download/decompress, build, compress/upload. This is a complex pipeline replacing turborepo orchestration. The lack of a --load flag means the script always runs the full flow. If the image is already loaded in Docker (e.g., from a previous step or retry), there is no early exit check visible in the diff -- the old imageExists() function was removed." + }, + { + "file": "scripts/turbo-cache.mjs", + "line": 68, + "severity": "major", + "comment": "getStream (high risk entry point per plan) returns a readable stream from the turbo cache API. This is called by docker-image-cache.js to pipe directly into zstd | docker load. If the HTTP response is not 200 (e.g., 404 for cache miss, or network error), the caller must handle this gracefully. The plan shows no calledBy for this function, suggesting it is an exported API -- verify all consumers handle non-200 responses." + }, + { + "file": "scripts/turbo-cache.mjs", + "line": 123, + "severity": "minor", + "comment": "healthCheck (high risk entry point per plan) exercises exists/put/get in sequence. This is good for CI validation but the 43 lines of test logic in a production module suggest it could be extracted to a separate test script to keep turbo-cache.mjs focused." + }, + { + "file": "scripts/docker-native-build.js", + "line": 89, + "severity": "minor", + "comment": "ensureDockerImage was simplified (high risk entry point, -18 lines). The function previously had fallback logic. With the simplification, verify that the docker-image-cache.js script is always called before this function in the CI pipeline, since ensureDockerImage no longer has its own cache restoration path." + }, + { + "file": "scripts/docker-image-cache.js", + "line": 32, + "severity": "minor", + "comment": "CACHE_INPUTS includes docker-image-cache.js and docker-native-build.js alongside the Dockerfile and rust-toolchain.toml. The original turbo.jsonc task only tracked the Dockerfile, build script, and toolchain. Adding the cache script itself as an input means any change to the caching logic forces a full Docker rebuild (~2.8GB upload). This is overly conservative." + }, + { + "file": "scripts/turbo-cache.mjs", + "line": 85, + "severity": "minor", + "comment": "put function (medium risk per plan, 34 lines added) uploads artifacts to turbo cache. For a ~500MB compressed Docker image, verify that the turbo cache API supports streaming uploads of this size without timeout. The workflow previously had --remote-cache-timeout 90 which is now gone." + }, + { + "file": ".github/workflows/build_and_deploy.yml", + "line": 252, + "severity": "major", + "comment": "The workflow previously used `pnpm dlx turbo@${TURBO_VERSION}` which handled turbo authentication via environment. The new direct node invocation needs TURBO_TOKEN and TURBO_TEAM (or equivalent) available. The turbo-cache.mjs module must read these from the environment. If missing, the script should fail fast with a clear error rather than silently skipping cache." + }, + { + "file": "scripts/docker-image-cache.js", + "line": 76, + "severity": "minor", + "comment": "pipeToShell (leaf node per plan) spawns shell processes for zstd decompression. The ~2.8GB uncompressed image flowing through pipes means any interruption could leave partial state. Ensure cleanup of temp files on process exit (SIGTERM handler or try/finally in main)." + } + ], + "summary": "The PR replaces turborepo-mediated Docker image caching with a direct turbo cache API client, motivated by cache corruption issues. The flow analysis reveals three independent entry points in turbo-cache.mjs that form a new caching abstraction layer. Key risks are around error handling in the streaming pipeline (getStream -> zstd -> docker load), ensuring cache credentials are available without turborepo's env handling, and the removal of the imageExists() early-exit check that previously prevented redundant work." + }, + "review_plan": "see prompt file plan section", + "judge": { + "baseline_scores": { + "completeness": 5, + "flow_awareness": 4, + "risk_identification": 5, + "actionability": 6, + "efficiency": 6, + "overall": 5.2 + }, + "flow_guided_scores": { + "completeness": 7, + "flow_awareness": 8, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.2 + }, + "reasoning": "The flow-guided review outperforms in several dimensions: (1) It identifies the removal of imageExists() as a missing early-exit, which the baseline misses entirely -- this is a real regression risk if the script is called twice. (2) It traces the getStream -> pipe -> docker load pipeline across two files (turbo-cache.mjs and docker-image-cache.js), identifying the cross-file error handling gap. (3) It correctly flags the upload timeout concern by connecting the removed --remote-cache-timeout 90 flag to the new put() function. (4) It identifies the ensureDockerImage simplification as a dependency ordering concern in CI. The baseline review catches the @ts-check removal and the self-referential cache input issue, which are valid but lower-impact observations. Both reviews identify the credential/environment variable concern as major, which is the most actionable finding. The baseline lacks awareness of how the files interact as a system.", + "winner": "flow_guided" + } +} diff --git a/evals/vercel__next.js__92080.json b/evals/vercel__next.js__92080.json new file mode 100644 index 0000000..46ff1d2 --- /dev/null +++ b/evals/vercel__next.js__92080.json @@ -0,0 +1,189 @@ +{ + "pr": { + "url": "https://github.com/vercel/next.js/pull/92080", + "owner": "vercel", + "repo": "next.js", + "number": 92080, + "title": "Enable prefetching in dev mode with background limit of 1", + "files_changed": 14, + "additions": 200, + "deletions": 8, + "language": "typescript" + }, + "timestamp": "2026-03-30T09:15:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/next/src/build/define-env.ts", + "line": 248, + "severity": "major", + "comment": "The expression `!dev || (isTurbopack && Boolean(config.experimental.turbopackPrefetchInDev))` evaluates to boolean, but process.env values are conventionally strings. Need to verify all bundler paths correctly inline this as a boolean literal. If any path serializes it as a string, the negation check still works, but it would be more robust to use Boolean(...) wrapping." + }, + { + "file": "packages/next/src/build/define-env.ts", + "line": 248, + "severity": "minor", + "comment": "Flag is named turbopackPrefetchInDev but env var is generic __NEXT_PREFETCH. Creates subtle coupling — in production __NEXT_PREFETCH is always true. If another prefetch feature keys off the same var, it could conflict." + }, + { + "file": "packages/next/src/client/app-dir/link.tsx", + "line": 695, + "severity": "minor", + "comment": "Splitting the original single if into two separate returns is clean, but need to confirm __NEXT_PREFETCH is treated as compile-time constant for tree-shaking by both Webpack and Turbopack." + }, + { + "file": "packages/next/src/client/components/segment-cache/scheduler.ts", + "line": 459, + "severity": "major", + "comment": "Background concurrency limit of 1 is via runtime NODE_ENV check while rest of PR uses compile-time __NEXT_PREFETCH. Inconsistency noted. Also no way for a developer to configure a higher limit." + }, + { + "file": "packages/next/src/client/components/segment-cache/scheduler.ts", + "line": 459, + "severity": "nit", + "comment": "The ternary would benefit from named constants (DEV_PREFETCH_CONCURRENCY=1, PROD_PREFETCH_CONCURRENCY=4) for clarity and tuning." + }, + { + "file": "packages/next/src/client/components/links.ts", + "line": 254, + "severity": "positive", + "comment": "Good that the TODO comment ('Investigate re-enabling this') was removed along with the code change that actually re-enables it." + }, + { + "file": "packages/next/src/server/config-shared.ts", + "line": 418, + "severity": "minor", + "comment": "turbopackPrefetchInDev lacks JSDoc comment unlike neighboring experimental flags." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/segment-cache-prefetch-dev.test.ts", + "line": 241, + "severity": "minor", + "comment": "Empty test body for webpack skip is misleading — produces a falsely passing test. Consider using it.skip or describe.skip." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/segment-cache-prefetch-dev.test.ts", + "line": 256, + "severity": "minor", + "comment": "The act! non-null assertion is fragile — if beforePageLoad were never called, it would throw a confusing error. A guard assertion would be safer." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/segment-cache-prefetch-dev.test.ts", + "line": 277, + "severity": "positive", + "comment": "HMR staleness test is well-designed with try/finally for file restoration and retry with 30s timeout for Turbopack recompilation latency." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/next.config.js", + "line": 5, + "severity": "minor", + "comment": "No negative test case verifying that prefetching remains disabled when the flag is off. This would strengthen confidence in the gating." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/app/layout.tsx", + "line": 9, + "severity": "nit", + "comment": "Root layout wrapping html inside Suspense is unusual — worth a brief comment explaining why it's needed for this test." + } + ], + "summary": "Well-scoped feature flag with appropriate guardrails. Main concerns: verifying __NEXT_PREFETCH is correctly inlined across all bundler paths for dead-code elimination, and absence of negative test case confirming prefetching remains disabled when flag is not set." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/next/src/build/define-env.ts", + "line": 248, + "severity": "positive", + "comment": "The boolean expression correctly handles all four cases: production (always true), dev+turbopack+flag (true), dev+turbopack+no-flag (false), dev+webpack (false regardless of flag)." + }, + { + "file": "packages/next/src/build/define-env.ts", + "line": 248, + "severity": "minor", + "comment": "Other defines use Boolean() wrapping for consistency (e.g., __NEXT_DYNAMIC_ON_HOVER, __NEXT_USE_OFFLINE). Suggests matching the pattern." + }, + { + "file": "packages/next/src/client/app-dir/link.tsx", + "line": 695, + "severity": "positive", + "comment": "Good refactoring: splitting the combined condition into two separate guards makes the logic clearer and the __NEXT_PREFETCH check independent of prefetch-enabled prop." + }, + { + "file": "packages/next/src/client/components/links.ts", + "line": 254, + "severity": "minor", + "comment": "Comment says 'disabled in dev unless turbopackPrefetchInDev' but the guard !process.env.__NEXT_PREFETCH isn't dev-specific — it also controls production. Suggests more general wording like 'Prefetching is not enabled for this build configuration.'" + }, + { + "file": "packages/next/src/client/components/segment-cache/scheduler.ts", + "line": 459, + "severity": "major", + "comment": "Using NODE_ENV here is inconsistent with rest of PR migrating to __NEXT_PREFETCH. Maintenance risk if __NEXT_PREFETCH derivation changes. Suggests dedicated define (e.g., __NEXT_PREFETCH_BG_LIMIT) or cross-reference comment." + }, + { + "file": "packages/next/src/client/components/segment-cache/scheduler.ts", + "line": 459, + "severity": "minor", + "comment": "Magic numbers 1 and 4 should be named constants (DEV_BG_PREFETCH_LIMIT, PROD_BG_PREFETCH_LIMIT)." + }, + { + "file": "packages/next/src/client/components/app-router-utils.ts", + "line": 32, + "severity": "positive", + "comment": "Clean replacement of process.env.NODE_ENV === 'development' with !process.env.__NEXT_PREFETCH. Updated comment accurately describes new semantics." + }, + { + "file": "packages/next/src/server/config-shared.ts", + "line": 418, + "severity": "nit", + "comment": "turbopackPrefetchInDev lacks JSDoc unlike sibling properties like partialFallbacks." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/segment-cache-prefetch-dev.test.ts", + "line": 241, + "severity": "minor", + "comment": "Bare it() with empty body should be it.skip so test runner reports it as skipped rather than falsely passing." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/segment-cache-prefetch-dev.test.ts", + "line": 256, + "severity": "nit", + "comment": "act! non-null assertion could use guard expect(act).toBeDefined() before first use for clearer failure messages." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/segment-cache-prefetch-dev.test.ts", + "line": 277, + "severity": "positive", + "comment": "Staleness test is well-designed — editing files on disk, navigating, verifying fresh content. try/finally to restore files is good practice." + }, + { + "file": "test/development/app-dir/segment-cache-prefetch-dev/next.config.js", + "line": 5, + "severity": "nit", + "comment": "cacheComponents is at top level while turbopackPrefetchInDev is inside experimental — worth verifying this is the intended config shape." + } + ], + "summary": "Well-structured PR that cleanly introduces experimental flag gated behind turbopackPrefetchInDev, consistently replaces hard-coded NODE_ENV checks with build-time __NEXT_PREFETCH define. Main concern is inconsistency in scheduler.ts where NODE_ENV is still used directly for concurrency limit. Solid test coverage including staleness test. Low-risk, safe to ship after addressing consistency point." + }, + "review_plan": "see /api/agent/review-plan?owner=vercel&repo=next.js&pull=92080&verbosity=standard", + "judge": { + "baseline_scores": { + "completeness": 7, + "flow_awareness": 7, + "risk_identification": 6, + "actionability": 6, + "efficiency": 5, + "overall": 6.2 + }, + "flow_guided_scores": { + "completeness": 8, + "flow_awareness": 8, + "risk_identification": 7, + "actionability": 7, + "efficiency": 7, + "overall": 7.4 + }, + "reasoning": "Both reviews identify the core scheduler.ts NODE_ENV inconsistency and share several observations (missing JSDoc, magic numbers, empty test body, act! assertion). Key differences: (1) Review 2 covers app-router-utils.ts which Review 1 misses entirely. (2) Review 1's [major] on boolean-vs-string is a false positive at that severity — define-env uses compile-time replacement, not runtime env vars. (3) Review 2 correctly identifies that the links.ts comment is misleading because the guard isn't dev-specific, showing deeper semantic understanding. (4) Review 2 correctly verifies the four boolean expression cases (prod, dev+turbo+flag, dev+turbo+noflag, dev+webpack), demonstrating it traced the logic through the define system. (5) Review 1 has more noise with lower-signal comments (Suspense wrapping, naming concerns).", + "winner": "flow_guided" + } +} diff --git a/evals/vuejs__core__14628.json b/evals/vuejs__core__14628.json new file mode 100644 index 0000000..8c9d2d1 --- /dev/null +++ b/evals/vuejs__core__14628.json @@ -0,0 +1,115 @@ +{ + "pr": { + "url": "https://github.com/vuejs/core/pull/14628", + "owner": "vuejs", + "repo": "core", + "number": 14628, + "title": "fix(transition-group): handle v-if dynamic slots", + "files_changed": 3, + "additions": 114, + "deletions": 16, + "language": "typescript" + }, + "timestamp": "2026-03-30T18:00:00Z", + "baseline_review": { + "comments": [ + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 148, + "severity": "major", + "comment": "The new guard condition `isMounted && tag === currentTag && slot === currentSlot` relies on reference equality for `slots.default`. If the compiler generates a new function reference for the slot on every render (e.g., when the parent re-renders without the slot itself changing), this check will never bail out, causing unnecessary teardown and re-creation of the transition group content on every render cycle. Confirm that the Vapor compiler guarantees stable slot function references when the slot template has not changed." + }, + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 150, + "severity": "minor", + "comment": "The `container` element is now created unconditionally on every `renderEffect` execution (when `tag` is truthy), even when the early return on line 149 would have bailed out. Previously, `createElement` was called inside `frag.update()`, so it only ran when the fragment actually updated. This means a new DOM element is allocated and immediately discarded on every effect run that passes the guard. Move the `createElement` call after the guard check or inside the `frag.update` callback to avoid the wasted allocation." + }, + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 165, + "severity": "minor", + "comment": "The second argument to `frag.update()` (the key) has been removed. Previously, `tag ?? null` was passed to prevent `undefined` from falling back to the render function as the key. If `DynamicFragment.update` uses the key for reconciliation or caching, removing it may cause the fragment to always treat updates as new content rather than patching existing content. Verify that this removal is intentional and that `DynamicFragment` handles the missing key correctly." + }, + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 155, + "severity": "nit", + "comment": "The `slot` variable captures `slots.default` at the top of the effect, but `slot` could be `undefined` when no default slot is provided. The `(slot && slot()) || []` pattern handles this, but a more explicit guard like `slot ? slot() : []` would be slightly clearer about the intent." + }, + { + "file": "packages-private/vapor-e2e-test/__tests__/transition-group.spec.ts", + "line": 267, + "severity": "minor", + "comment": "The e2e test only covers the v-if toggling from false to true and adding items after. It does not test toggling v-if back to false (removing the slot content) or rapid toggling (true -> false -> true) which could reveal issues with transition leave hooks or stale `currentSlot` references." + } + ], + "summary": "The fix correctly identifies the problem -- TransitionGroup was not tracking slot identity changes, so v-if toggling on a dynamic slot was invisible to the render effect. However, the refactored code creates a new container element on every effect run regardless of the early-return guard, and the removal of the key argument to `frag.update()` needs verification to ensure it does not break fragment reconciliation." + }, + "flow_guided_review": { + "comments": [ + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 148, + "severity": "major", + "comment": "Step 1 (setup, entry point, high risk): The core fix adds `slot === currentSlot` to the guard condition inside the `renderEffect`. Since `slots.default` is a reactive property on the slots proxy, reading it inside `renderEffect` establishes a dependency -- when v-if toggles the slot, the slot reference changes, the effect re-runs, and the guard no longer bails out. This is the correct reactive mechanism. However, the plan identifies `setup` as high-risk because it orchestrates all transition lifecycle. If the Vapor compiler ever wraps slot functions in per-render closures (breaking reference stability), this guard would never skip, causing the entire transition group to be torn down and rebuilt on every parent render." + }, + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 150, + "severity": "major", + "comment": "Step 1 continued (setup calls applyGroupTransitionHooks): The `container = createElement(tag)` is now hoisted outside `frag.update()`, meaning a fresh DOM element is created on every effect execution -- including runs where only tag or slot changed but also runs that pass the guard and proceed to update. When the tag has not changed but the slot has, this creates a new container unnecessarily; the old container with its existing children is discarded. This could cause a visible flash as children are re-inserted into a fresh element, disrupting in-progress FLIP animations that `applyGroupTransitionHooks` and `recordPosition`/`applyTranslation` rely on. The container should be reused when only the slot changes." + }, + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 165, + "severity": "major", + "comment": "Step 1 (setup, dependency on DynamicFragment): The previous code passed `tag ?? null` as a key to `frag.update()`. Looking at the call chain, `DynamicFragment.update` uses this key to decide whether to teardown the old block and create a new one vs. patch in place. By removing the key, every call to `frag.update()` may now be treated as a key change (if the default becomes `undefined`), forcing full teardown/recreation even when only slot content changed. This contradicts the goal of smooth transitions -- existing children would lose their transition state. Verify how `DynamicFragment` interprets a missing second argument." + }, + { + "file": "packages/runtime-vapor/src/components/TransitionGroup.ts", + "line": 138, + "severity": "minor", + "comment": "The new `BlockFn` import type is used for `currentSlot`. This correctly types the variable as `BlockFn | undefined` which matches the shape of `slots.default`. Good addition for type safety." + }, + { + "file": "packages-private/vapor-e2e-test/transition-group/cases/vapor-transition-group/dynamic-slot-with-v-if.vue", + "line": 9, + "severity": "minor", + "comment": "The test component uses `