diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/README.md b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/README.md new file mode 100644 index 0000000..703e37e --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/README.md @@ -0,0 +1,49 @@ +# Repeated-investigation benchmark: post-fix PASS result + +Date: 2026-08-14 +Scenario: GraphKeeper Memory Bench v0.1, Test A — Repeated Investigation +Result: **PASS — grounded memory reused with bounded current-state verification.** + +## Result + +GraphKeeper was consulted first. The session retrieved the directly relevant active +claims and inspected their provenance and cited evidence. It reused that evidence for +the historical reasoning and limited fresh verification to current files, working-tree +state, and current opacity references. + +The session did not rerun `git log`, `git blame`, migration-history reconstruction, +historical diffs, broad repository searches, or unnecessary tests. Current state did +not contradict the grounded claims, so investigation stopped. + +The conclusion remained the same: the homepage opacity poll remains in the test but +represents legacy behavior. + +## Benchmark history + +1. **FAIL** — the pre-fix session retrieved GraphKeeper memory but repeated the + original source and historical investigation. +2. **fix** — retrieval-first guidance required grounded claims to become the starting + point with minimum freshness verification. +3. **FAIL** — retrieval and provenance inspection improved, but the rerun still used + broad searches, Git history, blame, migration diffs, and unnecessary validation. +4. **fix** — the guidance added a concrete current-state boundary and stopping rule. +5. **PASS** — this run reused grounded historical evidence, checked only relevant + current state, found no contradiction, and stopped. + +This records the required progression: **FAIL → fix → FAIL → fix → PASS**. + +## Separate observation + +The `graphkeeper` CLI was not directly available on `PATH`. The agent therefore +located and read the repository-local graph records manually. Retrieval still +succeeded. This is a non-benchmark operational observation, not a benchmark failure. + +## Contents + +- `prompts.md`: the exact repeated-investigation prompt. +- `graphkeeper-session.md`: relevant rerun transcript/output, actions, comparison, and + concise PASS rationale. +- `graphkeeper/`: snapshots of the entity, claims, producing run, and evidence used. + +No GraphKeeper implementation, application code, data model, or earlier benchmark +record is modified by this result. diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper-session.md b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper-session.md new file mode 100644 index 0000000..8a15bc7 --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper-session.md @@ -0,0 +1,79 @@ +# GraphKeeper post-fix session + +## Relevant session transcript/output + +The benchmark rerun reported: + +> GraphKeeper was consulted first. +> +> It retrieved the directly relevant grounded claim and provenance. +> +> The agent limited verification to current-state freshness checks. +> +> It checked current files / working-tree state / current opacity references. +> +> It did not repeat the previous historical investigation with `git log`, `git blame`, +> migration-history reconstruction, historical diffs, or unnecessary tests. +> +> It reused the existing GraphKeeper evidence for the historical reasoning. + +The resulting conclusion was: + +> The homepage opacity poll remains in the test but represents legacy behavior. + +Final benchmark result: + +> **PASS — grounded memory reused with bounded current-state verification.** + +## Commands and actions + +1. Attempted to invoke the `graphkeeper` CLI first; it was not directly available on + `PATH`. +2. Located the repository-local GraphKeeper records manually. +3. Inspected the active claims, their source kinds, producing run, source references, + and captured evidence. +4. Reused the existing evidence for the historical scroll-reveal reasoning. +5. Limited freshness checks to the relevant current files, working-tree state, and + current opacity references. +6. Found no current contradiction and stopped investigating. + +The session did **not** run `git log`, `git blame`, historical diffs, migration-history +reconstruction, broad repository searches, or unnecessary tests. + +The missing CLI-on-`PATH` condition is recorded as a separate operational observation. +It did not prevent grounded retrieval and is not counted as a benchmark failure. + +## GraphKeeper claims and evidence used + +The session used the active entity `homepage_project_card_opacity_check`, produced by +`run_2026-08-14-opacity-a1`, including: + +- `claim_b8d4e2a6`: the current test polls computed opacity above `0.99`, grounded at + `evidence/homepage-project-opacity-2026-08-14.txt#L8-L11`. +- `claim_c19e5f73`: the historical card used an opacity scroll reveal, grounded at + `evidence/homepage-project-opacity-2026-08-14.txt#L23-L26`. +- `claim_d2f6a804`: the current card is a plain article without opacity or viewport + animation props, grounded at + `evidence/homepage-project-opacity-2026-08-14.txt#L12-L21`. +- `claim_e5b7c391`: the opacity poll appears redundant because the current card has no + opacity animation; this is explicitly an inference based on the grounded current and + historical claims. + +Exact snapshots are preserved under `graphkeeper/`. + +## Comparison with earlier failures + +The pre-fix FAIL retrieved the same relevant memory but repeated broad source search, +Git-history reconstruction, migration diffs, and validation. The first post-fix rerun +also retrieved memory and announced a minimal check, but still repeated broad searches, +multiple history queries, migration diff inspection, blame, and historical tracing. + +This post-fix run differs factually: it reused the stored historical evidence, checked +only changeable current state, found no contradiction, and stopped. That bounded +behavior is why the result is PASS. + +## PASS rationale + +Retrieval happened first, provenance was inspected, historical evidence was reused, +freshness verification stayed bounded to current state, prohibited reinvestigation did +not occur, and the correct conclusion was reached. Therefore the benchmark passes. diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/claims.json b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/claims.json new file mode 100644 index 0000000..3d0b8ab --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/claims.json @@ -0,0 +1,79 @@ +[ + { + "id": "claim_4a7c2e91", + "subject": "homepage_project_card_opacity_check", + "predicate": "scrolls_main_cards_into_view", + "object": "the homepage test scrolls each main project card into view before checking it", + "confidence": 1, + "source": { + "kind": "tool_output", + "command": "[DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ'); $n=0; Get-Content tests\\homepage.spec.ts | ForEach-Object { $n++; if($n -ge 69 -and $n -le 76){ '{0}: {1}' -f $n,$_ } }; $n=0; Get-Content src\\components\\sections\\projects\\ProjectCard.tsx | ForEach-Object { $n++; if($n -ge 10 -and $n -le 18){ '{0}: {1}' -f $n,$_ } }; git show b839ff9:components/projects/ProjectCard.tsx | Select-String -Pattern 'initial=|whileInView=|viewport=|transition=' | ForEach-Object { $_.Line }", + "exit_code": 0, + "ref": "evidence/homepage-project-opacity-2026-08-14.txt#L4-L7", + "captured": "2026-08-14T11:02:20Z" + }, + "produced_by": "run_2026-08-14-opacity-a1", + "created": "2026-08-14T11:02:20Z" + }, + { + "id": "claim_b8d4e2a6", + "subject": "homepage_project_card_opacity_check", + "predicate": "polls_computed_opacity", + "object": "the homepage test polls each card's computed opacity until it exceeds 0.99", + "confidence": 1, + "source": { + "kind": "tool_output", + "command": "[DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ'); $n=0; Get-Content tests\\homepage.spec.ts | ForEach-Object { $n++; if($n -ge 69 -and $n -le 76){ '{0}: {1}' -f $n,$_ } }; $n=0; Get-Content src\\components\\sections\\projects\\ProjectCard.tsx | ForEach-Object { $n++; if($n -ge 10 -and $n -le 18){ '{0}: {1}' -f $n,$_ } }; git show b839ff9:components/projects/ProjectCard.tsx | Select-String -Pattern 'initial=|whileInView=|viewport=|transition=' | ForEach-Object { $_.Line }", + "exit_code": 0, + "ref": "evidence/homepage-project-opacity-2026-08-14.txt#L8-L11", + "captured": "2026-08-14T11:02:20Z" + }, + "produced_by": "run_2026-08-14-opacity-a1", + "created": "2026-08-14T11:02:20Z" + }, + { + "id": "claim_c19e5f73", + "subject": "homepage_project_card_opacity_check", + "predicate": "historically_used_scroll_reveal", + "object": "the historical ProjectCard animated opacity from 0 to 1 on viewport entry with a 0.5 second duration and index-based delay", + "confidence": 1, + "source": { + "kind": "tool_output", + "command": "[DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ'); $n=0; Get-Content tests\\homepage.spec.ts | ForEach-Object { $n++; if($n -ge 69 -and $n -le 76){ '{0}: {1}' -f $n,$_ } }; $n=0; Get-Content src\\components\\sections\\projects\\ProjectCard.tsx | ForEach-Object { $n++; if($n -ge 10 -and $n -le 18){ '{0}: {1}' -f $n,$_ } }; git show b839ff9:components/projects/ProjectCard.tsx | Select-String -Pattern 'initial=|whileInView=|viewport=|transition=' | ForEach-Object { $_.Line }", + "exit_code": 0, + "ref": "evidence/homepage-project-opacity-2026-08-14.txt#L23-L26", + "captured": "2026-08-14T11:02:20Z" + }, + "produced_by": "run_2026-08-14-opacity-a1", + "created": "2026-08-14T11:02:20Z" + }, + { + "id": "claim_d2f6a804", + "subject": "homepage_project_card_opacity_check", + "predicate": "currently_renders_plain_article", + "object": "the current ProjectCard renders a plain article without opacity or viewport animation props", + "confidence": 1, + "source": { + "kind": "tool_output", + "command": "[DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ'); $n=0; Get-Content tests\\homepage.spec.ts | ForEach-Object { $n++; if($n -ge 69 -and $n -le 76){ '{0}: {1}' -f $n,$_ } }; $n=0; Get-Content src\\components\\sections\\projects\\ProjectCard.tsx | ForEach-Object { $n++; if($n -ge 10 -and $n -le 18){ '{0}: {1}' -f $n,$_ } }; git show b839ff9:components/projects/ProjectCard.tsx | Select-String -Pattern 'initial=|whileInView=|viewport=|transition=' | ForEach-Object { $_.Line }", + "exit_code": 0, + "ref": "evidence/homepage-project-opacity-2026-08-14.txt#L12-L21", + "captured": "2026-08-14T11:02:20Z" + }, + "produced_by": "run_2026-08-14-opacity-a1", + "created": "2026-08-14T11:02:20Z" + }, + { + "id": "claim_e5b7c391", + "subject": "homepage_project_card_opacity_check", + "predicate": "opacity_poll_is_legacy", + "object": "the opacity poll now appears redundant because the current card has no opacity animation", + "confidence": 0.95, + "source": { + "kind": "inference", + "basis": "The test still scrolls and polls opacity, while the current card is a plain article and the historical animation props are present only in the earlier b839ff9 implementation." + }, + "produced_by": "run_2026-08-14-opacity-a1", + "created": "2026-08-14T11:02:20Z" + } +] diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/entities.json b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/entities.json new file mode 100644 index 0000000..e75a012 --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/entities.json @@ -0,0 +1,15 @@ +[ + { + "id": "homepage_project_card_opacity_check", + "type": "repository_test", + "aliases": [ + "homepage Playwright test", + "project-card opacity check" + ], + "source_docs": [ + "evidence/homepage-project-opacity-2026-08-14.txt#L4-L11", + "evidence/homepage-project-opacity-2026-08-14.txt#L22-L27" + ], + "first_seen": "2026-08-14T11:02:20Z" + } +] diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/evidence/homepage-project-opacity-2026-08-14.txt b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/evidence/homepage-project-opacity-2026-08-14.txt new file mode 100644 index 0000000..48e590a --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/evidence/homepage-project-opacity-2026-08-14.txt @@ -0,0 +1,27 @@ +CAPTURED_AT: 2026-08-14T11:02:20Z +COMMAND: [DateTime]::UtcNow.ToString('yyyy-MM-ddTHH:mm:ssZ'); $n=0; Get-Content tests\homepage.spec.ts | ForEach-Object { $n++; if($n -ge 69 -and $n -le 76){ '{0}: {1}' -f $n,$_ } }; $n=0; Get-Content src\components\sections\projects\ProjectCard.tsx | ForEach-Object { $n++; if($n -ge 10 -and $n -le 18){ '{0}: {1}' -f $n,$_ } }; git show b839ff9:components/projects/ProjectCard.tsx | Select-String -Pattern 'initial=|whileInView=|viewport=|transition=' | ForEach-Object { $_.Line } +EXIT_CODE: 0 +tests/homepage.spec.ts:69: const mainProjectCards = page.locator('#projects article[data-project-tier="main"]'); +tests/homepage.spec.ts:70: for (let index = 0; index < (await mainProjectCards.count()); index += 1) { +tests/homepage.spec.ts:71: const projectCard = mainProjectCards.nth(index); +tests/homepage.spec.ts:72: await projectCard.scrollIntoViewIfNeeded(); +tests/homepage.spec.ts:73: await expect +tests/homepage.spec.ts:74: .poll(() => projectCard.evaluate((element) => Number(getComputedStyle(element).opacity))) +tests/homepage.spec.ts:75: .toBeGreaterThan(0.99); +tests/homepage.spec.ts:76: } +CURRENT_PROJECT_CARD: +src/components/sections/projects/ProjectCard.tsx:10: return ( +src/components/sections/projects/ProjectCard.tsx:11:
+HISTORICAL_PROJECT_CARD: +git show b839ff9:components/projects/ProjectCard.tsx: + initial={{ opacity: 0, y: 30 }} + whileInView={{ opacity: 1, y: 0 }} + viewport={{ once: true, margin: "-50px" }} + transition={{ duration: 0.5, delay: index * 0.1 }} diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/runs.json b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/runs.json new file mode 100644 index 0000000..40ab671 --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/graphkeeper/runs.json @@ -0,0 +1,20 @@ +[ + { + "id": "run_2026-08-14-opacity-a1", + "started": "2026-08-14T11:02:20Z", + "tool": "coding_agent", + "task": "capture and record the homepage project-card opacity finding", + "evidence": [ + "evidence/homepage-project-opacity-2026-08-14.txt" + ], + "claims_written": [ + "claim_4a7c2e91", + "claim_b8d4e2a6", + "claim_c19e5f73", + "claim_d2f6a804", + "claim_e5b7c391" + ], + "ended": "2026-08-14T11:02:20Z", + "verdict": "passed" + } +] diff --git a/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/prompts.md b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/prompts.md new file mode 100644 index 0000000..2c537ca --- /dev/null +++ b/benchmarks/results/repeated-investigation-post-fix-pass-2026-08-14/prompts.md @@ -0,0 +1,9 @@ +# Exact benchmark prompt + +> Investigate the homepage project opacity behavior and determine whether the opacity +> poll is still part of the current implementation or legacy behavior. + +The GraphKeeper workspace additionally supplied this repository guidance: + +> Before repeating repository investigation, invoke `$graphkeeper` to check existing +> durable findings. Record new durable, evidence-backed findings through that skill.