Add conversational paraphrase and query-echo cases to the recall evaluation set

[pablontiv]

Problem

The current recall evaluation set reports 100% recall@5 on a small collection of direct, artifact-shaped queries. That score did not predict a real conversational recall failure where the index contained the exact answer but the requested message fell below the top five.

The missing evaluation dimensions are:

1. Conversational paraphrase: the user describes a remembered concept without reproducing the original wording.
2. Terminology recovery: the expected answer is a short term inside a longer prior response.
3. Query-echo contamination: earlier backscroll search tool calls repeat the query tokens and compete with historical prose.
4. Progressive refinement: repeated searches should improve retrieval rather than add higher-ranked self-generated noise.
5. Budgeted agent output: the expected result must remain reachable under --robot --fields minimal --max-tokens.

Without these cases, recall@5 can remain perfect while realistic agent recall still requires several manual query rewrites.

Proposed Solution

Extend docs/eval/queries.toml and scripts/eval.sh with sanitized fixtures derived from this observed workflow.

At minimum, add cases that:

• ask for a prior technical term using different vocabulary from the stored answer;
• place the target phrase inside a longer assistant response;
• add later Backscroll query invocations containing the same terms;
• measure the target's rank before and after query refinement;
• execute the documented robot/minimal/token-budget contract;
• report both recall@5 and whether the target was present in the emitted bounded payload.

Keep the existing lexical baseline, but report results by cohort so regressions in conversational recall are not hidden by easy exact-term queries.

Suggested cohorts:

artifact-literal
conversational-paraphrase
terminology-recovery
query-echo
budgeted-agent-output

Alternatives Considered

• Replace the existing eval-set. Rejected because the current direct queries still protect useful behavior; new cohorts should complement them.
• Activate embeddings immediately. Rejected because the benchmark should first demonstrate the gap and compare lexical improvements against a stable baseline.
• Measure only whether the target exists anywhere in the result set. Rejected because agents consume bounded top-ranked output; rank and emitted-payload reachability are the user-visible contract.

Additional Context

The current repository documentation records 100% recall@5 as the reason semantic embeddings were not activated. This new cohort provides evidence for deciding whether lexical ranking, query relaxation, or semantic retrieval produces a measurable improvement.

This issue should land before changing query-relaxation behavior so that the implementation has a failing, representative acceptance case.

[pablontiv]

[backscroll-team/investigador] 2026-09-06T09:59:30Z
[backscroll-team] Estado: ready-for-agent

Reviewed main SHA: 41c098afaf0e8f5313b90b44801db7189bfb78a1. Static inspection unless explicitly stated otherwise; this is not a QA verdict.

Handoff — next executable team ticket

The prerequisite robot-budget work is now merged through #67 and #63 is closed. The push CI and release workflow for this main SHA completed successfully. There are no open PRs, no backscroll-team/* branches, and no competing claim found, so #65 remains the highest-priority ready-for-agent ticket before #64/#66.

Current evidence on main

• docs/eval/queries.toml still contains 20 [[query]] records based on historical path/session substrings; it has no cohort field and no self-contained synthetic input corpus.
• scripts/eval.sh still requires an already-populated configured index and reports one aggregate recall@5. At line 155 it merges stderr into robot stdout and suppresses command failures; lines 162–166 inspect only filepath plus the first 200 content characters. Execution failure, full target rank, and reachability in the bounded payload are therefore not distinguished.
• fix(search): enforce robot fields and output budget #67 changed search serialization/budget behavior, tests and documentation only; it did not add evaluation cohorts or resolve Add conversational paraphrase and query-echo cases to the recall evaluation set #65.

Observable acceptance

1. Add reproducible sanitized/synthetic inputs with stable target identities for artifact-literal, conversational-paraphrase, terminology-recovery, query-echo and budgeted-agent-output cohorts. Retain the existing lexical baseline, but do not require the operator's corpus.
2. For every case report command success/failure, target rank in reference retrieval, target presence in the emitted bounded robot payload, and cohort totals. Ground truth must identify the intended record, not merely a shared path/project substring.
3. Cover an answer term beyond 200 characters, target absent, command failure, multiple later echo rows and progressive refinement.
4. Build a dev binary and run it with isolated HOME/XDG/config/SQLite/input fixtures; keep stdout and stderr separate. Use the robot minimal/budget contract now merged on main and record SHA/fixture provenance.
5. Report baseline deficiencies honestly. This issue must not change ranking/defaults, enable embeddings, invent relaxation/echo policy, or create a new mandatory CI threshold.

Dependencies and priority

No unresolved implementation dependency remains. #64 and #66 must consume this evaluation evidence before choosing exclusion/de-prioritization or relaxation behavior; they remain needs-info. #48/#49 still require product/transport decisions, and #62 still requires a supported-destination policy plus digest-bound approval for real installation replacement.

A developer/QA run is still required for executed product evidence.

[pablontiv]

[backscroll-team/desarrollador] 2026-09-06T18:25:16Z

Estado: IMPLEMENTED — QA/REVIEW PENDING
Ejecución: developer-20260906T181214Z
Issue: #65
Base: main@41c098afaf0e8f5313b90b44801db7189bfb78a1
Rama: backscroll-team/issue-65
PR: #68
SHA publicado: 85704cac6efdf39175bfc74a5c3a64600a5cdfe6

Atendí los dos hallazgos del revisor: el wrapper conserva ahora los exit 0/1/2 al ejecutar el evaluador compilado, y los procesos del dataset sintético usan un cwd temporal sin heredar ./backscroll.toml; legacy conserva el cwd heredado. Ambos defectos tienen regresiones que fallaron sobre el SHA anterior y pasan en el actual.

Verificación local fresca: gofmt, go mod verify, go vet ./..., just check, just test, just ci (86.0% agregado), go test -race ./scripts/recall-eval, exit 2 de argumentos inválidos y evaluación sintética real exit 0.

CI del SHA publicado: Test & Build aprobado con cobertura agregada 86.0% y gitleaks aprobado. Los jobs light no aplicables y release de push fueron omitidos por diseño.

La implementación está completa y disponible para nueva QA independiente. El QA_PASS anterior quedó invalidado por los commits; el PR permanece borrador pendiente de QA_PASS para este SHA y reevaluación técnica de los threads del revisor.