Robot search output ignores --fields minimal and --max-tokens

[pablontiv]

Description

backscroll search --robot does not honor the documented agent-output contract for --fields minimal or --max-tokens.

runSearch builds robot output from modelResults, whose Content field is populated from the complete r.Text, and then writes the formatted lines directly. The robot branch therefore bypasses both the minimal r.Snippet projection used by JSON and the picokit formatter that enforces the token budget.

Relevant implementation: cmd/backscroll/search.go in runSearch, particularly the r.Text assignment and direct robot-line output.

Steps to Reproduce

1. Use an index containing several long messages that match a known term.

2. Run:

   backscroll search "<known-term>" \
     --all-projects \
     --robot \
     --fields minimal \
     --max-tokens 100 \
     --limit 10 > /tmp/backscroll-robot.out
   wc -c /tmp/backscroll-robot.out

3. Compare with text output using the same budget:

   backscroll search "<known-term>" \
     --all-projects \
     --max-tokens 100 \
     --limit 10 > /tmp/backscroll-text.out
   wc -c /tmp/backscroll-text.out

In the observed corpus, robot output was 16,350 bytes while text output was 388 bytes under the same --max-tokens 100 budget.

Expected Behavior

• --robot --fields minimal emits only the documented minimal fields: source path, bounded snippet, score, role, and timestamp.
• --max-tokens bounds the complete robot payload and emits an explicit truncation indicator when results do not fit.
• Agent callers can trust the declared output budget.

Actual Behavior

• Robot output includes complete message content.
• --fields minimal does not alter the robot payload.
• --max-tokens does not constrain robot output because the branch writes directly to stdout.
• One long high-ranked result can consume the caller's context before later relevant results are reached.

Environment

• Backscroll version: 3.5.1
• OS: macOS (darwin/arm64)
• Go implementation from current main

Acceptance criteria

• Robot mode uses the minimal snippet projection when --fields minimal is selected.
• Robot mode exposes full content only when --fields full is selected.
• --max-tokens applies to the complete robot payload, not independently per field or result.
• Truncation preserves parseable result_N_field=value output and reports that additional results were omitted.
• A regression test uses long matching messages and proves both minimal-field selection and a bounded total payload.
• JSON and text output behavior remains unchanged.

[pablontiv]

[backscroll-team/investigador] 2026-09-06 UTC
[backscroll-team] Estado: ready-for-agent

Reviewed main SHA: 1e698e58504bbb7ec83f835a675bbea32219f2f2. Static inspection unless explicitly stated otherwise; this is not a QA verdict.

Handoff — first implementation priority

The defect is still present and has a bounded, already-requested contract. No open PR or team claim was found. No dependency on MCP, embeddings, schema changes, or the ranking experiments in #64/#66.

Current evidence

• runSearch, lines 196–245: modelResults.Content = r.Text; only the JSON/minimal branch selects r.Snippet. The robot branch writes resultsToLines(modelResults, FormatRobot) directly with fmt.Fprintln, never applying maxTokens or fields.
• Current search contract: approximate total-output budget (characters / 4), stop before a result exceeding the budget, no partial-result output. Product-owned skill, section 4, explicitly invokes robot/minimal with a budget.
• Existing tests: TestSearchOutputRespectsTokenLimit exercises text, not robot, and its out == "" && len(out) > 0 assertion cannot fire. The robot-unwrapped test may skip when its fixture produces no result. Neither demonstrates this issue's acceptance.

Implementation boundary
Satisfy the issue's existing acceptance criteria in the robot serialization path: selected minimal/full projection, complete-payload budgeting after escaping (including envelope/newlines and truncation metadata), a parseable omission indicator, and unchanged JSON/text behavior. Preserve the existing once-wrapped result_N_field=value envelope and string escapes. Do not route already-formatted robot lines through a formatter that double-wraps them. Align CLI help and living docs with the implemented minimal-field contract. Existing consumers such as scripts/eval.sh read filepath/content; do not silently break them when implementing the requested projection.

Required regression/QA fixture
Use an isolated config/input directory and temporary DB with several long matching records and a unique target inside each record. First prove ingestion and a nonempty unbounded query. Compare:

backscroll search --text budgetsentinel --all-projects --robot --fields minimal --max-tokens 100 --limit 10
backscroll search --text budgetsentinel --all-projects --robot --fields full --max-tokens 100 --limit 10
backscroll search --text budgetsentinel --all-projects --robot --fields minimal --max-tokens 0 --limit 10

Check actual emitted payload, not just exit status: minimal snippet versus full content; budgeted versus unlimited; first oversized result; a later result that would overflow; no matches; multiline/backslash/non-ASCII values; no partial result or double wrapping; explicit omission metadata. Budget accounting must follow the documented approximate estimator, not an invented tokenizer. Retain JSON/text regression coverage. New tests must fail on this SHA and not skip merely because fixture setup failed.

Validation honesty
A clean checkout of this exact SHA was obtained. Go is absent from this runtime; the official toolchain network probe could not complete because network approval was cancelled. No binary, product scenario, or Go test was executed here. The issue's historical byte counts have NOT been independently reproduced. Developer/QA must provide real execution evidence and the normal build/tests/aggregate-coverage >=85%/gitleaks gates before merge.

[pablontiv]

[backscroll-team/desarrollador] 2026-09-06T09:30:41.820Z
[backscroll-team] Estado: implementation-complete; awaiting-qa-review
Ejecución: developer-20260906T0912Z-issue63
Issue: #63; PR: #67; rama: backscroll-team/issue-63.
SHA entregado: 152c4208f3167ba9cfdab1f6e568940dc7c73fe4; base main: 1e698e58504bbb7ec83f835a675bbea32219f2f2.

Implementación completa y reclamación de implementación liberada. Corregí el redondeo del payload completo solicitado por revisión, con regresión RED antes del cambio y GREEN después. La descripción del PR #67 conserva pruebas, fixtures, resultados y comparación controlada entre Go 1.26.2/1.27.1. CI 34024643936: build/tests, cobertura agregada 86.2% y gitleaks exitosos para este SHA. No modifiqué controles ni hice merge.

Próximo paso: QA independiente del SHA entregado y reevaluación técnica del revisor. El PR queda en borrador solamente por esas validaciones pendientes; una vez satisfechas, el revisor puede quitar draft e integrar con todos los controles. Este comentario no es QA_PASS ni resuelve la revisión.