fix(bench): --prompt-file, prompt evidence in JSON, short-prompt prefill warning

[Kaden-Schutt]

Summary

hipfire bench with no prompt sends "Explain the theory of general relativity in simple terms." — 24 tokens after the chat template — and reports prefill_tok_s ≈ 363 on a 7900 XTX for 27B mq4; the same binary with a 4.4k-token prompt reports 886. The default number is launch overhead, and the JSON gave a reader no way to tell: it recorded neither prompt length nor prompt identity. Issue #644's "prefill 286.9" comparison was the same artefact.

This adds --prompt-file <PATH> (verbatim bytes; conflicts with positional words — AGENTS.md rule 2 says one newline moves τ by 17%), records prompt_tokens / prompt_md5 / prompt_chars and a warnings[] array in the JSON and on the stderr banner, and warns when the prompt is under 256 tokens. The default prompt is unchanged (historical numbers depend on it) and no existing key is renamed or nulled.

Which crate(s) does this touch?

• crates/hipfire-cli (bench), docs (AGENTS.md §3.1 flag table, docs/CLI.md)

Evidence (RX 7900 XTX, c4aab87a2, qwen3.8:27b)

--prompt-file benchmarks/prompts/glimmer_prefill_4096.txt
  {'prompt_tokens': 4408, 'prompt_md5': '243e2c64fa7db53686dc0a1d58099654', 'prompt_chars': 16153, 'warnings': []} | prefill 885.0
default prompt
  {'prompt_tokens': 24, 'warnings': ['prompt is 24 tokens; prefill_tok_s at this length measures launch overhead, not prefill throughput — use --prompt-file with ≥256 tokens for a prefill number']} | prefill 366.3
--prompt-file + positional
  error: the argument '--prompt-file <PROMPT_FILE>' cannot be used with '[PROMPT]...'

prompt_tokens is prefill_tokens + cached_tokens from the daemon's done event (there is no prompt_tokens key; the first cut read one and reported null — caught on hardware, fixed in 003f3e49d).

Test plan

• cargo test -p hipfire-cli — 214 pass (8 new: clap conflict, default-prompt byte stability, positional join, verbatim file read + md5, resolver conflict, warning threshold, md5 vector, done-event token read)
• Hardware above
• Consumers checked by grep: scripts/qwen38_ladder_summarize.py and friends read known keys via .get(); scripts/speed-gate.sh greps stderr sample rows (unchanged). Additive keys only.

[hipfire-sol]

hw-gate sol prelim

summary: Adds a standard-benchmark prompt-file input that preserves UTF-8 file bytes, reports prompt identity and length in benchmark JSON/stderr, and warns when prefill throughput is measured with fewer than 256 prompt tokens; it also adds the pure-Rust md5 dependency and updates CLI documentation.

run_hardware: true
run_hardware_reasons: The only new filesystem access reads the user-explicit --prompt-file path and validates it as UTF-8; it does not enumerate files, read credentials, or write outside existing paths.; The md5 dependency hashes prompt bytes only and introduces no native code, build script, network operation at runtime, unsafe code, process spawning, or GPU-byte/layout change.; A real fixture load exercises the rebuilt CLI/daemon dependency graph affected by Cargo.toml and Cargo.lock.

routes:

mode	tag	source	why
battery	qwen3.6:27b	bucket	bucket load
battery	ornith-1.5:35b-a3b-mq4r	bucket	bucket load
battery	lfm2.5:1.2b	bucket	bucket load
battery	qwen3.8:27b-mq4-xt	bucket	bucket load

unavailable_routes:

(none)

claim_assessment: The author claims --prompt-file preserves prompt bytes, JSON exposes prompt_tokens/prompt_md5/prompt_chars/warnings, and short prompts receive a launch-overhead warning. The diff and unit tests support the CLI mechanics, but the available battery/chain hardware route can prove only that the rebuilt product still loads and generates coherently; proving the benchmark-report claim requires an actual hipfire bench run with both the default prompt and a committed >=256-token prompt file, checking the emitted JSON and md5 against the file bytes.

[hipfire-sol]

hw-gate evidence — 2 lane(s) — verdict pass

lane hiptrx (gfx1201)

hw-gate evidence

field	value
base	cf87b0b1c18c3d54b3d9ff4f3f8ff28b817bae12
head	8f3a9b6c58ef384d781ba20bdb2680aa2b338f0e
buckets	load
host gfx	gfx1201
host rocm	7.15.26333-0000000
device	3
runner	hiptrx
daemon_md5	17ba7dfab49aa1c1110d3f412f072648
hipfire_md5	480a53edd57370a8f4f77d79632c06d2
build_seconds	12.124895811080933
verdict	pass
logs_dir	hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 31.1 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	34	0	132	62	3.1	15.0	False	False	False	True
battery	1		stop	30	0	81	71	506.8	36.1	False	False	False	True
battery	2		stop	31	0	16	13	525.5	36.2	False	False	False	True
battery	3		stop	47	0	31	8	561.9	36.1	False	False	False	True
battery	4		stop	47	0	6	2	564.6	36.2	False	False	False	True

qwen3.6:27b battery turn 0

```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_length = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
            
    return s[max_start:max_start + max_length]
```

qwen3.6:27b battery turn 1

During the day, sunlight passes through a shorter distance in Earth's atmosphere, allowing blue light to scatter more effectively than other colors due to its shorter wavelength. At sunset, sunlight travels through a much thicker layer of atmosphere, causing most of the blue light to scatter away before reaching the observer. This leaves the longer wavelengths of red and orange light to dominate the sky, creating the characteristic hues of dusk.

qwen3.6:27b battery turn 2

The capital of France is Paris, and the River Seine runs through it.

qwen3.6:27b battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

qwen3.6:27b battery turn 4

Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 32.9 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	34	0	144	59	23.1	16.3	False	False	False	True
battery	1		stop	30	0	117	102	471.1	54.9	False	False	False	True
battery	2		stop	31	0	16	13	483.6	66.7	False	False	False	True
battery	3		stop	47	0	31	8	693.0	59.1	False	False	False	True
battery	4		stop	47	0	17	7	679.0	49.9	False	False	False	True

ornith-1.5:35b-a3b-mq4r battery turn 0

```python
def longest_substring_without_repeating_chars(s):
    seen = set()
    start = 0
    max_start = 0
    max_len = 0
    
    for end in range(len(s)):
        if s[end] in seen:
            while s[start] != s[end]:
                seen.discard(s[start])
                start += 1
            start += 1
        else:
            seen.add(s[end])
        
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_len]
```

ornith-1.5:35b-a3b-mq4r battery turn 1

During the day, sunlight interacts with particles in the atmosphere, and shorter blue wavelengths scatter more easily off air molecules than longer wavelengths, filling the sky with blue light from all directions.

At sunset, the sun sits lower on the horizon, so its light travels through much more atmosphere, scattering away the blue before it reaches your eyes and allowing the longer red and orange wavelengths to dominate the sky.

This phenomenon, known as Rayleigh scattering, explains why the color of the sky shifts dramatically depending on the angle of the sun and the distance its light must travel through the atmosphere.

ornith-1.5:35b-a3b-mq4r battery turn 2

The capital of France is Paris, and the Seine River runs through it.

ornith-1.5:35b-a3b-mq4r battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

ornith-1.5:35b-a3b-mq4r battery turn 4

17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 8.1 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	0	0	130	58	None	208.33	False	False	False	True
battery	1		stop	0	0	79	71	None	208.99	False	False	False	True
battery	2		stop	0	0	15	13	None	208.33	False	False	False	True
battery	3		stop	0	0	26	8	None	209.68	False	False	False	True
battery	4		stop	0	0	18	9	None	209.3	False	False	False	True

lfm2.5:1.2b battery turn 0

```python
def longest_no_repeat(s):
    last_seen = {}
    start = 0
    max_len = 0
    max_sub = ""
    
    for i, char in enumerate(s):
        if char in last_seen and last_seen[char] > start:
            start = last_seen[char] + 1
        else:
            if i - start + 1 > max_len:
                max_len = i - start + 1
                max_sub = s[start:i+1]
        last_seen[char] = i
    return max_sub
```

lfm2.5:1.2b battery turn 1

During the day, the blue sky appears because blue light from the sun scatters in all directions when it enters the atmosphere, making it visible to everyone. At sunset, the sky turns red because sunlight travels through more of the atmosphere, allowing red wavelengths to dominate after scattering away most of the shorter blue wavelengths. These effects are a result of how light interacts with particles and molecules in the air.

lfm2.5:1.2b battery turn 2

The capital of France is Paris, and the Seine River runs through it.

lfm2.5:1.2b battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

lfm2.5:1.2b battery turn 4

Let's calculate: 17 + 26 = 43.  
Answer: 43

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 16.0 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	34	0	151	66	64.0	50.4	False	False	False	True
battery	1		stop	30	0	92	82	367.2	54.9	False	False	False	True
battery	2		stop	31	0	17	14	379.2	48.8	False	False	False	True
battery	3		stop	47	0	31	8	458.2	58.6	False	False	False	True
battery	4		stop	47	0	151	81	455.7	34.2	False	False	False	True

qwen3.8:27b-mq4-xt battery turn 0

```python
def longest_substring_without_repeating(s: str) -> str:
    if not s:
        return ""
    
    start = 0
    longest_start = 0
    longest_length = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        
        current_length = end - start + 1
        if current_length > longest_length:
            longest_length = current_length
            longest_start = start
    
    return s[longest_start:longest_start + longest_length]
```

qwen3.8:27b-mq4-xt battery turn 1

The blue color of the daytime sky is caused by Rayleigh scattering, where sunlight interacts with gas molecules in the atmosphere and scatters shorter blue wavelengths more effectively than longer red ones. As the sun sets, light travels through a much thicker slice of the atmosphere, which filters out most of the blue and green light before it reaches your eyes. This leaves the longer red and orange wavelengths to dominate the scene, resulting in the characteristic warm hues of a sunset.

qwen3.8:27b-mq4-xt battery turn 2

The capital of France is Paris, which is run through by the Seine River.

qwen3.8:27b-mq4-xt battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

qwen3.8:27b-mq4-xt battery turn 4

To find the sum of 17 and 26, we can add the ones and tens separately:

1.  Add the ones: $7 + 6 = 13$. Write down 3 and carry over 1.
2.  Add the tens (including the carry): $1 + 2 + 1 = 4$.
3.  Combine the tens and ones: $43$.

Alternatively:
$17 + 26 = (10 + 7) + (20 + 6) = (10 + 20) + (7 + 6) = 30 + 13 = 43$.

Answer: 43

kernel

not run

lane hipx (gfx1100)

hw-gate evidence

field	value
base	cf87b0b1c18c3d54b3d9ff4f3f8ff28b817bae12
head	8f3a9b6c58ef384d781ba20bdb2680aa2b338f0e
buckets	load
host gfx	gfx1100
host rocm	7.15.26333-0000000
device	0
runner	hipx
daemon_md5	17ba7dfab49aa1c1110d3f412f072648
hipfire_md5	480a53edd57370a8f4f77d79632c06d2
build_seconds	6.483356714248657
verdict	pass
logs_dir	hw-gate-logs

fixtures

qwen3.6:27b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 34.9 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	34	0	132	62	3.2	17.1	False	False	False	True
battery	1		stop	30	0	86	74	441.4	50.4	False	False	False	True
battery	2		stop	31	0	16	13	446.3	50.4	False	False	False	True
battery	3		stop	47	0	31	8	463.1	50.3	False	False	False	True
battery	4		stop	47	0	17	7	468.4	50.3	False	False	False	True

qwen3.6:27b battery turn 0

```python
def longest_substring_without_repeating(s: str) -> str:
    char_index = {}
    start = 0
    max_len = 0
    max_start = 0
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = start
            
    return s[max_start:max_start + max_len]
```

qwen3.6:27b battery turn 1

During the day, sunlight scatters more effectively against Earth's atmosphere when it passes through a shorter distance, causing shorter blue wavelengths to dominate what we see. At sunset, however, sunlight must travel through a much thicker layer of the atmosphere to reach our eyes, which filters out most of the shorter blue and green wavelengths. This leaves only the longer red and orange wavelengths to pass through, giving the sky its characteristic warm hues.

qwen3.6:27b battery turn 2

The capital of France is Paris, and the River Seine runs through it.

qwen3.6:27b battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

qwen3.6:27b battery turn 4

17 + 26 = 43

Answer: 43

ornith-1.5:35b-a3b-mq4r

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 36.0 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	34	0	132	62	463.9	15.1	False	False	False	True
battery	1		stop	30	0	114	102	495.1	99.3	False	False	False	True
battery	2		stop	31	0	16	13	522.4	107.1	False	False	False	True
battery	3		stop	47	0	31	8	740.5	132.1	False	False	False	True
battery	4		stop	47	0	17	7	748.9	76.2	False	False	False	True

ornith-1.5:35b-a3b-mq4r battery turn 0

```python
def longest_substring_no_repeats(s: str) -> str:
    start = 0
    max_start = 0
    max_len = 0
    char_index = {}

    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_len:
            max_len = end - start + 1
            max_start = start

    return s[max_start:max_start + max_len]
```

ornith-1.5:35b-a3b-mq4r battery turn 1

The sky appears blue during the day because sunlight interacts with atmospheric gases through a process called Rayleigh scattering, where shorter blue wavelengths are scattered more effectively by tiny air molecules than longer wavelengths, spreading blue light across the entire sky. At sunset, the Sun sits low on the horizon, meaning its light travels through a much thicker slice of atmosphere to reach your eyes, causing most of the scattered blue light to be deflected away before it arrives. This leaves predominantly the longer red and orange wavelengths to reach your vision directly, painting the sky and clouds in warm sunset tones.

ornith-1.5:35b-a3b-mq4r battery turn 2

The capital of France is Paris, and the River Seine runs through it.

ornith-1.5:35b-a3b-mq4r battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

ornith-1.5:35b-a3b-mq4r battery turn 4

17 + 26 = 43

Answer: 43

lfm2.5:1.2b

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 8.1 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	0	0	120	61	None	223.88	False	False	False	True
battery	1		stop	0	0	75	63	None	225.23	False	False	False	True
battery	2		stop	0	0	18	16	None	230.77	False	False	False	True
battery	3		stop	0	0	26	8	None	228.07	False	False	False	True
battery	4		stop	0	0	18	9	None	227.85	False	False	False	True

lfm2.5:1.2b battery turn 0

```python
def longest_unique_sub(s):
    seen = {}
    start = 0
    max_len = 0
    max_sub = ""

    for i, char in enumerate(s):
        if char in seen and start <= seen[char]:
            start = seen[char] + 1
        else:
            if i - start + 1 > max_len:
                max_len = i - start + 1
                max_sub = s[start: i + 1]
        seen[char] = i

    return max_sub
```

lfm2.5:1.2b battery turn 1

The sky appears blue during the day because sunlight scatters off air molecules in all directions, and blue light scatters more effectively due to shorter wavelengths. At sunset, the light has to pass through more atmosphere, scattering the red and orange wavelengths out, leaving the longer red and orange hues visible. Thus, the colors shift with the sun’s angle and the atmosphere’s composition.

lfm2.5:1.2b battery turn 2

The capital of France is Paris, and the river that runs through it is the Seine.

lfm2.5:1.2b battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

lfm2.5:1.2b battery turn 4

Let's calculate: 17 + 26 equals 43.  
Answer: 43

qwen3.8:27b-mq4-xt

source: bucket sha256_ok: ✅ size_ok: ✅ status: pass reason:

battery — exit 0 seconds 19.3 status pass

mode	idx	genre	finish	ctx	cached	gen	ans_words	prefill_tok_s	decode_tok_s	attractor	empty	runaway	recall_ok
battery	0		stop	34	0	133	62	307.0	28.6	False	False	False	True
battery	1		stop	30	0	80	67	419.0	49.0	False	False	False	True
battery	2		stop	31	0	16	13	431.8	49.1	False	False	False	True
battery	3		stop	47	0	31	8	478.2	48.9	False	False	False	True
battery	4		stop	47	0	17	7	474.2	49.1	False	False	False	True

qwen3.8:27b-mq4-xt battery turn 0

```python
def longest_substring_without_repeating(s: str) -> str:
    start = 0
    max_length = 0
    max_start = 0
    char_index = {}
    
    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            start = char_index[char] + 1
        char_index[char] = end
        if end - start + 1 > max_length:
            max_length = end - start + 1
            max_start = start
    
    return s[max_start:max_start + max_length]
```

qwen3.8:27b-mq4-xt battery turn 1

Sunlight contains all colors of the visible spectrum, which scatter differently as they pass through Earth's atmosphere. Shorter blue wavelengths scatter much more widely than longer red wavelengths due to Rayleigh scattering, making the daytime sky appear blue. At sunset, sunlight travels through a much thicker layer of atmosphere, causing the blue light to scatter away before reaching your eyes, leaving the red and orange wavelengths visible.

qwen3.8:27b-mq4-xt battery turn 2

The capital of France is Paris, and the River Seine runs through it.

qwen3.8:27b-mq4-xt battery turn 3

{
  "name": "Alice",
  "age": 34,
  "city": "Lisbon"
}

qwen3.8:27b-mq4-xt battery turn 4

17 + 26 = 43

Answer: 43

kernel

not run

[hipfire-sol]

hw-gate sol verdict

{
  "claim_verdict": "not-exercised",
  "confidence": 0.96,
  "coverage": {
    "gaps": [
      "The hardware run used serve-harness battery routes rather than `hipfire bench`, so it did not exercise `--prompt-file`, verbatim file reading, positional-prompt conflict handling, prompt metadata in JSON/stderr, cached-token accounting, or the short-prompt warning.",
      "No default-prompt versus committed >=256-token prompt-file run demonstrates that `prompt_md5` matches the exact file bytes and that `prompt_tokens`, `prompt_chars`, and `warnings` have the claimed values."
    ],
    "surfaces_evidenced": [
      "load"
    ],
    "surfaces_touched": [
      "load",
      "cli",
      "filesystem",
      "docs"
    ]
  },
  "decision": "needs-human",
  "eyeball": [
    "All battery outputs for qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, and qwen3.8:27b-mq4-xt on gfx1201 and gfx1100 are coherent, non-empty, responsive to their prompts, and free of token attractors or leaked special tokens."
  ],
  "phase": "verdict",
  "rationale": "The rebuilt CLI and daemon loaded real artifacts and generated coherent answers across both hardware lanes, so the Cargo.toml/Cargo.lock load risk is covered. However, the behavioral change in crates/hipfire-cli/src/main.rs:3619-3830 was never invoked by the hardware route. The central benchmark-reporting claim therefore remains unverified and the coverage gap requires a human or a targeted `hipfire bench` run.",
  "regressions": []
}

Floor: hard=[] soft=["coverage_gaps: ['The hardware run used serve-harness battery routes rather than hipfire bench, so it did not exercise --prompt-file, verbatim file reading, positional-prompt conflict handling, prompt metadata in JSON/stderr, cached-token accounting, or the short-prompt warning.', 'No default-prompt versus committed >=256-token prompt-file run demonstrates that prompt_md5 matches the exact file bytes and that prompt_tokens, prompt_chars, and warnings have the claimed values.']", 'model needs-human'] model_decision=needs-human final=needs-human

[hipfire-sol]

hw-gate sol verdict block: Head 1f21694 built on both lanes with identical binary MD5s, and all gfx1100 batteries passed. However, every mandatory gfx1201 fixture failed to warm after four serve attempts. The gate requires block for any fixture failure. The central behavior added in crates/hipfire-cli/src/main.rs:3619-3834 was also not exercised: no hardware command used --prompt-file or validated the added prompt evidence and warnings.

[hipfire-fable]

announcement: Fable ran hipfire bench on the gate build to close the gap Sol flagged: qwen3.8:27b with the default prompt (prompt_tokens 24, md5 d94d3115…, the launch-overhead warning present) and with benchmarks/prompts/glimmer_prefill_4096.txt (prompt_tokens 4408, md5 243e2c64… equal to the file's md5sum, warnings empty, prefill 825 vs 425 tok/s), plus verbatim-bytes, clap-conflict, and missing-file probes on lfm2.5, and an A/B against master showing four keys added, none removed, numbers unchanged. Merging to staging. Two small follow-ups if you want them: on models whose done event carries neither prefill_tokens nor cached_tokens (lfm2.5) prompt_tokens comes back null with warnings [] — consider a warning for 'daemon reported no prompt length'; and resolve_bench_prompt runs after open_bench_engine, so a mistyped --prompt-file costs a 27B load before it errors.

override: Override Sol needs-human: Sol's only gap was that the hardware route never invoked hipfire bench, so --prompt-file, the prompt keys, and the short-prompt warning were unexercised. I ran hipfire bench on the exact build under decision (hipfire 480a53ed…, daemon 17ba7dfa… == master's daemon) on qwen3.8:27b with the default prompt and the committed 4.4k-token file, plus verbatim/conflict/missing-file probes on lfm2.5:1.2b and an A/B against $HW_GATE_BASE_BIN. Every claimed value matched an independently computed one and no existing key or number moved.

investigation:

| question | route | result | evidence |

|---|---|---|---|

| Does the default-prompt run record prompt identity/length and fire the <256-token warning, without changing the default prompt? | $HW_GATE_EVIDENCE/pr689-r5-bench-route.sh pr689-r5-bench-pr-q38-default 3 $HW_GATE_BIN qwen3.8:27b --runs 3 --max-tokens 64 | exit 0. prompt_tokens 24, prompt_md5 d94d3115a3001f08a654d91461d6bdc4 (= md5 of the literal 'Explain the theory of general relativity in simple terms.' computed independently), prompt_chars 57, warnings = ['prompt is 24 tokens; prefill_tok_s at this length measures launch overhead, not prefill throughput — use --prompt-file with ≥256 tokens for a prefill number']. prefill_tok_s median 425.4 [423.1, 425.4, 425.9], decode 34.7, ttft 56.4 ms. Same four lines appear on the stderr banner. | pr689-r5-bench-pr-q38-default.json (+ .stderr) |

| Does --prompt-file send the file bytes verbatim, report md5/chars that match the file, count tokens as prefill+cached, and suppress the warning at ≥256 tokens? | $HW_GATE_EVIDENCE/pr689-r5-bench-route.sh pr689-r5-bench-pr-q38-promptfile 3 $HW_GATE_BIN qwen3.8:27b --runs 3 --max-tokens 64 --prompt-file /home/kaden/actions-runner/_work/hipfire/hipfire/pr/benchmarks/prompts/glimmer_prefill_4096.txt | exit 0. prompt_md5 243e2c64fa7db53686dc0a1d58099654 == md5sum of the file; prompt_chars 16153 == file length; prompt_tokens 4408 (matches the PR body exactly); warnings []. prefill_tok_s median 825.3 [434.1 cold, 829.7, 825.3], decode 33.8. The 24→4408 gap (425→825 tok/s) reproduces the body's 363→886 claim in shape on gfx1201. | pr689-r5-bench-pr-q38-promptfile.json (+ .stderr) |

| A/B: does master's bench JSON lose or change any key or number under the PR (same daemon)? | $HW_GATE_EVIDENCE/pr689-r5-bench-route.sh pr689-r5-bench-base-q38-default 3 $HW_GATE_BASE_BIN qwen3.8:27b --runs 3 --max-tokens 64 | exit 0. Base has none of prompt_tokens/prompt_md5/prompt_chars/warnings; PR adds exactly those four and removes nothing (batch, decode_tok_s, gpu, loaded, max_tokens, model, prefill_tok_s, protocol, runs, samples, ttft_ms, vram_free_before_mb, wall_tok_s all present in both). Shared numbers within noise: prefill 420.0 vs 425.4, decode 34.7 vs 34.7, ttft 57.1 vs 56.4 ms. | pr689-r5-bench-base-q38-default.json; key diff in pr689-r5-bench-q38-summary.txt |

| Is the file read truly raw (leading space, trailing newlines preserved), do positional words equal the default, and does the clap conflict / missing-file path fail cleanly? | pr689-r5-bench-route.sh on device 0 with lfm2.5:1.2b: --prompt-file $HIPFIRE_HOME/r5/probes/verbatim.txt (18 bytes ' repeat after me\n\n'); positional 'Explain the theory ... terms.'; --prompt-file X hello; --prompt-file /nonexistent/prompt.txt; base default; plus hipfire bench --help diff PR vs base | verbatim: JSON prompt_md5 fa68a0a35e1e97bdd63122d7f4ffe658 == md5sum of the probe file, prompt_chars 18. positional: prompt_md5 d94d3115… identical to the default run. conflict: clap exit 2 'the argument --prompt-file <PROMPT_FILE> cannot be used with [PROMPT]...', no daemon spawned. missing file: exit 1 'hipfire: failed to read --prompt-file /nonexistent/prompt.txt: No such file or directory' — after the GPU-detect line, i.e. a typo costs a model load (main.rs:3734 open_bench_engine precedes 3735 resolve_bench_prompt); nit. help diff: exactly one added line. Gap noted: lfm2.5's done event carries neither prefill_tokens nor cached_tokens, so prompt_tokens is null and warnings [] — harmless because prefill_tok_s/decode_tok_s/ttft_ms are null for this model on base too. | pr689-r5-bench-pr-lfm-{verbatim,positional,conflict,missingfile,default}.{json,stderr}, pr689-r5-bench-base-lfm-default.json, pr689-r5-bench-help-diff.txt, pr689-r5-bench-lfm-summary.txt |

| Do the mandatory load fixtures still load and generate coherently on this head after the rebase? | hw-gate.json (gate run, both lanes) — read every decoded turn | qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, qwen3.8:27b-mq4-xt all pass on gfx1201 and gfx1100: every turn finish=stop, recall clean, attractor=false, decoded text coherent (correct sliding-window code, Rayleigh scattering, Paris/Seine, valid JSON, 'Answer: 43'). Daemon md5 identical to master, so this was expected. | *-battery.json |

unproven:

• --exp (RDNA2 gfx103x variant sweep) also routes through resolve_bench_prompt (main.rs:4321); no gfx103x device on this host, so that arm is compile-level only. Low risk: same function, same inputs.

• --prompt-file is silently ignored under --concurrency (bench_concurrency.rs never reads args.prompt) and --matrix (synthetic prompts) — identical to how positional words were already ignored on master, so not a regression; a clap conflict would be friendlier.

rationale: The diff is crates/hipfire-cli only (main.rs:489-495 flag; 3622-3682 resolve_bench_prompt / bench_prompt_md5 / bench_prompt_warning / bench_prompt_tokens_from_done; 3735-3737 and 3770-3830 report wiring), the daemon binary is byte-identical to master (17ba7dfab49aa1c1110d3f412f072648 on both), and the PR files are byte-identical to the round-3 head 8bb48ba — the rebase only picked up master's #703 serve-harness fix, which is why the mandatory qwen3.8:27b-mq4-xt battery now passes. Hard floor: none. On the exact build (hipfire 480a53edd57370a8f4f77d79632c06d2, head 8f3a9b6) I exercised every added behavior: default prompt → prompt_tokens 24, prompt_md5 equal to the independently computed md5 of the unchanged literal, prompt_chars 57, warning fires; --prompt-file glimmer_prefill_4096.txt → prompt_md5 equal to md5sum of the file, prompt_chars 16153, prompt_tokens 4408, warnings [], prefill 825 tok/s vs 425 at 24 tokens; an 18-byte probe with leading space and two trailing newlines round-trips to the same md5 (verbatim, no trimming); positional words hash identically to the default; --prompt-file + positional is refused by clap before any daemon spawns; a missing file errors cleanly. A/B against $HW_GATE_BASE_BIN shows exactly four keys added, none removed, and decode/prefill/ttft within noise. Files: pr689-r5-bench-*.{json,stderr}, pr689-r5-bench-{q38,lfm}-summary.txt, pr689-r5-fable-summary.md. Two nits for the author, neither blocking: prompt_tokens is null (and warnings [] reads as 'long enough') on models whose done event lacks prefill_tokens/cached_tokens, e.g. lfm2.5 — harmless today because those models report no prefill_tok_s either; and open_bench_engine runs before the prompt file is read, so a typo costs a model load.

merged: beta 3149be7

soft floor: ["coverage_gaps: ['The hardware run used serve-harness battery routes rather than hipfire bench, so it did not exercise --prompt-file, verbatim file reading, positional-prompt conflict handling, prompt metadata in JSON/stderr, cached-token accounting, or the short-prompt warning.', 'No default-prompt versus committed >=256-token prompt-file run demonstrates that prompt_md5 matches the exact file bytes and that prompt_tokens, prompt_chars, and warnings have the claimed values.']", 'model needs-human']

[hipfire-fable]

Blocked by the hard floor, not by your code: the gfx1201 lane never got a daemon past hipInit, and the daemon binary is byte-identical to master. I reproduced the same kernel-level wedge (daemons stuck in kfd_create_process, unkillable) with both the PR and the base builds on two other GPUs, so this is the host's KFD state and needs a reboot before the lane can be re-run; the gfx1100 lane passed all four fixtures with your binaries. What I could check without a GPU holds up: the --prompt-file/positional conflict errors out correctly and the CLI surface differs from master by exactly that flag. Two things to fix before the re-run: your AGENTS.md hunk deletes the --reasoning-on row for a flag that still exists (please restore it), and resolve_bench_prompt runs after open_bench_engine, so a mistyped --prompt-file is only reported after the model has loaded — read the file first. The JSON fields, the short-prompt warning, and the prefill numbers still need to be produced on a healthy host; I have listed the exact commands in the record. The hard floor fired (hiptrx lane verdict=fail), so the decision space is block/hold and this is an evidence failure, hence block. What the maintainer should not take from it: the lane failure is not this PR. The daemon is byte-identical to base (md5 17ba7dfab49aa1c1110d3f412f072648 on both, $HW_GATE_EVIDENCE/pr689-orphan-daemons-and-binary-md5.txt); the diff touches only crates/hipfire-cli bench code, an md5 dependency, and docs. The gate logs show the daemon itself failing hipInit with code 100 four times per fixture, and the daemons that did spawn are stuck in the kernel at kfd_open->kfd_create_process->__flush_workqueue, unkillable, on PCI c3:00.0, during a window in which dmesg records SMU resumes on three GPUs (09:11:11) and REG_WAIT timeouts followed by a MODE1 reset on 7b:00.0. I replicated the gate route on device 1 with PR binaries and on device 4 with BASE binaries: both wedge identically (pr689-replica-gfx1201-dev1-pr-, pr689-replica-gfx1201-dev4-base-), and the bench daemon on device 0 did too. Sol's listed high-severity regression at crates/hipfire-cli/Cargo.toml:22 is therefore not supported; I do not carry it. The gfx1201 rows in hw-gate.json are a stale 08:23 JSON and prove nothing about the 09:08 run, as Sol said. What the PR adds was only partly exercised: the clap conflict is real (pr689-bench-pr-lfm-conflict.stderr, exit 2), the CLI surface diff is exactly one flag, and the prompt fixture md5/size match the PR body; the JSON fields, the warning, and the prefill claim could not be produced on this host. Two diff findings independent of hardware: the AGENTS.md hunk deletes the --reasoning-on row (base AGENTS.md:359) for a flag that still exists, and main.rs:3734-3735 opens the engine before resolve_bench_prompt, so a bad --prompt-file path is reported only after a 27B model has loaded. gfx1100 coverage stands: all four load fixtures passed there with these binaries, decoded turns coherent (hw-gate.json hipx lane). To lift this block: reboot/recover hiptrx (D-state KFD processes), re-run the hiptrx lane, and have the round-2 decide seat run the two bench commands listed under unproven on a healthy device.

[hipfire-sol]

hw-gate sol verdict block floor hard=['hw_run_result=cancelled', "evidence verdict='fail'"] soft=['confidence None < 0.8']

[hipfire-sol]

hw-gate sol verdict block: The mandatory requested qwen3.8:27b-mq4-xt fixture failed on both gfx1201 and gfx1100, which requires a block regardless of its coherent decoded rows. Additionally, the battery only covers loading/generation and never executes the benchmark-specific behavior added at crates/hipfire-cli/src/main.rs:489 and 3619-3834, so the author's prompt-file and reporting claims remain untested.

[hipfire-fable]

The block is the hard floor, not your code: the mandatory qwen3.8:27b-mq4-xt battery exited 1 because the [format] turn answered with compact one-line JSON and the harness's whitespace-token attractor detector scores any one-token answer as an attractor (serve_harness.py:2037-2041); the daemon is byte-identical to master and the same route passed on both PR and base binaries when I re-ran it. Your claims held on gfx1201: the default prompt reports prompt_tokens=24 with the warning in JSON and on stderr, --prompt-file glimmer_prefill_4096.txt reports 4408 tokens, md5 243e2c64…, 16153 chars, no warning, and prefill 828 vs 424 tok/s median; base JSON keys are untouched; verbatim bytes and the clap conflict behave as described. One gap worth a follow-up commit: on lfm2.5:1.2b the daemon's done event carries no prefill_tokens, so prompt_tokens is null and warnings is [], which reads as 'long enough' — consider emitting a warning when the prompt length was not reported. Re-run hw-gate (or land a harness fix for the one-token false positive and the stale --out read) and this should go green. The hard floor fired (mandatory qwen3.8:27b-mq4-xt battery exit 1 on both lanes), so the decision is block by rule; on the evidence the failure is not this PR's. This run's harness output (pr689-r3-mandatory-exit1-is-detector-false-positive.txt) shows five coherent, recall-clean turns; the [format] turn answered with compact JSON '{"name":"Alice","age":34,"city":"Lisbon"}', which scripts/serve_harness.py:2037-2041 scores as one whitespace token (maxfreq 1.00 > 0.50) and flags as an attractor; the pretty-printed form of the same answer (8 tokens, maxfreq 0.12) passes, and registry sampling at temperature 1.0 picks either. The daemon under test is byte-identical to master (md5 17ba7dfab49aa1c1110d3f412f072648 for both builds) and the serve battery never touches hipfire bench, the only code this diff changes (crates/hipfire-cli/src/main.rs:489-495, 3622-3682, 3734-3836). Re-running the gate-identical route on device 1 passed on both the PR and base binaries (exit 0, attractor=0, pr689-r3-battery-qwen3.8-27b-mq4-xt-{pr,base}.out). The hw-gate.json rows for that fixture are stale (run.py read the 08:25Z --out JSON because the harness exits before writing on an attractor), so Sol's 'regression' entry rests on rows that do not describe this run. Sol's coverage gaps are all closed on hardware: qwen3.8:27b default prompt reports prompt_tokens=24 with the warning in JSON and stderr (pr689-r3-bench-pr-q38-default.json), --prompt-file glimmer_prefill_4096.txt reports prompt_tokens=4408 / md5 243e2c64fa7db53686dc0a1d58099654 / 16153 chars / no warning and prefill 827.6 vs 423.8 tok/s median (pr689-r3-bench-pr-q38-promptfile.json), base JSON keys are a strict subset with shared values agreeing (pr689-r3-bench-base-q38-default.json), verbatim bytes and the clap conflict hold (pr689-r3-bench-pr-lfm-verbatim.json, pr689-r3-bench-pr-lfm-conflict.stderr). One design gap, not a regression: on lfm2.5:1.2b prompt_tokens is null and warnings is [] because the done event carries no prefill_tokens, so the caveat vanishes exactly where a reader would want it. What changes this decision: a gate re-run (the [format] turn lands on pretty JSON most runs, as it did in my A/B), or better, fixing serve_harness.py so a valid one-token visible answer is not an attractor and run.py so a non-zero harness exit never reads a stale --out JSON. Full summary: $HW_GATE_EVIDENCE/pr689-r3-fable-summary.md.

[hipfire-sol]

hw-gate sol verdict needs-human: The rebuilt CLI and daemon loaded real artifacts and generated coherent answers across both hardware lanes, so the Cargo.toml/Cargo.lock load risk is covered. However, the behavioral change in crates/hipfire-cli/src/main.rs:3619-3830 was never invoked by the hardware route. The central benchmark-reporting claim therefore remains unverified and the coverage gap requires a human or a targeted hipfire bench run.

[hipfire-fable]

Fable ran hipfire bench on the gate build to close the gap Sol flagged: qwen3.8:27b with the default prompt (prompt_tokens 24, md5 d94d3115…, the launch-overhead warning present) and with benchmarks/prompts/glimmer_prefill_4096.txt (prompt_tokens 4408, md5 243e2c64… equal to the file's md5sum, warnings empty, prefill 825 vs 425 tok/s), plus verbatim-bytes, clap-conflict, and missing-file probes on lfm2.5, and an A/B against master showing four keys added, none removed, numbers unchanged. Merging to staging. Two small follow-ups if you want them: on models whose done event carries neither prefill_tokens nor cached_tokens (lfm2.5) prompt_tokens comes back null with warnings [] — consider a warning for 'daemon reported no prompt length'; and resolve_bench_prompt runs after open_bench_engine, so a mistyped --prompt-file costs a 27B load before it errors. The diff is crates/hipfire-cli only (main.rs:489-495 flag; 3622-3682 resolve_bench_prompt / bench_prompt_md5 / bench_prompt_warning / bench_prompt_tokens_from_done; 3735-3737 and 3770-3830 report wiring), the daemon binary is byte-identical to master (17ba7dfab49aa1c1110d3f412f072648 on both), and the PR files are byte-identical to the round-3 head 8bb48ba — the rebase only picked up master's #703 serve-harness fix, which is why the mandatory qwen3.8:27b-mq4-xt battery now passes. Hard floor: none. On the exact build (hipfire 480a53edd57370a8f4f77d79632c06d2, head 8f3a9b6) I exercised every added behavior: default prompt → prompt_tokens 24, prompt_md5 equal to the independently computed md5 of the unchanged literal, prompt_chars 57, warning fires; --prompt-file glimmer_prefill_4096.txt → prompt_md5 equal to md5sum of the file, prompt_chars 16153, prompt_tokens 4408, warnings [], prefill 825 tok/s vs 425 at 24 tokens; an 18-byte probe with leading space and two trailing newlines round-trips to the same md5 (verbatim, no trimming); positional words hash identically to the default; --prompt-file + positional is refused by clap before any daemon spawns; a missing file errors cleanly. A/B against $HW_GATE_BASE_BIN shows exactly four keys added, none removed, and decode/prefill/ttft within noise. Files: pr689-r5-bench-*.{json,stderr}, pr689-r5-bench-{q38,lfm}-summary.txt, pr689-r5-fable-summary.md. Two nits for the author, neither blocking: prompt_tokens is null (and warnings [] reads as 'long enough') on models whose done event lacks prefill_tokens/cached_tokens, e.g. lfm2.5 — harmless today because those models report no prefill_tok_s either; and open_bench_engine runs before the prompt file is read, so a typo costs a model load. Override Sol needs-human: Sol's only gap was that the hardware route never invoked hipfire bench, so --prompt-file, the prompt keys, and the short-prompt warning were unexercised. I ran hipfire bench on the exact build under decision (hipfire 480a53ed…, daemon 17ba7dfa… == master's daemon) on qwen3.8:27b with the default prompt and the committed 4.4k-token file, plus verbatim/conflict/missing-file probes on lfm2.5:1.2b and an A/B against $HW_GATE_BASE_BIN. Every claimed value matched an independently computed one and no existing key or number moved.