Stop skills reaching "covered" after two exchanges

[Dustymon111]

Stacked on #5 (base: feat/runtime-config) — merge that first.

Problem

A 5-skill interview was finishing in ~10 minutes: roughly 2 minutes per skill. Three causes compound.

1. A skill could jump initiated → covered in a single turn

resolve_state walked forward as many states as Flash proposed, applying the same probe_count >= 2 check at every hop:

(current_idx + 1).upto(proposed_idx) do |i|
  break unless valid_transition?(from: result, to: candidate, probe_count: probe_count)
  result = candidate
end

So when Flash proposed covered for a skill sitting at initiated, the loop passed initiated→partial (probe 2 ≥ 2 ✓) and then immediately partial→covered (probe 2 ≥ 2 ✓). Two candidate turns covered a skill. The analyzer runs once per candidate turn and caps probe_count at +1 per run, so 5 skills × 2 turns ≈ 10 minutes — exactly what was observed.

The second-order effect is worse than the arithmetic: no skill ever persisted in partial, so the coverage map never once told the interviewer "partial — you've probed but signal is still thin. Go deeper. Do NOT wrap up." Every partial instruction in the system prompt was dead code in practice.

2. covered reused the gate meant for leaving initiated

coverage.min_probes_to_advance (2) was the only depth requirement anywhere. Nothing expressed "this skill has been probed enough to be done".

Corroborating symptom: the portfolio grades high confidence at probe_count >= 3, so with every skill finishing at 2 probes, no skill could ever be rated high confidence.

3. Coverage completion ended the session regardless of the clock

all_covered? triggered wrap-up as soon as the agenda was complete. A generous analyzer could therefore close a 45-minute interview at minute 15 and leave 30 minutes unused.

Fix

1. Cap advancement at one state per analyzer run — new coverage.max_state_advance_per_turn (default 1). A skill must sit at partial for at least one turn, which is the only point at which the interviewer is told to go deeper on it. Walking forward one step (rather than rejecting a multi-step proposal outright) still prevents the "stuck state" case the original loop was written for; it just costs one extra turn.

2. Separate gate for covered — new coverage.min_probes_to_cover (default 4), clamped to never fall below min_probes_to_advance whatever the config says. CoverageAnalyzerWorker#advance_stale_partials writes covered directly without going through StateEngine, so it now applies the same floor.

3. Hold the wrap-up while time remains — new session.min_elapsed_ratio_before_wrap_up (default 0.6). While held, the coverage map carries wrap_up_blocked: true and deepen_next: <thinnest skill id>, and the system prompt instructs the interviewer to push that skill harder rather than close — looking for the ceiling of the candidate's ability, not a recap. Set to 0 to restore the old behaviour.

One trap worth flagging for review: once every skill is covered the analyzer freezes them (next if map.state == 'covered'), so probe_count stops incrementing and nothing in the coverage fingerprint can ever change again. refresh_coverage_cache returns early on an unchanged fingerprint, so without care the session would never re-evaluate and never notice the hold expiring. The hold flag is therefore part of coverage_fingerprint.

4. Tighter confidence test in the analyzer prompt — evidence must span more than one situation (not the same story retold) and the candidate must have been pushed at least once beyond their first answer. Unsure → partial.

5. System prompt — states the new depth requirement with concrete follow-up shapes, explains wrap_up_blocked / deepen_next, and adds the one exception to "never ask about a covered skill".

Verification

Turn-by-turn simulation through StateEngine (defaults: advance 2, cover 4, max advance 1):

Turn	State	Flash proposes	probe	Before	After
1	not_yet	initiated	1	initiated	initiated
2	initiated	covered	2	covered ← the bug	partial
3	partial	covered	3	—	partial
4	partial	covered	4	—	covered
—	not_yet	covered	9	covered	initiated

Wrap-up hold, 45-minute limit, all skills covered: 10m → held, 20m → held, 27m → releases. Payload while held carries wrap_up_blocked: true, deepen_next: "s1" (the skill with the lowest probe count). Fingerprint verified to change when the hold expires. Tenant override min_probes_to_cover = 6 verified: probe 4 and 5 stay partial, covered at 6.

Impact on interview length

Roughly skills × min_probes_to_cover candidate turns at ~1 minute each — 5 skills ≈ 20+ minutes at the defaults, against ~10 before.

⚠️ Check time_limit_min on short assessments. A 10-minute assessment with 5 skills cannot fit 20 turns, so no skill reaches covered and the session ends on the time ceiling instead. That is safe — no hang, transcript and portfolio still generate — but every skill will be rated low/medium confidence. For short assessments lower coverage.min_probes_to_cover for that tenant, or configure fewer skills.

All three thresholds are AppConfig settings, so the depth can be tuned per tenant at runtime without a deploy — 3 vs 4 vs 5 can be compared on staging directly.

🤖 Generated with Claude Code