feat(agent): verify-before-done + loop quality (1.9.8)

[Rippy1911]

Summary

• Why Combo-X recon/hallucinates vs ns-agent: finish was empty tool_calls with no proof gate; stuck streak counted each parallel read (fought batching); truncation could drop _repeat / dialogOpened.
• 1.9.6 loop rebuild: batch-aware observation streak (warn@3 turns / block@8), verify-before-done runtime gate (open tasks / unverified mutations / silent clicks), DEFAULT_SYSTEM VERIFY BEFORE CLAIM, critical envelope preserved on truncate, stronger dialogOpened:false miss hint.
• Detail: portfolio _memory/combo-x-agent-loop-quality.md.

Test plan

• npm test — 679 passed
• npm run build — dist manifest 1.9.6, bundle contains VERIFY BEFORE CLAIM
• Merge → reload unpacked extension/dist → dogfood a multi-step UI task (open doing-task + early “done” prose should continue)
• Healthtree meta tags: console script still recommended for the shop job; loop dogfood separate

Acceptance

Must ship	Evidence
Batch-aware stuck (parallel reads ≠ ACT NOW)	repeatGuard.test.ts
Verify-before-done continues on open tasks	loop.test.ts
Truncation keeps _repeat / dialogOpened	resultShaping.test.ts
Prompt verify-before-claim	defaultSystem.test.ts
Dist 1.9.6	local build; CI

[nextsolutions-studio]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🏅 Score: 78

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Stuck open tasks Open todo/doing/blocked tasks always produce a verify signal, and after MAX_VERIFY_NUDGES (2) the loop accepts finish anyway. A model that never calls update_task can still claim completion while tasks remain open; the gate only delays hallucination by two turns rather than requiring a task-status change or an honest blocked update. export function verifyBeforeDoneSignals(opts: { evidence: RunEvidence; openTaskTitles: string[]; }): string[] { const signals: string[] = []; if (opts.openTaskTitles.length > 0) { signals.push( `Open tasks still active (${opts.openTaskTitles.length}): ${opts.openTaskTitles .slice(0, 6) .map((t) => `"${t}"`) .join("; ")}${opts.openTaskTitles.length > 6 ? "…" : ""}`, ); } if (opts.evidence.mutationsOk > 0 && !opts.evidence.obsAfterMutation) { signals.push( "You mutated the page (click/type/navigate) but never re-read to verify the effect", ); } if (opts.evidence.silentClicks > 0 && !opts.evidence.obsAfterMutation) { signals.push( `${opts.evidence.silentClicks} click(s) reported dialogOpened:false with no follow-up re-scan — treat as miss until proven`, ); } return signals; } Silent click logic silentClicks is incremented on dialogOpened:false but the corresponding signal is only emitted when obsAfterMutation is still false. Any later successful observation (including an unrelated read) clears obsAfterMutation and drops the miss signal, so a failed modal open can be forgotten and the run allowed to finish without re-probing the control. /** Update evidence after a tool returns (call for every executed tool). */ export function noteToolEvidence(ev: RunEvidence, name: string, result: unknown): void { if (VERIFY_MUTATION_TOOLS.has(name)) { if (!toolOk(result)) return; ev.mutationsOk += 1; ev.obsAfterMutation = false; if (name === "click" || name === "click_index") { if (dialogOpenedFlag(result) === false) ev.silentClicks += 1; } return; } if (isObservationTool(name) && toolOk(result) && ev.mutationsOk > 0) { ev.obsAfterMutation = true; } } export const MAX_VERIFY_NUDGES = 2; /** * Signals that mean "finishing now would likely hallucinate completion". * Empty array → allow finish. */ export function verifyBeforeDoneSignals(opts: { evidence: RunEvidence; openTaskTitles: string[]; }): string[] { const signals: string[] = []; if (opts.openTaskTitles.length > 0) { signals.push( `Open tasks still active (${opts.openTaskTitles.length}): ${opts.openTaskTitles .slice(0, 6) .map((t) => `"${t}"`) .join("; ")}${opts.openTaskTitles.length > 6 ? "…" : ""}`, ); } if (opts.evidence.mutationsOk > 0 && !opts.evidence.obsAfterMutation) { signals.push( "You mutated the page (click/type/navigate) but never re-read to verify the effect", ); } if (opts.evidence.silentClicks > 0 && !opts.evidence.obsAfterMutation) { signals.push( `${opts.evidence.silentClicks} click(s) reported dialogOpened:false with no follow-up re-scan — treat as miss until proven`, ); } Nudge in user role The verify-before-done gate is pushed as role:"user". That can be treated as a new user instruction, pollute session history/memory, and blur the DEFAULT_SYSTEM rule that the gate is not the user. Prefer an internal/system (or tool) message so runtime gates stay distinct from real user turns. if (signals.length > 0) { runCtx.evidence.verifyNudges += 1; messages.push({ role: "assistant", content: finalText }); emit({ type: "assistant_delta", message: finalText }); void logUsage({ kind: "message", role: "assistant" }); const nudge = buildVerifyNudge(signals); messages.push({ role: "user", content: nudge }); emit({ type: "status", message: `Verify-before-done gate (${runCtx.evidence.verifyNudges}/${MAX_VERIFY_NUDGES}): continuing — ${signals[0]}`, }); continue; Premature batch block check() blocks when observationBatches >= 8 before the current batch is finalized, so the first tool of the 9th observation turn is refused and the streak resets. Remaining parallel reads in that same model turn then run under a reset counter and never get the stuck annotation, weakening the batch-aware guard under multi-read turns. check(name: string, args: Record<string, unknown>): RepeatVerdict { if (!OBSERVATION_TOOLS.has(name)) { this.observationBatches = 0; this.waitInStreak = false; } else if (this.observationBatches >= RepeatGuard.STUCK_BLOCK_AT && !this.waitInStreak) { // The refusal IS the intervention — reset so recovery reads (list_tabs to // find a lost tab, a scoped re-read) are not themselves refused next. const observations = this.observationBatches; this.observationBatches = 0; this.waitInStreak = false; return { kind: "block", repeats: observations, result: { ok: false, error: "stuck_loop_blocked", observations, hint: `Refused: ${observations} consecutive read-only turns with no click/type/navigation. ` + `The page does not change by reading it again. Mutate (click_index/type_index), map the form with ` + `list_form_fields, cut noise with within/excludeSelector, or report BLOCKED with what you tried. ` + `Wrong tab? list_tabs then activate_tab or navigate back — this refusal reset the streak.`,

[nextsolutions-studio]

Suggestion: check still resets observationBatches on every non-observation tool before the batch finishes, so a turn that mixes a read with a later click clears the streak mid-batch and can prevent the hard block from ever firing. Only finalizeObservationBatch should mutate batch counters; keep check limited to the identical-call fingerprint path plus the block ceiling test. [possible issue, importance: 8]

Suggested change

	check(name: string, args: Record<string, unknown>): RepeatVerdict {
	if (!OBSERVATION_TOOLS.has(name)) {
	this.observationStreak = 0;
	this.observationBatches = 0;
	this.waitInStreak = false;
	} else if (this.observationStreak >= RepeatGuard.STUCK_BLOCK_AT && !this.waitInStreak) {
	} else if (this.observationBatches >= RepeatGuard.STUCK_BLOCK_AT && !this.waitInStreak) {
	check(name: string, args: Record<string, unknown>): RepeatVerdict {
	if (OBSERVATION_TOOLS.has(name) && this.observationBatches >= RepeatGuard.STUCK_BLOCK_AT && !this.waitInStreak) {

[Rippy1911]

1.9.7 follow-up: removed hard stuck_loop_blocked (field: mid-paging get_page refuse). Soft _repeat ACT NOW kept. Worker default → DeepSeek Flash; Settings clarifies worker is parse-only (orch runs all browser tools). Reload dist 1.9.7 after merge.

[Rippy1911]

Addressed pr-agent / ns-agent review (1.9.8)

Finding	Action
Open tasks survive after MAX_VERIFY_NUDGES	Fixed — exhausted gate auto-blockeds doing/todo + appends UNVERIFIED closeout
Silent click cleared by unrelated read	Fixed — unresolvedSilentClicks only clears on later dialogOpened:true
Nudge as role:user	Fixed — injected as role:system + “(NOT the user)” banner
Premature stuck_loop_blocked batch block	Already removed in 1.9.7 (operator field: mid-paging refuse) — soft ACT NOW only

CI + merge next. Operator deploy = reload unpacked extension/dist 1.9.8. Agentic meta-tags prompt: _memory/healthtree-pl-meta-tags-combo-prompt.md v5 (no console/page-ext batch).