feat(suggestions): hybrid retrieval-grounded follow-ups behind feature flag

[yashgoel1331]

Summary

• Adds a hybrid suggestions pipeline that can ground follow-up questions in distilled search_documents retrieval evidence plus recent conversation context.
• Gates rollout behind SUGGESTIONS_HYBRID_ENABLED (default false) so production stays on conversation-only baseline until explicitly enabled.
• Adds observability for suggestions timing and input-mode selection without changing chat streaming behavior.

This PR contains exactly 2 commits cherry-picked onto current main:

1. 02f46a8 — suggestions scope guardrail (non-milk transaction limits)
2. 4ebd1d6 — hybrid suggestions pipeline + feature flag + docs

What changed

• Retrieval evidence extraction (app/services/chat.py): after each chat turn, distill current-turn search_documents tool returns into bounded snippets and cache as suggestions_shadow_{session_id}_{target_lang}.
• Hybrid input + quality gate (app/tasks/suggestions.py): when flag is enabled and retrieval passes deterministic checks, suggestions use last 3 conversation pairs + distilled evidence; otherwise fallback to last 5 pairs (conversation-only).
• Prompt guardrails (assets/prompts/suggestions_system.md): hybrid input usage rules, answerability constraints, and milk/cooperative scope limits.
• Feature flag (app/config.py, example.env): SUGGESTIONS_HYBRID_ENABLED.
• Instrumentation logs: suggestions_task_queued, chat_stream_complete, suggestions_shadow_evidence, suggestions_task_started, suggestions_input_mode, suggestions_cache_written.
• Docs: updated API_DOC.md and docs/CHAT_ENDPOINT_FE_INTEGRATION.md with current /api/suggest/ behavior and pipeline details.

Design notes

• Chat stream path is unchanged; suggestions run as FastAPI background tasks after streaming completes.
• Suggestions agent remains tool-free.
• Hybrid mode requires usable retrieval (min snippets/chars/overlap); weak/no retrieval automatically falls back.

Test plan

• With SUGGESTIONS_HYBRID_ENABLED=false, verify suggestions_input_mode ... mode=conversation_only retrieval_reason=hybrid_feature_flag_disabled.
• With SUGGESTIONS_HYBRID_ENABLED=true and a retrieval-heavy ag query, verify mode=hybrid retrieval_reason=ok.
• With hybrid enabled but no/weak retrieval, verify fallback to conversation_only with explicit retrieval_reason.
• Confirm chat streaming latency unchanged.
• Confirm /api/suggest/ returns JSON string array after generation.
• Spot-check suggestions avoid unsupported personal balance/passbook lookup prompts.

Made with Cursor

[KDwevedi]

Solid on the things that matter most — error isolation is good (task queueing try/except'd in the generator tail, broad except → [] in the task, pending-key cleared in finally), chat latency is untouched (all new stream-path work is after the last yield), and the flag default is off. Branch is 23 commits behind main but test-merges clean. Two things block it as-is:

1. Major — the flag doesn't gate all new behavior. Commit 2's system-prompt changes (assets/prompts/suggestions_system.md: answerability guardrails, language-switch ban, hybrid-input section) and the per-turn shadow-evidence extraction + Redis SET (app/services/chat.py:1084-1109) run unconditionally. With SUGGESTIONS_HYBRID_ENABLED=false, prod is not the pre-PR conversation-only baseline, so flipping the flag off doesn't restore old behavior and any on/off comparison is confounded. Please gate the prompt delta and the shadow writes behind the same flag.
2. Major — stale vs the micro-loan feature. suggestions_system.md:22-24 ("do not generate questions about bank accounts or non-milk financial transactions") predates the micro-loan flow that merged to main since this branched (4a68746, 8e41557, 8e52db9) — bank/loan questions are now a supported product flow, and this guardrail would suppress follow-ups for it. Needs rebase-time reconciliation with product.

Also worth fixing:

• No turn identifier in the shadow payload (write chat.py:1094, read app/tasks/suggestions.py:149-151): if the generator tail is skipped (client disconnect mid-stream), the background task still runs and can ground suggestions in the previous turn's 10-min-TTL evidence. A per-request nonce passed to the task makes the gate turn-exact.
• _extract_query_from_search_header / _extract_snippets_from_search_return (chat.py:180-190) are string-coupled to the exact output formatting of agents/tools/search.py:527,545 — any format change silently yields 0 snippets and permanently degrades hybrid to fallback with no alert. Fail-safe but silent; at least log a counter.
• suggestions.py:149-156 does the Redis GET + gate computation before the flag check — check hybrid_enabled first so flag-off costs nothing.
• Snippets hard-truncate mid-sentence at 600 chars and drop the document name from evidence; and the hybrid path passes any target_lang (:172) while the new prompt declares only English/Gujarati supported.