Answer quality: leaner prompt, consent-only handoff, company facts, credential checks - #474
Merged
Merged
Conversation
An answer that offered the team ("I can connect you with them if you want")
set suggest_handoff from the model's own words, and the widget opened the
form with nobody saying yes (CleanStart x-tech-mttd-sla, 2026-09-17 eval).
The offer now only waits for the visitor: it is counted as
handoff_offer_awaiting_consent and kept out of the QA cache, and a bare
"yes" to it opens the form through the existing affirmation path.
After the support reply, a new question such as "i need the escalation
matrix now" got the repeat form line. The repeat is now kept for a visitor
who asks for a person or reports a problem again; any other message is
answered while the form stays in the chat (support_repeat_answered).
A bare "yes", "sure" or "ok" to an offer with options (the greeting's
"Want to hear about our services, see recent work, or chat with the team?")
got "Got it." It now continues with the first option, so the greeting and
the name welcome answer "what services do you offer". A team-first offer is
still a handoff, and "ok" after an ordinary answer is still an ack.
On a managed SOC's bot (about 7,900 chunks) "where are the soc centers ?" was refused as off-topic and "list soc centers" was answered with the services list. The SOC cities are on the contact page, but hybrid search and rerank never put that chunk in the top 15. A question about the company's locations, contact details or people is now detected by a linear prefilter and counts as clearly on scope. On the retrieval path it also reads up to four chunks of the bot's own contact, locations, about and team pages (chosen by URL path) and puts them ahead of the retrieved chunks, after the rerank, so the relevance judge sees them. The list keeps its 15-result length, CAG-lite is left alone, and each pin emits company_facts_pinned.
From the 2026-09-17 evaluation, on-scope turns that still got the scope line: - "whats the margin for channel partners" after a partner program answer: partnership words now count as on scope. - "what certifications do you have": the assurance check now also reads the term before "do you", and knows accreditations, awards and empanelment. - "paid or unpaid? and is remote ok" and "d'accord, et c'est disponible en France ?" after an answer: retrieval searched the fragment's own words, so no chunk bore on it. The judge is now told that a follow-up about the business's offering scores 0.5 when the previous reply is about that subject, and a rejected fragment shaped as a continuation gets the no-info pivot instead of the scope line.
…ration
The 2026-09-17 evaluation found the model stating certifications and reports
as the company's own when the knowledge base only had guides, mapping pages
or services ("Our SOC 2 Type 2 report is typically shared under NDA").
credential_facts adds a linear prefilter for questions about the company's
own certifications, a fenced gate-tier check over the credential sentences
of the retrieved chunks (HELD with a verified quote, OFFERED, NOT_FOUND),
precision-first fallback rules and a self-contained CREDENTIAL FACTS block.
The test suite defaults the check's model to down, like the price and
support classifiers.
A credential question starts the check once the chunks are final, runs it alongside the relevance gate and puts the block after the reference information. The question neither reads nor writes the QA cache, so an answer cached before the check is never replayed past it. A non-English turn skips the check like the other English-tuned judges. The metric credential_facts_checked carries the verdict counts.
The production review of 2026-09-17 (240 conversations on two live bots) traced most answer defects to the prompt itself: five blocks each claimed to override everything, the SERVICES list made a bot deny services its website sells, length rules starved follow-ups, the gap path fired on facts that were present, and team offers read like a handoff already under way. Prompt (system 8,027 and 8,514 tokens down to 3,665 and 4,023 on the two bots): - One PRIORITY ORDER near the top replaces every "HIGHEST PRIORITY", "OVERRIDES" and "wins over every other instruction" claim. - FEATURED SERVICES is explicitly not exhaustive; a denial needs both the list and the reference material silent; location, SOC centre, office and team questions are not services questions. - RULE 1 LENGTH AND DEPTH: short facts stay short, lists are complete, "tell me more" and ordinal follow-ups resolve the item from history and give at least three facts, never only a link. - RULE 5a gap path only for an absent fact; "sits with our team" retired; reports and office or SOC locations are own claims; listicles, templates, comparison articles and provider-choice pages describe the topic. - TEAM OFFERS: a question the visitor can accept; no form announced unless the visitor asked for a person. Example offers are recognised by intent_service.bot_offers_handoff and not by _HANDOFF_RESPONSE_RE. - SCOPE: small talk merged in, FRUSTRATION gets an acknowledgement and a next step, general coding help is out of scope, comparisons stay in. - ABOUT replaces COMPANY CONTEXT and says the summary may be third person and ranks below the reference material. - Dates said once (RULE 6); RULE 0, 2b, 6a to 6c, 7 to 11 and the style block's duplicates removed; examples that used banned openers or invented roles removed; no dashes in the per-turn qualification text. - New per-turn line for a mixed pricing turn (gate outcome escalate_deferred): answer the rest, state no figure. Wired through build_hybrid_prompt(pricing_mixed=...). - The prompt leak detector keys on the new PRIORITY ORDER heading. QA_PROMPT_VERSION 5 -> 6 with a new fingerprint; 5 is kept. Tests changed on purpose: - test_answer_prompt_structure.py (new): priority order, length rule, gap rule, team offers against both regexes, featured services, scope, frustration, ABOUT, mixed pricing line and its wiring, dashes, invented roles, leak sentinel. - test_answer_prompt_claims_capabilities_comparisons.py: SCOPE anchor renamed; the services test pinned "say plainly that we do not offer it", the behaviour that produced the false denials, and now pins the both-sources condition instead. - test_brand_tone_guarded.py, test_rag_prompt_hardening.py: SCOPE anchor renamed. The "SCOPE line says above and below" check became a check that the priority order ranks grounding and scope above business instructions. - test_em_dash_prompt_rules.py: RULE 11 duplicated the style block's rule and was removed, so the check reads the assembled prompt (rule present once) instead of the builder source. - test_prompt_has_one_owner_per_rule.py: "1-3 sentences" became "1 to 3 sentences", stated once; the style block's "fewest words" is gone. - test_prompt_budget.py: ceilings 7,300/7,900/8,300/8,800 lowered to 3,000/3,350/3,750/4,050; media delta bound 900 lowered to 500. - test_qa_cache_prompt_version.py: fingerprint for version 6.
Eval 2026-09-17 (defect 6): "this bot is absolute trash" got "Sorry to hear that." and nothing else, "stupid bot" got a scope refusal, and a visitor asking which sleeping tablets to take got the off-topic redirect. The router now reads insults aimed at the bot as frustration, and its frustration and abuse replies acknowledge the feeling in one clause, ask for one thing at a time and offer the team when the plan has one. A reply right after a reaction reply uses new words and no second apology, in the router and in the dissatisfied reply. Praise sent with an annoyed face is left to the dissatisfaction check instead of being thanked. Self-harm and medical emergency signals get care and a pointer to the local emergency number or a crisis helpline, with no sales route, before moderation and before the name request. A request for medication gets a suggestion to see a doctor or pharmacist.
…ry reply "do u remember me?" missed the remember route, reached the relevance gate and opened the unhelped handoff form in production. The route now reads the "u" and "ya" forms, a bare "remember me" and common misspellings, as the whole message only, so a question about a "remember me" login still reaches retrieval.
… the team'
The canned no-info reply for an on-scope question the knowledge base cannot
answer opened with "That specific detail sits with the team", the evasive line
the answer prompt now forbids. It now says "I don't have that detail here."
On a paid plan it ends with a question the visitor can accept ("Want me to
loop in the team on this?"), which bot_offers_handoff reads as an offer, so a
following yes opens the form; Free plans keep the contact link or the
something-else line and never offer the team. The widget's own fallback
pattern does not match the new wording, so it does not open the form by itself.
…pped A message that asks the price and something else lost the something else: the price guard replaced the whole answer with the escalation. On a MIXED price decision a sentence redactor now drops only the sentences that state a figure (whole leads and tables where a drop would leave them dangling), and the escalation follows the rest. Stream and whole-answer outcomes stay equal; PRICE turns are unchanged.
…urns together
A Hinglish request for time ("kal team se call pe baat ho sakti hai
kya", "mujhe demo chahiye", "meeting set karo") was not read as a
meeting request, so the handoff reply opened the form on bots with a
scheduler. meeting_gate now reads those shapes, refusals and things
("call nahi chahiye", "call center") excluded, and the stream attaches
the booking card when the model leaves it out.
"send the datasheet and book a demo" kept only one of the two. The
files are now picked before the meeting gate, and the document reply
carries the booking card, or the meeting pivot when no scheduler is
configured.
A conversation held in another language keeps its model: the meeting pivot and the handoff skip the Hinglish shapes lead to are English.
…y the bot's own files
- media_cards: a card's identity is its normalised URL and, for a name
that says what the file is about, its normalised title. The document
pick, the secondary chip and the per-session dedupe use it, so one
case study stored at two URLs is offered once.
- A document request that points back ("is there a pdf of this i can
share with my boss") reuses the card the last reply carried, or the
topic of the visitor's previous message.
- The media catalog, the whitelist, the topical card and the chip admit
a file only on the bot's own registrable domains (website and allowed
domains) or a website builder or storage host. A bot with no website
is not filtered. Videos are unchanged.
…n accept The LIVE SUPPORT block told the model to say a team member would be with the visitor shortly. Model text no longer opens the handoff form and that sentence is no question, so the visitor never said yes and no form opened. The block now asks "Would you like to speak with our team now?", and the offline block ends with "Want me to take a message for our team?". Both examples are read by bot_offers_handoff and not by _HANDOFF_RESPONSE_RE. The v6 prompt fingerprint is re-recorded; v6 has not been deployed.
…offers offered_option_question turned actions into "tell me about your book an appointment" and split a listed topic at its commas. It now continues only when the first option is a topic (a stripped verb such as hear or see, or an opening such as interested in or exploring), keeps an "A, B and C" list whole, and returns None for an action so the handoff check decides. bot_offers_handoff now reads question offers to talk with, discuss something with, or arrange a call with our team, so a following sure is a handoff.
…sitives The crisis reply lived in the intent router, behind the urgent route, the support route and the name step. "my name is Sam, i want to kill myself" saved the name and answered the deferred question; "i want to kill myself, my account got hacked" and "i want to die, you charged me twice" alerted the team with no care. rag_pipeline_stream now checks the visitor's own message right after saving it and sends the router's crisis reply (repeat wording included) with no classifier, moderation, model call or alert. Healthcare and retail questions no longer get the crisis reply: breathing "through", "in" or "with" something, "a stroke of" and chest pain beside a booking or service question with nothing acute. That last one is answered under a short emergency line and skips the QA cache. A dosage question that says "your" or names the business reaches retrieval.
…he rule log-only Wix, GoDaddy, WordPress.com, Dropbox, DigitalOcean Spaces and Firebase file hosts and a same-name domain on another TLD now count as the bot's own. A file the rule rejects is counted as media_file_offsite and logged with its host, and still offered while ENFORCE_OWNED_FILES is False.
…ice requests Gift, completion, participation, SSL/TLS and insurance certificates are not the business's credentials, and supporting, selling or offering a compliant product is a request for the product. The check's deadline drops to 2.5 s with the model call at 2 s.
…pany's facts A bare phone, whatsapp or owners counts only at the end of a request, a head count needs the company as its object, and work, deliver, ship and available count as a location question only with a country, city, region or market.
…person A talk with the team asks for time only with a medium (call pe, demo ke liye) or a time word (kal, shaam 5 baje), so a visitor who wants a person is no longer handed a calendar.
jadhavgaurav
added a commit
that referenced
this pull request
Sep 17, 2026
…ly the search rewrite
With history, the meeting check read the search rewrite, which can come back as
keywords ('datasheet and demo booking next week') that name no request. The
production smoke test after PR #474 sent 'send me your datasheet and also book a
demo for next week' on Eventus and CleanStart and got the file cards without the
booking card. The raw message now counts as well as the rewrite.
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answer-quality batch from the 2026-09-17 production evaluation (reviewed pass rate before this PR: Eventus 80%, CleanStart 77%) and the same-day review of the answer system prompt.
What changes for visitors
QA_PROMPT_VERSION5 to 6). About 3,700 to 4,000 tokens, down from 8,000 to 8,500. One priority order instead of five competing "highest priority" blocks; one length and depth rule ("tell me more" gets real detail, ordinals resolve from history, never a bare link); the gap line is used only when a fact is truly missing; first-person claims (certifications, reports, SLAs, terms, locations) only from the company's own pages; team offers are questions the visitor can accept; featured services are not exhaustive, so a bot no longer denies services it sells; frustration gets a next step; general coding help is out of scope; mixed pricing turns answer the non-price part.media_file_offsite) but not dropped yet.Review
An independent review of the combined diff found crisis-route ordering, a live-support promise the code no longer backed, garbled option continuations, file-host false negatives, credential and company-facts false positives and a Hinglish misread. All are fixed in the last commits.
Deploy notes
company_facts_pinned,credential_facts_checked,media_file_offsite,handoff_offer_awaiting_consent,affirmed_offered_option,support_repeat_answered, and the redactor and card metrics.Checks
test_superadmin_impersonate_mint::test_redirect_url_uses_the_shared_app_url_setting, which fails locally on the base commit too).Follow-ups (not in this PR)