fix(chat): borrowed figures, false gap lines and scope refusals - #476
Merged
Merged
Conversation
…rowed figures out of our terms The 2026-09-17 production evaluation found Eventus Security quoting generic pages as its own commitments. "Whats the guaranteed P1 response time in the contract" got "a documented P1 acknowledge target of 10 min", an "e.g." from the buyer guide /best-soc-as-a-service-providers-2025. A CISO checklist's MTTD and MTTR figures were called "the closest published targets", a generic article's "remediate critical findings within 48 hours" became the patch SLA, "fully air-gapped?" got "Yes" when the knowledge base only says on-premises, and a French follow-up got "we support France" with nothing about France. Every document reached the model with the same header, so it could not tell a listicle from a contract. A new page_kind module reads a crawled page's URL path: a guide, listicle, comparison, glossary, checklist, "what is" or "how to" page, or a blog post, now carries "general article, not the company's own terms" in its <<<DOCUMENT n | name>>> header. Uploaded files and pages whose path names the company are never tagged, and a knowledge hub is not a signal on its own (CleanStart publishes its SLA tiers there). credential_facts shares the token set and path parsing instead of keeping its own copy. RULE 5a now says the tag marks such a page, that SLAs, response or remediation times, guarantees and countries served are ours only when an untagged page states them, and that an example or best-practice figure is never ours, not even as a closest target. RULE 5c says a yes must match the exact capability asked. QA_PROMPT_VERSION is 7 with the new fingerprint; the minimal prompt is 2,898 tokens against its 3,000 ceiling.
…ead of the gap line The 2026-09-17 evaluation on CleanStart and Eventus Security found the canned "I don't have that detail here. Want me to loop in the team?" on turns the conversation or the knowledge base answers. Four of the five came from the canned pivot after the relevance judge rejected the turn and nothing overruled it; one was the model's own gap line: - "the third one. how exactly do u help them" after a list of four industries - "sorry not hospital, we are a bank. what changes" (both bots); it was not even rewritten against the conversation, so retrieval searched its words - "so which one am i talking to", answered by the bot's own configuration - "why should i pay, cant i just use free open source tools for this" - "whats the hr mail id" (model), with the careers address on the contact page Changes: - _refers_to_an_earlier_turn: an ordinal pointer at a listed item or a correction of the visitor's details. It makes a turn a follow-up (rewritten, judged beside the last reply, kept out of the QA cache) and, right after a bot reply, a topical follow-up whose refusal does not stand when chunks exist. - _asks_which_business_this_is: "who am i talking to", "which company is this", "is this <us> or X". Such a turn reaches generation even when the judge rejects the chunks or retrieval found none; the prompt carries the company name, bot name and description. - The comparison shape now covers paying the business against free, open source or in-house tools, and the company-facts contact shape covers an HR or careers address, so contact pages are pinned. - RULE 5a: the gap path needs the fact absent from the conversation too, and is never for a listed item, a correction or who the bot is. QA_PROMPT_VERSION 7 with its fingerprint.
…lls and material requests
Evaluation 2026-09-17 on the Eventus Security and CleanStart production bots:
"thanks thats all for now" got the scope refusal, which re-opens a
conversation the visitor just closed; "hello again" on a returning visit got
"Welcome back, Eva!" and then the refusal; a Hinglish request for a call
("bhai aaj shaam ko aapki team ke saath ek call fix ho sakta hai kya?") was
refused on both bots; CleanStart refused a request for a downloadable breach
cost report and "whats the difference between BAS and red teaming"; and
Eventus wrote a 300 word class essay on cyber security.
- The intent router gets a closing route: a whole-message goodbye ("thats all
for now", "ok bye thanks", "no thats it thank you") gets one warm line with
no question and no offer. A bare "thanks" stays the ack, and a message
ending in "?" is never a closing. A greeting followed by "again" and "i'm
back" are greetings. Closing replays no deferred question and gets no
welcome-back opener.
- The meeting gate reads the Hinglish request, but it only answers for a bot
without a scheduler. A bot with one fell through to the relevance judge,
which found no chunk about a call and refused. A bookable meeting request
now skips the judge and the empty-context refusal and reaches generation,
where the booking-card safety net attaches the card.
- Asking the business for its own material ("do you have any report ... i
can download", "can i download your ebook") is clearly on scope, so a turn
the judge rejects with chunks in hand reaches the model.
- The relevance judge scores a question about a term in the business's field
as related (gate prompt version 4).
- The answer prompt declines essays, assignments and homework even on a topic
in the business's field, and allows a brief explanation of a field term that
says whether the business offers it. QA_PROMPT_VERSION is 7 with the new
fingerprint recorded.
…sion 7 The three fixes in this batch each changed the answer prompt and each recorded its own version 7 fingerprint; this records the one for the combined prompt.
…ignals
Review findings fixed on this branch:
- page_kind: "how", "what", "best", "top" and "learn" alone no longer tag a
page. A page is a general article only with a publishing section
(blog, guide, glossary, article, insights, resources, news, webinar) or an
article slug (what-is-, how-to-, top-N-, best/top ... providers or tools,
best ... year, -vs-, -comparison, -compared, -checklist, -explained,
-tutorial). credential_facts keeps its broader word set.
- _refers_to_an_earlier_turn: "i meant" and "actually i'm/we're" need a short
noun phrase ending the clause; "X, not Y" needs the business speaking
("we are", "our company is"); not really/now/yet/bad/sure/interested and
"a bit" are never corrections.
- _ASKS_OUR_MATERIALS_RE: a report, guide, research, resources or checklist
request needs an ownership marker (your, you publish, download); only
whitepapers, ebooks, brochures, datasheets and case studies stand alone.
- Booking bypass: the judge is skipped only for a request addressed to the
business (you, your team, can we, with you, a bare request, a Hinglish
shape) or naming it, so "how do I schedule a meeting in outlook?" is judged.
A meeting request on a bot with a scheduler never reads or writes the QA
cache.
- Answer prompt: a borrowed figure points at the team only on a plan with a
team path, otherwise "we don't publish that figure here". The gap path
counts only facts the bot already gave; a visitor's claim never counts.
Fingerprint for version 7 re-recorded (not yet deployed).
- _ASKS_WHICH_BUSINESS_RE no longer matches "what company is this laptop
from"; the closing route treats "that's it?!" as a question; an echoed
general-article tag is caught by the leak check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The next three answer-quality fixes from the 2026-09-17 production evaluation of Eventus and CleanStart (reviewed after PR #474: Eventus 79%, CleanStart 75% on 163 cases).
What changes for visitors
Review
An independent review of the combined diff found over-broad signals, all fixed in the last commit: "how/what/best" alone tagged company pages as articles; loose correction and "your material" patterns let off-topic questions past the judge; "how do I schedule a meeting in outlook?" bypassed the judge; a booking turn could be cached without its card; "our exact terms come from our team" appeared on plans with no team; the gap rule counted the visitor's claims as facts. The general-article tag is also a leak marker now.
Deploy notes
QA_PROMPT_VERSION6 to 7 (answer prompt changed); the relevance-gate prompt version 3 to 4. Cached answers and gate verdicts from the old prompts are not reused.Checks
test_superadmin_impersonate_mint::test_redirect_url_uses_the_shared_app_url_setting, a known local ordering failure).Known limits
/cybersecurity/vulnerability-management/computer-security,/soc-as-a-service-ciso) have no article shape in their URL, so only the prompt rule protects them./knowledge-hub/how-enterprises-patch-containers) is not tagged, to keep company "how it works" pages untagged.