fix(chat): borrowed commitment figures, phone-picker countries, contact addresses and field-question refusals - #477
Merged
Merged
Conversation
…eference context Eventus Security, 2026-09-17: "d'accord, et c'est disponible en France ?" was answered "We serve France." The only France on the company's pages is the phone country-code picker of its contact and talk-to-sales forms, crawled as about 250 lines like "* France+33". CleanStart, 2026-09-17: "want to apply for a devops role, whats the hr mail id" got the message form. The careers address is on every page only as a link with a truncated label, "[careers@](mailto:careers@cleanstart.com)". The reference context now runs each chunk through tidy_reference_text before the length cap: a run of three or more country-code lines is dropped (a single address line or a full phone number stays), and a mailto or tel link shows the full address, as "label (address)" when the label is a real label. Applied at prompt-build time, so no re-crawl is needed. Knowledge base check: Eventus, 14 picker runs (603 lines) removed in 14 chunks, no other line removed; CleanStart, no runs, 609 links rewritten (551 careers@).
CleanStart, 2026-09-17: "want to apply for a devops role, whats the hr mail id" got "I'll open a quick message form for you." while the careers address was on the pinned contact page. LEAVE A MESSAGE now says to give an email or phone the reference material holds for what was asked (HR, careers, support, sales) and not open the form, and its example no longer asks for a support address. Eventus Security, 2026-09-17: "We serve France." came from a contact form's phone-code list. RULE 5c now says a country in a phone-code list or a form is not one we serve. The prompt stays within its budget ceilings (two lines were shortened). QA_PROMPT_VERSION is 8 with the new fingerprint recorded.
… replies Production, 2026-09-17, Eventus Security bot: "whats ur SLA for patching critical CVEs? like in how many hours" got "We remediate critical-severity findings within 48 hours. Want me to take a message for our team?" The only source was a general best-practices page (/cybersecurity/vulnerability-management/computer-security/) listing "Alert Triage SLAs: Remediate critical-severity findings within 48 hours". Its URL has no article shape, so the general-article tag missed it, and the prompt rule did not stop the model. A new deterministic guard (commitment_guard.py, no model call) reads the finished answer. A sentence that speaks for the company (we, our, us, the company name) and states a commitment figure (a bounded or SLA-worded duration, or an uptime percentage) keeps it only when a retrieved chunk that is not a general article states the same figure as the company's own: an uploaded file, a terms page by path (sla, terms, pricing, tiers, service-level), or a sentence with a first-person or company-name subject. Text the owner wrote for the bot also counts. Otherwise the first such sentence becomes "I don't have our exact figure for that." and later ones are dropped; the rest of the answer stays. rag_pipeline_stream runs it after the media card step and carries the corrected text through answer_override, the saved message and the QA cache, so the widget and POST /chat match the transcript. Metric: commitment_figure_redacted. Checked against the 2026-09-17 knowledge base exports: every figure on CleanStart's two SLA pages (190 of 190) counts as supported, Eventus 48 hours is supported nowhere in its 7,917 chunks, and across 1,034 evaluated replies only one changes (Eventus "documented P1 acknowledge target of <=10 min").
…business's field Production, 2026-09-17, CleanStart (hardened container images and software supply chain security), after the name step: "whats the difference between BAS and red teaming" got "Let's keep this about CleanStart. Would it help to hear about our work and our services, or should I connect you with the team?", logged as off_topic_refusal reason=gate_fired gate_score=0.00. The judge prompt (gate version 4) was applied: refusals are never cached and CAG-lite and retrieval share one prompt. But the judge is never told what the business does. It infers the field from the chunks, and with no BAS or red teaming material the retrieved chunks were weakly related build notes, so the field-term rule had no field to match and the judge scored 0.00. A turn the judge is about to refuse with chunks in hand now asks the gate-tier model one question (field_question): is this about a concept, term, practice or technology in the field this business works in? It sees the company name, description and featured services the answer prompt uses, with the business and the message fenced as data. Temperature 0, one attempt under 2s, a worker thread under a 2.5s deadline. On YES the turn reaches generation, where RULE 5c explains briefly and says whether the business offers it, and it counts as a relaxed turn. A failure, a stall, a bot with no description or services, and an empty retrieval keep the refusal. Essays, homework, how-to help with other tools and other industries are NO in the prompt. Ordinary turns cost nothing. The answer prompt and the gate prompt are unchanged, so neither version moves.
… guard
Review findings on this branch, each with a failing test first:
- commitment_guard: a figure on a crawled page that is not a general article
now counts as the company's own unless its sentence or bullet reads as
advice or an example (e.g., for example, should, must, look for, best
practice, typically, aim for, demand, a from/to change, or a bare imperative
at the start or after a colon). CleanStart's roadmap, why-cleanstart and
release-notes figures and the Eventus ransomware page are kept; the
production best-practice list and the buyer-guide example stay redacted.
Refund and cancellation pages count as terms pages; "48h" is a duration.
- commitment_guard: the visitor's own timeline is skipped, a duration needs a
commitment word in its clause or a firm bound ("in 3 days" alone is not
one), a percentage needs an uptime or SLA word within three words, only the
unsupported clause of a sentence joined by ";", ", and" or ", but" is
replaced, and durations are compared in minutes (14 days is 2 weeks).
- commitment_guard: no sentence split after e.g., i.e., approx., vs., Mr.,
Dr., "etc." before a lowercase word or "No." before a number; the
Devanagari danda ends a sentence.
- rag_service: the leave-message safety net no longer forces the form when
the reply gives an email address or a phone number.
- field_question: the prompt says NO to step-by-step attack or harm
instructions, and a message under three words or one the intent router
answers is not sent to the classifier.
- cleaner: a phone-code run is dropped only when every line names a country
or territory (kb_quality.starts_with_country_name) and the run has at
least five lines, so results lists and short office lists are kept.
jadhavgaurav
added a commit
that referenced
this pull request
Sep 17, 2026
… out of replies Production, 2026-09-17 14:25 UTC, Eventus Security, after "tell me about managed SOC and cybersecurity services": "d'accord, et c'est disponible en France ?" got "Yes, France is listed among the countries we serve." The phone-code picker was already dropped (PR #477), but /cybersecurity/security-operations-market/ still carries a plain country dropdown, one name per line. The other France mentions are an MSSP listicle and incident write-ups. - cleaner: strip_country_name_runs drops runs of ten or more lines that are each only a country or territory name (kb_quality.is_country_name: the whole line, optional bullet, bracketed qualifiers and "Macao S.A.R., China" style options allowed). tidy_reference_text runs it after the phone-code rule. On the 2026-09-17 exports it drops 5 runs (318 lines, 5 chunks) on Eventus and nothing on CleanStart; no other run of name-only lines is longer than 3. - commitment_guard: redact_unsupported_country_claims checks a reply sentence where the company speaks, a coverage word appears (serve, cover, operate, available, presence, customers, clients, offices, based, countries) and a country is named, with no denial and not as the visitor's own place. Each country needs a mention in a chunk that is not a general article, an event page or a page about other companies (providers, companies, jobs), within 120 characters of the company speaking, or with a serve, presence or office word on its line or an office word under a bare country heading, and not in a list of three or more country names in a row. An uploaded file may support a listed country; text the owner wrote supports any. The first unsupported sentence becomes "I don't have a statement about serving <Country> here.", later ones go, the rest of the answer stays. - rag_service runs it right after the commitment guard, through the same answer_override path. Metric: country_claim_redacted. Replay over the 1,032 reviewed bot replies: 17 carry a checked claim, all office claims (Eventus: India, UAE, Saudi Arabia, Qatar, US, Singapore; CleanStart: US, Singapore, India, UAE). None changes with each bot's whole knowledge base, nor with 15 random chunks plus the page naming the country. The production France answer is redacted with the whole Eventus knowledge base and in 25 of 25 sampled trials, one set per page naming France.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four answers still wrong after PR #476, from the live check on Eventus and CleanStart (2026-09-17).
What changes for visitors
commitment_guard.py, no model call) checks each reply sentence that states a response, remediation, onboarding or uptime figure as the company's own. The figure stays when a company page states it (a terms, SLA or refund page, a sentence with the company as subject, or any non-article page where the line is not advice or an example); otherwise the sentence or clause becomes "I don't have our exact figure for that." The corrected text is sent asanswer_override(the live widget applies it), saved and cached. Metric:commitment_figure_redacted.[careers@](mailto:careers@cleanstart.com)reads ascareers@cleanstart.com; 609 links on CleanStart). The prompt gives an email or phone from the reference for what was asked instead of opening the message form, and the leave-message safety net no longer forces the form on a reply that contains an address or phone number.field_question.py) with the business description and services decides whether the question is about the business's field; YES reaches the answer step. It says NO to essays, homework, other tools, other industries and step-by-step attack instructions, and it is skipped for messages under three words and turns the router answers.Review
An independent review found that the first version of the guard redacted figures that company pages state without a subject (CleanStart "Critical vulnerabilities are patched within 24 hours", Eventus "Response from IR expert within 10 min"), plus false positives (a visitor's own timeline, event dates, unrelated percentages), sentence splitting after "e.g." and the Hindi danda, the form forced after an address, and a cleaner rule that could drop short lists of "+N" lines. All fixed in the last commit. Replay over 1,032 evaluated replies: 0 redactions against each bot's whole knowledge base, 0 to 2 with 15 sampled pages.
Deploy notes
QA_PROMPT_VERSION7 to 8.Checks
test_superadmin_impersonate_mint::test_redirect_url_uses_the_shared_app_url_setting, a known local ordering failure).