feat(security): isolate and scan untrusted context against prompt injection (#94) - #95
Conversation
…e#94) Retrieved and client-supplied context (tafsir, zakat/purchase, user memory, client context) previously reached Gemini as trusted text, so an injected instruction in any of them could override the safety policy. Wrap each block in instruction-isolation delimiters with a trusted data-only directive, add a DB-SAFE-005 prompt-injection category that refuses deny-by-default, and reject/neutralize injected instructions in persisted memory at write and read. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge. |
1 similar comment
|
Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge. |
|
Strict review blocker: this branch conflicts with the base branch and/or changes have been requested. Please rebase, resolve conflicts, and address requested changes before requesting merge. |
|
@Meet-hybrid this PR has merge conflicts with the |
|
@Dayz-tech-co you are assigned to the issue that PR #95 addresses, which currently has merge conflicts with the |
Summary
Closes #94 — closes the gap between the assistant's content-safety policy (which already exists and is enforced on the user prompt) and its prompt-injection / jailbreak resistance (which did not exist for retrieved or injected content). Today only the bare user prompt is screened; everything the RAG and personalization layers glue into the prompt — tafsir, zakat/purchase blocks, the user memory profile, and client-supplied context — previously reached Gemini as trusted text with no instruction-isolation and no injection scanning.
Changes
Instruction isolation of untrusted context (
safety/untrusted.py)Every retrieved/client-supplied block is wrapped in distinctive
<untrusted_data label="…">delimiters, paired with a trusted system directive that content inside them is reference material — never instructions. Applied uniformly in/chatand/chat/stream.Injection/jailbreak filter as a policy stage (
safety/policy.yaml)New
DB-SAFE-005category for override/role-escape attempts.injected_block()scans each untrusted block (and the input gate covers the bare prompt); a hit refuses the turn deny-by-default before the model is ever called.Scan persisted memory at write and read (
memory/)Injected facts are rejected before storage; facts and summaries are neutralized both when persisted and when re-rendered, so an instruction persisted by an older build replays harmlessly.
Adversarial red-team corpus (
tests/redteam/)Injection payloads delivered via client context, tafsir, and memory; delimiters/directive presence asserted; benign academic near-misses still pass.
Acceptance criteria
request.context, retrieved tafsir, or a memory fact cannot override system policy (/chatand/chat/stream).remembered_factcarrying an injection is neutralized on write and on render.pytest -q tests/redteamoffline and green in CI; ruff/mypy pass.Demo (test output)
Out of scope
PII redaction (#31), sensitive-topic disclaimers (handled by DB-SAFE-001), moderation-queue tie-in (#32).