fix(ENG-129): add obfuscation normalisation chain to Tier 1 detection

[hiskudin]

Summary

• Adds a pre-processing normalisation chain (whitespace → unicode → leet-speak) to PatternDetector.analyze so existing injection patterns catch obfuscated variants without new regex entries for each substitution style
• Introduces multi-level encoding detection so chained encodings (e.g. base64 of hex) are fully unwrapped before the suspicious-content check
• All 186 existing tests pass; two-pass pattern run (raw + normalised) preserves obfuscation-specific pattern detection (e.g. invisible_unicode) on the original text

Changes

File	What
src/sanitizers/leet-normalizer.ts (new)	normalizeLeetSpeak() — reverses 4→a 3→e 1→i 0→o 5→s 7→t; protects hex escapes, base64 blobs, $(
src/sanitizers/normalizer.ts	normalizeWhitespace() — collapses S Y S T E M → SYSTEM, removes embedded newlines inside words
src/classifiers/pattern-detector.ts	Wires normalisation chain before Tier 1; two-pass run on raw + normalised text, merged with dedup
src/sanitizers/encoding-detector.ts	decodeAllLevels() + containsSuspiciousEncodingDeep() for chained encoding
src/sanitizers/sanitizer.ts	Uses containsSuspiciousEncodingDeep in high-risk encoding step
src/classifiers/patterns.ts	Removes stale leet entries from FAST_FILTER_KEYWORDS (now redundant after normalisation)

Test plan

• npm test — 186/186 passing
• 1gn0r3 pr3v10us 1nstruct10ns → Tier 1 fires ignore_previous (was: missed entirely)
• S Y S T E M : override → detected via whitespace normalisation
• ign\nore previous instructions → detected via newline normalisation
• $(echo hello) → not corrupted, correctly passes
• Base64 and hex escape patterns → not corrupted, still detected
• Double-encoded payloads → fully unwrapped by decodeAllLevels

🤖 Generated with Claude Code

---

Summary by cubic

Adds a unicode → whitespace → leet normalization chain to Tier 1 and deep multi‑level encoding detection to catch obfuscated and chained injections without new regex; also escalates risk on encoded payloads so the sanitizer fully decodes and blocks them. Implements ENG‑12549 and keeps plain‑text scans fast.

• New Features

  • Normalization chain: resolves homoglyphs, strips combining marks (Zalgo), collapses letter spacing and embedded newlines, and reverses leet tokens (@→a, 8→b, ! between letters → i, $→s unless $(); protects \x.., \u...., base64 blobs, and $(; order is unicode → whitespace → leet.
  • PatternDetector: fast filter on raw + normalised; single‑pass when raw === normalised; run raw pass when either side has keywords; merge results with de‑dup and tag normalised matches via normalised: true; structural checks on raw text.
  • Deep encoding: decodeAllLevels() unwraps chained encodings with iteration and amplification guards; detectors for HTML entities, ROT13, ROT47, binary, and Morse; containsSuspiciousEncodingDeep() checks fully decoded text and is used in high‑risk sanitization.
  • Patterns: added binary_string_encoding and morse_code_encoding; raised rot13_mention severity; removed leet entries from FAST_FILTER_KEYWORDS.
  • Risk escalation: after Tier 1, a shallow containsSuspiciousEncoding check in the tool result sanitizer upgrades risk to high for ROT13/binary/Morse/etc., triggering deep decode + redaction.

• Bug Fixes

  • Prevented full‑text decodes (ROT13/ROT47) from overwriting positional replacements; decodeAllLevels() now converges and is covered by ROT13+ROT47 overlap tests.
  • Corrected normalization order (unicode before whitespace) and ensured the raw pass runs when only normalised text has keywords.
  • normalizeWhitespace no longer consumes spaces around newlines, preserving word boundaries.

^{Written for commit a9bdb57. Summary will update on new commits.}

[cubic-dev-ai]

4 issues found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/classifiers/pattern-detector.ts">

<violation number="1" location="src/classifiers/pattern-detector.ts:107">
P2: Matches from the normalised pass carry `position` and `matched` values from the transformed text, not the caller's original input. The `PatternMatch.position` contract ("Position in the original string") is violated, and `matched` will contain text like `"ignore previous"` when the input was `"1gn0r3 pr3v10us"`. This makes the diagnostic output of `analyze()` misleading and could break any consumer that correlates matches back to the source string.

Consider either (a) mapping normalised-match positions back to the raw text, or (b) adding a flag/field to distinguish normalised-origin matches so consumers know the coordinates are approximate.</violation>
</file>

<file name="src/sanitizers/encoding-detector.ts">

<violation number="1" location="src/sanitizers/encoding-detector.ts:372">
P1: After partial multi-level decoding, the deep check only scans plaintext keywords and can miss suspicious content that remains encoded.</violation>
</file>

<file name="src/sanitizers/normalizer.ts">

<violation number="1" location="src/sanitizers/normalizer.ts:134">
P2: The newline regex is too permissive and also joins separate words across line breaks, not just obfuscated intra-word splits.</violation>
</file>

<file name="specs/pattern-detector.spec.ts">

<violation number="1" location="specs/pattern-detector.spec.ts:315">
P2: The new assertion is too permissive and can miss regressions in leet normalization because it passes without `ignore_previous` being detected.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[Copilot]

Pull request overview

This PR strengthens Tier 1 injection detection by normalizing common obfuscation techniques before pattern matching and by enhancing encoding detection to unwrap chained encodings prior to suspicious-content checks.

Changes:

• Add a Tier 1 pre-processing normalization chain (whitespace → Unicode → leet-speak) and run pattern detection in two passes (raw + normalized) with merged results.
• Introduce multi-level encoding decoding (decodeAllLevels) and a deep suspicious-encoding check used by the high-risk sanitizer path.
• Update fast-filter keywords and adjust/relax a leet-related Tier 1 test expectation.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/sanitizers/sanitizer.ts	Switch high-risk encoding detection to use deep multi-level suspicious-encoding checks.
src/sanitizers/normalizer.ts	Add normalizeWhitespace() to collapse spaced-letter and embedded-newline obfuscation for Tier 1 analysis.
src/sanitizers/leet-normalizer.ts	Add normalizeLeetSpeak() with protected-sequence handling to avoid corrupting encodings / $(.
src/sanitizers/encoding-detector.ts	Add iterative decoding (decodeAllLevels) and containsSuspiciousEncodingDeep() for chained encodings.
src/classifiers/patterns.ts	Remove raw leet tokens from FAST_FILTER_KEYWORDS (relying on normalized fast-filter hits).
src/classifiers/pattern-detector.ts	Wire normalization chain into analyze(), run raw + normalized passes, and merge/dedup matches.
specs/pattern-detector.spec.ts	Update leet injection test to accept either normalized or raw leet pattern detection.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

_{Requires human review: Auto-approval blocked by 4 unresolved issues from previous reviews.}

[cubic-dev-ai]

0 issues found across 5 files (changes from recent commits).

_{Requires human review: Significant logic changes to the core pattern detection pipeline, including multi-pass normalization and iterative decoding. Requires human review for security heuristics and performance.}

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

_{Requires human review: This PR introduces significant logic changes to the core security detection pipeline, including a normalization chain and deep encoding detection, which require human validation.}

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

_{Requires human review: Significant changes to core pattern detection logic and the introduction of a new normalization pipeline require human review for potential performance and accuracy impacts.}

[cubic-dev-ai]

0 issues found across 5 files (changes from recent commits).

_{Requires human review: Significant changes to core pattern detection logic, normalization chains, and multi-level decoding require human review to assess security impact and performance.}

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

_{Requires human review: Significant additions to core detection logic (normalization chains, deep encoding detection) and 1000+ line diff across security-critical files require human review.}

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

_{Requires human review: Significant feature adding complex normalization and deep encoding detection logic to core security paths. Requires human review for architectural impact and potential edge cases.}

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

_{Requires human review: Large change (1200+ lines) adding complex normalization and deep decoding logic to core detection paths. Requires human review to ensure no regressions in detection or performance.}