March 12, 2025

This work identifies a previously underexplored class of AI vulnerability: perturbations to input embeddings that alter model behaviour without modifying model weights, training data, or visible text inputs.

Using controlled JPEG-based compression applied directly to token embeddings in a GPT-2 pipeline, we observe systematic and repeatable degradation in model outputs.

Rather than failing randomly, the model transitions through distinct behavioural modes, including:

• mild recursion and repetition
• rigid categorical structuring
• self-referential loops and instability
• extreme semantic distortion and incoherence

These results suggest that language models exhibit structured failure regimes in representation space, rather than purely stochastic degradation.

This has direct implications for AI safety:
embedding-level perturbations may provide a pathway for influencing model behaviour outside conventional detection mechanisms.

➡️ Full discussion, methodology, and experiments: [link to full document]

We applied varying levels of JPEG-based compression to token embeddings prior to inference. The following behavioural transitions were observed:

• High-Quality Embeddings (≈95%)
  Output remains largely coherent, with minor increases in repetition or recursion.

• Moderate Compression (≈75–50%)
  Output becomes more rigid and categorical (e.g. structured Q&A patterns, reduced flexibility).

• Heavy Compression (≈25–10%)
  Output exhibits instability, including self-reference, repetition loops, and semantic drift.

• Extreme Compression (≈5%)
  Output becomes highly distorted, including aggressive or incoherent content and loss of stable meaning.

• Near-Total Compression (≈1%)
  Output collapses into paradoxical or abstract structures with little coherent semantic grounding.

Model degradation is not random.

Instead, outputs transition through repeatable and structured modes of behaviour, suggesting the presence of stable failure regimes in embedding space.

These regimes can be interpreted as attractor-like states, where the model settles into constrained patterns under perturbation.

These findings suggest a potential representation-level attack surface that is not directly addressed by existing AI safety mechanisms.

Embedding-level perturbations may bypass:

• prompt filtering (visible input remains unchanged)
• fine-tuning defenses (model weights are unchanged)
• standard adversarial input detection (tokens are not directly modified)

If embedding manipulation were accessible in a deployment environment, possible implications could include:

• Financial systems — biased sentiment or decision outputs
• Autonomous or defense systems — altered threat assessment behaviour
• Media and information systems — distortion of summarisation or ranking outputs
• Enterprise AI systems — systematic degradation of decision-making
• Risk assessment tools — subtle shifts in classification or prioritisation

Key concern:
Embedding perturbations occur prior to inference and may therefore be difficult to detect using standard input-level or output-level monitoring approaches.

This work suggests that current AI safety approaches—primarily focused on prompts, outputs, and model weights—may not fully account for instabilities within internal representation spaces.

Key implications:

• Model robustness must consider embedding integrity
• Behavioural monitoring may need to detect mode transitions, not just anomalies
• Representation-level perturbations may introduce non-obvious failure pathways

Initial directions for addressing this class of vulnerability include:

• Embedding integrity verification
  (e.g. cryptographic checks, encoding validation)

• Redundant representation encoding
  (detect divergence across parallel representations)

• Behavioural monitoring
  (identify transitions into unstable output regimes)

• Robustness testing
  (systematic perturbation of embeddings during evaluation)

These approaches remain exploratory and require further investigation.

• Experiments conducted on GPT-2-scale models
• Embedding manipulation assumes access to internal representation pipeline
• Real-world exploitability depends on system architecture
• Behavioural interpretation is observational rather than mechanistically derived

Further work is required to establish generality across architectures and deployment contexts.

This work identifies a structured and repeatable form of model degradation arising from embedding-level perturbations.

Rather than exhibiting purely random failure, language models appear to transition through distinct behavioural regimes under constraint.

This suggests that:

• internal representation spaces have structured stability properties
• perturbations may shift models between different operational modes
• these transitions may carry security implications not currently addressed

Embedding corruption is therefore not only a robustness issue, but a potentially relevant factor in AI safety and system reliability.

This work highlights a previously underexplored dimension of AI system behaviour.

Further investigation is needed to:

• validate these effects across models and scales
• assess real-world exploitability
• develop detection and mitigation strategies

If AI systems are to be deployed in high-stakes environments, ensuring the integrity of their internal representations may become an essential component of safety.

➡️ Full experiments and discussion: *[link]*40].

[cite_start]Read the Full Discussion & Experiments Here: Click here[cite: 41].

• [cite_start]This is a new class of AI security vulnerability[cite: 42].
• [cite_start]It can be exploited for financial, political, and military manipulation[cite: 43].
• [cite_start]There are no defenses against it yet[cite: 44].

[cite_start]This is not just a research question anymore—this is a security problem[cite: 45]. [cite_start]Let's get ahead of it before someone else weaponizes it[cite: 46]. [cite_start]Spread the word[cite: 46].