Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ exclude = [
"/assets",
"PYTHON.md",
"NODE.md",
"WASM.md",
"CLI.md",
"DOCKER.md",
]
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ The good news: we have the specialized equipment. And it is written in Rust, so
| ------------------ | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **A: Unicode** | ZWSP, bidi controls, tag chars, variation selectors, private-use codepoints, **dash homoglyphs** (U+2011 non-breaking hyphen, en-dash, em-dash, etc.), **punctuation homoglyphs** (curly quotes, ellipsis U+2026, etc.), **mathematical alphanumerics** (饾懆鈫扐), Braille blank U+2800 | Deterministic, lossless exorcism 馃Ч |
| **File: Metadata** | C2PA manifests, EXIF, XMP, document properties, the digital equivalent of a tracking ankle bracelet | Stripped from PNG, JPEG, WebP, SVG, PDF, DOCX, ODT, HTML, Markdown |
| **B: Statistical** | Token-sampling watermarks (SynthID-Text, KGW), watermarks baked into the actual word choices | Best-effort via stochastic synonym replacement (400+ English entry table + ES/FR/DE/AR multilingual support) |
| **B: Statistical** | Token-sampling watermarks (SynthID-Text, KGW), watermarks baked into the actual word choices | Best-effort via stochastic synonym replacement (~30 000-entry Moby Thesaurus II + ES/FR/DE/AR multilingual support) |
| **Pixel** | SynthID-Image, StegaStamp, Tree-Ring, StableSignature: pixel-domain perturbations invisible to the eye | Decode鈫抮aw RGBA鈫抣ossless PNG re-encode via `pixel-scrub` feature |

> **Fun fact:** some of those invisible characters are technically in the Unicode "Tag" block, which was originally designed for plane tickets in 1997 and then deprecated. AI providers found a new use for them. The Unicode Consortium is presumably very proud.
Expand Down Expand Up @@ -142,7 +142,7 @@ println!("Substituted {} words", output.words_substituted);
println!("{}", output.text);
```

The English table has **400+ curated entries** covering common verbs, nouns, and adjectives. A two-tier fallback uses `/usr/share/dict/` system wordlists for same-length substitution when no curated synonym exists.
The English table is the full **Moby Thesaurus II** (~30 000 head words, ~2.5 M synonym tokens, public domain) embedded at compile time as a PHF map: O(1) lookups, zero runtime I/O. A two-tier fallback uses `/usr/share/dict/` system wordlists for same-length substitution when no Moby entry exists.

### 馃實 Multilingual Support

Expand All @@ -164,7 +164,7 @@ assert_eq!(lang, LanguageHint::Arabic);

| Language | Identifier | Entries |
| -------- | --------------------- | ------- |
| English | `LanguageHint::English` | 400+ |
| English | `LanguageHint::English` | ~30 000 (Moby Thesaurus II, public domain) |
| Spanish | `LanguageHint::Spanish` | 35 |
| French | `LanguageHint::French` | 32 |
| German | `LanguageHint::German` | 32 |
Expand Down
4 changes: 4 additions & 0 deletions build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
// option. This file may not be copied, modified, or distributed
// except according to those terms.

//! # Build Script
//!
//! Emits link flags required by napi-rs when the `node` feature is enabled.

fn main() {
#[cfg(feature = "node")]
napi_build::setup();
Expand Down
33 changes: 20 additions & 13 deletions src/text/stochastic.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@
//!
//! ## Key Optimisation: Compile-time PHF Synonym Table
//!
//! The curated synonym table is stored as a `phf::Map` with `&'static [&'static str]`
//! values, compiled into the binary at build time. This replaces a runtime `HashMap`
//! that would be rebuilt on each call with a **zero-allocation, O(1)** lookup.
//! The English synonym table is the full **Moby Thesaurus II** (~30 000 head words,
//! ~2.5 M synonym tokens, public domain, Grady Ward) stored as a `phf::Map` with
//! `&'static [&'static str]` values, compiled into the binary at build time by
//! `build.rs`. This provides a **zero-allocation, O(1)** lookup with no runtime I/O.
//!
//! ## Stop-word Detection
//!
Expand All @@ -25,17 +26,17 @@
//! ## Enhancement Pipeline
//!
//! For each non-stop word in the input text, [`StochasticEnhancer`] samples a uniform
//! Bernoulli draw with probability `p` (default 0.5). On a "hit", the curated table
//! is consulted first; if no entry exists, a same-length word from the system wordlist
//! is substituted. Case style (ALL_CAPS, Capitalised, lowercase) is mirror-copied to
//! the substituted word.
//! Bernoulli draw with probability `p` (default 0.5). On a "hit", the Moby Thesaurus
//! table is consulted first; if no entry exists, a same-length word from the system
//! dictionary is substituted. Case style (ALL_CAPS, Capitalised, lowercase) is
//! mirror-copied to the substituted word.
//!
//! ## Wordlist Source
//!
//! On Linux and macOS, the system dictionary at `/usr/share/dict/american-english`
//! (or the first path from [`SYSTEM_DICT_PATHS`] that exists) is loaded at
//! [`SynonymBank::new`] construction time. On WASM targets the wordlist is omitted
//! and only the curated table is used.
//! [`SynonymBank::new`] construction time as a second-tier fallback. On WASM targets
//! the system wordlist is omitted and only the Moby Thesaurus table is used.
//!
//! ## Example
//!
Expand Down Expand Up @@ -275,12 +276,18 @@ impl SynonymBank {
pub fn candidate<R: Rng>(&self, word: &str, rng: &mut R) -> Option<&str> {
let table = synonyms_for(self.language);
if let Some(synonyms) = table.get(word) {
let idx = rng.random_range(0..synonyms.len());
return Some(synonyms[idx]);
let filtered: Vec<&&str> = synonyms.iter().filter(|&&w| w != word).collect();
if !filtered.is_empty() {
let idx = rng.random_range(0..filtered.len());
return Some(*filtered[idx]);
}
}
if let Some(bucket) = self.wordlist.get(&word.len()).filter(|b| !b.is_empty()) {
let idx = rng.random_range(0..bucket.len());
return Some(&bucket[idx]);
let filtered: Vec<&String> = bucket.iter().filter(|&w| w != word).collect();
if !filtered.is_empty() {
let idx = rng.random_range(0..filtered.len());
return Some(filtered[idx].as_str());
}
}
None
}
Expand Down
Loading