AI-Text-Detector Robustness Harness

A research tool that builds a perplexity/burstiness/lexical AI-text detector and measures how its accuracy degrades under generic, benign text transformations. The point is to quantify detector fragility — useful for evaluating whether such detectors are reliable enough to act on (e.g., in education or moderation policy debates).

What this is / is not

• Is: an adversarial-robustness study of a detector we build ourselves.
• Is not: a "humanizer", an evasion model, or anything tuned against a specific commercial detector. The perturbations are standard generic probes and never read detector feedback.

Quick start

pip install -r requirements.txt
python -m nltk.downloader punkt wordnet omw-1.4
python -m detector_robustness.cli --seed --out out --fake-scorer   # instant
python -m detector_robustness.cli --seed --out out_real            # real model
pytest -v                                                          # tests

CLI options

Flag	Meaning	Default
--seed	Use the built-in 12-example seed corpus	off
--data PATH	Use a JSONL corpus instead (--data or --seed required)	—
--out DIR	Output directory for report.md and degradation.png	out
--fake-scorer	Deterministic length-based scorer; no model download	off
--test-frac F	Fraction of the corpus held out for the test split	0.4
--seed-value N	RNG seed for the split and the perturbations (note: distinct from --seed, which selects the corpus)	42

Using a real corpus

Provide a JSONL file, one object per line: {"text": "...", "label": 0, "domain": "essays"} where label is 0 for human and 1 for AI. Then: python -m detector_robustness.cli --data corpus.jsonl --out out. For credible results use a balanced, multi-domain corpus (hundreds+ per class) and report on a held-out test split.

Interpreting results

report.md shows clean AUC/accuracy and, per probe, the AUC and the change in AI-class recall. Large negative deltas indicate the detector misses AI text after a trivial benign edit — i.e., it is not robust enough to rely on for consequential decisions.

Limitations & responsible use

• The seed corpus is tiny and illustrative only; real conclusions need a real corpus.
• This measures our detector, not any third-party product, and should not be represented as such.
• Findings about detector unreliability are intended to inform responsible policy and detector improvement, not to facilitate academic dishonesty or deceptive content.