Pivot to natural-language vs LLM-prompt detection (warp style)

[ealmloff]

Summary

Complete pivot of betlang's classes and data: the 48-class source-language student is replaced by a 2-class detector that decides whether text is ordinary prose or a prompt for a language model (the Warp-style "should this input go to the AI?" routing decision).

Model / data (new training pipeline, no Magika teacher):

• scripts/build_prompt_corpus.py builds a ~105k-sample corpus from public HF datasets — prompts from Alpaca, Dolly, awesome-chatgpt-prompts; prose from WikiText-103, AG News, IMDB, Yelp. Long prose samples also emit a random 1–2 sentence slice so length can't become a proxy for the label (this fixed a strong short-text→prompt bias in the first training run).
• scripts/train_prompt_student.py trains the same wordseq-b1024-k3-m2048-tiny-3conv-hidden architecture from scratch with hard labels + label smoothing, QAT for the second half of the schedule, and exports through the existing MSQ1 serializer (head now 96 -> 2).
• New embedded artifact (45,448 bytes): test_accuracy=0.983, recall natural_language=0.976 / prompt=0.995. MODEL_CARD/TRAINING docs rewritten.

Rust API (Language → Kind):

pub enum Kind { NaturalLanguage = 0, Prompt = 1 }   // slugs: natural_language, prompt

let detection = betlang::detect("Write a short poem about the ocean.");
assert_eq!(detection.kind(), Some(betlang::Kind::Prompt));
detection.top_kinds() // ranked (probability, Kind) pairs

• src/language.rs → src/kind.rs; Detection::language()/top_languages() → kind()/top_kinds(); CLASSES: 48 → 2. Runtime/tokenizer/quantized layers unchanged.
• 48 language fixtures/tests replaced with prompt & natural-language fixtures under tests/fixtures/{prompt,natural_language}/; example CLI tree mode now reports a prompt-vs-prose breakdown; bench uses a prompt snippet.

cargo fmt --check, cargo test (all pass incl. doctests), cargo clippy --all-targets --all-features -- -D warnings, and RUSTDOCFLAGS="-D warnings" cargo doc are green locally.

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/2d01d14ba6444015ae43aac7fc183d60
Requested by: @ealmloff

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

[staging-devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring