Pivot to shell-command vs LLM-prompt detection (warp style)

[ealmloff]

Summary

Complete pivot of betlang's classes and data: the 48-class source-language student is replaced by a 2-class detector matching Warp's Agent Mode auto-detection task — is terminal input a shell command or a prompt for a language model? (Warp's public input_classifier crate does exactly this with a 17MB BERT-tiny ONNX model + heuristics; this model is ~45KB.)

Model / data (new training pipeline, no Magika teacher):

• scripts/build_prompt_corpus.py builds a ~148k-sample corpus:
  • prompt: real user prompts from OpenAssistant oasst1 (English first turns), ShareGPT first human turns, and no_robots, plus Alpaca/Dolly instructions and awesome-chatgpt-prompts personas.
  • shell_command: real bash one-liners from NL2Bash and tldr-pages example commands ({{placeholder}} flattened).
• scripts/train_prompt_student.py trains the existing wordseq-b1024-k3-m2048-tiny-3conv-hidden architecture from scratch with hard labels + label smoothing, QAT for the second half of the schedule, exporting through the existing MSQ1 serializer (head now 96 -> 2).
• New embedded artifact (45,448 bytes): test_accuracy=0.992, recall prompt=0.989 / shell_command=0.995.

Rust API (Language → Kind):

pub enum Kind { Prompt = 0, ShellCommand = 1 }   // slugs: prompt, shell_command

assert_eq!(betlang::detect("Write a short poem about the ocean.").kind(), Some(Kind::Prompt));
assert_eq!(betlang::detect("git log --oneline | head -5").kind(), Some(Kind::ShellCommand));
detection.top_kinds() // ranked (probability, Kind) pairs

• src/language.rs → src/kind.rs; Detection::language()/top_languages() → kind()/top_kinds(); CLASSES: 48 → 2. Runtime/tokenizer/quantized layers unchanged.
• Language fixtures/tests replaced with tests/fixtures/{prompt,shell_command}/; example CLI tree mode reports a prompt-vs-shell breakdown; bench uses a prompt snippet.

Supersedes #9 (natural-language vs prompt) after feedback to match Warp's actual task and use real-user prompt datasets.

cargo fmt --check, cargo test, cargo clippy --all-targets --all-features -- -D warnings, and RUSTDOCFLAGS="-D warnings" cargo doc are green locally.

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/2d01d14ba6444015ae43aac7fc183d60
Requested by: @ealmloff

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

[staging-devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring