textmate-lang builds TextMate grammars in Rust and emits them as tmLanguage.json, the highlighting format that VS Code, Sublime Text, TextMate, Shiki, and many other editors and highlighters load without tree-sitter. Describe a language's tokens with matches, begin/end regions, captures, and includes; call to_json; get a file an editor can load.
A grammar file cannot check itself: a typo in a regular expression, a capture of a group that does not exist, an include of an entry that is not there, or two pattern lists that include each other all load silently and then highlight wrongly, or hang the tokenizer. Emitting here validates the whole grammar first and refuses each of these with an error that points into the output. Regular expressions are held to a documented subset of Oniguruma syntax that the engines loading TextMate grammars read the same way. The JSON writer is the crate's own and escapes every string it writes, so no name, pattern, or scope can change the structure of the output. It is the sibling of treesitter-lang in the -lang family.
MSRV is 1.85+ (Rust 2024 edition). no_std-compatible (needs only alloc), #![forbid(unsafe_code)], no dependencies.
Status: pre-1.0 (0.2.0, the foundation). The grammar model, validation, and JSON emission are complete; semantic-token legends and embedded-language helpers come in 0.5. The API may change before1.0.0. SeeROADMAPandCHANGELOG.md.
- A
Grammaris a scope name (source.demo), top-level patterns, a repository of named rules, and injections.to_jsonvalidates it and returns the JSON. - A
Ruleis one entry of a pattern list: aMatch, aBeginEndorBeginWhileregion, an include of a repository entry,$self,$base, or another grammar, or a list of rules. ACapturesays what to do with a capture group. - An
Errorsays what is wrong and where, as a JSON Pointer into the output (/repository/strings/end); a regular-expression error also gives the byte offset in the pattern. validate_regexandescape_regexcheck one pattern, and turn literal text (keywords, operators) into one.
What it guarantees, and how each guarantee is checked:
| Guarantee | How it is held |
|---|---|
| No input can change the structure of the output. | Property tests build random grammars whose names, file types, and pattern text are arbitrary strings (quotes, backslashes, control characters, U+2028/U+2029, any Unicode), emit them pretty and compact, read them back with a strict RFC 8259 parser written in the tests, and compare with the expected value built directly. |
| Output is deterministic. | Repository entries and injections are sorted by name and captures by group; property tests add them in random orders and get identical bytes. |
| Every regular expression is in the documented Oniguruma subset. | Random syntax trees of the subset print to patterns the checker accepts with the right group count; removing any closing parenthesis is refused; arbitrary input never panics and errors point at character boundaries. |
escape_regex output matches its input literally, in every context. |
Property-tested: any text, and any concatenation of escaped texts, is accepted in match and end patterns and in extended mode. |
| Include cycles that would hang a tokenizer are refused; recursion through regions is not. | Property tests compare the cycle check with a reachability computation on random include graphs, and check that each reported cycle is real. |
| Scope names follow TextMate conventions, and substitutions name real groups. | Property tests compare the checker with a direct reading of the rules. |
| No grammar can overflow the stack. | Validation, emission, and dropping are iterative; 500,000-deep rules are refused without recursion, and 100,000-entry grammars validate and emit. |
[dependencies]
textmate-lang = "0.2"Without the standard library:
[dependencies]
textmate-lang = { version = "0.2", default-features = false }use textmate_lang::{BeginEnd, Grammar, Match, Rule};
let json = Grammar::new("source.demo")
.name("Demo")
.file_type("demo")
.pattern(Rule::include("comments"))
.pattern(Rule::include("strings"))
.pattern(Match::new(r"\b(if|else|while|return)\b").scope("keyword.control.demo"))
.pattern(
Match::new(r"\b(fn)\s+([a-z_]\w*)")
.capture(1, "storage.type.function.demo")
.capture(2, "entity.name.function.demo"),
)
.repository("comments", Match::new("#.*$").scope("comment.line.number-sign.demo"))
.repository(
"strings",
BeginEnd::new("\"", "\"")
.scope("string.quoted.double.demo")
.pattern(Match::new(r"\\.").scope("constant.character.escape.demo")),
)
.to_json()?;
assert!(json.contains("\"scopeName\": \"source.demo\""));
// std::fs::write("syntaxes/demo.tmLanguage.json", json)?;
# Ok::<(), textmate_lang::Error>(())use textmate_lang::{BeginEnd, Error, Grammar, Match, RegexError, RegexErrorKind, Rule};
// A pattern outside the subset: `x{2}?` is lazy in some engines and
// optional in Oniguruma.
let err = Grammar::new("source.demo").pattern(Match::new("x{2}?")).validate().unwrap_err();
assert_eq!(
err,
Error::InvalidRegex {
path: "/patterns/0/match".into(),
error: RegexError { offset: 1, kind: RegexErrorKind::AmbiguousQuantifier },
},
);
// `end` back-references name `begin` groups, and there is only one.
let err = Grammar::new("source.demo").pattern(BeginEnd::new("(['\"])", r"\2")).validate().unwrap_err();
assert!(matches!(err, Error::InvalidRegex { ref path, .. } if path == "/patterns/0/end"));
// Two pattern lists that include each other expand forever.
let err = Grammar::new("source.demo")
.repository("a", Rule::patterns([Rule::include("b")]))
.repository("b", Rule::patterns([Rule::include("a")]))
.validate()
.unwrap_err();
assert_eq!(err.to_string(), "includes form a cycle that consumes no text: #a -> #b -> #a");Keyword and operator lists usually come from somewhere else; escape_regex makes literal text safe to match:
use textmate_lang::{Grammar, Match, escape_regex};
let keywords = ["let", "fn", "match"];
let operators = ["=>", "->", "::", "+="].map(escape_regex);
let json = Grammar::new("source.demo")
.pattern(Match::new(format!(r"\b(?:{})\b", keywords.join("|"))).scope("keyword.demo"))
.pattern(Match::new(operators.join("|")).scope("keyword.operator.demo"))
.to_json_compact()?;
assert!(json.contains(r#""match":"\\=\\>|\\-\\>|\\:\\:|\\+\\=""#));
# Ok::<(), textmate_lang::Error>(())Runnable programs in examples/:
| Example | What it shows |
|---|---|
calc |
The Tier-1 path: a small language's grammar, built and emitted. cargo run --example calc > calc.tmLanguage.json |
errors |
What validation reports for common mistakes. cargo run --example errors |
Validation is one pass over the grammar plus one sort of the repository; includes resolve by binary search over a contiguous name index, and the include graph is searched once, iteratively. The regex checker is a single left-to-right pass with an explicit group stack, reused across a grammar's patterns. Emission writes straight into one String from a work stack, copying unescaped runs of each string in bulk.
Measured with the benchmarks in benches/, Windows x86_64, Rust stable, release profile. Throughput is relative to the size of the pretty JSON (for the regex rows, of the pattern or text):
| Benchmark | What it measures | Time | Throughput |
|---|---|---|---|
large_100k/validate |
Validate 100,000 repository entries (matches with captures, regions with nested patterns), 100,000 includes; 45 MB of JSON. | ~136 ms | ~332 MiB/s |
large_100k/to_json_pretty |
Validate and emit the same grammar, pretty. | ~255 ms | ~177 MiB/s |
large_100k/to_json_compact |
Validate and emit it, compact. | ~249 ms | ~182 MiB/s |
typical_300/to_json_pretty |
Validate and emit a 300-entry grammar, the size of a hand-written language grammar. | ~0.38 ms | ~356 MiB/s |
regex/validate_keywords |
Check one alternation of 100,000 keywords (1.3 MiB). | ~3.9 ms | ~344 MiB/s |
regex/validate_mixed |
Check 400 KiB of classes, groups, look-around, and quantifiers. | ~2.0 ms | ~203 MiB/s |
regex/escape |
Escape 1.1 MiB of text. | ~2.3 ms | ~484 MiB/s |
A grammar the size real languages have validates and emits in well under a millisecond. On the 100,000-entry grammar, include resolution dominates validation: each include is a binary search over 100,000 names that no longer fit in cache. A hash index would be faster there, but its worst case on hostile names is quadratic; the binary search's is not.
cargo bench --bench benchCriterion writes per-benchmark reports to target/criterion/. Numbers vary by CPU and between runs on this machine by about ±10%; use the trend, not a single absolute.
- A subset, on purpose. Oniguruma accepts many spellings that other engines refuse or read differently:
x{3}?is an optionalx{3}, not a lazy one; plain groups stop being numbered when named groups appear unless the loader asks otherwise (VS Code does, others do not). Refusing these keeps a grammar's meaning the same in every editor. The subset is listed indocs/API.md. - VS Code's quirks are checked, not assumed away. VS Code substitutes
begincaptures intoendandwhilepatterns by searching the text for a backslash and digits, so\\1there is misread; it is refused. A top-level|in a selector silently drops the rest; it is refused.+in a selector's scopes is dropped by its tokenizer; it is refused. - Which cycles are errors. Recursion through a region (parentheses within parentheses) consumes text at every level and is how nested constructs are written. A cycle of pattern lists that include each other consumes nothing: the tokenizer expands the lists while building its scanner and never stops. Only the second is refused.
- Escaping is not conditional on validation. Every string goes through the JSON escaper whatever the validator decided, so a gap in validation could produce a grammar an editor rejects, never JSON with a different structure.
- No recursion where input reaches. Validation and emission run from explicit work stacks; dropping a rule moves nested rules onto a stack; nesting is capped at levels editors handle.
The suite runs on Windows, Linux, and macOS through the CI matrix, on stable and the 1.85 MSRV:
cargo test # unit + integration + property + doctests
cargo test --no-default-features # no_std + alloc
cargo clippy --all-targets --all-features -- -D warnings
cargo bench --bench benchThe property tests in tests/proptests.rs hold the emitter to a strict RFC 8259 reader and a direct construction of the expected JSON, the regex checker to random syntax trees of the subset, the cycle check to a reachability computation, and the scope checker to a direct reading of the naming rules. PROPTEST_CASES=20000 runs a longer soak. Every rust example in this README and in docs/API.md is compiled and run as a doctest.
What the tests cannot show: that every editor's loader accepts every grammar emitted. The subset is chosen from Oniguruma's documented syntax and VS Code's loader source; running emitted grammars through vscode-textmate itself is planned for 0.9 (ROADMAP).
- Linux (x86_64, aarch64)
- macOS (x86_64, Apple Silicon)
- Windows (x86_64)
The crate uses no operating-system facilities and no platform-specific code; a grammar emits the same bytes on every platform.
See REPS.md and dev/DIRECTIVES.md for the engineering standards every change is held to, and dev/ROADMAP.md for what comes next. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.
Licensed under either of
- Apache License, Version 2.0 — LICENSE-APACHE
- MIT License — LICENSE-MIT
at your option.