Home

TinyTokenizer

A high-performance syntax tree library for .NET 8+ with fluent queries, pattern matching, and undo/redo editing — built on a zero-allocation tokenizer with SIMD optimization.

Features

Feature	Description
TinyAst	Red-green syntax tree with fluent queries, editing, and undo/redo
Query API	CSS-like selectors with combinators, lookahead, and repetition
SyntaxEditor	Batch mutations with atomic commit and full undo/redo support
Syntax Nodes	Pattern-based AST matching (function calls, property access, etc.)
Keyword System	Schema-defined keywords with categorization and efficient lookup
Schema System	Unified configuration for tokenization + keywords + syntax defs
TreeWalker	DOM-style filtered tree traversal
High Performance	Zero-allocation parsing with SIMD-optimized SearchValues<char>
Error Recovery	Gracefully handles malformed input and continues parsing

Quick Example

using TinyTokenizer.Ast;

// Parse source into a syntax tree
var tree = SyntaxTree.Parse("function foo() { return 1; }");

// Query nodes with CSS-like selectors
var idents = tree.Select(Query.AnyIdent);  // [Ident("function"), Ident("foo"), Ident("return")]

// Fluent mutations with undo support
tree.CreateEditor()
    .Replace(Query.Ident("foo"), "bar")  // Matches: Ident("foo")
    .InsertAfter(Query.BraceBlock.First().Start(), "console.log('enter');")
    .Commit();

// Undo/redo
tree.Undo();
tree.Redo();

Getting Started

→ Getting Started — Installation and basic usage

Documentation

TinyAst

• TinyAst Guide — SyntaxTree, red/green nodes, basic navigation
• Schema — Unified tokenization + syntax definitions
• Query API — CSS-like selector API with all combinators
• SyntaxEditor — Batch mutations, undo/redo
• Syntax Nodes — Pattern-based AST matching
• TreeWalker — DOM-style filtered traversal
• Trivia — Whitespace and comment preservation

Low-Level Tokenization

• Token Types — SimpleToken vs Token, all concrete types
• Configuration — TokenizerOptions, operators, comments, symbols
• Async Streaming — Stream, PipeReader, IAsyncEnumerable
• Error Handling — ErrorToken and recovery patterns

Reference

• Architecture — Two-level design, red-green trees, SIMD optimization
• API Reference — Core types, enums, operators, comments

Requirements

• .NET 8.0 or later
• CommunityToolkit.HighPerformance (automatically included)

License

MIT — See LICENSE