-
Notifications
You must be signed in to change notification settings - Fork 0
NewDesign
(disclosure: written with assistance of Claude/Sonnet)
The goal isn't to patch Irony. It's to take what was genuinely right about it — grammar as a C# class, BNF through operator overloading, terminals as composable mini-scanners — and rebuild everything else on a foundation that can actually support the use cases Irony couldn't.
Here's what that means concretely.
- Grammar as a C# class, BNF via operator overloading. This abstraction earned its place. It stays unchanged in spirit.
-
Terminals as mini-scanners. The scanner engine's model of delegating to individual terminal implementations once a candidate is selected is sound. It's the right decomposition, and with
Span<char>it can be made significantly more efficient.
Everything else is on the table.
Irony originally targeted .NET Standard to bridge the full framework 4.6 and Core 5.0+ worlds. That compromise is no longer worth carrying. The rebuild targets .NET 10+, which means first-class access to Span<T>, ReadOnlySpan<char>, System.IO.Pipelines, and the broader set of allocation-reduction APIs that have matured over the last several releases. The scanner and terminal layer in particular should be re-architected around spans end-to-end — source positions, token values, and lexeme references should be span slices into the source buffer, not heap-allocated strings.
Irony had no preprocessing layer at all. The rebuild needs one. C/C++-style macro expansion and #include processing require a source stream abstraction sitting between the raw input and the scanner — a layer that can splice, substitute, and redirect the character stream before terminals ever see it. Preprocessor directives like #if/#endif add another dimension: they require evaluating boolean expressions over a defined symbol set, which means a small embedded expression evaluator as part of the preprocessor engine. None of this is exotic, but it demands a clean, explicit API boundary between the source stream layer, the preprocessor, and the scanner — something the original architecture simply didn't have.
The current approach to shift/reduce conflicts is essentially "fix your grammar or provide explicit resolution hints." That's too rigid. The rebuild should treat conflict states as a first-class runtime concern rather than a construction-time hard failure. Concretely: the parser initializer emits a warning for each conflict state, suppressible with an explicit annotation that says "this is known and intentional." At parse time, when execution reaches a conflict state with multiple valid transitions, the engine explores each path in turn, backtracking on failure. This is a meaningful architectural shift — the parser state machine needs to support lightweight checkpointing — but it unlocks a large class of practical grammars that LALR(1) strict-determinism currently rejects.
LALR(1) with Penello construction remains the right default. But the grammar object — the BNF rule graph — is independent of the parsing algorithm that consumes it. There's no fundamental reason the same grammar definition couldn't be used to drive an LL(n) parser, a recursive descent generator, or an Earley recognizer for ambiguous cases. The rebuild should make the boundary between grammar definition and parsing algorithm explicit, so alternative parsing backends are at least structurally possible without re-architecting the grammar layer.
The model I have in mind: non-terminals carry an optional FromVersion attribute. The parser is initialized with a target language version. Productions gated to a higher version are either excluded from the state graph entirely or flagged at parse time with a diagnostic. This keeps the grammar definition as a single source of truth for all versions rather than requiring separate grammar classes per version.
HTML embedding JavaScript and CSS is the motivating case, but the pattern is general. The design I'm considering: a special non-terminal type that signals a language boundary, referencing a separate Grammar instance for the embedded language. When the parser encounters that non-terminal, it hands control to the embedded grammar's scanner and parser until the embedded region closes, then returns to the outer grammar. The tricky part is the scanner/parser interaction boundary — see below.
This is the most architecturally sensitive piece in the rebuild. The naive model — scan the entire input into a flat token stream, then run the parser over it — breaks immediately for multi-language files and preprocessor directives. In the HTML+JS case, the parser needs to detect a <script> tag and switch the scanner to the JavaScript grammar mid-stream. For preprocessor directives, the scanner needs to surface #if lines to a preprocessor layer before they reach the parser, and that preprocessor may alter which tokens the scanner produces next.
The clean solution is an explicit co-routine-style protocol between scanner and parser: the parser drives token consumption and can signal scanner context switches; the scanner can surface preprocessor events back to an interception layer before forwarding tokens downstream. Getting this API right is probably the highest-leverage design decision in the entire rebuild — it needs careful thought before any code is written.
One concrete example of where strict LALR(1) grammar expression breaks down: parsing modifier keyword lists in C#. Consider the sequence of keywords — public, static, readonly, abstract, sealed, partial, and so on — that can appear in front of a class, method, property, or field declaration. Each declaration type has its own valid subset of modifiers. Expressing those restrictions directly in BNF is painful at best and practically impossible at worst. The parser reads the keywords before it knows what kind of declaration it's looking at, so any attempt to encode per-element-type restrictions into the grammar produces shift/reduce conflicts and ambiguity that LALR(1) cannot resolve deterministically.
The solution I have in mind is a pair of custom NonTerminal subtypes: KeywordList and KeywordSubList.
KeywordList holds the union of all possible modifier keywords across all declaration types. KeywordSubList references a parent KeywordList but carries a restriction set specific to a declaration type — the allowed keywords for a method, or a field, or a class. In the BNF rules, you use the KeywordSubList for each declaration type. At parse time, when the parser encounters a KeywordSubList, it parses it using the parent KeywordList — consuming all recognized modifier keywords without trying to validate them yet. Once the full declaration is parsed and the declaration type is known, the engine makes a second pass over the collected keyword set and applies the KeywordSubList restrictions, emitting diagnostics for any keywords that don't belong.
The LALR automaton never sees the restriction problem. It operates on the permissive parent list and defers the semantic validation to a post-production step. The grammar stays unambiguous; the language-specific rules are enforced, just not at the shift/reduce level.
I'm calling this trick two-stage parsing — parse permissively first, validate against a tighter contract once enough context is available. The modifier list case is the clearest example, but the pattern is likely applicable elsewhere: argument label validation, attribute target restrictions, annotation sets with context-dependent legality. Any situation where the valid set of some syntactic element depends on context that isn't yet available when the element is being recognized is a candidate. It's worth building KeywordList and KeywordSubList as first-class constructs in the new grammar model and watching for other places the same abstraction fits.
Irony's parser produces a parse tree — a direct structural reflection of the grammar rules applied to the source text. In most real use cases, that parse tree is an intermediate artifact. What you actually want is an Abstract Syntax Tree: a typed object graph that represents the semantics of the source, not its syntax. The distinction matters. A VB.NET source file and a C# source file produce completely different parse trees, but if you're building a .NET semantic model — classes, methods, properties, type references — the AST for both should be identical. The parse tree is grammar-specific; the AST is domain-specific.
Old Irony had rudimentary support for this. You could attach an AST node type or a creator method reference to a NonTerminal, and the engine would attempt construction during tree building. In practice — and my experience building NGraphQL is the direct evidence here — this was painful. The mapping between parse tree structure and AST node construction required a significant amount of repetitive plumbing code that the framework should have been handling automatically.
The design I'm considering decouples AST construction from the grammar definition entirely. The grammar classes carry no references to AST types. Instead, AST construction logic lives in a separate class — an AST provider — that is associated with the grammar dynamically at initialization time. The engine scans the provider class using reflection, discovers methods decorated with grammar-binding attributes, and wires them to the corresponding NonTerminal productions automatically.
An AST builder method would be a typed function returning an AST node, with parameters that correspond — by position, name, or explicit attribute — to the elements of the BNF production for its bound non-terminal. The binding attribute on the method identifies which non-terminal it handles. The engine takes care of invoking the right method with the right parse tree children, handling list productions, optional elements, and token value extraction as part of the plumbing layer rather than leaving it to the implementor.
The result: AST construction logic reads as a clean set of factory methods, each focused purely on constructing one semantic concept from its syntactic children. No manual parse tree traversal, no positional child indexing, no repetitive null checks on optional elements. The grammar stays grammatical; the AST layer stays semantic; the engine owns the connection between them.
GrammarExplorer needs to be rebuilt from scratch. The WinForms implementation served its purpose but it's not the right platform going forward. My current thinking is Blazor WebAssembly for the UI, which gives browser-based access without a deployment dependency, combined with a desktop host — likely the new .NET MAUI hybrid app model or Electron, depending on how the ecosystem looks at the time. The core feature set — synchronized state graph, parse log navigation, parse tree, source position tracking — stays. But the rebuild is also an opportunity to add grammar diff views, conflict visualization, and potentially live grammar editing with incremental re-construction of the state graph.
Irony had none. A handful of demo grammars, some CodePlex discussion threads, and whatever you could reverse-engineer from the source. For developers trying to do anything non-trivial, that was a real barrier, and it almost certainly limited adoption.
The rebuild should provide documentation as a first-class deliverable, not an afterthought. That means API reference, conceptual guides covering the grammar model and parser architecture, worked examples for common patterns (expression languages, configuration formats, statement-based DSLs), and tutorials that walk through building a grammar from scratch through to a working evaluator.
This is also one area where AI tooling changes the economics meaningfully. Generating first drafts of API reference from method signatures and XML doc comments, turning grammar samples into annotated tutorials, producing "how do I..." guides — all of that is tractable and easier now with AI. The documentation still needs an author who understands the system deeply enough to catch what the AI gets wrong. But the raw effort of going from zero to comprehensive coverage is substantially lower than it used to be, and there's no good excuse for shipping without it.
The rebuild is a larger undertaking than Irony was. The original was a proof of concept that outgrew its architecture. This one needs to be designed for the full problem space from the start.