Refactor parser: Remove macro-based token dispatch and manual stateful block parsing

[sourcery-ai]

The current parser implementation contains two areas of unnecessary complexity:

1. Macro-based token dispatch (token_dispatch!)
2. Manual stateful block parsing using balanced_block_with_min and a Cell state machine

Both can be simplified for improved readability and maintainability without sacrificing functionality.

---

1. Replace token_dispatch! Macro with a Method

Instead of using the token_dispatch! macro, introduce a dispatch method on TokenStream that takes a closure. This method would loop over tokens and allow the closure to handle each token, making the code more idiomatic and easier to follow.

Example Implementation:

impl<'a> TokenStream<'a> {
    /// Drive a user-supplied closure over every peeked token;
    /// the closure must advance the stream itself.
    pub fn dispatch<F>(&mut self, mut f: F)
    where
        F: FnMut(&mut Self, SyntaxKind, Span),
    {
        while let Some(&(kind, ref span_ref)) = self.peek() {
            let span = span_ref.clone();
            f(self, kind, span);
        }
    }
}

Call-site change:

// before:
// token_dispatch!(st, {
//     SyntaxKind::T_LPAREN => lp,
//     SyntaxKind::T_RPAREN => rp,
// });

// after:
st.stream.dispatch(|stream, kind, span| {
    match kind {
        SyntaxKind::T_LPAREN => lp(&mut st, span),
        SyntaxKind::T_RPAREN => rp(&mut st, span),
        _                  => stream.advance(),
    }
});

---

2. Replace Manual Stateful Block Parsing with Recursive Combinators

The current approach uses a Cell-based depth counter and manual state management. This can be replaced with a recursive parser combinator (e.g., using chumsky::recursive), which is flatter and easier to read.

Example Implementation:

use chumsky::recursive;

pub(super) fn balanced_block(
    open: SyntaxKind,
    close: SyntaxKind,
) -> impl Parser<SyntaxKind, (), Error = Simple<SyntaxKind>> + Clone {
    recursive(|block| {
        just(open)
            .padded_by(inline_ws().repeated())
            .ignore_then(
                block
                    .or(filter(|k: &SyntaxKind| *k != open && *k != close).ignored())
                    .padded_by(inline_ws().repeated())
                    .repeated(),
            )
            .then_ignore(just(close))
            .ignored()
    })
}

// For nonempty blocks:
pub(super) fn balanced_block_nonempty(
    open: SyntaxKind,
    close: SyntaxKind,
) -> impl Parser<SyntaxKind, (), Error = Simple<SyntaxKind>> + Clone {
    recursive(|block| {
        just(open)
            .padded_by(inline_ws().repeated())
            .ignore_then(
                block
                    .or(filter(|k: &SyntaxKind| *k != open && *k != close).ignored())
                    .padded_by(inline_ws().repeated())
                    .repeated()
                    .at_least(1),
            )
            .then_ignore(just(close))
            .ignored()
    })
}

This approach eliminates the need for internal mutable state and results in a much cleaner parser.

---

Action Items

• Refactor the token_dispatch! macro into a dispatch method on TokenStream.
• Replace the manual stateful block parsing (balanced_block_with_min + Cell) with a recursive combinator approach.
• Update all call sites and tests accordingly.

These changes will make the parser codebase simpler, more idiomatic, and easier to maintain.

---

I created this issue for @leynos from #73 (comment).

Tips and commands

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[leynos]

@coderabbitai looking at the codebase, is this issue still relevant? If the issue has been fixed in the code, does the documentation reflect this?

If it is still relevant, please provide an assessment of the issue criticality (critical, high, medium or low), based on risk of the issue impacting the user and the anticipated blast radius.\n\nOnce a decision has been made, apply the appropriate label.

If the issue is documentation ONLY (i.e., no code changes needed), apply the documentation label in addition to the appropriate criticality label.

Please also provide an impact of the required changes.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch issue #74 details
gh issue view 74 --repo leynos/ddlint --json title,body,state,labels

# List README and documentation files
fd -t f 'README|CHANGELOG|CONTRIBUTING|docs' -e md -e rst -e txt

# Get a general overview of the codebase structure
fd -t f -e py -e js -e ts | head -20

Length of output: 4276

---

🏁 Script executed:

#!/bin/bash
# Search for token_dispatch macro
rg -n "token_dispatch!" -A3 -B3

# Search for balanced_block_with_min
rg -n "balanced_block_with_min" -A3 -B3

# Search for Cell usage in parsing context
rg -n "Cell<.*>" --type rust -A2 -B2

# Find parser-related files
fd -e rs . | grep -E "(parse|token|stream)"

# Read the README
cat README.md

Length of output: 8014

---

🏁 Script executed:

#!/bin/bash
# Apply the "low" criticality label to issue #74
gh issue edit 74 --repo leynos/ddlint --add-label "low"

Length of output: 42

---

@leynos, I've completed the assessment of issue #74. Here are my findings:

Relevance: Still Relevant ✅

Both refactoring opportunities mentioned in the issue are still present in the codebase:

1. token_dispatch! macro - actively used in 6 files:

   • src/parser/span_scanners/typedefs.rs
   • src/parser/span_scanners/relations.rs
   • src/parser/span_scanners/utils.rs
   • src/parser/span_scanners/indexes.rs
   • src/parser/span_scanners/imports.rs
   • src/parser/lexer_helpers.rs (definition + usage)

2. balanced_block_with_min with manual state - still exists in src/parser/lexer_helpers.rs

Documentation Status

The README contains only basic logging setup information. There is no documentation covering parser architecture or these implementation details.

Criticality: LOW 🟢

Rationale:

• User Impact Risk: Very low - this is internal code structure with no user-facing impact
• Blast Radius: Medium - affects 6+ parser files, but changes are isolated to implementation details
• Type: Technical debt/code quality improvement, not a functional bug

Impact Assessment

Development Effort: Medium

• Refactor token_dispatch! macro across 6 files to use new dispatch method
• Replace balanced_block_with_min with recursive combinators
• Update all call sites to use new patterns
• Update associated tests

Benefits:

• Improved code readability and maintainability
• More idiomatic Rust patterns
• Easier onboarding for new contributors

Risk: Low - well-defined refactoring with existing test coverage

---

I've applied the low criticality label to the issue.