[Medium] Three token-stream implementations are highly similar

[coderabbitai]

Description

Three separate token-stream implementations contain highly similar logic for lookahead, expect, slice, EOF span, and error handling. This duplication creates maintenance overhead and increases the risk of inconsistent behaviour across parsers.

Priority

Medium

Details

Duplicate implementations:

• src/parser/token_stream.rs:23 (main token stream)
• src/parser/expression/token_stream.rs:12 (expression token stream)
• src/parser/pattern/mod.rs:17 (pattern token stream)

Repeated logic:

• Lookahead operations
• Expect/consume operations
• Token slicing
• EOF span handling
• Error handling patterns

Impact

• Maintenance burden: Changes to token-stream logic must be applied in three locations
• Drift risk: Implementations can diverge, leading to inconsistent parser behaviour
• Testing complexity: Each implementation requires separate test coverage
• Code duplication: Violates DRY principle and increases codebase size

Actionable fix

Introduce a shared core token cursor trait/struct and thin adapters for parser-specific behaviour.

Approach:

1. Extract common token-stream operations into a core trait or struct (e.g., TokenCursor)
2. Implement shared logic for lookahead, expect, slice, EOF span, and error handling
3. Create thin adapter wrappers for expression-specific and pattern-specific behaviour
4. Migrate existing implementations to use the shared core

Benefits:

• Single source of truth for token-stream operations
• Consistent behaviour across all parsers
• Easier to maintain and extend
• Reduces code duplication
• Simplifies testing

References

• PR: Add ADR: parser crate split for multi-consumer use #212
• Requested by @leynos

Requested by

@leynos

[traycerai]

Warning

Rate limit exceeded. Please try after 513246 seconds.