feat: add TokenStream snapshot support#884
feat: add TokenStream snapshot support#884gaetschwartz wants to merge 24 commits intomitsuhiko:masterfrom
Conversation
Add `assert_token_snapshot!` macro for testing proc_macro2::TokenStream values.
- File-based mode: Pretty-prints using prettier-please, falls back to
TokenStream::to_string()
- Inline mode with `@{...}` syntax: Compares TokenStreams semantically
(ignoring whitespace differences)
- Extend cargo-insta to recognize @{...} brace syntax for inline updates
New files:
- insta/src/tokenstream.rs: pretty_print() and tokens_equal() helpers
Usage:
// File-based
assert_token_snapshot!(tokens);
assert_token_snapshot!("name", tokens);
// Inline with semantic comparison
assert_token_snapshot!(tokens, @{ struct Foo; });
Add comprehensive functional tests for the tokenstream feature:
- test_tokenstream_inline_empty_to_populated: Tests @{} gets updated
- test_tokenstream_inline_matching: Tests matching tokens pass
- test_tokenstream_file_based: Tests .snap file creation
- test_tokenstream_semantic_comparison: Tests whitespace-insensitive comparison
- test_tokenstream_multiline_inline: Tests multiline token streams
…shots
Add InlineFormat enum to control inline snapshot output format:
- InlineFormat::Text: standard string literal format (@"...")
- InlineFormat::Tokens: brace format for TokenStream (@{ ... })
cargo-insta now detects the original format from the source AST and
preserves it when updating snapshots, instead of always converting
to string format.
- Fix indentation to use @{ line's leading whitespace, not macro start
- Single-line content uses compact format: @{ content }
- Multiline content properly indents: content +4 spaces, closing } aligns with @{
- Add pretty_print_for_inline() for proper multiline handling
- Add tests for same-line and separate-line @{ formatting
…ions - Split to_inline() into separate functions for clarity - Fix doc comments to use backticks for TokenStream - Format test assertions for better readability
…m fallback Tests for tokens like `Vec<u8>` that aren't valid expressions/items: - test_tokenstream_non_expression_fallback: verifies fallback to TokenStream::to_string() - test_tokenstream_non_expression_semantic_comparison: verifies semantic comparison works
…content
Add functional tests verifying that untokenizable content inside @{}
produces compile-time errors:
- Unclosed string literal triggers E0765 error
- Unclosed delimiter triggers mismatched delimiter error
Uses cargo build -q to capture clean error output for snapshot testing.
Add a new setting `ignore_docs_for_tokens` (enabled by default) that strips `#[doc = "..."]` attributes from TokenStreams before comparison. This allows snapshots to focus on code structure without being affected by documentation changes. - Add `ignore_docs_for_tokens` field to Settings with getter/setter - Add `DocRemover` visitor and `remove_docs()` function - Integrate doc stripping into `tokens_equal()` - Add `visit-mut` feature to syn dependency - Add unit tests and functional tests
Revert accidental regression from ad13147 that added '\n' to the raw string condition. Multiline content should not use raw string prefixes unless it contains backslashes or quotes, matching upstream/master behavior from PR mitsuhiko#828.
- Update assert_token_snapshot! to accept any type implementing quote::ToTokens, not just proc_macro2::TokenStream - Add ignore_docs_for_tokens method to ActualSettings to enable use with the with_settings! macro - Update tests to use with_settings! instead of manual Settings binding - Add functional test for ToTokens support with syn types
Use &val_str[..] instead of val_str.as_str() to prevent conflicts with traits that define as_str() methods. This fixes issues when users have traits like TokensAsStr (blanket impl for ToTokens) in scope, which can shadow String's inherent as_str() method due to Rust's method resolution. Also simplifies the macro by using fully-qualified ToTokens::to_token_stream instead of importing the trait.
Simplify assert_token_snapshot! inline mode by delegating to _assert_snapshot_base! like other assertion macros do. - Remove tokens_equal() pre-check - Remove direct assert_snapshot() call - Pre-compute reference string and pass InlineValue(&ref_str) - Reduces macro from ~50 lines to ~20 lines
Remove the ($name, $value, @{ tokens }) variant since inline snapshots
don't use names - the snapshot is stored in source code, not a file.
- Explicitly declare syn "full" feature instead of relying on transitive activation via prettier-please - Remove unused tokenstream_tokens_equal export from _macro_support - Default ignore_docs_for_tokens to false (conservative default) - Fix "whitepace" typo in test name
When set to false, TokenStream snapshots use raw TokenStream::to_string() output instead of parsing and formatting with prettier-please. Defaults to true (formatted output).
Drop the doc-stripping feature entirely — it's premature without real user demand. If users want to ignore doc attributes in token snapshots, they can strip them before passing tokens to the macro. This removes the DocRemover visitor, the RemovableDocs trait, the setting, and all related tests. Also drops the syn "visit-mut" feature which was only needed for the visitor.
…ings - Remove unused `tokens_equal` function and its tests - Drop `syn` `extra-traits` feature (only needed by `tokens_equal`) - Make `tokenstream` module `pub(crate)` instead of `pub` - Fix stale comment about cargo-insta brace format behavior - Fix 35 `needless_raw_strings` clippy warnings in test code
- Fix inline snapshot comparison always falling through to legacy matching by trimming trailing newline from prettier-please output - Move InlineFormat to _cargo_insta_support instead of public API - Add CHANGELOG entry for tokenstream feature - Fix broken intra-doc link for TokenStream::to_string() - Add backticks around TokenStream in settings doc comments - Delete stale .pending-snap file
When `to_inline_tokens` indents snapshot content in source files, `/** */` block doc comments absorb that indentation as part of their string value (since `quote!` preserves whitespace verbatim). This causes inline token snapshots to never stabilize: the indentation accumulates on each accept cycle. Fix by stripping common leading whitespace from `#[doc = "..."]` attribute values (similar to what rustdoc does) before passing the AST to `prettier-please`. This makes the comparison idempotent regardless of source-level indentation.
|
we generally avoid having macros for types; instead we have them for formsts. what's the advantage of this vs serializing to yaml? |
The types you'd snapshot with this macro (e.g.
I'd argue this actually is a format macro. The conversion to/from |
- Fix clippy needless_raw_strings lint in tokenstream.rs
- Pin quote to 1.0.41 for MSRV compatibility (1.0.42 requires rustc 1.68+)
- Add RUSTFLAGS to clean_env() to prevent -D warnings from leaking into
generated test projects and causing unused import errors
- Add --color=never and env cleaning to compile-error tests that use
Command::new("cargo") directly, preventing ANSI codes in snapshots
|
Yes, I think that's a reasonable argument. But what is the boundary condition? This could be a separate project? I'd be happy to link there. Or if there's a substantial following over time, we could consider adding it to the main project; that would at least help alleviate the additional support burden... |
Summary
Adds a new
tokenstreamfeature (disabled by default) that enables snapshot testing ofproc_macro2::TokenStreamvalues:assert_token_snapshot!macro accepting anyToTokenstypeprettier-please, falls back toTokenStream::to_string()with a warning@{ ... }brace syntax for semantic token comparisonformat_tokenssetting to disable pretty-printingcargo-instaextended to detect and preserve@{ ... }brace format during updatesUsage
New dependencies (all optional, gated behind
tokenstreamfeature)syn(full, visit-mut)quoteprettier-pleaseproc-macro2Test plan
cargo nextest run --features tokenstream -p insta)