Skip to content

feat: add TokenStream snapshot support#884

Open
gaetschwartz wants to merge 24 commits intomitsuhiko:masterfrom
gaetschwartz:feat/tokenstream-snapshot-support
Open

feat: add TokenStream snapshot support#884
gaetschwartz wants to merge 24 commits intomitsuhiko:masterfrom
gaetschwartz:feat/tokenstream-snapshot-support

Conversation

@gaetschwartz
Copy link
Copy Markdown

Summary

Adds a new tokenstream feature (disabled by default) that enables snapshot testing of proc_macro2::TokenStream values:

  • New assert_token_snapshot! macro accepting any ToTokens type
  • File-based mode: pretty-prints using prettier-please, falls back to TokenStream::to_string() with a warning
  • Inline mode with @{ ... } brace syntax for semantic token comparison
  • format_tokens setting to disable pretty-printing
  • cargo-insta extended to detect and preserve @{ ... } brace format during updates
  • Doc attribute whitespace normalization for stable inline round-trips

Usage

// File-based
assert_token_snapshot!(tokens);

// Inline with semantic comparison
assert_token_snapshot!(tokens, @{ struct Foo; });

// Multiline
assert_token_snapshot!(tokens, @{
    fn hello() {
        println!("world");
    }
});

New dependencies (all optional, gated behind tokenstream feature)

  • syn (full, visit-mut)
  • quote
  • prettier-please
  • proc-macro2

Test plan

  • 125 unit tests pass (cargo nextest run --features tokenstream -p insta)
  • 845 lines of functional tests covering: inline update, matching, file-based, semantic comparison, multiline, format_tokens disabled, compile-time errors for invalid content, ToTokens acceptance, non-expression fallback
  • CI passes on all targets

Add `assert_token_snapshot!` macro for testing proc_macro2::TokenStream values.

- File-based mode: Pretty-prints using prettier-please, falls back to
  TokenStream::to_string()
- Inline mode with `@{...}` syntax: Compares TokenStreams semantically
  (ignoring whitespace differences)
- Extend cargo-insta to recognize @{...} brace syntax for inline updates

New files:
- insta/src/tokenstream.rs: pretty_print() and tokens_equal() helpers

Usage:
  // File-based
  assert_token_snapshot!(tokens);
  assert_token_snapshot!("name", tokens);

  // Inline with semantic comparison
  assert_token_snapshot!(tokens, @{ struct Foo; });
Add comprehensive functional tests for the tokenstream feature:
- test_tokenstream_inline_empty_to_populated: Tests @{} gets updated
- test_tokenstream_inline_matching: Tests matching tokens pass
- test_tokenstream_file_based: Tests .snap file creation
- test_tokenstream_semantic_comparison: Tests whitespace-insensitive comparison
- test_tokenstream_multiline_inline: Tests multiline token streams
…shots

Add InlineFormat enum to control inline snapshot output format:
- InlineFormat::Text: standard string literal format (@"...")
- InlineFormat::Tokens: brace format for TokenStream (@{ ... })

cargo-insta now detects the original format from the source AST and
preserves it when updating snapshots, instead of always converting
to string format.
- Fix indentation to use @{ line's leading whitespace, not macro start
- Single-line content uses compact format: @{ content }
- Multiline content properly indents: content +4 spaces, closing } aligns with @{
- Add pretty_print_for_inline() for proper multiline handling
- Add tests for same-line and separate-line @{ formatting
…ions

- Split to_inline() into separate functions for clarity
- Fix doc comments to use backticks for TokenStream
- Format test assertions for better readability
…m fallback

Tests for tokens like `Vec<u8>` that aren't valid expressions/items:
- test_tokenstream_non_expression_fallback: verifies fallback to TokenStream::to_string()
- test_tokenstream_non_expression_semantic_comparison: verifies semantic comparison works
…content

Add functional tests verifying that untokenizable content inside @{}
produces compile-time errors:
- Unclosed string literal triggers E0765 error
- Unclosed delimiter triggers mismatched delimiter error

Uses cargo build -q to capture clean error output for snapshot testing.
Add a new setting `ignore_docs_for_tokens` (enabled by default) that
strips `#[doc = "..."]` attributes from TokenStreams before comparison.

This allows snapshots to focus on code structure without being affected
by documentation changes.

- Add `ignore_docs_for_tokens` field to Settings with getter/setter
- Add `DocRemover` visitor and `remove_docs()` function
- Integrate doc stripping into `tokens_equal()`
- Add `visit-mut` feature to syn dependency
- Add unit tests and functional tests
Revert accidental regression from ad13147 that added '\n' to the raw
string condition. Multiline content should not use raw string prefixes
unless it contains backslashes or quotes, matching upstream/master
behavior from PR mitsuhiko#828.
- Update assert_token_snapshot! to accept any type implementing
  quote::ToTokens, not just proc_macro2::TokenStream
- Add ignore_docs_for_tokens method to ActualSettings to enable
  use with the with_settings! macro
- Update tests to use with_settings! instead of manual Settings binding
- Add functional test for ToTokens support with syn types
Use &val_str[..] instead of val_str.as_str() to prevent conflicts with
traits that define as_str() methods. This fixes issues when users have
traits like TokensAsStr (blanket impl for ToTokens) in scope, which can
shadow String's inherent as_str() method due to Rust's method resolution.

Also simplifies the macro by using fully-qualified ToTokens::to_token_stream
instead of importing the trait.
Simplify assert_token_snapshot! inline mode by delegating to
_assert_snapshot_base! like other assertion macros do.

- Remove tokens_equal() pre-check
- Remove direct assert_snapshot() call
- Pre-compute reference string and pass InlineValue(&ref_str)
- Reduces macro from ~50 lines to ~20 lines
Remove the ($name, $value, @{ tokens }) variant since inline snapshots
don't use names - the snapshot is stored in source code, not a file.
- Explicitly declare syn "full" feature instead of relying on
  transitive activation via prettier-please
- Remove unused tokenstream_tokens_equal export from _macro_support
- Default ignore_docs_for_tokens to false (conservative default)
- Fix "whitepace" typo in test name
When set to false, TokenStream snapshots use raw TokenStream::to_string()
output instead of parsing and formatting with prettier-please. Defaults
to true (formatted output).
Drop the doc-stripping feature entirely — it's premature without real
user demand. If users want to ignore doc attributes in token snapshots,
they can strip them before passing tokens to the macro. This removes
the DocRemover visitor, the RemovableDocs trait, the setting, and all
related tests. Also drops the syn "visit-mut" feature which was only
needed for the visitor.
…ings

- Remove unused `tokens_equal` function and its tests
- Drop `syn` `extra-traits` feature (only needed by `tokens_equal`)
- Make `tokenstream` module `pub(crate)` instead of `pub`
- Fix stale comment about cargo-insta brace format behavior
- Fix 35 `needless_raw_strings` clippy warnings in test code
- Fix inline snapshot comparison always falling through to legacy
  matching by trimming trailing newline from prettier-please output
- Move InlineFormat to _cargo_insta_support instead of public API
- Add CHANGELOG entry for tokenstream feature
- Fix broken intra-doc link for TokenStream::to_string()
- Add backticks around TokenStream in settings doc comments
- Delete stale .pending-snap file
When `to_inline_tokens` indents snapshot content in source files,
`/** */` block doc comments absorb that indentation as part of their
string value (since `quote!` preserves whitespace verbatim). This
causes inline token snapshots to never stabilize: the indentation
accumulates on each accept cycle.

Fix by stripping common leading whitespace from `#[doc = "..."]`
attribute values (similar to what rustdoc does) before passing the
AST to `prettier-please`. This makes the comparison idempotent
regardless of source-level indentation.
@max-sixty
Copy link
Copy Markdown
Collaborator

we generally avoid having macros for types; instead we have them for formsts. what's the advantage of this vs serializing to yaml?

@gaetschwartz
Copy link
Copy Markdown
Author

what's the advantage of this vs serializing to yaml?

The types you'd snapshot with this macro (e.g. syn::DeriveInput, custom AST nodes, etc.) don't implement serde::Serialize — they live in the proc-macro2/syn ecosystem, not serde. So serializing to YAML isn't an option. The macro works with any type that implements quote::ToTokens/syn::Parse, just like assert_yaml_snapshot! works with any type that implements serde::Serialize.

we generally avoid having macros for types; instead we have them for formsts.

I'd argue this actually is a format macro. The conversion to/from proc_macro2::TokenStream via quote::ToTokens/syn::Parse (and by extension to/from string) is a serialization format, just like serde's conversion to/from YAML/JSON. The macro snapshots any type that implements those traits — it's not specific to one type.

- Fix clippy needless_raw_strings lint in tokenstream.rs
- Pin quote to 1.0.41 for MSRV compatibility (1.0.42 requires rustc 1.68+)
- Add RUSTFLAGS to clean_env() to prevent -D warnings from leaking into
  generated test projects and causing unused import errors
- Add --color=never and env cleaning to compile-error tests that use
  Command::new("cargo") directly, preventing ANSI codes in snapshots
@max-sixty
Copy link
Copy Markdown
Collaborator

Yes, I think that's a reasonable argument.

But what is the boundary condition?

This could be a separate project? I'd be happy to link there. Or if there's a substantial following over time, we could consider adding it to the main project; that would at least help alleviate the additional support burden...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants