reduce the amount of panics in {TokenStream, Literal}::from_str calls#147859
Open
cyrgani wants to merge 1 commit intorust-lang:mainfrom
Open
reduce the amount of panics in {TokenStream, Literal}::from_str calls#147859cyrgani wants to merge 1 commit intorust-lang:mainfrom
{TokenStream, Literal}::from_str calls#147859cyrgani wants to merge 1 commit intorust-lang:mainfrom
Conversation
Collaborator
|
rust-analyzer is developed in its own repository. If possible, consider making this change to rust-lang/rust-analyzer instead. cc @rust-lang/rust-analyzer |
rustbot
added
S-waiting-on-review
Collaborator
|
r? @davidtwco rustbot has assigned @davidtwco. Use |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Contributor
Author
|
@rustbot author |
rustbot
added
S-waiting-on-author
Collaborator
|
Reminder, once the PR becomes ready for a review, use |
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
e2c0009 to
80b67bb
Compare
Contributor
Author
|
@rustbot ready |
rustbot
added
S-waiting-on-review
bjorn3
reviewed
Oct 20, 2025
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
ba098bc to
0274d78
Compare
Contributor
Author
|
This does not fix #58736 entirely yet: TokenStream::from_str("0b2");is still a compile error because it internally calls |
Member
|
I'm not familiar enough with this part of the compiler r? compiler |
cyrgani
force-pushed
the
nonfatal-tokenstream-parse
branch
from
0274d78 to
13d3d06
Compare
This comment has been minimized.
This comment has been minimized.
cyrgani
changed the title
TokenStream::from_str panics with LexErrors{TokenStream, Literal}::from_str calls
jhpratt
added a commit
to jhpratt/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
jhpratt
added a commit
to jhpratt/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
Zalathar
added a commit
to Zalathar/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
JaclynCodes
added a commit
to JaclynCodes/rust
that referenced
this pull request
Nov 17, 2025
add larger test for `proc_macro` `FromStr` implementations Currently, there are only few tests that check the output of `TokenStream::from_str` and `Literal::from_str` (which is somewhat understandable as the rustc implementation just delegates these calls to the parser). In preparation for both the standalone backend (rust-lang#130856) which will probably need to reimplement this logic as well as for removing panics from these functions (rust-lang#58736), this PR adds a test which shows the various messy ways of how these functions report errors and the return values for successful parses. Followup PRs such as rust-lang#147859 will change more and more of these "diagnostic + error"s into `LexErrors`. The test structure with the extra module is used to allow reusing it later easily for the standalone backend.
This comment has been minimized.
This comment has been minimized.
rust-cloud-vms
bot
force-pushed
the
nonfatal-tokenstream-parse
branch
from
33ff2a3 to
519a3f7
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
rust-cloud-vms
bot
force-pushed
the
nonfatal-tokenstream-parse
branch
from
519a3f7 to
cabd6e7
Compare
Collaborator
|
This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
cyrgani
added
S-waiting-on-review
petrochenkov
added
S-waiting-on-author
rust-cloud-vms
bot
force-pushed
the
nonfatal-tokenstream-parse
branch
from
cabd6e7 to
f1c80f5
Compare
cyrgani
added
S-waiting-on-review
petrochenkov
added
T-libs-api
Member
|
We reviewed this in today's @rust-lang/libs-api meeting. This change seems reasonable. We'd like to see it run through crater, though. Separate from that, in a different PR, we would also like to see a change to remove the case that does rustc fixups and returns a valid token stream but sets a compile error in the background, and make that produce an error all the time. |
Contributor
|
@bors try |
rust-bors bot
pushed a commit
that referenced
this pull request
Feb 17, 2026
reduce the amount of panics in `{TokenStream, Literal}::from_str` calls
This comment has been minimized.
This comment has been minimized.
Contributor
Contributor
|
@craterbot check |
Collaborator
|
👌 Experiment ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more |
craterbot
added
S-waiting-on-crater
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this PR, calling
TokenStream::from_strorLiteral::from_strwith an invalid argument would always cause a compile error, even if theTokenStreamis not used afterwards at all.This PR changes this so it returns a
LexErrorinstead in some cases.This is very theoretically a breaking change, but the doc comment on the impl already says
Fixes some cases of #58736.