Python: Add support for PEP-758 exception syntax #20990
Draft
+45,140
−37,684
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Extends the scanner with a new token kind representing the start of a template string. This is used to distinguish template strings from regular strings (because only a template string will start with a `_template_string_start` external token). - Cleans up the logic surrounding interpolations (and the method names) so that format strings and template strings behave the same in this case. Finally, we add two new node types in the tree-sitter grammar: - `template_string` behaves like format strings, but is a distinct type (mainly so that an implicit concatenation between template strings and regular strings becomes a syntax error). - `concatenated_template_string` is the counterpart of `concatenated_string`. However, internally, the string parts of a template strings are just the same `string_content` nodes that are used in regular format strings. We will disambiguate these inside `tsg-python`.
Adds three new AST nodes to the mix: - `TemplateString` represents a t-string in Python 3.14 - `TemplateStringPart` represents one of the string constituents of a t-string. (The interpolated expressions are represented as `Expr` nodes, just like f-strings.) - `JoinedTemplateString` represents an implicit concatenation of template strings. Importantly, we _completely avoid_ the complicated construction we currently do for format strings (as well as the confusing nomenclature). No extra injection of empty strings (so that a template string is a strict alternation of strings and expressions). A `JoinedTemplateString` simply has a list of template string children, and a `TemplateString` has a list of "values" which may be either `Expr` or `TemplateStringPart` nodes. If we ever find that we actually want the more complicated interface for these strings, then I would much rather we reconstruct this inside of QL rather than in the parser.
We do the usual thing. Downgrade scripts remove the relevant relations; upgrade scripts do nothing.
Not actually based on any measurements, just the usual 100/1000 stuff.
See https://peps.python.org/pep-0758/ for more details. We implement this by extending the syntax for exceptions and exception groups so that the `type` field can now contain either an expression (which matches the old behaviour), or a comma-separated list of at least two elements (representing the new behaviour). We model the latter case using a new node type `exception_list`, which in `tsg-python` is simply mapped to a tuple. This means it matches the existing behaviour (when the tuple is surrounded by parentheses) exactly, hence we don't need to change any other code. As a consequence of this, however, we cannot directly parse the Python 2.7 syntax `except Foo, e: ...` as `except Foo as e: ...`, as this would introduce an ambiguity in the grammar. Thus, we have removed support for the (deprecated) 2.7-style syntax, and only allow `as` to indicate binding of the exception. The syntax `except Foo, e: ...` continues to be parsed (in particular, it's not suddenly a syntax error), but it will be parsed as if it were `except (Foo, e): ...`, which may not give the correct results. In principle we could extend the QL libraries to account for this case (specifically when analysing Python 2 code). In practice, however, I expect this to have a minor impact on results, and not worth the additional investment at this time.
Note in particular that the `exceptions.py` test is unaffected.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See https://peps.python.org/pep-0758/ for more details.
We implement this by extending the syntax for exceptions and exception groups so that the
typefield can now contain either an expression (which matches the old behaviour), or a comma-separated list of at least two elements (representing the new behaviour).We model the latter case using a new node type
exception_list, which intsg-pythonis simply mapped to a tuple. This means it matches the existing behaviour (when the tuple is surrounded by parentheses) exactly, hence we don't need to change any other code.As a consequence of this, however, we cannot directly parse the Python 2.7 syntax
except Foo, e: ...asexcept Foo as e: ..., as this would introduce an ambiguity in the grammar. Thus, we have removed support for the (deprecated) 2.7-style syntax, and only allowasto indicate binding of the exception. The syntaxexcept Foo, e: ...continues to be parsed (in particular, it's not suddenly a syntax error), but it will be parsed as if it wereexcept (Foo, e): ..., which may not give the correct results.In principle we could extend the QL libraries to account for this case (specifically when analysing Python 2 code). In practice, however, I expect this to have a minor impact on results, and not worth the additional investment at this time.