parser: close binlog-relevant statement gaps (XA, tablespace, ALTER VIEW, SRS, admin) - #1142
Draft
morgo wants to merge 35 commits into
Draft
parser: close binlog-relevant statement gaps (XA, tablespace, ALTER VIEW, SRS, admin)#1142morgo wants to merge 35 commits into
morgo wants to merge 35 commits into
Conversation
Copies github.com/block/tidb/pkg/parser @ e528fd979fc8 (upstream
pingcap/tidb master as of 2026-05-04 plus our spatial type/index
support) into pkg/parser, rewrites all imports to
github.com/block/spirit/pkg/parser, and removes the go.mod
require+replace of the external module.
goyacc (needed only to regenerate parser.go from parser.y) becomes a
nested Go module so its modernc.org dependencies stay out of spirit's
dependency graph; its one import of the parser's format package is
replaced by a local copy of the 3-line Formatter interface.
Two mechanical fixes for Go 1.26 vet (which go test enforces):
Errorf(rangeErrMsg) -> Errorf("%s", ...) in parser.y/parser.go, and an
unused Sprintf result in digester_test.go.
Content is otherwise verbatim from the fork so that follow-up commits
that strip unused functionality have reviewable diffs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cases Grammar: remove TiDB-only statements (ADMIN, BRIE, IMPORT INTO, placement policies, resource groups, bindings, stats ops, FLASHBACK, SPLIT REGION, sequences, traffic, batch DML, procedures, HELP...) and in-statement features (TiFlash replicas, AUTO_RANDOM, SHARD_ROW_ID_BITS, TTL, clustered/global index, AS OF, TABLESAMPLE, vector/columnar/HYPO indexes, partial indexes), plus MariaDB-isms (PAGE_CHECKSUM, PAGE_COMPRESSED, TRANSACTIONAL, SEQUENCE= table option, IETF_QUOTES). MySQL surface is kept in full, including ALTER TABLE ... ANALYZE PARTITION, account management, FLUSH, KILL, EXPLAIN [ANALYZE|FOR CONNECTION], LOAD DATA, LOCK TABLES, prepared statements, BINLOG, and SECONDARY_LOAD/UNLOAD. Keywords: drop 277 keyword tokens that no surviving grammar rule references (TiDB/MariaDB statement vocabulary plus builtin-function names that reach the grammar through btFuncTokenMap); these words now lex as plain identifiers, which is strictly more MySQL-compatible. Keyword-list %prec annotations (DATE/TIME/TIMESTAMP vs string literals, PASSWORD/REUSE vs eq) are preserved. keywords.go regenerated. Lexer: drop the /*T![feature] */ TiDB special-comment machinery (now a plain comment, as MySQL treats it), the AS OF / TO TSO / TO TIMESTAMP multi-token lookahead, the &^ operator, and the CREATE BINDING hint special case. MEMBER OF handling is kept. Parse table shrinks from 3,797,221 to 1,722,509 entries (-55%); parser.y from 17.7k to 12k lines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Fold test_driver into ast: concrete ValueExpr/ParamMarkerExpr/Datum/MyDecimal replace the hook-var driver registration; drop blank imports across spirit. - Remove TiDBKeyword grammar section, BEGIN OPTIMISTIC/PESSIMISTIC, CAUSAL CONSISTENCY, BINLOG MONITOR (MariaDB), TOKUDB row formats, SetMariaDB mode, and the /*T![feature_id] special-comment machinery. - Trim terror to the error-class registry the parser needs; drop pingcap/log + zap from our code (remaining go.mod entries are transitive via go-mysql-org/go-mysql). - Delete auth crypto (keep UserIdentity/RoleIdentity), digester, ast/util.go. - Prune tests of removed features; keyword consistency tests updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports from pingcap/tidb pkg/parser commits since our fork base: - e2b6ce7333: bound parentheses nesting (10000) and AST depth to prevent parser DoS via deeply nested expressions (also covers the optimizer-hint scanner) - 551d10a652: INSERT ... VALUES/SET row aliases (MySQL 8.0.19+), e.g. INSERT INTO t VALUES (1,2) AS new(m,n) ON DUPLICATE KEY UPDATE; rejected for REPLACE like MySQL - b254c43931: dual-password syntax (MySQL 8.0.14+): ALTER USER ... RETAIN CURRENT PASSWORD / DISCARD OLD PASSWORD and SET PASSWORD ... RETAIN CURRENT PASSWORD, with grammar-level enforcement that RETAIN requires a cleartext (BY-form) password and CREATE USER accepts neither clause - 9bbc86e96e: SET_VAR optimizer hint accepts decimal/float values, e.g. /*+ SET_VAR(optimizer_prune_level=0.3) */; other hints still reject non-integer numerics Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…itted goyacc binary - untrack pkg/parser/goyacc/goyacc (4.6MB compiled binary committed by mistake in the fork commit); ignore it plus y.output - delete .editorconfig, test.sh (TiDB CI leftovers) and the duration package (nothing imports it since TTL grammar removal) - remove the plan-cache HashEquals machinery (util/hash64.go, Hash64/ Equals methods on FieldType, CIStr, SelectLockInfo) - remove masking-policy AST nodes and grammar scaffolding (block-TiDB extension; grammar rules were already gone) - orphan-token sweep: drop ATTRIBUTES, CLUSTER, COLUMNAR, LABELS, TTL, VECTOR, TIDB_CURRENT_TSO tokens and the vector type/index plumbing (TypeTiDBVectorFloat32, ETVectorFloat32, IndexTypeVector/HNSW/Hypo/ Inverted) - remove sequence functions (NEXTVAL/LASTVAL/SETVAL, NEXT VALUE FOR): MariaDB/TiDB feature, MySQL has no sequences - KILL: drop the KILL TIDB extension and TiDBExtension AST field - ShowStmt: drop 38 Show* types whose grammar is gone (stats, bindings, placement, import, region, sequence, procedure-status etc.) plus dead fields and the unused NeedLimitRSRow helper - drop the TiDB warning for CREATE/ALTER USER WITH <resource-options> (valid MySQL syntax, parse it silently) - remove TiDBStrictIntegerDisplayWidth global: display widths now always round-trip, matching the flag's default behavior - trim goleak ignores for dependencies we no longer have Parse table: 1,611,644 -> 1,566,523 entries; keywords 507 -> 496. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…code Second brutal-pass batch over the forked parser: - mysql/const.go: drop the TiDB version/server-build machinery (removes the coreos/go-semver direct dependency), client/server protocol capability flags, Com* command bytes, cursor types, and other wire-protocol constants the parser never touches. - mysql/type.go, util.go, error.go, charset.go: drop TypeInt24 bounds, IsAuthPluginClearText, ErrBadConn/ErrMalformPacket, and a dead collation alias. - mysql/locale_format.go: delete (locale-aware number formatting, unused). - ast: remove the expression-flag system (flag.go, SetFlag/GetFlag on ExprNode, FlagHas* consts). Only flag.go itself ever read these flags; removing it also saves a full-AST visitor walk per parse. - main_test.go: goleak ignores for long-gone dependencies removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These are fresh errors with no wrapping or stack-trace semantics, so the stdlib is equivalent. Removes spirit-proper's last direct use of pingcap/errors outside the forked parser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ire token - hintparser.y: keep only MySQL 8.0 hint names (plus the SEMIJOIN strategies). TiDB hints (INL_JOIN, MEMORY_QUOTA, READ_FROM_STORAGE, LEADING, USE_TOJA, QUERY_TYPE, ...) now degrade to the generic "unsupported hint" warning or a hint syntax error; the statement itself still parses either way. Hint parse table shrinks 28,934 -> 6,235 entries. The TiDB partition qualifier in hint tables and the MB/GB/TRUE/FALSE/TIKV/TIFLASH/OLAP/OLTP helper tokens are gone too. - ast: remove LeadingList/FlattenLeadingList/HintTimeRange and the Restore cases for removed hints; HintTable loses PartitionList. - errname: hint warning no longer says "by TiDB"; drop the MEMORY_QUOTA overflow error (8063). - keywords.go/generate_keyword: deleted. parser.Keywords was only read by its own tests; nothing in spirit consumes it. - parser.y/misc.go: rename the REQUIRE token from require to requireKwd so internal tests can import testify's require package without aliasing (consistent_test.go, lexer_test.go, reserved_words_test.go de-aliased). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The terror package (an error-class registry from TiDB) and the
github.com/pingcap/errors dependency are gone:
- mysql.ParseError is the one typed error now: a sentinel template
carrying a MySQL error code, instantiated with GenByArgs (standard
template) or GenByFormat (custom message). Instances match their
sentinel through errors.Is, replacing terror.ErrorEqual and
*Error.Equal. Rendering is unchanged ("[parser:1064]..."), so
messages and tests are stable.
- GenWithStackByArgs/FastGenByArgs -> GenByArgs, GenWithStack ->
GenByFormat. No caller inspected stacks; the names now say what
the methods do.
- errors.Annotate/Annotatef -> fmt.Errorf("...: %w", err);
errors.Trace(err) -> err; errors.Errorf -> fmt.Errorf;
errors.New -> stdlib.
- errname.go drops the ErrMessage/redaction indirection: MySQLErrName
is a plain map[uint16]string (no entry ever set RedactArgPos).
- mysql.SQLError/NewErr/NewErrf and mysql/state.go had no callers
left and are deleted.
- goyacc (nested module) converted too; its go.mod no longer needs
pingcap/errors.
github.com/pingcap/* and go.uber.org/* now appear in spirit's go.mod
only as // indirect via go-mysql-org/go-mysql.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The charset name/collation catalog is retained in full: any MySQL DDL can still reference gbk, gb18030, big5, etc. and parse correctly. What is removed is the byte-level transcoding support that only mattered when the *client* connection charset was gbk/gb18030 (a TiDB-oriented feature; spirit always connects with utf8mb4): - delete encoding_gbk.go, encoding_gb18030.go, encoding_gb18030_data.go (~76KB of tables) and the x/text encoding lookup in encoding_table.go - FindEncoding now falls back to the pass-through binary encoding for any charset without a transcoder, the same behavior big5/latin2/etc. always had - simplify encodingBase.Foreach: the GB18030-specific transformer hack is gone - drop AddCharset/RemoveCharset/AddCollation/AddSupportedCollation: dynamic charset registration is a TiDB experimental feature with no MySQL equivalent; the collation registry is now built once in init() - remove GBK-dependent tests and the dead gbkEncodingChecker, subqueryChecker and windowFrameBoundChecker test helpers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PARTITION BY SYSTEM_TIME (plus HISTORY/CURRENT partition definitions) is MariaDB system-versioning syntax, and RANGE ... INTERVAL (...) FIRST/LAST PARTITION LESS THAN is TiDB's interval partitioning; neither exists in MySQL. Removes the grammar rules, the SYSTEM_TIME token, PartitionInterval/PartitionIntervalExpr/PartitionDefinitionClauseHistory AST nodes, the PartitionMethod Unit/Limit/Interval fields, the now-unused MariaDB error-code block, and the corresponding pkg/statement handling. Parse table shrinks from 1,566,523 to 1,542,591 entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stem golangci-lint (uncapped) reported 518 issues across pkg/parser. This fixes all of them; `golangci-lint run` is now clean repo-wide with no warnings. The biggest change: ExprNode.Format(io.Writer) and its 33 implementations are deleted. It was TiDB's legacy pretty-printer, fully parallel to Restore (which spirit uses), had no consumers, and accounted for ~70 of the unchecked-error findings on its own. Op.Format and format_test.go go with it; the fulltext tests now assert on Restore output. Everything else, by linter: - staticcheck/ST1005 (309): lowercase the upstream "An error occurred" message family; the sql_mode message keeps MySQL's capitalization with a nolint - errcheck: propagate Restore errors in ddl.go/dml.go; explicitly ignore builder writes in format.go; require.NoError in tests - exhaustive: //nolint:exhaustive on render-if-set switches, matching the existing repo convention - unused (21): delete isAllPlacementOptions, the __DEPRECATED_* enum tombstones, showTpCount, exprCleaner, lazyBuf, setKeepHint, eof, and dead test helpers - errorlint: errors.Is/errors.As for strconv.ErrRange and parserDepthLimitError (drops the pingcap-era Cause() method) - gocritic/QF*: if-else chains to switches (labeled where break targets the loop), embedded-field selectors simplified, assignOp/valSwap/ newDeref cleanups - testifylint/modernize: mechanical assertion and range-over-int fixes - remove stale '//nolint: all_revive' directives (unknown-linter warning) - gofmt: import ordering in pkg/fmt, pkg/lint, pkg/migration left over from the fork's import rewrite Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deletes everything the deadcode tool and a TiDB-reference audit turned up: - Dead statement nodes never constructed by the grammar: TraceStmt, SetSessionStatesStmt, StringOrUserVar, ShowSlow(+Type/Kind consts), TableNameExpr, StatisticsSpec/StatsType*, StatementScope. - TiDB-only grammar: FLUSH CLIENT_ERRORS_SUMMARY, FLUSH TIDB PLUGINS, and the ILIKE operator (PatternLikeOrIlikeExpr renamed back to PatternLikeExpr, IsLike discriminator dropped). Parse table shrinks 1,542,591 -> 1,522,413 entries. - Dead API: ast.NewDatum/NewBytesDatum/NewStringDatum/MakeDatums, GetStmtLabel, TrimComment(+regexps), CharsetClient, ColumnChoice, SensitiveStmtNode/SecureText, FieldType.PartialEqual, charset GetSupportedCharsets/GetSupportedCollations/GetDefaultCharsetAndCollate/ GetCharsetInfoByID/GetCollationByID(+backing maps), mysql CharsetNameToID/CharsetIDs/Collations/CollationNames/RangeGraph, Del/SetSQLMode, FormatSQLModeStr(+CombinationSQLMode), Str2Priority, seven unused type-flag helpers, system-table name consts. - TiDB builtin-function name consts (tidb_*, vitess_hash, format_nano_time, current_resource_group); DATE/TIME/TIMESTAMP literal markers rebranded 'tidb` -> 'spirit`. - LoadDataStmt loses the TiDB FORMAT field; FileLocServerOrRemote renamed FileLocServer. - utf8mb4_zh_pinyin_tidb_as_cs collation and TiFlashSupportedCharsets removed; TiDB-flavored comments reworded throughout. - reserved_words_test.go: TiDBKeyword marker removed, stale exceptions dropped, MYSQL_DSN override added; now passes against MySQL 8.0.45. deadcode -test ./... now reports only the four interface marker methods; golangci-lint clean; all parser + dependent tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pkg/parser/README.md now documents the hard fork: credit to PingCAP for creating and maintaining the parser, the fork base, what was stripped and why, usage, goyacc regeneration, and the reserved-words test that compares the grammar against a live MySQL server. Root README, AGENTS.md, and pkg/statement/README.md now point at pkg/parser instead of the external TiDB parser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the govulncheck CI failure: GO-2026-6090 (crypto/tls) and GO-2026-5972 (encoding/asn1) are both stdlib vulnerabilities fixed in go1.26.6. Bumps the go.mod directive, the govulncheck and linter workflow pins, and the Dockerfile base images. The release workflow already follows go.mod via go-version-file. govulncheck ./... now reports 0 vulnerabilities affecting our code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MySQL LIST partitioning has no DEFAULT partition; that is a MariaDB
feature TiDB adopted. Removes the bare 'PARTITION p DEFAULT' clause,
DEFAULT inside VALUES IN (...) (DefaultOrExpression{,List} rules), the
matching Restore/Validate special cases, and verifies MySQL LIST
COLUMNS row-constructor syntax still parses. Parse table shrinks to
1,519,313 entries.
Also deletes orphaned comments left by earlier removals (IF NOT EXISTS
on Constraint/CreateIndexStmt, 'MariaDB specific options' label) and
points the INTERSECT/EXCEPT docs at MySQL 8.0.31+ set-operations
documentation instead of the MariaDB knowledge base.
The only remaining MariaDB mention in code is the lexer comment
explaining that /*M! comments are skipped, which is MySQL-compatible
behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… 2b285ed389e0) The separator literal in OptGConcatSeparator was constructed with an empty charset/collation instead of the connection's. Port the upstream fix: thread parser.charset/parser.collation through, and restore the separator with RestoreStringWithoutCharset since the grammar only accepts a plain string literal after SEPARATOR (a charset introducer there would not re-parse). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MySQL 8.0.13+ distinguishes literal defaults (DEFAULT '{}') from
expression defaults (DEFAULT ('{}')); BLOB/TEXT/JSON/GEOMETRY columns
only accept the parenthesized form. The grammar discarded the
parentheses ('(' SignedLiteral ')' -> $2), so the --statement path
restored ADD COLUMN j JSON DEFAULT ('{}') as DEFAULT '{}' and MySQL
rejected it with Error 1101.
Parser:
- DefaultValueExpr keeps the parentheses as an ast.ParenthesesExpr, for
both CREATE/ADD COLUMN defaults and ALTER COLUMN SET DEFAULT (expr).
- AlterTableSpec.Restore no longer re-wraps an expression that already
carries its own parentheses (single set, not two).
- Upstream still has this bug (pingcap/tidb#57768); noted in README.
Statement layer:
- isExpressionDefault treats ParenthesesExpr as an expression default,
so DefaultIsExpr now distinguishes DEFAULT ('x') from DEFAULT 'x'.
- Extraction unwraps the parentheses before reading the value (emission
re-adds them from DefaultIsExpr), and emission quotes string-valued
expression defaults: DEFAULT ('{}').
- lint_zero_date unwraps parentheses so DEFAULT ('0000-00-00') is still
caught.
Validated against MySQL 8.0.45: the issue's exact statement now
round-trips and executes, and a table created via the restored DDL
extracts identically from SHOW CREATE TABLE (which renders these as
DEFAULT (_utf8mb4'{}')), so declarative diffs converge. Note MySQL
folds ALTER COLUMN ... SET DEFAULT (literal) into a plain literal
default server-side; spirit only emits MODIFY COLUMN, so this does not
create diff loops.
Fixes block#542.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test documented the old TiDB-parser limitation: ALTER ... ADD COLUMN
c BLOB DEFAULT ('abc') via --statement was expected to fail with Error
1101 because the restore dropped the parentheses. The parser fork
preserves them (block#542), so that path now succeeds — assert success and
keep the --table/--alter variant as a separate column. CREATE TRIGGER
remains unparsable and still asserts an error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Functions whose names lex as keyword tokens rather than plain
identifiers were unparsable in two overlapping ways:
1. DEFAULT expressions only accepted raw-identifier function names
(upstream special-cased just REPLACE), so DEFAULT (point(0,0)) —
the MySQL manual's own expression-default example — failed while
SELECT point(0,0) parsed fine.
2. The spatial constructors other than POINT (LINESTRING, POLYGON,
MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, GEOMETRYCOLLECTION)
became keyword tokens with the GIS fork but were never added to
FunctionNameConflict, so they failed as function calls in *every*
expression context: SELECT lists, generated columns, and CHECK
constraints, not just defaults.
MySQL accepts all of these forms and SHOW CREATE TABLE emits them, so
an affected table was invisible to spirit everywhere — table-info
loading and declarative diffs, not just --statement rewriting.
Fix: split FunctionNameConflict into FunctionNameConflictNonNow +
builtinNow, add the six spatial constructors to it, and let
BuiltinFunction (the DEFAULT-expression grammar) accept
FunctionNameConflictNonNow '(' ExpressionListOpt ')' in place of the
old REPLACE special case. NOW stays excluded from the DEFAULT path so
DEFAULT (now()) keeps folding to CURRENT_TIMESTAMP. Grammar regen is
conflict-free (parse table 1,519,313 -> 1,532,595 entries, +0.9%).
Validated against MySQL 8.0.45: every fixed form executes, spirit's
restored output executes and converges with SHOW CREATE TABLE of the
original, and MySQL's own SHOW CREATE output (charset introducers
included) parses and round-trips stably.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports pingcap/tidb@52b9a887b3b0 ("parser, bindinfo: normalize binding parentheses with AST restore"): the opt-in RestoreSkipRedundantParentheses restore flag, the RestoreCtx plumbing it needs (ParentBinaryOp, ParentBinarySide, InUnaryOperation), and the MySQL precedence table that decides whether a ParenthesesExpr can be dropped in its position. With the flag set, `a + (b * c)` restores as `a + b * c` while `(a + b) * c` and `a - (b - c)` keep their parentheses; same-precedence right children only drop them for operators that regroup safely (AND/OR/XOR, bitwise and logical), never for arithmetic. Unary operands, unknown operators, and anything below a kept pair of parentheses are treated conservatively. Default restore behaviour is unchanged — the flag is opt-in, so `DEFAULT ('{}')` keeps its parentheses as before. Adapted for the fork: stdlib wrapped errors, PatternLikeExpr in place of upstream's PatternLikeOrIlikeExpr, and //nolint:exhaustive on the two opcode switches that deliberately fall through to a default. The tests are ours; upstream's cover the flag through binding normalization, which this fork does not have. Each case asserts the canonical text, that re-restoring it is a fixed point, and — outside the deliberate reassociation cases — that the text parses back to the same expression structure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The expression normalizer stripped every parenthesis and re-wrapped every operator node, so a CHECK constraint converged with MySQL's stored form as `CHECK ((`a`=1) OR ((`b`=2) AND (`c`=3)))` — stable, but not DDL anyone would write. It now renders the same tree with RestoreSkipRedundantParentheses, giving `CHECK (`a`=1 OR `b`=2 AND `c`=3)`. The strip-and-wrap pass stays: it is what makes the form canonical, by erasing the input's parenthesization before anything is rendered. Restore never invents parentheses, so without it `-(a)` and `-a`, or `f((a + b))` and `f(a + b)`, would each canonicalize two ways — and MySQL emits the first of each pair. What the new flag adds is only how much of the (already canonical) structure has to be spelled out. Also fixes a pre-existing gap in that pass: MEMBER OF, quantified comparisons (`= ANY (...)`), and COLLATE were not re-wrapped, so `a = (1 MEMBER OF (j))` rendered as `a = 1 MEMBER OF (j)`, which reads left to right as the different `(a = 1) MEMBER OF (j)`. Every node whose parentheses the renderer reasons about is now wrapped. Verified against MySQL 8.0.45: TestRoundTrip_ExpressionParenShapes runs 28 expression shapes through both directions of the round trip — the shape as written must converge with MySQL's stored form, and the canonical text must apply as DDL and converge after MySQL stores it — which is what checks the ported precedence table against MySQL's own. Generated-column expressions get the same treatment. Two shapes are documented as out of reach because MySQL rewrites the expression itself when it stores it (De Morgan on NOT, REGEXP into REGEXP_LIKE). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MyDecimal.FromString panicked on any literal whose digits exceeded the 9-word buffer, e.g. SELECT with a 90-digit number — valid MySQL that can appear in binlogged statements. pkg/change parses every binlog Query event with no recover, so one such statement killed the process. Port upstream TiDB's clamping instead: an oversized integer part returns ErrDataOutOfRange (toDecimal already clamps those to the max decimal value with a warning, like MySQL), and an oversized fraction is truncated to the words that remain, now surfaced as a warning too. The remaining panic branches (empty input, no digits, scientific notation) are unreachable from the lexer and now return an error instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All of these are valid MySQL that is written to the binary log as Query
events, so pkg/change must at least parse them:
- REPAIR [NO_WRITE_TO_BINLOG|LOCAL] TABLE(S) ... [QUICK][EXTENDED][USE_FRM]
- RENAME TABLES and ANALYZE TABLES plural spellings
- ANALYZE ... UPDATE HISTOGRAM: MANUAL/AUTO UPDATE (8.4+) and USING DATA
- BEGIN/COMMIT/ROLLBACK WORK spellings; START TRANSACTION with
comma-separated characteristics
- FLUSH USER_RESOURCES / OPTIMIZER_COSTS / RELAY LOGS [FOR CHANNEL] /
TABLES ... FOR EXPORT
- ALTER DATABASE ... READ ONLY = {0|1|DEFAULT}
- ALTER INSTANCE: ROTATE {INNODB|BINLOG} MASTER KEY, RELOAD TLS FOR
CHANNEL, {ENABLE|DISABLE} INNODB REDO_LOG, RELOAD KEYRING
New keyword tokens follow live-MySQL reservation status (OPTIMIZER_COSTS,
SYSTEM and UNDO are reserved there; the rest are unreserved), keeping
reserved_words_test green. mysql-test corpus pass rate: 90.55% -> 90.72%.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These statement classes appear in the binary log (or, for XA, in
workloads spirit must detect and refuse) but previously failed to
parse, which blocks making pkg/change strict about unparseable
statements:
- ALTER VIEW (with algorithm/definer/security/check option)
- XA {START|BEGIN|END|PREPARE|COMMIT|ROLLBACK|RECOVER}, including
JOIN/RESUME/SUSPEND [FOR MIGRATE]/ONE PHASE/CONVERT XID and
string/hex xids with an optional formatID
- CREATE [OR REPLACE] / DROP SPATIAL REFERENCE SYSTEM
- CREATE/ALTER/DROP [UNDO] TABLESPACE and CREATE/ALTER/DROP
LOGFILE GROUP with the size/engine/wait option set
mysql-test corpus pass rate: 90.72% -> 91.39% (18274 -> 16953
failures out of 196996 statements).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both switches intentionally rely on default: the first errors on the invalid zero value, the second prints all size options one way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # pkg/migration/singleversion_test.go
Review feedback on block#1126: the parser README claimed CI verifies parser.go/hintparser.go are in sync with the grammar, but no workflow did. Add a parser-regen job that deletes the generated files, rebuilds them with make, and fails on any diff, and update the README to describe it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The %r verb rework (block#1037) landed on main using errors.Errorf; this branch removed the pingcap/errors import from sqlescape as part of the fork, so the merge compiled against a missing identifier. Use fmt.Errorf like the rest of the file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR
Stacked on #1126 → #1134. Only the top 3 commits are new here (
575ae4a1,9c768c74,b235f016); everything below them is the parser hard fork and paren canonicalization under review in those PRs.Why
pkg/changecurrently ignores binlog Query events it can't parse. To make that strict (fail on unparseable statements) the forked parser needs to recognize everything MySQL writes to the binary log as a statement. An audit of the fork against the MySQL 9.6 grammar (sql_yacc.yy) and the full mysql-test suite corpus found a parse-time panic plus a set of missing statement classes. This PR fixes the panic and closes the statement-class gaps, except stored programs (see scope note below).What's in it
P0 — decimal literal panic (
575ae4a1): a literal wider than 81 digits (e.g.SELECT 1e0 + <82 nines>) panickedMyDecimal.FromStringinstead of returning an error. Ported the upstream TiDB clamping behavior: out-of-range literals clamp to the max decimal with a warning, over-precision fractions truncate with a warning. A strict pkg/change must never panic on server-accepted input.Grammar gaps in existing statements (
9c768c74):REPAIR [NO_WRITE_TO_BINLOG|LOCAL] TABLE ... [QUICK|EXTENDED|USE_FRM]RENAME TABLESspelling;ANALYZE TABLEhistogram clauses (UPDATE HISTOGRAM ... [USING DATA '...'|MANUAL/AUTO UPDATE],DROP HISTOGRAM)BEGIN/COMMIT/ROLLBACK WORK,START TRANSACTIONoption lists (READ ONLY,READ WRITE,WITH CONSISTENT SNAPSHOT)FLUSH USER_RESOURCES | OPTIMIZER_COSTS | RELAY LOGS [FOR CHANNEL ...],FLUSH TABLES ... FOR EXPORTALTER DATABASE ... READ ONLYALTER INSTANCE(RELOAD TLS [FOR CHANNEL] [NO ROLLBACK ON ERROR], RELOAD KEYRING, ROTATE INNODB/BINLOG MASTER KEY, ENABLE/DISABLE INNODB REDO_LOG)New statement classes (
b235f016):ALTER VIEW(algorithm/definer/SQL security/check option)XA START/BEGIN/END/PREPARE/COMMIT/ROLLBACK/RECOVERincl.JOIN,RESUME,SUSPEND [FOR MIGRATE],ONE PHASE,CONVERT XID, string/hex xids, and hex formatIDs. spirit refuses XA workloads (Refuse XA workloads in the binlog stream instead of applying prepared rows #1079), but refusing cleanly requires recognizing the statements rather than failing to parse them.CREATE [OR REPLACE] / DROP SPATIAL REFERENCE SYSTEMCREATE/ALTER/DROP [UNDO] TABLESPACE,CREATE/ALTER/DROP LOGFILE GROUPwith the size/engine/wait option setKeyword changes follow live MySQL:
OPTIMIZER_COSTS,SYSTEM, andUNDObecome reserved (they are reserved in MySQL 8.0+); ~35 new unreserved keywords.INNODB/REDO_LOGare matched semantically as identifiers, mirroringsql_yacc.yy, so they stay non-keywords.Verification
go build ./...,go vet ./..., fullpkg/parser,pkg/statement,pkg/changetest suites pass; new round-trip restore tests for every added statement class.reserved_words_test(live-MySQL keyword contract) passes against mysql:8.0.45.Out of scope
Stored procedures/functions/triggers/events (compound-statement grammar — large, deliberately deferred), ACL-family clause gaps, MySQL 9.x-only features (VECTOR, LIBRARY, $-quoting), INVISIBLE columns,
DEFAULT (expr), and the charset/collation registry. These are tracked from the gap-analysis report and can land as follow-ups before pkg/change turns on strict parsing.🤖 Generated with Claude Code