feat(lineage): add TiDB and MariaDB lineage analyzers - #2
Merged
Conversation
Add backend/plugin/lineage/{tidb,mariadb}, each a dialect copy of the
omni-backed MySQL analyzer using omni's per-dialect parser and typed AST, and
register Engine_TIDB / Engine_MARIADB. OceanBase is dropped by decision and
stays unsupported.
- All three analyzers run the same golden corpus so the dialects stay
behaviorally in sync: TiDB 73/73, MariaDB 69/73.
- MariaDB records four explicit parser gaps: omni's MariaDB parser rejects join
trees nested three or more parentheses deep, which is what mysqldump emits for
views. MySQL's parser accepts any depth.
- testutil gains RunLineageTestSuitesFromYAMLDirSkipping so a dialect's parser
gaps are named in the test rather than hidden by a narrowed corpus.
- plan/mysql_family_dialect_lineage_plan.md documents the design and findings.
Verified: go build ./..., the full hermetic suite, golangci-lint (0 issues), the
release build, and make test-integration-mysql against real MySQL 8.4.
Add a golden corpus per dialect, run alongside the shared MySQL corpus, so each dialect's own syntax is exercised rather than only the dialect-neutral subset. MariaDB: system-versioned FOR SYSTEM_TIME in all four temporal forms, with and without an alias. Note the MariaDB syntax is `table FOR SYSTEM_TIME ... [AS] alias`, so the alias follows the temporal clause; a temporal base table inside a derived table, and a temporal table joined with an alias, are covered too. Also INSERT/REPLACE/DELETE ... RETURNING, FOR PORTION OF on UPDATE/DELETE, CREATE OR REPLACE VIEW, INTERSECT/EXCEPT, sequence expressions, JSON_EXTRACT, NATURAL JOIN. TiDB: READ_FROM_STORAGE / SET_VAR / HASH_JOIN hints, PARTITION selection, INSERT IGNORE, REPLACE ... SELECT, ON DUPLICATE KEY UPDATE ... VALUES(), FOR UPDATE, LOCK IN SHARE MODE, INSERT ... SELECT *, multi-table DELETE/UPDATE, INTERSECT/EXCEPT, CREATE OR REPLACE VIEW. Both dialect packages now run two corpora (shared + own) via a dialectCorpusDir helper. Every form was validated against a real MariaDB 11.8 server. One omni MariaDB parser gap remains and is deliberately not covered: applying FOR SYSTEM_TIME to a derived table (`FROM (subquery) FOR SYSTEM_TIME ... AS alias`), which MariaDB accepts but omni rejects.
Ranxy
force-pushed
the
tidb_mariadb_lineage_omni
branch
from
September 14, 2026 09:12
009f78a to
ec8091b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add TiDB and MariaDB column-level lineage, one package each, as dialect copies of the omni-backed MySQL analyzer, plus a golden corpus per dialect for its own syntax. OceanBase is dropped by decision and stays unsupported.
Design + findings:
plan/mysql_family_dialect_lineage_plan.md.Changes
backend/plugin/lineage/tidb— omnitidb/{parser,ast}, registersEngine_TIDB.backend/plugin/lineage/mariadb— omnimariadb/{parser,ast}, registersEngine_MARIADB.backend/server/ultimate.goblank-imports both, soMYSQL/TIDB/MARIADBresolve to their analyzers;OCEANBASEstaysErrorEngineNotSupported(recorded skip).testutil.RunLineageTestSuitesFromYAMLDirSkippinglets a dialect name the corpus cases its parser cannot handle.Dialect-specific coverage
Every case carries full expected edges (source/target column, relation type, transform, temp).
MariaDB (
mariadb/testdata/analyze/, 20 cases): system-versionedFOR SYSTEM_TIMEin all four temporal forms (AS OF/ALL/FROM..TO/BETWEEN..AND), with and without an alias, a temporal table joined with an alias, a temporal base table inside a derived table,INSERT/REPLACE/DELETE ... RETURNING,UPDATE/DELETE ... FOR PORTION OF,CREATE OR REPLACE VIEW,INTERSECT/EXCEPT, sequence expressions,JSON_EXTRACT,NATURAL JOIN.TiDB (
tidb/testdata/analyze/, 15 cases):READ_FROM_STORAGE/SET_VAR/HASH_JOINhints,PARTITION (...)selection,INSERT IGNORE,REPLACE ... SELECT,ON DUPLICATE KEY UPDATE ... VALUES(),FOR UPDATE,LOCK IN SHARE MODE,INSERT ... SELECT *, multi-tableDELETE/UPDATE,INTERSECT/EXCEPT,CREATE OR REPLACE VIEW.Why separate packages (copies), not one shared analyzer
omni ships a separate parser + AST per dialect, and the ASTs already diverge (MariaDB adds
Returning/ForPortionOf/ temporalSystemTime; TiDB omitsTableSource/ValuesSource/Quantifier). Every difference is an addition or removal of a field the analyzer does not read, so the traversal is source-compatible — but expressing it as a single implementation would require an adapter/IR layer that the MySQL migration deliberately avoided. The corpora are the behavioral sync guard.Known omni MariaDB parser gaps (verified against MariaDB 11.8)
Both are parser gaps, not analyzer gaps, and both are left uncovered explicitly rather than hidden.
Join trees nested three or more parentheses deep, which is what mysqldump emits for views:
FROM (t1 JOIN t2 ON ...)FROM ((t1 JOIN t2 ON ...))FROM (((t1 JOIN t2 ON ...)))FROM (((t1 JOIN t2 ON ...)) LEFT JOIN t3 ON ...)The four affected shared-corpus cases are listed in
mariadb/analyze_test.go(knownParserGaps).FOR SYSTEM_TIMEapplied to a derived table, which MariaDB accepts:FROM (SELECT ...) FOR SYSTEM_TIME AS OF TIMESTAMP '...' AS x→ omni rejects it withexpected UPDATE or SHARE after FOR. The supported forms — including a temporal base table inside the derived table — are covered above.Under the hard-fail parse policy, statements hitting either gap get no lineage until omni's parser is fixed. Worth filing upstream.
Verification
go test ./...(hermetic) green: MySQL 73/73; TiDB 73/73 shared + 15/15 dialect; MariaDB 69/73 shared (4 recorded gaps) + 20/20 dialect.golangci-lint0 issues;gofmtclean; release build ok.make test-integration-mysqlagainst real MySQL 8.4: 5/5.go.mod/go.sumchange — the dialect parsers live in the already-pinned omni module and add no dependencies.Follow-ups
knownParserGapsentries once fixed (the shared corpus returns to 73/73).RETURNING) is not needed for parity: the base DML lineage is produced andRETURNINGis ignored.