Skip to content

feat(lineage): add TiDB and MariaDB lineage analyzers - #2

Merged
Ranxy merged 2 commits into
mainfrom
tidb_mariadb_lineage_omni
Sep 14, 2026
Merged

feat(lineage): add TiDB and MariaDB lineage analyzers#2
Ranxy merged 2 commits into
mainfrom
tidb_mariadb_lineage_omni

Conversation

@Ranxy

@Ranxy Ranxy commented Sep 14, 2026

Copy link
Copy Markdown
Owner

What

Add TiDB and MariaDB column-level lineage, one package each, as dialect copies of the omni-backed MySQL analyzer, plus a golden corpus per dialect for its own syntax. OceanBase is dropped by decision and stays unsupported.

Design + findings: plan/mysql_family_dialect_lineage_plan.md.

Changes

  • backend/plugin/lineage/tidb — omni tidb/{parser,ast}, registers Engine_TIDB.
  • backend/plugin/lineage/mariadb — omni mariadb/{parser,ast}, registers Engine_MARIADB.
  • backend/server/ultimate.go blank-imports both, so MYSQL / TIDB / MARIADB resolve to their analyzers; OCEANBASE stays ErrorEngineNotSupported (recorded skip).
  • testutil.RunLineageTestSuitesFromYAMLDirSkipping lets a dialect name the corpus cases its parser cannot handle.
  • Each dialect runs two corpora: the shared MySQL corpus (behavioral sync guard) plus its own dialect corpus.

Dialect-specific coverage

Every case carries full expected edges (source/target column, relation type, transform, temp).

MariaDB (mariadb/testdata/analyze/, 20 cases): system-versioned FOR SYSTEM_TIME in all four temporal forms (AS OF / ALL / FROM..TO / BETWEEN..AND), with and without an alias, a temporal table joined with an alias, a temporal base table inside a derived table, INSERT / REPLACE / DELETE ... RETURNING, UPDATE / DELETE ... FOR PORTION OF, CREATE OR REPLACE VIEW, INTERSECT / EXCEPT, sequence expressions, JSON_EXTRACT, NATURAL JOIN.

MariaDB's temporal syntax is table FOR SYSTEM_TIME ... [AS] alias — the alias follows the temporal clause. Writing the alias first (FROM t e FOR SYSTEM_TIME ...) is invalid MariaDB and does not parse.

TiDB (tidb/testdata/analyze/, 15 cases): READ_FROM_STORAGE / SET_VAR / HASH_JOIN hints, PARTITION (...) selection, INSERT IGNORE, REPLACE ... SELECT, ON DUPLICATE KEY UPDATE ... VALUES(), FOR UPDATE, LOCK IN SHARE MODE, INSERT ... SELECT *, multi-table DELETE / UPDATE, INTERSECT / EXCEPT, CREATE OR REPLACE VIEW.

Why separate packages (copies), not one shared analyzer

omni ships a separate parser + AST per dialect, and the ASTs already diverge (MariaDB adds Returning / ForPortionOf / temporal SystemTime; TiDB omits TableSource / ValuesSource / Quantifier). Every difference is an addition or removal of a field the analyzer does not read, so the traversal is source-compatible — but expressing it as a single implementation would require an adapter/IR layer that the MySQL migration deliberately avoided. The corpora are the behavioral sync guard.

Known omni MariaDB parser gaps (verified against MariaDB 11.8)

Both are parser gaps, not analyzer gaps, and both are left uncovered explicitly rather than hidden.

  1. Join trees nested three or more parentheses deep, which is what mysqldump emits for views:

    SQL mysql mariadb
    FROM (t1 JOIN t2 ON ...) ok ok
    FROM ((t1 JOIN t2 ON ...)) ok ok
    FROM (((t1 JOIN t2 ON ...))) ok FAIL
    FROM (((t1 JOIN t2 ON ...)) LEFT JOIN t3 ON ...) ok FAIL

    The four affected shared-corpus cases are listed in mariadb/analyze_test.go (knownParserGaps).

  2. FOR SYSTEM_TIME applied to a derived table, which MariaDB accepts:
    FROM (SELECT ...) FOR SYSTEM_TIME AS OF TIMESTAMP '...' AS x → omni rejects it with expected UPDATE or SHARE after FOR. The supported forms — including a temporal base table inside the derived table — are covered above.

Under the hard-fail parse policy, statements hitting either gap get no lineage until omni's parser is fixed. Worth filing upstream.

Verification

  • go test ./... (hermetic) green: MySQL 73/73; TiDB 73/73 shared + 15/15 dialect; MariaDB 69/73 shared (4 recorded gaps) + 20/20 dialect.
  • golangci-lint 0 issues; gofmt clean; release build ok.
  • make test-integration-mysql against real MySQL 8.4: 5/5.
  • No go.mod / go.sum change — the dialect parsers live in the already-pinned omni module and add no dependencies.

Follow-ups

  • Report the two MariaDB parser gaps upstream; drop the knownParserGaps entries once fixed (the shared corpus returns to 73/73).
  • Modelling dialect-specific outputs (e.g. MariaDB RETURNING) is not needed for parity: the base DML lineage is produced and RETURNING is ignored.
  • Optional: dialect benchmarks.

Add backend/plugin/lineage/{tidb,mariadb}, each a dialect copy of the
omni-backed MySQL analyzer using omni's per-dialect parser and typed AST, and
register Engine_TIDB / Engine_MARIADB. OceanBase is dropped by decision and
stays unsupported.

- All three analyzers run the same golden corpus so the dialects stay
  behaviorally in sync: TiDB 73/73, MariaDB 69/73.
- MariaDB records four explicit parser gaps: omni's MariaDB parser rejects join
  trees nested three or more parentheses deep, which is what mysqldump emits for
  views. MySQL's parser accepts any depth.
- testutil gains RunLineageTestSuitesFromYAMLDirSkipping so a dialect's parser
  gaps are named in the test rather than hidden by a narrowed corpus.
- plan/mysql_family_dialect_lineage_plan.md documents the design and findings.

Verified: go build ./..., the full hermetic suite, golangci-lint (0 issues), the
release build, and make test-integration-mysql against real MySQL 8.4.
Add a golden corpus per dialect, run alongside the shared MySQL corpus, so each
dialect's own syntax is exercised rather than only the dialect-neutral subset.

MariaDB: system-versioned FOR SYSTEM_TIME in all four temporal forms, with and
without an alias. Note the MariaDB syntax is `table FOR SYSTEM_TIME ... [AS]
alias`, so the alias follows the temporal clause; a temporal base table inside a
derived table, and a temporal table joined with an alias, are covered too.
Also INSERT/REPLACE/DELETE ... RETURNING, FOR PORTION OF on UPDATE/DELETE,
CREATE OR REPLACE VIEW, INTERSECT/EXCEPT, sequence expressions, JSON_EXTRACT,
NATURAL JOIN.

TiDB: READ_FROM_STORAGE / SET_VAR / HASH_JOIN hints, PARTITION selection,
INSERT IGNORE, REPLACE ... SELECT, ON DUPLICATE KEY UPDATE ... VALUES(),
FOR UPDATE, LOCK IN SHARE MODE, INSERT ... SELECT *, multi-table DELETE/UPDATE,
INTERSECT/EXCEPT, CREATE OR REPLACE VIEW.

Both dialect packages now run two corpora (shared + own) via a dialectCorpusDir
helper.

Every form was validated against a real MariaDB 11.8 server. One omni MariaDB
parser gap remains and is deliberately not covered: applying FOR SYSTEM_TIME to
a derived table (`FROM (subquery) FOR SYSTEM_TIME ... AS alias`), which MariaDB
accepts but omni rejects.
@Ranxy
Ranxy force-pushed the tidb_mariadb_lineage_omni branch from 009f78a to ec8091b Compare September 14, 2026 09:12
@Ranxy
Ranxy merged commit b0076b8 into main Sep 14, 2026
5 checks passed
@Ranxy
Ranxy deleted the tidb_mariadb_lineage_omni branch September 14, 2026 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant