Skip to content

fix: prevent data loss when MySQL tables have triggers (#285)#311

Draft
ljluestc wants to merge 1 commit intobrokercap:masterfrom
ljluestc:fix/mysql-trigger-data-loss
Draft

fix: prevent data loss when MySQL tables have triggers (#285)#311
ljluestc wants to merge 1 commit intobrokercap:masterfrom
ljluestc:fix/mysql-trigger-data-loss

Conversation

@ljluestc
Copy link

Problem

When MySQL tables have triggers and binlog_format is STATEMENT or MIXED, DML operations (INSERT/UPDATE/DELETE) are logged as QUERY_EVENT instead of row-based events (WRITE_ROWS_EVENT, UPDATE_ROWS_EVENT, DELETE_ROWS_EVENT). Bifrost did not handle this case, causing two cascading bugs that resulted in silent data loss.

Bug 1 — Binlog parser drops the commit signal (Bristol/mysql/conn_dump.go):
After a DML QUERY_EVENT is dispatched via callback, the post-callback switch falls through to the default case and resets commitEventOk = false. When the subsequent COMMIT or XID_EVENT arrives, it sees commitEventOk == false and skips the commit entirely. Downstream plugins never receive the transaction.

Bug 2 — ClickHouse plugin flushes data prematurely (plugin/clickhouse/src/clickhouse.go):
For any QUERY_EVENT that does reach the ClickHouse plugin (e.g. DML or transaction control statements like SAVEPOINT), the Query() method falls through to AutoCommit(). This flushes whatever partial row data is in the buffer and resets it mid-transaction, losing data.

Fixes #285

Changes

Bristol/mysql/conn_dump.go

Add a QUERY_EVENT case in the post-callback switch. If event.TableName != "" (i.e. the event carries DML data for a specific table), preserve commitEventOk = true so the subsequent COMMIT/XID event is properly propagated. For control queries like BEGIN/COMMIT that have no table name, reset commitEventOk = false as before.

plugin/clickhouse/src/sql.go

Add IsNonDDLQuery() helper function that identifies:

  • DML statements: INSERT, UPDATE, DELETE, REPLACE
  • Trigger-related transaction control: SAVEPOINT, RELEASE SAVEPOINT, ROLLBACK TO

These are matched case-insensitively via prefix after trimming whitespace.

plugin/clickhouse/src/clickhouse.go

Call IsNonDDLQuery() early in Query(). If the query is non-DDL, return (nil, nil, nil) immediately — no AutoCommit() call, no buffer flush. Pending row data is preserved for the eventual COMMIT to flush correctly.

Tests (7 new test functions)

  • TestIsNonDDLQuery — 20 cases covering DDL, DML, transaction control, edge cases, and case insensitivity
  • TestTranferQuerySql_DML — verifies TranferQuerySql returns empty strings for all DML/transaction-control inputs
  • TestQuery_NonDDLStatements — verifies 7 non-DDL query types don't trigger AutoCommit or flush buffered data
  • TestQuery_TransactionControlSkipped — verifies BEGIN/COMMIT don't flush pending data
  • TestQuery_AutoCreateTableFalse — verifies all queries are no-ops when AutoCreateTable is disabled

When MySQL tables have triggers and binlog_format is STATEMENT or MIXED,
DML operations are logged as QUERY_EVENT instead of row events. This
caused two cascading bugs resulting in data loss:

1. Bristol/mysql/conn_dump.go: After a DML QUERY_EVENT callback,
   commitEventOk was reset to false, causing the subsequent COMMIT/XID
   event to be skipped entirely. Downstream plugins never received the
   commit signal.

2. plugin/clickhouse/src/clickhouse.go: Non-DDL QUERY_EVENTs (DML and
   transaction control like SAVEPOINT) triggered premature AutoCommit(),
   flushing pending row data and resetting the buffer mid-transaction.

Fixes:
- Add QUERY_EVENT case in conn_dump.go post-callback switch to preserve
  commitEventOk=true for DML queries (identified by event.TableName!="")
- Add IsNonDDLQuery() helper to skip DML and transaction control
  statements without calling AutoCommit in the ClickHouse plugin
- Add 7 new tests covering the fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MySQL同步clickhouse,当表上有触发器时,数据不会同步

1 participant