support arbitrary string parallel snapshotting by jgao54 · Pull Request #4522 · PeerDB-io/peerdb

jgao54 · 2026-06-30T22:27:32Z

Add parallel snapshotting support for arbitrary string using an adaptive bisection strategy.

The partitions are determined by starting with a single partition using min/max fetched values; and then continuously bisecting the largest partition into 2 smaller partitions, until we reached the target number of partitions or until they are no longer splittable.

To determine largest partition, we leverage mysql's query planner's estimated row count for a given range.
To determine the midpoint, we encode the first 8 characters (after shared prefix) as a base-95 integer (one digit per character over printable ASCII); bytes outside of the range get rounded to the nearest edge.
To ensure correctness in coverage, we always fetch the next real key after the interpolated midpoint to ensure server-side collation is used.

There are known edge cases that can lead to skewed partitions (see unit tests), but never compromises on correctness. We sacrifice some balancing optimization for a more simple, deterministic partitioning algorithm.

Fixes: DBI-885

codecov · 2026-06-30T22:48:04Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
3144	2	3142	379

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH

Stack Traces | 0.02s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH (0.02s)
2026/07/21 10:00:00 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 10:00:00 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_api_qm2qrofg.original

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_BinlogIncident

Stack Traces | 31.1s run time

=== RUN   TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_BinlogIncident
=== PAUSE TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_BinlogIncident
=== CONT  TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_BinlogIncident
2026/07/21 09:59:12 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/21 09:59:12 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    clickhouse_mysql_test.go:2282: 
        	Error Trace:	.../flow/e2e/mysql.go:130
        	            				.../flow/e2e/clickhouse_mysql_test.go:2282
        	            				.../hostedtoolcache/go/1.26.5.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	MySQL execute error: readInitialHandshake: io.ReadFull(header) failed. err EOF: connection was bad
        	Test:       	TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_BinlogIncident
2026/07/21 09:59:40 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mych_9vugq2ax.test_update_pkey_chunking_initial_load_enabled
--- FAIL: TestPeerFlowE2ETestSuiteMySQL_CH/Test_MySQL_BinlogIncident (31.12s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-06-30T22:50:58Z

❌ Test Failure

Analysis: The PR's own new feature test (Test_MySQL_String_Partition_Key_Arbitrary_Parallel_Snapshot) fails deterministically across all three relevant suites with a value-correctness mismatch (expected 8, got 1), indicating a real bug in the arbitrary-string parallel-snapshot logic rather than a flake.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-06-30T22:54:20Z

❌ Test Failure

Analysis: A real bug: the new Test_MySQL_String_Partition_Key_Arbitrary_Parallel_Snapshot test (the exact feature this branch implements) fails deterministically across two suite variants with an identical partition-count assertion mismatch (expected 8, got 1), indicating the arbitrary-string-key parallel snapshot isn't splitting into multiple partitions — no timeout, race, or network signature.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-06-30T23:36:26Z

❌ Test Failure

Analysis: The new Test_MySQL_String_Partition_Key_Arbitrary_Parallel_Snapshot test added by this PR fails deterministically across all four CH suite variants with a consistent partition-key string value mismatch, indicating a real logic bug in the PR's arbitrary-string parallel snapshot feature rather than a flaky/timeout/race failure.
Confidence: 0.93

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-01T00:48:45Z

❌ Test Failure

Analysis: The new test Test_MySQL_String_Partition_Key_Arbitrary_Parallel_Snapshot (matching the PR's own feature branch) fails deterministically with assertion mismatches on partition/row counts across all three suites and matrix jobs, indicating a real bug in the PR's new feature rather than flakiness.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-01T02:10:07Z

❌ Test Failure

Analysis: The PR's own new test Test_MySQL_String_Partition_Key_Arbitrary_Parallel_Snapshot fails deterministically with partition/row-count assertion mismatches across every suite and compatibility matrix, indicating a real bug in the arbitrary-string-parallel-snapshot feature rather than flakiness.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

claude · 2026-07-07T02:45:07Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

github-actions · 2026-07-08T03:50:59Z

❌ Test Failure

Analysis: A deterministic, pure unit test (TestBuildAdaptiveStringPartitions_MixedCase_UUID, 0.00s, no network/concurrency) newly added by this PR fails because adaptive string partitioning produced 3 partitions instead of the expected 5 for mixed-case UUIDs — a real logic bug in the PR's own feature, not flakiness.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

jgao54 · 2026-07-08T06:31:20Z

The last commit 7e97b79 also fixes the flaky test scenario TestBuildAdaptiveStringPartitions_MixedCase_UUID reported above, which happens occasionally if the generated UUIDs have a midpoint that does not return a valid fetchNextRealKey; but would return a valid fetchPrevRealKey.

ilidemi

A bunch of Fable finds

ultracode an adversarial code review of the changes on this branch, especially the string wrangling logic. report the findings here

ilidemi

huh forgot to post

claude · 2026-07-18T01:06:59Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

github-actions · 2026-07-18T04:46:20Z

❌ Test Failure

Analysis: The PR's new test Test_MySQL_String_Partition_Key_Special_Chars_Parallel_Snapshot fails deterministically across all 4 MySQL suites and all 3 matrix jobs because the new string-partition-key parallel snapshot drops one record (destination reaches 49/50 rows), so it's a real bug in the feature this PR adds, not flakiness.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

if fetchNextRealKey does not find a valid key; try fetchPrevRealKey before deeming the partition as unsplittable. This is useful for narrow ranges

…e sensitive data, make test verify contiguous chain

github-actions · 2026-07-21T10:06:40Z

🔄 Flaky Test Detected

Analysis: The MySQL_BinlogIncident e2e test failed with a connection-level EOF during the initial MySQL handshake ("connection was bad") in test setup, an infrastructure/network flake that affected only 1 of 3 matrix jobs and is unrelated to the PR's changes.
Confidence: 0.85

✅ Automatically retrying the workflow

View workflow run

ilidemi

Great round of improvements!

ilidemi · 2026-07-23T20:42:00Z

+	// mysql.Escape must NOT be used because MySQL connector session has sql_mode set
+	// to NO_BACKSLASH_ESCAPES (see setSessionSettings). Only quotes needs to be escaped.


Claude likes adding these types of comments but this really should be a task and a sweep (def too big for this PR)

ilidemi · 2026-07-23T22:16:08Z

+		// The interpolated midpoint may be unrepresentable in the column charset,
+		// in which case the server cannot transcode the string literal for comparison
+		// and fails with ERROR 3854 (ER_CANNOT_CONVERT_STRING). Treat it as "no key found".
+		if mErr, ok := errors.AsType[*mysql.MyError](err); ok && mErr.Code == 3854 {


Awesome fallback, just don't split the partition if we're in a complete encoding rabbithole

This is still an issue today that I didn't address in this PR yet. It doesn't impact correctness when that happens (this == consecutively invalid partition, not just transcode error), but we are unnecessarily creating a bunch of single-row partitions with only 1 bulk partition when that happen. I'll take this as a follow-up to not expand this PR further. Should be a relatively small fix where we check min/max to see if they are splittable upfront, and if they are both over, or both under the acceptable ascii, then fallback to single partition.

actually nvm, the transcoding error rabbithole is not a real issue because with found == false, it would end partitioning logic quickly. (vs. the issue with all PK being outside of the ascii bound)

ilidemi · 2026-07-23T22:31:47Z

 	if _, err := conn.Execute("SET NAMES utf8mb4"); err != nil {
 		c.logger.Warn("utf8mb4 not supported, ignoring", slog.Any("error", err))
 	}


Not introduced in this PR but related: if this fails, this gives us more cases where we have to deal with invalid utf-8, so safer to just error out here. Statement itself is supported everywhere and the cases it's being hit are all network errors/driver bugs https://c.house/NEzPf5hrTg

good point, will address this as a follow-up

…row partitions (#4617) Follow up to #4522 Main thing to clean up is the edge case where if both start and end boundary string's first differing byte (the most significant byte for determining the midpoint) are below or above printable ascii, continuing with midpoint interpolation in the extreme case can result in generating a lot of single-row partitions, until we are left with one large final partition. Terminate early when this happens to avoid introducing unnecessary latency. It may still be a full table snapshot, but we don't want to generate a lot of meaningless partitions. --------- Co-authored-by: Ilia Demianenko <ilia.demianenko@clickhouse.com>

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from bd316b6 to 3c9cf7d Compare June 30, 2026 22:29

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 3c9cf7d to 87683c6 Compare June 30, 2026 23:09

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 87683c6 to 4b4de52 Compare July 1, 2026 00:25

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 4b4de52 to 2817e2e Compare July 1, 2026 01:47

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 2817e2e to 29d182c Compare July 7, 2026 00:52

jgao54 marked this pull request as ready for review July 7, 2026 01:59

jgao54 requested a review from a team as a code owner July 7, 2026 01:59

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 29d182c to 83aeb72 Compare July 8, 2026 02:03

jgao54 commented Jul 8, 2026

View reviewed changes

Comment thread flow/connectors/mysql/qrep_partition.go

ilidemi reviewed Jul 9, 2026

View reviewed changes

Comment thread flow/connectors/mysql/qrep_partition.go Outdated

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 7e97b79 to 547be48 Compare July 18, 2026 00:25

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 14261af to 7640e12 Compare July 18, 2026 04:45

jgao54 added 6 commits July 20, 2026 23:44

support arbitrary string parallel snapshotting

c4f0a6d

remove dupe code

94d07ba

partitioning improvement

516b515

if fetchNextRealKey does not find a valid key; try fetchPrevRealKey before deeming the partition as unsplittable. This is useful for narrow ranges

review feedback: hard failure instead of fallback, add comment, remov…

fbde577

…e sensitive data, make test verify contiguous chain

catalog 0x00 NUL byte handling

c0f471f

special char wrongly escaped should fail

9d07fc1

jgao54 added 2 commits July 20, 2026 23:44

address ESCAPE_WITH_NO_BACKLASH session setting

303620c

address multibyte midpoint and unrepresentable midpoint edge cases

590974f

jgao54 force-pushed the arbitrary-string-parallel-snapshot branch from 7640e12 to 590974f Compare July 21, 2026 09:44

jgao54 requested a review from ilidemi July 21, 2026 17:42

ilidemi approved these changes Jul 23, 2026

View reviewed changes

jgao54 merged commit c7a801b into main Jul 24, 2026
32 of 33 checks passed

jgao54 deleted the arbitrary-string-parallel-snapshot branch July 24, 2026 00:52

jgao54 mentioned this pull request Jul 24, 2026

fix string partitioning edge case to avoid rabbit-holing into single-row partitions #4617

Merged

		// mysql.Escape must NOT be used because MySQL connector session has sql_mode set
		// to NO_BACKSLASH_ESCAPES (see setSessionSettings). Only quotes needs to be escaped.

Uh oh!

Conversation

jgao54 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Jun 30, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jun 30, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jun 30, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 1, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 1, 2026

❌ Test Failure

Uh oh!

claude Bot commented Jul 7, 2026

Code review

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026

❌ Test Failure

Uh oh!

jgao54 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilidemi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilidemi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot commented Jul 18, 2026

Code review

Uh oh!

github-actions Bot commented Jul 18, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 21, 2026

🔄 Flaky Test Detected

Uh oh!

ilidemi left a comment

Choose a reason for hiding this comment

Uh oh!

ilidemi Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

ilidemi Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgao54 Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilidemi Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 commented Jun 30, 2026 •

edited

Loading

codecov Bot commented Jun 30, 2026 •

edited

Loading

jgao54 commented Jul 8, 2026 •

edited

Loading

ilidemi left a comment •

edited

Loading

jgao54 Jul 24, 2026 •

edited

Loading

jgao54 Jul 24, 2026 •

edited

Loading