Skip to content

Binary to string mappings - #4179

Draft
sm745052 wants to merge 7 commits into
mainfrom
bin-string
Draft

Binary to string mappings#4179
sm745052 wants to merge 7 commits into
mainfrom
bin-string

Conversation

@sm745052

@sm745052 sm745052 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes data corruption issues when mapping source database binary and bit types to Spanner STRING columns, and ensures they successfully round-trip during reverse migrations.

The Problem

Previously, when forward migrating binary/blob/bit data into a Spanner STRING column, the pipeline was calling .toString() on the underlying Java ByteBuffer. This destroyed the actual binary data by inserting literal Java memory references (e.g., "java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]") into Spanner. Consequently, reverse migrations back to the source database were completely broken, and the related integration tests were commented out or ignored.

The Fix

1. Forward Migrations (AvroToValueMapper.java)

  • Intercepts ByteBuffer and byte[] types before they are cast to strings.
  • Safely encodes the raw binary data into a Hexadecimal String before writing it to Spanner.

2. Reverse Migrations (DML Generators)

  • MySQL: Updates MySQLDMLGenerator to detect when a Spanner string is being mapped back to a blob or binary column. It now wraps the generated value in the UNHEX() function (e.g., UNHEX('7835383030')) to natively restore the raw bytes.
  • PostgreSQL: Updates PostgreSQLDMLGenerator to map hex-encoded Spanner strings back to bytea by wrapping the output in the decode(..., 'hex') function.
  • Bit Handling: Ensures BIT and VARBIT arrays correctly map to strings and migrate back to their native representations without issue.

3. Integration Tests

  • Restored and uncommented previously ignored binary-to-string integration tests across MySQLDataTypesIT, PostgreSQLDataTypesIT, and SpannerToMySqlDataTypesIT.
  • Updated test expectations to assert successful end-to-end hex encoding round-trips.

@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.90%. Comparing base (9f81d23) to head (2515803).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...t/v2/templates/source/mysql/MySQLDMLGenerator.java 81.81% 0 Missing and 2 partials ⚠️
...plates/source/postgres/PostgreSQLDMLGenerator.java 66.66% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4179      +/-   ##
============================================
+ Coverage     55.90%   61.90%   +6.00%     
+ Complexity     7475     3452    -4023     
============================================
  Files          1134      580     -554     
  Lines         70135    34860   -35275     
  Branches       8014     3878    -4136     
============================================
- Hits          39207    21581   -17626     
+ Misses        28381    12167   -16214     
+ Partials       2547     1112    -1435     
Components Coverage Δ
spanner-templates 84.63% <83.33%> (-0.02%) ⬇️
spanner-import-export ∅ <ø> (∅)
spanner-live-forward-migration 88.64% <100.00%> (-0.02%) ⬇️
spanner-live-reverse-replication 81.24% <83.33%> (-0.03%) ⬇️
spanner-bulk-migration 89.05% <100.00%> (-0.02%) ⬇️
gcs-spanner-dv 87.85% <100.00%> (-0.03%) ⬇️
Files with missing lines Coverage Δ
.../v2/spanner/migrations/avro/AvroToValueMapper.java 94.56% <100.00%> (+0.16%) ⬆️
...t/v2/templates/source/mysql/MySQLDMLGenerator.java 95.48% <81.81%> (-1.10%) ⬇️
...plates/source/postgres/PostgreSQLDMLGenerator.java 90.32% <66.66%> (-1.02%) ⬇️

... and 571 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sm745052 sm745052 changed the title added binary to string mappings Binary to string mappings Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant