[SpannerToSourceDb] Add support for URL-encoded connection properties and enable MySQL SSL based authentication

[darshan-sj]

Implement MySQL Client-to-Server SSL Connection support

This PR implements support for MySQL client-to-server SSL based connections in the spanner-to-sourcedb template.

Steps to Connect to MySQL over Client-to-Server TLS

To establish a secure client-to-server TLS connection with MySQL, follow these steps:

1. Upload the JKS truststore file, which is needed to verify the server-side certificate, to a GCS bucket.
2. Provide the GCS path to this file using the extraFilesToStage parameter in your Dataflow execution command. For example:

   --parameters="extraFilesToStage=gs://your-bucket/path/to/truststore.jks"
   

3. In your jdbc-shard-config.json file, configure the connectionProperties to reference the trust store. The path must be prefixed with file:/extra_files/ followed by the filename:

   "connectionProperties": "useSSL=true&requireSSL=true&trustCertificateKeyStoreUrl=file:/extra_files/truststore.jks&trustCertificateKeyStorePassword=your-truststore-password"

4. Ensure that the trust store contains a valid CA certificate that can verify the MySQL server's certificate, and that the MySQL hostname matches the Common Name (CN) or Subject Alternative Name (SAN) on the server's certificate.

Overview of the Changes

1. The standard MySQL JDBC driver handles this verification natively via properties passed in the connection URL string.
2. We use extraFilesToStage as a mechanism to mount the trust store on the Dataflow worker machines.
3. The JdbcConnectionHelper is updated to correctly split and parse standard URL-encoded parameters (delimited by & or ;), allowing standard JDBC SSL parameters to be loaded into the HikariConfig for the pipeline runtime.
4. Similarly, MySQLSpToSrcSourceConnector was updated to parse these URL-encoded properties to ensure the initial pre-flight validation and schema discovery connections also successfully authenticate over SSL.

Unit Testing

All newly introduced code paths and property parsing logic in this commit are fully covered by unit tests.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the database connection configuration logic to support URL-encoded properties. By allowing these properties to be parsed and applied to the Hikari connection pool, it enables better integration with databases requiring specific configurations, such as SSL-based authentication for MySQL. The changes ensure that connection strings can be passed in a standard URL format, improving flexibility for users configuring their source database connections.

Highlights

• URL-Encoded Connection Properties: Added support for parsing URL-encoded connection properties (using '&' or ';' delimiters) in both the JDBC connection helper and the MySQL source connector.
• MySQL SSL Authentication: Enabled support for MySQL SSL-based authentication by allowing properties like 'useSSL' and 'requireSSL' to be passed through the connection configuration.
• Robustness Improvements: Implemented logic to handle malformed connection property strings gracefully by ignoring parameters that do not follow the key-value format.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for parsing connection properties containing delimiters like & or ; in both JdbcConnectionHelper and MySQLSpToSrcSourceConnector, along with corresponding unit tests. The reviewer feedback suggests URL-decoding the parsed keys and values to prevent connection failures when special characters are encoded, updating the unit tests to verify this decoding behavior, and refactoring the connector test to mock DriverManager for verifying connection properties.

[codecov]

Codecov Report

❌ Patch coverage is 91.42857% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.15%. Comparing base (d9fa0f1) to head (460ece5).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...er/migrations/connection/JdbcConnectionHelper.java	92.30%	2 Missing ⚠️
...ates/source/mysql/MySQLSpToSrcSourceConnector.java	88.88%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4184      +/-   ##
============================================
+ Coverage     55.90%   63.15%   +7.24%     
+ Complexity     7477     2786    -4691     
============================================
  Files          1134      562     -572     
  Lines         70135    32645   -37490     
  Branches       8014     3652    -4362     
============================================
- Hits          39212    20616   -18596     
+ Misses        28377    10984   -17393     
+ Partials       2546     1045    -1501     

Components	Coverage Δ
spanner-templates	84.70% <91.42%> (+0.05%)	⬆️
spanner-import-export	∅ <ø> (∅)
spanner-live-forward-migration	88.69% <92.30%> (+0.02%)	⬆️
spanner-live-reverse-replication	81.39% <91.42%> (+0.12%)	⬆️
spanner-bulk-migration	89.08% <92.30%> (+0.01%)	⬆️
gcs-spanner-dv	87.90% <92.30%> (+0.02%)	⬆️

Files with missing lines	Coverage Δ
...ates/source/mysql/MySQLSpToSrcSourceConnector.java	90.76% <88.88%> (+12.19%)	⬆️
...er/migrations/connection/JdbcConnectionHelper.java	91.13% <92.30%> (+3.41%)	⬆️

... and 593 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bharadwaj-aditya]

Can you also check if you can bring up the test coverage. In cases where flags etc parsing is involved, it is good to cover all scenarios.

[bharadwaj-aditya]

LGTM.

Just check the usernames and passwords are not getting logged.

Also, as a follow up, we should do this for other sources as well.