Terraform module for gcs-spanner-dv

[aasthabharill]

b/514941482

Since validation job is source agnostic, we only need this sample folder. For sharded flow or postgres flow, nothing will really change in the setup, so there's no point creating new samples

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.09%. Comparing base (b612c89) to head (0af93bd).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4183      +/-   ##
============================================
+ Coverage     55.90%   63.09%   +7.19%     
+ Complexity     7476     2784    -4692     
============================================
  Files          1134      562     -572     
  Lines         70135    32614   -37521     
  Branches       8014     3645    -4369     
============================================
- Hits          39209    20579   -18630     
+ Misses        28380    10991   -17389     
+ Partials       2546     1044    -1502     

Components	Coverage Δ
spanner-templates	84.64% <ø> (-0.01%)	⬇️
spanner-import-export	∅ <ø> (∅)
spanner-live-forward-migration	88.65% <ø> (-0.02%)	⬇️
spanner-live-reverse-replication	81.25% <ø> (-0.02%)	⬇️
spanner-bulk-migration	89.06% <ø> (-0.01%)	⬇️
gcs-spanner-dv	87.85% <ø> (-0.02%)	⬇️
see 593 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds Terraform support for the GCS to Spanner Data Validator, providing users with a repeatable and automated way to deploy the Dataflow pipeline. By including a structured samples directory, it simplifies the setup of required IAM roles, network configurations, and Dataflow job parameters, ensuring a consistent deployment experience.

Highlights

• Terraform Samples Added: Introduced a new terraform/samples/ directory containing a comprehensive example for deploying the GCS to Spanner Data Validator pipeline.
• Simplified Deployment: Added a simple-validation-job sample that includes all necessary resource definitions, IAM role bindings, and configuration variables to streamline the deployment process.
• Documentation Updates: Updated the main GCS Spanner Data Validator README to include a reference to the new Terraform samples directory.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces Terraform modules and samples for deploying a GCS to Spanner Data Validation Dataflow job, including a reusable module, a simple validation job sample, and associated documentation. The code review identified several critical issues in the Terraform configuration, including potential string interpolation failures when optional variables like working_directory_prefix are null, the use of an unsupported additional_pipeline_options argument in the google_dataflow_flex_template_job resource, and inadequate validation for empty string values in parameters like local_session_file_path and working_directory_bucket.

[gemini-code-assist]

Code Review

This pull request introduces a new Terraform sample for deploying a GCS to Spanner Data Validation Dataflow job, complete with documentation, configuration files, and variable templates. The review feedback suggests improving the robustness of the Terraform configuration by handling full self-links or relative paths for networks and subnetworks, stripping the 'gs://' prefix from user-provided bucket names to prevent invalid URIs, and updating the minimum required Terraform version to '~> 1.3' to support the optional object attributes with default values used in the variables definition.

[gemini-code-assist]

Code Review

This pull request introduces Terraform modules and samples for the GCS to Spanner Data Validator, including a complete sample deployment with detailed documentation. Additionally, it updates existing migration sample READMEs to standardize documentation on schema changes. The review feedback suggests constructing the default Compute Engine service account email dynamically using the project number rather than relying on the google_compute_default_service_account data source, which can cause race conditions if the Compute Engine API is enabled in the same run.