iac(tofu): bring environments/* and bootstrap/* to the same offline-validated bar as envs/*

[mdheller]

Stacked on #1108 (fix/tofu-provider-pinning). Retarget to main once that lands.

The stack is not optional. #1108 removes .terraform.lock.hcl from infra/tofu/.gitignore; without that line gone, every lock file in this PR is silently dropped by git add. Basing on main would have meant either editing .gitignore too — a direct conflict with #1108's only shared file — or git add -f, which leaves eight tracked files that .gitignore still claims to ignore. Stacking avoids both.

The decision this implements

"we should have the 4 providers on same ground, but we don't need to run live connections to their clouds." — Michael

So the bar is offline-validated IaC, uniformly: required_version present, required_providers pinned, lock committed, tofu init -backend=false && tofu validate clean. Live plan/apply, credentials, and federation are explicitly out of scope, and no credential path needs to work.

#1108 met that bar for the four envs/* roots CI validates and deliberately left the other nine unlocked. This is the remaining gap: six environments/* and three bootstrap/*.

Parity matrix

All measurement with OpenTofu 1.8.3 — the TOFU_VERSION in tofu-plan.yml — from a clean .terraform, no plugin cache, tofu init -backend=false && tofu validate.

Before (origin/main @ 55a3d086):

root	required_version	required_providers	lock committed	init	validate
envs/gcp-landing	❌	❌	❌	FAIL	FAIL
envs/shared	❌	❌	❌	ok	ok
envs/prod	❌	❌	❌	ok	ok
environments/aws-eks	✅	⚠️ partial	❌	ok	ok
environments/azure-aks	✅	✅	❌	ok	ok
environments/gcp-ephemeral	✅	✅	❌	ok	ok
environments/gcp-gke	✅	✅	❌	ok	ok
environments/gcp-persistent	✅	✅	❌	ok	ok
environments/ibm-iks	✅	✅	❌	FAIL	FAIL
bootstrap/aws	❌	✅	❌	ok	ok + warning
bootstrap/azure	❌	✅	❌	ok	ok
bootstrap/gcp	❌	✅	❌	ok	ok

aws-eks is "partial" because three of its seven providers were undeclared — see the transitive section.

After (this PR, on top of #1108):

root	required_version	required_providers	lock committed	init	validate
envs/gcp-landing	✅	✅	✅	FAIL (#1103)	FAIL (#1103)
envs/shared	✅	✅	✅	ok	ok
envs/prod	✅	✅	✅	ok	ok
environments/aws-eks	✅	✅	✅ new	ok	ok
environments/azure-aks	✅	✅	✅ new	ok	ok
environments/gcp-ephemeral	✅	✅	✅ new	ok	ok
environments/gcp-gke	✅	✅	✅ new	ok	ok
environments/gcp-persistent	✅	✅	✅ new	ok	ok
environments/ibm-iks	✅	✅	❌ (#1107)	FAIL (#1103)	FAIL (#1107)
bootstrap/aws	✅ new	✅	✅ new	ok	ok, warning gone
bootstrap/azure	✅ new	✅	✅ new	ok	ok
bootstrap/gcp	✅ new	✅	✅ new	ok	ok

The two remaining FAILs are inherited, not caused: gcp-landing is #1103's parse error and ibm-iks is #1103 + #1107. Neither is touched here.

End state, all four lanes merged locally (#1103 + #1107 + #1108 + this, merged clean, measured, then discarded):

root	init	validate		root	init	validate
envs/local	ok	ok		environments/gcp-ephemeral	ok	ok
envs/gcp-landing	ok	ok		environments/gcp-gke	ok	ok
envs/shared	ok	ok		environments/gcp-persistent	ok	ok
envs/prod	ok	ok		environments/ibm-iks	ok	ok
shared/	ok	ok		bootstrap/aws	ok	ok
environments/aws-eks	ok	ok		bootstrap/azure	ok	ok
environments/azure-aks	ok	ok		bootstrap/gcp	ok	ok

14 of 14 roots clean, zero warnings, and tofu fmt -check -recursive infra/tofu/ → exit 0. That is the parity Michael asked for, and it needs all four PRs.

The four asymmetries

1. Three google majors — decided: one, at 6.x

bootstrap/gcp was on ~> 5.0, environments/gcp-* on ~> 6.0, envs/* unconstrained and resolving 7.42.0 until #1108 held them at 6.x. Nothing recorded whether any of that was intended. Two of the three were clearly accidents: envs/* had no constraint at all, and shared/versions.tf documented a ~> 6.0 pin it could not apply to anything.

bootstrap/gcp moves to ~> 6.0. 6.x is the majority and the version #1108 settled on, so bootstrap follows rather than the estate splitting the difference.

⚠️ This changes a resolved major: google 5.45.2 → 6.50.0. Validated at both, clean init, no plugin cache:

bootstrap/gcp @ ~> 5.0  → hashicorp/google v5.45.2 → Success! The configuration is valid.  exit 0
bootstrap/gcp @ ~> 6.0  → hashicorp/google v6.50.0 → Success! The configuration is valid.  exit 0

No warnings either side. The root creates two long-stable resources (google_storage_bucket, google_storage_bucket_iam_member).

2. Two azurerm majors — same call, same evidence

Not on the brief's list, but it is the identical defect: bootstrap/azure on ~> 3.0 while environments/azure-aks was already on ~> 4.0. Moves to ~> 4.0.

⚠️ Also a resolved-major change: azurerm 3.117.1 → 4.81.0. Validated at both:

bootstrap/azure @ ~> 3.0  → hashicorp/azurerm v3.117.1 → Success! The configuration is valid.  exit 0
bootstrap/azure @ ~> 4.0  → hashicorp/azurerm v4.81.0  → Success! The configuration is valid.  exit 0

azurerm 4.0 made provider subscription_id mandatory; this root already sets it from var.subscription_id, which is why the move is clean.

Surfaced by the bump, deliberately not fixed: azurerm_storage_container sets storage_account_name, which 4.x deprecates and 5.x is expected to remove. Read off the real schema rather than from memory — tofu providers schema -json, azurerm 4.81.0:

storage_account_name -> {"deprecated": true, "optional": true}
storage_account_id   -> {"optional": true}

Swapping the attribute changes which identifier addresses an existing container and can produce a diff against state that already exists — an apply-time question this PR has no way to answer offline. The deprecation exists in 4.x whether or not this root points at it; the bump surfaces it, it does not create it. Filed, not hidden.

3. bootstrap/* had no required_version

It was the only tier without one — all six environments/* had it, and #1108 gave it to all four envs/*. These three were the only roots an older OpenTofu would happily parse.

The terraform {} block moves out of main.tf into a versions.tf in each bootstrap root, carrying required_version = ">= 1.8.0". That is not cosmetic: after this, all 14 roots declare versions in the same filename, which is what makes the matrix above checkable by looking rather than by grepping. provider blocks stay in main.tf, matching envs/* post-#1108.

4. environments/aws-eks transitive constraints — the sharpest one

terraform-aws-modules/{vpc,eks} require four providers the root never declared, all with a floor and no ceiling. Straight from init:

- Finding hashicorp/tls versions matching ">= 3.0.0"...        → installed 4.3.0
- Finding hashicorp/time versions matching ">= 0.9.0"...       → installed 0.14.0
- Finding hashicorp/cloudinit versions matching ">= 2.0.0"...  → installed 2.4.0
- Finding hashicorp/null versions matching ">= 3.0.0"...       → installed 3.3.0

tls was resolving 4.3.0 against a floor of 3.0.0 — already a full major above its stated minimum, and it would have crossed into 5.x unannounced on the day 5.0 shipped. A floor without a ceiling is not a pin, and no amount of local pinning of aws/helm/kubernetes constrained these.

All four are now declared at the root with ceilings. tls = "~> 4.0" and null = "~> 3.2" are not new inventions — they are the constraints already written in shared/versions.tf, finally applied somewhere they load. The lock now records the intersection:

hashicorp/tls        4.3.0   constraints = ">= 3.0.0, ~> 4.0"
hashicorp/time       0.14.0  constraints = ">= 0.9.0, ~> 0.14"
hashicorp/cloudinit  2.4.0   constraints = ">= 2.0.0, ~> 2.4"
hashicorp/null       3.3.0   constraints = ">= 3.0.0, ~> 3.2"

No resolved version changed — the ceiling was added above where resolution already sat. aws-eks goes from 3 declared / 7 resolved to 7 / 7.

bootstrap/aws was the only root validating with a warning

Warning: Invalid Attribute Combination
  with aws_s3_bucket_lifecycle_configuration.tofu_state, on main.tf line 44
No attribute specified when one (and only one) of [rule[0].filter, rule[0].prefix] is required
This will be an error in a future version of the provider

Fixed with filter {} — the provider's documented way to say "every object in the bucket", which is what the rule already meant, so this records the existing scope rather than narrowing it. bootstrap/aws now returns a bare Success! The configuration is valid.

A hole #1108's un-ignore does not reach

environments/gcp-ephemeral/ and environments/gcp-persistent/ each carry their own .gitignore that also lists .terraform.lock.hcl. A nested .gitignore overrides its parent, so #1108 removing the line from infra/tofu/.gitignore does not un-ignore these two — both locks would have been dropped silently by git add, and the matrix above would have quietly shown ten locks instead of twelve.

Both lines removed. *.tfstate stays ignored in both: the lock pins the provider build, not the state.

Locks

Same platform set as #1108 — -platform=linux_amd64 -platform=darwin_arm64 -platform=darwin_amd64 (CI plus both Mac architectures), generated with 1.8.3.

Verified, not assumed: all 21 provider entries across the eight new locks carry exactly 3 h1: hashes. And they are stable — a second clean init from a deleted .terraform leaves all eight byte-identical (sha256 compared before/after):

root	providers	root	providers
environments/aws-eks	7	environments/gcp-persistent	1
environments/azure-aks	3	bootstrap/aws	1
environments/gcp-ephemeral	3	bootstrap/azure	1
environments/gcp-gke	4	bootstrap/gcp	1

environments/aws-eks and environments/azure-aks need tofu init -backend=false before providers lock — providers lock does not install modules, and both roots take providers from module blocks. Worth knowing before anyone regenerates them.

---

environments/ibm-iks — untouched, and the lock is here for #1107

Not modified. #1103 fixes its init, #1107 its validate errors, and both are open against exactly these paths. Nothing in this PR touches environments/ibm-iks/ or modules/ibm-trusted-profile/.

Its lock is the one piece of the matrix I cannot land, because it cannot be generated on this branch — init fails until #1103's provider-source declaration exists. I generated it on a throwaway merge of all four lanes and verified it:

provider "registry.opentofu.org/ibm-cloud/ibm"        version = "1.89.0"  constraints = "~> 1.70"
provider "registry.opentofu.org/hashicorp/helm"       version = "2.17.0"  constraints = "~> 2.13"
provider "registry.opentofu.org/hashicorp/kubernetes" version = "2.38.0"  constraints = "~> 2.30"

3 providers, 9 h1: hashes (3 x 3 platforms), 2840 bytes, sha256 f9ef4b7e8c8aa7b675cba1870da60d6d54eccce2672f0af9ea98bd7daaa11dbc.

Recommendation: whichever of #1103/#1107 lands last adds it with
tofu providers lock -platform=linux_amd64 -platform=darwin_arm64 -platform=darwin_amd64.
ibm-iks has no nested .gitignore, so #1108's un-ignore is sufficient there.

Recommendation: the IBM auth pieces

Keep modules/ibm-trusted-profile and environments/ibm-iks. Inert, and labelled. Do not remove.

The brief is right that they exist only to enable live connections, and under this decision that purpose is gone. But removal is still the wrong move:

1. Deleting the module deletes the fourth lane. environments/ibm-iks is a validating root only because that module exists. Removing it to achieve parity would remove IBM from the set being brought to parity.
2. fix(iac): make ibm-iks pass tofu validate — and stop it claiming OIDC federation IBM cannot do #1107 already does exactly the right thing and is open. It removes the invalid Profile-OIDC claim rule rather than swapping in one that passes the validator and federates nothing, keeps the profile, and labels the non-federation in four places plus the IBM console description string. That is "inert and clearly labelled", already written. Deleting the module would throw that work away and collide with an open PR.
3. Offline-validated IaC is a real deliverable. The module's job under this posture is to be valid HCL that records intent. It does that without ever authenticating.

One amendment I would suggest to #1107's owner, not a blocker: its README and comments frame the non-federation as an IBM limitation ("IBM Cloud cannot federate GitHub Actions OIDC"). Both are true, but the operative reason is now the decision — we are not running live connections against any of the four clouds. Saying "out of scope by decision, and also not possible on IBM" is the honest ordering, and it stops the note reading as a blocked item someone should go fix.

Recommendation: the drift workflow's cloud-auth jobs

⚠️ .github/workflows/infra-drift-detect.yml is #1090's lane (fix/drift-exit1-is-failure, open). Not touched here. This is a recommendation for its owner.

The important point: this is not an IBM problem. infra-drift-detect.yml has four jobs — gcp-landing, aws-eks, azure-aks, ibm-iks — and every one runs tofu plan against a live cloud. Under a no-live-connections posture the whole workflow has no purpose, not just its IBM step. Removing only the cr-token block would be inconsistent, and worse, it would imply the other three are fine.

The IBM step is independently broken regardless of posture — it posts a GitHub OIDC JWT to grant_type=urn:ibm:params:oauth:grant-type:cr-token, a grant that expects a platform-injected compute-resource token (#1107 documents this in detail) — but fixing it is now moot.

Recommend: make the whole workflow inert rather than delete the file.

• Drop the schedule: trigger; leave workflow_dispatch: so it can be run deliberately if live access is ever wanted.
• Header comment stating the posture and pointing at this decision.
• Keep the file. Lines 17-23 are the only written inventory of which tofu output feeds which CI variable (GCP_WIF_PROVIDER, AWS_TOFU_ROLE_ARN, AZURE_CLIENT_ID, IBM_TRUSTED_PROFILE_ID, ...). Deleting it loses reference material that is genuinely useful the day live connections come back into scope, and costs nothing to keep once the schedule is gone.

Nothing is lost by doing this: per #1097's audit the workflow has 27 runs, 26 startup_failure, and has never once succeeded. It has never detected drift. Making that explicit is strictly better than leaving a scheduled job that looks like a control and is not one — the same defect class #1107 declined to create.

Recommendation for tofu-plan.yml (#1097's lane — not edited)

⚠️ Not touched. #1097 owns that file.

#1107 named nine roots to add and verified eight pass locally. With this PR the count is twelve, and all twelve pass. fmt -check -recursive already walks all of infra/tofu/, but validate covers 4 of 14 roots, which is precisely why a root that had never validated survived indefinitely.

Add these eight to fmt-and-validate (tofu init -backend=false && tofu validate each) — all verified clean here:

infra/tofu/environments/aws-eks
infra/tofu/environments/azure-aks
infra/tofu/environments/gcp-ephemeral
infra/tofu/environments/gcp-gke
infra/tofu/environments/gcp-persistent
infra/tofu/bootstrap/aws
infra/tofu/bootstrap/azure
infra/tofu/bootstrap/gcp

Plus infra/tofu/environments/ibm-iks once #1103 + #1107 land — verified clean on the combined merge above — and infra/tofu/shared, which validates clean and costs nothing.

A matrix over root directories beats twelve hand-written steps, and is the only version of this that stops a new root landing unvalidated. That is the failure mode that actually happened here; enumerating roots by hand just resets the clock.

Lane hygiene

git merge-tree --write-tree against all four open PRs, from this branch head:

against	result	overlapping files
#1103 fix/tofu-fmt-validate-blackout	clean	none
#1107 fix/ibm-iks-tofu-validate	clean	none
#1108 fix/tofu-provider-pinning	clean	none (it is this branch's base)
#1097 fix/tofu-plan-setup-opentofu-blocked	clean	none

Zero conflicts, zero overlapping files with any lane. The four lanes also merge cleanly with each other and with this branch — that is how the end-state matrix above was measured.

Every change is inside infra/tofu/.

⚠️ Evidence is local only

GitHub Actions is at a spend cap — hundreds of jobs queued, zero running. No workflow ran for this branch, and none will until the cap clears. Everything above is a local OpenTofu 1.8.3 binary with .terraform deleted and no plugin cache, replicating the CI command sequence. It is not a CI result and should not be read as one.

Specifically not covered, and deliberately so under the no-live-connections decision:

• tofu plan / tofu apply anywhere. Schema compatibility across the two major bumps is confirmed; apply-time differences (changed defaults, newly-required fields) are not.
• Any credential path, on any of the four clouds.
• Whether the committed locks resolve on linux_amd64. The hashes are in the lock and were fetched from the registry for that platform, but only darwin_arm64 was actually initialised from them here.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Aligns environments/* and bootstrap/* OpenTofu roots with the repo’s “offline-validated IaC” standard by ensuring consistent version/provider pinning and making .terraform.lock.hcl trackable where it was previously ignored.

Changes:

• Stop ignoring .terraform.lock.hcl in nested environments/gcp-* .gitignore files so committed locks aren’t silently dropped.
• Add versions.tf to bootstrap/{aws,azure,gcp} with required_version and pinned required_providers, and remove the inline terraform {} blocks from main.tf.
• Declare previously-transitive providers explicitly in environments/aws-eks/versions.tf to ensure ceilings exist at the root.

Reviewed changes

Copilot reviewed 9 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
infra/tofu/environments/gcp-persistent/.gitignore	Removes nested ignore of .terraform.lock.hcl and documents why lockfiles must be tracked here.
infra/tofu/environments/gcp-ephemeral/.gitignore	Removes nested ignore of .terraform.lock.hcl and documents why lockfiles must be tracked here.
infra/tofu/environments/aws-eks/versions.tf	Adds explicit root-level provider constraints for transitively used providers to avoid floor-only resolution.
infra/tofu/bootstrap/gcp/versions.tf	Introduces consistent required_version/provider pinning for the bootstrap GCP root.
infra/tofu/bootstrap/gcp/main.tf	Removes inline terraform {} in favor of versions.tf.
infra/tofu/bootstrap/azure/versions.tf	Introduces consistent required_version/provider pinning for the bootstrap Azure root.
infra/tofu/bootstrap/azure/main.tf	Removes inline terraform {} in favor of versions.tf.
infra/tofu/bootstrap/aws/versions.tf	Introduces consistent required_version/provider pinning for the bootstrap AWS root.
infra/tofu/bootstrap/aws/main.tf	Removes inline terraform {} in favor of versions.tf and adds an empty lifecycle filter {} to eliminate a validation warning.

Files not reviewed (8)

• infra/tofu/bootstrap/aws/.terraform.lock.hcl: Generated file
• infra/tofu/bootstrap/azure/.terraform.lock.hcl: Generated file
• infra/tofu/bootstrap/gcp/.terraform.lock.hcl: Generated file
• infra/tofu/environments/aws-eks/.terraform.lock.hcl: Generated file
• infra/tofu/environments/azure-aks/.terraform.lock.hcl: Generated file
• infra/tofu/environments/gcp-ephemeral/.terraform.lock.hcl: Generated file
• infra/tofu/environments/gcp-gke/.terraform.lock.hcl: Generated file
• infra/tofu/environments/gcp-persistent/.terraform.lock.hcl: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.