Skip to content

Merge upstream turbot v1.31.0 into catio fork (SDK v5→v6, Go 1.26)#8

Open
samsar wants to merge 165 commits into
mainfrom
merge/upstream-v1.31.0
Open

Merge upstream turbot v1.31.0 into catio fork (SDK v5→v6, Go 1.26)#8
samsar wants to merge 165 commits into
mainfrom
merge/upstream-v1.31.0

Conversation

@samsar

@samsar samsar commented Jul 15, 2026

Copy link
Copy Markdown

What this does

Brings the catio fork current with upstream turbot/steampipe-plugin-aws v1.31.0 (162 upstream commits), while preserving all 52 of our fork-specific commits. To see only our changes on top of the upstream, see this view.

Merge (not rebase) so conflicts are resolved once in the final state and there's no history rewrite / force-push on a branch our pipelines depend on.

The big upstream changes this pulls in

  • steampipe-plugin-sdk v5 → v6 (major bump — repo-wide import change)
  • Go 1.23 → 1.26 (go directive in go.mod)
  • aws-sdk-go (v1) dependency removed upstream; xxxv1.EndpointsID replaced by constants (AWS_EC2_SERVICE_ID, AWS_LOGS_SERVICE_ID)

Conflicts resolved (5 files)

File Resolution
go.mod / go.sum Took upstream wholesale (v6 + Go 1.26 + dep overhaul), then go mod tidy re-added our extra deps (pganalyze/pg_query_go/v6, testify)
Makefile Kept both our docker-build-steampipe and upstream's dev target
table_aws_availability_zone.go Kept our region-scoping design (GetMatrixItemFunc + EC2Client, no ParentHydrate); dropped dead aws-sdk-go v1 import; ec2v1.EndpointsIDAWS_EC2_SERVICE_ID. Upstream's shouldIgnoreErrorPluginDefault handling auto-merged and retained
table_aws_vpc_flow_log_event.go Kept our column/hydrate additions on upstream's v6 imports + IgnoreConfig + AWS_LOGS_SERVICE_ID

Silent (non-conflict) fixes the SDK bump required

These files merged clean but wouldn't compile against v6:

  • Migrated aws/operations_analyzer.go and aws/build_filter_test.go from steampipe-plugin-sdk/v5v6.
  • Bumped docker/steampipe/Dockerfile builder image golang:1.23golang:1.26 (go.mod now requires go 1.26.0).

Verification

  • go build ./... — pass
  • go vet ./... — pass
  • go test ./aws/vpcflowlogs/... — pass
  • go test ./aws/ unit tests — pass except TestAnalyzeQueriesFromJSON, which is pre-existing/environmental (reads ../../extractor-steampipe/..., a sibling repo only present in the CI pipeline — fails identically on main)
  • Full Docker build for linux/amd64 (matches CI) — pass; go build completes in golang:1.26 container, image builds, and steampipe plugin list recognizes the built turbot/aws@latest plugin

Note

The AWS plugin shows no connections in steampipe plugin list inside the image because no aws.spc is baked in — credentials/connection config are injected at runtime. Expected, not a regression.

pdecat and others added 30 commits April 23, 2025 18:57
…on table (turbot#2158)

Co-authored-by: misraved <ved@turbot.com>
Co-authored-by: ParthaI <partha@turbot.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.36.0 to 0.38.0.
- [Commits](golang/net@v0.36.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… and update Parliament IAM permissions to latest
Co-authored-by: ParthaI <partha@turbot.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…elegated_services_for_account tables (turbot#2421) (turbot#2477) (turbot#2496)

Co-authored-by: Muhammad Fuad Bin Abdullah <67743055+FuadAbdullah@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Keep Focused <partha@turbot.com>
Co-authored-by: ParthaI <partha@turbot.com>
Co-authored-by: Ved misra <47312748+misraved@users.noreply.github.com>
…eriod_end as optional quals) in all the table aws_cost_* tables (turbot#2168)

Co-authored-by: misraved <ved@turbot.com>
kaidaguerre and others added 29 commits May 20, 2026 13:15
* Add CHANGELOG for v1.30.3

* Add Dependencies section noting SDK v6 + Go 1.26 requirement
…istent formatting (turbot#2742)

This commit also includes:
- Fix tags for `aws_secretsmanager_secret` table
- Fix logging output string in `aws_cloudtrail_trail` table
Co-authored-by: ppapishe <praneeth.papishetty@wbd.com>
…date `aws_ecr_registry`, `aws_eks_pod_identity_association`, `aws_msk_topic` tables with various fixes
…s columns

- Return the nested Canary from getSyntheticsCanary so get queries populate
- Change success_retention_period_in_days from JSON to INT
- Add not-found IgnoreConfig and title/akas standard columns
- Add turbot test plus aws_region and canary_arn outputs
…ixes

awsCommercialRegionPrefixes was missing the il and mx prefixes, so region
wildcards like il-* or mx-* fell through to an unknown partition instead of
resolving to the commercial partition. Added both prefixes and a unit test
covering all commercial/gov/china/iso prefixes.
Add new tables `aws_ecr_registry`, `aws_eks_pod_identity_association`, `aws_msk_topic`, `aws_rds_global_cluster`, and `aws_synthetics_canary`.

This commit also fixes bugs with the `aws_cloudwatch_log_subscription_filter` table and wildcard resolution for the `il` and `mx` regions.
…2746)

urllib3 2.6.3 -> 2.7.0 in generate_parliament_iam_permissions. Security fix (HIGH), version-only diff, codegen tooling not shipped in plugin.
urllib3 2.6.3 -> 2.7.0 in generate_aws_supported_endpoints. Security fix (HIGH), version-only diff, codegen tooling not shipped in plugin.
lodash 4.17.23 -> 4.18.1 in aws-test. Security fix (HIGH). tint.js _.template usage verified unaffected; 4.18.1 includes the fix for the 4.18.0 template regression.
idna 3.7 -> 3.15 in generate_parliament_iam_permissions. Security fix (MEDIUM), version-only diff, codegen tooling not shipped in plugin.
requests 2.32.4 -> 2.33.0 in generate_aws_supported_endpoints. Security fix (MEDIUM), version-only diff, codegen tooling not shipped in plugin.
picomatch 2.3.1 -> 2.3.2 in aws-test. Security fix (HIGH), lock-file-only diff, test harness not shipped in plugin.
…#2722)

requests 2.32.4 -> 2.33.0 in generate_parliament_iam_permissions. Security fix (MEDIUM), version-only diff, codegen tooling not shipped in plugin.
Bumps [github.com/go-jose/go-jose/v4](https://github.com/go-jose/go-jose) from 4.1.3 to 4.1.4.
- [Release notes](https://github.com/go-jose/go-jose/releases)
- [Commits](go-jose/go-jose@v4.1.3...v4.1.4)

---
updated-dependencies:
- dependency-name: github.com/go-jose/go-jose/v4
  dependency-version: 4.1.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Brings the fork current with upstream (162 commits), including the
steampipe-plugin-sdk v5 -> v6 major bump, Go 1.23 -> 1.26, and removal
of the aws-sdk-go (v1) dependency.

Conflict resolutions (5 files):
- go.mod / go.sum: took upstream (v6 + Go 1.26 + dep overhaul), then
  'go mod tidy' re-added our extra deps (pganalyze/pg_query_go/v6, testify).
- Makefile: kept both our 'docker-build-steampipe' and upstream 'dev' targets.
- table_aws_availability_zone.go: kept our region-scoping design
  (GetMatrixItemFunc + EC2Client, no ParentHydrate); dropped dead aws-sdk-go
  v1 import; ec2v1.EndpointsID -> AWS_EC2_SERVICE_ID. Upstream's
  shouldIgnoreErrorPluginDefault handling auto-merged and retained.
- table_aws_vpc_flow_log_event.go: kept our columns/hydrate additions on
  upstream's v6 imports + IgnoreConfig + AWS_LOGS_SERVICE_ID.

Silent (non-conflict) fixes required by the SDK bump:
- Migrated operations_analyzer.go and build_filter_test.go from
  steampipe-plugin-sdk/v5 -> v6.

Verified: go build ./... , go vet ./... , go test ./aws/vpcflowlogs/... all
pass. Our aws-package unit tests pass except TestAnalyzeQueriesFromJSON,
which is pre-existing and environmental (reads a sibling repo only present
in the CI pipeline).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
go.mod now requires 'go 1.26.0' (raised by the upstream merge), so the
plugin-builder stage must use golang:1.26 — golang:1.23 would refuse to
build the module.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
golangci-lint (errcheck) flagged unchecked error returns on 8 io/gzip
Close() calls in our fork-specific aws/vpcflowlogs/flow_log_events_s3.go.
All are cleanup-path closes where the error is safe to discard, so make
that explicit with '_ ='. Fixed all 8 (CI only surfaced 6 due to
golangci-lint's default max-same-issues=3).

golangci-lint v2.12.2 (CI-identical) now reports 0 issues.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New test.yml workflow runs go test ./... on every pull request and push
to main. TestAnalyzeQueriesFromJSON now skips (instead of failing) when
the sibling extractor-steampipe fixture is absent, so the suite is green
on plugin-only checkouts and in this repo's CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The installer previously fetched latest at build time, making image
rebuilds non-reproducible and exposing us to untested CLI releases.
v2.4.4 is the current stable release (2026-05-25) and what recent
unpinned builds shipped; plugin-sdk v6 is compatible with CLI v2.x.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@samsar

samsar commented Jul 16, 2026

Copy link
Copy Markdown
Author

All verification is complete, including the changelog review and the SRE pass. Here's the full picture.

Verdict: safe to merge ✅

PR #8 is a faithful, well-executed sync. Every check I ran came back clean, with one systemic (not merge-blocking) CI gap and a short list of post-merge watch items.

Merge integrity — verified

  • Exact tag merged: the merge commit's second parent is precisely the upstream v1.31.0 tag. Previous sync point was v1.11.0, so this brings in 20 releases at once (797 files) — the size is expected, not suspicious.
  • Fork customizations fully preserved: the fork's delta vs. upstream is the same 19 files / +5,995 lines before and after the merge. A diff-of-diffs on every shared file shows the only content changes are the required migrations: ec2v1.EndpointsIDAWS_EC2_SERVICE_ID (aws-sdk-go v1 was removed upstream) and SDK v5v6 imports. Zero sdk/v5 or aws-sdk-go v1 references remain anywhere.
  • Conflict resolution was correct in both directions: only 7 files needed hands-on resolution, and I confirmed upstream's improvements inside our modified tables survived — the shouldIgnoreErrorPluginDefault fix in aws/table_aws_availability_zone.go:135 and the new IgnoreConfig in aws/table_aws_vpc_flow_log_event.go:61. No conflict markers anywhere.
  • Builds, tests, lint: go build, go vet, and go mod tidy -diff are all clean on Go 1.26. Tests: exactly one failure — TestAnalyzeQueriesFromJSON, the documented expected local failure (needs the extractor-steampipe sibling repo, which isn't checked out here). Everything else passes, including the entire fork-owned aws/vpcflowlogs suite. golangci-lint passed in CI on the exact head commit.
  • The PR will land cleanly on today's main: main gained one commit (7534e4e3, gitsha-only image publishing) after this branch was cut. I simulated the actual PR merge with git merge-tree — it's conflict-free, keeps main's local/steampipe:local tag and gitsha-only workflows, and upstream's new Makefile dev: target. Nothing gets reverted.
  • Post-merge commits are safe: the Dockerfile bump (golang:1.26, matching go.mod) and the errcheck fix (8 mechanical _ = Close() on cleanup paths — no behavior change).

One correction to my interim note: add-issue-to-project.yml was not deleted by upstream — it's a new upstream workflow (added after v1.11.0) that will appear on main. It's harmless in the fork (fires on issue-open, calls a turbot reusable workflow that will no-op/fail without turbot's secrets), but see the suggestion below.

Upstream behavior changes worth knowing (v1.12.0 → v1.31.0)

No tables or columns were removed or renamed across the entire range, no breaking config changes, and ~60 new tables are purely additive. The changes that could surprise a downstream consumer:

  • aws_lambda_function.code no longer exposes the Location sub-field (v1.30.1 security fix — it was a presigned S3 URL). Any extractor query reading code -> 'Location' now gets NULL. This is the only silent data removal in 20 releases.
  • Rate limiting is now actually enforced (an SDK fix in v1.15.0), and default rate-limiter configs were added for IAM, CloudFormation, Kinesis, Route 53, WAF, and Lambda tables. Expect fewer AWS throttling errors but possibly slower extraction on those services.
  • Result-set semantics fixed in a few places: aws_cloudformation_stack now also returns DELETE_COMPLETE stacks (row counts change); boolean WHERE NOT x qualifiers now filter correctly (v1.13.0); several tables that used to error now return rows; aws_health_* tables only query us-east-1/us-east-2/us-gov-west-1.
  • Credential handling improved (v1.30.3): credentials refresh on every request-signing call instead of being captured at config construction — this directly fixes ExpiredToken on long-running queries with rotated STS creds, which is a real win for your long VPC flow log scans.
  • Deprecations only, nothing dropped: aws_organizations_account.status (use state) and aws_cost_usage.search_start/end_time (use period_start/end) — both old columns still exist; I verified in the merged code.
  • Runtime requirements: SDK v6 + Go 1.26 (stated in the v1.30.3 changelog entry, not v1.31.0's). No minimum Steampipe CLI version is documented anywhere; the Docker image installs the latest CLI at build time, so compatibility is satisfied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.