Skip to content

feat(runtime): add tenant-scoped S3 artifact provider - #119

Merged
XnLemon merged 9 commits into
mainfrom
codex/113-s3-artifact
Sep 3, 2026
Merged

feat(runtime): add tenant-scoped S3 artifact provider#119
XnLemon merged 9 commits into
mainfrom
codex/113-s3-artifact

Conversation

@XnLemon

@XnLemon XnLemon commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Closes #113

Summary

Add a tenant-scoped S3-compatible provider for durable runtime artifacts and objects. The provider uses the AWS SDK v2 API and supports AWS S3, MinIO, and S3-compatible OSS endpoints.

S3 remains opt-in: existing Session, Memory, Summary, Knowledge, Audit, and Vector providers are unchanged.

Issue Ledger

  • Implement tenant-scoped ArtifactStore and ObjectStore with an owned S3 client.
  • Enforce endpoint, region, bucket, size, MIME, checksum, overwrite/idempotency, deletion, timeout, cancellation, and close lifecycle contracts.
  • Use stable tenant-scoped object keys with separate objects and artifacts namespaces.
  • Reject encoded S3 keys over 1,024 bytes before remote I/O.
  • Validate Artifact metadata against a conservative 1.8 KiB S3 user-metadata budget.
  • Preserve CreatedAt on object replacement and keep artifact version/timestamp semantics stable.
  • Validate S3 endpoint, bucket, secret scope, and tenant binding at the profile/materialization boundaries.
  • Fail closed on invalid bindings, unavailable stores, mismatched scope, missing objects, corrupt metadata, and remote errors; never fall back to in-memory storage.
  • Wire bootstrap registration, readiness probing, shutdown ownership, Compose MinIO support, Kubernetes examples, and runtime/deployment/operations documentation.
  • Add contract, boundary, cancellation, pagination, defensive-copy, error-redaction, and optional live-conformance tests.
  • Keep Redis Session/Memory and all other existing capability providers unchanged; vector indexing and native OSS APIs remain out of scope.

Implementation

  • New provider: trpcservice/runtime/storage/s3.
  • Backend Profile provider name: s3, capability: artifact.
  • Required binding fields: HTTPS endpoint, tenant-scoped SecretRef, and bucket; local HTTP requires explicit allow_insecure=true.
  • S3 credentials are resolved only through the existing SecretResolver boundary and are never included in profiles, execution plans, logs, audit payloads, or returned errors.
  • Transfers are bounded by max_bytes and operation deadlines. Content is buffered before the single PUT request, and SHA-256 metadata validates reads.
  • Object replacement preserves the original CreatedAt; identical writes are idempotent.
  • Provider recreation can recover persisted artifacts and attachment objects from the same bucket.

Validation

  • go test ./... -count=1
  • go test -race ./trpcservice/runtime/storage/s3 ./trpcservice/bootstrap ./trpcservice/backend -count=1
  • go vet ./...
  • bash scripts/format.sh --check
  • C:\Users\16040\go\bin\golangci-lint.exe run ./...
  • bash scripts/build.sh
  • docker compose -f deploy/docker-compose.yml config --quiet
  • docker compose --profile s3 --env-file deploy/example.env -f deploy/docker-compose.yml config --quiet
  • python -m mkdocs build --strict -f docs/mkdocs.yml
  • GitHub CI on head 23d4ad3: Build/Test/Coverage, Race, Format/Lint, Vet-equivalent build gates, Docs, deployment validation, fault-injection E2E, WeCom/Telegram E2E, gitleaks, and Codecov patch all pass.

The optional MinIO live conformance test is not run by default because it requires a reachable Docker/S3 environment and explicit S3_RUNTIME_TEST_* variables. When configured, it verifies durable Artifact/Object recovery after provider recreation and the attachment reader path.

Compatibility and Non-goals

  • Summary, Knowledge, Audit, Vector, Session, and Memory continue using their existing providers.
  • No native Alibaba OSS-only APIs, lifecycle policies, CDN, presigned browser uploads, replication, archival tiers, cross-region transfer, migration/cutover tooling, or vector indexing are included.
  • The branch is synchronized with origin/main at 117b411 by merge commit 23d4ad3; the mainline WeCom AI Bot changes are preserved.

@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 06:26 — with GitHub Actions Active
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.86828% with 70 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
trpcservice/runtime/storage/s3/store.go 88.76% 25 Missing and 24 partials ⚠️
trpcservice/bootstrap/environment.go 84.55% 12 Missing and 9 partials ⚠️

📢 Thoughts on this report? Let us know!

@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 07:06 — with GitHub Actions Active

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two focused inline findings on the S3 profile validation and artifact ID compatibility contracts; not LGTM yet.

Comment thread trpcservice/bootstrap/environment.go
Comment thread trpcservice/runtime/storage/s3/store.go Outdated
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 07:40 — with GitHub Actions Active

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One focused P1 inline finding remains on the exact PR head; not LGTM.

Comment thread trpcservice/bootstrap/environment.go Outdated

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One focused P1 inline finding remains on the exact PR head; not LGTM.

Comment thread trpcservice/bootstrap/environment.go Outdated
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 11:39 — with GitHub Actions Active
Comment thread trpcservice/bootstrap/environment.go Outdated
Comment thread trpcservice/bootstrap/environment.go
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 12:16 — with GitHub Actions Active

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three focused P1 inline findings on the exact PR head; not LGTM.

Comment thread trpcservice/runtime/storage/s3/store.go
Comment thread trpcservice/runtime/storage/s3/store.go Outdated
Comment thread trpcservice/runtime/storage/s3/store.go
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 13:11 — with GitHub Actions Active
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 13:28 — with GitHub Actions Active
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 13:37 — with GitHub Actions Active

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required CI remains non-green on this exact head; the two inline notes below are tied to the failing gocyclo findings in the required Format & Lint job.

Comment thread trpcservice/runtime/storage/s3/store.go
Comment thread trpcservice/runtime/storage/s3/store_test.go

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# Conflicts:
#	trpcservice/bootstrap/environment.go
@XnLemon
XnLemon deployed to telegram-e2e September 3, 2026 14:06 — with GitHub Actions Active

@XnLemon XnLemon left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@XnLemon XnLemon self-assigned this Sep 3, 2026
@XnLemon
XnLemon merged commit 81abdf1 into main Sep 3, 2026
12 checks passed
@XnLemon
XnLemon deleted the codex/113-s3-artifact branch September 4, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime/storage: add tenant-scoped S3 artifact provider

1 participant