Skip to content

Implement static website hosting with VersityGW#2019

Open
pr0ton11 wants to merge 19 commits intoversity:mainfrom
pr0ton11:feature/s3-website-hosting
Open

Implement static website hosting with VersityGW#2019
pr0ton11 wants to merge 19 commits intoversity:mainfrom
pr0ton11:feature/s3-website-hosting

Conversation

@pr0ton11
Copy link
Copy Markdown

@pr0ton11 pr0ton11 commented Apr 5, 2026 •
edited
Loading

Implements the S3 static website hosting API operations: PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite, along with a middleware for index document resolution and redirect handling.

  • Full WebsiteConfiguration support: IndexDocument, ErrorDocument, RedirectAllRequestsTo, and RoutingRules (up to 50 rules)
  • Backend implementations for posix and s3proxy; Azure returns BackendUnsupported; ScoutFS inherits from posix
  • Index document middleware rewrites directory-like keys on GetObject/HeadObject and handles RedirectAllRequestsTo (301)
  • 13 end-to-end integration tests replacing the previous NotImplemented stubs

Uses the existing s3:PutBucketWebsite and s3:GetBucketWebsite policy actions already defined in auth/bucket_policy_actions.go. Delete uses the PutBucketWebsite action (matching AWS behavior).

Fully backwards compatible. The "website" metadata attribute simply doesn't exist on pre-upgrade buckets GetBucketWebsite returns NoSuchWebsiteConfiguration (404), and DeleteBucketWebsite is a no-op. No migration needed.

  • All unit tests pass: go test ./s3response/... ./s3api/...
  • All packages build: go build ./cmd/... ./s3api/... ./backend/... ./tests/integration/...
  • 13 new e2e integration tests cover: non-existing bucket, validation errors (empty suffix, slash in suffix, invalid protocol, redirect+index mutual exclusion), put/get round-trips, redirect-all round-trips, delete idempotency

Still in progress:

  • Error page return when object is not found
  • RedirectRules evaluation and following the pattern
  • Seperate the website endpoint with DNS support to allow proxy integration without exposing the S3 API on static websites

The PR is now ready to review.

pr0ton11 added 5 commits April 5, 2026 11:34
@pr0ton11
Add WebsiteConfiguration types, validation, and S3 error codes
0ff3695 
Add S3 bucket website configuration types with XML serialization support
in s3response/website.go. Includes IndexDocument, ErrorDocument,
RedirectAllRequestsTo, and RoutingRules with full validation matching
AWS S3 behavior.

Add corresponding S3 error codes: ErrNoSuchWebsiteConfiguration,
ErrInvalidWebsiteConfiguration, ErrInvalidWebsiteSuffix, and
ErrInvalidWebsiteRedirectCode.

Unit tests cover validation logic, XML round-trip, and parsing.

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Add website backend interface and implementations for posix and s3proxy
a14c063 
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite methods
to the Backend interface with BackendUnsupported stubs that return
ErrNotImplemented.

Posix backend stores website config as a metadata attribute (key:
'website') following the same pattern as CORS. ScoutFS inherits via
embedding.

S3Proxy backend stores website config in the metadata bucket with
prefix 'vgw-meta-website-', consistent with existing ACL/policy/CORS
metadata storage. Returns ErrNoSuchWebsiteConfiguration when not found.

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Add website API controllers and wire into router
93a2229 
Add PutBucketWebsite, GetBucketWebsite, and DeleteBucketWebsite
controller methods following the same pattern as CORS. Controllers
parse and validate WebsiteConfiguration XML, check IAM authorization,
and delegate to the backend.

Replace the three HandleErrorRoute(ErrNotImplemented) stubs in the
router with the new controller methods. Regenerate the backend mock
to include the new interface methods.

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Add website index document middleware and wire into router
01fc515 
Add ResolveWebsiteIndex middleware that rewrites directory-like object keys
(empty or ending with /) to include the IndexDocument suffix when website
hosting is enabled. Also handles RedirectAllRequestsTo by returning 301.

Wire the middleware into both GetObject and HeadObject handler chains in
the router, positioned after BucketObjectNameValidator and before auth.

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Add website integration tests and remove NotImplemented stubs
5c8ad1a 
Replace PutBucketWebsite, GetBucketWebsite, DeleteBucketWebsite
NotImplemented test stubs with comprehensive integration tests covering:
- non-existing bucket errors
- validation (empty suffix, suffix with slash, invalid protocol, mutual
  exclusion of RedirectAllRequestsTo and IndexDocument)
- successful put/get round-trips for both index+error and redirect-all configs
- delete idempotency and verification

Signed-off-by: Marc Singer <marc@singer.gg>
@versity-github
Copy link
Copy Markdown
Collaborator

versity-github commented Apr 5, 2026

This won't automatically run in continuous integration without approval. A member of the Versity organization must allow it.

pr0ton11 added 3 commits April 5, 2026 12:49
@pr0ton11
Add error document serving, routing rules, and integration tests
6819f37 
Implement Features 1 and 2 of S3 static website hosting:

- WebsiteErrorDocument controller wrapper intercepts 4xx errors on
  website-enabled buckets and serves the configured error document or
  evaluates post-request routing rules (error code match redirects)
- ResolveWebsiteIndex middleware now caches parsed WebsiteConfiguration
  in context, handles RedirectAllRequestsTo, evaluates pre-request
  routing rules (key prefix match redirects), and rewrites directory
  keys for index document
- MatchPreRequestRule and MatchPostRequestRule methods on
  WebsiteConfiguration for routing rule evaluation
- 14 unit tests for routing rule matching
- 7 integration tests covering error document, routing rules,
  redirect-all, and index document behavior

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Add separate website hosting endpoint with virtual-host routing
5d62140 
Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Support catch-all mode for website endpoint when --website-domain is …
3b717ae 
…omitted

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Copy link
Copy Markdown
Author

pr0ton11 commented Apr 5, 2026

Now the implementation for the website buckets has it's own port and does not affect existing S3 handlers anymore. I found this approach much more clean, since I anyway have the reverse-proxy handle TLS and host based forwarding.
I build the binary for linux-amd64 already on my repo and am testing it at the moment.

@pr0ton11
Remove website middleware from S3 API router
2ead059 
Website hosting behavior (index document rewriting, error document serving,
routing rule redirects) does not belong on the S3 API port. These semantics
are exclusively handled by the dedicated --website endpoint.

Remove ResolveWebsiteIndex middleware and WebsiteErrorDocument controller
wrapper from the GetObject/HeadObject handler chains, and clean up the
now-unused files and context key.

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11 pr0ton11 force-pushed the feature/s3-website-hosting branch from 0d00a9e to 2ead059 Compare April 5, 2026 16:51
pr0ton11 added 3 commits April 5, 2026 18:55
@pr0ton11
Rewrite website integration tests for dedicated endpoint and add docs
10ed96f 
Rewrite the 7 WebsiteHosting integration tests to target the dedicated
website endpoint using plain HTTP with Host-header-based bucket resolution,
instead of the S3 API port with signed requests. The website middleware was
removed from the S3 API router in the previous commit, so these tests must
now exercise the separate website server.

Changes:
- Add websiteEndpoint field to S3Conf and WithWebsiteEndpoint option
- Add --website-endpoint / VGW_TEST_WEBSITE_ENDPOINT flag to test CLI
- Rewrite all tests in WebsiteHosting.go to use websiteGet() helper
  that issues unsigned HTTP requests with the bucket as Host header
- Gate TestWebsiteHosting group on websiteEndpoint being configured
- Document VGW_WEBSITE_* environment variables in extra/example.conf

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Add website policy actions to WebUI documentation and example policy
44771fc 
Add s3:PutBucketWebsite and s3:GetBucketWebsite to the policy
documentation panel and example policy template so users can
discover the correct action names for website configuration.

Signed-off-by: Marc Singer <marc@singer.gg>
@pr0ton11
Fix nil pointer dereference in website handler GetObject call
b6b51db 
The website handler called GetObject without setting the Range field,
causing a nil pointer dereference in the posix backend when it
dereferences *input.Range. Provide an empty Range string to avoid the
panic. Also fix 'web-site' typo in request logger format.

Signed-off-by: Marc Singer <marc@singer.gg>
@benmcclelland
Copy link
Copy Markdown
Member

benmcclelland commented Apr 9, 2026

Thanks for working on this. We will get this reviewed as soon as we can.

@pr0ton11
Copy link
Copy Markdown
Author

pr0ton11 commented Apr 9, 2026

No problem. Thank you for providing VersityGateway. It's a super awesome product.
In the meantime I deployed it on my instance and I can report that it's working for my use case (which might not use the full WebsiteConfig implementation yet. So take this report with a grain of salt).

niksis02
niksis02 requested changes Apr 9, 2026
View reviewed changes
Comment thread s3response/website.go Outdated
Comment thread tests/integration/DeleteBucketWebsite.go Outdated
Comment thread tests/integration/GetBucketWebsite.go Outdated
Comment thread tests/integration/PutBucketWebsite.go Outdated
Comment thread tests/integration/WebsiteHosting.go Outdated
Comment thread website/server.go Outdated
Comment thread website/handler.go
Comment thread website/handler.go Outdated
Comment thread website/handler.go Outdated
Comment thread s3response/website.go
@pr0ton11
Copy link
Copy Markdown
Author

pr0ton11 commented Apr 10, 2026

@niksis02 Thank you for your review. I implemented the changes you suggested.

@pr0ton11 pr0ton11 requested a review from niksis02 April 10, 2026 07:02
@pr0ton11
Fix review comments
f666804 
Signed-off-by: Marc Singer <marc@singer.gg>
@niksis02
Copy link
Copy Markdown
Contributor

niksis02 commented Apr 10, 2026

@niksis02 Thank you for your review. I implemented the changes you suggested.

Thank you. I'll give it another review by testing the feature e2e and let you know. By the way thank you for the feature implementation.

@pr0ton11
Copy link
Copy Markdown
Author

pr0ton11 commented Apr 16, 2026

@niksis02 were you able to test the PR yet? Thank you

@niksis02
Copy link
Copy Markdown
Contributor

niksis02 commented Apr 20, 2026 •
edited
Loading

@niksis02 were you able to test the PR yet? Thank you

I'm sorry. I've been busy recently with other things. I will try to find some free time and e2e test the new feature. Not sure when exactly I'll be able to, TBH ((.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@niksis02 niksis02 Awaiting requested review from niksis02

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pr0ton11 @versity-github @benmcclelland @niksis02