feat: AWS S3 + Glue lakehouse backend with S3 Files mount tutorial

[everettVT]

Summary

• Adds AWS S3 + Glue as a third interchangeable lakehouse backend alongside the existing SQLite (local) and GCS BigLake modes. One env var (AWS_S3_BUCKET) toggles it on; the rest of the pipeline runs unchanged.
• Pairs naturally with the new AWS S3 Files capability (launched 2026-04-07): mount your lakehouse bucket locally as a POSIX filesystem, write to it from agents/scripts with strong consistency, read it back with Daft via the Glue catalog.
• Ships a standalone daft.File knowledge-base example showing how one glob + one DataFrame handles every file type (code, docs, PDFs, media) in a single bucket.

Why

S3 Files turns an S3 bucket into a local filesystem with strong consistency. That's the missing piece for AI workloads that need shared mutable storage — agent memory, knowledge bases, intermediate artifacts. Daft already reads from S3 unchanged; this PR makes the lakehouse story land alongside it: one bucket, one source of truth, three access paths (POSIX mount, S3 API, Iceberg catalog).

What's in the PR

File	Change
pipelines/catalog.py	New elif AWS_S3_BUCKET branch using PyIceberg's Glue catalog with s3:// warehouse
pipelines/lakehouse_analytics/README.md	Three-backend table, AWS quick start, nine-step end-to-end S3 Files mount tutorial with full IAM JSON, security group rules, fstab persistence, and troubleshooting table
examples/files/daft_file_knowledge_base.py	Mixed-content processing example — one glob, every file type, branch by extension
.env.example	AWS_S3_BUCKET + AWS_REGION
README.md, tests/registry.py	Wired up the new example

Stacks on

This PR is based on everettVT/ship-lakehouse (#35). Merge that first.

Test plan

• uv run examples/files/daft_file_knowledge_base.py runs against public HF data (42 files across doc/pdf/audio/video/other)
• Local SQLite mode still works (no env vars set)
• BigLake mode untested in this PR (covered by parent [codex] Add lakehouse analytics pipeline #35)
• AWS Glue mode requires real AWS infra — verified that catalog.py constructs the PyIceberg call correctly, runtime path needs a real bucket to validate
• S3 Files mount tutorial commands need a real AWS account to validate end-to-end

Marketing companion

Eventual-Inc/marketing#297 — blog post tracking the "one bucket for everything" angle.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6db5140c73

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Add Glue dependencies to the lakehouse extra

When users follow the new README command AWS_S3_BUCKET=... uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest, uv installs the project's lakehouse extra from pyproject.toml, which still has pyiceberg[sql-sqlite,gcsfs,bigquery] and not the glue/S3 extras shown only in this PEP 723 header. In a fresh checkout the new AWS branch can therefore select type="glue" without the Glue dependencies (notably boto3) being installed, so the AWS mode fails before it can connect; update the project extra/lockfile as well as the inline metadata.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Pass AWS region using PyIceberg's current keys

For the AWS branch, region_name is a removed/ignored PyIceberg catalog property in current versions; PyIceberg expects keys such as glue.region or client.region (and s3.region if configuring FileIO separately). With the documented/default AWS_REGION value only copied into this region_name entry, users who do not also have a region configured in their AWS profile/environment can get a missing or wrong-region Glue client even though the pipeline claims to default to us-east-1.

Useful? React with 👍 / 👎.