Once financial data is sitting in partitioned Iceberg tables on S3, a second problem shows up: different roles need to access that same Gold dataset in different ways, and locking everyone into a single query engine doesn't work well in practice. A Data Analyst running ad-hoc SQL doesn't want to spin up a Docker cluster, a Data Engineer doing heavier local analytics doesn't want to pay for cloud compute on every query, and an ML Engineer just needs read access to pull training data, not a full query interface at all.
This project was built to solve that access problem directly. It's a lakehouse layer where a single, versioned, partitioned Gold dataset in Iceberg can be queried from multiple independent engines, cloud and on-premises, without duplicating the data or maintaining separate pipelines per engine. Specifically, it needed to:
- Store the Gold dataset once, in a format (Iceberg on S3) that supports ACID guarantees, partitioning, and multiple concurrent readers
- Expose that same dataset to a serverless cloud query engine (Athena) for teams that want SQL without managing infrastructure
- Expose it to on-premises engines (Trino and Spark, via Docker) for teams that want local, high-performance or large-scale distributed querying
- Enforce access per role (Data Engineer/Admin, Data Engineer/Analyst, ML/AI Engineer) through S3 and Glue IAM policies, rather than per query engine
- Keep an ML/AI training path available off the same Gold data, separate from the three query engines
Everything below, from ingestion through the Glue ETL job and the three query engines, is the system built to solve that problem end to end.
This project demonstrates a cloud-native Data Lakehouse on AWS, built around the Gold layer of the Medallion Architecture. It ingests financial data from a Trading API/Platform, converts raw CSV to Parquet via an AWS Glue ETL job, and exposes a curated Gold dataset to multiple consumer roles via three independent query engines, with an optional ML/AI path.
%20Mar%202026.png)
Raw financial data is streamed from a Trading API/Platform and stored as CSV in S3. An AWS Glue ETL job converts the raw CSV into Parquet format and loads it into the Gold S3 bucket. Access to both the raw bucket and Glue job is governed by S3 + Glue IAM policies scoped per user/role. From the Gold dataset, three independent query engines are available:
- QE 1 — AWS Athena (Cloud): Data Engineers and Analysts query the Gold dataset via Apache Iceberg tables with partitioning using AWS Athena, enabling versioned, ACID-compliant serverless SQL analytics directly in the cloud.
- QE 2 — On-Premises Trino (Docker): Data Engineers and Analysts can run high-performance SQL queries locally via a Trino query engine deployed in Docker, connecting back to the Gold S3 dataset.
- QE 3 — On-Premises Spark (Docker): Data Engineers and Analysts can run large-scale distributed analytics locally via an Apache Spark query engine deployed in Docker, also connecting to the Gold S3 dataset. Additionally, ML/AI Engineers have read access to the Gold dataset to optionally train and deploy ML models — this path is separate from the three query engines.
| Component | Description |
|---|---|
| Trading API / Platform | Source of real-time financial market data |
| S3 Raw Bucket | Landing zone storing ingested data as CSV |
| Component | Description |
|---|---|
| AWS Glue ETL Job | Converts raw CSV to Parquet format |
| S3 Gold Bucket | Stores the curated, query-optimised Gold dataset |
| S3 + Glue IAM Policies | Scoped access policies enforced per user/role |
| Component | Description |
|---|---|
| AWS Athena + Apache Iceberg Tables (QE 1) | Serverless cloud SQL querying on partitioned, versioned Iceberg tables in S3 |
| On-Prem Trino Query Engine (QE 2) | Local high-performance SQL analytics via Docker |
| On-Prem Spark Query Engine (QE 3) | Local distributed data processing and analytics via Docker |
| ML/AI Model Training (Optional) | Model training and deployment on the Gold dataset |
| Role | Access |
|---|---|
| Data Engineer + Admin | Full access — ingestion, raw S3 bucket, Glue ETL job |
| Data Engineer / Data Analyst | Gold S3 bucket, Iceberg tables (QE 1), Trino (QE 2), Spark (QE 3) |
| ML / AI Engineer | Read access to Gold S3 dataset |
- Amazon S3 — object storage for raw and Gold layers
- AWS Glue — serverless ETL (CSV → Parquet)
- AWS IAM — role-based access control via S3 + Glue policies
- Apache Iceberg — open table format with partitioning and versioning (QE 1)
- AWS Athena — serverless cloud SQL query engine for Iceberg tables (QE 1)
- Trino — on-premises SQL query engine via Docker (QE 2)
- Apache Spark — on-premises distributed query engine via Docker (QE 3)
- Docker — container runtime for on-premises query engines
- Trading API / Platform — financial market data source
For further assistance, please visit my YouTube channel, I have a fully detailed video explaining how to setup the components of this repository. YouTube Channel: https://www.youtube.com/@BDB5905