This repository contains a full-stack ETL (Extract, Transform, Load) data pipeline that ingests Reddit data and transforms it into structured insights within an Amazon Redshift data warehouse. This architecture is designed for scalability, automation, and robust data processing, leveraging a combination of industry-standard tools and AWS services.
The pipeline is designed to:
- Real-time Data Extraction: Retrieves data from Reddit using its API.
- Scalable Orchestration: Manages workflows and task distribution using Apache Airflow and Celery.
- Efficient Data Storage: Temporarily stages data in PostgreSQL and archives raw data in S3.
- Automated Data Transformation: Processes raw data using AWS Glue and Athena for efficient querying.
- Comprehensive Analytics: Loads structured data into Amazon Redshift for advanced analytics and business intelligence.
- Reddit API : Data source for extracting posts and comments.
- Apache Airflow & Celery: Orchestration for the entire ETL pipeline.
- PostgreSQL: Temporary staging and metadata management.
- Amazon S3 : Long-term raw data storage.
- AWS Glue : Automated ETL and data cataloging.
- Amazon Athena : SQL-based data transformation and analysis.
- Amazon Redshift –: Scalable data warehousing for complex analytics.
- Reddit API credentials : Data Extraction
- AWS Account with the following service permissions:
- S3, Glue, Athena, and Redshift
- Docker : For containerized deployment
- Python 3.9+ : For the pipeline and related scripts
- Clone the repository.
git clone https://github.com/AjDevCode/RedditETLPipeline.git
- Create a virtual environment.
python3 -m venv venv
- Activate the virtual environment.
source venv/bin/activate - Install the dependencies.
pip install -r requirements.txt
- Rename the configuration file and the credentials to the file.
mv config/config.conf.example config/config.conf
- Starting the containers
docker-compose up -d
- Launch the Airflow web UI.
open http://localhost:8080