This project is designed to process mediation data using SQLite and pandas. It involves reading data from CSV files, inserting them into a local SQLite database, performing aggregations, and writing the results to CSV.
- Reads mediation data and application data.
- Inserts the data into a SQLite database.
- Performs aggregations based on the data.
- Writes the aggregated data into a new CSV file: 'exercise_1/data/processed/mediation_aggregation.csv'.
Ensure you have Python 3.12 installed.
-
Clone this repository:
git clone git@github.com:noam-meron/data_eng_home_assignment.git
-
Create virtual environment:
python -m venv venv
-
Activate the virtual environment:
source venv/bin/activate -
Install the required packages:
pip install -r requirements.txt
-
Run the main script:
python -m exercise_1.main