AeroPipe

An Ingestion pipeline built with Apache Airflow that pulls NASA's Astronomy Picture of the Day (APOD) data daily and loads it into a Postgres database on AWS RDS. Airflow runs locally via the Astronomer CLI for development and deploys to Astro Cloud for production.

What it does

The DAG aeropipe_nasa_apod runs on a daily schedule (catchup=False) and executes four tasks in order:

1. create_table: creates the apod_data table in RDS if it doesn't already exist
2. extract_apod: calls GET /planetary/apod on NASA's API via SimpleHttpOperator, returns JSON
3. transform_apod_data: pulls out title, explanation, url, date, and media_type from the response
4. load_data_to_postgres: inserts the record into apod_data using PostgresHook

Stack

• Apache Airflow on Astro Runtime 12.1.1 (Astronomer CLI)
• Postgres 13 on AWS RDS
• Operators: SimpleHttpOperator, PostgresHook, TaskFlow API (@task)

Local development

1. Set up Airflow connections

NASA API:

• Connection ID: nasa_api
• Conn type: HTTP
• Host: https://api.nasa.gov
• Extra: {"api_key": "your_key_here"}

Postgres (RDS):

• Connection ID: my_postgres_connection
• Conn type: Postgres
• Host: your RDS endpoint (e.g. your-instance.xxxx.us-east-1.rds.amazonaws.com)
• Port: 5432
• DB / Login / Password: as configured in your RDS instance

2. Start Airflow

astro dev start

Airflow UI runs at http://localhost:8080. Enable aeropipe_nasa_apod there or trigger it manually.

The docker-compose.yml in this repo spins up a local Postgres container for development if you don't want to point at RDS during testing.

Deploy to Astro Cloud

astro login
astro deployment create
astro deploy

After deploying, re-add both connections (nasa_api and my_postgres_connection) in the Astro UI under your deployment's Connections settings. Make sure your RDS security group allows inbound traffic on port 5432 from Astro Cloud's IP range.

Project structure

aeropipe/
├── dags/
│   └── etl.py              # DAG definition
├── docker-compose.yml       # Local Postgres (dev only)
├── Dockerfile               # Astro Runtime 12.1.1
├── requirements.txt
└── .astro/
    └── config.yaml