I delivered on a complete, end-to-end, financial ETL pipeline. I learned
- Apache Airflow
- Scala
- Apache Spark
- Grafana
- Kafka (soon)
- Broker-controller (static scaling)
- Calculate important events to trigger email alerts
- Loads 20+ years of stock price data from the S&P 500
- Performs quantitative analysis (currently, parameterized SMA, EMA)
- Visualizes the results
I knew PostgreSQL, Docker (and compose), and Python (yeah, Duh!)
Pretty nifty? There's more to come!