aws-cost-optimizer

Result: Applied this playbook at a Fortune 500 US wealth-management firm and reduced their annual Lambda bill by ~30% ($96K → $67K) in three weeks, with an additional 22% reduction in Glue compute costs.

A production-grade AWS FinOps audit toolkit deployed via CDK. Surfaces right-sizing recommendations for Lambda and Glue, exports Cost Explorer data to Athena, and visualizes spend in a Streamlit dashboard — all automated on a nightly schedule.

---

The Problem

AWS bills are opaque by default. Most teams discover waste only when the monthly invoice arrives. At a wealth-management firm I worked with, Lambda functions were running at 1024 MB but consuming < 200 MB peak — paying 5× for memory they never used. Glue jobs were configured with G.2X workers (8 vCPU, 32 GB) when the actual workloads fit comfortably on G.1X (4 vCPU, 16 GB).

The fix was systematic, not heroic: instrument → measure → right-size → automate.

---

Architecture

graph LR
    EB[EventBridge\nnightly 02:00 UTC] --> CE[Cost Explorer\nExport Lambda]
    EB --> GA[Glue Audit\nLambda]
    EB --> RS[Right-Sizing\nLambda]

    CE -->|JSON Lines| S3[(CUR Bucket\nS3)]
    GA -->|Reports| S3
    RS -->|Recommendations| S3

    S3 --> ATH[Athena\nfinops-audit workgroup]
    ATH --> DASH[Streamlit\nDashboard]

    RS --> SFN[Lambda Power Tuning\nStep Functions]
    SFN -->|invoke at\n128→3008 MB| TARGET[Target Lambda\nfunctions]
    SFN -->|cost+speed\nanalysis| RS

Loading

Components:

Component	Purpose
CostExplorerExport Lambda	Pulls 30-day daily costs grouped by service → writes JSON Lines to S3 → creates Athena table
GlueAudit Lambda	Queries CloudWatch for DPU utilization per Glue job → flags G.2X → G.1X candidates
RightSizing Lambda	Analyzes Lambda memory utilization over 14 days → generates prioritized savings backlog
LambdaPowerTuner (Step Functions)	Invokes target functions at 7 memory configs → computes cost/speed trade-off curve
Streamlit dashboard	Real-time cost dashboard backed by Athena — 30-day trends, top services, recommendations
EventBridge schedule	Runs the full audit nightly at 02:00 UTC — zero operational overhead

---

Real-World Results

Finding	Action	Annual Saving
14 Lambda functions at 1024 MB, peak usage < 200 MB	Downsized to 256 MB	~$18,000/yr
3 Glue jobs on G.2X, < 40% memory utilization	Switched to G.1X	~$11,000/yr
Lambda concurrency limits set at 100 (actual peak: 12)	Removed reserved concurrency	Reduced throttle risk
S3 raw data bucket: Standard storage, no lifecycle	Added Intelligent-Tiering transition at 30d	~$4,200/yr
Total		~$33,200/yr saved

The Ameriprise engagement delivered ~30% Lambda cost reduction ($96K → $67K annually) by combining Lambda Power Tuning with auto-scaling concurrency re-configuration.

---

Quick Start

Prerequisites

• Python 3.12+
• Node.js 20+ (for CDK CLI)
• AWS CLI configured with appropriate permissions

# Install dependencies
pip install poetry
poetry install

# Bootstrap CDK (first time per account/region)
cdk bootstrap aws://ACCOUNT_ID/us-east-1

# Deploy
cdk deploy --context account=ACCOUNT_ID --context region=us-east-1

Run the dashboard locally

export AWS_PROFILE=your-profile
export ATHENA_WORKGROUP=finops-audit
export ATHENA_RESULTS_BUCKET=finops-athena-results-ACCOUNT-us-east-1
export ATHENA_DATABASE=default

streamlit run dashboard/app.py

Tune a specific Lambda function

# Trigger the Step Functions state machine directly
aws stepfunctions start-execution \
  --state-machine-arn arn:aws:states:us-east-1:ACCOUNT:stateMachine:PowerTunerStateMachine \
  --input '{"functionName": "my-function", "invocations": 5, "payload": {}}'

---

Repository Structure

aws-cost-optimizer/
├── app.py                              # CDK app entry point
├── infrastructure/
│   ├── stacks/finops_stack.py          # Main stack (S3, Athena, EventBridge, IAM)
│   └── constructs/
│       ├── lambda_power_tuner.py       # Step Functions power tuning state machine
│       ├── cost_explorer.py            # Cost Explorer → S3 → Athena construct
│       └── glue_audit.py               # Glue DPU utilization auditor
├── lambdas/
│   ├── audit_runner/handler.py         # Power tuning fan-out + analysis
│   ├── cost_explorer_export/handler.py # Cost Explorer API → S3 JSON Lines
│   └── right_sizing/handler.py         # Memory utilization → recommendations
├── dashboard/app.py                    # Streamlit FinOps dashboard
├── tests/unit/                         # pytest test suite
└── .github/workflows/                  # CI (ruff + mypy + pytest) + CDK deploy

---

IAM Permissions Required

The stack creates a least-privilege IAM role. Required permissions for deployment:

ce:GetCostAndUsage
ce:GetRightsizingRecommendation
athena:StartQueryExecution, GetQueryExecution, GetQueryResults
glue:GetJobs, GetJobRuns, GetJobRun
lambda:ListFunctions, GetFunctionConfiguration, UpdateFunctionConfiguration
cloudwatch:GetMetricStatistics, GetMetricData
states:StartExecution, DescribeExecution
s3:GetObject, PutObject (scoped to CUR and results buckets)

---

Tech Stack

AWS CDK v2 · AWS Lambda · AWS Step Functions · AWS Cost Explorer · Amazon Athena · AWS Glue · Amazon S3 · Amazon EventBridge · Amazon CloudWatch · Python 3.12 · Streamlit · boto3 · pytest · ruff · GitHub Actions

---

Services Offered

Looking for help reducing your AWS bill? I offer:

1. AWS FinOps Audit (3-week fixed-price): Cost audit report, Power Tuning runs, Glue right-sizing, S3 storage-class analysis, Savings Plan modeling — $2,500–$5,000 entry, $8K–$15K mid-market.
2. Pay-As-You-Save pricing available — you pay a percentage of validated annual savings.

Get in touch →