A scalable, event-driven architecture that automates metadata extraction from structured documents (e.g., CSV) using AWS services like S3, SQS, Lambda (Dockerized), and DynamoDB. The system processes uploaded files in real-time and stores extracted metadata in a NoSQL database — with zero manual effort.
🎥 Live Demo: Watch on YouTube
📦 GitHub Repo: https://github.com/Nagamedha/CC_Project
- Fully automated pipeline: File upload → Trigger → Metadata extraction → Storage
- Serverless-first design: Uses AWS Lambda, S3, SQS, DynamoDB
- Modular infrastructure: Supports routing and processing Lambdas per data type (e.g., sales, inventory)
- Fault-tolerant: DLQ handling for failed events
- Cloud-native scalability: Built using VPC, subnets, ECR, CloudFormation stacks
- Real-time storage: Output saved in DynamoDB and visualized using DynamoDB Workbench
Om-insights-backend/
├── src/ # Main Lambda entry point
├── file_processor/ # Metadata extraction logic
├── infrastructure/ # CDK infrastructure (VPC, S3, Lambda, SQS)
├── tests/ # Unit and integration tests
├── Dockerfile # Docker config for Lambda
├── requirements.txt # Python dependencies
└── README.md # You're here!
| Layer | Tool/Service Used |
|---|---|
| File Storage | Amazon S3 |
| Message Queue | Amazon SQS + Dead Letter Queue |
| Compute | AWS Lambda (Docker container via ECR) |
| Storage | Amazon DynamoDB |
| Monitoring | Amazon CloudWatch |
| Dev Tools | Python, PyCharm Community, GitHub |
| Visualization | DynamoDB Workbench |
| Deployment | AWS CDK (CloudFormation stacks) |
This section is for anyone who wants to download, set up, and run the project locally or deploy it to AWS.
git clone https://github.com/Nagamedha/CC_Project.git
cd CC_Project/Om-insights-backendpython3 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txtYou’ll need the following tools:
| Tool | Purpose | Install Command / Link |
|---|---|---|
| AWS CLI | Access & deploy to AWS | Install Guide |
| Docker | Build Lambda images | Download Docker |
| Node.js | For AWS CDK | Download Node.js |
| AWS CDK | Deploy infrastructure | npm install -g aws-cdk |
| AWS Account | Create & configure AWS profile | Run aws configure to set keys |
aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-ecr-repo-uri>Replace <your-region> and <your-ecr-repo-uri> with actual values.
From the project root:
docker build -t routing-lambda ./file_processor/
docker tag routing-lambda:latest <your-ecr-repo-uri>:latest
docker push <your-ecr-repo-uri>:latestNavigate to the infrastructure directory:
cd infrastructure/
npm install
cdk bootstrap
cdk deploy --allThis command deploys:
- VPC & subnets
- S3 bucket
- SQS queues + DLQs
- Lambda functions (Routing + Processing)
- DynamoDB tables (e.g.,
sales_table)
- Upload a CSV file to S3 (via AWS Console)
- S3 triggers an event, message added to SQS
- Lambda is triggered, Docker image is used to process
- Metadata is extracted and saved to DynamoDB
- View the result in DynamoDB Workbench or via:
aws dynamodb scan --table-name sales_table --region <your-region>docker run -v "$PWD":/var/task routing-lambda:latest- Add a new file type: create a new Processor Lambda under
file_processor/and register it in the router. - Add a new DynamoDB table: update the CDK stack in
infrastructure/processing_stack.py. - Want an API? Add API Gateway in CDK and connect to routing Lambda.
- 100% automation from upload to storage
- ~90% manual effort reduced
- Processed in seconds, not hours
- Supports multiple file types and projects
- Easily scalable across teams or use cases
- Support for PDF, Excel, and XML
- AI-based validation and smart column mapping
- Frontend for upload, status tracking, and querying
- Email or SNS notifications post-processing
- Enterprise-scale support with usage-based logging
Thank you for exploring our project!