Data Science, AI & Cloud Project Experiments

This repository is dedicated to experiments, implementations, and hands-on projects spanning Data Science, AI, and Cloud technologies.

---

🚀 Event-Driven Document Processing Pipeline on Google Cloud

A serverless, event-driven pipeline that ingests files uploaded to a Cloud Storage bucket, triggers document analysis using Pub/Sub notifications, processes the files in a Python FastAPI Cloud Run service (simulated OCR & metadata extraction), and streams the extracted structured metadata into a BigQuery partitioned table.

📐 Architecture Design

graph TD
    User([User / Client]) -->|Upload File| GCS[Cloud Storage Ingestion Bucket]
    GCS -->|Object Created Notification| PubSub[Pub/Sub Topic]
    PubSub -->|Push Subscription OIDC Token Delivery| CloudRun[Cloud Run Processor - Python FastAPI]
    CloudRun -->|Read File Content| GCS
    CloudRun -->|Simulated OCR & Tag Extraction| GCS
    CloudRun -->|Stream Structured Rows| BigQuery[(BigQuery Partitioned Table)]

Loading

📂 Repository Structure

• src/: Python source code of the Cloud Run microservice.
  • main.py: FastAPI service entry point with simulated OCR logic and BigQuery streaming integration.
  • requirements.txt: Python package requirements.
  • Dockerfile: Packaging configuration for containerization.
• terraform/: Infrastructure as Code configs.
  • main.tf: Resources definition (GCS, Pub/Sub, BigQuery, Cloud Run, IAM).
  • variables.tf: Deploy configurations.
  • outputs.tf: Endpoint outputs.
• scripts/: Development and testing helper tools.
  • emulate_upload.py: Local offline testing script that mocks GCS events.

---

🛠️ Local Development & Testing

You can run the FastAPI processor locally without any GCP credentials. In this mode, the server uses fallback mock data and prints output to the terminal rather than invoking actual storage or database APIs.

1. Install Python dependencies:

   python3 -m pip install -r src/requirements.txt

2. Start the FastAPI microservice:

   python3 -m uvicorn src.main:app --port 8080

3. Simulate a GCS upload notification: Open a separate terminal window and run:

   python3 scripts/emulate_upload.py

---

🚢 Google Cloud Platform Deployment

Refer to the complete deployment walkthrough in walkthrough.md for full commands.

1. Build and push the Docker image to Artifact Registry:

   gcloud artifacts repositories create doc-pipeline-repo --repository-format=docker --location=us-central1
   gcloud auth configure-docker us-central1-docker.pkg.dev
   docker build -t us-central1-docker.pkg.dev/YOUR_PROJECT_ID/doc-pipeline-repo/processor:latest ./src
   docker push us-central1-docker.pkg.dev/YOUR_PROJECT_ID/doc-pipeline-repo/processor:latest

2. Initialize and apply Terraform provisioning:

   cd terraform
   terraform init
   terraform apply \
       -var="project_id=YOUR_PROJECT_ID" \
       -var="bucket_name=YOUR_UNIQUE_UPLOAD_BUCKET" \
       -var="container_image=us-central1-docker.pkg.dev/YOUR_PROJECT_ID/doc-pipeline-repo/processor:latest"