Skip to content

README.md

Latest commit

 

History

History
331 lines (290 loc) · 14.1 KB

README.md

File metadata and controls

331 lines (290 loc) · 14.1 KB

Automation Executor

A standalone automation executor for JSON-defined workflows with HTTP requests, webhooks, scheduled executions, and AI agent automations. Build powerful automations using simple JSON configurations with multi-provider AI support.

🏗️ System Architecture

                    ┌─────────────────────────────────────────────────────────┐
                    │              AUTOMATION EXECUTOR SYSTEM                 │
                    │                 (1000+ RPS Capacity)                   │
                    └─────────────────────────────────────────────────────────┘
                                              │
                    ┌─────────────────────────┼─────────────────────────┐
                    │                         │                         │
            ┌───────────────┐         ┌─────────────┐         ┌───────────────┐
            │  👥 Web Apps  │         │ 📱 Mobile   │         │ 🔗 API Clients│
            │   (React)     │────────▶│    Apps     │◀────────│   (cURL)     │
            └───────────────┘         └─────────────┘         └───────────────┘
                                              │
                                   ┌─────────────────┐
                                   │ 🌐 Load Balancer │
                                   │   (Auto-Scale)   │
                                   └─────────────────┘
                                              │
                                   ┌─────────────────┐
                                   │ 🚪 API Gateway  │
                                   │ • Authentication │
                                   │ • Rate Limiting  │
                                   │ • Request Valid. │
                                   └─────────────────┘
                                              │
        ┌───────────────────────────────────────────────────────────────────────────┐
        │                           🎯 EXECUTOR CORE                               │
        ├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
        │ 📋 Workflow  │ ⚡ Execution │ 🔐 Auth      │ 🤖 AI Agent │ 🔧 Node     │
        │    Manager   │    Queue     │   Service    │   Service    │  Executors   │
        │              │              │              │              │              │
        │ • Templates  │ • 4 Priority │ • API Keys   │ • OpenAI     │ • Code Exec  │
        │ • User Flows │ • 20 Workers │ • Quotas     │ • Google AI  │ • HTTP Calls │
        │ • Categories │ • Round Robin│ • Rate Limit │ • OpenRouter │ • Webhooks   │
        │ • Dynamic    │ • Timeout    │ • JWT        │ • Multi-Tool │ • Scheduling │
        │   Loading    │   Handling   │   Support    │   Support    │ • AI Agents  │
        └──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘
                                              │
        ┌───────────────────────────────────────────────────────────────────────────┐
        │                          💾 STORAGE LAYER                               │
        ├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
        │ 📁 Workflows │ 👥 User Data │ ⚡ Cache     │ 🔒 Secrets   │ 📊 Metrics  │
        │              │              │              │              │              │
        │ • Templates  │ • Profiles   │ • Results(1h)│ • API Keys   │ • Usage Stats│
        │ • Custom     │ • Usage      │ • Queue      │ • Configs    │ • Performance│
        │ • Public     │ • Quotas     │   Status     │ • Encrypted  │ • Health     │
        │ (File System)│ (Database)   │ (Memory)     │ (Vault)      │ (Time Series)│
        └──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘
                                              │
        ┌───────────────────────────────────────────────────────────────────────────┐
        │                      🌍 MULTI-CLOUD DEPLOYMENT                          │
        ├─────────────────┬─────────────────┬─────────────────┬─────────────────────┤
        │ ☁️ Google Cloud │ ☁️ Amazon AWS   │ ☸️ Kubernetes   │ 🐳 Docker Compose │
        │                 │                 │                 │                     │
        │ • Cloud Run     │ • ECS Fargate   │ • Any Provider  │ • Local/VPS        │
        │ • 0-1000 scale  │ • 2-100 tasks   │ • Full Control  │ • Development      │
        │ • Serverless    │ • VPC Control   │ • Portable      │ • Testing          │
        │ • Pay-per-req   │ • 35% cheaper   │ • Complex       │ • Simple Setup     │
        └─────────────────┴─────────────────┴─────────────────┴─────────────────────┘

🤖 AI Agent Capabilities

The system includes advanced AI agent functionality with multi-tool support:

Multi-Provider AI Support

  • OpenAI: GPT-3.5, GPT-4, GPT-4-turbo models
  • Google AI: Gemini Pro, Gemini Ultra models
  • OpenRouter: Access to 150+ models (Claude, Llama, etc.)

AI Agent Tools

  • Web Search: Real-time internet search capabilities
  • HTTP Requests: API calls with authentication
  • Code Execution: Run Python/JavaScript in sandbox
  • File Operations: Read/write files securely
  • Database Queries: Connect to external databases
  • Email/SMS: Send notifications and alerts

Agent Workflow Examples

{
  "id": "ai-research-agent",
  "name": "AI Research Assistant",
  "nodes": [
    {
      "type": "aiAgent",
      "parameters": {
        "provider": "openai",
        "model": "gpt-4",
        "tools": ["web_search", "http_request", "code_execution"],
        "prompt": "Research and analyze the latest trends in {{topic}}"
      }
    }
  ]
}

🏛️ Architecture Explanation

System Overview

The Automation Executor is a multi-cloud, serverless automation platform designed to handle 1000+ requests per second with automatic scaling and high availability.

🎯 Core Components

1. API Gateway Layer

  • Load Balancer: Distributes traffic across multiple instances
  • Authentication: API key-based auth with JWT support
  • Rate Limiting: Per-user quotas (Free: 1K, Pro: 10K, Enterprise: 100K)
  • Request Validation: JSON payload validation and sanitization

2. Workflow Management System

  • Template Engine: 8+ pre-built workflow templates
  • Dynamic Loading: Workflows loaded from file system
  • User Workflows: Custom workflows per user
  • Categories: AI, Tools, Scheduled, HTTP workflows

3. Execution Engine

  • Priority Queue: 4 levels (urgent → high → normal → low)
  • Worker Pool: Configurable workers (default: 20 per instance)
  • Round-Robin: Load balancing across workers
  • Timeout Handling: 30-second default timeout
  • Result Caching: 1-hour memory cache for results

4. AI Integration Layer

  • Multi-Provider Support: OpenAI, Google AI, OpenRouter
  • Model Selection: Dynamic model routing
  • Token Management: Usage tracking per provider
  • Error Handling: Fallback and retry logic

📊 Performance Architecture

Horizontal Scaling

Load Balancer
     │
     ├── Instance 1 (80 concurrent)
     ├── Instance 2 (80 concurrent)  
     ├── Instance 3 (80 concurrent)
     └── Instance N (80 concurrent)
     
Total Capacity = N × 80 concurrent requests
Target: 1000+ RPS sustained

Execution Flow

  1. Request → API Gateway (auth + validation)
  2. Workflow Resolution → Template/Custom/Main workflow
  3. Queue Assignment → Priority-based queuing
  4. Worker Assignment → Round-robin selection
  5. Execution → Isolated workflow execution
  6. Result Storage → 1-hour result caching
  7. Response → JSON result or execution ID

☁️ Multi-Cloud Deployment

Google Cloud Platform (Recommended for startups)

  • Service: Cloud Run (serverless containers)
  • Scaling: 0-1000 instances, instant scaling
  • Cost: Pay-per-request model
  • Deployment: 3 commands to deploy

Amazon Web Services (Recommended for enterprise)

  • Service: ECS Fargate (managed containers)
  • Scaling: 2-100 tasks, gradual scaling
  • Cost: 35% cheaper for sustained workloads
  • Deployment: Full VPC control, more setup steps

Kubernetes (Any provider)

  • Flexibility: Complete control over infrastructure
  • Portability: Deploy anywhere (Azure, DigitalOcean, on-premise)
  • Complexity: Requires K8s expertise

🔒 Security Architecture

  • API Key Hashing: SHA-256 hashed keys
  • Secret Management: Cloud-native secret stores
  • VM Sandboxing: Isolated code execution
  • CORS Configuration: Web security
  • Non-root Containers: Security best practices

📈 Monitoring & Observability

  • Health Endpoints: /health, /api/status, /api/v1/queue/status
  • Metrics: Request rate, response time, error rate, queue depth
  • Alerting: High error rate, response time, queue backlog
  • Logging: Structured JSON logs with request tracing

Features

  • Code Execution: Run JavaScript code with sandbox isolation
  • HTTP Requests: Make HTTP calls to external APIs
  • Webhooks: Trigger workflows via HTTP webhooks
  • Scheduling: Schedule workflows with cron expressions
  • JSON Workflows: Define workflows in JSON format
  • Multi-AI Providers: OpenAI, Google AI, OpenRouter support
  • Auto-scaling: Handle 1000+ requests per second
  • Multi-cloud: Deploy on GCP, AWS, or Kubernetes

Installation

npm install
npm run build

Usage

Start the server

npm start
# or for development
npm run dev

Execute a workflow

curl -X POST http://localhost:3000/execute \
  -H "Content-Type: application/json" \
  -d '{
    "workflow": {
      "id": "test-workflow",
      "name": "Test Workflow",
      "nodes": [
        {
          "id": "code-1",
          "type": "code",
          "position": [0, 0],
          "parameters": {
            "code": "return { message: \"Hello World\", input: $input };"
          }
        }
      ],
      "connections": []
    },
    "data": { "test": "data" }
  }'

Schedule a workflow

curl -X POST http://localhost:3000/schedule \
  -H "Content-Type: application/json" \
  -d '{
    "workflow": {
      "id": "scheduled-workflow",
      "name": "Scheduled Test",
      "nodes": [
        {
          "id": "code-1",
          "type": "code",
          "position": [0, 0],
          "parameters": {
            "code": "console.log(\"Scheduled execution at\", new Date()); return { executed: true };"
          }
        }
      ],
      "connections": []
    },
    "schedule": "*/5 * * * *"
  }'

Webhook execution

curl -X POST http://localhost:3000/webhook/my-workflow-id \
  -H "Content-Type: application/json" \
  -d '{ "message": "Hello from webhook" }'

Workflow JSON Format

{
  "id": "unique-workflow-id",
  "name": "My Workflow",
  "nodes": [
    {
      "id": "node-1",
      "type": "code|http|webhook",
      "position": [x, y],
      "parameters": {
        // Node-specific parameters
      }
    }
  ],
  "connections": [
    {
      "source": "node-1",
      "target": "node-2"
    }
  ]
}

Node Types

Code Node

{
  "id": "code-1",
  "type": "code",
  "parameters": {
    "code": "return { result: $input.value * 2 };"
  }
}

HTTP Node

{
  "id": "http-1",
  "type": "http",
  "parameters": {
    "url": "https://api.example.com/data",
    "method": "POST",
    "headers": { "Authorization": "Bearer token" },
    "body": { "key": "value" }
  }
}

Deployment to Google Cloud

  1. Build the project: npm run build
  2. Create a Dockerfile for containerization
  3. Deploy to Google Cloud Run or App Engine
  4. Set environment variables as needed