Quelm

A distributed multi-agent workflow platform where AI agents collaborate asynchronously to execute complex workflows at scale. Built with event-driven architecture, message queues, real-time orchestration, and a visual workflow builder.

Status: Active Development | Stack: TypeScript · Next.js · Express · BullMQ · PostgreSQL · Redis · Groq · Vitest

Overview

Quelm is a general-purpose distributed workflow platform where users visually compose workflows by connecting AI agents on a canvas, trigger runs with a JSON input payload, and watch them execute live in real time.

Under the hood, tasks flow through BullMQ queues on Redis, workers process them independently with automatic retries and exponential backoff, and an event-driven orchestrator dispatches downstream tasks as dependencies complete. Every execution is fully persisted in PostgreSQL with a complete audit trail.

Features

Visual Workflow Builder — drag and drop agent nodes onto a canvas, connect them with edges, configure each node with prompts, URLs, or transformation logic
Live Run Monitor — watch a workflow execute in real time with per-node status updates via Server-Sent Events
Distributed Workers — each agent type runs as an independent worker process, scales horizontally without code changes
Dependency Resolution — parallel execution, fan-out, and sequential chains all handled automatically based on the graph structure
Retry Logic — exponential backoff with configurable max attempts per task, dead-letter handling for permanently failed jobs
Critical Task Control — mark any node as critical to fail the entire workflow on failure, or non-critical to skip and continue
Agent Heartbeat — every worker reports its liveness every 30 seconds, the dashboard shows real-time agent health
Modular Agent System — extend BaseAgent to add any new agent type without touching the orchestrator
Structured Observability — structured JSON logs in production, per-request logging, full task input/output history in the database

Architecture

flowchart TB
    subgraph Client["Client (Next.js + Vercel)"]
        WB["Workflow Builder\nReact Flow canvas"]
        RM["Run Monitor\nLive SSE updates"]
        DB["Dashboard\nMetrics & agents"]
    end

    subgraph API["REST API (Express)"]
        WR["/api/workflows"]
        RR["/api/runs"]
        AR["/api/agents"]
        SSE["/api/runs/:id/stream\nSSE endpoint"]
    end

    subgraph Orchestrator["Orchestrator"]
        OC["Workflow engine\nDependency resolver"]
        QE["QueueEvents\nCompletion listener"]
        RE["RunEmitter\nEvent broadcast"]
    end

    subgraph Queues["BullMQ Queues (Redis)"]
        LQ["LLM_AGENT"]
        HQ["HTTP_AGENT"]
        TQ["TRANSFORM_AGENT"]
        EQ["EXTRACTION_AGENT"]
        NQ["NOTIFICATION_AGENT"]
        SQ["STORAGE_AGENT"]
    end

    subgraph Workers["Agent Workers"]
        LA["LLM Agent\nGroq API"]
        HA["HTTP Agent"]
        TA["Transform Agent"]
    end

    subgraph Storage["Storage"]
        PG[("PostgreSQL\nWorkflows · Runs · Tasks · Agents")]
        RD[("Redis\nJob queues · State")]
    end

    WB -->|"POST /api/workflows"| WR
    WB -->|"POST /api/workflows/:id/run"| WR
    RM -->|"GET /api/runs/:id/stream"| SSE
    DB -->|"GET /api/agents"| AR
    DB -->|"GET /api/runs"| RR

    WR --> OC
    OC -->|"Dispatch first tasks"| LQ
    OC -->|"Dispatch first tasks"| HQ
    QE -->|"completed / failed"| OC
    OC -->|"Dispatch next tasks"| LQ
    OC --> RE
    RE -->|"Push status event"| SSE

    LQ -->|"Pull job"| LA
    HQ -->|"Pull job"| HA
    TQ -->|"Pull job"| TA

    LA -->|"Update task"| PG
    HA -->|"Update task"| PG
    TA -->|"Update task"| PG

    OC -->|"Update run state"| PG
    API --> PG
    Queues --- RD

Tech Stack

Layer	Technology	Purpose
Language	TypeScript	End-to-end type safety
Package Manager	pnpm	Fast installs, strict workspace support
Frontend	Next.js + Tailwind + shadcn/ui	Full-stack React, deploys to Vercel
Workflow Canvas	React Flow	Node-based visual editor
Backend	Node.js + Express	REST API server
Queue System	BullMQ + Redis	Task distribution, retries, job state
Database	PostgreSQL + Prisma	Persistent workflow and run state
AI Provider	Groq (LLaMA 3.3 70B)	Fast LLM inference, generous free tier
Real-time	Server-Sent Events	Live run monitor updates
Containerization	Docker Compose	Local Postgres and Redis
Testing	Vitest + Supertest	Unit, integration, and API tests
Deployment	Vercel + Render	Frontend and backend hosting

Project Structure

quelm/
├── client/                         # Next.js frontend
│   ├── __tests__/                  # Test Suite
│   ├── app/                        # App router pages
│   ├── components/                 # React components
│   ├── hooks/                      # Custom hooks for API calling
│   ├── lib/                        # Utility methods and types
│   ├── providers/                  # Context providers
│   └── package.json
├── server/                         # Express backend
│   ├── agents/                     # Agent worker implementations
│   │   ├── base.agent.ts           # Abstract base class
│   │   ├── llm.agent.ts            # Groq LLM agent
│   │   └── registry.ts             # Agent startup registry
│   ├── __tests__/                  # Test suite
│   │   ├── api/                    # API integration tests (supertest)
│   │   ├── helpers/                # Mock factories and app builder
│   │   ├── orchestrator/           # Orchestrator workflow tests
│   │   ├── services/               # Service layer unit tests
│   │   ├── utils/                  # Utility function tests
│   │   └── setup.ts                # Global mocks and env config
│   ├── agents/                     # Agent worker implementations
│   │   ├── base.agent.ts           # Abstract base class
│   │   ├── llm.agent.ts            # Groq LLM agent
│   │   └── registry.ts             # Agent startup registry
│   ├── api/                        # Modular REST API
│   │   ├── workflow/               # Workflow module
│   │   ├── run/                    # Run module
│   │   └── agent/                  # Agent module
│   ├── config/                     # App configuration
│   │   ├── index.ts                # Typed env variables
│   │   ├── logger.config.ts        # Winston logger
│   │   ├── prisma.config.ts        # Prisma singleton
│   │   ├── redis.config.ts         # Redis singleton
│   │   └── groq.config.ts          # Groq client singleton
│   ├── events/
│   │   └── run.emitter.ts          # In-process event emitter for SSE
│   ├── orchestrator/
│   │   └── index.ts                # Workflow orchestration engine
│   ├── queue/
│   │   └── index.ts                # BullMQ queue abstraction
│   ├── middleware/
│   │   └── error.middleware.ts     # Global error handler
│   ├── prisma/
│   │   └── schema.prisma           # Database schema
│   ├── utils/
│   │   ├── errors.ts               # Typed API error classes
│   │   ├── types.ts                # Shared TypeScript types
│   │   └── template.utils.ts       # Prompt interpolation
│   └── index.ts                    # Server entry point
├── docker-compose.yml              # Postgres + Redis
├── .eslintrc.js                    # Shared ESLint config
├── .prettierrc                     # Shared Prettier config
├── tsconfig.base.json              # Base TypeScript config
└── pnpm-workspace.yaml             # pnpm workspace config

Getting Started

Prerequisites

Node.js 20+
pnpm 8+
Docker Desktop

Installation

# Clone the repository
git clone https://github.com/yourusername/quelm.git
cd quelm

# Install all dependencies
pnpm install

Environment Variables

Create the following .env files:

Root .env — Docker Compose infrastructure:

POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=agent_platform

server/.env — Application config:

# App
PORT=8000

# Database
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/agent_platform

# Redis
REDIS_URL=redis://localhost:6379

# Client URL (deployed frontend, for CORS)
CLIENT_URL=http://localhost:3000

# Groq
GROQ_API_KEY=your_groq_api_key_here

# JWT
JWT_SECRET=your_jwt_secret
JWT_REFRESH_SECRET=your_jwt_refresh_secret

client/.env.local — Frontend config:

NEXT_PUBLIC_API_URL=http://localhost:8000

Copy the example files as a starting point:

cp .env.example .env
cp server/.env.example server/.env
cp client/.env.local.example client/.env.local

Get a free Groq API key at console.groq.com.

Running Locally

1. Start infrastructure:

docker compose up -d

2. Run database migrations:

cd server
pnpm prisma migrate dev

3. Start the server:

# From root
pnpm dev:server

4. Start the client:

# From root
pnpm dev:client

5. Run tests (no infrastructure required):

# Run all backend tests
pnpm test:server

# Watch mode
pnpm test:server:watch

# Run all frontend tests
pnpm test:client

# Watch mode
pnpm test:client:watch

Tests use mocked dependencies — no database, Redis, or API keys needed.

The API will be available at http://localhost:8000 and the frontend at http://localhost:3000.

API Reference

All endpoints return responses in the following shape:

{
  "success": true,
  "message": "Human readable message",
  "data": {}
}

Errors follow this shape:

{
  "success": false,
  "message": "Error description",
  "errorCode": "NOT_FOUND"
}

Authentication

Method	Endpoint	Description	Auth Required
`POST`	`/api/auth/register`	Create a new account with email/password	No
`POST`	`/api/auth/login`	Login with email/password	No
`POST`	`/api/auth/refresh`	Exchange refresh token for access token	No
`POST`	`/api/auth/logout`	Invalidate refresh token and clear cookie	No
`GET`	`/api/auth/me`	Get current user profile	Yes
`GET`	`/api/auth/google`	Redirect to Google OAuth consent	No
`GET`	`/api/auth/google/callback`	Handle Google OAuth callback	No
`GET`	`/api/auth/github`	Redirect to GitHub OAuth consent	No
`GET`	`/api/auth/github/callback`	Handle GitHub OAuth callback	No

Register:

POST /api/auth/register
Content-Type: application/json

{
  "email": "user@example.com",
  "password": "securepassword",
  "name": "Jane Doe"
}

Login:

POST /api/auth/login
Content-Type: application/json

{
  "email": "user@example.com",
  "password": "securepassword"
}

Refresh token:

POST /api/auth/refresh
Content-Type: application/json

{
  "refreshToken": "your_refresh_token"
}

The refresh token is also read from the refreshToken HTTP-only cookie if present.

Logout:

POST /api/auth/logout

No body required. Clears the refreshToken cookie and invalidates the stored token.

Get current user:

GET /api/auth/me
Authorization: Bearer <accessToken>

Google OAuth:

Initiate the OAuth flow by redirecting the user to /api/auth/google. After granting access, Google redirects back to /api/auth/google/callback with an authorization code. The server exchanges the code for tokens, finds or creates the user, and redirects to {CLIENT_URL}/auth/callback?token={accessToken}.

Requires GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET environment variables. Returns 503 if not configured.

GitHub OAuth:

Same flow as Google but uses /api/auth/github and /api/auth/github/callback. The server requests the user:email read:user scope to retrieve the user's primary verified email.

Requires GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET environment variables. Returns 503 if not configured.

Workflows

Method	Endpoint	Description
`GET`	`/api/workflow`	List all workflow definitions
`GET`	`/api/workflow/:id`	Get a single workflow definition
`POST`	`/api/workflow`	Create a new workflow definition
`POST`	`/api/workflow/:id/run`	Trigger a workflow run

Create a workflow:

POST /api/workflow
Content-Type: application/json

{
  "data": {
    "name": "Company Research Pipeline",
    "description": "Researches a company and analyses sentiment",
    "definition": {
      "nodes": [...],
      "edges": [...]
    }
  }
}

Trigger a run:

POST /api/workflow/:id/run
Content-Type: application/json

{
  "data": {
    "input": {
      "company_name": "Anthropic"
    }
  }
}

Runs

Method	Endpoint	Description
`GET`	`/api/runs`	List all workflow runs
`GET`	`/api/runs/:id`	Get a run with all its tasks
`GET`	`/api/runs/workflow/:workflowId`	Get all runs for a workflow
`GET`	`/api/runs/:id/stream`	SSE stream for live run updates

Agents

Method	Endpoint	Description
`GET`	`/api/agents`	List all registered agents
`GET`	`/api/agents/:id`	Get a single agent

Agent Types

Type	Description	Use Case
`LLM_AGENT`	Calls Groq with a configurable prompt template	Summarisation, classification, extraction, generation
`HTTP_AGENT`	Makes an HTTP request to any external API	Fetching data, sending webhooks, third-party integrations
`TRANSFORM_AGENT`	Transforms and reshapes JSON data	Filtering fields, mapping values, formatting
`EXTRACTION_AGENT`	Extracts structured data from unstructured sources	Web scraping, PDF parsing, raw text parsing
`NOTIFICATION_AGENT`	Sends notifications	Emails, Slack messages, webhooks
`STORAGE_AGENT`	Reads from or writes to external storage	Saving results, reading files, S3 operations

Workflow Definition Schema

A workflow definition is a JSON object with nodes and edges:

{
  "nodes": [
    {
      "id": "node_1",
      "type": "LLM_AGENT",
      "name": "Research Company",
      "critical": true,
      "config": {
        "promptTemplate": "Research {{company_name}} and summarise their recent news in 3 bullet points.",
        "model": "llama-3.3-70b-versatile",
        "maxTokens": 500
      }
    },
    {
      "id": "node_2",
      "type": "LLM_AGENT",
      "name": "Sentiment Analysis",
      "critical": true,
      "config": {
        "promptTemplate": "Analyse the sentiment of this text — return positive, negative, or neutral with a one sentence reason: {{text}}",
        "model": "llama-3.3-70b-versatile",
        "maxTokens": 100
      }
    }
  ],
  "edges": [
    {
      "id": "edge_1",
      "source": "node_1",
      "target": "node_2"
    }
  ]
}

Prompt templates support {{variable}} placeholders. The first node receives the run input as its variables. Subsequent nodes receive the output of their dependency tasks.

How It Works

Creating a Workflow

Open the Workflow Builder, drag agent nodes onto the canvas, connect them with edges, and configure each node's prompt or settings in the side panel. Save the workflow — it gets stored as a JSON definition in PostgreSQL.

Triggering a Run

Click Run on any saved workflow, fill in the input payload, and hit Execute. The API creates a WorkflowRun row, instantiates Task rows for every node, builds a dependency map from the edges, and dispatches the first tasks (those with no incoming edges) to the appropriate BullMQ queues. The run starts asynchronously and the API returns immediately with a runId.

Live Run Monitor

The frontend opens an SSE connection to /api/runs/:id/stream. As each task completes, the worker updates the database and the orchestrator emits an event on the RunEmitter. The SSE endpoint pushes that event to the browser — the canvas node changes colour (blue for running, green for completed, red for failed) without a single page refresh.

Retry and Failure Handling

Every task has a configurable maxAttempts (default 3). BullMQ retries failed jobs with exponential backoff. If a task marked critical: true exhausts all attempts, the entire workflow run is marked FAILED and all pending tasks are cancelled. Non-critical task failures are logged and skipped — the run continues with remaining tasks.

Stale Run Timeout

Runs that have been in RUNNING status for more than 10 minutes with no task activity (no task has reached RUNNING or COMPLETED) are automatically marked as FAILED and all pending tasks are cancelled. A background job runs every 5 minutes to detect and clean up stale runs. This covers edge cases where invalid configuration, type mismatches, or worker unavailability prevent any task from progressing.

Testing

The project includes both backend and frontend test suites powered by Vitest. Tests are designed to run locally and in CI without requiring external infrastructure such as databases, Redis instances, or third-party API keys.

Backend Test Structure

server/__tests__/
├── setup.ts                # Global mocks (Prisma, Redis, BullMQ, Groq, Winston)
├── helpers/
│   ├── prisma.ts           # Mock PrismaClient factory
│   └── app.ts              # Express app builder for integration tests
├── utils/                  # Error classes, template interpolation
├── services/               # Workflow, auth, run, agent, dashboard
├── orchestrator/           # Workflow engine, dependency resolution, event handling
└── api/                    # Integration tests via supertest

Frontend Test Structure

client/__tests__/
├── helpers.tsx             # Shared render utilities and mock factories
├── LoginPage.test.tsx
├── RegisterPage.test.tsx
├── RunPage.test.tsx
├── AgentsPage.test.tsx
└── StatCard.test.tsx

Coverage Areas

Backend

Layer	Tests	What's tested
Utils	11	Error classes, `{{placeholder}}` interpolation
Services	38	CRUD validation, ownership checks, auth flow, stats
Orchestrator	13	triggerRun, dependency graph, task dispatch, failure handling
API	37	Auth, workflows, agents, dashboard — via supertest

Frontend

Area	What's tested
Auth Pages	Login and registration flows, validation, redirects, and toasts
Runs Page	Loading, error, empty, and populated states, duration formatting
Agents Page	Agent listing, loading states, and empty-state rendering
Components	Dashboard components such as StatCard

Running Tests

# Run all tests
pnpm test

# Run backend tests
pnpm test:server

# Run frontend tests
pnpm test:client

# Backend watch mode
pnpm test:server:watch

# Frontend watch mode
pnpm test:client:watch

# Run a specific backend test file
pnpm --filter server vitest run __tests__/services/workflow.service.test.ts

# Run a specific frontend test file
pnpm --filter client vitest run __tests__/LoginPage.test.tsx

CI Integration

The test suite is automatically executed in GitHub Actions on every push and pull request to ensure that frontend and backend changes do not introduce regressions.

Mock Strategy

Backend

@prisma/client — globally mocked; PrismaClient returns configured mock methods
bullmq — Queue, Worker, and QueueEvents are mocked; no Redis connection required
ioredis — Redis constructor returns a no-op mock
groq-sdk — Groq returns a mock chat.completions.create
winston — createLogger returns a no-op logger
bcryptjs — hash and compare are stubbed to avoid real crypto operations

Frontend

next/navigation — mocked router and navigation hooks
next/link — rendered as a standard anchor element for assertions
sonner — toast notifications mocked to prevent portal rendering
@xyflow/react — React Flow components mocked for lightweight testing
@monaco-editor/react — Monaco editor replaced with a simple textarea
React Query — wrapped with a dedicated test QueryClientProvider
Shared mock factories provide reusable workflow, run, agent, and dashboard data

Services use dependency injection and isolated mocks, allowing tests to run quickly and deterministically without external services.

Deployment

Frontend — Vercel

Push your code to GitHub
Go to vercel.com and import the repository
Set the root directory to client
Add the environment variable NEXT_PUBLIC_API_URL pointing to your Render backend URL
Deploy — Vercel auto-deploys on every push to main

Backend — Render

Go to render.com and create a new Web Service
Connect your GitHub repository and set the root directory to server
Set the build command to pnpm install && pnpm prisma generate && pnpm build
Set the start command to pnpm start
Add a PostgreSQL database from the Render dashboard — it injects DATABASE_URL automatically
Add a Redis instance from the Render dashboard — copy the external URL into REDIS_URL
Add remaining environment variables — GROQ_API_KEY, PORT, NODE_ENV=production
Deploy

Both Vercel and Render have free tiers that cover this project with no credit card required.

Roadmap

User authentication (JWT + refresh tokens)
HTTP Agent implementation
Transform Agent implementation
Backend testing infrastructure (Vitest, 83 tests)
Node name editing in the workflow builder
Workflow versioning
Scheduled workflow triggers (cron)
Webhook triggers
Multi-tenancy support

Contributing

Contributions, issues, and feature requests are welcome. See CONTRIBUTING.md for setup instructions, coding standards, branching strategy, and a step-by-step guide to adding new agent types.

Fork the repository
Create a feature branch — git checkout -b feature/my-feature
Commit your changes — git commit -m 'feat: add my feature'
Push to the branch — git push origin feature/my-feature
Open a Pull Request against main

Name		Name	Last commit message	Last commit date
Latest commit History 86 Commits
.github		.github
client		client
server		server
.env.example		.env.example
.eslintignore		.eslintignore
.gitignore		.gitignore
.prettierignore		.prettierignore
.prettierrc		.prettierrc
CONTRIBUTING.md		CONTRIBUTING.md
INTERVIEW_PREP.md		INTERVIEW_PREP.md
README.md		README.md
docker-compose.yml		docker-compose.yml
eslint.config.mjs		eslint.config.mjs
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
preview.png		preview.png

Folders and files

Latest commit

History

Repository files navigation

Quelm

Table of Contents

Overview

Features

Architecture

Tech Stack

Project Structure

Getting Started

Prerequisites

Installation

Environment Variables

Running Locally

API Reference

Authentication

Workflows

Runs

Agents

Agent Types

Workflow Definition Schema

How It Works

Creating a Workflow

Triggering a Run

Live Run Monitor

Retry and Failure Handling

Stale Run Timeout

Testing

Backend Test Structure

Frontend Test Structure

Coverage Areas

Backend

Frontend

Running Tests

CI Integration

Mock Strategy

Backend

Frontend

Deployment

Frontend — Vercel

Backend — Render

Roadmap

Contributing

About

Topics

Resources

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages