Skip to content

API.md

Latest commit

 

History

History
1201 lines (883 loc) · 25.4 KB

API.md

File metadata and controls

1201 lines (883 loc) · 25.4 KB

LabelFleet API Reference

LabelFleet exposes a REST API that covers every admin operation. All endpoints return JSON. This document provides full request/response schemas for programmatic and agentic use.

Authentication

Most endpoints require a NextAuth session. To authenticate programmatically:

# 1. Get a session cookie via the NextAuth credentials endpoint
curl -c cookies.txt -X POST http://localhost:3000/api/auth/callback/credentials \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "email=admin@fleet.local&password=admin&csrfToken=<token>"

# 2. Use the cookie in subsequent requests
curl -b cookies.txt http://localhost:3000/api/annotators

Exceptions: /api/health and /api/version (no auth), /api/webhook (uses X-Webhook-Secret header).

All authenticated endpoints return 401 Unauthorized if the session is missing or invalid.

Pagination

All list endpoints support pagination via query parameters:

Param Type Default Description
limit integer 100 Results per page (1-500)
offset integer 0 Number of results to skip

Paginated responses are wrapped in an envelope:

{
  "data": [ ... ],
  "pagination": {
    "total": 150,
    "limit": 100,
    "offset": 0,
    "hasMore": true
  }
}

Annotators

Annotators represent individual labelers. Each annotator gets their own isolated Label Studio container.

List Annotators

GET /api/annotators

Response: 200 OK

[
  {
    "id": 1,
    "name": "Alice",
    "email": "alice@example.com",
    "roleId": 1,
    "status": "active",
    "containerId": "fleet-ls-1",
    "containerIp": "172.18.0.5",
    "containerPort": 8080,
    "subdomain": "alice",
    "databaseName": "ls_annotator_1",
    "apiToken": "abc123...",
    "lsUserId": 1,
    "notes": null,
    "lastActiveAt": "2026-04-06T12:00:00.000Z",
    "rejectionCount": 2,
    "reviewedCount": 50,
    "createdAt": "2026-04-01T00:00:00.000Z",
    "updatedAt": "2026-04-06T12:00:00.000Z",
    "role": { "id": 1, "name": "labeler", "color": "#3B82F6" }
  }
]

Create Annotator

POST /api/annotators

Request Body:

Field Type Required Description
name string yes Display name (1-255 chars)
email string yes Email address (unique)
roleId integer no Role ID to assign
notes string no Free-text notes (max 2000 chars)

Response: 201 Created

{
  "id": 2,
  "name": "Bob",
  "email": "bob@example.com",
  "roleId": 1,
  "status": "created",
  ...
}

Errors:

  • 400 — Validation failed (missing name, invalid email, etc.)
  • 409 — Email already exists

Get Annotator

GET /api/annotators/:id

Response: 200 OK — Annotator object with nested assignments array and role.

Errors: 404 — Annotator not found

Update Annotator

PUT /api/annotators/:id

Request Body: Same fields as create, all optional. At least one field required.

Response: 200 OK — Updated annotator object.

Delete Annotator

DELETE /api/annotators/:id

Deletes the annotator record and cleans up any associated containers (stops and removes from Docker/ECS).

Response: 200 OK

{ "success": true }

Start Container

POST /api/annotators/:id/start

Provisions and starts an isolated Label Studio container for the annotator. This:

  1. Creates a per-annotator PostgreSQL database
  2. Deploys a Label Studio container (Docker or ECS depending on ORCHESTRATOR_BACKEND)
  3. Configures Traefik routing for the annotator's subdomain
  4. Waits for the container to become healthy
  5. Triggers auto-assignment of the first workflow batch

Response: 200 OK

{ "success": true, "status": "starting" }

The container creation is asynchronous. Poll /api/annotators/:id/status to track progress.

Errors:

  • 400 — Annotator already has an active container
  • 404 — Annotator not found

Stop Container

POST /api/annotators/:id/stop

Stops the annotator's Label Studio container. In ECS mode, scales the service to 0 (preserving the target group and listener rule for quick restart). In Docker mode, stops the container.

Response: 200 OK

{ "success": true }

Get Container Status

GET /api/annotators/:id/status

Returns the current container lifecycle status.

Response: 200 OK

{
  "status": "active",
  "containerId": "fleet-ls-1",
  "containerIp": "172.18.0.5",
  "subdomain": "alice",
  "url": "http://alice.fleet.localhost"
}

Status values: created, starting, active, stopping, stopped, error, deleted

Report Token (internal)

POST /api/annotators/:id/token

Called by the Label Studio entrypoint script to report its API token back to the admin app. Authenticated via X-Webhook-Secret header (not NextAuth session).

Request Headers:

  • X-Webhook-Secret — The annotator's webhook secret

Request Body:

Field Type Required Description
token string yes Label Studio API token

Response: 200 OK

Workflows

Workflows define how data flows to annotators. They specify the S3 source path, batch size, gold task injection rate, and optional chaining/rejection routing.

List Workflows

GET /api/workflows

Response: 200 OK

[
  {
    "id": 1,
    "name": "Image Classification",
    "roleId": 1,
    "labelingConfigId": 1,
    "previousWorkflowId": null,
    "s3SourcePath": "images/classification/",
    "s3ExportPath": "exports/classification/",
    "imagesPerProject": 100,
    "maxProjects": null,
    "estimatedTimeMinutes": 30,
    "autoAssign": true,
    "goldStandardRate": 0.1,
    "shuffleImages": true,
    "isActive": true,
    "totalImagesAvailable": 5000,
    "totalImagesAssigned": 1200,
    "rejectionMode": "none",
    "rejectionArchiveBucket": null,
    "rejectionRouteWorkflowId": null,
    "rejectionReturnToOriginal": false,
    "rejectionFlagAnnotator": true,
    "maxReworkIterations": null,
    "createdAt": "2026-04-01T00:00:00.000Z",
    "role": { "id": 1, "name": "labeler", "color": "#3B82F6" },
    "labelingConfig": { "id": 1, "name": "Image Classification" }
  }
]

Create Workflow

POST /api/workflows

Request Body:

Field Type Required Default Description
name string yes — Workflow name (1-255 chars)
roleId integer no — Target role for auto-assignment
labelingConfigId integer no — Label Studio config template ID
previousWorkflowId integer no null Upstream workflow for chaining
s3SourcePath string yes — S3 prefix for source images
s3ExportPath string yes — S3 prefix for annotation exports
imagesPerProject integer no 100 Images per batch (1-10000)
maxProjects integer no null Max batches per annotator
estimatedTimeMinutes integer no null Estimated completion time
autoAssign boolean no true Auto-assign on annotator ready
goldStandardRate number no 0 Gold task injection rate (0-1)
shuffleImages boolean no true Randomize image order
rejectionMode enum no "none" "none", "archive", or "route_to_workflow"
rejectionArchiveBucket string no null S3 path for archived rejections
rejectionRouteWorkflowId integer no null Target workflow for rejected tasks (required when mode is "route_to_workflow")
rejectionReturnToOriginal boolean no false Route rejected work back to original annotator
rejectionFlagAnnotator boolean no true Increment annotator's rejection count
maxReworkIterations integer no null Max rework cycles before escalation

Response: 201 Created — Created workflow object.

Validation: If rejectionMode is "route_to_workflow", then rejectionRouteWorkflowId is required.

Get Workflow

GET /api/workflows/:id

Response: 200 OK — Workflow object with the full chain traced (from origin to terminal node).

Update Workflow

PUT /api/workflows/:id

Request Body: Same fields as create, all optional. Additionally accepts:

  • isActive (boolean) — Pause/resume the workflow

Delete Workflow

DELETE /api/workflows/:id

Response: 200 OK

{ "success": true }

Trigger Assignment

POST /api/workflows/:id/assign

Manually trigger an assignment for a specific annotator.

Request Body:

Field Type Required Description
annotatorId integer yes Target annotator ID

Response: 200 OK

{
  "success": true,
  "assignment": {
    "id": 5,
    "workflowId": 1,
    "annotatorId": 2,
    "lsProjectId": 12,
    "taskCount": 100,
    "goldTaskCount": 10,
    "status": "pending"
  }
}

Errors:

  • 400 — Annotator not active or not in the right role
  • 404 — Workflow not found
  • 409 — No images available for assignment

Get Workflow Graph

GET /api/workflows/graph

Returns the complete workflow DAG as nodes and edges, suitable for visualization.

Response: 200 OK

{
  "nodes": [
    {
      "id": 1,
      "name": "Image Classification",
      "type": "workflow",
      "roleId": 1,
      "roleName": "labeler",
      "roleColor": "#3B82F6",
      "isActive": true,
      "rejectionMode": "route_to_workflow",
      "stats": {
        "totalAssignments": 50,
        "completedAssignments": 45,
        "activeAnnotators": 5,
        "rejectionRate": 0.12
      }
    },
    {
      "id": "archive-1",
      "name": "Archive",
      "type": "terminal"
    }
  ],
  "edges": [
    {
      "source": 1,
      "target": 2,
      "type": "chain",
      "label": "accept"
    },
    {
      "source": 2,
      "target": 3,
      "type": "rejection",
      "label": "reject"
    }
  ]
}

Roles

Roles govern which annotators receive which workflows.

List Roles

GET /api/roles

Response: 200 OK

[
  {
    "id": 1,
    "name": "labeler",
    "description": "General image labeling",
    "color": "#3B82F6",
    "createdAt": "2026-04-01T00:00:00.000Z"
  }
]

Create Role

POST /api/roles

Request Body:

Field Type Required Description
name string yes Role name (1-100 chars, unique)
description string no Role description (max 500 chars)
color string no Hex color code (e.g., #3B82F6)

Response: 201 Created

Get Role

GET /api/roles/:id

Response: 200 OK

Update Role

PUT /api/roles/:id

Request Body: Same fields as create, all optional.

Delete Role

DELETE /api/roles/:id

Response: 200 OK

Labeling Configs (Templates)

Label Studio XML configuration templates. These define the annotation interface (e.g., classification, bounding boxes, NER).

List Configs

GET /api/labeling-configs

Response: 200 OK

[
  {
    "id": 1,
    "name": "Image Classification",
    "description": "Single-label image classification",
    "configXml": "<View>\n  <Image name=\"image\" value=\"$image\"/>\n  <Choices name=\"label\" toName=\"image\">\n    <Choice value=\"cat\"/>\n    <Choice value=\"dog\"/>\n  </Choices>\n</View>",
    "createdAt": "2026-04-01T00:00:00.000Z"
  }
]

Create Config

POST /api/labeling-configs

The XML is validated to ensure it is well-formed, has a <View> root, and contains at least one Label Studio object tag (Image, Text, Audio, Video, etc.).

Request Body:

Field Type Required Description
name string yes Config name (1-100 chars)
description string no Description (max 500 chars)
configXml string yes Label Studio XML config (max 50000 chars)

Response: 201 Created

Errors:

  • 400 — Invalid XML (malformed, missing <View>, no object tags)

Get Config

GET /api/labeling-configs/:id

Response: 200 OK — Config object with usage counts:

{
  "id": 1,
  "name": "Image Classification",
  "configXml": "...",
  "workflowCount": 3,
  "goldTaskCount": 10,
  "isLocked": true
}

isLocked is true when gold tasks reference this config (XML edits are blocked to preserve scoring consistency).

Update Config

PUT /api/labeling-configs/:id

Request Body: Same fields as create, all optional.

Errors:

  • 400 — Invalid XML
  • 409 — Cannot update configXml when gold tasks reference this config

Delete Config

DELETE /api/labeling-configs/:id

Errors:

  • 409 — Cannot delete: config is referenced by workflows or gold tasks

Gold Tasks

Gold standard tasks are known-good labeled data injected into annotation batches for quality scoring.

List Gold Tasks

GET /api/gold-tasks

Query Parameters:

Param Type Description
labelingConfigId integer Filter by labeling config
category string Filter by category

Response: 200 OK

[
  {
    "id": 1,
    "name": "Gold - Cat",
    "s3ImageKey": "gold/cat_001.jpg",
    "expectedLabels": [{ "from_name": "label", "to_name": "image", "type": "choices", "value": { "choices": ["cat"] } }],
    "labelingConfigId": 1,
    "category": "animals",
    "difficulty": "easy",
    "createdAt": "2026-04-01T00:00:00.000Z"
  }
]

Create Gold Task

POST /api/gold-tasks

Request Body:

Field Type Required Description
name string no Display name (max 255 chars)
s3ImageKey string yes S3 key of the gold image
expectedLabels array yes Expected Label Studio annotation result (array of objects, min 1)
labelingConfigId integer yes Associated labeling config
category string no Category for filtering (max 100 chars)
difficulty enum no "easy", "medium", or "hard"

Response: 201 Created

Get Gold Task

GET /api/gold-tasks/:id

Response: 200 OK

Update Gold Task

PUT /api/gold-tasks/:id

Request Body: Same fields as create, all optional. At least one field required.

Errors:

  • 400 — Validation failed or labeling config not found

Delete Gold Task

DELETE /api/gold-tasks/:id

Response: 200 OK

Assignments

Track in-flight annotation work across all annotators and workflows.

List Assignments

GET /api/assignments

Query Parameters:

Param Type Description
workflowId integer Filter by workflow
annotatorId integer Filter by annotator
status enum Filter by status: pending, in_progress, completed, failed, cancelled
limit integer Results per page (default 100)
offset integer Offset for pagination

Response: 200 OK — Paginated list of assignments with nested annotator and workflow.

Get Assignment

GET /api/assignments/:id

Response: 200 OK — Assignment with nested annotator (with role), workflow (with labeling config), tasks, and goldResults.

Cancel Assignment

DELETE /api/assignments/:id

Cancels a pending or in-progress assignment. Completed, failed, or already cancelled assignments cannot be cancelled.

Response: 200 OK

{ "success": true }

Errors:

  • 409 — Cannot cancel assignment with current status

Rejections

Audit trail for rejected annotations in review workflows.

List Rejections

GET /api/rejections

Query Parameters:

Param Type Description
workflowId integer Filter by reviewer's workflow
annotatorId integer Filter by original annotator
actionTaken enum Filter by action: none, archive, route_to_workflow
limit integer Results per page (default 100)
offset integer Offset for pagination

Response: 200 OK — Paginated list of rejections with originalAnnotator and reviewerAnnotator.

Get Rejection

GET /api/rejections/:id

Response: 200 OK — Rejection with full relations: originalAnnotator, reviewerAnnotator, reviewerAssignment (with workflow), originalAssignment, routedToAssignment.

Events

Unified audit log for container lifecycle and annotation events.

List Events

GET /api/events

Query Parameters:

Param Type Description
type enum Filter by type: "container" or "annotation"
annotatorId integer Filter by annotator
limit integer Results per page (default 100)
offset integer Offset for pagination

When no type is specified, both container and annotation events are merged and sorted by createdAt descending.

Response: 200 OK — Paginated list of events, each with a type field.

System Settings

Key-value store for system configuration. Settings are persisted as JSON values.

List Settings

GET /api/settings

Response: 200 OK

[
  {
    "id": 1,
    "key": "s3.bucket",
    "value": "my-annotation-bucket",
    "description": null,
    "updatedAt": "2026-04-06T12:00:00.000Z"
  }
]

Get Setting

GET /api/settings/:key

Response: 200 OK — Single setting object.

Errors: 404 — Setting not found.

Upsert Setting

PUT /api/settings/:key

Creates the setting if it doesn't exist, or updates it if it does.

Request Body:

Field Type Required Description
value any yes Setting value (any valid JSON)
description string no Description (max 500 chars)

Response: 200 OK — Created or updated setting object.

Bulk Operations

Bulk Create Annotators

POST /api/annotators/bulk

Request Body:

{
  "annotators": [
    { "name": "Alice", "email": "alice@example.com", "roleId": 1 },
    { "name": "Bob", "email": "bob@example.com", "roleId": 1 }
  ]
}

Max 100 annotators per request.

Response: 201 Created

{ "created": 2, "annotators": [ ... ] }

Errors: 409 — Duplicate email address in batch.

Bulk Delete Annotators

DELETE /api/annotators/bulk

Request Body:

{ "ids": [1, 2, 3] }

Max 100 IDs per request. Sets status to "deleted".

Response: 200 OK

{ "deleted": 3 }

Bulk Create Workflows

POST /api/workflows/bulk

Request Body:

{
  "workflows": [
    { "name": "Stage 1", "s3SourcePath": "data/", "s3ExportPath": "out/", ... },
    { "name": "Stage 2", "s3SourcePath": "data/", "s3ExportPath": "out/", "previousWorkflowId": 1, ... }
  ]
}

Max 50 workflows per request. Validates all referenced labelingConfigId and previousWorkflowId values exist.

Response: 201 Created

{ "created": 2, "workflows": [ ... ] }

Version

Get Version

GET /api/version

No authentication required.

Response: 200 OK

{
  "version": "0.1.0",
  "apiVersion": "v1",
  "features": [
    "container-orchestration",
    "workflow-chaining",
    "gold-standard-quality",
    "rejection-routing",
    "bulk-operations",
    "assignment-tracking",
    "event-audit-log",
    "system-settings"
  ],
  "node": "v20.x.x"
}

Monitoring

Real-time metrics for annotation throughput and system health.

System Overview

GET /api/monitoring

Response: 200 OK

{
  "activeAnnotators": 5,
  "todayAnnotations": 342,
  "inProgressAssignments": 8,
  "annotationsPerHour": [
    { "hour": "2026-04-06T10:00:00.000Z", "count": 45 },
    { "hour": "2026-04-06T11:00:00.000Z", "count": 52 }
  ],
  "annotatorSummaries": [
    {
      "id": 1,
      "name": "Alice",
      "todayCount": 87,
      "avgTimeMs": 12500,
      "completionRate": 0.95
    }
  ]
}

Per-Annotator Detail

GET /api/monitoring/annotator/:id

Response: 200 OK

{
  "annotator": { "id": 1, "name": "Alice", "status": "active" },
  "containerUptime": "4h 32m",
  "todayAnnotations": 87,
  "annotationsPerHour": 21.5,
  "timeDistribution": [
    { "bucket": "0-10s", "count": 15 },
    { "bucket": "10-30s", "count": 42 },
    { "bucket": "30-60s", "count": 25 },
    { "bucket": "60s+", "count": 5 }
  ],
  "assignmentStats": {
    "total": 12,
    "completed": 10,
    "inProgress": 2
  },
  "recentEvents": [
    {
      "eventType": "annotation_created",
      "lsTaskId": 456,
      "timeSpentMs": 15000,
      "createdAt": "2026-04-06T12:30:00.000Z"
    }
  ]
}

Quality

Aggregated quality metrics including gold standard accuracy and rejection rates.

Get Quality Metrics

GET /api/quality

Response: 200 OK

{
  "overallAccuracy": 0.87,
  "accuracyThreshold": 0.7,
  "belowThresholdCount": 1,
  "reworkRate": 0.05,
  "annotatorScores": [
    {
      "annotatorId": 1,
      "annotatorName": "Alice",
      "accuracy": 0.92,
      "evaluationCount": 15,
      "rejectionCount": 2,
      "reviewedCount": 50,
      "rejectionRate": 0.04
    }
  ],
  "recentEvaluations": [
    {
      "annotatorId": 1,
      "annotatorName": "Alice",
      "goldTaskName": "Gold - Cat",
      "score": 1.0,
      "evaluatedAt": "2026-04-06T12:00:00.000Z"
    }
  ],
  "recentRejections": [
    {
      "originalAnnotatorId": 2,
      "originalAnnotatorName": "Bob",
      "reviewerAnnotatorName": "Carol",
      "imageKey": "images/001.jpg",
      "reason": "Bounding box too loose",
      "actionTaken": "route_to_workflow",
      "createdAt": "2026-04-06T11:30:00.000Z"
    }
  ]
}

Reconciliation

Force-sync the admin database with Label Studio's actual state.

Run Reconciliation

POST /api/reconcile

Compares in-progress assignments against each annotator's Label Studio instance. Updates completion counts and triggers exports/auto-assignment if projects are complete. Uses a PostgreSQL advisory lock to prevent concurrent runs.

Response: 200 OK

{
  "reconciled": 3,
  "completed": 1,
  "errors": []
}

Webhook

Receives annotation events from Label Studio instances.

Ingest Event

POST /api/webhook

Request Headers:

  • X-Annotator-Id — The annotator's numeric ID
  • X-Webhook-Secret — The annotator's webhook secret

Request Body: Label Studio webhook payload:

Field Type Description
action enum "ANNOTATION_CREATED", "ANNOTATION_UPDATED", or "ANNOTATION_DELETED"
annotation object { id, created_at, updated_at, lead_time, result, task, completed_by }
project object { id, title }
task object { id, data }

Response: 200 OK

{ "ok": true }

Processing is idempotent: duplicate webhooks for the same annotation+event are safely ignored via a unique constraint.

What happens on ingestion:

  1. Records the annotation event with timing data
  2. Marks the assignment task as completed
  3. If all tasks in the assignment are done: exports annotations, triggers auto-assignment of next batch, and triggers downstream workflow chaining
  4. If the annotation contains a review_decision result: records accept/reject and triggers rejection routing

Health

Health Check

GET /api/health

No authentication required.

Response: 200 OK

{
  "status": "ok",
  "database": "connected"
}

Returns 503 if the database is unreachable.

Common Error Responses

All errors follow this shape:

{
  "error": "Human-readable error message",
  "details": [ ... ]  // Optional: Zod validation issues
}
Status Meaning
400 Bad request (validation failure, invalid params)
401 Unauthorized (missing or invalid session)
404 Resource not found
409 Conflict (duplicate email, resource in use, no images available)
500 Internal server error

Agentic Usage Example

Here is an example of how an automated agent could set up a complete annotation pipeline:

BASE=http://localhost:3000

# 1. Create a role
curl -b cookies.txt -X POST $BASE/api/roles \
  -H "Content-Type: application/json" \
  -d '{"name": "labeler", "description": "Image labelers", "color": "#3B82F6"}'

# 2. Create a labeling config
curl -b cookies.txt -X POST $BASE/api/labeling-configs \
  -H "Content-Type: application/json" \
  -d '{"name": "Object Detection", "configXml": "<View><Image name=\"image\" value=\"$image\"/><RectangleLabels name=\"label\" toName=\"image\"><Label value=\"car\"/><Label value=\"person\"/></RectangleLabels></View>"}'

# 3. Create a workflow
curl -b cookies.txt -X POST $BASE/api/workflows \
  -H "Content-Type: application/json" \
  -d '{"name": "Car Detection", "roleId": 1, "labelingConfigId": 1, "s3SourcePath": "images/cars/", "s3ExportPath": "exports/cars/", "imagesPerProject": 50, "goldStandardRate": 0.1}'

# 4. Create an annotator
curl -b cookies.txt -X POST $BASE/api/annotators \
  -H "Content-Type: application/json" \
  -d '{"name": "Alice", "email": "alice@example.com", "roleId": 1}'

# 5. Start the container (async — poll status)
curl -b cookies.txt -X POST $BASE/api/annotators/1/start

# 6. Monitor progress
curl -b cookies.txt $BASE/api/monitoring

# 7. Check quality
curl -b cookies.txt $BASE/api/quality