- Base URL:
/api/v1 - Authentication: Bearer JWT token in
Authorizationheader (or API key inX-API-Keyheader). - Pagination: All list endpoints accept
?page=1&page_size=50(default page_size=20, max 100). - Sorting:
?sort_by=created_at&sort_order=desc. - Filtering:
?field=value. Complex filters use a JSON-encodedfilterparameter. - Responses: Standard envelope:
{"data": ..., "meta": {"page": ..., "page_size": ..., "total": ...}}. - Errors:
{"error": {"code": "VALIDATION_ERROR", "message": "...", "details": [...]}}. - Links: HATEOAS-inspired
linksobject included in single-resource responses. - API versioning: Via URL prefix (
/api/v1/). Version is bumped only on backward-incompatible changes. Additionally, aAccept-Versionheader can be used for content negotiation within a major version (e.g.,Accept-Version: 1.2). - Rate limits: 1000 req/min per team (default), configurable. Returned via
X-RateLimit-*headers. - Long-running operations: Endpoints that trigger background work (evaluation runs, exports) return
202 Acceptedwith aLocationheader pointing to the resource and aworkflow_idfield referencing the Temporal workflow execution. Clients poll the resource endpoint or subscribe via WebSocket for completion. - Idempotency: Mutating endpoints accept an
Idempotency-Keyheader. If the same key is received within 24 hours, the previous response is returned (cached). This is critical for CI/CD pipeline safety.
Create a new user account. Returns a JWT and refresh token.
Authenticate with email/password. Returns JWT (access) and refresh token.
Exchange a refresh token for a new JWT.
Invalidate the current refresh token (server-side blacklist).
Return the current user's profile and team memberships.
Update the current user's profile (name, avatar, preferences).
Change the current user's password. Requires current password confirmation.
List API keys for the authenticated team.
Create a new API key. The raw key is returned in the response (shown once).
Revoke an API key.
List teams the current user belongs to.
Create a new team. The creator becomes the owner.
Get team details.
Update team settings (name, billing info).
Soft-delete a team (owner only).
List team members with roles.
Change a member's role.
Remove a member from the team.
List projects within the authenticated team. Supports ?search= query.
Create a new project. Body: {name, description, tags?}.
Get project details, including summary stats (total runs, last run date, pass rate).
Update project metadata.
Soft-delete a project.
List prompts in a project.
Create a new prompt. Body: {name, template, variables, commit_message?}.
Get prompt details and current version.
Update the prompt (creates a new version).
Soft-delete a prompt.
List all versions of a prompt.
Get a specific prompt version with the template snapshot.
List datasets.
Create a dataset. Supports:
Content-Type: application/jsonwith rows inline.Content-Type: multipart/form-datawith a CSV/JSONL file upload.
Body/Form: {name, description, file?, rows?}.
Get dataset metadata.
Update dataset metadata (creates new version only if rows change).
Soft-delete a dataset.
List dataset versions.
Paginated row retrieval. Supports ?offset=0&limit=100.
List configured providers for a project (returns provider names, never API keys).
Add/configure a provider. Body: {provider_name, config, encrypted_api_key?}.
Update provider configuration. Re-encrypts the API key if provided.
Remove a provider configuration.
Send a test prompt to verify connectivity. Returns the response, latency, and token count without persisting anything.
Return a list of all supported providers (no auth required). Used for UI dropdowns.
List available models for a provider (from the Model Catalog). Returns model name, capabilities, pricing, rate limits, context window.
Query models by capability. Returns all models across providers that match the required capabilities.
List models available to a project (intersection of configured providers + their active models).
List experiments. Supports filters: ?status=active&tags=regression.
Create a new experiment. Body: {name, description?, hypothesis?, baseline_run_id?, tags?}.
Get experiment details, including linked run count, status, and baseline comparison summary.
Update experiment metadata, conclusion, or status.
Soft-delete an experiment (cancels associated pending runs).
List evaluation runs within an experiment. Supports standard pagination and filtering.
Trigger a new evaluation run within this experiment. Same body as evaluation run creation, with experiment_id implied.
Return side-by-side comparison of all runs in the experiment. Shows metric scores per run per model, with delta from baseline.
List evaluation profiles. Includes built-in system profiles and project-custom profiles.
Create a custom profile. Body: {name, description, extends? (parent profile name), configuration: {...}}.
Get profile details and resolved configuration.
Update a custom profile. System profiles are read-only.
Delete a custom profile. System profiles cannot be deleted.
Preview the resolved configuration for a profile (useful for understanding inherited settings before running).
List evaluation runs. Supports filters: ?status=completed&metrics=hallucination,toxicity&experiment_id=uuid.
Create and trigger a new evaluation run. Body:
{
"name": "GPT-4 vs Claude Hallucination",
"experiment_id": "uuid (optional, for grouping)",
"profile_id": "uuid (optional, overrides inline config)",
"prompt_version_id": "uuid",
"dataset_version_id": "uuid",
"providers": [
{"provider": "openai", "model": "gpt-4o"},
{"provider": "anthropic", "model": "claude-3-opus"}
],
"metrics": ["hallucination", "faithfulness", "answer_relevancy"],
"thresholds": {"hallucination": {"lte": 0.3}},
"webhook_url": "https://..."
}Returns 202 Accepted with {run_id, workflow_id, status: "pending"}. The run executes asynchronously via Temporal.
Get evaluation run status and summary (aggregate scores per metric per provider).
Cancel a running evaluation (sends cancellation signal to Temporal workflow) or soft-delete a completed one.
List evaluation runs. Supports filters: ?status=completed&metrics=hallucination,toxicity.
Create and trigger a new evaluation run. Body:
{
"name": "GPT-4 vs Claude Hallucination",
"prompt_version_id": "uuid",
"dataset_version_id": "uuid",
"providers": ["openai/gpt-4", "anthropic/claude-3-opus"],
"metrics": ["hallucination", "faithfulness", "answer_relevancy"],
"thresholds": {"hallucination": {"lte": 0.3}},
"webhook_url": "https://..."
}Returns the evaluation run ID. The run executes asynchronously.
Get evaluation run status and summary (aggregate scores per metric per provider).
Cancel a running evaluation or soft-delete a completed one.
List individual evaluation tasks (one per prompt row per provider). Supports pagination.
Get a single task with full input, output, latency, token count, cost, and metric results.
List metric results for a single task.
Subscribe to real-time progress updates for an evaluation run. Messages:
{
"type": "task_completed",
"task_id": "uuid",
"provider": "openai/gpt-4",
"row_index": 42,
"progress": {"completed": 42, "total": 100}
}{
"type": "run_completed",
"run_id": "uuid",
"summary": { "overall_pass": true }
}{
"type": "error",
"task_id": "uuid",
"message": "Rate limit exceeded"
}Return the list of all available metrics from the MetricRegistry. Each entry includes: name, display_name, description, category, evaluator_type, required_inputs, version, and metadata.
Get details for a specific metric, including version history if applicable.
List threshold configurations for the project.
Bulk update threshold configurations. Body: [{"metric_name": "hallucination", "comparison": "lte", "value": 0.2, "severity": "critical"}, ...].
List Temporal workflow executions. Supports filters: ?status=running&type=evaluation. For operators and debugging.
Get workflow execution details, including status, start/completion time, and error details. Links to Temporal Web UI if configured.
Request cancellation of a running workflow.
Query domain events. Supports filters: ?event_type=EvaluationCompleted&entity_type=evaluation_run&entity_id=uuid&from=ISO&to=ISO. Paginated. For auditing and debugging.
Get a single event with full payload and processing metadata (which subscribers processed it, when).
Get all events sharing a correlation ID. Enables distributed tracing across publisher → subscriber chains.
List campaigns.
Create a campaign. Body: {name, target_provider, target_model, strategy, adversarial_dataset_id, schedule?}.
Get campaign details and summary statistics.
Update campaign configuration.
Trigger a campaign execution immediately.
List findings from this campaign, optionally filtered by severity.
Triage a finding (mark as reviewed, dismiss, assign).
List saved reports.
Create a new report configuration. Body: {name, type, configuration: {...}}.
Get report data (executes the report query and returns results).
Update report configuration.
Delete a report configuration.
Export report data. Query param ?format=csv|json. Returns a file download.
Return aggregate statistics for the project dashboard:
- Total runs (last 30 days)
- Pass/fail counts
- Average scores per metric
- Total cost (last 30 days)
- Average latency per provider
Time-series data for charting.
Side-by-side metric comparison of two or more evaluation runs.
List configured webhooks.
Create a webhook. Body: {url, events: [...], secret?}.
Update webhook configuration.
Delete a webhook.
Send a test event to verify the endpoint.
List audit log entries. Supports filters: ?actor_id=&action=&resource_type=&from=&to=. Paginated.
Simple health check (returns {"status": "ok"}).
Readiness check (verifies DB connection, Redis connection, Temporal Server health, and at least one Temporal worker connected).
Return the current application version and commit SHA.
Return the status of all system dependencies (PostgreSQL, Redis, Temporal Server, Temporal Workers) with latency. For operational dashboards.