Skip to content

INTEGRATION.md

Latest commit

 

History

History
631 lines (480 loc) · 25.3 KB

INTEGRATION.md

File metadata and controls

631 lines (480 loc) · 25.3 KB
Raw

Backend Integration Guide

This app is a static frontend. It does not own the data system. A compatible backend only needs to provide field metadata, accept query payloads, and stream results as JSON Lines events.

The current checked-in endpoint remains the default example/testing backend. New deployments can point the same frontend at another backend without changing field logic in the frontend.

The easiest user-facing path is the in-app API Settings panel. It saves a compatible API URL in the browser, can test get_fields, can run a compatibility report, and can build a launch link that includes only the API override.

For private deployments, use docs/AUTH.md before exposing real library-system data. The recommended path is a same-origin authenticated API route where existing credentials are handled by the backend, reverse proxy, or identity gateway, not by the static frontend.

For non-browser workflows, the CLI in docs/CLI.md talks to the same backend contract and uses the same payload/result/export modules as the app. It can list fields, run compatibility checks, inspect status, cancel runs, export saved results, list templates, run query JSON configs, apply local post filters, and export JSONL, JSON, CSV, or XLSX. That gives deployments a scriptable path for scheduled reports without adding a separate API.

For AI agents, tool-calling systems, and workflow automation, start with docs/AI_API.md. It keeps this same JSONL backend contract but shows how to expose it as MCP tools, strict model function tools, or OpenAPI-importable actions.

Recommended Contract

For new integrations, keep the backend adapter small and boring:

  1. Accept JSON POST requests at one URL.
  2. Implement get_fields so the frontend can discover fields.
  3. Implement run so the frontend can execute the selected fields and filters.
  4. Stream result events as JSONL with one meta event, zero or more row events, and a final done event.
  5. Add query IDs, history, cancellation, saved results, and templates only when your deployment needs those panels.

The machine-readable schema is docs/schemas/query-api.schema.json. It uses JSON Schema draft 2020-12 and intentionally allows extra properties so a backend can include deployment-specific metadata without forking the frontend. An OpenAPI 3.1 description for documentation and AI/tool importers lives at docs/schemas/query-api.openapi.json.

Fastest Setup Path

  1. Implement POST JSON handling for get_fields.
  2. Implement POST JSON handling for run.
  3. Return application/x-ndjson result streams with ordered columns and row value arrays.
  4. Serve this repository as a static site.
  5. Open API Settings, enter https://your.example.org/query-api or a same-origin route such as /api/query, save it, and reload fields.
  6. Run Compatibility Check in API Settings.
  7. Add status, cancel, get_results, and template actions only when those panels need to work against your backend.

The frontend does not require backend-specific code for field definitions. Your get_fields response defines available fields, filter operators, field warnings, dynamic/buildable field inputs, and any fields that return multi-value cells.

Minimum Working Backend

A minimal backend only needs these two request/response pairs.

The repository also includes a runnable Node example at examples/minimal-backend/:

npm run example:backend

Use this URL in the API Settings panel:

http://127.0.0.1:8787/query-api

Request:

{ "action": "get_fields" }

Response:

{
  "fields": [
    {
      "name": "Title",
      "type": "string",
      "category": "Bibliographic",
      "filters": ["contains", "equals"]
    }
  ]
}

Request:

{
  "action": "run",
  "result_format": "jsonl",
  "display_fields": ["Title"],
  "filters": [
    { "field": "Title", "operator": "=", "value": "*history*" }
  ]
}

Response:

{"type":"meta","version":1,"format":"jsonl","query_id":"query-1","columns":["Title"]}
{"type":"row","values":["Example title"]}
{"type":"done","rows":1}

That is enough for field loading, query building, result display, post filters, copy, and Excel export. The rest of this guide documents optional features and deployment choices.

Integration Goals

  • Use JSON request bodies for actions.
  • Stream query results as JSON Lines events.
  • Keep field definitions backend-driven. The frontend must not ship a built-in field catalog.
  • Make optional features clear: core querying can work without template persistence, but the template panel needs template actions.

Choosing an API URL

The default endpoint is still defined in src/core/backendApi.js as DEFAULT_API_URL.

From the app UI:

  1. Open API Settings.
  2. Enter an absolute API URL or a same-origin path.
  3. Use Test Connection to verify that get_fields returns metadata.
  4. Use Run Compatibility Check to verify browser access, JSONL streaming, event order, multi-value arrays, and optional workflow actions.
  5. Save the endpoint.
  6. Reload fields so the app refreshes metadata from that backend.

For a static deployment, the frontend can use another compatible backend by providing one of these browser settings:

?api_url=https://your.example.org/query-api
?query_api_url=https://your.example.org/query-api

The URL can be absolute or same-origin relative:

https://reports.example.org/index.html?api_url=/api/query

When a valid URL is supplied, the app stores it in localStorage under:

query-project.api-url

You can also pre-seed that localStorage value in your deployment shell. The default endpoint remains the fallback if no override is supplied. Do not put secrets or API keys in the settings screen or URL; use normal authenticated sessions, reverse proxies, or server-side API credentials instead.

Your backend must allow browser requests from the deployed frontend origin through normal CORS rules, unless it is served from the same origin.

The app's browser fetches use credentials: "same-origin" for Query API requests. That is intentional: private deployments should normally expose the API through the same origin as the app, such as /api/query, and keep session cookies scoped to that origin. Cross-origin credentialed requests require an intentional frontend and CORS policy change.

Compatibility Report

The API Settings panel includes a compatibility report that sends safe diagnostic requests to the configured endpoint.

Core checks:

Check What it verifies
CORS / browser access The browser can complete a POST request to the API URL
Field metadata get_fields returns a non-empty field list
JSONL stream run returns protocol version 1 JSONL with a meta event
Event order The stream uses meta first, row values as arrays, and done last
Multi-value arrays At least one sampled row contains an array-valued cell, when present in the result sample

Optional checks:

Check Action
Query status status
Cancellation cancel
Saved results get_results
Templates list_templates

The run diagnostic sends a small request with compatibility_check: true, limit: 5, and max_rows: 5. Backends should honor those hints when possible so compatibility tests stay fast and avoid large result streams. If a backend does not have sample multi-value data, the multi-value check can report a warning rather than a failure.

Transport

All app actions are sent as JSON POST requests:

POST /query-api
Content-Type: application/json

Every request has an action property. Successful non-result JSON endpoints should return Content-Type: application/json. Query result endpoints must return Content-Type: application/x-ndjson; charset=utf-8.

Errors should use normal HTTP status codes. A useful error body is:

{
  "error": "Human-readable error message"
}

Rate limits should return HTTP 429. The frontend understands either retry_after_seconds or retry_after:

{
  "error": "Too many requests from this IP.",
  "retry_after_seconds": 30
}

Schema Reference

Use docs/schemas/query-api.schema.json as the shared contract between the frontend and a backend adapter. The schema includes definitions for:

  • field metadata returned by get_fields
  • run payloads and backend filters
  • streaming JSONL result events
  • optional query status/progress payloads
  • optional cancellation, saved-result loading, template payloads, and error responses

The schema is permissive by design outside the core stream protocol. Required fields are limited to the contract the frontend needs, while additionalProperties allows backend-specific IDs, timing data, diagnostics, auth context, or deployment metadata. JSONL examples in this guide are validated in the architecture test suite against the jsonlEvent schema definition.

For AI integrations, use docs/AI_API.md with docs/schemas/query-api.openapi.json. The AI-facing examples keep endpoint selection and credentials outside model-controlled arguments while still using the same get_fields, run, status, cancel, get_results, and list_templates actions.

Required Core Actions

These two actions are the minimum needed for the main query builder.

get_fields

Request:

{
  "action": "get_fields"
}

Response:

{
  "fields": [
    {
      "name": "Title",
      "label": "Title",
      "type": "string",
      "category": "Bibliographic",
      "desc": "Main title displayed to users",
      "filters": ["contains", "starts", "equals", "does_not_equal"],
      "aliases": ["title"],
      "parts": 1
    }
  ]
}

The response may also be a bare array of field definitions.

Supported field metadata:

Property Purpose
name Required canonical field name sent back in query payloads
label Optional display label; defaults to name
desc or description Field help text used by search/tooltips
category String or string array for grouping in the picker
type string, date, number, money, boolean, or another backend-defined text type
filters or operators Backend-supported filter operators for the field
values Optional scalar or descriptive object values for dropdown/pill controls
aliases Old or alternate names that should resolve to this field
allowValueList Allows comma/newline values for equality filters; the frontend sends bulk lists as value arrays
multiSelect Allows selecting multiple values from values
groupValues Lets the UI group selector values with dashed labels
parts Optional backend-specific segment hint for fields that are assembled from multiple backend values
numberFormat or numericFormat integer, decimal, year, or currency style hints
sensitive UI hint that the field may expose protected/staff-only information
requiresAuth or authRequired UI hint that the field needs an authenticated backend session
authorized Set to false when the backend wants to show the field but prevent use until sign-in or stronger access
requiredScopes Optional backend-defined scopes, roles, or groups needed for the field
authMessage or accessMessage Optional user-facing explanation shown when a field is unavailable
access Optional string or object for deployment-specific access metadata
builder Defines dynamic/buildable fields
filterGroup Optional backend-defined group metadata for related fields that can use independent all/any matching

Selector values may be scalars or objects. Descriptive objects use RawValue for the exact filter value, Name for the visible label, Description for delayed hover help, Group for grouping, SearchText for matching, and optional structured Metadata. Lowercase aliases are also accepted. Libraries use this generic contract, but it is available to every field with enumerated values.

Sensitive And Auth-Gated Fields

The frontend can display badges and disable controls for protected fields, but it is not the security boundary. Backends must enforce authorization for every get_fields, run, status, cancel, get_results, template, and export action.

Recommended behavior:

  • If the user is not signed in, either omit sensitive fields from get_fields or return them with authorized: false.
  • If a signed-in user can use a sensitive field, return sensitive: true, requiresAuth: true, and any useful requiredScopes.
  • If a user sends an unauthorized field manually in display_fields or filters, reject the request server-side with HTTP 401 or 403.
  • If saved results can contain sensitive columns, protect get_results and history/status metadata too; knowing a query_id must not be enough to retrieve protected data.

Example:

{
  "name": "Checkout User Name",
  "type": "string",
  "category": "User",
  "filters": ["equals"],
  "sensitive": true,
  "requiresAuth": true,
  "authorized": false,
  "requiredScopes": ["reports:sensitive"],
  "authMessage": "Sign in with an authorized staff account to use checkout user fields."
}

The included Sirsi backend helper marks checkout-user fields as sensitive and denies them unless the backend request context includes reports:sensitive or an administrator intentionally enables the sensitive-report bypass. The static frontend only reads the metadata and shows the right badges/messages; it does not decide who is allowed.

Buildable Fields

Use buildable field metadata when users need to create dynamic output fields from inputs. The frontend renders the inputs generically and sends the generated field name back to the backend. It does not need to know what backend tool extracts the value.

{
  "name": "Custom Field",
  "type": "string",
  "category": "Dynamic",
  "desc": "Create a custom output field",
  "filters": ["contains", "equals"],
  "builder": {
    "outputFieldIdTemplate": "Custom {code}${subfield}",
    "displayLabelTemplate": "Custom {code}${subfield}",
    "matchPattern": "^Custom\\s+[A-Z0-9]+(?:\\$[A-Za-z0-9])?$",
    "inputs": [
      {
        "id": "code",
        "label": "Code",
        "type": "text",
        "pattern": "^[A-Z0-9]+$",
        "placeholder": "LOCAL",
        "error_msg": "Enter a valid code"
      },
      {
        "id": "subfield",
        "label": "Subfield",
        "type": "text",
        "pattern": "^[A-Za-z0-9]$",
        "placeholder": "a",
        "optional": true
      }
    ]
  }
}

Related buildable fields can declare a generic filter group. The client renders the label, help text, and match control from this metadata; it does not inspect field names or know what kind of backend data the fields represent.

{
  "filterGroup": {
    "id": "local_metadata",
    "label": "Local metadata conditions",
    "description": "Choose whether every related condition or at least one must match.",
    "defaultLogic": "all",
    "minConditions": 2
  }
}

When a query uses fields in that group, the client sends:

{
  "filter_group_logic": {
    "local_metadata": "any"
  }
}

Backends own the field-family semantics and must validate supported group IDs and apply the requested logic. Frontends treat every group uniformly.

If a user enters LOCAL and leaves the optional subfield blank, the frontend creates and displays Custom LOCAL. If the user enters subfield a, it creates Custom LOCAL$a.

The backend should recognize the generated field name in display_fields and return that column in results.

run

Request:

{
  "action": "run",
  "name": "Optional query name",
  "result_format": "jsonl",
  "display_fields": ["Title", "Custom LOCAL"],
  "filters": [
    {
      "field": "Title",
      "operator": "=",
      "value": "*history*"
    },
    {
      "field": "Record Date",
      "operator": ">=",
      "value": "20240101"
    }
  ]
}

Supported backend operators currently sent by the frontend:

Operator Meaning
= equals, contains, starts-with, or date never depending on value
!= does not equal or does not contain
> greater than or after
< less than or before
>= greater than/equal or on/after
<= less than/equal or on/before

For text contains/starts filters, the frontend sends wildcard values such as *needle* or needle*. For date fields, the frontend sends normalized YYYYMMDD values or NEVER. For fields with allowValueList, an equals filter may send value as an array. Backends should treat those arrays as a bulk input list for that field instead of applying the smaller limit normally used for repeated ad hoc filter values.

The frontend requests streaming JSONL results with result_format: "jsonl". Result responses must use Content-Type: application/x-ndjson; charset=utf-8 and emit the event format below. JSON object error responses are still valid before result streaming starts.

Post filters are intentionally not sent to the backend. They operate only on already-loaded result rows and are cleared between query runs.

Streaming JSONL Result Format

Backends stream newline-delimited JSON events. This keeps the old streaming performance characteristics while avoiding delimiter escaping problems and giving a natural representation for multi-value cells. The first non-empty line must be a meta event with version: 1 and format: "jsonl" so clients can reject incompatible streams before processing rows.

{"type":"meta","version":1,"format":"jsonl","query_id":"query-1","columns":["Title","Public Note","Custom LOCAL"]}
{"type":"row","values":["Example title",["First note","Second note"],["Local value one","Local value two"]]}
{"type":"progress","rows":1000,"message":"Rows streamed"}
{"type":"done","rows":1000,"elapsed_ms":12345}

Event types:

  • meta: required first event; includes version: 1, format: "jsonl", optional query_id, and ordered columns.
  • row: one result row; includes canonical values, an array matching the meta.columns order.
  • progress: optional progress metadata for long-running work.
  • warning: optional recoverable issue metadata.
  • error: failure after streaming has started.
  • done: required final success event with the total row count.

Supported cell shapes:

{
  "Single value": "text",
  "Multiple values as array": ["one", "two"],
  "Multiple values as object": { "values": ["one", "two"] },
  "Wrapped single value": { "value": "text" }
}

Multi-value arrays are normalized by the frontend and work with the virtual table, copy cell, post filters, split-column views, and Excel export.

Long-Running Query Actions

The core query builder can show results from the direct run response. Query history, cancellation, and loading saved results work best when the backend also supports IDs and status actions.

Query IDs

If run returns an X-Query-Id header, the frontend tracks that query in history and can poll/cancel/load it later.

status

Request:

{
  "action": "status"
}

Response:

{
  "queries": {
    "q-123": {
      "name": "Optional query name",
      "status": "complete",
      "start_time": "2026-06-05T02:30:00Z",
      "end_time": "2026-06-05T02:30:07Z",
      "row_count": 25,
      "progress": {
        "schema_version": 1,
        "stage": "complete",
        "label": "Complete",
        "current": 25,
        "unit": "rows",
        "counters": {
          "emitted_rows": 25
        },
        "updated_at": "2026-06-05T02:30:07Z"
      },
      "request": {
        "name": "Optional query name",
        "display_fields": ["Title"],
        "filters": []
      },
      "ui_config": {
        "DesiredColumnOrder": ["Title"],
        "Filters": []
      }
    }
  }
}

Recognized statuses include running, complete, failed, and canceled.

Running queries may include a backend-neutral progress object. The frontend treats this as a generic status contract and does not assume anything about the backend's query engine, database, or enrichment tools.

Recommended progress fields:

Field Meaning
schema_version Optional contract version. Use 1 for the current shape.
stage Machine-readable stage such as queued, base_query, loading_dynamic_fields, filtering_results, streaming_results, complete, failed, or canceled. Custom stage names are allowed.
label Short user-facing status such as Finding matching rows or Loading requested field values.
detail Optional extra user-facing detail. Keep it backend-neutral when possible.
current Optional numeric progress count.
total Optional numeric total for determinate work.
percent Optional numeric percent from 0 to 100. The frontend can derive this from current and total when both are present.
unit Optional unit label such as rows, records, or items.
counters Optional object of extra neutral counters, for example candidate_rows, lookup_keys, lookup_records, matched_rows, emitted_rows, or skipped_records.
updated_at Optional timestamp for the latest progress update.

This contract is intentionally generic. A backend can use any internal tools it wants, but the status response should describe progress in terms users can understand without exposing implementation details.

cancel

Request:

{
  "action": "cancel",
  "query_id": "q-123"
}

Any HTTP success response is treated as a successful cancellation.

get_results

Request:

{
  "action": "get_results",
  "query_id": "q-123"
}

Response uses the same streaming JSONL event format as run.

Template Actions

Templates are optional. If your deployment does not support them, the main query builder can still run. The template panel expects these actions when enabled:

Action Request fields Expected response
list_templates none { "templates": [...], "categories": [...] }
create_template name, description, categories, ui_config, pinned, pin_order saved template object or { "template": {...} }
update_template template_id plus the create fields saved template object or { "template": {...} }
delete_template template_id, optional name success object
reorder_pinned_templates template_ids optional { "templates": [...] }
create_template_category name, description saved category object
update_template_category category_id, name, description saved category object
delete_template_category category_id success object

Template objects may use either camelCase or snake_case for common keys:

{
  "id": "template-1",
  "name": "Monthly report",
  "description": "Reusable report",
  "categories": [
    { "id": "cat-1", "name": "Reports", "description": "" }
  ],
  "ui_config": {
    "DesiredColumnOrder": ["Title"],
    "Filters": []
  },
  "pinned": true,
  "pin_order": 0,
  "created_at": "2026-06-05T02:30:00Z",
  "updated_at": "2026-06-05T02:31:00Z"
}

Integration Options

Native JSONL Backend

Best option for new systems. Implement the actions above and stream JSONL result events. This gives clean multi-value cells, progress/error events, and immediate row delivery.

Adapter or Proxy Backend

Use a small backend adapter that accepts the app's JSON payload, translates it to your internal system, then streams JSONL events. This is usually the cleanest path for systems that already have their own query language.

Adapter sketches are available in examples/adapters/, including Node/Express, Python/FastAPI, legacy delimited output, and SQL/reporting API shapes.

Static Deployment with Same-Origin API

Serve the app and API under the same origin, then launch the app with:

https://reports.example.org/index.html?api_url=/api/query

Static Deployment with Cross-Origin API

Launch the app with:

https://reports.example.org/index.html?api_url=https://api.example.org/query

Your API must allow the frontend origin with CORS. If users authenticate through cookies, configure credentials and same-site policy on your backend/proxy.

Compatibility Guardrails

  • The frontend field catalog comes from get_fields.
  • Dynamic/buildable fields come from backend builder metadata.
  • The schema contract lives in docs/schemas/query-api.schema.json.
  • The architecture test tests/architecture/noHardcodedFrontendFields.mjs fails if production frontend code adds backend field-name literals or a local field catalog.
  • src/core/queryResultParser.js hydrates rows only from JSONL-derived payloads.
  • npm test runs the integration guard, parser tests, payload tests, and browser smoke tests.

Minimal Backend Checklist

  1. Implement POST JSON handling.
  2. Support get_fields.
  3. Support run.
  4. Return result rows as JSONL events with meta, row, and done.
  5. Add X-Query-Id, status, cancel, and get_results if you want history and cancellation.
  6. Add template actions only if you want saved templates.
  7. Return useful { "error": "..." } bodies for failures.
  8. Configure CORS or serve the API from the same origin as the static app.
  9. Honor compatibility_check, limit, and max_rows hints when possible.