Skip to content

Repository files navigation

Agent Harness — Containerized Java Agent Execution Engine

Agents are managed state machines. Not chat interfaces.

A high-performance, Spring Boot 4 / Java 25 workflow execution engine built for enterprise-grade, siloed multi-tenant deployments. Every tenant runs inside a fully isolated containerized environment with a dedicated JVM process and a dedicated PostgreSQL database instance. There is no shared fate, no cross-tenant process bleed, and no shared data tier.


Table of Contents

  1. Infrastructure Overview
  2. Core Architectural Highlights
  3. Department Isolation and Thread Resilience
  4. Chaos Test Scenarios — Logistics and Transportation
  5. Extensibility and Downstream Implementations

1. Infrastructure Overview

Siloed Tenancy Model

Each tenant deployment is a fully isolated unit:

┌─────────────────────────────────────────────────────────────────────┐
│  Tenant: freight-customer-acme                                      │
│                                                                     │
│  ┌─────────────────────────────┐   ┌────────────────────────────┐  │
│  │  Spring Boot JVM Process    │   │  PostgreSQL Instance       │  │
│  │  agent-harness:8138         │   │  (dedicated, no sharing)   │  │
│  │                             │   │                            │  │
│  │  WorkflowRuntimeExecutor    │──▶│  model_registry            │  │
│  │  WorkflowQueueConsumer      │   │  llm_connections           │  │
│  │  BlackboardCompactionSvc    │   │  workflow_instances        │  │
│  │  WorkflowStepAdapterRegistry│   │  workflow_execution_queue  │  │
│  └─────────────────────────────┘   │  agent_traces              │  │
│                                    │  blackboard_compaction_...  │  │
│                                    └────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  Tenant: freight-customer-zyx   (separate container, separate DB)   │
│  ... identical isolation boundary ...                               │
└─────────────────────────────────────────────────────────────────────┘

Zero shared-fate: No connection pool sharing, no row-level tenant discriminator columns, no cross-contamination at the process or data tier between tenants.

Internal compartmentalization within a tenant boundary operates at the Department level using RequestContext — a thread-bound scoping primitive that carries deptId, userId, sessionId, and queueId through every execution context.

Quick-start (single tenant container)

# Start infrastructure
podman-compose up -d

# Start agent harness (Windows)
cd services-java/agent-harness
./mvnw.cmd spring-boot:run

# Verify health
curl http://localhost:8138/actuator/health

2. Core Architectural Highlights

2.1 Execution Model — State Machine, Not Chat Loop

WorkflowRuntimeExecutor processes a WorkflowRuntimeDefinition as an ordered list of WorkflowRuntimeStep records loaded from the tenant database. Each step is dispatched to exactly one of four execution paths:

WorkflowRuntimeExecutor.execute(def, inputContext, inputBlackboard)
         │
         ▼
   ┌─────────────────────────────────────┐
   │  for each WorkflowRuntimeStep       │
   │                                     │
   │  isGateStep?   ──▶  PauseForReview  │
   │  isSetStep?    ──▶  applySet()      │
   │  isForeachStep?──▶  applyForeach()  │
   │  isAdapterStep?──▶  applyAdapter()  │  ← WorkflowStepAdapterRegistry lookup
   │  else          ──▶  applyLlm()      │  ← UserLlmSelector → BrainLayer routing
   └─────────────────────────────────────┘

Step results accumulate in two maps — context and blackboard — that flow through the entire run. Conditional branching throws BranchToStepException; HITL pause throws WorkflowPauseException. Both are caught cleanly without killing the thread.

2.2 Database-Backed Brain Registry

The harness does not hard-code model configuration. Every LLM routing decision is resolved at runtime from two tables inside the tenant's private PostgreSQL instance:

model_registry

Column Type Purpose
id uuid Primary key
dept_id uuid Department FK — enforces dept boundary
connection_id uuid FK → llm_connections
brain_set_name varchar(128) Named brain profile (e.g. "freight-default", "customs-audit")
brain_layer varchar(128) Role in execution (e.g. EXECUTION, REASONING, PERCEPTION)
context_window_tokens integer Ceiling fed to BlackboardCompactionService
max_output_tokens integer Hard output cap
temperature / top_p numeric(3,2) Per-department inference tuning
input_token_cost_per_million numeric(10,4) Cost tracking for quota ledger
is_default boolean Fallback when no layer match exists
fallback_model_id uuid Self-referential FK — hot-swap on failure

llm_connections

Column Purpose
dept_id Department scoping — NOT NULL, FK-enforced
endpoint_url Can point to local Ollama (e.g. NVIDIA RTX 3090), cloud API, or internal proxy
api_key Scoped to this department's operational boundary
timeout_ms Per-connection network timeout
provider anthropic, openai, ollama, custom

UserLlmSelector.getClient(BrainLayer) resolves the correct LlmClientInterface via O(1) map lookup from the session's pre-loaded llmProfile. No string scanning, no dynamic queries during step execution.

2.3 WorkflowStepAdapter Registry

All non-LLM execution is delegated to typed adapters registered in WorkflowStepAdapterRegistry. Adapter type is resolved from step.kind() using normalized string matching:

Adapter Step Kind Purpose
McpWorkflowStepAdapter "mcp" Dispatch to external MCP server tools; merges skill tool catalog with dept MCP catalog
RagIngestShareStepAdapter "rag_ingest_share" Index shared enterprise file shares into pgvector; route through RagIngestionProcessor
ScriptEngineStepAdapter "script" Execute sandboxed algorithm via SandboxedScriptEngineService with per-step timeoutMs
MessageQueueStepAdapter "message_queue" Broadcast async events through validated MessageQueueRegistry
StorageBucketStepAdapter "storage_bucket" Stage / retrieve files from object storage
HumanApprovalStepAdapter "review" Serialize state → throw WorkflowPauseException → park thread
HttpWorkflowStepAdapter "http" Outbound HTTP calls to external systems
DbWorkflowStepAdapter "db" Direct tenant-scoped database operations

Adding a new adapter requires implementing WorkflowStepAdapter, registering it as a Spring @Component, and declaring a new step kind in the workflow YAML definition. No changes to WorkflowRuntimeExecutor.

2.4 Blackboard Compaction

Mid-run context inflation is intercepted by BlackboardCompactionService before token ceilings defined in model_registry.context_window_tokens are breached:

BlackboardCompactionService.compactIfNeeded(scope, blackboard, config, extraMustKeepKeys)
    │
    ├── totalBytes ≤ config.maxBytes?  →  return as-is
    │
    ├── for each key in blackboard:
    │       mustKeep? → carry forward verbatim
    │       hotKey?   → carry forward if ≤ maxKeyBytes, else truncate
    │       else      → move to summarySource
    │
    ├── summarize(summarySource) → compacted["_summary"]
    │
    └── assertMustKeepIntegrity() → throws if required keys were dropped

Compaction events emit Micrometer counters and summaries under blackboard.compaction.* tags with scope and outcome dimensions. Connect to Prometheus/Grafana for per-workflow visibility.

2.5 Adaptive Reflection Loop

LLM steps can opt in to a critique-and-revise loop via ReasoningOrchestrator:

# Workflow step YAML
- id: tariff_classification
  kind: llm
  instructions: "Classify HS code for {{commodity_description}}"
  reflection:
    enabled: true
    maxRevisions: 2

ReflectionRuntimeMetadata is persisted on each AgentTrace and WorkflowStepState record, including revisionsAttempted, satisfactory, maxRevisionsReached, and critiqueSummary. Full auditability with zero config overhead if reflection is disabled.

2.6 End-to-End Flow

flowchart TD
    A[Inbound HTTP / gRPC Request] --> B[UserIsolationFilter / UserApiTokenInterceptor]
    B --> C[RequestContext.doAs\ndeptId · userId · sessionId · queueId]
    C --> D[WorkflowHarness]
    D --> E[WorkflowRuntimeExecutor]

    E --> F{Step Kind?}
    F -->|adapter| G[WorkflowStepAdapterRegistry]
    F -->|llm| H[UserLlmSelector]
    F -->|gate| I[WorkflowPauseException]
    F -->|set / foreach| J[Context Mutation]

    G --> G1[McpWorkflowStepAdapter]
    G --> G2[ScriptEngineStepAdapter]
    G --> G3[RagIngestShareStepAdapter]
    G --> G4[StorageBucketStepAdapter]
    G --> G5[MessageQueueStepAdapter]

    H --> K[model_registry lookup\nby dept + brain_layer]
    K --> L[llm_connections endpoint\nOllama / Cloud / Proxy]

    I --> M[markQueuePaused\nWorkflowExecutionQueue]
    M --> N[Thread Released to Pool]
    N --> O[WorkflowLifecycleController\nResurrection Webhook]
    O --> P[WorkflowQueueConsumer\npollResumeQueue @Scheduled]
    P --> C

    E --> Q[AgentTraceService\nWorkflowStepStateService\nAuditService]

    classDef core fill:#1a1d23,stroke:#4a9eff,color:#e0e0e0,stroke-width:1px;
    classDef infra fill:#1a1d23,stroke:#6b7280,color:#9ca3af,stroke-width:1px;
    class A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q core;
    class G1,G2,G3,G4,G5 infra;
Loading

3. Department Isolation and Thread Resilience

3.1 Database Compartmentalization Blueprint

Within a single tenant's PostgreSQL instance, every table that participates in workflow execution carries a dept_id uuid NOT NULL column with a FK to departments(id). Flyway migrations enforce this progressively:

Migration What It Enforces
V20 department_rate_limits rename, dept_id NOT NULL, FK on workflow_instances
V22 Phase-1 schema scope alignment across all execution tables
V25 agent_trace.dept_id NOT NULL enforcement + FK cascade
V27 blackboard_compaction_policy.scope_id UUID alignment

Every query that touches a tenant-scoped collection filters on dept_id. Cross-department data leakage is structurally impossible within a single deployment.

3.2 Thread-Bound RequestContext

RequestContext is a ThreadLocal-backed context carrier. The custom task decorator propagates it safely across thread-pool boundaries so asynchronous worker loops never suffer context amnesia:

// WorkflowQueueConsumer — resuming a parked workflow
RequestContext.doAs(payload.deptId(), sessionId, payload.userId(), () -> {
    RequestContext.withUserApiToken(payload.securityToken(), () -> {
        RequestContext.withSessionData(sessionData, () -> {
            RequestContext.withQueueId(queueId, () -> {
                workflowHarness.resume(payload.runId(), payload.reviewData());
            });
        });
    });
});

Every lambda boundary restores and clears context correctly. There is no risk of a Dept A context leaking into a Dept B execution slot regardless of thread pool reuse.

3.3 Human-In-The-Loop (HITL) — Thread-Safe Parking and Resurrection

When a workflow reaches a HumanApprovalStepAdapter (step kind "review"), the entire execution thread is released back to the pool without blocking:

sequenceDiagram
    participant Q as WorkflowQueueConsumer
    participant E as WorkflowRuntimeExecutor
    participant A as HumanApprovalStepAdapter
    participant DB as workflow_execution_queue
    participant W as WorkflowLifecycleController

    Q->>E: execute(def, context, blackboard, startFromStepId)
    E->>A: adapter.execute(step, context)
    A->>A: context.put("reviewData", params + pausedStepId)
    A-->>E: throw WorkflowPauseException("PAUSED_FOR_REVIEW")
    E->>DB: markQueuePaused(queueId, stepId, context, blackboard)
    E-->>Q: return WorkflowExecutionResult(paused=true, "PAUSED_FOR_REVIEW")
    Q->>Q: inFlightResumePermits.release() → thread returned to pool

    Note over DB: State serialized. Thread is free. No blocking.

    W->>DB: POST /lifecycle/{runId}/resume → updateStatus("RESUME_QUEUED")
    Q->>DB: pollResumeQueue @Scheduled(1000ms)
    DB-->>Q: WorkflowExecutionQueue items (RESUME_QUEUED)
    Q->>Q: orderForDepartmentFairness() → round-robin by deptId
    Q->>Q: inFlightResumePermits.tryAcquire() [semaphore: maxInFlight=8]
    Q->>E: execute(def, context, blackboard, startFromStepId=nextStepId)
Loading

Token quota enforcement gates every resume attempt before context is re-bootstrapped:

Quota Window Behavior
minute Retry with minuteQuotaBackoff (default 30s), status → RESUME_QUOTA_BACKOFF
day Retry with dayQuotaBackoff (default 600s)
billing_month Hard reject, status → RESUME_REJECTED

3.4 Department Fairness in the Queue Consumer

When multiple departments have concurrent resumable workflows, orderForDepartmentFairness() buckets items by deptId and interleaves them round-robin before dispatching. No department monopolizes the in-flight semaphore:

Input:  [D1-wf1, D1-wf2, D2-wf1, D1-wf3, D3-wf1]
Output: [D1-wf1, D2-wf1, D3-wf1, D1-wf2, D1-wf3]

4. Chaos Test Scenarios — Logistics and Transportation

These scenarios are built to exercise the harness under adversarial conditions representative of enterprise freight forwarding deployments.

Scenario 1: Automated Customs Manifest Audit Gate

Domain: International freight / customs compliance
Goal: Validate that a multi-step manifest audit pipeline runs correctly under concurrent cross-department load, that tariff lookups via pgvector survive high-cardinality vector space, and that the HITL gate triggers correctly when a classification confidence threshold is breached.

Workflow topology:

workflow:
  id: customs-manifest-audit
  brain_set_name: customs-audit

steps:
  - id: fetch_manifest
    kind: storage_bucket
    params:
      bucketId: "{{manifestBucketId}}"
      objectKey: "{{shipmentId}}/manifest.pdf"

  - id: ingest_tariff_schedule
    kind: rag_ingest_share
    params:
      shareId: "{{tariffScheduleShareId}}"
      namespace: "hs_tariff_2024"

  - id: classify_commodities
    kind: llm
    instructions: >
      Using the indexed tariff schedule, classify each line item in
      {{fetch_manifest.content}} with its 6-digit HS code and applicable duty rate.
    reflection:
      enabled: true
      maxRevisions: 2
    outputs:
      classifications:
        type: array

  - id: audit_gate
    kind: review
    params:
      reason: low_confidence_classification
      threshold: 0.85

  - id: broadcast_audit_result
    kind: message_queue
    params:
      queueId: "{{customsAuditQueueId}}"
      payload: "{{classify_commodities}}"

What to inject:

  • Run 12 concurrent manifest workflows across 3 departments simultaneously.
  • Kill the tariffScheduleShareId file share mid-ingest on department 2.
  • Verify RagIngestShareStepAdapter surfaces the error, WorkflowRuntimeExecutor catches the RuntimeException, evaluates pauseOnErrors config, and parks the run at ingest_tariff_schedule.
  • Resume department 2 via WorkflowLifecycleController webhook after file share is restored.
  • Assert department 1 and department 3 pipelines completed without any cross-department blackboard contamination. Check agent_traces.dept_id for every completed step.

Assertions:

-- No cross-department trace leakage
SELECT dept_id, COUNT(*) FROM agent_traces
WHERE session_id IN (/* run session IDs */)
GROUP BY dept_id
HAVING COUNT(DISTINCT dept_id) > 1; -- must return 0 rows

-- All classifications persisted with correct dept scope
SELECT COUNT(*) FROM workflow_step_states wss
JOIN workflow_instances wi ON wi.id = wss.instance_id
WHERE wi.dept_id != /* expected dept UUID */; -- must return 0

Scenario 2: Multi-Leg Intermodal Routing Disruption

Domain: Intermodal transportation — rail/ocean/trucking demurrage
Goal: Verify that ScriptEngineStepAdapter correctly sandboxes a compute-heavy demurrage penalty calculation, that BlackboardCompactionService triggers and survives when accumulated routing context inflates past token ceiling, and that a mid-run connection pool drop exercises HITL parking and state re-hydration without data loss.

Workflow topology:

workflow:
  id: intermodal-routing-disruption-audit
  brain_set_name: logistics-ops

steps:
  - id: ingest_routing_events
    kind: llm
    instructions: >
      Parse the following EDI 315 ocean status messages and extract
      container ID, vessel, POD, and ETA for each leg: {{edi_payload}}
    outputs:
      routing_legs: { type: array }

  - id: calculate_demurrage
    kind: script
    params:
      scriptId: "{{demurragePenaltyScriptUUID}}"
      inputKey: routing_legs
      timeoutMs: 3000          # sandbox kills script at 3 s — no JVM escape

  - id: llm_disruption_analysis
    kind: llm
    instructions: >
      Given the demurrage penalties in {{calculate_demurrage.output}},
      identify the highest-risk legs and recommend re-routing options.
    reflection:
      enabled: true
      maxRevisions: 1

  - id: ops_review_gate
    kind: review
    params:
      reason: high_demurrage_exposure
      exposureThreshold: 50000

  - id: notify_freight_ops
    kind: message_queue
    params:
      queueId: "{{freightOpsQueueId}}"

What to inject:

  1. Sandboxed script abuse: Supply a demurrage script that contains an infinite loop. Assert SandboxedScriptEngineService terminates execution at timeoutMs=3000 and surfaces a typed error that triggers branchOnErrors routing to a fallback step.

  2. Blackboard inflation: Feed 40+ routing legs with dense ETA/delay history so llm_disruption_analysis pushes the blackboard past context_window_tokens before the LLM call. Assert BlackboardCompactionService fires (blackboard.compaction.events{outcome="compacted"} counter increments), must_keep keys survive intact, and the LLM step completes successfully with the compacted context.

  3. Connection pool drop during HITL: After ops_review_gate parks the workflow, drop the LLM connection defined in llm_connections for this department (simulate network partition to local Ollama). Resume the workflow via webhook. Assert WorkflowQueueConsumer re-bootstraps UserSessionData, the UserLlmSelector falls back to model_registry.fallback_model_id, and the notify_freight_ops step completes without re-executing the already-completed steps.

Assertions:

-- Compaction fired on the disruption analysis step
SELECT * FROM blackboard_compaction_policies
WHERE scope_id = /* dept UUID */
  AND last_triggered_at > NOW() - INTERVAL '10 minutes';

-- HITL parking and resurrection recorded correctly
SELECT status, retry_count, resumed_at FROM workflow_execution_queue
WHERE run_id = /* run UUID */
ORDER BY created_at;
-- Expected: PAUSED → RESUME_QUEUED → RESUMED

-- Demurrage script step shows FAILED status with correct error message
SELECT step_status, error_message FROM workflow_step_states
WHERE step_id = 'calculate_demurrage'
  AND error_message LIKE '%timeout%';

5. Extensibility and Downstream Implementations

5.1 Writing a Custom WorkflowStepAdapter

Implement the interface and register as a Spring component. No changes to the executor:

@Component
public class ShipmentTrackingStepAdapter implements WorkflowStepAdapter {

    private final ShipmentTrackingClient client;

    public ShipmentTrackingStepAdapter(ShipmentTrackingClient client) {
        this.client = client;
    }

    @Override
    public String getType() {
        return "shipment_track";   // step kind in YAML
    }

    @Override
    public Map<String, Object> execute(WorkflowRuntimeStep step, Map<String, Object> context) {
        String trackingNumber = Objects.requireNonNull(
            (String) context.get("trackingNumber"), "trackingNumber required");
        ShipmentStatus status = client.track(trackingNumber);
        return Map.of(
            "carrier",    status.carrier(),
            "location",   status.lastKnownLocation(),
            "eta",        status.estimatedDelivery(),
            "exceptions", status.activeExceptions()
        );
    }
}

The adapter is immediately available to any workflow YAML using kind: shipment_track.

5.2 Extending the MCP Server Layer

Each department can configure its own MCP tool catalog in llm_connections. The McpServiceCatalog.getToolSpecsForCurrentDept() call resolves tool specifications for the active department from RequestContext.deptId() — so adding freight-specific MCP servers (track-and-trace APIs, customs databases, port authority feeds) requires only a new llm_connections row and a corresponding MCP server process. No harness code changes.

# Workflow step using a department-specific MCP tool
- id: query_port_authority
  kind: mcp
  params:
    tool: port_authority.get_vessel_eta
    args:
      vessel_imo: "{{vesselImo}}"
      port_code: "{{destinationPort}}"

5.3 TUI Dashboard — REST and gRPC Execution Trace Layer

The harness exposes execution state through two queryable surfaces suited for a TUI dashboard implementation:

REST endpoints (via WorkflowLifecycleController)

Endpoint Purpose
GET /workflow/instances/{runId} Full run state with step-level status
GET /workflow/instances/{runId}/steps Step-by-step WorkflowStepState records with input/output JSON
POST /workflow/instances/{runId}/resume Inject HITL review decision, enqueue for resurrection
GET /agent/traces?sessionId=... Per-session AgentTrace records with timing, tool calls, reflection metadata

gRPC streaming (when agent.grpc.enabled=true)

Live execution events stream over gRPC for low-latency TUI rendering. Connect a TUI client (Bubble Tea, Textual, or custom) to the gRPC stream and render:

  • Active workflow step name and status
  • Blackboard key count and estimated byte size (compaction pressure gauge)
  • BrainLayer currently executing and which llm_connections endpoint it's hitting
  • Tool call log with durationMs per call
  • HITL queue depth by department

Minimal TUI polling loop (Go / Bubble Tea example)

func (m model) pollWorkflowStatus() tea.Cmd {
    return func() tea.Msg {
        resp, _ := http.Get(fmt.Sprintf(
            "http://localhost:8138/workflow/instances/%s/steps", m.runId))
        var steps []WorkflowStepState
        json.NewDecoder(resp.Body).Decode(&steps)
        return stepsMsg{steps: steps}
    }
}

5.4 Adding a New Brain Set for a Department

No code deployment required. Insert rows into the tenant's private PostgreSQL instance:

-- Register a local Ollama endpoint for the customs audit department
INSERT INTO llm_connections (id, dept_id, provider, endpoint_url, api_key, timeout_ms)
VALUES (
    uuid_generate_v4(),
    'dept-uuid-customs',
    'ollama',
    'http://10.0.1.42:11434',     -- NVIDIA RTX 3090 host
    'ollama-local',
    45000
);

-- Bind a model to the EXECUTION brain layer for this department
INSERT INTO model_registry (
    id, connection_id, dept_id, brain_set_name, brain_layer,
    is_default, context_window_tokens, max_output_tokens,
    input_token_cost_per_million, output_token_cost_per_million,
    temperature, is_active
) VALUES (
    uuid_generate_v4(),
    '/* connection id above */',
    'dept-uuid-customs',
    'customs-audit',
    'EXECUTION',
    false,
    32768,
    4096,
    0.0000,    -- local Ollama — no token cost
    0.0000,
    0.70,
    true
);

The next session bootstrapped for dept-uuid-customs with brain_set_name = 'customs-audit' will route EXECUTION layer calls to the local Ollama instance — without a redeploy.


Build and Test Reference

# Fast compile check (agent-harness)
cd services-java/agent-harness
./mvnw.cmd -q -DskipTests compile

# Unit tests only
./mvnw.cmd test

# Full integration test suite (Failsafe — *IT, *IntegrationTest)
./mvnw.cmd verify

# Single test class
./mvnw.cmd "-Dtest=WorkflowE2EIT" test

# Single method
./mvnw.cmd "-Dtest=WorkflowHarnessTest#resumeRejectsInvalidToken" test

H2 test profile: Integration tests use H2 in PostgreSQL compatibility mode (MODE=PostgreSQL, ddl-auto=create-drop). Flyway applies all migrations against H2. No local PostgreSQL required for CI.

gRPC in tests: Disabled by default via src/test/resources/application-test.yml. Do not flip the runtime default to fix test failures.

About

AI Services Platform

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages