Agents are managed state machines. Not chat interfaces.
A high-performance, Spring Boot 4 / Java 25 workflow execution engine built for enterprise-grade, siloed multi-tenant deployments. Every tenant runs inside a fully isolated containerized environment with a dedicated JVM process and a dedicated PostgreSQL database instance. There is no shared fate, no cross-tenant process bleed, and no shared data tier.
- Infrastructure Overview
- Core Architectural Highlights
- Department Isolation and Thread Resilience
- Chaos Test Scenarios — Logistics and Transportation
- Extensibility and Downstream Implementations
Each tenant deployment is a fully isolated unit:
┌─────────────────────────────────────────────────────────────────────┐
│ Tenant: freight-customer-acme │
│ │
│ ┌─────────────────────────────┐ ┌────────────────────────────┐ │
│ │ Spring Boot JVM Process │ │ PostgreSQL Instance │ │
│ │ agent-harness:8138 │ │ (dedicated, no sharing) │ │
│ │ │ │ │ │
│ │ WorkflowRuntimeExecutor │──▶│ model_registry │ │
│ │ WorkflowQueueConsumer │ │ llm_connections │ │
│ │ BlackboardCompactionSvc │ │ workflow_instances │ │
│ │ WorkflowStepAdapterRegistry│ │ workflow_execution_queue │ │
│ └─────────────────────────────┘ │ agent_traces │ │
│ │ blackboard_compaction_... │ │
│ └────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ Tenant: freight-customer-zyx (separate container, separate DB) │
│ ... identical isolation boundary ... │
└─────────────────────────────────────────────────────────────────────┘
Zero shared-fate: No connection pool sharing, no row-level tenant discriminator columns, no cross-contamination at the process or data tier between tenants.
Internal compartmentalization within a tenant boundary operates at the Department level using RequestContext — a thread-bound scoping primitive that carries deptId, userId, sessionId, and queueId through every execution context.
# Start infrastructure
podman-compose up -d
# Start agent harness (Windows)
cd services-java/agent-harness
./mvnw.cmd spring-boot:run
# Verify health
curl http://localhost:8138/actuator/healthWorkflowRuntimeExecutor processes a WorkflowRuntimeDefinition as an ordered list of WorkflowRuntimeStep records loaded from the tenant database. Each step is dispatched to exactly one of four execution paths:
WorkflowRuntimeExecutor.execute(def, inputContext, inputBlackboard)
│
▼
┌─────────────────────────────────────┐
│ for each WorkflowRuntimeStep │
│ │
│ isGateStep? ──▶ PauseForReview │
│ isSetStep? ──▶ applySet() │
│ isForeachStep?──▶ applyForeach() │
│ isAdapterStep?──▶ applyAdapter() │ ← WorkflowStepAdapterRegistry lookup
│ else ──▶ applyLlm() │ ← UserLlmSelector → BrainLayer routing
└─────────────────────────────────────┘
Step results accumulate in two maps — context and blackboard — that flow through the entire run. Conditional branching throws BranchToStepException; HITL pause throws WorkflowPauseException. Both are caught cleanly without killing the thread.
The harness does not hard-code model configuration. Every LLM routing decision is resolved at runtime from two tables inside the tenant's private PostgreSQL instance:
| Column | Type | Purpose |
|---|---|---|
id |
uuid |
Primary key |
dept_id |
uuid |
Department FK — enforces dept boundary |
connection_id |
uuid |
FK → llm_connections |
brain_set_name |
varchar(128) |
Named brain profile (e.g. "freight-default", "customs-audit") |
brain_layer |
varchar(128) |
Role in execution (e.g. EXECUTION, REASONING, PERCEPTION) |
context_window_tokens |
integer |
Ceiling fed to BlackboardCompactionService |
max_output_tokens |
integer |
Hard output cap |
temperature / top_p |
numeric(3,2) |
Per-department inference tuning |
input_token_cost_per_million |
numeric(10,4) |
Cost tracking for quota ledger |
is_default |
boolean |
Fallback when no layer match exists |
fallback_model_id |
uuid |
Self-referential FK — hot-swap on failure |
| Column | Purpose |
|---|---|
dept_id |
Department scoping — NOT NULL, FK-enforced |
endpoint_url |
Can point to local Ollama (e.g. NVIDIA RTX 3090), cloud API, or internal proxy |
api_key |
Scoped to this department's operational boundary |
timeout_ms |
Per-connection network timeout |
provider |
anthropic, openai, ollama, custom |
UserLlmSelector.getClient(BrainLayer) resolves the correct LlmClientInterface via O(1) map lookup from the session's pre-loaded llmProfile. No string scanning, no dynamic queries during step execution.
All non-LLM execution is delegated to typed adapters registered in WorkflowStepAdapterRegistry. Adapter type is resolved from step.kind() using normalized string matching:
| Adapter | Step Kind | Purpose |
|---|---|---|
McpWorkflowStepAdapter |
"mcp" |
Dispatch to external MCP server tools; merges skill tool catalog with dept MCP catalog |
RagIngestShareStepAdapter |
"rag_ingest_share" |
Index shared enterprise file shares into pgvector; route through RagIngestionProcessor |
ScriptEngineStepAdapter |
"script" |
Execute sandboxed algorithm via SandboxedScriptEngineService with per-step timeoutMs |
MessageQueueStepAdapter |
"message_queue" |
Broadcast async events through validated MessageQueueRegistry |
StorageBucketStepAdapter |
"storage_bucket" |
Stage / retrieve files from object storage |
HumanApprovalStepAdapter |
"review" |
Serialize state → throw WorkflowPauseException → park thread |
HttpWorkflowStepAdapter |
"http" |
Outbound HTTP calls to external systems |
DbWorkflowStepAdapter |
"db" |
Direct tenant-scoped database operations |
Adding a new adapter requires implementing WorkflowStepAdapter, registering it as a Spring @Component, and declaring a new step kind in the workflow YAML definition. No changes to WorkflowRuntimeExecutor.
Mid-run context inflation is intercepted by BlackboardCompactionService before token ceilings defined in model_registry.context_window_tokens are breached:
BlackboardCompactionService.compactIfNeeded(scope, blackboard, config, extraMustKeepKeys)
│
├── totalBytes ≤ config.maxBytes? → return as-is
│
├── for each key in blackboard:
│ mustKeep? → carry forward verbatim
│ hotKey? → carry forward if ≤ maxKeyBytes, else truncate
│ else → move to summarySource
│
├── summarize(summarySource) → compacted["_summary"]
│
└── assertMustKeepIntegrity() → throws if required keys were dropped
Compaction events emit Micrometer counters and summaries under blackboard.compaction.* tags with scope and outcome dimensions. Connect to Prometheus/Grafana for per-workflow visibility.
LLM steps can opt in to a critique-and-revise loop via ReasoningOrchestrator:
# Workflow step YAML
- id: tariff_classification
kind: llm
instructions: "Classify HS code for {{commodity_description}}"
reflection:
enabled: true
maxRevisions: 2ReflectionRuntimeMetadata is persisted on each AgentTrace and WorkflowStepState record, including revisionsAttempted, satisfactory, maxRevisionsReached, and critiqueSummary. Full auditability with zero config overhead if reflection is disabled.
flowchart TD
A[Inbound HTTP / gRPC Request] --> B[UserIsolationFilter / UserApiTokenInterceptor]
B --> C[RequestContext.doAs\ndeptId · userId · sessionId · queueId]
C --> D[WorkflowHarness]
D --> E[WorkflowRuntimeExecutor]
E --> F{Step Kind?}
F -->|adapter| G[WorkflowStepAdapterRegistry]
F -->|llm| H[UserLlmSelector]
F -->|gate| I[WorkflowPauseException]
F -->|set / foreach| J[Context Mutation]
G --> G1[McpWorkflowStepAdapter]
G --> G2[ScriptEngineStepAdapter]
G --> G3[RagIngestShareStepAdapter]
G --> G4[StorageBucketStepAdapter]
G --> G5[MessageQueueStepAdapter]
H --> K[model_registry lookup\nby dept + brain_layer]
K --> L[llm_connections endpoint\nOllama / Cloud / Proxy]
I --> M[markQueuePaused\nWorkflowExecutionQueue]
M --> N[Thread Released to Pool]
N --> O[WorkflowLifecycleController\nResurrection Webhook]
O --> P[WorkflowQueueConsumer\npollResumeQueue @Scheduled]
P --> C
E --> Q[AgentTraceService\nWorkflowStepStateService\nAuditService]
classDef core fill:#1a1d23,stroke:#4a9eff,color:#e0e0e0,stroke-width:1px;
classDef infra fill:#1a1d23,stroke:#6b7280,color:#9ca3af,stroke-width:1px;
class A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q core;
class G1,G2,G3,G4,G5 infra;
Within a single tenant's PostgreSQL instance, every table that participates in workflow execution carries a dept_id uuid NOT NULL column with a FK to departments(id). Flyway migrations enforce this progressively:
| Migration | What It Enforces |
|---|---|
V20 |
department_rate_limits rename, dept_id NOT NULL, FK on workflow_instances |
V22 |
Phase-1 schema scope alignment across all execution tables |
V25 |
agent_trace.dept_id NOT NULL enforcement + FK cascade |
V27 |
blackboard_compaction_policy.scope_id UUID alignment |
Every query that touches a tenant-scoped collection filters on dept_id. Cross-department data leakage is structurally impossible within a single deployment.
RequestContext is a ThreadLocal-backed context carrier. The custom task decorator propagates it safely across thread-pool boundaries so asynchronous worker loops never suffer context amnesia:
// WorkflowQueueConsumer — resuming a parked workflow
RequestContext.doAs(payload.deptId(), sessionId, payload.userId(), () -> {
RequestContext.withUserApiToken(payload.securityToken(), () -> {
RequestContext.withSessionData(sessionData, () -> {
RequestContext.withQueueId(queueId, () -> {
workflowHarness.resume(payload.runId(), payload.reviewData());
});
});
});
});Every lambda boundary restores and clears context correctly. There is no risk of a Dept A context leaking into a Dept B execution slot regardless of thread pool reuse.
When a workflow reaches a HumanApprovalStepAdapter (step kind "review"), the entire execution thread is released back to the pool without blocking:
sequenceDiagram
participant Q as WorkflowQueueConsumer
participant E as WorkflowRuntimeExecutor
participant A as HumanApprovalStepAdapter
participant DB as workflow_execution_queue
participant W as WorkflowLifecycleController
Q->>E: execute(def, context, blackboard, startFromStepId)
E->>A: adapter.execute(step, context)
A->>A: context.put("reviewData", params + pausedStepId)
A-->>E: throw WorkflowPauseException("PAUSED_FOR_REVIEW")
E->>DB: markQueuePaused(queueId, stepId, context, blackboard)
E-->>Q: return WorkflowExecutionResult(paused=true, "PAUSED_FOR_REVIEW")
Q->>Q: inFlightResumePermits.release() → thread returned to pool
Note over DB: State serialized. Thread is free. No blocking.
W->>DB: POST /lifecycle/{runId}/resume → updateStatus("RESUME_QUEUED")
Q->>DB: pollResumeQueue @Scheduled(1000ms)
DB-->>Q: WorkflowExecutionQueue items (RESUME_QUEUED)
Q->>Q: orderForDepartmentFairness() → round-robin by deptId
Q->>Q: inFlightResumePermits.tryAcquire() [semaphore: maxInFlight=8]
Q->>E: execute(def, context, blackboard, startFromStepId=nextStepId)
Token quota enforcement gates every resume attempt before context is re-bootstrapped:
| Quota Window | Behavior |
|---|---|
minute |
Retry with minuteQuotaBackoff (default 30s), status → RESUME_QUOTA_BACKOFF |
day |
Retry with dayQuotaBackoff (default 600s) |
billing_month |
Hard reject, status → RESUME_REJECTED |
When multiple departments have concurrent resumable workflows, orderForDepartmentFairness() buckets items by deptId and interleaves them round-robin before dispatching. No department monopolizes the in-flight semaphore:
Input: [D1-wf1, D1-wf2, D2-wf1, D1-wf3, D3-wf1]
Output: [D1-wf1, D2-wf1, D3-wf1, D1-wf2, D1-wf3]
These scenarios are built to exercise the harness under adversarial conditions representative of enterprise freight forwarding deployments.
Domain: International freight / customs compliance
Goal: Validate that a multi-step manifest audit pipeline runs correctly under concurrent cross-department load, that tariff lookups via pgvector survive high-cardinality vector space, and that the HITL gate triggers correctly when a classification confidence threshold is breached.
Workflow topology:
workflow:
id: customs-manifest-audit
brain_set_name: customs-audit
steps:
- id: fetch_manifest
kind: storage_bucket
params:
bucketId: "{{manifestBucketId}}"
objectKey: "{{shipmentId}}/manifest.pdf"
- id: ingest_tariff_schedule
kind: rag_ingest_share
params:
shareId: "{{tariffScheduleShareId}}"
namespace: "hs_tariff_2024"
- id: classify_commodities
kind: llm
instructions: >
Using the indexed tariff schedule, classify each line item in
{{fetch_manifest.content}} with its 6-digit HS code and applicable duty rate.
reflection:
enabled: true
maxRevisions: 2
outputs:
classifications:
type: array
- id: audit_gate
kind: review
params:
reason: low_confidence_classification
threshold: 0.85
- id: broadcast_audit_result
kind: message_queue
params:
queueId: "{{customsAuditQueueId}}"
payload: "{{classify_commodities}}"What to inject:
- Run 12 concurrent manifest workflows across 3 departments simultaneously.
- Kill the
tariffScheduleShareIdfile share mid-ingest on department 2. - Verify
RagIngestShareStepAdaptersurfaces the error,WorkflowRuntimeExecutorcatches theRuntimeException, evaluatespauseOnErrorsconfig, and parks the run atingest_tariff_schedule. - Resume department 2 via
WorkflowLifecycleControllerwebhook after file share is restored. - Assert department 1 and department 3 pipelines completed without any cross-department blackboard contamination. Check
agent_traces.dept_idfor every completed step.
Assertions:
-- No cross-department trace leakage
SELECT dept_id, COUNT(*) FROM agent_traces
WHERE session_id IN (/* run session IDs */)
GROUP BY dept_id
HAVING COUNT(DISTINCT dept_id) > 1; -- must return 0 rows
-- All classifications persisted with correct dept scope
SELECT COUNT(*) FROM workflow_step_states wss
JOIN workflow_instances wi ON wi.id = wss.instance_id
WHERE wi.dept_id != /* expected dept UUID */; -- must return 0Domain: Intermodal transportation — rail/ocean/trucking demurrage
Goal: Verify that ScriptEngineStepAdapter correctly sandboxes a compute-heavy demurrage penalty calculation, that BlackboardCompactionService triggers and survives when accumulated routing context inflates past token ceiling, and that a mid-run connection pool drop exercises HITL parking and state re-hydration without data loss.
Workflow topology:
workflow:
id: intermodal-routing-disruption-audit
brain_set_name: logistics-ops
steps:
- id: ingest_routing_events
kind: llm
instructions: >
Parse the following EDI 315 ocean status messages and extract
container ID, vessel, POD, and ETA for each leg: {{edi_payload}}
outputs:
routing_legs: { type: array }
- id: calculate_demurrage
kind: script
params:
scriptId: "{{demurragePenaltyScriptUUID}}"
inputKey: routing_legs
timeoutMs: 3000 # sandbox kills script at 3 s — no JVM escape
- id: llm_disruption_analysis
kind: llm
instructions: >
Given the demurrage penalties in {{calculate_demurrage.output}},
identify the highest-risk legs and recommend re-routing options.
reflection:
enabled: true
maxRevisions: 1
- id: ops_review_gate
kind: review
params:
reason: high_demurrage_exposure
exposureThreshold: 50000
- id: notify_freight_ops
kind: message_queue
params:
queueId: "{{freightOpsQueueId}}"What to inject:
-
Sandboxed script abuse: Supply a demurrage script that contains an infinite loop. Assert
SandboxedScriptEngineServiceterminates execution attimeoutMs=3000and surfaces a typed error that triggersbranchOnErrorsrouting to a fallback step. -
Blackboard inflation: Feed 40+ routing legs with dense ETA/delay history so
llm_disruption_analysispushes the blackboard pastcontext_window_tokensbefore the LLM call. AssertBlackboardCompactionServicefires (blackboard.compaction.events{outcome="compacted"}counter increments),must_keepkeys survive intact, and the LLM step completes successfully with the compacted context. -
Connection pool drop during HITL: After
ops_review_gateparks the workflow, drop the LLM connection defined inllm_connectionsfor this department (simulate network partition to local Ollama). Resume the workflow via webhook. AssertWorkflowQueueConsumerre-bootstrapsUserSessionData, theUserLlmSelectorfalls back tomodel_registry.fallback_model_id, and thenotify_freight_opsstep completes without re-executing the already-completed steps.
Assertions:
-- Compaction fired on the disruption analysis step
SELECT * FROM blackboard_compaction_policies
WHERE scope_id = /* dept UUID */
AND last_triggered_at > NOW() - INTERVAL '10 minutes';
-- HITL parking and resurrection recorded correctly
SELECT status, retry_count, resumed_at FROM workflow_execution_queue
WHERE run_id = /* run UUID */
ORDER BY created_at;
-- Expected: PAUSED → RESUME_QUEUED → RESUMED
-- Demurrage script step shows FAILED status with correct error message
SELECT step_status, error_message FROM workflow_step_states
WHERE step_id = 'calculate_demurrage'
AND error_message LIKE '%timeout%';Implement the interface and register as a Spring component. No changes to the executor:
@Component
public class ShipmentTrackingStepAdapter implements WorkflowStepAdapter {
private final ShipmentTrackingClient client;
public ShipmentTrackingStepAdapter(ShipmentTrackingClient client) {
this.client = client;
}
@Override
public String getType() {
return "shipment_track"; // step kind in YAML
}
@Override
public Map<String, Object> execute(WorkflowRuntimeStep step, Map<String, Object> context) {
String trackingNumber = Objects.requireNonNull(
(String) context.get("trackingNumber"), "trackingNumber required");
ShipmentStatus status = client.track(trackingNumber);
return Map.of(
"carrier", status.carrier(),
"location", status.lastKnownLocation(),
"eta", status.estimatedDelivery(),
"exceptions", status.activeExceptions()
);
}
}The adapter is immediately available to any workflow YAML using kind: shipment_track.
Each department can configure its own MCP tool catalog in llm_connections. The McpServiceCatalog.getToolSpecsForCurrentDept() call resolves tool specifications for the active department from RequestContext.deptId() — so adding freight-specific MCP servers (track-and-trace APIs, customs databases, port authority feeds) requires only a new llm_connections row and a corresponding MCP server process. No harness code changes.
# Workflow step using a department-specific MCP tool
- id: query_port_authority
kind: mcp
params:
tool: port_authority.get_vessel_eta
args:
vessel_imo: "{{vesselImo}}"
port_code: "{{destinationPort}}"The harness exposes execution state through two queryable surfaces suited for a TUI dashboard implementation:
| Endpoint | Purpose |
|---|---|
GET /workflow/instances/{runId} |
Full run state with step-level status |
GET /workflow/instances/{runId}/steps |
Step-by-step WorkflowStepState records with input/output JSON |
POST /workflow/instances/{runId}/resume |
Inject HITL review decision, enqueue for resurrection |
GET /agent/traces?sessionId=... |
Per-session AgentTrace records with timing, tool calls, reflection metadata |
Live execution events stream over gRPC for low-latency TUI rendering. Connect a TUI client (Bubble Tea, Textual, or custom) to the gRPC stream and render:
- Active workflow step name and status
- Blackboard key count and estimated byte size (compaction pressure gauge)
BrainLayercurrently executing and whichllm_connectionsendpoint it's hitting- Tool call log with
durationMsper call - HITL queue depth by department
func (m model) pollWorkflowStatus() tea.Cmd {
return func() tea.Msg {
resp, _ := http.Get(fmt.Sprintf(
"http://localhost:8138/workflow/instances/%s/steps", m.runId))
var steps []WorkflowStepState
json.NewDecoder(resp.Body).Decode(&steps)
return stepsMsg{steps: steps}
}
}No code deployment required. Insert rows into the tenant's private PostgreSQL instance:
-- Register a local Ollama endpoint for the customs audit department
INSERT INTO llm_connections (id, dept_id, provider, endpoint_url, api_key, timeout_ms)
VALUES (
uuid_generate_v4(),
'dept-uuid-customs',
'ollama',
'http://10.0.1.42:11434', -- NVIDIA RTX 3090 host
'ollama-local',
45000
);
-- Bind a model to the EXECUTION brain layer for this department
INSERT INTO model_registry (
id, connection_id, dept_id, brain_set_name, brain_layer,
is_default, context_window_tokens, max_output_tokens,
input_token_cost_per_million, output_token_cost_per_million,
temperature, is_active
) VALUES (
uuid_generate_v4(),
'/* connection id above */',
'dept-uuid-customs',
'customs-audit',
'EXECUTION',
false,
32768,
4096,
0.0000, -- local Ollama — no token cost
0.0000,
0.70,
true
);The next session bootstrapped for dept-uuid-customs with brain_set_name = 'customs-audit' will route EXECUTION layer calls to the local Ollama instance — without a redeploy.
# Fast compile check (agent-harness)
cd services-java/agent-harness
./mvnw.cmd -q -DskipTests compile
# Unit tests only
./mvnw.cmd test
# Full integration test suite (Failsafe — *IT, *IntegrationTest)
./mvnw.cmd verify
# Single test class
./mvnw.cmd "-Dtest=WorkflowE2EIT" test
# Single method
./mvnw.cmd "-Dtest=WorkflowHarnessTest#resumeRejectsInvalidToken" testH2 test profile: Integration tests use H2 in PostgreSQL compatibility mode (
MODE=PostgreSQL,ddl-auto=create-drop). Flyway applies all migrations against H2. No local PostgreSQL required for CI.
gRPC in tests: Disabled by default via
src/test/resources/application-test.yml. Do not flip the runtime default to fix test failures.