Skip to content

worker-guide.md

Latest commit

 

History

History
271 lines (181 loc) · 14.4 KB

worker-guide.md

File metadata and controls

271 lines (181 loc) · 14.4 KB
Raw

Worker Guide

Guide for deploying, managing, and troubleshooting AgentTeams Worker Agents.

Overview

Workers are lightweight, stateless containers that:

  • Connect to the Manager via Matrix for task communication
  • Sync configuration from centralized MinIO storage
  • Use AI Gateway for LLM access
  • Use mcporter CLI for MCP Server tool calls (GitHub, etc.)

Declarative create / update (v1.1.0+)

Workers are CRD-backed. Besides asking the Manager in Matrix, you can:

  • Run agt create worker / agt update worker inside agentteams-controller or agentteams-manager (see faq.md).
  • Apply YAML with install/agentteams-apply.sh (forwards to agt apply -f in the Manager container).

Full field reference: Declarative Resource Management.

Filesystem layout by spec.runtime

Runtime Primary workspace Notes
openclaw /root/agentteams-fs/agents/<worker-name>/ (HOME points here) openclaw.json, SOUL.md, AGENTS.md, skills, .openclaw/ live under this tree. Shared data: /root/agentteams-fs/shared/.
copaw /root/.agentteams-worker/<worker-name>/ (runtime config in .copaw/) Legacy compatibility path. A symlink /root/agentteams-fs → the per-worker tree keeps scripts that assume OpenClaw-style paths working.
qwenpaw /root/agentteams-fs/agents/<worker-name>/ (QwenPaw config in .qwenpaw/) QwenPaw 2.x path. When switching from copaw, persisted state is restored before the legacy .copaw/ state is migrated to .qwenpaw/. Shared data: /root/agentteams-fs/shared/.
hermes /root/agentteams-fs/agents/<worker-name>/ (HOME equals workspace, same mirror root as OpenClaw) Hermes policy/state under .hermes/ inside that directory (e.g. .hermes/config.yaml, state.db).

The Controller contains an OpenHuman backend, but the shipped Worker CRD enum does not currently accept an explicit spec.runtime: openhuman. Until that contract is fixed in a separate business-code change, do not create an OpenHuman Worker through the normal Worker CR flow.

Installation

Workers are created by the Manager Agent or the controller via declarative APIs. The Manager handles all infrastructure setup (Matrix account, Higress consumer, config files, etc.) and can either create the Worker container directly or provide a command for manual execution.

Method 1: Direct Creation (Recommended for Local Development)

If the Manager has access to the host's container runtime socket (default when using make install), it can create Worker containers directly:

  1. Tell Manager: "Create a new Worker named alice for frontend dev. Create it directly."
  2. Manager creates all infrastructure and starts the container automatically
  3. No manual steps needed

Method 2: Docker Run Command (for Manual or Edge Deployment)

If the Manager doesn't have socket access, it will reply with a docker run command:

  1. Tell Manager: "Create a new Worker named alice for frontend dev"
  2. Manager creates infrastructure and provides a docker run command
  3. Copy and run the command on the target host:
docker run -d --name agentteams-worker-alice \
  -e AGENTTEAMS_WORKER_NAME=alice \
  -e AGENTTEAMS_FS_ENDPOINT=http://<MANAGER_HOST>:9000 \
  -e AGENTTEAMS_FS_ACCESS_KEY=<ACCESS_KEY> \
  -e AGENTTEAMS_FS_SECRET_KEY=<SECRET_KEY> \
  agentteams/worker-agent:latest

The Manager will provide all the specific values in its reply.

Installing Skills on a Worker

There are currently two stable ways to install a Skill on an existing Worker:

Method Best for Persistent result
Distribute through the Manager Let the Manager validate, distribute, and maintain a declarative assignment record Uploads the complete Skill and updates Worker.spec.skills
Distribute through the Dashboard Select a target Worker in the UI when you already have a ZIP Uploads directly to Worker object storage and triggers a reload; does not update Worker.spec.skills

Both methods ultimately write the Skill files to the Worker's persistent storage, from which the Worker runtime synchronizes and loads them. They differ mainly in their entry point and whether they maintain the spec.skills assignment record.

Method 1: Distribute through the Manager

Provide the Skill to the Manager in either of these ways:

  1. Put the complete third-party skill under $AGENTTEAMS_WORKSPACE_DIR/worker-skills/<skill-name>/ on the Manager host. The default is ~/agentteams-manager/worker-skills/<skill-name>/; or
  2. Send the Manager a ZIP attachment containing one complete Skill root with SKILL.md and any optional scripts/ or references/.

Then ask the Manager to install that Skill for a named Worker and verify the assignment. For a ZIP attachment, explicitly ask the Manager to safely extract and validate it before distribution.

For example:

Install the alert-fusion skill from ~/worker-skills/alert-fusion/ for Worker amy-ai. Verify that the files were uploaded and the Worker assignment was updated.

Or, after sending the ZIP attachment:

Install the Skill from the ZIP attachment I just sent for Worker amy-ai. Safely extract and validate it, distribute the complete Skill, and verify the Worker assignment.

The Manager uploads and verifies the files before updating Worker.spec.skills. This ordering prevents a Worker from receiving an assignment that points to missing content. QwenPaw Workers then synchronize the assigned skill into their native workspace and refresh and enable it automatically.

You can check the assignment by asking the Manager:

List the skills currently assigned to Worker amy-ai and confirm whether alert-fusion is included.

For operator-side inspection or troubleshooting, use the equivalent CLI query:

agt get workers amy-ai -o json | jq '.skills'

agt get workers amy-ai -o json | jq '.skills' shows the spec.skills assignment record maintained through the Manager or declarative API. For a Skill distributed directly through the Dashboard, verify it through the Dashboard or Worker runtime as described below.

Method 2: Distribute through the Dashboard

This method requires an enabled Dashboard with credentials for the AgentTeams object store. Embedded installations created by the AgentTeams Bash installer configure this connection automatically.

Prepare the Skill ZIP

The upload package must meet these requirements:

  • The filename ends in .zip, and the archive is no larger than 64 MB.
  • The archive contains one complete Skill root. SKILL.md may be at the ZIP root or inside that Skill root directory.
  • SKILL.md starts with YAML frontmatter containing non-empty name and description fields.
  • name contains only letters, numbers, dots, underscores, and hyphens, and starts with a letter or number.
  • Include scripts/, references/, and other supporting files with SKILL.md; the Dashboard preserves the complete file structure under the Skill root.

Distribute from the Skill Center

  1. Open the Dashboard and select 技能中心 (Skill Center) in the left navigation.
  2. Click 分发技能 (Distribute Skill) beside the page title.
  3. Select the target Worker in the dialog.
  4. Drop or choose the Skill ZIP and confirm that the filename and size appear.
  5. Click 分发技能 (Distribute Skill) and wait for the result and reload note.

You can also use Workers → target Worker → 详情 (Details) → 上传技能包 (Upload Skill Package) to enter the same Worker-specific distribution flow.

Note: 上传技能 (Upload Skill) above the Skill Center list adds a Skill to the Dashboard's centralized Skill catalog; it does not distribute that Skill to a Worker. To install a Worker Skill, use 分发技能 (Distribute Skill) beside the page title or 上传技能包 (Upload Skill Package) in Worker details.

Load and verify

The Dashboard validates the ZIP and SKILL.md, then writes the files to agents/<worker-name>/skills/<skill-name>/ in object storage. After upload, it attempts to put the Worker to sleep and wake it again so the runtime loads the Skill immediately. Distribute only while the Worker is idle to avoid interrupting an active task.

  • On a successful reload, the UI reports that the Worker was notified to load the new Skill.
  • If sleep or wake fails, the uploaded files remain available; the UI reports that periodic synchronization should discover them within about five minutes.

Reopen Workers → target Worker → 详情 (Details) and confirm the Skill under 已分发技能 (Distributed Skills). To verify runtime availability rather than only file presence, ask that Worker to confirm it can discover and use the Skill.

Direct Dashboard distribution does not modify Worker.spec.skills, so the Skill may not appear in agt get workers <name> -o json | jq '.skills'. Use Manager distribution when you require a declarative assignment record, future Manager-driven redistribution, or assignment auditing.

For an end-to-end example that starts with packaging and verifies distribution, runtime discovery, and actual use, follow Use case 6: Add and use a custom Skill.

Troubleshooting

Worker won't start

# Check container logs
docker logs agentteams-worker-alice

# Common issues:
# - "openclaw.json not found": Manager hasn't created config yet
# - "mc: command not found": Image build issue
# - Connection refused: Manager container not running or ports not exposed

Worker can't connect to Matrix

# Verify Matrix server is reachable from Worker (via gateway port)
docker exec agentteams-worker-alice curl -sf http://matrix-local.agentteams.io:18080/_matrix/client/versions

# Check Worker's openclaw.json for correct Matrix config
docker exec agentteams-worker-alice cat /root/agentteams-fs/agents/alice/openclaw.json | jq '.channels.matrix'

Worker can't access LLM

# Test AI Gateway access with Worker's key
# Note: these commands run inside the Worker container where domain names resolve to Manager's internal IP
docker exec agentteams-worker-alice curl -sf \
  -H "Authorization: Bearer $(jq -r '.models.providers."agentteams-gateway".apiKey' /root/agentteams-fs/agents/alice/openclaw.json)" \
  http://aigw-local.agentteams.io:8080/v1/models

# If 401: Check that Worker's consumer key in openclaw.json matches the one in Higress.
# If 403: Worker may not be authorized for the AI route. Ask Manager to add.

Worker can't access MCP (GitHub)

# Test mcporter connectivity (run inside Worker container)
docker exec agentteams-worker-alice mcporter --transport http \
  --server-url "http://aigw-local.agentteams.io:8080/mcp-servers/mcp-github/mcp" \
  --header "Authorization=Bearer <WORKER_KEY>" \
  call list_repos '{"owner": "test"}'

# If 403: Worker not authorized for this MCP Server. Ask Manager.

Resetting a Worker

# Stop and remove the container
docker stop agentteams-worker-alice
docker rm agentteams-worker-alice

# Then ask Manager to recreate the Worker:
# "Please recreate the alice worker container"
# Manager will re-run create-worker.sh which regenerates credentials and restarts the container

Note: Worker config and task data live in MinIO, not in the container. Removing the container does not lose any work.

Lifecycle Management

The Manager automatically manages Worker container lifecycle:

  • Auto-stop: Idle Workers (no active finite tasks) are stopped after a configurable timeout to save resources
  • Auto-start: When a task is assigned to a stopped Worker, the Manager wakes it up before sending the task
  • Auto-recreate on restart: When the Manager container restarts, it checks all registered Workers and recreates any whose container is missing or whose Manager IP has changed

You can also manually control Workers by asking the Manager:

  • "Stop the alice worker"
  • "Start the alice worker"
  • "What is the status of all workers?"

Architecture Details

Startup Sequence

Each runtime has its own entrypoint, but they all perform the following work:

  1. Obtain object-storage credentials and restore configuration and persistent state from agents/<name>/.
  2. Prepare the runtime-specific workspace, agent prompt files, and skills.
  3. Translate model, Matrix channel, MCP Server, and Team context into the runtime's native configuration.
  4. Start file synchronization or the runtime configuration update loop.
  5. Launch the selected OpenClaw, CoPaw, QwenPaw, or Hermes runtime.

See Filesystem layout by spec.runtime above for the concrete paths. When troubleshooting, use the logs and configuration paths for the selected runtime instead of applying OpenClaw's openclaw.json consumption model to every Worker.

File Sync

  • Local to Remote: Real-time via mc mirror --watch
  • Remote to Local: Periodic pull every 5 minutes

Config Hot-Reload

When Manager updates Worker's config in MinIO:

  1. MinIO receives the updated file
  2. mc mirror pulls it to Worker's local filesystem (next 5-min cycle, or immediately if Manager pushes)
  3. OpenClaw detects file change (~300ms) and hot-reloads config

Environment Variables

Variable Description
AGENTTEAMS_WORKER_NAME Worker identifier (e.g., alice)
AGENTTEAMS_MATRIX_URL Matrix Homeserver URL (e.g., http://matrix-local.agentteams.io:18080)
AGENTTEAMS_AI_GATEWAY_URL AI Gateway URL (e.g., http://aigw-local.agentteams.io:18080)
AGENTTEAMS_FS_ENDPOINT MinIO endpoint URL (e.g., http://<MANAGER_HOST>:9000)
AGENTTEAMS_FS_BUCKET Bucket name for non-default storage layouts
AGENTTEAMS_FS_ACCESS_KEY MinIO access key (Worker-specific, generated by Manager)
AGENTTEAMS_FS_SECRET_KEY MinIO secret key (Worker-specific, generated by Manager)

All values are generated by the Manager and provided in the docker run command or set automatically during direct creation. You should not need to set these manually.

Runtime scripts now use AGENTTEAMS_MATRIX_URL and AGENTTEAMS_AI_GATEWAY_URL directly; legacy aliases are no longer part of the main contract.

Syncing Files Manually

Inside the Worker container, run agentteams-sync to pull the latest config and skill files from MinIO:

docker exec agentteams-worker-alice agentteams-sync

This is useful after the Manager pushes updated skills or config to MinIO and you want to apply them immediately without waiting for the next sync cycle.