Knowledge Agent MVP

Phase A MVP for a secure, on-prem ready RAG stack built with vLLM, Dify, and Qdrant.

Objectives

• Run Qwen2.5-72B-Instruct-AWQ locally through vLLM.
• Provide a Dify UI and API for agent orchestration.
• Enrich documents before ingestion using ingest.py.
• Scale from single A100 to 4x GPU cluster without code changes.

Requirements

• Docker and Docker Compose.
• NVIDIA drivers and CUDA runtime for vLLM.
• Local model weights available at MODEL_PATH.

Quick Start

1. Copy env file and set secrets.

cp .env.example .env

2. Set MODEL_PATH to your local quantized model directory.

3. Start the stack.

docker compose up -d

4. Open Dify console.

• Web UI: http://localhost:3000.
• API: http://localhost:5001.

5. Configure the model provider in Dify UI.

• Provider: OpenAI compatible.
• Base URL: http://vllm:8000/v1.
• API key: local-key.
• Model name: Qwen2.5-72B-Instruct-AWQ.

Air-Gapped Dify Config

Dify configuration is split into two files so you can transfer the full folder without relying on external services.

• dify-api.env, used by dify-api and dify-worker.
• dify-web.env, used by dify-web.

These files keep all LLM traffic inside the private network by targeting http://vllm:8000/v1.

Ingestion

One-shot

python ingest.py \
  --input-dir ./data/input \
  --output-file ./data/enriched/enriched.jsonl

Watch mode

python ingest.py --watch

Upload to Dify dataset

python ingest.py --upload-to-dify

Make sure DIFY_API_KEY and DIFY_DATASET_ID are set in .env.

Phase A Acceptance Checklist

• docker compose up -d succeeds.
• vLLM responds on http://localhost:8000/v1/models.
• Dify UI available on http://localhost:3000.
• Ingest creates JSONL records in ./data/enriched/enriched.jsonl.
• Upload mode inserts documents into a Dify dataset.
• Dify query returns a grounded answer from enriched documents.

On-Prem Migration

No code changes are required. Only environment changes are needed:

• TENSOR_PARALLEL_SIZE=4.
• Adjust GPU visibility and runtime to match the 4x GPU host.
• Update MODEL_PATH to on-prem storage location.
• Apply on-prem secrets and network constraints.

Optional Extensions

• Dify sandbox and plugin daemon are not included in this MVP compose.
• Add them later if you need code execution or plugin support.

Remote Dify Recovery Scripts

Use these scripts on the remote server to avoid manual env edits and shell source parsing failures.

1. Normalize Dify env files.

/workspace/knowledge_agent/scripts/repair_dify_env.sh \
  /workspace/dify/api/.env \
  /workspace/dify/api/.env.runtime \
  /workspace/dify-plugin-daemon/.env

2. Restart only Dify API and worker with safe env loading.

/workspace/knowledge_agent/scripts/restart_remote_dify_services.sh

3. Recheck health.

curl --max-time 5 -s http://127.0.0.1:5001/console/api/setup | jq .
curl --max-time 5 -s http://127.0.0.1:5001/console/api/system-features | jq '.enable_marketplace, .plugin_manager'

Fast Remote Setup (No Docker, Vast Runtime)

For the remote /workspace layout used in this project, use the runbook script instead of long manual command chains.

1. Baseline startup plus env normalization and API or worker restart.

scripts/ops_runbook.sh start

2. Select runtime mode.

# Indexing mode: embed on :8001, chat off :8000
scripts/ops_runbook.sh mode-indexing

# Chat mode: chat on :8000, embed off :8001
scripts/ops_runbook.sh mode-chat

3. Run health checks.

scripts/ops_runbook.sh health

4. Create shutdown snapshot backup.

scripts/ops_runbook.sh backup

Reference details and incident history are documented in REMOTE_RUNTIME_HANDOFF.md.

Security Hardening and Rollback

The setup flow remains one-command (scripts/ops_runbook.sh start), but now supports safer defaults with explicit rollback switches.

What changed

• scripts/ops_runbook.sh start now auto-loads secrets from SECRETS_FILE (default: /workspace/.secrets.env) when present.
• Startup now validates effective runtime values for:
  • SECRET_KEY (must not be placeholder values)
  • DB_PASSWORD (must not be default dify or placeholder values)
• scripts/restart_remote_dify_services.sh now runs run_with_env_file.sh with ENV_OVERRIDE_MODE=preserve, so pre-exported or sourced secrets are not overwritten by template values in env files.
• scripts/ops_runbook.sh mode-chat now supports CHAT_TRUST_REMOTE_CODE toggle.
• docker-compose.yml now binds internal services (5432, 6379, 6333) to ${INTERNAL_BIND_IP:-127.0.0.1}.

Recommended secrets file

Create /workspace/.secrets.env on the server (do not commit it):

cat >/workspace/.secrets.env <<'EOF'
SECRET_KEY=<strong-random-secret>
DB_PASSWORD=<strong-db-password>
EOF

Rollback switches

If anything breaks and you need legacy behavior quickly:

1. Skip startup secret validation:

ALLOW_INSECURE_DEFAULTS=1 scripts/ops_runbook.sh start

2. Force legacy model trust flag behavior:

CHAT_TRUST_REMOTE_CODE=1 scripts/ops_runbook.sh mode-chat

3. Re-open internal Docker service bindings to all interfaces:

INTERNAL_BIND_IP=0.0.0.0 docker compose up -d

4. Revert env merge behavior when using run_with_env_file.sh directly:

ENV_OVERRIDE_MODE=overwrite scripts/run_with_env_file.sh <env-file> <command>

Backup Snapshot Usage

Backup archives like backup_<timestamp>.tgz are runtime snapshots, not source code artifacts.

• Keep them outside git.
• Download them before stopping a paid instance.
• Use them to restore env files, inspect logs, and restore DB state.

Download example:

scp -i ~/.ssh/<your-key> -P <port> \
root@<server-ip>:/workspace/exports/backup_<timestamp>.tgz \
~/Downloads/

Extract example:

tar -xzf backup_2026-02-14_014252.tgz

If dify.sql exists in the extracted backup folder, restore example:

runuser -u postgres -- psql -d dify < dify.sql

What Must Be in GitHub

To make future instances reproducible with only clone and launch, keep these tracked in the repo:

• scripts/ops_runbook.sh
• scripts/repair_dify_env.sh
• scripts/restart_remote_dify_services.sh
• REMOTE_RUNTIME_HANDOFF.md
• tests for operational scripts under tests/

Do not commit runtime backups or secrets:

• backup_*.tgz
• *.sql dumps from live environments
• copied .env files with secrets

Evidence (Option A, Dify References)

This MVP uses Dify built in references, not a custom citation UI.

Steps:

• Create or open an App in Dify.
• Attach the target Dataset to the App Knowledge Base.
• Enable the references or sources toggle in the App response settings.
• Ask a question, then open the references list under the answer to see source snippets.

Note: This provides document level or chunk level references. It is not a side panel viewer.

Tests

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -q