Skip to content

Container Architecture

Jump to bottom
Chris Zinda edited this page Mar 7, 2026 · 3 revisions

Container Architecture

This page documents all container images, custom-built containers, compose file organization, volume management, and healthcheck patterns used in the Certificate Revocation Lab.

Table of Contents

Container Images and Sources

PKI and Identity

Image Registry Version Variable Default Purpose
quay.io/dogtagpki/pki-ca Quay.io PKI_VERSION latest Dogtag PKI CA, OCSP, KRA, EST RA, ACME RA
quay.io/389ds/dirsrv Quay.io DS_VERSION latest 389 Directory Server (LDAP backend)
quay.io/freeipa/freeipa-server Quay.io IPA_VERSION fedora-43 FreeIPA identity management
localhost/dogtag-pki-pq Local build PQ_PKI_IMAGE localhost/dogtag-pki-pq:latest Custom Dogtag with ML-DSA-87 support

Event Streaming

Image Registry Version Variable Default Purpose
docker.io/confluentinc/cp-kafka Docker Hub KAFKA_VERSION 7.5.0 Apache Kafka broker
docker.io/confluentinc/cp-zookeeper Docker Hub ZOOKEEPER_VERSION 7.5.0 ZooKeeper coordination service

Automation

Image Registry Version Variable Default Purpose
quay.io/ansible/awx-ee Quay.io AWX_EE_VERSION latest AWX Execution Environment
quay.io/ansible/ansible-rulebook Quay.io EDA_VERSION v1.0.0 Event-Driven Ansible rulebook engine

Databases and Caching

Image Registry Version Variable Default Purpose
quay.io/hummingbird/postgresql Quay.io (Hummingbird) POSTGRES_VERSION latest PostgreSQL (AWX database)
quay.io/hummingbird/valkey Quay.io (Hummingbird) VALKEY_VERSION latest Valkey -- Redis-compatible cache (AWX)

Note: The lab uses Hummingbird project images for PostgreSQL and Valkey. Valkey is a Redis-compatible fork; the container name remains redis for AWX compatibility but runs the valkey-cli binary.

Monitoring

Image Registry Version Variable Default Purpose
quay.io/prometheus/prometheus Quay.io PROMETHEUS_VERSION latest Prometheus metrics collection
docker.io/grafana/grafana Docker Hub GRAFANA_VERSION latest Grafana dashboards
docker.io/grafana/loki Docker Hub LOKI_VERSION latest Loki log aggregation
docker.io/grafana/promtail Docker Hub PROMTAIL_VERSION latest Promtail log shipping agent

Development

Image Registry Version Variable Default Purpose
quay.io/jupyter/minimal-notebook Quay.io JUPYTER_VERSION latest Jupyter Lab for demos and documentation

Custom-Built Containers

These containers are built from Containerfile definitions in the containers/ directory. They are built automatically by podman-compose build or start-lab.sh.

Container Build Context Host Port Description
mock-edr containers/mock-edr/ 8082 Mock Endpoint Detection and Response (FastAPI)
mock-siem containers/mock-siem/ 8083 Mock Security Information and Event Management (FastAPI)
mock-ct-log containers/mock-ct-log/ 8086 Mock Certificate Transparency Log (FastAPI)
iot-client containers/iot-client/ 8085 IoT device simulator with EST-first enrollment
pki-exporter containers/pki-exporter/ 9091 Prometheus exporter scraping all 9+ Dogtag CAs
crl-server containers/crl-server/ 8088 CRL Distribution Point server (RFC 5280)
policy-engine containers/policy-engine/ 8089 Certificate policy evaluation engine
chain-visualizer containers/chain-visualizer/ 8090 Web UI for visualizing certificate chains
dnsmasq containers/dnsmasq/ 5353 Lightweight DNS for wildcard *.cert-lab.local
mtls-proxy containers/mtls-proxy/ 9443/8087 Nginx-based mTLS reverse proxy demo
pin-validator containers/pin-validator/ 8091 Certificate pinning validator with Kafka integration
kmip-server containers/kmip-server/ 8092/5696 PyKMIP key management server with REST API
kryoptic-hsm containers/kryoptic-hsm/ (internal) Kryoptic PKCS#11 HSM (built from Rust source)
dogtag-pki-pq containers/dogtag-pq/ -- Custom Dogtag with ML-DSA-87 (used by PQ compose)

Building Custom Containers

# Build all custom containers in main compose
podman-compose build

# Build specific containers
podman-compose build mock-edr mock-siem

# Build the post-quantum Dogtag image
podman build -t localhost/dogtag-pki-pq:latest containers/dogtag-pq/

Compose File Organization

The lab is split across five compose files to separate rootless and rootful podman requirements.

Compose File Podman Mode Network Description
podman-compose.yml Rootless lab-network (172.20.0.0/16) Main stack: DNS, Kafka, EDA, AWX, monitoring, mock services, Jupyter
pki-compose.yml Rootful pki-net (172.26.0.0/24) RSA-4096 PKI: Root CA, Intermediate CA, IoT CA, ACME RA, EST RA, OCSP, KRA + 389 DS instances
pki-ecc-compose.yml Rootful pki-ecc-net (172.28.0.0/24) ECC P-384 PKI: Root CA, Intermediate CA, IoT CA, EST RA, OCSP, KRA + 389 DS instances
pki-pq-compose.yml Rootful pki-pq-net (172.27.0.0/24) ML-DSA-87 PKI: Root CA, Intermediate CA, IoT CA, EST RA, OCSP, KRA + 389 DS instances
freeipa-compose.yml Rootful freeipa-net (172.25.0.0/24) FreeIPA server
federation-compose.yml Rootful federation-net (172.29.0.0/24) Partner Org + Bridge CA for federated trust

Starting the Lab

The start-lab.sh script orchestrates all compose files with proper ordering:

# Start everything (all three PKI hierarchies)
./start-lab.sh --all

# Start specific PKI types
./start-lab.sh --rsa          # RSA only
./start-lab.sh --ecc          # ECC only
./start-lab.sh --pqc          # Post-quantum only
./start-lab.sh --dual         # RSA + PQ
./start-lab.sh --rsa --ecc    # RSA + ECC

# Clean start (removes all data and volumes)
./start-lab.sh --clean --all

Semaphore UI

All start/stop operations are also available as Semaphore task templates:

Template Playbook Description
Lab Start ansible/playbooks/ops/lab-start.yml Start with pki_mode and clean options
Lab Stop ansible/playbooks/ops/lab-stop.yml Stop with optional volume cleanup
Lab Status ansible/playbooks/ops/lab-status.yml Health check all services

See Ansible Semaphore for setup and full template list.

Manual Compose Commands

# Main stack (rootless)
podman-compose up -d
podman-compose logs -f kafka

# RSA PKI (rootful)
sudo podman-compose -f pki-compose.yml up -d
sudo podman-compose -f pki-compose.yml logs -f dogtag-root-ca

# ECC PKI (rootful)
sudo podman-compose -f pki-ecc-compose.yml up -d

# PQ PKI (rootful)
sudo podman-compose -f pki-pq-compose.yml up -d

# FreeIPA (rootful)
sudo podman-compose -f freeipa-compose.yml up -d

Stopping the Lab

./stop-lab.sh              # Stop all containers
./stop-lab.sh --rsa        # Stop RSA PKI only
./stop-lab.sh --ecc        # Stop ECC PKI only
./stop-lab.sh --pqc        # Stop PQ PKI only
./stop-lab.sh --clean      # Stop and remove all volumes

Volume Management and Data Persistence

Named Volumes

Named volumes persist data across container restarts. They are defined in each compose file.

Main Stack Volumes

Volume Container Purpose
ds-root-data ds-root 389 DS data (Root CA LDAP)
ds-intermediate-data ds-intermediate 389 DS data (Intermediate CA LDAP)
ds-iot-data ds-iot 389 DS data (IoT CA LDAP)
pki-root-data dogtag-root-ca PKI instance data (/var/lib/pki)
pki-root-logs dogtag-root-ca PKI logs (/var/log/pki)
pki-intermediate-data dogtag-intermediate-ca PKI instance data
pki-intermediate-logs dogtag-intermediate-ca PKI logs
pki-iot-data dogtag-iot-ca PKI instance data
pki-iot-logs dogtag-iot-ca PKI logs
freeipa-data freeipa FreeIPA server data
zookeeper-data zookeeper ZooKeeper state
zookeeper-log zookeeper ZooKeeper transaction logs
kafka-data kafka Kafka message data
postgres-data postgres PostgreSQL database
redis-data redis Valkey/Redis cache
awx-projects awx-web, awx-task AWX project files
prometheus-data prometheus Prometheus TSDB
grafana-data grafana Grafana dashboards and state
loki-data loki Loki log chunks

RSA PKI Volumes (pki-compose.yml)

Additional volumes for ACME RA, EST RA, OCSP, and KRA follow the pattern: pki-{acme,est,ocsp,kra}-{data,logs} and ds-{ocsp,kra}-data.

ECC / PQ PKI Volumes

Follow the same pattern with ecc- or pq- prefix: pki-ecc-root-data, ds-pq-intermediate-data, etc.

Bind Mounts

Several directories are bind-mounted from the host for configuration and data sharing:

Host Path Container Path Mode Purpose
./configs/pki/ /etc/pki-configs ro PKI configuration files
./scripts/pki/ /scripts ro Initialization and helper scripts
./data/certs/ /certs rw Certificate output (shared across CAs)
./data/certs/ecc/ /certs rw ECC certificate output
./data/certs/pq/ /certs rw PQ certificate output
./ansible/rulebooks/ /rulebooks ro EDA rulebook definitions
./ansible/playbooks/ /playbooks ro Ansible playbooks
./data/logs/ /opt/cert-lab/logs rw EDA execution logs
./data/eda-ssh/ /app/.ssh ro SSH keys for EDA bridge
./configs/prometheus/ /etc/prometheus/ ro Prometheus configuration
./configs/grafana/ /etc/grafana/provisioning/ ro Grafana auto-provisioning

Clean Restart

To remove all volumes and start fresh:

./stop-lab.sh --clean
# or manually:
podman-compose down -v
sudo podman-compose -f pki-compose.yml down -v
sudo podman-compose -f pki-ecc-compose.yml down -v
sudo podman-compose -f pki-pq-compose.yml down -v
sudo podman-compose -f freeipa-compose.yml down -v

Healthcheck Patterns

The lab uses several healthcheck patterns depending on the service type.

Dogtag PKI CAs

CAs check the Dogtag status REST endpoint:

healthcheck:
  test: ["CMD-SHELL", "curl -sk https://localhost:8443/ca/admin/ca/getStatus 2>/dev/null | grep -q running || exit 1"]
  interval: 10s
  timeout: 10s
  retries: 5
  start_period: 120s    # CAs need time for pkispawn initialization

Note: CA healthchecks show "unhealthy" until pkispawn completes initialization. The start_period: 120s allows time for this without triggering restarts.

OCSP Responders

healthcheck:
  test: ["CMD-SHELL", "curl -sk https://localhost:8443/ocsp/admin/ocsp/getStatus 2>/dev/null | grep -q running || exit 1"]

EST Registration Authorities

EST RAs check for the CA certificates endpoint:

healthcheck:
  test: ["CMD-SHELL", "curl -sk https://localhost:8443/.well-known/est/cacerts 2>/dev/null | head -1 | grep -qE 'BEGIN|MIIB|MIIC|MIID' || exit 1"]

ACME Registration Authorities

ACME RAs check the ACME directory:

healthcheck:
  test: ["CMD-SHELL", "curl -sk https://localhost:8443/acme/directory 2>/dev/null | grep -q newNonce || exit 1"]

389 Directory Server

DS instances check using the built-in health command or LDAP query:

healthcheck:
  test: ["CMD-SHELL", "/usr/libexec/dirsrv/dscontainer -H || ldapsearch -x -H ldap://localhost:3389 -b '' -s base > /dev/null 2>&1"]
  interval: 10s
  timeout: 10s
  retries: 10
  start_period: 120s

FastAPI Services (EDR, SIEM, CT Log, IoT Client)

Python-based services check the /health endpoint and verify Kafka connectivity:

healthcheck:
  test: ["CMD-SHELL", "python -c \"import json, urllib.request; d=json.load(urllib.request.urlopen('http://localhost:8000/health')); exit(0 if d.get('kafka_connected') else 1)\""]
  interval: 10s
  timeout: 10s
  retries: 5
  start_period: 60s

Kafka and ZooKeeper

# ZooKeeper
healthcheck:
  test: ["CMD", "nc", "-z", "localhost", "2181"]

# Kafka
healthcheck:
  test: ["CMD", "kafka-topics", "--bootstrap-server", "localhost:9092", "--list"]

Monitoring (Prometheus, Grafana, Loki)

# Prometheus
healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://localhost:9090/-/healthy || exit 1"]

# Grafana
healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://localhost:3000/api/health || exit 1"]

# Loki
healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://localhost:3100/ready || exit 1"]

Valkey/Redis

healthcheck:
  test: ["CMD", "valkey-cli", "ping"]

Known Limitation

podman-compose may not fully honor condition: service_healthy in depends_on directives. The start-lab.sh script provides its own DS readiness checks (LDAP probes) as a safety net, and PKI init scripts call wait_for_ds() internally before running pkispawn.

See Also

Home

Getting Started
Architecture
PKI Operations
Security & Compliance
Operations & Testing
Key & HSM Management
Monitoring
Advanced Topics

Clone this wiki locally