Docker image request

[menciuschow]

Can you make a docker image?

[davide221]

Yes, we can integrate it! Added to the feature list

[jkyamog]

I have been using it with Proxmox (LXC) for a while. I can tidy it up and remove my specific setup, need guidance where you want me to put this. Here is what I have.

Dockerfile

ARG UBUNTU_VERSION=24.04
ARG CUDA_VERSION=13.0.1
ARG LUCEBOX_REPO=https://github.com/Luce-Org/lucebox-hub
ARG LUCEBOX_REF=main

FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu${UBUNTU_VERSION} AS build-base

ENV DEBIAN_FRONTEND=noninteractive \
    PIP_NO_CACHE_DIR=1 \
    PYTHONUNBUFFERED=1

RUN apt-get update && apt-get install -y \
    git \
    git-lfs \
    cmake \
    ninja-build \
    build-essential \
    python3 \
    python3-venv \
    python3-pip \
    curl \
    ca-certificates \
    libgomp1 \
    && rm -rf /var/lib/apt/lists/*

FROM build-base AS source
ARG LUCEBOX_REPO
ARG LUCEBOX_REF

WORKDIR /app
RUN set -eux; \
    git clone --recurse-submodules "${LUCEBOX_REPO}" /app/lucebox-hub; \
    cd /app/lucebox-hub; \
    git checkout "${LUCEBOX_REF}"; \
    git submodule update --init --recursive

FROM build-base AS pydeps

RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"

RUN pip install --upgrade pip && \
    pip install fastapi uvicorn transformers gguf jinja2 huggingface_hub torch torchvision

FROM build-base AS cppbuild

WORKDIR /app/lucebox-hub/dflash
COPY --from=source /app/lucebox-hub /app/lucebox-hub

RUN cmake -B build -S . -G Ninja \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CUDA_ARCHITECTURES=86 \
      -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined \
      -DCMAKE_SHARED_LINKER_FLAGS=-Wl,--allow-shlib-undefined && \
    cmake --build build --target test_dflash -j"$(nproc)"

FROM nvidia/cuda:${CUDA_VERSION}-runtime-ubuntu${UBUNTU_VERSION} AS runtime

ENV DEBIAN_FRONTEND=noninteractive \
    PIP_NO_CACHE_DIR=1 \
    PYTHONUNBUFFERED=1 \
    DFLASH_TARGET=/models/Qwen3.6-27B-Q4_K_M.gguf \
    DFLASH_DRAFT=/models/draft/model.safetensors \
    DFLASH27B_KV_TQ3=1 \
    DFLASH27B_PREFILL_UBATCH=16

RUN apt-get update && apt-get install -y \
    python3 \
    python3-venv \
    python3-pip \
    curl \
    ca-certificates \
    libgomp1 \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app/lucebox-hub/dflash

COPY --from=pydeps /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"

COPY --from=cppbuild /app/lucebox-hub /app/lucebox-hub

EXPOSE 8000

CMD ["python3", "scripts/server.py", "--port", "8000", "--daemon", "--max-ctx", "65536", "--budget", "16", "--fa-window", "0"]

docker-compose.yaml

services:
  dflash:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        LUCEBOX_REPO: https://github.com/jkyamog/lucebox-hub.git
        LUCEBOX_REF: fix-reasoning-stream
    container_name: dflash
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ./models:/models
      - ./models/draft:/app/lucebox-hub/dflash/models/draft
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
      DFLASH_TARGET: /models/Qwen3.6-27B-Q4_K_M.gguf
      DFLASH_DRAFT: /models/draft/model.safetensors
      DFLASH27B_KV_K: "q4_0"
      DFLASH27B_KV_V: "q4_0"
      # DFLASH27B_PREFILL_UBATCH: "8"
    gpus: all
    shm_size: "1gb"
    ulimits:
      memlock: -1
      stack: 67108864
    restart: unless-stopped
    networks:
      - internal-network

networks:
  internal-network:
    external: true      

[jkyamog]

btw DFLASH_DRAFT doesn't seem to honored, so I had to mount to where it looking for the draft model. ./models/draft:/app/lucebox-hub/dflash/models/draft. Maybe this needs to be checked as well. This is modeled after llama.cpp docker file and I just run them on docker compose in particular having this internal-network (my poor man K8s flat pod network).

[davide221]

Closing, considering #334