Skip to content

adding villar light curve fitting using batch GPU#426

Closed
frenbox wants to merge 49 commits into
boom-astro:mainfrom
frenbox:villar_pso_gpu
Closed

adding villar light curve fitting using batch GPU#426
frenbox wants to merge 49 commits into
boom-astro:mainfrom
frenbox:villar_pso_gpu

Conversation

@frenbox

@frenbox frenbox commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator

Trying to fit lightcurves to Villar model as a batch using GPU for ZTF sources

mcoughlin and others added 22 commits March 13, 2026 11:51
Introduces a runtime-configurable GPU/CPU worker mode:

- gpu.enabled=true: A single GPU worker owns all ONNX models (ACAI, BTSBot)
  and processes batched inference requests via Redis. Enrichment workers
  become CPU-only, pushing pre-extracted features to the GPU queue and
  polling for results. This avoids CUDA context contention between threads.

- gpu.enabled=false (default): No behavioral change. Enrichment workers
  load and run ONNX models inline as before.

New files:
- src/gpu/mod.rs, src/gpu/worker.rs: GPU worker implementation with
  batched inference, configurable batch size and timeout

Modified:
- src/conf.rs: GpuConfig struct (enabled, batch_size, batch_timeout_ms)
- src/utils/worker.rs: WorkerType::Gpu variant
- src/scheduler/base.rs: GPU worker spawn path
- src/bin/scheduler.rs: Conditional GPU worker pool (exactly 1 worker)
- src/enrichment/ztf.rs: Refactored to support both CPU-inline and
  GPU-deferred classification paths
- config.yaml: gpu section with defaults

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
33 tests covering:
- GpuConfig deserialization and defaults (4 tests)
- GpuInferenceRequest/Response JSON roundtrip (5 tests)
- Queue naming and result key helpers (2 tests)
- ACAI metadata extraction with missing fields (7 tests)
- BTSBot metadata extraction with derived features (4 tests)
- Cross-extractor consistency (1 test)
- GPU request construction from extracted metadata (1 test)
- WorkerType::Gpu display and copy (2 tests)
- WorkerCmd display (1 test)
- should_terminate signal handling (3 tests)

All tests are pure unit tests — no MongoDB, Redis, or ONNX required.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GpuConfig gains `device_id: i32` (default 0) for multi-GPU nodes
- load_model_on_device(path, device_id) passes device_id to
  CUDAExecutionProvider::default().with_device_id(dev)
- AcaiModel::new_on_device / BtsBotModel::new_on_device constructors
- GPU worker reads device_id from config and loads all models on that device
- Existing load_model(path) unchanged (device 0, backwards compatible)
- config.yaml documents device_id field

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@frenbox

frenbox commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator Author

Build and test the GPU version

docker build --build-arg USE_GPU=true --target dev -t boom-dev --no-cache .
docker compose --profile api up -d

docker run --rm -it --gpus all --network boom_boom   -v $(pwd)/config.yaml:/app/config.yaml   -v $(pwd)/data:/app/data   -v $(pwd)/.env:/app/.env   boom-dev bash

Inside bash
(the features gpu is not really necessary for producer and consumer)

cargo run --release --features gpu --bin kafka_producer ztf 20240617 public --server-url broker:29092

cargo run --release --features gpu --bin kafka_consumer ztf 20240617 --programids public

cargo run --release --features gpu --bin scheduler ztf --config config.yaml

Example of an output

image

@Theodlz

Theodlz commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator

Build and test the GPU version

docker build --build-arg USE_GPU=true --target dev -t boom-dev --no-cache .
docker compose --profile api up -d

docker run --rm -it --gpus all --network boom_boom   -v $(pwd)/config.docker.yaml:/app/config.yaml   -v $(pwd)/data:/app/data   -v $(pwd)/.env:/app/.env   boom-dev-gpu bash

Inside bash (the features gpu is not really necessary for producer and consumer)

cargo run --release --features gpu --bin kafka_producer ztf 20240617 public --server-url broker:29092

cargo run --release --features gpu --bin kafka_consumer ztf 20240617 --programids public

cargo run --release --features gpu --bin scheduler ztf --config config.yaml

Example of an output

image

How does one use the villar_fit array to select candidates?

@frenbox

frenbox commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator Author

I have to name them properly. That part is missing :). I spent the whole day figuring out various CUDA version and how it interacts with ONNX. I will get that done tomorrow morning.

In short, villar fit params represent the 7 parameter for g and r band for this equation

image

Reference: paper

You have signature rise time, fall time, peak magnitude, etc (in a normalized space .... will need to change in back properly. So final touchup remains)

@frenbox

frenbox commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator Author

Build and test the CPU on;y version

docker build --build-arg --target dev -t boom-dev --no-cache .
docker compose --profile api up -d

docker run --rm -it  --network boom_boom   -v $(pwd)/config.docker.yaml:/app/config.yaml   -v $(pwd)/data:/app/data   -v $(pwd)/.env:/app/.env   boom-dev bash

Inside bash

cargo run --release  --bin kafka_producer ztf 20240617 public --server-url broker:29092

cargo run --release --bin kafka_consumer ztf 20240617 --programids public

cargo run --release  --bin scheduler ztf --config config.yaml

Example of an output

image

@frenbox

frenbox commented Apr 3, 2026

Copy link
Copy Markdown
Collaborator Author
image

Theodlz and others added 23 commits April 3, 2026 16:19
…'s gpu.enabled value; rename the gpu docker compose file to compose.cuda.yaml; add a cuda overwrite for compose in throughput test, use it in _run.sh if platform is linux and gpu is enabled; accept more truthy values
Co-authored-by: Pete Bachant <pbachant@caltech.edu>
…throughput script's GPU validation wait logic
…ng (boom-astro#432)

Upgrading `ort` to rc.12 and adding `load-dynamic` caused `ort-sys` to
download `libonnxruntime.so` at **test runtime** (~20-28 min) when
`ORT_DYLIB_PATH` is unset, hitting the 30-minute CI timeout on every
run. Additionally, the throughput test workflow was failing before any
jobs started due to an invalid expression in the workflow-level `env:`
block.

## Fix

### Unit tests (`test.yaml`)

Added a CI step before testing that installs the CPU
`onnxruntime==1.24.4` Python wheel and exports `ORT_DYLIB_PATH` pointing
to the installed library, so `ort-sys` resolves it immediately instead
of downloading. Also pins `BOOM_GPU__ENABLED=false` as a workflow env
var to prevent CUDA EP initialization on CPU-only runners.

```yaml
- name: Install ONNX Runtime (CPU) for tests
  run: |
    pip install onnxruntime==1.24.4
    ORT_CAPI=$(python3 -c "import pathlib, onnxruntime; print((pathlib.Path(onnxruntime.__file__).parent / 'capi').as_posix())")
    ORT_LIB=$(ls "${ORT_CAPI}"/libonnxruntime.so.* 2>/dev/null | head -1)
    echo "ORT_DYLIB_PATH=${ORT_LIB}" >> "$GITHUB_ENV"
    echo "BOOM_GPU__ENABLED=false" >> "$GITHUB_ENV"
```

### Throughput test (`test-throughput.yaml`)

The workflow had `BOOM_GPU__ENABLED: ${{ env.BOOM_GPU__ENABLED ||
'false' }}` in the workflow-level `env:` block. GitHub Actions does not
allow referencing the `env` context at that scope — this caused a
workflow evaluation error before any jobs started (runs showed `failure`
with 0 jobs) and silently suppressed `pull_request` event triggers for
subsequent commits. Fixed by replacing it with a literal `'false'`,
since CI runners never have GPU hardware.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Theodlz <49785117+Theodlz@users.noreply.github.com>
… a Dockerfile.gpu that only ships scheduler+migrations but in a CUDA-enabled env
adding NVIDIA toolkit for docker

added docker arg for gpu based build

working docker system

adding GPU docker image

added cfg to avoid gpu based fits for cpu only build

made final changes

added keys to villar output
@frenbox

frenbox commented Apr 24, 2026

Copy link
Copy Markdown
Collaborator Author

Another PR was created (#448). Closing this.

@frenbox frenbox closed this Apr 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants