adding villar light curve fitting using batch GPU#426
Closed
frenbox wants to merge 49 commits into
Closed
Conversation
Introduces a runtime-configurable GPU/CPU worker mode: - gpu.enabled=true: A single GPU worker owns all ONNX models (ACAI, BTSBot) and processes batched inference requests via Redis. Enrichment workers become CPU-only, pushing pre-extracted features to the GPU queue and polling for results. This avoids CUDA context contention between threads. - gpu.enabled=false (default): No behavioral change. Enrichment workers load and run ONNX models inline as before. New files: - src/gpu/mod.rs, src/gpu/worker.rs: GPU worker implementation with batched inference, configurable batch size and timeout Modified: - src/conf.rs: GpuConfig struct (enabled, batch_size, batch_timeout_ms) - src/utils/worker.rs: WorkerType::Gpu variant - src/scheduler/base.rs: GPU worker spawn path - src/bin/scheduler.rs: Conditional GPU worker pool (exactly 1 worker) - src/enrichment/ztf.rs: Refactored to support both CPU-inline and GPU-deferred classification paths - config.yaml: gpu section with defaults Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
33 tests covering: - GpuConfig deserialization and defaults (4 tests) - GpuInferenceRequest/Response JSON roundtrip (5 tests) - Queue naming and result key helpers (2 tests) - ACAI metadata extraction with missing fields (7 tests) - BTSBot metadata extraction with derived features (4 tests) - Cross-extractor consistency (1 test) - GPU request construction from extracted metadata (1 test) - WorkerType::Gpu display and copy (2 tests) - WorkerCmd display (1 test) - should_terminate signal handling (3 tests) All tests are pure unit tests — no MongoDB, Redis, or ONNX required. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GpuConfig gains `device_id: i32` (default 0) for multi-GPU nodes - load_model_on_device(path, device_id) passes device_id to CUDAExecutionProvider::default().with_device_id(dev) - AcaiModel::new_on_device / BtsBotModel::new_on_device constructors - GPU worker reads device_id from config and loads all models on that device - Existing load_model(path) unchanged (device 0, backwards compatible) - config.yaml documents device_id field Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Collaborator
Author
Collaborator
Collaborator
Author
|
I have to name them properly. That part is missing :). I spent the whole day figuring out various CUDA version and how it interacts with ONNX. I will get that done tomorrow morning. In short, villar fit params represent the 7 parameter for g and r band for this equation
Reference: paper You have signature rise time, fall time, peak magnitude, etc (in a normalized space .... will need to change in back properly. So final touchup remains) |
Collaborator
Author
Collaborator
Author
…'s gpu.enabled value; rename the gpu docker compose file to compose.cuda.yaml; add a cuda overwrite for compose in throughput test, use it in _run.sh if platform is linux and gpu is enabled; accept more truthy values
Co-authored-by: Pete Bachant <pbachant@caltech.edu>
…and disable CPU fallback only on Linux
…script and binary
….enabled from config
…throughput script's GPU validation wait logic
…to gpu-setup-fix
…ng (boom-astro#432) Upgrading `ort` to rc.12 and adding `load-dynamic` caused `ort-sys` to download `libonnxruntime.so` at **test runtime** (~20-28 min) when `ORT_DYLIB_PATH` is unset, hitting the 30-minute CI timeout on every run. Additionally, the throughput test workflow was failing before any jobs started due to an invalid expression in the workflow-level `env:` block. ## Fix ### Unit tests (`test.yaml`) Added a CI step before testing that installs the CPU `onnxruntime==1.24.4` Python wheel and exports `ORT_DYLIB_PATH` pointing to the installed library, so `ort-sys` resolves it immediately instead of downloading. Also pins `BOOM_GPU__ENABLED=false` as a workflow env var to prevent CUDA EP initialization on CPU-only runners. ```yaml - name: Install ONNX Runtime (CPU) for tests run: | pip install onnxruntime==1.24.4 ORT_CAPI=$(python3 -c "import pathlib, onnxruntime; print((pathlib.Path(onnxruntime.__file__).parent / 'capi').as_posix())") ORT_LIB=$(ls "${ORT_CAPI}"/libonnxruntime.so.* 2>/dev/null | head -1) echo "ORT_DYLIB_PATH=${ORT_LIB}" >> "$GITHUB_ENV" echo "BOOM_GPU__ENABLED=false" >> "$GITHUB_ENV" ``` ### Throughput test (`test-throughput.yaml`) The workflow had `BOOM_GPU__ENABLED: ${{ env.BOOM_GPU__ENABLED || 'false' }}` in the workflow-level `env:` block. GitHub Actions does not allow referencing the `env` context at that scope — this caused a workflow evaluation error before any jobs started (runs showed `failure` with 0 jobs) and silently suppressed `pull_request` event triggers for subsequent commits. Fixed by replacing it with a literal `'false'`, since CI runners never have GPU hardware. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Theodlz <49785117+Theodlz@users.noreply.github.com>
… a Dockerfile.gpu that only ships scheduler+migrations but in a CUDA-enabled env
adding NVIDIA toolkit for docker added docker arg for gpu based build working docker system adding GPU docker image added cfg to avoid gpu based fits for cpu only build made final changes added keys to villar output
Collaborator
Author
|
Another PR was created (#448). Closing this. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.





Trying to fit lightcurves to Villar model as a batch using GPU for ZTF sources