Turn conda environments into optimized container images for development and HPC deployment.
Absconda bridges the gap between conda's reproducible environments and container-based workflows. Define your scientific computing environment once with conda, then deploy it anywhere—Docker, Singularity/Apptainer, HPC clusters—with production-ready optimizations.
🚀 Multi-Stage Builds - Automatic optimization that reduces image sizes by 40-60%
🔐 Policy Validation - Enforce security and compliance rules organization-wide
🏗️ Remote Builders - Offload builds to cloud instances with automatic provisioning
🧪 HPC Integration - Singularity/Apptainer support with module files and wrappers
📦 R + renv Support - Combine conda environments with R package management
🎯 Flexible Deployment - Multiple modes: full-env, tarball, requirements, export-explicit
🔧 Custom Templates - Jinja2-based system for advanced customization
# Install from PyPI
pip install absconda
# Or with pipx (recommended)
pipx install absconda
# Verify installation
absconda --version1. Create a conda environment file:
# environment.yaml
name: my-analysis
channels:
- conda-forge
- bioconda
dependencies:
- python=3.11
- numpy=1.26
- pandas=2.1
- scikit-learn=1.32. Build a Docker image:
absconda build \
--file environment.yaml \
--repository ghcr.io/myorg/my-analysis \
--tag latest \
--push3. Use the image:
docker run --rm ghcr.io/myorg/my-analysis:latest python -c "import numpy; print(numpy.__version__)"For HPC with Singularity:
# Build, push, pull SIF, and generate wrappers + module in one step
absconda deploy \
--file environment.yaml \
--repository ghcr.io/myorg/my-analysis \
--tag latest \
--commands python,pip,jupyter
# Or build/push first, then deploy from the image reference
absconda publish \
--file environment.yaml \
--repository ghcr.io/myorg/my-analysis \
--tag latest
absconda deploy ghcr.io/myorg/my-analysis:latest \
--commands python,pip,jupyterSee the Quick Start Guide for a complete walkthrough.
- Installation - Get up and running
- Quick Start - 5-minute tutorial
- Core Concepts - Understanding absconda
- Basic Usage - Essential workflows
- Building Images - Docker build process
- HPC Deployment - Singularity on HPC clusters
- Remote Builders - Cloud-based builds
- R + renv Integration - Combining conda and renv
- Requirements Mode - Deployment mode comparison
- Advanced Templating - Custom Dockerfiles
- Multi-Stage Builds - Optimize image size
- Custom Base Images - GPU and specialized bases
- Secrets Management - Handle credentials safely
- CI/CD Integration - Automate with GitHub Actions
Complete working examples with explanations:
- Minimal Python - Simple Python environment
- Data Science Stack - NumPy/pandas/scikit-learn
- R + Bioconductor - RNA-seq analysis
- GPU + PyTorch - Deep learning with CUDA
- HPC Workflow - Complete HPC deployment
- CLI Reference - Complete command documentation
- Environment Files - YAML specification
- Configuration - System configuration
- Policies - Policy validation system
- Contributing - How to contribute
- Testing - Running and writing tests
- Architecture - Technical design documentation
Scientific computing has conflicting requirements:
- Reproducibility: Need exact package versions
- Portability: Must run on laptops, HPC clusters, cloud
- Performance: Large conda environments create huge containers
- HPC Reality: Singularity/Apptainer, not Docker; module systems, not Docker Compose
Absconda solves this by:
- Starting with conda - Use the ecosystem you already know
- Optimizing automatically - Multi-stage builds reduce image sizes by 40-60%
- Targeting HPC - Native Singularity support with modules and wrappers
- Enforcing policies - Organization-wide security and compliance
- Enabling remote builds - Build on powerful cloud instances, not your laptop
| Feature | Absconda | repo2docker | docker-conda | Manual Dockerfile |
|---|---|---|---|---|
| Multi-stage optimization | ✅ Automatic | ❌ No | ❌ No | |
| Singularity integration | ✅ Built-in | ❌ No | ❌ No | |
| HPC modules | ✅ Built-in | ❌ No | ❌ No | |
| Policy enforcement | ✅ Yes | ❌ No | ❌ No | ❌ No |
| Remote builders | ✅ Yes | ❌ No | ❌ No | ❌ No |
| R + renv support | ✅ Yes | ❌ No | ||
| Custom templates | ✅ Jinja2 | ❌ No | ❌ No | ✅ Full control |
Complete workflow for deploying to NCI Gadi HPC:
# 1. Build optimized Docker image (locally or on GCP)
absconda build \
--file rnaseq-env.yaml \
--repository ghcr.io/lab/rnaseq \
--tag v1.0 \
--remote-builder gcp-builder \
--push
# 2. Deploy to HPC: pull SIF, generate wrappers and module
ssh gadi.nci.org.au
absconda deploy ghcr.io/lab/rnaseq:v1.0 \
--commands python,pip \
--image-cache /apps/rnaseq/v1.0 \
--output-dir /apps/rnaseq/v1.0/wrappers \
--module-dir /apps/Modules/modulefiles
# 3. Use in PBS job
module load rnaseq/v1.0
python analysis.py # Uses containerized environment transparentlyThe result: reproducible, optimized, compliant environments deployed consistently from development through production.
- Reproducible analysis pipelines
- Sharing environments with collaborators
- Publishing with computational papers
- Archive environments for long-term reproducibility
- Centralized environment management
- Policy enforcement across teams
- Module system integration
- Singularity deployment at scale
- Build in cloud (fast, powerful VMs)
- Deploy to HPC (Singularity)
- Consistent environments across platforms
- Cost-optimized with remote builders
- Bioconda + R/Bioconductor workflows
- GPU-accelerated analysis
- Large-scale genomics pipelines
- Compliance with data policies
Absconda is production-ready and actively maintained. It powers scientific computing workflows for research teams at the Garvan Institute and beyond.
Current version: 0.2.5
Python support: 3.10, 3.11, 3.12, 3.13+
License: MIT
- Documentation: docs/
- Issues: GitHub Issues
- Discussions: GitHub Discussions
Contributions welcome! See Contributing Guide.
Areas needing help:
- Documentation improvements
- Example workflows
- Testing on different platforms
- Feature requests and feedback
Developed at the Garvan Institute of Medical Research by the Swarbrick Lab.
Built with:
- micromamba - Fast conda package manager
- Jinja2 - Template engine
- Singularity/Apptainer - HPC containers
- Terraform - Infrastructure as code
MIT License - see LICENSE for details.
Ready to get started? → Installation Guide
Have questions? → GitHub Discussions