More detailed documentation is available at https://pub.current.posit.team/public/shiny-remote-hpc/

Turbocharge your Shiny Apps with Remote HPC Submission

This repository demonstrates how to offload compute-intensive calculations from Shiny applications to remote High Performance Computing (HPC) clusters. By leveraging HPC resources, you can maintain app responsiveness while handling demanding computational workloads.

Overview

Modern Shiny applications often need to perform compute-intensive calculations on demand. However, the hosting environment (local laptop, Posit Connect, Shiny Server) typically has limited resources. This project shows how to:

• Offload heavy computations to dedicated HPC clusters
• Maintain Shiny app interactivity and stability
• Manage package dependencies across different environments
• Implement both tight and loose integration patterns

Repository Structure

.
├── app-0/          # Basic local parallel example using clustermq
├── app-1/          # Loose integration with SSH (clustermq)
├── app-2/          # Tight integration example (clustermq)
├── app-3/          # SSH integration using mirai
├── app-4/          # Additional mirai examples
├── Offload.qmd     # Detailed documentation (Quarto)
└── README.md       # This file

Integration Patterns

Tight Integration

The Shiny app runs on an environment with direct access to the HPC scheduler (e.g., on a login or service node). This eliminates the need for SSH but requires more infrastructure setup.

Loose Integration

The Shiny app interacts with the HPC cluster via SSH connections to login nodes and submits jobs remotely. More flexible but requires SSH configuration.

Prerequisites

Software Requirements

• R (version 4.5+ recommended)
• SSH client (for loose integration)
• Access to an HPC cluster with SLURM scheduler

R Packages

• shiny
• clustermq or mirai (for parallel execution)
• renv (for package management)
• pak (installed automatically on HPC)

Quick Start

1. Clone the Repository

git clone https://github.com/sol-eng/shiny-hpc-offload.git
cd shiny-hpc-offload

2. Try the Local Example

# Navigate to app-0 directory
setwd("app-0")

# Initialize renv
renv::init()

# Run the app
shiny::runApp()

This example uses the Palmer Penguins dataset and runs GLM bootstrap calculations locally.

3. Set Up SSH Access (for Loose Integration)

Generate SSH key:

ssh-keygen -N '' -t rsa -f ~/.ssh/my_hpc

Copy public key to HPC:

ssh-copy-id -i ~/.ssh/my_hpc -p 443 username@your-hpc-hostname

Test connection:

ssh -i ~/.ssh/my_hpc -p 443 username@your-hpc-hostname

4. Configure HPC Integration

Edit the SSH template (app-1/ssh.tmpl) and app configuration to match your HPC environment:

options(
  clustermq.scheduler = "ssh",
  clustermq.ssh.host = "username@your-hpc-hostname",
  clustermq.ssh.log = "~/cmq_ssh.log",
  clustermq.template = "path/to/ssh.tmpl"
)

5. Install clustermq on HPC

On the remote HPC cluster:

install.packages("clustermq")

Key Features

Automatic Package Management

The hpc_setup() and init() functions automatically:

• Detect locally installed packages and versions
• Create isolated package directories on HPC (~/.shiny-<uuid>)
• Install matching package versions using pak
• Prevent interference with other HPC workflows

UUID-Based Isolation

Each app deployment gets a unique identifier, ensuring:

• Package version consistency
• No conflicts between different apps
• Clean separation of environments

Usage Examples

Using clustermq (Loose Integration)

library(clustermq)

# Configure SSH backend
options(
  clustermq.scheduler = "ssh",
  clustermq.ssh.host = "user@hpc-host"
)

# Run parallel computation
results <- Q(
  fun = my_function,
  x = 1:100,
  n_jobs = 10
)

Using mirai (SSH Integration)

library(mirai)

# Create SSH configuration
ssh_cfg <- ssh_config(
  remotes = "ssh://hpc-host",
  tunnel = TRUE,
  rscript = "/path/to/Rscript"
)

# Launch daemons
daemons(n = 4, url = "tcp://127.0.0.1:0", remote = ssh_cfg)

# Execute remote computation
m <- mirai({ mean(rnorm(1e7)) })
m[]

Troubleshooting

Connection Timeout

Check the cmd_ssh.log file in your home directory on the HPC cluster. Common issues:

• Missing clustermq package on HPC
• Incorrect SSH configuration
• Firewall blocking connections

Package Version Mismatches

Ensure R versions are similar on both Shiny server and HPC. The automatic package management handles version matching.

SLURM Job Failures

Check SLURM logs and ensure:

• SLURM binaries are in $PATH
• Resource requests are appropriate
• Queue/partition names are correct

Documentation

For detailed explanations, architecture diagrams, and step-by-step tutorials, see the Quarto documentation.

To render the documentation:

quarto::quarto_render("Offload.qmd")