PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

This repository contains the code artifact for the SenSys '26 paper:

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

Paper link: https://doi.org/10.1145/3774906.3802783

PELM targets power-efficient Large Language Model (LLM) inference on mobile and edge platforms, where sustained generation is constrained not only by compute capacity but also by power and thermal limits. Conventional DVFS methods adjust processor frequencies, but they leave the LLM workload itself unchanged. PELM expands this optimization space by jointly controlling hardware frequency and decoding-side computation.

The key observation is that not all generated tokens require full-depth inference to preserve generation quality. PELM therefore combines three knobs: CPU/GPU DVFS, speculative decoding, and dynamic verification depth. A lightweight online controller selects these settings according to runtime speed, utilization, power, and temperature feedback. Across the evaluated models, datasets, and hardware platforms, PELM improves on prior power governing methods with up to 23.1% speedup and 52.4% energy reduction while maintaining comparable task performance.

System Requirements

The artifact was tested with the following software stack:

• NVIDIA Jetson AGX Orin Developer Kit
• JetPack 6.2.1 / Jetson Linux R36.4.4 / Ubuntu 22.04.5 (aarch64)
• CUDA 12.6 / cuDNN 9.3
• Python 3.10.12
• PyTorch 2.8.0
• Transformers 4.53.2
• Datasets 4.0.0
• tqdm 4.67.1
• protobuf 6.31.1
• sentencepiece 0.2.1

Environment Setup

Install Docker Engine and the NVIDIA Container Toolkit on the Jetson. The NVIDIA Container Toolkit installation guide contains the runtime installation instructions. After installing the packages, configure and restart Docker:

sudo apt-get update
sudo apt-get install -y docker.io nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
sudo usermod -aG docker "$USER"

Log out and back in after changing Docker group membership. Then follow DOCKER_BUILD.md to build the PELM image with the tested dependency versions. After the build completes, set PELM_IMAGE to the full image name and tag created by the build:

export PELM_IMAGE="<your-built-image-name>:<your-built-image-tag>"

Choose an absolute host directory for downloaded model weights and datasets (the persistent model/dataset cache), and locate the cloned project. Replace both placeholder paths below with paths on your machine:

mkdir -p /path/to/model-and-dataset-cache
cd /path/to/PELM

Run Experiment

1. Select a DVFS nvpmodel mode

Edit /etc/nvpmodel.conf directly and add a custom mode (for example, ID 4 named DVFS) with the required CPU/GPU frequency limits. The tested mode keeps all 12 cores online and uses these limits:

< POWER_MODEL ID=4 NAME=DVFS >
CPU_A78_0 MIN_FREQ 115200
CPU_A78_0 MAX_FREQ -1
CPU_A78_1 MIN_FREQ 115200
CPU_A78_1 MAX_FREQ -1
CPU_A78_2 MIN_FREQ 115200
CPU_A78_2 MAX_FREQ -1
GPU MIN_FREQ 0
GPU MAX_FREQ -1

Keep the mode's other Orin constraints, such as all CPU_ONLINE entries, TPC, EMC, DLA, and PVA settings. Use an ID that is not already defined, then select and verify the mode:

sudo nvpmodel -m 4
sudo nvpmodel -q --verbose

2. Stop automatic fan control

PELM writes the fan PWM directly, so stop the service before starting the privileged experiment container:

sudo systemctl stop nvfancontrol.service
systemctl is-active nvfancontrol.service  # expected: inactive

3. Start the privileged container

Start the container with the NVIDIA runtime, cache, and project directory:

docker run --rm -it \
  --network host \
  --runtime nvidia \
  --privileged \
  -v "/path/to/model-and-dataset-cache:/cache" \
  -v "/path/to/PELM:/workspace/PELM" \
  -w /workspace/PELM \
  "$PELM_IMAGE" bash

Inside the container, run the experiment:

bash run_experiment.sh

run_experiment.sh temporarily sets /sys/class/thermal/thermal_zone8/policy to user_space and writes the desired 0-255 PWM value to /sys/devices/platform/pwm-fan/hwmon/hwmon0/pwm1. These sysfs paths and the --privileged flag are required for direct PWM/DVFS control. Restore the thermal policy and fan PWM if the script is interrupted. After a normal completion, the script restores both settings itself.

After the experiment, restart nvfancontrol.service if it was originally active:

sudo systemctl start nvfancontrol.service

Note

Adapting to another Jetson model

The current settings target the tested 12-core AGX Orin (3 CPU clusters, 4 cores each). Other models, including Jetson Nano variants, may have different cluster layouts, valid CPU/GPU frequency ranges, governors, and sysfs paths for temperature, frequency, power, thermal policy, and fan control. The frequency values shown above are specific to the tested AGX Orin; use frequency values supported by the target device. Check these locations before running:

• /etc/nvpmodel.conf: replace the Part 1 CPU_A78_0..2 and CPU/GPU MIN_FREQ/MAX_FREQ fields with names and ranges supported by the target.
• run_experiment.sh lines 21-32 and 55-67: adapt the fan PWM and thermal-zone temp/policy paths, cooling threshold, PWM values, and policy names.
• utils/shared.py lines 3-15 and utils/Manager.py lines 29-78, 89-214, and 288-325: adapt CPU/GPU frequency and governor nodes, fan node, INA3221 path/channel mapping, thermal zones, and GPU-utilization node.
• utils/Manager.py lines 29 and 158-166 plus PELM/pelm_dqn.py lines 30-36, 54-83, and 92-150: adapt cores per cluster, representative core IDs (0/4/8), state layout, and the three CPU CSV fields.
• If the CPU cluster count is not three, also adapt the three cpu_f0..2 branches and fixed state indexes in PELM/agent_dqn.py lines 85-102 and 150-295, then match the CPU-frequency summary fields in eval.py lines 311-320.

Setting target speed and temperature

Adjust TARGET_SPEED, WARN_TEMP, and the TEMP_LIMIT hard-stop temperature in PELM/pelm_dqn.py lines 20 and 44-45 for the target model and device. Keep WARN_TEMP consistent with warning_temperature in PELM/agent_dqn.py lines 67-68.

4. Evaluation

Pass either one generated session directory or a parent data directory to the evaluator. It writes results/eval_summary.csv and results/eval_summary.json next to eval.py:

python3 eval.py data/<session-directory>

The evaluator reports system metrics like elapsed time, test sample ID, power, speed, speculation, frequency, temperature, and the dataset-specific (GSM8K, NQ Open, HumanEval, WMT14, and CNN/DM) metrics.

Citation

@inproceedings{10.1145/3774906.3802783,
author = {Yang, Weisi and Xia, Stephen},
title = {PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling},
year = {2026},
isbn = {9798400723094},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3774906.3802783},
doi = {10.1145/3774906.3802783},
booktitle = {Proceedings of the 2026 ACM/IEEE International Conference on Embedded Artificial Intelligence and Sensing Systems},
pages = {438–451},
numpages = {14},
series = {SenSys '26}
}