Skip to content

Support AGX Orin and Thor T5000: core layouts, Jetson tuning, one-command provision - #326

Merged
shawnpatel merged 1 commit into
mainfrom
add-agx-orin-thor-t5000
Sep 24, 2026
Merged

shawnpatel merged 1 commit into
mainfrom
add-agx-orin-thor-t5000

Conversation

@shawnpatel

@shawnpatel shawnpatel commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds AGX Orin (12 cores) and Thor T5000 (14 cores) as compute options alongside Raspberry Pi 5 and Orin NX 16GB. axol provision is now the only setup command a host needs.

Core layout (utils/affinity.py)

Host cores CAN control IK relay recorder camera pool
Raspberry Pi 5 4 2-3 1 1 0 0 –
Orin NX 16GB / AGX Orin 32GB 8 6-7 2 3 4-5 0-1 0-1, 5
AGX Orin 64GB / Industrial 12 10-11 2 3 4-5 8-9 0-1, 5-7
Thor T5000 14 12-13 2 3 4-5 10-11 0-1, 5-9
  • Before this change, 12- and 14-core hosts used the 8-core groups unchanged and left cores 6-9 / 6-11 unassigned.
  • On 12+ cores the two cores below CAN are the recorder's, and no SCHED_FIFO camera thread may run there. On 8 cores, camera threads preempt the recorder on every core it has: it got about 5% of each core on the 2026-09-14 policy ops. That's why it currently borrows the IK core. With a pair of its own it no longer needs to, so share_ik_core has no effect on these hosts. The normal-priority GStreamer threads can still use the recorder cores.
  • The Rust trace writers moved to the recorder cores.
  • The layout is built over the kernel's online CPU list, so a power mode that offlines a CPU in the middle can't produce pins to missing CPUs.
  • The camera pool now avoids whichever CPU the CAN interrupt is actually on (irqbalance, a manual move), not only CPU0.
  • The 8-core layout is unchanged.

Jetson tuning on AGX Orin and Thor (utils/jetson.py)

  • AGX Orin uses the same chip as the Orin NX, so every existing step applies unchanged.
  • Thor has no NVENC clock node. Encode is clocked by gpu-nvd-* and CUDA by gpu-gpc-*, and the *.nvenc / *.gpu patterns missed both. Before this change, a Thor pinned only its VIC. Those two domains now get the performance governor, as NVIDIA's Thor power guide prescribes.
  • Neither the pin nor the runtime check (jetson_diag) touches those nodes' frequency files. Those files take the devfreq lock, and a JetPack 7.2 Thor hang has been seen blocking on it.
  • Thor's USB host is the same tegra-xusb xHCI driver, so the CAN interrupt lookup carries over. MAXN is mode 0 on both boards.

One command: axol provision

  • axol provision applies the per-boot host tuning as its last step. It logs the board and the core layout, e.g. core layout: 14 cores online: CAN 12-13, control 2, IK 3, relay 4-5, recorder 10-11, camera 0-1,5-9.
  • New axol provision --boot runs only the tuning, with no installs and no update lock. The installer's unit now uses it as ExecStartPre and is ordered After=nvpmodel.service; the docs already claimed that ordering. The installer's separate axol jetson.setup step is removed.
  • axol jetson.setup stays as an alias, because units written by older installers call it.
  • The self-updater's provision pass inside axol serve skips the tuning. It can overlap a robot session, and the service restart that ends an update re-tunes the host before serve starts.

Test plan

  • uv run pytest: 1591 passed, 1 skipped
  • ruff check . / ruff format --check . (0.9.7)
  • bash -n web/app/public/install
  • axol provision --boot and axol jetson.setup on a non-Jetson VM: log the host and layout, apply no tuning, exit 0
  • On an AGX Orin 64GB: axol provision logs the 12-core layout; the CAN interrupt lands on CPU 11; nvargus-daemon is confined to the camera cores
  • On a Thor T5000: gpu-gpc-0 / gpu-nvd-0 report performance, VIC is pinned, the CAN interrupt lands on CPU 13, and a recording runs without skipped exposures or CAN faults
  • Reboot both: the ExecStartPre (axol provision --boot) re-applies the tuning

Not covered

  • Thor GPU hang. If the GPU driver is already hung (the JetPack 7.2 bug) before the first governor switch, that write can still block. Then axol serve won't start. NVIDIA's kernel patch is the fix.
  • Raspberry Pi 5. Still gets no host tuning; this isn't a regression.
  • Interrupt moved mid-session. The camera pool is read when the relay starts, so an interrupt moved during a session isn't tracked.

🤖 Generated with Claude Code


Note

High Risk
Changes real-time CPU partitioning, Jetson clock/interrupt tuning, and boot-time systemd hooks on production robot hosts; mis-tuning can affect CAN timing, camera encode latency, and recording throughput.

Overview
axol provision is now the single host setup path: installs stay the same, but the last step applies per-boot real-time tuning (Jetson power mode, engine/CPU clocks, CAN interrupt steering, Argus SCHED_FIFO). axol provision --boot runs only that tuning (no update lock) and replaces axol jetson.setup in the installer’s ExecStartPre; jetson.setup remains as an alias for older units.

CPU affinity gains explicit recorder and camera groups: on 12+ core Jetsons (AGX Orin 64GB, Thor T5000), middle cores join the throughput pool and two cores below CAN are dedicated to the dataset recorder and Rust trace writers, so FIFO camera work no longer starves recording. Layouts use the kernel online CPU list; the camera pool avoids whichever CPU(s) actually carry the CAN USB interrupt, not only CPU0.

Thor (JetPack 7) tuning pins gpu-gpc-* / gpu-nvd-* via the performance devfreq governor (no min_freq on those nodes); runtime pin checks match. Docs, .superset/config.json, and web/app/public/install drop the separate jetson.setup install step and order the unit after nvpmodel.service.

Provisioning skips live host tuning when axol serve is already running (self-updater); the service restart + ExecStartPre re-applies tuning instead.

Reviewed by Cursor Bugbot for commit 832e4a1. Configure here.

…mand provision

Core layout (utils/affinity.py): 12-core AGX Orin 64GB and 14-core Thor
T5000 used the 8-core groups unchanged and left cores 6-9 / 6-11
unassigned. Every extra core now goes to throughput work; the two below
CAN become a dedicated recorder pair with no SCHED_FIFO camera work (new
"recorder" and "camera" groups, pin_recorder), so the recorder's CPU is
guaranteed instead of borrowed from IK. Control, IK, relay and CAN keep
their Orin NX CPUs. The layout is built over the kernel's online CPU list,
and the real-time camera pool now avoids whichever CPU the CAN interrupt
is actually delivered to (not only CPU0).

Jetson tuning (utils/jetson.py): Thor has no NVENC devfreq node; encode
and CUDA are clocked by gpu-nvd-*/gpu-gpc-*, which the *.nvenc/*.gpu globs
missed. Those domains get the performance governor (NVIDIA's Thor guide),
and neither the pin nor the runtime check reads their frequency files,
which take the devfreq lock a JetPack 7.2 hang blocks on.

Provision (cli/provision.py, installer, unit): `axol provision` applies
the per-boot host tuning as its last step, so it is the only command a
host needs. `axol provision --boot` (tuning only, no lock) is the unit's
ExecStartPre, ordered after nvpmodel.service; `axol jetson.setup` stays as
an alias for older units. The self-updater's pass inside serve skips the
tuning, since it can overlap a session and the restart re-tunes anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
axol Ready Ready Preview Sep 24, 2026 7:35pm UTC

Request Review

@shawnpatel
shawnpatel merged commit ad01bfe into main Sep 24, 2026
6 checks passed
@shawnpatel
shawnpatel deleted the add-agx-orin-thor-t5000 branch September 24, 2026 19:40

This branch was successfully deployed

1 active deployment
Preview — 832e4a13 Deployed Sep 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant