Support AGX Orin and Thor T5000: core layouts, Jetson tuning, one-command provision - #326
Merged
Merged
Conversation
…mand provision Core layout (utils/affinity.py): 12-core AGX Orin 64GB and 14-core Thor T5000 used the 8-core groups unchanged and left cores 6-9 / 6-11 unassigned. Every extra core now goes to throughput work; the two below CAN become a dedicated recorder pair with no SCHED_FIFO camera work (new "recorder" and "camera" groups, pin_recorder), so the recorder's CPU is guaranteed instead of borrowed from IK. Control, IK, relay and CAN keep their Orin NX CPUs. The layout is built over the kernel's online CPU list, and the real-time camera pool now avoids whichever CPU the CAN interrupt is actually delivered to (not only CPU0). Jetson tuning (utils/jetson.py): Thor has no NVENC devfreq node; encode and CUDA are clocked by gpu-nvd-*/gpu-gpc-*, which the *.nvenc/*.gpu globs missed. Those domains get the performance governor (NVIDIA's Thor guide), and neither the pin nor the runtime check reads their frequency files, which take the devfreq lock a JetPack 7.2 hang blocks on. Provision (cli/provision.py, installer, unit): `axol provision` applies the per-boot host tuning as its last step, so it is the only command a host needs. `axol provision --boot` (tuning only, no lock) is the unit's ExecStartPre, ordered after nvpmodel.service; `axol jetson.setup` stays as an alias for older units. The self-updater's pass inside serve skips the tuning, since it can overlap a session and the restart re-tunes anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds AGX Orin (12 cores) and Thor T5000 (14 cores) as compute options alongside Raspberry Pi 5 and Orin NX 16GB.
axol provisionis now the only setup command a host needs.Core layout (
utils/affinity.py)SCHED_FIFOcamera thread may run there. On 8 cores, camera threads preempt the recorder on every core it has: it got about 5% of each core on the 2026-09-14 policy ops. That's why it currently borrows the IK core. With a pair of its own it no longer needs to, soshare_ik_corehas no effect on these hosts. The normal-priority GStreamer threads can still use the recorder cores.Jetson tuning on AGX Orin and Thor (
utils/jetson.py)gpu-nvd-*and CUDA bygpu-gpc-*, and the*.nvenc/*.gpupatterns missed both. Before this change, a Thor pinned only its VIC. Those two domains now get theperformancegovernor, as NVIDIA's Thor power guide prescribes.jetson_diag) touches those nodes' frequency files. Those files take the devfreq lock, and a JetPack 7.2 Thor hang has been seen blocking on it.tegra-xusbxHCI driver, so the CAN interrupt lookup carries over. MAXN is mode 0 on both boards.One command:
axol provisionaxol provisionapplies the per-boot host tuning as its last step. It logs the board and the core layout, e.g.core layout: 14 cores online: CAN 12-13, control 2, IK 3, relay 4-5, recorder 10-11, camera 0-1,5-9.axol provision --bootruns only the tuning, with no installs and no update lock. The installer's unit now uses it asExecStartPreand is orderedAfter=nvpmodel.service; the docs already claimed that ordering. The installer's separateaxol jetson.setupstep is removed.axol jetson.setupstays as an alias, because units written by older installers call it.axol serveskips the tuning. It can overlap a robot session, and the service restart that ends an update re-tunes the host before serve starts.Test plan
uv run pytest: 1591 passed, 1 skippedruff check ./ruff format --check .(0.9.7)bash -n web/app/public/installaxol provision --bootandaxol jetson.setupon a non-Jetson VM: log the host and layout, apply no tuning, exit 0axol provisionlogs the 12-core layout; the CAN interrupt lands on CPU 11;nvargus-daemonis confined to the camera coresgpu-gpc-0/gpu-nvd-0reportperformance, VIC is pinned, the CAN interrupt lands on CPU 13, and a recording runs without skipped exposures or CAN faultsExecStartPre(axol provision --boot) re-applies the tuningNot covered
axol servewon't start. NVIDIA's kernel patch is the fix.🤖 Generated with Claude Code
Note
High Risk
Changes real-time CPU partitioning, Jetson clock/interrupt tuning, and boot-time systemd hooks on production robot hosts; mis-tuning can affect CAN timing, camera encode latency, and recording throughput.
Overview
axol provisionis now the single host setup path: installs stay the same, but the last step applies per-boot real-time tuning (Jetson power mode, engine/CPU clocks, CAN interrupt steering, ArgusSCHED_FIFO).axol provision --bootruns only that tuning (no update lock) and replacesaxol jetson.setupin the installer’sExecStartPre;jetson.setupremains as an alias for older units.CPU affinity gains explicit
recorderandcameragroups: on 12+ core Jetsons (AGX Orin 64GB, Thor T5000), middle cores join the throughput pool and two cores below CAN are dedicated to the dataset recorder and Rust trace writers, so FIFO camera work no longer starves recording. Layouts use the kernel online CPU list; the camera pool avoids whichever CPU(s) actually carry the CAN USB interrupt, not only CPU0.Thor (JetPack 7) tuning pins
gpu-gpc-*/gpu-nvd-*via theperformancedevfreq governor (nomin_freqon those nodes); runtime pin checks match. Docs,.superset/config.json, andweb/app/public/installdrop the separatejetson.setupinstall step and order the unit afternvpmodel.service.Provisioning skips live host tuning when
axol serveis already running (self-updater); the service restart +ExecStartPrere-applies tuning instead.Reviewed by Cursor Bugbot for commit 832e4a1. Configure here.