Available for consulting on Jetson/Orin OS bring-up and inference optimization.
Get in touch

J3011 Custom OS — A Minimal Yocto Linux for the Jetson Orin Nano

A from-scratch, minimal Yocto Linux build for a Jetson Orin Nano 8GB (Seeed reComputer J401 carrier, JetPack 7.2 / L4T R39.2.0), built to leave the maximum possible share of an 8GB unified-memory budget free for on-device LLM inference — no desktop environment, no unused subsystems, nothing shipped that a GPU-accelerated inference workload doesn't need.

This covers the actual OS bring-up: getting it to boot at all, getting CUDA to work on it, and validating that TensorRT-Edge-LLM runs correctly on top of it — the foundation everything else in this project is built on.

Result

$ uname -r
6.8.12-1021-tegra
$ cudaGetDeviceCount()
err=0 (no error), count=1
Device 0: Orin, compute capability 8.7

CUDA works. TensorRT-Edge-LLM builds and runs natively on-device. Rootfs auto-resizes to the full NVMe partition on first boot. CPU/GPU clocks lock to max performance automatically. All of it survives a clean reflash with zero manual intervention.

None of this was straightforward — JetPack 7.2 / L4T R39.2 was, at the time of this build, NVIDIA's first-ever officially supported Yocto release for Jetson (wrynose, previously community-only via OE4T). Several of the bugs below were genuinely new territory.

The flashing saga

The very first blocker was flashing at all. meta-tegra's own initrd-flash reliably hung during boot — visible only via a UART console wired directly to the board's debug header, since the host-side tooling just showed an indefinite "waiting for USB storage device" stall with no other signal.

Getting a usable UART trace took real hardware debugging on its own. Tegra's Combined UART multiplexes multiple firmware log streams (MB1, SPE, BPMP, TrustZone, etc.) onto a single physical wire — plain screen just shows the raw interleaved mess. Building tcu_muxer from source to properly demux the stream turned "garbled fragments" into clean, per-component logs, which is what made the rest of this debugging possible at all.

With a working trace, the board was hanging right at the very start of UEFI, with no fixed, reproducible point — six captured attempts across two different Kconfig experiments (BuildMinimal.conf's missing USB/NVMe/PCIe options, and a SECURITY_MM_OPTEE vs SECURITY_MM_NONE mismatch) each hung at a different spot in the OP-TEE→UEFI handoff. That pattern — non-deterministic hang location regardless of which static config changed — pointed at a timing-sensitive firmware race rather than a fixable Kconfig flag, past the point where further blind rebuild-and-reflash cycles were productive. Filed a full writeup upstream (OE4T/meta-tegra#2296) with the complete attempt-by-attempt pattern.

Breakthrough came from a deliberate control test, not more UEFI debugging: flashing the exact same board via Seeed's official DevelopTool (NVIDIA's stock JetPack firmware) succeeded cleanly and booted to a login prompt over USB networking. That conclusively proved the hardware, module, carrier, and cabling were all fine — the hang was a real, narrow bug specific to this build's flash-environment UEFI variant, not a hardware or general JetPack 7.2 problem.

First real boot came from a workaround, not a fix: substituting the custom-built core-image-minimal rootfs directly into NVIDIA's own official flashing mechanism (l4t_initrd_flash.sh, the same one DevelopTool uses under the hood) — bypassing meta-tegra's broken UEFI path entirely by riding on a boot chain already proven to work. That meant reverse-engineering how the flash package's system.img is actually consumed (despite the .img name, it's a zstd-compressed rootfs tarball detected by magic bytes, not a raw disk image) and rebuilding it to match exactly.

The actual root cause was found later, as a side effect of chasing a different bug (see below): the flashing UEFI variant only matched the right board correctly once the build's MACHINE was switched to jetson-orin-nano-devkit-nvme — at which point meta-tegra's own initrd-flash started working cleanly too, closing out the original flashing bug for good.

The device-tree bug

Even once flashing worked, /proc/device-tree/compatible reported the wrong board — a p3767-0003 (Orin Nano) module identifying itself partly as an Orin NX. Root-caused to Seeed's massflash BSP package: its QSPI cpu-bootloader blob is a single generic file with no SKU number in its name, unlike sibling firmware blobs in the same package that correctly bake the SKU in — i.e. per-SKU blob generation happened for some firmware components but not that one. Switching to the correct MACHINE and meta-tegra's own (by-then-fixed) flashing bypassed the defective blob entirely — both compatible and model came back fully correct, better than stock JetPack's own output.

The CUDA bug

With flashing and device-tree both solid, cudaGetDeviceCount() still failed with CUDA_ERROR_NOT_SUPPORTED — a separate, unrelated problem.

Isolated via a direct A/B hardware test: the exact same physical board, same module, same device tree content, reflashed to stock JetPack — CUDA worked immediately. Reflashed back to this custom build — CUDA failed identically. Same hardware, same silicon, two different software stacks, one working and one not. That conclusively ruled out hardware and pointed at the from-source kernel build (linux-noble-nvidia-tegra + nvidia-kernel-oot) — most likely a subtle ABI/behavioral difference in the from-source-built nvgpu.ko versus NVIDIA's official prebuilt driver.

Root-caused the actual dependency chain empirically, not from documentation. Rather than guessing what CUDA/TensorRT need at runtime, instrumented the known-good stock JetPack system directly — strace -f -e trace=openat,open,mmap,ioctl,mknod on a CUDA test binary, plus a full lsmod capture — to get ground truth on exactly which device nodes and kernel modules are load-bearing (/dev/nvmap, /dev/nvgpu/*, nvgpu/nvmap/host1x/tegra_drm/nvidia*, thirteen modules total out of ~150 resident). That turned an open-ended "why doesn't this work" into a concrete, verifiable target.

Fix: swap in NVIDIA's official prebuilt kernel and a curated module subset, via a new Yocto recipe (nvidia-prebuilt-kernel), instead of building the kernel from source at all. Three real bugs surfaced only by actually building and flashing the result, invisible to bitbake -e dependency resolution:

• Shipping NVIDIA's original modules.dep (referencing hundreds of modules outside the curated subset) caused a kernel panic in the flashing initrd itself — regenerating it via the project's own depmod against just the shipped modules fixed it.
• The flashing environment needed the Tegra QSPI controller driver (spi-tegra210-quad.ko) that the booted OS itself never touches — missed on the first pass because nobody anticipated the flashing initrd having different driver needs than the final OS.
• A stale bitbake hash-equivalence cache produced a spurious "metadata is not deterministic" error after rapid recipe iteration — a tooling false-positive, not a real recipe bug.

Last piece: a red herring in disguise. The flash reported success, but the board dropped into an interactive UEFI shell instead of booting — reproduced identically across three power cycles, with the boot binary confirmed present and correctly named. Root-caused to stale UEFI NVRAM state left over from the session's many earlier interrupted flash attempts, not a defect in the build itself — fixed with a --qspi-only reflash that resets just the variable store without rewriting the whole rootfs.

With all of that resolved: cudaGetDeviceCount() returns success, on the team's own from-scratch minimal OS.

TensorRT-Edge-LLM, validated twice

Validated the inference stack itself first on stock JetPack (to decouple "does this framework work on this hardware class at all" from the kernel-swap fix still in progress), then reproduced the same validation on the finished custom OS as an acceptance test.

• Two-stage pipeline confirmed: model quantization/ONNX export needs an x86 host with a discrete NVIDIA GPU; the actual TensorRT engine build and inference run entirely on-device.
• Qwen3-1.7B-Instruct: 2,025 tok/s prefill, 62.2 tok/s decode (CUDA-graph captured, steady state, clocks locked via jetson_clocks).
• Qwen3-4B-Instruct-2507: first build attempt hit a genuine CUDA OOM at framework-default settings — real, concrete evidence that 4B-class models are tight on an 8GB unified-memory budget at default parameters. Fixed by trimming --maxKVCacheCapacity/--maxInputLen; built and ran correctly afterward.
• Confirmed constraint: this generation of Orin hardware doesn't support FP8/FP4 at runtime in this software release — FP16/INT8/INT4 only.

Reproducing the exact same validation methodology on the custom OS (same models, same benchmark) closed the loop: the kernel-swap fix didn't just make cudaGetDeviceCount() pass, it produced a genuinely working inference stack.

Performance tuning, baked in and verified across a real reflash

Locked max CPU/GPU clocks (NVPMODEL_CONFIG_DEFAULT, jetson_clocks autostart) and an aggressive fan curve (100% PWM above 65°C) into the image, then reflashed to verify — and caught a real bug doing it, rather than assuming the config took effect: jetson-clocks-autostart.service had a hard Requires= on nvpmodel.service, which reliably fails on this board's kernel/module set (a missing sysfs node) — meaning the clock-lock service silently never even attempted to start on any boot, despite being correctly "wired up" in every config file. Relaxing that to Wants= (preserving ordering without hard-blocking on the known-failing dependency) fixed it. Reflashed a second time to confirm: all six CPU cores locked to 1728MHz automatically on first boot, no manual intervention needed.

What this makes possible

A Jetson Orin Nano that boots directly into a minimal Linux userspace with a working, benchmarked CUDA/TensorRT-Edge-LLM stack, locked performance clocks, an auto-resizing rootfs, and headroom deliberately preserved for an on-device LLM workload instead of a desktop environment nothing here needs. This is the baseline the rest of the project's on-device agentic coding pipeline is built on top of.