initrd-flash hangs during MB1/OP-TEE boot on Orin Nano 8GB (J3011) — variable stopping point, OP-TEE logs "FIRMWARE INIT FAILED AT LEVEL 95 FEATURE ID 13" when it gets furthest

[Hi5808]

Flashing a Seeed reComputer J401 carrier board with a Jetson Orin Nano 8GB
module (P3767 SKU 0003) using initrd-flash from a core-image-minimal
build on the wrynose branch. The board hangs partway through boot on
every attempt.

Environment:

• meta-tegra: wrynose branch
• openembedded-core: wrynose branch
• bitbake: 2.18 branch
• L4T: R39.2.0 / JetPack 7.2
• MACHINE: p3509-a02-p3767-0000, TEGRA_BOARDSKU = "0003"
• Host: Ubuntu 26.04 x86_64
• Module: Jetson Orin Nano 8GB, board SKU 0003 (confirmed via EEPROM), FAB 301, revision F.1, Chip SKU D5

Symptom:

From the host side, initrd-flash gets through RCM boot (Step 2) cleanly
every time — BCT, MB1, PSC-BL1, blob all sent successfully — then hangs
indefinitely at:

== Step 3: Sending flash sequence commands ==
Waiting for USB storage device flashpkg from ff746551......

with zero USB re-enumeration activity on the host (checked via lsusb/
kernel log — the board's USB link goes completely silent, not a flaky
reconnect). This reproduces identically across three different host USB
ports (via a hub, and two different direct ports on two different machines).

With a serial console attached (115200 baud, J401 debug header J14,
pins 6/9/10), I can see the board's own boot progress. Note: the raw
output is a mix of interleaved fragments from Tegra's Combined UART (TCU)
multiplexing multiple firmware log streams onto one wire — I have not
decoded/demuxed it, so lines appear interleaved/truncated, but the content
is legible enough to follow. Two separate attempts with a freshly re-seated
connection produced two different stopping points:

Attempt A — got further, reaching UEFI:

1. MB1/MB2 load normally, BPMP/PSC/RCE firmware load, SPE init, combined
   UART enabled.
2. STORAGE: Failed to open SDMMC: 3 / OCR failed, error = 39390706 for
   device 1-3 — explicitly logged as non-fatal (Ignoring init failure for device 1-3), presumably an SD/eMMC slot probe unrelated to the NVMe
   boot drive.
3. A Rebooting : reboot-recovery + second MB1 boot cycle happens — looks
   like the expected get_board_info() RCM probe-and-reboot from
   tegra-flash-helper.sh, not an error.
4. TOS/OP-TEE authenticates and boots, but logs:

   *** FIRMWARE INIT FAILED AT LEVEL 95 FEATURE ID 13 ***
   

5. A camera ISP driver error also appears:
   ERROR: camera-ip/isp5/isp5.c:2709 [isp5_pm_init] "ERROR: Faile[d ...]
6. Primary CPU switching to normal world boot, then t23x_minimal UEFI
   begins printing — then nothing else. Confirmed via a 2-minute watch
   with a byte-for-byte unchanged log while the host side kept
   accumulating wait-dots.

Attempt B — same setup, hung earlier, during OP-TEE's own boot,
before reaching the FEATURE ID 13 message at all — last visible output
was mid-OP-TEE-init (...Print Task[...] CPU initializing). Also confirmed
hung via a 2-minute zero-new-bytes watch; the screen process was still
alive and holding the serial port the whole time (so it's not a dropped
capture session).

Question: the fact that these two attempts stopped at different points
is itself useful information but I'm not sure how to interpret it — does
it suggest a timing-sensitive/racy firmware issue (rather than a
deterministic hang at one specific spot), or is this more likely explained
by an unreliable physical debug connection on my end (my UART wiring is
functional — verified via TX/RX loopback — but the header connection may
still be marginal)? Does anyone recognize FIRMWARE INIT FAILED AT LEVEL 95 FEATURE ID 13 from OP-TEE, or know what depends on it? In attempt A the
boot visibly continued past two other non-fatal-looking issues (SDMMC,
camera-ISP) but never got further after this one.

Raw serial excerpt — Attempt A (reached UEFI start)

I> cpubl:  TSEC leyblob
I> Task: Load and authenticate registered FWs
I>v3 registers
I> Task: Audit firewall settings
I> Task: Clear dit
[garbled]is OP-T[garbled][TS:95215793[garbled]9869679616]SHA Task I[garbled]nit
*** FIRMWARE INIT FAILED AT LEVEL 95 FEATURE ID 13 ***"
[garbled]I/TC:dress is : 0x4003e000
...
I> tos: Authentication Finalize Done
I> BinarE arg3: 0xbe040020
I> NVRNG: Health check succeconsole to device: /serial@0c198000
I/TC: OP-TEE v1845885]Admin T[garbled]ask Started
[garbled]might[garbled][TS:22191[garbled]1741]RM DCE SC7 SH[garbled]A Disabled
[garbled]re!
[garbled][TS:22204997[garbled]2]RM Task Ru[garbled]ERROR: camera-ip/isp5/isp5.c:2709 [isp5_pm_init] "ERROR: Faile
 Primary CPU switching to normal world boot
[garbled]
t23x_minimal UEF [garbled]
(— nothing further, confirmed via 2-minute watch —)

Raw serial excerpt — Attempt B (stopped earlier, in OP-TEE)

ask: Measured Boot init
I> Task: fTPM silicon identity init
I>TSA values
I> LA/PTSA updates are done from mb2-bct
I> Task: Vask: Carveout setup
I> Program remaining OEM cae: SafeRTOS v8.4
[garbled]spe: init
[garbled]scheduler initialized
...
I> tos: Authentication Finalize Done
I> Binary
I> Binary eks loaded successfully at 0xbe020000
I> EKB detectloader handler
I> Task: Disable MSS perf stats
I> Task: PrograI/TC:
complete
[garbled]I/TC: [garbled][TS[garbled]:95605723330353561[garbled]6]Print Task ry CPU initializing
(— nothing further, confirmed via 2-minute watch, screen process still holding the port —)

[Hi5808]

Quick follow-up after digging further: I don't think FIRMWARE INIT FAILED AT LEVEL 95 FEATURE ID 13 is actually the cause of the hang.

I checked the optee_os source used in this build — no match for that string or "FEATURE ID" anywhere in it. Looking at the fuller surrounding context, the message actually appears twice in my logs (once during the preliminary get_board_info() RCM probe-reboot, once during the real boot), and boot visibly continues past it both times without issue. The lines right around it (RM Task, DCE ... SC7 SHA Disabled) suggest it's actually printed by the DCE (Display Compute Engine) firmware, not OP-TEE — the I/TC: line immediately after is most likely just Combined-UART interleaving with a concurrent OP-TEE log line, not evidence that OP-TEE itself is what's failing.

So my current best guess is this is a benign/expected warning (maybe a secure-display feature that's irrelevant on this headless NVMe-flash setup), not the actual root cause. The real hang — right after t23x_minimal UEFI starts printing, then nothing — is still unexplained on my end. I don't have tooling locally to demux Tegra's Combined UART, which would let me see UEFI's own log stream cleanly separated from SPE/BPMP/DCE/OP-TEE's interleaved output on the same wire — that seems like the most direct way to see what UEFI itself was doing right before going silent, if anyone knows of such a tool or a cleaner way to capture just the UEFI stream.

[Hi5808]

Update: isolated this to a build/flash-mechanism issue, not hardware.

I flashed the same board with NVIDIA's official JetPack 7.2 firmware via a different tool (Seeed's DevelopTool, which uses the standard NVIDIA flash mechanism rather than initrd-flash) over the same USB-C port. That flash completed successfully and the board booted all the way to a running Linux userspace — systemd, NVMe partitions detected, USB/PCIe devices enumerating, and a USB RNDIS network interface I could ping successfully. So the module, carrier board, NVMe, cabling, and board SKU are all confirmed fine, and JetPack 7.2 genuinely works on this hardware.

One thing that stood out comparing the two logs: the official flash's log never shows t23x_minimal UEFI at all — only t23x_general UEFI during the final real-OS boot. My initrd-flash build's hang happens specifically at t23x_minimal UEFI startup, which appears to be a UEFI variant used only by the temporary flash-environment blob (tegra-initrd-flash-initramfs), not the final OS boot path. That's my best lead right now — happy to dig into how that image gets built/selected in meta-tegra if anyone can point me at the relevant recipe/logic, or if this rings a bell for anyone.

[Hi5808]

Ran two targeted Kconfig experiments comparing t23x_minimal (used only by initrd-flash's temporary flash-environment blob, where I hang) against t23x_general (used by the real OS boot, confirmed working via an official DevelopTool flash on the same board). Neither fixed it, but the combined pattern across 6 UART-captured attempts is informative, so sharing in case it helps narrow things down.

Experiment 1 — missing USB/PCIe/NVMe drivers (disproven):
BuildMinimal.conf implies none of PCIE, NVME, SOC_USB, SELECT_ALL_USB_DEVICES, XHCI, EHCI, while BuildGeneral.conf implies all of them. Patched BuildMinimal.conf to add all six, rebuilt (bitbake edk2-firmware-tegra-rcmboot -c configure/compile/deploy -f + image rebuild), verified the generated .config actually has all six as y. Result: identical hang, same exact spot as unpatched. Ruled out — the hang happens too early for driver availability to matter.

Experiment 2 — SECURITY_MM_OPTEE vs SECURITY_MM_NONE (inconclusive but informative):
Diffing the full generated .config (not just the defconfig) surfaced CONFIG_SECURITY_MM_OPTEE=y (general) vs CONFIG_SECURITY_MM_NONE=y (minimal). Patched BuildMinimal.conf with imply DEFAULT_SECURITY_MM_OPTEE if SOC_T23X, rebuilt, verified the flip took effect. Flashed twice (same build, two separate attempts). Both hung, but at two different points, neither matching the unpatched hang location.

Full pattern across all 6 UART-captured attempts:

#	Build	Hung at
1	unpatched	t23x_minimal UEF[I] (UEFI banner starts, nothing after)
2	unpatched	mid ...CPU initializing (earlier, in OP-TEE)
3	USB/NVMe patch	t23x_minimal UEF[I] (same as #1)
4	MM_OPTEE patch	mid SHA Task Init
5	MM_OPTEE patch	mid Test OEM keys are being used

Every attempt hangs somewhere in the OP-TEE/TOS→normal-world transition zone, but at a genuinely different exact point each time, regardless of which Kconfig variant was flashed. That pattern feels more consistent with a timing-sensitive/racy issue in this specific boot chain than a static misconfiguration — but I don't have the EDK2/Tegra platform depth to take it further than that, and I don't have tooling to properly demux Tegra's Combined UART (multiple firmware log streams interleaved on one wire) which might be smearing a more consistent underlying stopping point into what looks like noise.

At this point I think I've exhausted what I can usefully test locally — happy to run anything more specific if someone has a concrete next step, otherwise this is where I'm stuck.

[joekale-pp]

Orin Nano does not have onboard eMMC. The MACHINE you reference p3509-a02-p3767-0000 is the Orin NX which does. You need to change your machine to the jetson-orin-nano-devkit-nvme if you will be using NVMe. Seeed boards are not officially supported by meta-tegra but the Seeed J401 may be close enough to the devkit to at least get you able to flash and work out what is missing with a custom MACHINE type.

[madisongh]

I'd just try MACHINE="p3768-0000-p3767-0003". The J401 carrier emulates the P3768 carrier, despite having an HDMI jack instead of a DisplayPort jack. It's worked for me, at least.

[Hi5808]

@joekale-pp thank you — that's a genuinely strong lead I hadn't considered. BuildMinimal.conf (the Kconfig for the t23x_minimal UEFI where I hang) implies EMMC support, so if p3509-a02-p3767-0000 really carries Orin NX defaults, the minimal UEFI trying to probe eMMC that doesn't physically exist on the Nano module would explain a hang that's specifically isolated to that build variant (the t23x_general UEFI, which boots the real OS and works fine, implies a much broader set of storage/USB drivers where a missing eMMC probably just gets skipped/ignored rather than hanging). I didn't get a chance to test switching to jetson-orin-nano-devkit-nvme with a custom MACHINE before landing on a workaround (below), but this seems like the right next place to look for anyone still chasing the actual root cause.

Closing the loop on my end: I worked around this rather than fixing it. Since I needed a working board and didn't want to keep guessing, I flashed the same core-image-minimal rootfs through NVIDIA's official l4t_initrd_flash.sh (extracted from a Seeed DevelopTool-generated mass-flash BSP package) instead of meta-tegra's own initrd-flash. That uses the t23x_general UEFI/bootloader chain end-to-end (never touches t23x_minimal at all), and it flashed and booted cleanly on the first try. So my actual board is up and running now, custom minimal rootfs and all — just not via meta-tegra's own flashing path. Leaving the thread open/unresolved on the t23x_minimal + Orin Nano NVMe question since your MACHINE-mismatch theory seems like the real lead, in case it helps someone else (or I circle back to it later with more time).

[Hi5808]

Mahalo for the advice, Got it up and running!
https://github.com/Hi5808/j3011-custom-os
@joekale-pp @madisongh @kekiefer @jonte