`esp-radio 0.18.0` (ESP32-S3): intermittent memory corruption in the Wi-Fi timer path — `InstrError` / `StoreProhibited` in `ets_timer_disarm`, `pp_timer_process`, `hal_enable_sta_tbtt`

[barak1412]

Bug description

On an ESP32-S3 running a Wi-Fi STA workload, the device panics intermittently
after anywhere from ~40 seconds to ~6 hours of uptime. Across four captured
crashes the faults all land in the Wi-Fi blob's timer / power-management /
beacon machinery, and two of them are unmistakable memory corruption:

• the CPU is directed to execute a data-RAM address (InstrError, PC in
  DRAM), i.e. a callback/function pointer has been overwritten, and
• a store through a bad pointer inside esp-radio's own timer shim
  (StoreProhibited at esp_radio::common_adapter::ets_timer_disarm).

The pattern looks like a use-after-free / stale-pointer on a timer object
shared between esp-radio's timer adapter and the closed-source Wi-Fi blob.

I understand pp_timer_process, txpwr_offset, pm_beacon_offset_get_expect
and hal_enable_sta_tbtt are inside the proprietary blob and not esp-rs code.
Filing here because the crash is reached through the esp-radio/esp-rtos timer
glue — and one crash faults inside esp_radio::common_adapter::ets_timer_disarm
itself — and because this seems the right path to escalate to the blob team.
Happy to be redirected.

To Reproduce

Honest caveat: it only reproduces inside the full application, and it is
intermittent, so confirming any change needs multi-hour soaks. I have not been
able to reduce it to a minimal example — my attempt at a standalone
Wi-Fi-only harness does not link (the blob's libprintf f64 helpers land
outside windowed-call range in a binary that small: dangerous relocation: windowed longcall crosses 1GB boundary), which is itself why the "reduced
build" above was derived from the full firmware instead.

Shape of it:

1. ESP32-S3 (N8R8), Wi-Fi STA associated to an ordinary 2.4 GHz AP.
2. Run an embassy executor with embassy-net plus a Matter (UDP/IPv6) workload.
3. Leave it running for hours.
4. Observe an eventual InstrError / StoreProhibited resolving into the blob
   timer, power-management or beacon paths.

Expected behavior

Indefinitely stable Wi-Fi STA operation with no corruption of timer callback
pointers or timer objects.

Captured crashes

All four were captured over serial and symbolicated with
xtensa-esp32s3-elf-addr2line against the exact ELF that was running.

1. StoreProhibited inside esp-radio's timer shim

ERROR - panicked at .../esp-hal-1.1.1/src/exception_handler/mod.rs:10:5:
Exception occurred on ProCpu 'StoreProhibited'
Context { PC: 0x42065082, A1(SP): 0x3FCDDBF0, A0: 0x40380788, ... }

register	value	symbol
PC	0x42065082	esp_radio::common_adapter::ets_timer_disarm
A0 (return addr)	0x40380788	pm_beacon_offset_get_expect (blob, power management)

The blob's power-management code called into esp-radio's ets_timer_disarm,
which then stored through a pointer it should not have — consistent with the
timer object having been freed or never validly initialised.

2. InstrError — executing data RAM

ERROR - panicked at .../esp-hal-1.1.1/src/exception_handler/mod.rs:10:5:
Exception occurred on ProCpu 'InstrError'
Context { PC: 0x3FCDB230, A1(SP): 0x3FCE6350, A0: 0x421670FD, ... }

register	value	symbol
PC	0x3FCDB230	data RAM — not executable, not in .text
A0 (return addr)	0x421670FD	hal_enable_sta_tbtt (blob, beacon timing / TBTT)

The blob's beacon-timing path called through a function pointer that had been
overwritten with something pointing into DRAM.

3–4. Earlier InstrErrors on the same build family

Exception occurred on ProCpu 'InstrError'
Context { PC: 0x3FCAFCC0, EXCVADDR: 0x3FCAFCC0, EXCCAUSE: 2,
          A0: 0x82166FF1, A8: <pp_timer_process>, ... }

• PC == EXCVADDR, again a DRAM address (executing data).
• A0 → txpwr_offset (blob, TX-power calibration).
• Reached from pp_timer_process (blob, periodic timer processing).
• Reproduced twice with the identical corrupted PC; across two builds the
  address moved by exactly the build's layout delta (0x3FCAFC40 vs
  0x3FCAFCC0), i.e. it consistently pointed at the same object — in our
  binary, storage belonging to an embassy task. That is what made us read it as
  a waker/callback-pointer mix-up rather than random corruption.

Environment

Chip	ESP32-S3 rev v0.2, N8R8 module (8 MB flash, 8 MB octal PSRAM, PSRAM in Octal SPI mode)
esp-hal	1.1.1
esp-radio	0.18.0 (features: esp-alloc, esp32s3, log-04, unstable, wifi)
esp-rtos	0.3.0 (features: embassy, esp-alloc, esp-radio, esp32s3, log-04)
esp-alloc	0.10.0
embassy-executor	0.10.0 / embassy-net 0.9.1
Bootloader	bundled ESP-IDF v5.5.1-838-gd66ebb86d2e
Build	no_std, release, opt-level = "s", lto = "fat"
Wi-Fi power save	default (PowerSaveMode::None), never changed

Application: an embassy executor running Wi-Fi STA + embassy-net +
rs-matter (Matter over UDP/IPv6) +
a small HTTP server. Heap is esp-alloc, with the large Matter allocations
placed in PSRAM.

[bugadani]

You likely have a stack overflow somewhere. Localising that would help. I'd also prefer not talking to an LLM, if possible.

[barak1412]

@bugadani Used LLM to summarize my investigation, but noted - from now on, I will use my words.

I rulled out stackoverflow, but I guess you still need repro?

[bugadani]

If possible, a reproducer would be useful, especially if we can crash the CPU deterministically and quickly. But first:

Heap is esp-alloc, with the large Matter allocations placed in PSRAM.

@ivmarkov is that safe? Would rs-matter end up bumping into #2027 ?

[ivmarkov]

If possible, a reproducer would be useful, especially if we can crash the CPU deterministically and quickly. But first:

Heap is esp-alloc, with the large Matter allocations placed in PSRAM.

@ivmarkov is that safe? Would rs-matter end up bumping into #2027 ?

TL;DR: rs-matter should be safe (it does not use any atomics - neither core::sync::atomic nor portable-atomic), but this does not mean that the user code is safe. If @barak1412 is having a custom AsyncHandler or Handler instances with (indirect) usage of core::sync::atomic or portable-atomic, it would be a problem.

Yes, using PSRAM with the s3 and putting any non-trivial structs code in PSRAM is my number 0 suspicion.

Also I'm not sure why rs-matter needs to be there? It takes ~ 60 to 70KB RAM, and then ~ 20KB of stack. On a chip with 512KB SRAM, that's not that much.

[barak1412]

@bugadani @ivmarkov thanks a lot, based on your recommendations, will dig into it and update.

[barak1412]

Tried enabling ESP_RTOS_CONFIG_SW_TASK_OVERFLOW_DETECTION, and after ~5.8 hours, my app crashed again, without a clue of stack overflow (the stack pointer was in a legal heap region).

Also no atomics in my app's PSRAM-resident structs.

It will be difficult, but I will try to write a reproducer.