feat: Persistent addr2line with ELF isolation and rebuild detection (IDFGH-18193)

[nebkat]

Why

PcAddressDecoder spawns a fresh addr2line for every monitor line that contains hex addresses. Nearly all of that cost is startup — addr2line loads and indexes the ELF each time — and it is paid again for every decoded backtrace line, on an ELF that has not changed. Keeping one process alive per ELF and feeding it addresses on stdin removes that cost from every decode after the first.

What

addr2line subprocess handling moves into a new esp_idf_panic_decoder/addr2line.py (Addr2LineRunner), leaving pc_address_decoder.py to do address matching and presentation:

• One process per ELF, kept alive and reused. Each lookup writes <addr>\n0xfefefefe\n and reads until the sentinel's 3-line echo (0xfefefefe / ?? / ??:0) marks the end of the response. -i emits a variable number of inlined frames, so a terminator is needed; 0xfefefefe is well outside any ESP code region and is rejected by PcAddressMatcher.is_executable_address, so it cannot collide with a real lookup.
• Falls back to the current one-shot subprocess.check_output per address if the persistent path fails (broken pipe, spawn error), so a decode still returns a result.
• addr2line opens a temp copy of the ELF, not the ELF itself. On Windows an open BFD handle can block the linker from rewriting the ELF during a rebuild — a real risk once the handle is long-lived rather than momentary.
• Rebuilds are picked up automatically. Each lookup stats the original (mtime, size); if it changed, the cached process is torn down and respawned against a fresh copy. Without this, a persistent process would happily keep answering from a stale ELF for the rest of the session.
• Cleanup via atexit, plus close() and __enter__/__exit__ for callers that want deterministic teardown.

Because the spawn cost is gone, the per-ELF batching in translate_addresses no longer earns its bookkeeping and collapses to a per-address loop. is_executable_address filtering and its first-ELF-wins precedence are unchanged.

API

PcAddressDecoder(...), translate_addresses, perform_addr2line, decode_address and pc_address_buffer keep their signatures and behaviour — esp-idf-monitor needs no changes and picks up the speedup as-is. Added: lookup_address(address, elf_file, is_rom) and close().

One removal: parse_addr2line_output. It is unused once addresses are looked up one at a time, and keeping it would mean a second copy of the frame parsing that now lives in addr2line.py. It was undocumented and not used by esp-idf-monitor, the only known consumer — happy to restore it as a wrapper if you would rather not break the signature.

Testing

Run against real backtraces from an esp-idf-monitor session: decoded output matches the previous implementation, with the speedup visible from the second decoded line onward. The parser was additionally diff-tested against the old implementation across inlined frames, unresolved ??/??, (discriminator N), CRLF, Windows paths and demangled C++ signatures, and the subprocess lifecycle (reuse, respawn on rebuild, respawn after a killed process, fallback paths, temp cleanup) was exercised against a stub addr2line.

The repo has no test suite to extend, so none of that is included here. Glad to add one if you would like it in the same PR.

[peterdragun]

Hi @nebkat, thank you for contributing. I would like to understand your use case better. Is this trying to solve the issue where the Monitor becomes unresponsive when in a boot loop? (as mentioned in espressif/esp-idf-monitor#45)

I definitely agree that running addr2line in a subprocess for each line is not ideal. But considering that this only runs on a couple of lines when the chip is booted and then again when there is a panic, I think the cost is not that high (at least in a standard scenario). We can consider using a simpler solution - we could introduce a cache, which would cover the bootloop scenario quite nicely without needing to change the logic that much.

I think that with a new "Force quit" option in IDF Monitor, this becomes less of an issue, so a simple cache could further improve the experience of using our tools without adding a lot of complexity.

Let me know what you think.

[nebkat]

Hi @peterdragun,

We are actually using backtraces at runtime for debugging non-fatal errors and event loop slowdowns so we get a lot more than just the panic. (Side note: this should be encouraged somewhere in the docs as it is quite the superpower! I think back to the days of inserting countless log lines to figure out where a failure was coming from...)

Without this change it becomes very hard to follow what is going on as the screen starts to lag while resolving each entry. Admittedly we are on the extreme end as we frequently end up printing backtraces and they are quite long, but, with this change it is all instant and works perfectly.

I appreciate this adds some complexity but it is overall nicely contained within this package and maintains the same external API so I think it could be a nice addition for these more advanced workflows.

Separately I am considering making a PR that allows selectively enabling/disabling the backtrace decoding, as there are times when we do not need them but don't want to quit out of the monitor to toggle it.