Border Router cannot route mesh→LAN traffic to hosts using its own advertised on-link prefix (route table in libopenthread_br never covers the self-advertised PIO)

[littleant]

Note

This issue was researched and written by an AI — Claude (Fable 5), not a human. It was produced by analyzing the ESP-IDF/OpenThread/lwIP sources and the symbol tables of the prebuilt libopenthread_br.a on behalf of the reporting user. The human user reviewed the write-up and personally verified the failure and the workaround on real hardware (packet captures, pings, flashing). Please read with the usual scrutiny applied to automated analysis.

Environment

• Example: examples/basic_thread_border_router (ESP32-S3 + RCP, Wi-Fi backbone)
• ESP-IDF: v5.5.4
• esp-thread-lib (components/openthread/lib): commit 2e2d91a ("update thread-lib for upstream a12ff0d0f")
• LAN: AVM FRITZ!Box advertising a GUA prefix and its own ULA prefix via RA

Symptom

Thread end devices become unreachable from LAN hosts (e.g. a Matter
controller) whenever the host sources its traffic from the Border Router's
self-advertised on-link prefix. The mesh looks healthy the whole time (BR is
leader, good link quality, devices "connected"), which makes this very hard
to diagnose.

OpenThread's routing manager derives a local on-link prefix from the extended
PAN ID (fd17:9499:2de5:90bf::/64 here) and advertises it as a PIO on the
backbone link whenever it has not discovered a favored on-link prefix from
another router — e.g. transiently at startup or after a Wi-Fi reconnect,
before the infrastructure router's RA has been received/processed. LAN hosts
then autoconfigure fd17:... addresses (1800 s lifetimes) and — by RFC 6724
rule 8 (longest prefix match against the Thread OMR ULA) — prefer them as
source for Thread destinations. Even after the BR later deprecates the
prefix, hosts hold the addresses until the valid lifetime expires, so the
breakage persists long after the triggering window.

Reproduction from a LAN host (ip -6 route get <omr-address> confirmed the
kernel picked the fd17 source):

ping source	result
host GUA (2a00:xxxx:xxxx:xx00::/64)	works, normal RTT
infra router ULA (fd06:1f1e:b316::/64)	works, normal RTT
BR on-link prefix (fd17:9499:2de5:90bf::/64)	100 % loss

Note the infrastructure prefixes are fully favored-eligible per
OnLinkPrefix::IsFavoredOver() (rdisc6 capture of the FRITZ!Box RA):

PIO	Valid lifetime	Preferred lifetime	Flags
2a00:xxxx:xxxx:xx00::/64 (GUA)	7200 s	3600 s	on-link + autonomous
fd06:1f1e:b316::/64 (ULA)	7200 s	3600 s	on-link + autonomous

so the local on-link prefix advertisement is the transient/redundancy case
that OpenThread intentionally supports — and a later capture showed the OTBR
no longer sending the fd17 PIO (deprecating), consistent with the
intermittent trigger. RFC 4862 address retention on the hosts turns that
transient into a long-lived outage.

Root cause

The Thread-side routing is correct: RoutePublisher publishes ::/0 /
fc00::/7 into network data (and keeps the ULA route published while the
on-link prefix is deprecating, precisely so return traffic still routes to
the BR). Device replies to fd17:... destinations therefore do reach the BR.
The packets are lost inside the BR's LAN-side forwarding.

lwip_hook_ip6_route() / lwip_hook_nd6_get_gw() are implemented in the
closed-source libopenthread_br.a (esp_openthread_route_table.c.obj).
Inspecting the library symbols shows the route table is populated exclusively
from received RAs:

• esp_openthread_route_table_add_route_entry is referenced only by
  esp_openthread_nd6.c.obj (the ICMPv6 ND receive path)
• nothing consumes otBorderRoutingGetOnLinkPrefix() to install a route for
  the prefix the BR itself advertises

Since the BR never receives its own RA, no route entry for
fd17:9499:2de5:90bf::/64 ever exists. Mesh→LAN packets destined to hosts in
that prefix either fail routing or are handed to the default gateway learned
from the infrastructure router, which has no route for the prefix — a silent
blackhole for exactly the addresses the BR told the hosts to use.

Expected behavior

Whenever the routing manager publishes/advertises (or is still deprecating)
its local on-link prefix, the BR must be able to deliver packets to that
prefix on-link on the backbone interface — e.g. by installing a corresponding
on-link route in the route table, or by assigning the BR an address inside
the prefix.

Workaround (verified)

Assigning the BR a static backbone-netif address inside the local on-link
prefix fixes it: lwIP gives statically configured addresses an implied
on-link /64 (RFC 5942), and nd6_get_next_hop_entry() checks on-link status
before the ND6_GET_GW hook, so replies are delivered directly via neighbor
discovery:

otIp6Prefix prefix;
esp_netif_t *backbone = esp_openthread_get_backbone_netif();
esp_ip6_addr_t lladdr, addr = {0};

esp_netif_get_ip6_linklocal(backbone, &lladdr);
otBorderRoutingGetOnLinkPrefix(esp_openthread_get_instance(), &prefix);
memcpy(addr.addr, prefix.mPrefix.mFields.m8, 8);
memcpy((uint8_t *)addr.addr + 8, (const uint8_t *)lladdr.addr + 8, 8);
esp_netif_add_ip6_address(backbone, addr, true);

(run on OPENTHREAD_EVENT_ATTACHED / OPENTHREAD_EVENT_DATASET_CHANGED,
refreshed if the prefix changes)

Flashed and verified on the affected network: pings to Thread devices sourced
from the fd17:... host addresses now get replies with normal Thread RTTs,
where they previously had 100 % loss.

This resolves the connectivity loss, but the proper fix belongs in
libopenthread_br, since every deployment where the BR's own on-link prefix
wins source selection on a host is affected.

[littleant]

As I understand it, the TBR published an address to my home network that my Matter server used as its source address (for whatever reason) when communicating with Thread devices. But the TBR didn't relay the answers back to this address. The AI found this to be the issue:

Your host was sending Matter traffic to Thread devices from the wrong IPv6 source address. The ESP border router advertises an on-link ULA prefix fd17:9499:2de5:90bf::/64 on the LAN, but it does not publish a return route for that prefix into the Thread mesh. `ip -6 route get` showed the kernel picking exactly that fd17 address as source for Thread destinations, so every reply from the devices was blackholed. Proof: pings to a device sourced from fd17 had 100% loss, while the identical ping sourced from the FritzBox ULA (fd06) or your global address worked with normal Thread RTTs. The mesh itself was healthy the whole time (OTBR is leader, 7 routers, good link quality) — which is why the devices looked "connected" to the border router.

I let Fable 5 find the root cause and it rechecked the claim from the first AI and responded with:

The return route into the Thread mesh *is* published; the blackhole is inside the border router itself, on the LAN side.

Sorry for the AI quotes, but I don't understand the IPv6 routing enough to debug this issue that deeply by myself. The matter server is this docker image: ghcr.io/matter-js/python-matter-server:stable, it is running as "network_mode: host" if that helps. Could this issue be avoided by preventing the TBR publishing addresses to my home network? Is there a config variable for that?

…

*zwx1995esp* left a comment (espressif/esp-thread-br#204) <#204 (comment)> Hi @littleant Is there any specific reason that you need the end device to access the TBR via the TBR on-link prefix address rather than a thread prefix address(ether OMR address or Mesh prefix address)?

[zwx1995esp]

Hi @littleant,

Thanks for the very detailed write-up and for verifying the workaround on real hardware — this is a great analysis to work from.

One correction on the root-cause direction: PIO handling for the on-link prefix isn't actually missing from the stack. It's implemented in lwIP itself (nd6_input() in components/lwip/lwip/src/core/ipv6/nd6.c), not in libopenthread_br.a — when the BR sends its own RA, it's looped back through the normal netif receive path and processed like any other received RA, including the PIO (on-link prefix) option. So the on-link-prefix logic your route-table symbol analysis was looking for does exist, just in a different layer than expected.

Given that, we'd like to reproduce this locally to see exactly where it breaks rather than guess further. Could you share the steps you used to reproduce it, as precisely as possible? In particular:

• How the BR and the host were both brought up (e.g. does the issue require a fresh BR boot / Wi-Fi reconnect, or can it also happen on a BR that's been running for a while)?
• Roughly how long after BR startup / reconnect you started seeing the fd17... addresses win source selection on the LAN host, and how long the outage lasted before you intervened?
• Whether it reproduces consistently on your setup, or only intermittently?

[littleant]

I'm unable to reproduce the issue right now. I flashed the old, unpached firmware onto the TBR and tried all combinations I could come up with, including getting a new IPv6 prefix from the ISP.

If I remember correctly, I never touched the TBR. No reboot, no flashing, nothing.
I did reboot the matter server, though. I don't know if the devices were unreachable right after the reboot. I think they became unreachable after some time?

It may have been 30min or so when I noticed that the devices were unreachable and I tasked the AI to find the cause. The AI ran for at least 19min before suggesting the command ip addrlabel add prefix fd17:xxxx:xxxx:xxxx::/64 label 99 to temporary fix the problem, which it did. (I did not add the prefix permanently.)

[zwx1995esp]

@littleant Thanks for your time on reproducing this issue. Ok, based your investigation now, the AI may not find the root cause. I think you can continue watching the device to see if the problem can be reproduced and I'm glad to help analyzing the root cause if you can provide some steps to reproduce this issue.