Merging latest Python Transport changes#63
Merged
Conversation
The _path_requests cull condition was using DESTINATION_TIMEOUT (now 1 week after Tier 1.A) which prevented stale outbound path request entries from ever expiring. Python uses a dedicated PATH_REQUEST_GATE_TIMEOUT of 120s for this table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matches Python Transport.set_network_identity / has_network_identity: the setter is no-op if a network identity has already been set, and the boolean check is exposed for callers gating network-destination creation. No call-site migration: _network_identity has no in-tree external readers; current internal uses inside Transport remain on direct member access. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an in-memory map of destination_hash to state (UNKNOWN / UNRESPONSIVE / RESPONSIVE) alongside the path table, with four accessors (mark_path_unresponsive, mark_path_responsive, mark_path_unknown_state, path_is_unresponsive) matching Python Reticulum. The state is marked UNRESPONSIVE in jobs() when a link establishment attempt times out against a path that was previously 1 hop away (and the receiving interface is not MODE_BOUNDARY), and is reset to UNKNOWN whenever a fresh announce installs or replaces a path entry. During announce ingestion, an announce with the same emission timestamp as an existing UNRESPONSIVE path now replaces that entry instead of being rejected as a duplicate, which lets a peer recover from a stale path without waiting for full expiry. State is RAM-only (matches Python; never persisted to microStore) and is garbage-collected in the existing tables-cull block when its destination is no longer in the path table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the synchronous, unthrottled drain of accumulated path requests at the end of jobs() with a bounded queue (default 32 entries via RNS_QUEUED_DISCOVERY_PRS_MAX) that is drained at a fixed rate of one transmission per DISCOVERY_PR_TX_THROTTLE interval (0.5s), matching the rate Python achieves with its sleeping background thread. Each queued entry now carries the receiving interface of the failed link as a blocked_interface, so when the destination was previously 1 hop away, the rediscovery request_path is fanned out across every other interface instead of the one that just failed. Entries with no blocked_interface (pending-link culls, "path now missing", and "local-client link" branches) still trigger a single unconstrained request_path. The accumulator in jobs() switches from std::set<Bytes> to std::map<Bytes, Interface> to carry blocked_if through to the queue. New PendingDiscoveryPREntry class and PendingDiscoveryPRs deque alias in Transport.h follow the existing container-naming pattern. handle_disovery_path_requests preserves the Python spelling so cross-references in either direction grep cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds class-level rx/tx byte totals and current bits-per-second rates to Transport, populated once per second from a new count_traffic() pass that snapshots each non-child interface's rxbytes/txbytes, computes the per-interval delta and rate, updates that interface's current_rx_speed/current_tx_speed, and folds the delta into Transport::_traffic_rxb / _traffic_txb. Child interfaces are skipped to avoid double-counting against their parent. State is RAM-only, matching Python. The per-interface snapshot lives on InterfaceImpl as five new fields (_traffic_counter_ts/_rxb/_txb, _current_rx_speed, _current_tx_speed) accessed through Interface getters and an update_traffic_counter(ts, rxb, txb) mutator. A new _traffic_check_interval (1.0s, gated in jobs()) drives the tick. The remote status payload now reports the new totals and rates directly: top-level rxb/txb pull from Transport's counters, top-level rxs/txs from the aggregate speeds, and each interface entry now carries per-interface rxs/txs. packMapSize values on both the per-interface map (11 -> 15) and the top-level map (5 -> 7) are brought into sync with the entries actually emitted; the prior values understated the count, exposed now by adding the new keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Changes Transport::_interfaces from std::map<Bytes, Interface> to std::vector<Interface> and adds prioritize_interfaces() which sorts the vector in place by Interface::bitrate() descending. With a hash-keyed map the container's iteration order was bytewise on the hash and could not be reordered; with a vector, sort once and every subsequent iteration in outbound, announce broadcast, discovery PR fanout, etc. sees the higher-bitrate interface first -- which is what Python's prioritize_interfaces achieves. Lookup paths that previously used map::find / map::count are routed through the existing public find_interface_from_hash, whose body now does std::find_if. is_interface_from_hash collapses to a one-line wrapper around it. register_interface dedupes by hash via find_interface_from_hash before push_back; deregister_interface uses erase-remove. The seven iteration sites in Transport.cpp plus one each in Reticulum.cpp and Provisioning/BuiltinNamespaces.cpp drop the structured-binding pair and bind directly to Interface. prioritize_interfaces() is invoked from start() after interface setup and once per jobs() tick, matching Python's call sites. The sort is O(n log n) over a tiny vector (typical n <= a few), well under the per-tick floor. Interface::bitrate(uint32_t) is promoted from protected to public to match the adjacent public mode(uint32_t) setter and enable runtime reconfiguration plus test access. The lookup cost change from O(log n) to O(n) for hash queries is dominated by per-packet crypto on this stack, so it does not impact realistic throughput. A new test_transport unit test (test_prioritize_interfaces) registers two interfaces with distinct bitrates, calls prioritize_interfaces, and asserts the higher-bitrate interface sorts first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-path random_blobs collection on DestinationEntry was an unbounded std::set, so it grew without limit and its byte-sorted iteration order discarded the information needed to know which blob was newest. Switches it to std::vector with newest at the back, capped at MAX_RANDOM_BLOBS via an erase-from-front trim when the limit is reached. Adds timebase_from_random_blob / timebase_from_random_blobs which extract the 5-byte big-endian emission timebase from offset 5 of a random_blob and return the max across the path's stored blobs. Inbound announce processing in the equal/less-hops branch now requires the announce_emitted timebase to exceed the path's known max blob timebase, in addition to the blob being unseen. An announce carrying a fresh blob but an older emission than what we've already accepted for the destination is now rejected, matching Python's replay-prevention semantic. DestinationEntry's storage codec keeps the same byte layout (count + length-prefixed blob sequence) so existing persisted entries decode without migration. The encoder additionally caps written blobs to PERSIST_RANDOM_BLOBS, taking only the tail-newest slice. The ArduinoJson-style serializer in Utilities/Persistence.h is updated to deserialize the field as std::vector as well, so the two codecs stay in sync. Test fixtures in test_persistence and test_rns_persistence updated their blob literals from std::set to std::vector. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the custom flat-binary codec for DestinationEntry with a MsgPack positional array of seven elements (timestamp, hops, expires, received_from, random_blobs, receiving_interface, announce_packet). The encode path uses MsgPack::Packer; the decode path uses MsgPack::Unpacker with bin_t<uint8_t> for binary fields. Library is the same hideakitai MsgPack@0.4.2 already used by Resource and Transport remote-status payloads. The motivation is schema evolution: appending future fields (for example a persisted path-state byte) becomes a one-line outer array length bump where old decoders simply read fewer elements rather than misinterpreting bytes. Endianness is also now explicitly handled by MsgPack rather than relying on native double / uint16 layout. Storage cost is ~+6 bytes per entry (~1-2%): four MsgPack type tags plus one extra byte for binary length prefixes. The encoded size of the test_codec_destination_entry fixture went from 97 to 103 bytes. Per-blob cost is identical (12 bytes either way), so the overhead does not scale with random_blobs count. Migration is by reset-on-upgrade: existing persisted segments will fail to decode under the new codec (decode() returns false cleanly), and the path table starts empty after the first boot post-upgrade. Path persistence is a runtime cache, so the table refills naturally from inbound announces within minutes. No version byte or magic sentinel is introduced. The PERSIST_RANDOM_BLOBS tail-trim on encode is preserved, as is the post-decode "announce was effectively re-received so bump hops" behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a per-Transport list of blackholed identity hashes that
blocks path-table population and provides an opt-in RNS-published
list. Each entry carries the source identity hash that authored
the blackhole, an optional unix-timestamp expiry, and an optional
human-readable reason.
Methods on Transport:
blackhole_identity(hash, until=0, reason="") -- add + persist + purge
unblackhole_identity(hash) -- remove + persist
is_blackholed(hash) -- O(log n) lookup
reload_blackhole() -- read persisted state
remove_blackholed_paths() -- scan + drop paths
persist_blackhole() -- write local file
blackhole_list_handler(...) -- RNS /list responder
Storage is a single bulk file at {storagepath}/blackhole_local,
serialised as a msgpack map of identity_hash -> [source, until,
reason]. Only entries whose source matches this instance are
persisted, leaving future multi-source ingestion safe to layer on.
Cross-platform string handling routes through MsgPack::str_t,
which aliases to std::string on native and Arduino's String on
embedded, then normalises to std::string at the boundary so the
internal type is stable.
The /list request handler endpoint is wired through the existing
_blackhole_destination scaffolding, now gated on
Reticulum::publish_blackhole_enabled() (new flag, defaults to
false). reload_blackhole() runs unconditionally during start() so
persisted entries take effect even if publishing is disabled, and
remove_blackholed_paths() then purges any path-table rows whose
associated identity is on the list. A 60s expiry sweep in jobs()
drops entries whose 'until' has passed.
// DIVERGENCE: announce ingestion now consults the blackhole list
inline. Python's Transport.py:313-337 only filters when loading
the path table from storage at startup; new announces from a
blackholed identity still populate the path table and stay there
until the next manual blackhole_identity() or reload_blackhole()
call. On a leaf node that is rarely-supervised this leaves the
blackhole effectively ineffective between purges, so the C++ port
adds an inline check in inbound() right before the path-table put:
if the announce's recalled identity is blackholed, drop the
should_add flag.
Multi-source ingestion (Python's per-source files under
blackholepath/, the BlackholeUpdater thread, and the
blackhole_sources allow-list) is intentionally deferred. The
persistence format does not preclude adding those later.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
detach_interfaces() now performs the clean-shutdown sequence: tear down every active and pending link so peers see the connection drop, drain for 150ms to allow teardown packets to leave the wire, then call detach() on every interface so it can release resources (sockets, hardware, worker threads) before destruction. Wired into exit_handler() so it runs alongside persist_data() during Reticulum shutdown. InterfaceImpl gains a default-empty virtual detach() hook; an Interface::detach() accessor on the value type calls through. Subclasses that own external resources can override; default is a no-op so existing interface implementations need no change. drop_announce_queues() empties each interface's announce_queue list and logs the count. Useful for clean shutdown without flushing buffered outbound announces, or when an interface is being deregistered. shared_connection_disappeared and shared_connection_reappeared remain stubs and are not invoked anywhere -- microReticulum does not support being a Python-style shared-instance client or server, so those state-resetting hooks have no caller. The Python reference is preserved as /*p */ blocks for future reference; when an external stack-sharing interface (e.g. WebSocket) is added it can decide whether equivalent reset semantics are needed. Python's detach_interfaces does additional ordering between LocalServerInterface and LocalClientInterface for its shared-instance protocol; without those interface types in this codebase the discrimination collapses to a single uniform loop over _interfaces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Uncomments the dormant block in Transport::start() that registers two IN/SINGLE destinations under a configured network identity: network.instance.<network_identity_hash> (specific to this node's participation) and network (shared across the named overlay). Both are added to the management-announce rotation so peers can discover members. The block was previously held in a /*p ... */ comment pending the network_identity accessors, which landed earlier. With those in place plus the `has_network_identity()` helper, the destinations can now actually be created when an application sets a network identity prior to Transport::start(). Python guards this with `not is_connected_to_shared_instance` so a shared-instance master can own these destinations on a client's behalf. microReticulum does not support being a shared-instance client, so that guard is dropped and the registration is gated only on has_network_identity(). If no network identity is set, has_network_identity() returns false and this block is skipped, preserving existing behavior for applications that don't use the network-identity overlay. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A side-by-side audit of every Transport mode-conditional branch against the Python reference found two real semantic drifts; the rest of the mode logic (link cull non-BOUNDARY filter, AP/ROAMING path-expiry TTLs, announce-outbound AP/ROAMING/BOUNDARY broadcast filters, roaming same-interface PR suppression) matches. 1. DISCOVER_PATHS_FOR was missing MODE_ROAMING. Python's Interface.DISCOVER_PATHS_FOR is the set of interface modes for which Transport.path_request() will initiate a search for an unknown destination. Python includes MODE_ROAMING in that set; the C++ value was MODE_ACCESS_POINT | MODE_GATEWAY only, meaning roaming-mode receivers silently dropped path discovery for unknown destinations they could otherwise have resolved. 2. PATH_REQUEST_RG (1.5s) and the associated retransmit-timeout grace were absent. When responding to a path request on a roaming-mode interface Python adds 1.5s to the retransmit timeout so better-connected peers have a chance to answer first. Without it a roaming-mode node answers as fast as any other peer, defeating the prioritization. Adds the PATH_REQUEST_RG constant in Type::Transport, includes MODE_ROAMING in DISCOVER_PATHS_FOR, and adds the conditional grace addition in the path-request response timing branch. A minor stylistic difference (Python's `not hasattr(iface, "mode")` vs C++'s `mode() == MODE_NONE`) is functionally equivalent and left as-is. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A side-by-side audit of every msgpack-serialised wire format against the Python reference (remote_status, remote_path table, remote_path rates, blackhole list, plus all local-file umsgpack-packb sites) found one real drift: the /list request handler was emitting each blackhole entry as a positional 3-element array [source, until, reason], but Python's Transport.blackholed_identities is a dict-of-dicts where each value is itself a map with named keys "source", "until", and "reason". A Python client decoding the C++ /list response would have gotten an array and tripped on field access. Switches the per-entry serialization to a 3-key map matching the Python wire format exactly. The on-disk blackhole_local format (written by persist_blackhole, read by reload_blackhole) is left as the compact positional array since local persistence is not a Python-interop surface in this codebase. Other wire formats in this codebase (remote_status_handler interface stats, remote_path table command, remote_path rates command) already use named maps with keys that match Python's get_interface_stats / get_path_table / get_rate_table emissions and required no change. The IDX_* positional indices in Python's Transport.py are purely internal abstractions over its list-typed table values; they never reach the wire. Tunnel wire format remains out of scope until B.8 lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the commented Python reference block in Transport::inbound() with the real C++ enforcement. When the receiving interface has announce_rate_target > 0 (opt-in by an interface implementation), the announce_rate_table is consulted on every non-PATH_RESPONSE announce: the destination's inter-announce interval is compared against the target, a violation counter is incremented or decayed accordingly, and once the count exceeds announce_rate_grace the destination is blocked for target+penalty seconds. rate_blocked then suppresses insertion into the announce table for rebroadcast, matching Python Transport.py:1838-1866. Three new fields on InterfaceImpl back the opt-in: _announce_rate_target, _announce_rate_grace, _announce_rate_penalty, with public getters and setters on Interface so a concrete interface subclass (or its configuration) can enable rate limiting without touching the Transport. Defaults are 0 (disabled), so existing interfaces inherit the prior unlimited behavior; only interfaces that explicitly set a target are affected. The RateEntry container and MAX_RATE_TIMESTAMPS macro were already present in the codebase from earlier work, so this commit only wires the new fields and replaces the inert /*p ... */ block with the corresponding C++ implementation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After every successful broadcast transmit in Transport::outbound(), the receiving interface is now notified via two new default-empty virtual hooks on InterfaceImpl: sent_announce() fires when the packet's type is ANNOUNCE, sent_path_request() fires when the destination is PLAIN and the packet was sent via Transport::request_path(). Matches Python Transport.py:1323-1324 which calls these on every interface.broadcast transmit. A new is_outbound_pr flag on Packet (matching Python's packet.is_outbound_pr) is set inside Transport::request_path() just before packet.send() so outbound() can distinguish a path request from any other PLAIN packet at transmit time. The hooks default to no-ops; subclasses that want to maintain per-interface frequency/burst counters (Python's oa_freq_deque and op_freq_deque) can override. This restores the per-interface stat-tracking surface that the audit found missing without prescribing what stats a particular interface implementation must keep. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a DATA-to-LINK packet arrives on an interface that does not match the link's attached_interface, Transport::inbound() now removes the packet hash from the dedup filter so the packet can still be received when it later arrives on the correct interface. Matches Python Transport.py:2141-2150. Without this branch, an interface failover (the link's attached interface is partly malfunctioning and the packet has been routed via another path to the same destination_hash) would be dropped as a duplicate when it finally arrives on the correct interface, because the hash was already inserted on the first arrival. Also adds an explicit break after the matching link_id is found so we don't continue iterating after the decision is made. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a LINKREQUEST arrives at Transport::inbound() for a local destination, the receiving interface now clamps the proposed path MTU before the destination parses the request. Matches Python Transport.py:2099-2118. The link request carries the initiator's proposed path MTU in its trailing LINK_MTU_SIZE (3) bytes. If our receiving interface asks for an autoconfigured or fixed MTU and the proposed path MTU exceeds our HW_MTU, the trailing bytes are rewritten by Link::signalling_bytes(nh_mtu, mode) so the destination's parser sees the clamped value. If the receiving interface has no HW_MTU configured, the trailing MTU bytes are stripped entirely so the destination sees a plain link request without MTU negotiation. Errors during the clamping step drop the packet. All the helpers required for this logic (Link::mtu_from_lr_packet, Link::mode_from_lr_packet, Link::signalling_bytes, the Type::Link::LINK_MTU_SIZE constant, and AUTOCONFIGURE_MTU / FIXED_MTU / HW_MTU accessors on Interface) already existed in the codebase; the change just wires them at the right spot in inbound() rather than deferring the work to destination.receive(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Interface subclasses can now publish per-packet signal-quality stats from the radio hardware via three new fields on the base InterfaceImpl (_r_stat_rssi, _r_stat_snr, _r_stat_q), each defaulting to NaN as a "not present" sentinel. The driver populates these synchronously with handle_incoming(bytes) so the values describe the same packet whose bytes are about to be delivered to Transport. Transport::inbound() now snapshots the three values onto the Packet at construction time -- the commented Python reference block has been replaced with three lines that copy each non-NaN metric onto the corresponding Packet field. Packet exposes rssi(), snr(), and q() getters plus fluent setters; the impl-side fields existed but weren't reachable through the value-type API. Python keeps three class-level deques (local_client_rssi_cache etc.) that map packet_hash -> value so shared-instance clients can look up signal stats by hash via RPC. microReticulum does not support being a shared-instance client, so the cache and the corresponding Reticulum::get_packet_rssi/snr/q lookup API are intentionally omitted. Consumer callbacks have the Packet directly and read packet.rssi() from it. This is the baseline implementation. Future extensions could populate Link::_rssi / _snr (already declared on LinkData) from the packet on Link::receive() and add per-interface RSSI/SNR/Q keys to the remote_status_handler payload so tools like rnstatus can display signal quality -- both deferred until a concrete consumer needs them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When Link::receive() processes an inbound packet, the packet's stamped signal-quality stats are now snapshotted onto the link impl's existing _rssi / _snr / _q fields so consumers can read the link's last-received signal quality without holding the original Packet. Mirrors Python's Link.py:257-258 (which keeps .rssi and .snr on the link object) plus the Q metric. The fields' defaults are switched from 0.0 to NaN as the "not present" sentinel, matching the convention established in the Packet and Interface layer. A NaN value means either no packet has been received on the link yet, or the source packet didn't carry that metric (e.g., link traffic over a non-radio interface such as UDP). No public accessors on Link are added yet -- the snapshot happens inside Link::receive() with direct impl access, and no in-tree consumer reads link.rssi/snr/q. Public accessors can be added when a concrete consumer needs them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The remote_status_handler per-interface entry now carries three additional msgpack keys -- rssi, snr, q -- sourced from the interface's r_stat_* fields populated by hardware drivers. packMapSize bumps from 15 to 18. Each metric is emitted as float64 when the receiving interface has reported it, or as nil when not present (NaN sentinel). A consumer can distinguish "not reported by this interface" (nil) from "reported but zero" (float 0.0). The bitrate field uses the same nil-on-absent pattern. These keys are a microReticulum-specific extension; Python's get_interface_stats doesn't emit them. Forward-compatible by construction -- Python clients (e.g. rnstatus, nomadnet) just ignore unknown keys when decoding the map; clients that explicitly look for the keys gain visibility into signal quality on radio interfaces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the path-table pattern (PathStore + NewPathTable):
- New Persistence/IdentityEntry.{h,cpp} hosts the value type, microStore
Codec specialization, and KnownStore / KnownDestinations aliases.
- Identity owns a static KnownStore + TypedStore pair, initialised from
Transport::start() when RNS_USE_FS and RNS_KNOWN_DESTINATIONS_PERSIST
are both set; falls back to BasicHeapStore otherwise.
- Capacity is bounded by microStore policy_max_recs, so the bespoke cull
loop (including its bad_alloc fallback) and the periodic save/load
scaffolding go away. Identity::persist_data and exit_handler are gone
with their last callers in Reticulum::persist_data and clean_caches.
- Provisioning exposes KnownDestinationsMaxsize / SegmentSize /
SegmentCount knobs on the existing TransportConfig namespace.
Same pattern as the path and known-destinations stores: a static HashlistStore (BasicFileStore when RNS_USE_FS and RNS_PERSIST_HASHLIST are set, BasicHeapStore otherwise) replaces the GenerationalSet<Bytes>. - insert/contains/erase call sites move to put/exists/remove with raw pointer + size, since the hashlist stores keys only (empty value). - Capacity is enforced by policy_max_recs; hashlist_maxsize() now forwards into set_max_recs. - The write_packet_hashlist() stub, the legacy packet_hashlist file load block in start(), and the matching unlink in clear_storage() are removed; microStore persists incrementally and clears its own segment files. - Provisioning gains HashlistSegmentSize / HashlistSegmentCount knobs alongside the existing HashlistMaxsize entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.