Password reset: reset the Identity admin, not just the legacy row (ON HOLD - decide whether to keep) - #1104
Open
tvancott42 wants to merge 63 commits into
Open
Password reset: reset the Identity admin, not just the legacy row (ON HOLD - decide whether to keep)#1104tvancott42 wants to merge 63 commits into
tvancott42 wants to merge 63 commits into
Conversation
Unchecking "an agent collects for this site" on the main site left the console still dialing the agent tunnel, so every console read failed once the agent was stopped. SiteTunnelRouting already asked whether the default site had been handed to its agent before honoring devices.via_agent; the console's own reader never did, and answered from the stored flag alone. The flag is deliberately kept rather than cleared when coverage goes off, so turning coverage back on restores the operator's choice - which is exactly why the flag on its own cannot decide this. IsConsoleViaAgentAsync now asks the same question its device-side counterpart does. Removing a site's last agent clears both routing flags, for any site rather than just the main one. They name a tunnel, so with no agent left they can only point at something that will never answer, and they outlived the agent on every path that removed one. SiteTunnelRouting gains an Invalidate so the cleared flag takes effect immediately rather than after its one-minute cache expires.
The /sites card read "0/1 agents online" for a site whose only agent had been deliberately disabled, and the Agents tile on the overview card counted it while Agents Online never could. Both reported a site as missing an agent it had been told to stop using. Disabled agents now drop out of both halves of the per-site count and out of both overview tiles. A site whose only agent is disabled reaches 0/0, which the markup already omits rather than rendering a count of nothing.
Adding the routing cleanup to DeleteAgentAsync gave AgentEnrollmentService two more dependencies and left the test that constructs it directly on the old signature, so the test project stopped compiling. The app itself built and deployed fine, which is exactly why it went unnoticed - src builds clean and only the test assembly breaks. The same empty service provider already used for coverage backs the routing cleanup here, so removal logs that it could not tidy the flags and carries on. That is the behavior worth having under test: tidying is best-effort and must never fail the removal.
Gating IsConsoleViaAgentAsync on agent coverage fixed the routing but broke the teardown hooks that shared it. Those ask whether the live client rides the tunnel; the setting answers whether the console should route that way from here on. The two diverge the moment coverage is switched off with a tunnel-routed console still connected, and the hooks then declined to tear anything down: the client stayed connected against a loopback proxy whose tunnel had died, with no way back, because every automatic reconnect is gated on being disconnected. Each console call then dialed the dead proxy and paid the full retry backoff. The connect paths now record how they built the client, and all three hooks ask that instead. This also closes a race in agent removal, where clearing the routing flags could beat the tunnel's own teardown to the same check. Changing coverage reconnects the console, so the switch takes effect on the existing connection rather than only on the next one - a console parked in awaiting-agent had nothing to move it. Not awaited: a reconnect takes seconds and this runs from a checkbox. Settings reads the SSH routing hint through the gated reader too. Read raw, it claimed tunnel routing on a main site whose agent no longer collects for it, contradicting the console hint beside it.
The default site was skipped when refreshing whether it has an enrolled agent, so its enrolled flag stayed false forever and only a LIVE tunnel counted as an agent being present. A main site handed to its agent therefore kept collecting whenever that agent was merely offline - and with its devices routed through the tunnel, every one of those polls dialed a loopback proxy with nothing behind it and came back as SSH.NET's "no identification string". A secondary site counts an enrolled agent whether or not it is connected and stands down cleanly, which is why an agent-down secondary site has always looked right and this did not. With coverage off the answer is false either way, so an install that has not opted in takes an unchanged path. Ticking "console via agent" now reconnects the console. That checkbox only appears once coverage is on, so coverage is necessarily switched first, and reconnecting there alone always ran against the console's old routing - the choice went unapplied until something else happened to reconnect, which is how a console configured for the tunnel stayed on the direct path with no error shown. The devices flag drops its cache when it changes, and when coverage changes, since coverage gates the answer without the flag itself being touched. It is consulted per SSH command and per modem poll, so the switch appeared to do nothing for up to a minute. Gateway SSH's awaiting-agent message no longer says SQM will connect once the agent is online. It is returned for every gateway SSH use, so naming one feature read as a non-sequitur in Test SSH Connection.
Reverts the collection half of 89d1365. Letting the default site count an enrolled-but-offline agent made it stand down like a secondary site, which took the site dark until the agent returned. The fallback is the better behavior here: a server that can still reach the network it monitors should keep collecting rather than go quiet because an agent it was handed to is down. The other three fixes in that commit stand - they were about settings taking effect, not about who collects.
…line" This reverts commit 5d3d98e.
…fline" This reverts commit 21e6d6f.
…wrong Gateway SSH already answers with the awaiting-agent message when the site's devices are tunnel-routed and no agent is online. The shared device SSH path did not, so it dialed the loopback proxy with nothing behind it and surfaced SSH.NET's 'no identification string' - true of the socket, useless to the reader. It is what qmicli cellular modem polling rides on, which is where it showed up. The cable modem, ONT and Starlink pollers are HTTP or gRPC and never had this shape.
The push path refuses to send targets to a main-site agent unless the site has been handed to it. The write path had no such check, so it recorded whatever arrived. Switching coverage off stops the config going out but does not stop an agent that already has targets: it keeps probing and pushing while the server resumes probing the same targets itself. Both then write one series at different cadences - identical means, mismatched sample rates - which renders as a sawtooth rather than as obvious duplicates. Needs an agent mid-probe when coverage flips, so it takes toggling to reach. Not paired with pushing a stop to the agent. That would turn a transiently false coverage read on a healthy site into a cleared target list and a real data gap, which is a worse trade than some wasted pings that are now discarded anyway.
…repo The Reverse proxy section jumped straight into hand-rolled Traefik/Caddy/nginx config without mentioning NetworkOptimizer-Proxy, which ships the agent tunnel route enabled by default. It also never stated plainly that the proxy is a prerequisite rather than optional polish, so a reader running the app on a bare LAN address had no cue that TLS termination was theirs to stand up. Also preserve the original Host on the Caddy gRPC route. Optional in practice - the tunnel service does no host-based routing and gRPC is exempt from canonical redirect enforcement - but it keeps the example explicit.
Unlike Traefik and Caddy, nginx sends the upstream address as the Host by default on both proxy_pass and grpc_pass, so the examples were the odd ones out. It is cosmetic on the gRPC route - the tunnel does no host-based routing and gRPC is exempt from canonical redirect enforcement - but it matters on the app route, where enforcement is active and keys on Host, and an install following the example verbatim would drive that off 127.0.0.1:8042.
…he proxy The variable is named in the opening paragraph as where the agent's server URL comes from, but nothing says to go set it once the proxy exists. That is the gap operators fall into: the proxy comes up, the Agents panel still shows a placeholder because the app has no canonical address, and they substitute the app's own LAN address - which the agent then dials on 443, where nothing is listening. Adds a short step at the end of the Reverse proxy section with the health-check curl to run before enrolling.
Publishing incrementally over a warm tree - no compilable change since the last build - recreates package content folders such as LatoFont EMPTY rather than leaving them alone. WiX then harvests the empty folder and packages an MSI that is missing files, with no warning and a build that reports success. The v2.5.3 MSI lost its 19 Lato font files this way, and the only reason it was caught was comparing the artifact size against the published asset. The release flow does not normally hit this, since the MSI is built once right after a merge that changed code. It does hit a rebuild after a failed upload, and a docs-only or template-only release, where nothing needs recompiling. Removing the publish folder first forces the publish target to repopulate it. The build output is untouched, so the cost is a file copy rather than a recompile. Verified against the exact failure condition: a second build on a warm tree now produces a byte-identical file table to a known-good MSI.
Nothing invokes the standalone cfspeedtest binary any more - uwnspeedtest superseded it for gateway WAN tests, and the app resolves that one (UwnClientRunner). There are no references to cfspeedtest in C#, Razor, or the WiX authoring at all, so every macOS install and update was spending a Go cross-compile and ~5.5 MB on a binary that is never run. The src/cfspeedtest module stays: uwnspeedtest imports its speedtest package (src/uwnspeedtest/go.mod), and both Dockerfiles copy it for that reason. Only the standalone binary build is dropped, and it is commented rather than deleted in case it is wanted again.
The DownloadTraefik target only fired when traefik.exe or a template was missing, and the script then skipped any template that already existed. Both gates were presence checks, never freshness checks, so the first MSI build on a machine staged the templates permanently. This build box shipped five-month-old templates that way, missing the multi-site agent tunnel route the companion repo had added in the meantime, and nothing surfaced it. Templates now re-download every build. They are a few KB and track Ozark-Connect/NetworkOptimizer-Proxy, so they are the part that drifts. The pinned 170 MB binary is skipped when already staged, which it previously was not once the target fired, so a build costs less network than before rather than more. Each template downloads to a temp file and is moved into place only on success, so a partial fetch cannot truncate a good staged copy. A failed fetch with a copy already staged warns and continues, keeping offline builds working; with nothing staged it stays fatal. Verified end to end: the target fires during the WiX build, skips the binary, re-fetches both templates, and the resulting MSI is byte-identical in file table to the published v2.5.3 artifact.
The coverage flag cached with a one-minute expiry, and the synchronous reader answered false on a miss while it refilled. Every expiry therefore opened a window in which a covered site read as uncovered: probes resolved to the local executor, device routing dialed direct instead of through the tunnel, and a console connecting in that window connected direct and stayed there. Harmless on a server sitting on the network it monitors. On the off-site server this feature exists for, it means probing from the wrong network and writing the result as that site's, plus dialing the site's RFC1918 addresses on the hosting provider's network. Every writer already calls Invalidate, so the expiry bought nothing. The cache now holds until invalidated and is warmed for all sites at startup, which closes the window rather than shortening it. What it gives up is noticing a value changed in the database behind the app's back - an operator editing SQLite by hand - and a restart settles that.
The cached flag no longer expires, so it now outlives the site that set it. A slug deleted and re-created would inherit the previous site's coverage answer until the next restart - the same shape as the per-site registries that already get swept, and previously hidden by the one-minute expiry. SiteAgentCoverage joins that sweep, which runs on removal and on creation. Nothing to tear down, so it evicts and returns null.
The handler returned silently when the WAN interface list was empty, so with the console unreachable the button did nothing at all - no error, no state change. The list comes from the console, and this is the one control on the page that depends on it, so it is also the one that has to explain itself. Run Test from Agent does not need the list, which is why it behaved. It distinguishes waiting for the site's agent from the console simply being unreachable, because those need different things from the reader.
…g there One question gated both, and they are not the same question. SNMP reads the device's own counters, identical whoever asks, so the server carrying on while the agent is offline is a real fallback - the site keeps its device history instead of going dark. A probe measures the path FROM whoever runs it. The server running one for a site its agent covers describes the server's route, not the site's, and stores it under the site's name regardless. On an off-site server that is a different network. The upstream tracer had the same shape, falling back to the local executor and tracing from the wrong place. Probing and tracing now stand down on the configuration alone and let the probe fail while the agent is away: a gap is honest, a number from the wrong vantage is not, and mixing the two is what put the sawtooth in the charts. SNMP call sites are unchanged.
Switching agent collection on reconnected the console immediately, and that reconnect read the flag through the synchronous reader - which answers false while an invalidated entry refills. So the console reconnected on the direct path and showed no waiting-for-agent banner, on a site that had just been handed to its agent. Probing was unaffected, which is what made it look inconsistent: the latency tier reads the flag on its own cadence, by which time the background refill had landed, so it stood down correctly. Same flag, different timing, different answer. The writer knows the value it just stored, so it records it. Invalidate stays for callers that only know the value changed - site eviction.
Clearing them on every site strands a secondary one. That site is reached ONLY through an agent, so the flags describe its sole access path rather than an option it took - and the replacement agent does not restore them, because the setup wizard writes them only when its proxy checkbox is ticked and that defaults off. Swap a secondary site's agent and it would sit on direct routing it cannot use, with generic connection failures rather than the waiting-for-the -agent messages, which need the flags set in order to fire. The main site keeps the clearing: direct access is a real fallback there, which is what made removing the flags right in the first place. Never released - the clearing landed after the v2.5.3 tag.
Both comments described the old behavior, where the gate asked whether an agent was connected. It asks about configuration now, so a secondary site offers Add Target and Discover before any agent exists and the probe fails with a reason. That is the intent rather than an oversight, so it is written down: a control that explains itself beats one that is silently absent, and targets added now are seeded ready for the agent that arrives later. Comment only.
The Site ID hint slugged the typed name inline, which ignores what is already taken - a name colliding with an existing site, or with the reserved 'main' slug, promised an ID the site would not get. PreviewSlugAsync already answers this (it runs the same generator creation does, suffix and all) and had no caller; the hint now asks it.
…ont of it (#1096) IAuditQueryService carried no gate at all. The surface was covered in practice - the export endpoints have RequireAuthorization(RequireAdmin) and the Audit Log tab sits behind an AuthorizeView on the main site - but by the endpoint and the page rather than by the service, which is the arrangement [MutatingService] exists to replace. Any new caller reaching this interface, a component on another page or a background job or a future endpoint, would have inherited nothing. That matters more here than for most reads. The log is the record of who did what across the whole install: actors, source addresses, target names, and since the site stamping work, the site each action touched. Reading it is closer to reading a credential store than a status page. Every member gets [RequireRole(Roles.Admin)] and the registration moves to AddMutatingService so the implementation is not resolvable ungated. No [AuditAction]: recording every read would write an entry for each page and each page-turn of the log itself, burying the actions the log is kept for. Both callers keep working. CallerContextMiddleware populates the caller for the export endpoints and CallerContextCircuitHandler does it for the interactive page, so the proxy has a principal on either path. Tests cover Admin allowed, Viewer and Operator refused, both exports refused separately from the page read - they leave as files through their own endpoint, so a gate covering only the interactive read would have missed the larger disclosure - and a reflection check that the attributes stay on the interface.
Both endpoints answer with Content-Disposition: attachment, so nothing was ever going to render in the new tab - it opened, received a download, and stayed behind empty. A same-tab link with the download attribute gets the file without navigating away from the log.
Monitors every WAN a site has, not just the primary one. 189 commits squashed. A WAN vantage names the WAN it measures and how its probes get there: bound to a source IP the gateway policy-routes out that WAN, or run by an agent sitting behind it. An agent on the gateway can bind probes to the WAN's own interface, so no policy-based route is needed there at all. Latency points carry a wan tag, upstream discovery runs per WAN, and ISP Health grades each WAN against its own counters, plan speeds, targets and hops instead of pairing one WAN's traffic with another's expectations. WHERE IT SHOWS UP Network Performance gains a vantage card and a WAN filter on Latency & Packet Loss: one WAN alone, several compared, or all. Shared hosts probed from every WAN line up side by side, one color per host and one line pattern per WAN. Latency Targets and Flaky Monitoring Targets both name the WAN a target belongs to. Upstream Path Discovery runs per WAN and opens on the one selected. Live View takes its own WAN selector with its own saved selection, so watching one WAN never moves the analysis views. Network Tools can run a probe from any vantage, which is how a policy-based route or interface bind gets confirmed. Live and analysis link both ways, carrying the moment, the WAN and the category, without either view's saved filter being written to. ISP HEALTH SCORING Several changes here are fleet-wide and affect single-WAN sites too. Loaded latency no longer answers "was any sample high", which one ICMP-deprioritized responder answers on its own. Samples are collapsed by instant across every non-LAN target on the WAN, since a queue on the access link sits in front of all of them. Magnitude comes from the targets that saw it and only the credence scales with the cohort, so the figure does not fall as more targets are monitored. A line that stays clean under load now reads as clean rather than as an absence of evidence, and a run of clean load episodes after elevated ones is read as the line having been fixed. Evidence is weighed by recency, by how loaded, and by how sustained. A WAN speed test can raise the figure where it read higher and reached 70% of plan in that direction, since the probes sample on their own cadence and a short event's peak queue can build and drain between two of them unseen. Off-path access hops stop feeding the packet loss pool, unless they are all a site has. Target edits and upstream discovery invalidate every WAN's cached report, not just the primary's. A report computed before the agent's console was up is dropped, since SNMP arrives through that console and is what classifies load. STARLINK Satellite is inferred from the access ASN, satellite hops are kept out of the candidate set, and the dish's fixed LAN-side MAC resolves to Starlink. Latency bands are set from measured dishes rather than estimated. Idle: 23 ms is the best the medium does at all and scores full, 42 ms is where a healthy Backup dish sits and scores 80. Loaded: 3 ms excellent, 12 ms acceptable, replacing a 25 ms ceiling that sat above the 95th percentile of real delta and could not fail anything. Loaded loss is unchanged. A dish reporting a reduced-speed plan tier is graded on whether it carries usable traffic rather than on ratio, whatever plan is configured. Latency is deliberately not tier-aware: a cheaper plan really is worse latency, and hiding that would never tell anyone the tier is the reason. METERED WANS Continuous probing costs about 5.4 GB a month at 25 targets on the 10s default, which is most of a small satellite or cellular plan. Satellite, cellular and fixed wireless each cost a rung, and a configured Data Usage cap costs another. A rung means fewer targets and a slower cadence, never smaller packets. The allowance rotates across access hops, transit and path endpoints so a site is not left with detail on its own first mile and no way to tell whether anything it reaches is up. SINGLE-WAN SITES Nothing changes. With no vantages and one WAN, none of the selectors render, the schema additions stay null, latency points carry no wan tag, and the scoped queries resolve to the same inputs as before, pinned by equivalence tests. AGENTS AgentProtocol gained an optional supports_source_bind capability and traceroute binding compiles into the agent, so test agents need new binaries. Older agents keep working: the capability reads as "did not say" and interface binding is not offered for them. ROLLBACK Additive nullable columns only, and the data migrations are one-way by design. An older build ignores the new columns, reads the normalized keys correctly, and loses multi-WAN behavior rather than data. STILL TO BUILD FOR THIS RELEASE Per-WAN outage alerting. Alerting still fires per target, so a secondary WAN going down announces itself as a handful of "target is down" alerts rather than one outage on that WAN. The design is settled and intended to ship with this work, not after it: without it a multi-WAN site gets more noise from a WAN failure than a single-WAN site did.
…y turned on (#1100) * Config Optimizer: catch Smart Queues that UniFi Network never actually turned on UniFi Network regularly accepts the Smart Queues toggle on a WAN without provisioning the queues. The setting reads as on, no shaper is ever created, and the connection runs unshaped with nothing on screen to say so - the user just sees a lopsided upload. Until now we only told anyone about it at the moment an Adaptive SQM deploy failed on the missing IFB device, so nobody who wasn't deploying Adaptive SQM ever heard it. Config Optimizer now finds it on its own. For every WAN with Smart Queues enabled it reads the gateway's traffic control over SSH and raises a Performance Suggestion when the htb root class isn't there, prescribing the QoS-rule workaround that un-wedges it. Interface resolution is the controller's, unchanged: the WAN's uplink_ifname - eth6 plain, eth6.100 VLAN-tagged, ppp0 for PPPoE - plus its ifb companion, exactly the devices Adaptive SQM and Monitoring already use. Egress (upload) rides the WAN interface, ingress (download) rides the ifb, and a WAN shaped in only one direction is reported naming the direction that isn't. Everything goes out in one SSH round trip, and the check stays silent whenever it can't see the answer: no WAN with Smart Queues on (no SSH at all then), gateway SSH off or without credentials, an agent tunnel that isn't up, a failed command, a truncated readout, or a WAN interface the gateway doesn't have. A direction UniFi was told to shape at 0 isn't expected to have a shaper either. Adds SmartqUpRateMbps to WanInterfaceInfo, which is what tells those two apart. Closes #1083 * Cover the shaper probe's preconditions Everything the probe refuses to do is the part that matters: a site with no WAN on Smart Queues, a gateway with SSH off or no credentials, an agent tunnel that isn't up, a failed command, and an interface name that has no business on a command line all have to cost nothing and produce no state. A finding raised from a failed read would accuse a healthy install. * Only read the gateway's shapers when a WAN actually has Smart Queues on Asking which interfaces to read costs a second device-list fetch from the controller, and an install with Smart Queues off everywhere can never produce the finding that would pay for it. The network configs this run already fetches say whether any WAN has it enabled, so waiting on that one small call buys the skip - the rest stays parallel. * Tour step for the Smart Queues check, and one QoS menu path everywhere The step only reaches installs it can mean something to: a new "smart-queues" predicate, which is UniFi's own toggle on some WAN and deliberately not the existing "sqm-enabled" (that one is our Adaptive SQM, and the whole point of this check is the WAN that has UniFi's Smart Queues on with nothing of ours deployed). Nothing stores UniFi's toggle locally so the predicate asks the console, which is affordable only because predicates resolve for a tour that is actually due, never on an ordinary Dashboard visit. The anchor sits on the Performance Suggestions checkbox rather than the results card: the card does not exist until you have run Analyze, so a first-visit tour would spotlight nothing. A step whose "requires" names a predicate that does not exist is silently dropped from every install forever, with nothing in the logs to say a tour lost a step, so the shipped tour JSON is now checked against the predicates that actually exist. Also settles the QoS rule menu path, which the app gave three ways: the two Adaptive SQM and Cellular Data Savings variants now read the same as the new finding.
An AP at a test site restarted after being power cycled and the Device Status card kept explaining it with a firmware upgrade from days before. The tracker clears the reason the moment it sees a device on a new boot, but its record is only as fresh as the last uptime sample it was fed, while the dashboard reads uptime straight off the console. In the gap between the restart and the next sample, the card pairs a two-minute uptime with the previous run's reason. The reason is now looked up against the boot the device is reporting: a reported boot later than the record's by more than the tracker's own match tolerance means a restart it has yet to account for, so nothing is shown until the new reason resolves. A reported boot that is earlier is the two uptime sources disagreeing, not news, and still shows. A device that reports no uptime at all is offline, and keeps its last reason. The MAC-only accessor is gone with it - it could only ever speak for the last boot the tracker was told about, which is the bug.
Swap the "Download" / "Upload" words for down / up arrows and shrink the rate unit at the mobile breakpoint, so the two cards stop wrapping on a phone. Both copies of the panel (Monitoring - Live View and the Dashboard card) get the same treatment.
Latency & Packet Loss: the eye sits after the controls, which take a full row on mobile, so it wrapped onto a line of its own while the space beside the filters went unused. It moves to the card's top right corner instead, the same absolute-corner treatment the settings link in that header already gets at this breakpoint. Scoped by the card's id so no other chart header moves. Live View: the icon's margin-top auto pushed it to the bottom of a row that gets tall once a multi-WAN filter is in it, leaving it sitting low against the pills. Dropped at the mobile breakpoint only, so single-WAN sites - where the row is short and the current placement is right - are untouched.
Follow-up to the placements landed in 9548d06, with the offsets tuned on a phone: the eye sits hard into the Latency & Packet Loss card's top right corner and needs no reserved padding in the header, and the Live View icon centers in its row rather than aligning to the top of it.
The WAN token repeated on every pill and every stacked row, which is a lot of "WAN" for a phone-width column that has a number to show. On mobile the pill token is hidden outright - the connection's name is what identifies it, and the token is the tie-breaker a narrow screen cannot afford - and the stacked rows fall back to the index alone, "WAN2" reading as "2" beside its value. The stacked label and value also step down a size so a three-WAN stack fits its tile. Hiding the pill token is not tab-scoped, so it also applies to the Latency Targets and Upstream Path Discovery pickers, where the same reasoning holds. Desktop is untouched everywhere.
…he All pill The play/pause and Historic cluster sits inset from the chart's right edge to clear the corner the single-WAN chart uses. Comparison mode does not use that corner, so the cluster moves out to 2.25rem there. It follows the existing comparing() path rather than introducing a second idea of multi-WAN: the class is set in syncModeUi, with calls added to both exits of doSetWans since remountChart rebuilds the chart but not the cluster, which is parented outside the mount div and survives. Mobile pins it at 0 in both modes - the compound selector is repeated in the mobile block so the comparison rule cannot outrank it on specificity. The All pill also takes flex 0.4 in every WAN filter (Live View, Network Performance, the Dashboard live card, Latency Targets), so a two-letter button stops taking the same width as a connection name.
A link's ?wan= kept describing the view long after the user had moved on, so a reload or the back button re-narrowed to the link's WAN over the choice they had made. It is now dropped on the same terms as the linked instant. Two triggers, deliberately not the same one. Taking the timeline drops both parameters in a single navigation - a second replaceState behind Blazor's router is what desynchronized the two views of the URL last time. Changing the WAN filter drops only ?wan=, since choosing which WAN is on screen says nothing about the instant being viewed; that path still moves the instant's arm to the new URL, or the navigation would re-arm the seek and pull the timeline back to the link's moment on every WAN change. Both tabs are covered, each at its own user-driven point: Network Performance through ApplyWanSelectionAsync where persist:true means the pick was theirs (link and restore pass false), and Live View through the WAN change callback. That callback also fires while the link is being applied, which would be the clear-on-arrival that cost a navigation per jump and tripped the browser's throttle, so it is guarded by a flag set around the link's own selection. Also in this batch, both mobile-only: the All pill's flex is repeated inside the mobile block so it beats .time-btn's flex 1, and the comparison count in the WAN rate cards reads "(4)" rather than "(4 WANs)" where the arrow has already said what the tile is.
…al at 100 ms Razor treats an @ that follows a word character as a literal - the email-address heuristic - and it compiles clean either way, so a build says nothing about it. Two spots were rendering their code as text: - The WAN rate tiles' comparison count, added in the previous commit, showed "Upload@WanRateCount()" on the label. - The Upstream Discovery link in the access-technology hint, which has been emitting its own href as text since well before this branch (it read discover=1@DiscoveryWanQuery() until the rename earlier today). Both take the explicit @(...) form the WAN Throughput title already uses. The neighbouring @expr@if and @expr@expr shapes are NOT affected: there the @ ends a code transition rather than following markup text, which two long-standing instances in Security Audit and the ISP Health profile line demonstrate. Also in this commit: the live RTT tiles keep two decimals below 100 ms and drop to one at or above it, which is the resolution worth reading at three digits and holds the tile width steady as a value crosses a hundred.
…y link Making that href dynamic in the previous commit changed how Blazor writes it. Static markup is emitted verbatim, so the hand-written & was decoded exactly once by the browser and the link worked; as an attribute VALUE it is HTML-encoded on the way out, so the ampersand was encoded a second time and the DOM held the literal text "&discover=1". The site-context script then parsed that to append its site parameter, read a parameter named "amp;discover", and percent-encoded the semicolon on the way back out - which is the &%3Bdiscover=1 seen in the bar. A plain & in the source is correct here: encoding a dynamic attribute is Blazor's job, not the markup's. The other hand-escaped ampersands in the codebase are unaffected because they are still static - the four discovery tooltips on Monitoring, and the map popup HTML built in FloorPlanEditor and SpeedTestMap, which is parsed as HTML once when it is injected.
…l hop A UniFi gateway reaches an attached 5G/LTE modem over a GRE tunnel, and nothing else on the gateway presents as gre*, so the interface name says what the medium is outright. Two things follow from it. The access technology is set to Cellular rather than left for the L2 neighbor vendor to guess, which on a cellular WAN it cannot: the neighbor is the modem. It still only fills an empty slot, so a technology someone chose is never overwritten. Hop 1 is dropped from the access hop pool. It is the modem's own tunnel endpoint - a CGNAT address on our side of the radio that answers every trace and tells us nothing about the carrier's first mile - so monitoring it measures the tunnel. Confined to gre* uplinks: everywhere else hop 1 IS the first-mile device, and carrier-side CGNAT hops beyond this one stay eligible, which is what makes the existing CGNAT allowance still worth having. Also adds the RBAC check discussed for WAN vantages. Add, edit, save and delete already sat inside SiteAdminOnly in the markup; they now re-check the same policy against the same site where the write happens, so the rule no longer depends on which wrapper a button is inside. It cannot reject anyone who had a button to press, and falls open when no authentication is in play at all.
…eachable Both halves of the previous commit were dead on the WANs they were written for. The access technology sat beside the L2 vendor inference, which returns early when no neighbor MAC is found - and a GRE tunnel has no ARP neighbor, so on a UniFi Cellular Modem WAN that step always returns before reaching it. It now runs where the WAN is identified, which is where the uplink name is known. The hop exclusion filtered the candidate pool, but the modem's tunnel endpoint does not arrive that way. It is CGNAT with no BGP attribution, so it enters through the positional pass for unannounced first-mile hops, which by design admits exactly that shape of address. The exclusion moves to where the access pool is finished, after all three contributors, so no path can bypass it, and logs what it dropped. The predicate moves to NetworkUtilities beside IsPppoeInterface, which is the same idea - a data-path interface name that identifies the connection - and is named for what it actually detects. gre* means a UniFi Cellular Modem attached to the gateway and nothing else; a third-party modem, a bridged carrier router or a dongle is equally cellular and presents as an ordinary Ethernet or PPPoE WAN, so a false result says nothing about the medium and the comments no longer imply otherwise. Verified against sample-device-response.txt, where the cellular WAN reports both ifname and uplink_ifname as gre1.
A Starlink discovery hung at "Attributing hops to ASNs" with nothing in the log. Two causes, one on top of the other. CGNAT was being sent to the whois fallback. RFC 6598 space is shared and not announced in BGP, so neither the offline database nor bgp.tools can attribute it - the call only pays a network round trip to learn that, once per distinct hop, and a Starlink first mile is almost entirely 100.64/10. The address class check now requires public space, which is what the comment above it already claimed and what the tracer's WAN IP lookup documents. Private addresses were already skipped; this puts CGNAT on the same footing. The whois call also had a deadline on its connect and nothing else. A rate-limited bgp.tools accepts the connection and then stays silent, so the read waited on the caller's token with no timeout of its own - holding one of the two permits indefinitely and blocking every later lookup behind it. Found with both permits held by sockets sitting at 0 bytes in either direction, which is why the phase stopped dead rather than slowing down, and why nothing was logged: a hang never reaches the catch. One 8 second budget now covers connect, write and read together, and a timeout returns null, which is already what an unattributable hop looks like. Also: the Latency Targets card's "N active" badge counts the targets the filter is showing rather than every target on the site. With All selected, or on a single-WAN site, the number is what it always was.
…tcome A whois call left no trace unless it threw, so a run that reached the network repeatedly, slowly, or not at all looked identical in the log - which is what made the CGNAT stall read as a silent hang rather than a lookup. Each fallback now logs the address, how long it took and what came back. The offline database has already answered everything it can by that point, so every line is a public address it did not know: none for CGNAT confirms those are skipped rather than merely fast, and a duration near the ceiling shows the timeout working instead of a hang.
…ored Nothing on the page said how to start monitoring another WAN, and the answer - enroll an agent for it from the site's own Configuration - is not somewhere you would look without being told. A card now says so, between the enable checkbox and the Sites table, naming the two buttons it is talking about. Shown only where it is news: more than one WAN on this site, no vantage configured for any of them, and the user has not dismissed it. A site that already has a vantage has been through this, and a site with one WAN has nothing to act on. Dismissal is per user, in the same UserUiHints table and service as the WAN filter hint - what one operator has learned says nothing about their colleagues. DismissAsync records the show limit rather than adding a second flag, so the existing ShouldShowAsync rule covers both kinds of hint and no migration is needed. The WAN lookup runs only while the Multi-Site tab is on screen, and once per visit. A console it cannot reach leaves the card hidden rather than guessing, since telling a single-WAN site about its second WAN is worse than saying nothing.
… refinements The card's second condition was wrong. It hid on any vantage existing, but a vantage with no agent behind it is precisely the state the card exists to get someone out of: they created one, nothing probes it, and the missing step is the one the card names. It now asks whether any vantage is agent-bound. TEMPORARY: that condition is commented out so the card can be seen on a site whose vantages are already bound. Restore before this ships. Also, following the CGNAT change earlier: CGNAT reaches the offline database again and is blocked only from the network call. The GeoLite2 lookup costs nothing and does sometimes know shared space a carrier has registered, so it is worth asking; whois is not, because RFC 6598 space is not announced in BGP. And when bgp.tools returns nothing usable, the first line of its reply is logged. "No row for this prefix", a notice instead of data, and a format change were indistinguishable before, which is most of why a rate limit took so long to identify.
…thout multi-site on The card used banner-text, which is display:flex column - a component built for a bold title stacked over a description. Every inline name in the sentence became its own row. It is a plain alert now, with a small class of its own for the icon, paragraph and Dismiss button, so emphasis inside the prose reads inline. It also said the same thing whether or not multi-site was on, naming a Configuration button and an Add Agent action that do not exist on the page until it is. The opening sentence is shared; with multi-site off it now leads with the checkbox directly above it and says why - the agent enrolls through multi-site, including on this site - and leaves the rest for when those controls appear. The wording swaps as soon as the box is ticked, since only the branch depends on the setting, not whether the card is shown. TEMPORARY, both restored before this ships: the WAN-count and bound-vantage conditions are commented out, so the card shows on any site. That is what makes it testable on a single-WAN single-site install.
The bar is shrink-wrapped to its content so a two or three WAN selector does not stretch the width of the card on a desktop. On a phone that fights the rule right beside it: the pills are told to share the row, and a max-content parent leaves them no free space to share, so they stay bunched in a corner of it. Width goes back to auto at the mobile breakpoint only. The desktop shrink-wrap and the max-width guard against a long WAN list are both untouched.
…t means Three columns on a phone left the sentence about two hundred pixels wide and reading as a narrow ribbon down the card. Icon and text now share the full width on one row and Dismiss sits under them, right aligned. The desktop row is unchanged; the icon and text are wrapped together so the button is the only thing that moves. The card also said "the site's Configuration" while pointing at a table of several. It names the current site now, falling back to "this" if the name has not loaded, so the sentence never has a hole in it.
Enabling multi-site loaded the site list but never set the current site's name, which only the init path did and only when multi-site was already on. So everything that names the site stayed blank until the page was left and returned to: the header's site badge, and the WAN vantage card, which had to fall back to wording that does not name one. The card takes a whole phrase now rather than dropping a name into a sentence. With no name to use, "the <blank> site's" reads as a missing word; "this site's" is just the weaker of two correct sentences.
Both were commented out to make the card visible for review on sites that would not otherwise show it. It is back to appearing only where it is news: more than one WAN on the site, no vantage with an agent bound to it, and not dismissed by this user.
…mplate "Open the Main Site site's Configuration" is what a template produces when the name is itself a noun, and people do name their sites that way. The name now takes the possessive on its own - "open Main Site's Configuration" - and with no name loaded the sentence reads "open this site's Configuration".
Two expectations were verbatim strings spanning a source-file newline, so they inherited the checkout's line endings and failed on a CRLF working copy while the emitted Flux filter always uses \n. Written with explicit escapes like their siblings.
…1101) * Monitoring: per-WAN outage evaluator replacing per-target alerts for WAN categories One access-layer outage used to page once per target. The WAN-facing categories (AccessIsp, Transit, InternetService, legacy Wan) now feed a per-WAN state machine that publishes monitoring.wan_outage / monitoring.wan_outage_partial / monitoring.wan_recovered - one open alert per (WAN, kind), partial superseded by total, site-level rollup when every WAN of a multi-WAN site drops in the same evaluation. Verdicts are classified from the per-target offline states against the persisted trace map, inheriting OutageDetector's attribution rules (AttributeBreak and the ASN independence key are now internal for that). Fabric and Custom targets keep their per-target alerts, and the per-target state machine keeps running underneath for every type. * Alerts: seed the WAN outage rules, and make rule seeding one-time per database Three enabled-by-default rules for the new event family: monitoring.wan_outage (Warning floor), monitoring.wan_outage_partial (Info floor, so a non-primary WAN's partial still matches) and monitoring.wan_recovered. Seeding now records every seeded pattern in a new SeededAlertRules table and only inserts patterns never recorded, so a rule the user deletes stays deleted across restarts instead of resurrecting on every boot; existing installs are backfilled from their current rules on first start. * Alerts: WAN outage alerts close themselves on recovery and supersede The wan_* family is an open/close family, unlike the rest of the catalog: monitoring.wan_recovered resolves the WAN's open outage and partial alerts (and any all-wans rollup, whose premise a single recovery invalidates), and a confirmed total outage resolves the partial it grew out of, so the two never stack in Active Alerts. Resolution runs before rule matching so it works even when the recovery rule is disabled, and re-derives incident status through a shared AlertCorrelationService helper the UI resolve path now also uses. Adds the three event types to the Rules tab pattern picker. * Monitoring: WAN outage evaluator tests, plus three review fixes Classifier and state-machine coverage: one WAN alert per access outage with per-target events silenced for the WAN categories, partial vs total classification, supersede, recovery, rollup open and staggered release, non-primary severity, flap and staleness guards, and Fabric / Custom left exactly as they were. Review fixes the tests surfaced: releasing the rollup no longer reopens an alert for a WAN whose verdict already cleared in the same pass (it used to publish a zero-target partial and an instant recovery); an upstream total now requires a failing internet destination, so an access hop plus rate-limited transit hops alone can never read as the internet being down; partial bodies say 'failing or degraded' since sustained loss also counts toward them. * Monitoring - ISP Health: point a WAN with no scheduled speed tests at the schedule The collecting-data banner now checks whether the selected WAN is covered by an enabled scheduled WAN speed test (gateway schedules by their wanGroup in TargetConfig, server-vantage schedules count for the primary) and, when it is not, adds a nudge plus the same Set up schedule link the WAN Speed Test banner uses, so throughput history builds while ISP Health waits on latency history. * Monitoring: cut WAN outage alerting to ~30 s, and grade load-balanced WANs by impact Timing: a target counts as failing for WAN verdicts after two failed probes, passes run at 10 s, and two held passes open. No WAN verdict ever rests on one target - a total needs the whole cohort failing at once - so the cohort agreement is the flap suppression the per-target 3-strikes rule was providing. Closing still takes three passes, so a flapping WAN gives one alert and one recovery. The per-target machine's own thresholds, which Fabric and Custom alerts use, are untouched. Severity now turns on whether the WAN carries traffic rather than on the primary role alone: under load balancing every WAN carries live sessions, so a backup going dark is a real service loss and grades Critical. An idle failover backup still grades Warning, unchanged. * Monitoring: drop the reassuring clause from partial outage alerts A partial is often a total still arriving, with the remaining targets a probe cycle behind, so telling the reader the connection itself looks fine can be contradicted a minute later. State the evidence only. * Monitoring: let the site rollup fire across a window, superseding per-WAN alerts Requiring every WAN to confirm its outage in the same evaluation made the rollup near-unreachable: WANs are polled at their own intervals, so a site that loses everything at once still confirms a minute apart. Each WAN now records when its total outage was confirmed and the rollup fires while those confirmations sit within 90 s of each other, folding in any WAN that already opened its own alert - the rollup event resolves the per-WAN alerts it supersedes, so the worst case is one alert then the rollup rather than one per WAN. * Monitoring: stop live ISP and Transit tiles reading off stale probe results A target that stops reporting kept presenting the reading it carried before it went quiet, so the tiles read healthy through an outage. Readings now expire after 90 s, half again the slowest poll interval a target can be given. The access ISP tile also read its loss off the nearest hop by RTT, and a hop with no RTT could never become the nearest - so with every hop dark it kept naming the one stale hop that still had an RTT, and reported that hop's 0% loss during a total outage. With nothing answering there is no nearest hop, and the mean of what did report is what it shows. Observed on a WAN whose ISP targets went quiet under a blackhole while its transit targets kept reporting 100%. The live WAN chart's mean now returns null rather than 0 when nothing fresh reported, and the chart plots a gap instead of a healthy zero. * Monitoring: carry the WAN selection when the live tiles navigate The ISP and Transit tiles are shared between the Monitoring page and the Dashboard. On Monitoring they jump within the page, so the WAN filter survives by staying put; from the Dashboard the same tiles navigate, and the link carried no WAN - so clicking one WAN's tile opened Latency and Packet Loss showing every WAN. They now build the link the way Monitoring's own jump does, selection and all. * Monitoring: move the ?wan= link building onto LiveWanScope The Live View tiles and the Monitoring page's own jump were building the same fragment two ways, and the tiles' copy arrived a commit ago. The scope owns the selection, so it owns how a link carries it: QueryValue for the set, QueryFragment for the fragment, with an override for a link that means every WAN rather than the current focus (the LAN jump). Behavior is unchanged on both call sites. * Monitoring: land a deep link on its WAN every time, and stop releasing it snapping the view back Two faults in how a link's ?wan= and the stored WAN filter took turns. Network Performance applied the link from a block that runs once per chart mount, and gave up when the WAN options had not loaded yet - so whether the link took came down to which finished first, and on the losing pass the stored filter, read a line earlier, stood in its place. The link is now held until the options and the multi-WAN answer arrive, and applied from whichever lands last. Releasing a link also reset the restore, so the stored set was read over the top of the current selection. The URL drops its ?wan= on ordinary interactions - changing the time filter, resuming Live - which yanked the view off the WAN being watched, mid-look. Releasing now clears the link claim only; the selection on screen stays until someone changes it, and storage is read again on the next load. * Agent relay: stop a batch vanishing without a word Three ways a relayed batch could be lost silently, all of them now either impossible or logged. Ownership is tri-state. AgentOwnsAnyContextAsync answered false when it could not read the site contexts at all, and false there drops the whole batch - the same answer as a definite "owns nothing". Only a definite false drops it now; an unreadable answer falls through to the per-result judgement, which asks the same question per target. The drop itself is logged. It was a bare return, so a main-site agent that owns no context lost every batch with nothing in the log to say so. An unconfigured InfluxDB client is logged too: the latency writes no-op on one, so the batch would be walked and stored nowhere. The batch still runs, since the live caches and alerting do not depend on Influx. * Monitoring: keep latency and loss on the WAN live chart across a throughput hole Chart rows were built from the throughput series alone, so a span with no throughput point produced no row - and dropped that span's latency and loss with it. The gateway's SNMP counters are collected by whoever collects for the site, and a site that leaves collection to the server has nothing buffering them, so a server restart leaves a real hole in throughput while the agent's probe results replay into it perfectly. The chart drew the hole across every series and the backfilled latency and loss were invisible. Latency points landing in such a hole now get a row of their own with no throughput on it. Only in a hole: a point with throughput either side within a couple of sample intervals still rides that throughput point, so ordinary operation keeps exactly the rows it had and the throughput line does not turn dotted. Both series were already in hand, so this adds no query. Also logs two silent drops in the agent relay: the batch dropped when a main-site agent owns no WAN context (a bare return until now), and a batch arriving while the site's InfluxDB client is unconfigured, whose writes no-op. Neither changes what is processed. * Alerts: re-derive each incident once per Resolve All, not once per alert Resolve All recalculated the incident for every alert it resolved, and each recalculation re-reads the incident plus every alert on it - so a list where twenty alerts shared one incident did that work twenty times, each read and write its own SQLite transaction. That is what made the button crawl. The alerts are written as before; the incidents they belong to are re-derived once each, afterwards. Also raises ISP Health MinDataHours to 5 TEMPORARILY, to bring the collecting-data banner back on a WAN that just crossed the real threshold so the schedule nudge can be checked. Revert to 4 after. * ISP Health: raise the temporary MinDataHours to 9 More history has accumulated since the 5 went in, so the banner needs a higher bar to come back. Still temporary - revert to 4 once the schedule nudge has been checked. * Alerts - Active Alerts: even out the per-alert buttons Acknowledge takes the tertiary style, and on desktop the three buttons share a 6.75rem minimum so View, Acknowledge and Resolve line up as a column instead of stepping in and out with their label lengths. Mobile lays them in a row and is left as it was. * Monitoring - ISP Health: only offer the schedule nudge to those who can act on it Creating a scheduled speed test is a site administrator's to do - Alerts & Schedule gates the whole section on it - so anyone else was being sent to a page that would not let them. Non-administrators keep the plain collecting-data line; the extra sentence and the Set up schedule button appear only for an administrator on a WAN that has no schedule. * Alerts: name the Acknowledged section's button for what it resolves It said Resolve All and took the whole unresolved list, acknowledged or not. Now it says Resolve Acknowledged and takes that section's alerts. The Active section's Resolve All still clears everything unresolved, so acknowledging the lot and resolving in one press is unchanged - which is what the wider scope here was for before the button said which alerts it meant. * ISP Health: put MinDataHours back to 4 The temporary raise had done its job: the collecting-data banner and its schedule nudge were checked on a WAN with more history than the real threshold needs. * Alerts: write bulk acknowledge and resolve in one round trip Every bulk button walked its alerts one at a time, and each of those is an UpdateAlertAsync - a SaveChanges, so a SQLite commit - against a change tracker that grew by an entity per iteration. A few hundred alerts meant a few hundred commits and a quadratic walk of the tracker, which is why the buttons locked up on a fast machine. SetAlertStatusAsync loads the rows once, stamps them, and saves once. Used by Acknowledge All and Resolve All in Active Alerts, Resolve Acknowledged, and both Acknowledge All and Resolve All under Incidents, which gather their incidents alerts first and write them together. Incident status is still re-derived once per affected incident. * Alerts: re-derive the incidents in one pass, and hold the bulk buttons while they work Batching the alert writes left the incident pass as the cost: a few hundred alerts usually means nearly as many incidents, since most hold one alert, and each was re-derived with two reads and a commit of its own. They are now read together, derived in memory, and saved once. All five bulk buttons also disable and show a spinner while one runs. They act on the same lists, so a second press mid-flight would work from a set already being written - and until now nothing on screen said the first press had been heard. * Alerts - Incidents: batch the reads and writes those buttons actually make The incident buttons were left doing per-incident work: a read of each incident's alerts, then a save of each incident, so four hundred incidents meant four hundred reads and as many commits. They now read every incident's alerts in one query and save the incidents together. UpdateIncidentsAsync also had to stop assuming its incidents were tracked. They come from the page, which reads them detached, so the single save it did would have persisted nothing; it attaches each one first, reading the tracked set once rather than scanning it per incident. * Alerts: poll the alert and incident lists without moving them under the reader The refresh timer reloaded Active Alerts straight into the bound list every thirty seconds. Entries arrive newest-first, so anything new pushed the list down - past the row being read, or out from under the button about to be pressed - and there was no sign it had happened. The poll now reads into a holding buffer and offers what it found as a pill. A read with nothing new applies itself, so acknowledgements and removals still keep up on their own; only genuinely new entries wait to be asked for. It skips entirely while a bulk action, a modal or an inline edit is open, and the Incidents tab is polled the same way rather than not at all. Rows carry their id as a key so the diff keeps the DOM it already has. * Alerts - History: page it in SQL instead of showing the newest slice History took the newest 200 rows and stopped, with nothing on screen to say the rest existed - a site running since February has thousands. It now reads a page at a time with the total the filters match, so the pager can say where you are, and a filter change starts again from the first page rather than leaving you on a page that no longer exists. Uses the pagination classes the client dashboard already uses; the only new rule is the pill the alert poll offers, which had nothing to borrow. * Alerts: never hold a poll's findings out of reach Two faults in the pill, both showing up on an empty list. It rendered inside the non-empty branch, so with nothing on screen it could not be reached at all; and the poll withheld new entries even then, though an empty list has nothing to push down. Together that left the tab reading All Clear with the alerts parked in a buffer and no way to ask for them. An empty list now takes the read straight away, and both pills sit outside the empty/non-empty split so held entries are always offered. * Alerts - Incidents: ask the database for unresolved incidents The tab read the newest fifty incidents whatever their status and filtered to unresolved afterwards, so an unresolved incident disappeared the moment fifty newer ones had been resolved. On the test site that was every one of them: twenty-two sat unresolved from February while the fifty-row window reached back only to August, and the tab read empty. Filtered in SQL now, so the limit applies to what the tab actually shows. * Alerts API: note the incidents endpoint's status-filter trap It takes the newest incidents whatever their status, so a caller after the unresolved ones loses them once that many newer ones are resolved - what the Incidents tab was suffering. Left as-is rather than changing what the resource returns by default; the TODO says what it needs. * Alerts - History: give the pager room below it Scoped to this one rather than the shared pagination class, which the client dashboard also uses between sections where it does not need it. * Alerts: send monitoring alerts to the moment and the WAN they are about Every monitoring alert linked to /monitoring?tab=performance and nothing more, so following one landed on the tab's default view of now - by which time the window that shows what happened has usually scrolled off the live view, and on a multi-WAN site it may not even be the right WAN's chart. The link now carries the category, the instant and the WAN, which the analysis page already reads. Per-target alerts use their own target's category and WAN; a WAN outage opens on the access hops at the moment the outage started, a partial on the destinations; a recovery opens at the START of the episode that just ended rather than at the recovery; and the all-WANs rollup spans every WAN. * Monitoring: LAN alerts ask for every WAN, like the page's own LAN jump A Fabric target is not reached over any one WAN, so its alert link carried no WAN at all and the analysis opened narrowed to whatever the stored filter was. It now asks for all of them, which is the choice Monitoring's own LAN jump already makes. * Alerts: count the poll's held entries against a set Both pending counts are read on every render and compared each held entry against the whole visible list, which is quadratic in a list that reaches the hundreds - the shape the bulk buttons were just cured of. The poll's own is-anything-new check did the same.
* Starlink: alert on what the dish reports about itself We collect a great deal from the dish and alert on none of it. Every other physical source has alerting - monitoring.sfp_rx_power, cellular.signal_poor - while Starlink, which reports more about itself than any of them, is silent. A dish can be obstructed, thermally shut down, deprioritized, negotiated at 100 Mbps on a gigabit service, or knocked out of alignment, and the product says nothing. Six new event types plus a closer, all starlink.*, hanging off the dish poll rather than off MonitoringAlertEvaluator: Starlink is usually a backup WAN with no vantage, no agent and no monitored targets, so the dish is the only sensor on that link and these have to fire for a WAN nothing else watches. There is deliberately no correlation with per-WAN outage alerting - both firing is correct, they carry different evidence. Severity does not follow the per-WAN outage table either: a degraded backup is silent by construction and is discovered at the moment it is needed, so backup dish problems keep real severity. - starlink.dish_alert - the dish's own verdict: its alert codes verbatim, a self-test that has started failing, and a disablement code other than Okay. Warning, Critical when out of service. - starlink.obstructed - a sustained obstruction fraction, or the dish's own persistently-low-SNR flag. Warning, Critical past the critical bar. One type because the remedy is the same either way. - starlink.alignment_drift - the current hourly median departing from the dish's own 7 day baseline by more than 2 degrees for 30 minutes. Warning. - starlink.eth_speed_degraded - a link negotiated below what this dish has been seen to reach. Warning. - starlink.outage_burst - the dish's own outage seconds over a rolling day, with the most recent cause. Warning. - starlink.service_restricted - the TRANSITION into a rate limit, never the standing state. Info. - starlink.recovered - closes the one condition it names, so a dish that clears its obstruction keeps whatever else it still has open. Every rule is written against a value, never against "the field is set". On the reference dish disablement_code reads Okay, alerts carries install_pending continuously, hardware_self_test reads Failed continuously, both restriction reasons are permanently populated, mobility_class reads Mobile on a bolted-down dish, and the boresight sits a steady 3.69 degrees off desired - all while nothing is wrong. Any "has a value" test would fire on day one and never stop, so the tests that matter most here are the ones asserting silence. Alignment is judged against the dish's own baseline for the same reason, and is gated on attitude uncertainty but explicitly NOT on mobility class - that gate would have silenced the alert on exactly the installation it was written for. WAN association is best effort: where exactly one Starlink WAN and one dish are configured the alerts carry the usual GatewayWanHelper label, otherwise they name the dish and fire regardless. * Starlink: calibrate the alerting constants against the reference dish Measured 30 days of the reference dish rather than reasoning about it, which caught a rule that would have shipped dead. The attitude-uncertainty gate on alignment drift was set at 1 degree on the intuition that uncertainty near the 2 degree trigger makes drift and confusion indistinguishable. A healthy dish is nowhere near that certain of its attitude: p50 0.70, p95 1.49, p99 1.83, max 2.71 degrees over 30 days. The bar sat inside the healthy distribution, so roughly a third of ordinary polls were gated out, and because a gated poll stalls the sustain the alert could never hold its 30 minute window. Raised to 4 degrees, about 1.5x the observed maximum, with the distribution recorded on the constant and a theory pinning the gate open across the real healthy range. The gate is worth keeping: mean uncertainty rises monotonically with how far the offset has strayed from its median (0.77 / 0.90 / 1.10 / 1.19 across deviation bands), so the excursions it must not mistake for movement do come with the dish saying it is less sure. Everything else measured out as the spec described, and the numbers are now on the constants they justify: - alerts: install_pending and nothing else in 30 days, which is real evidence that the motor and mast codes are not always-on for a fixed-tilt motorless install and belong outside the benign list. The comment no longer claims the other four entries were measured; they are judged on what the code means. - hardware_self_test Failed throughout, disablement_code Okay throughout, both restriction reasons constant, eth_speed_mbps a clean constant 1000. - fraction_obstructed median 0.06%, never above 0.1%, against a 2% bar. - outage seconds 1 to 31 a day, median 13, against a 300 second bar. - ~730 samples a day, so the hourly alignment median holds about thirty. Also: the WAN binding cache is now dropped when a dish is added, enabled or deleted, instead of ageing out over 30 minutes and labelling a second dish with the first one's WAN. Alignment drift copy now names one yardstick rather than switching between the dish's baseline and its ideal aim mid-sentence. * Starlink: close the review findings on the alerting rules Four fixes from reviewing the branch. A superseded alert could be closed and its replacement then dropped. These types keep one open alert per (dish, condition) by having a re-publish supersede its own predecessor, and AlertProcessingService resolves the old row BEFORE rules are consulted - while cooldown keys are per (site, rule, device), so the replacement shares the key of the alert it just closed. An obstruction escalating Warning to Critical inside the hour resolved the Warning and had the Critical suppressed, leaving a critically obstructed dish with no open alert at all. The WAN outage family never had this because a total supersedes a PARTIAL, which is a different rule and a different key. Every Starlink rule now ships with no cooldown. It is redundant as well as harmful: the evaluator publishes on state changes only, and is where the real throttling lives - sustain windows and hysteresis on the gated conditions, "new evidence only" on the dish's own codes, edge-triggering on restriction. A test pins the invariant so nobody restores a cooldown without reading why. That also fixes recoveries quietly going missing. starlink.recovered is one rule closing six different conditions, all sharing the dish's device id, so any cooldown swallowed the second recovery whenever two cleared together. The alert still closed - resolution runs ahead of rule evaluation - but nobody was told. Per-dish evaluator state is now serialized. The timer poll is single-flighted against itself, but the Starlink Stats panel calls PollStarlinkAsync directly (Refresh, moving between terminals, first paint on an empty cache) with no such guard, so two polls of the same dish can be in flight together. The sample windows are plain Queues, whose concurrent Enqueue/Dequeue corrupts their internal indices rather than merely racing. An obstruction alert raised purely on persistently low SNR no longer quotes the obstruction fraction, which is healthy in that case, against the poor obstruction threshold. * Starlink: make the alerting observable on a dish with nothing wrong A healthy dish publishes no alerts, which is correct and completely useless for telling a working evaluator from a broken one - both are silent. Two Debug lines close that. Per poll, after the checks so the open set is current: the WAN the dish was bound to, the obstruction fraction and SNR flag, the alignment median against its baseline with the sample count and whether attitude uncertainty gated it, negotiated against capable Ethernet speed, outage seconds over the rolling day, the restriction state, and which conditions are currently open. Built only when Debug is enabled, since it medians the alignment window and sums the outage window to produce it. Per baseline refresh: what came back from Influx and how many points it rests on. A null median there is the difference between alignment drift watching and alignment drift being switched off, and the two are otherwise indistinguishable from outside. * Starlink: stop saying Starlink twice where the wording already says it Dishes and Starlink WANs get named "Starlink Roof", "Roof Starlink", or just "Starlink", so anywhere our own wording opens with the word, the name doubles it: "Starlink has taken Starlink Roof out of service", and a diagnostic line reading "Starlink Starlink Roof alerting". The name is trimmed at both ends, because our templates put the word in front and the two arrangements double up identically. A name that is nothing but "Starlink" strips to nothing and falls back to a generic noun rather than leaving a hole in the sentence. The word is left alone anywhere else in a name - "My Starlink Dish" stays whole, since cutting from the middle mangles names more often than it tidies them. Only the sentences and log templates that carry the word themselves are trimmed. Alert TITLES keep the full name: a title is often all that reaches a notification channel, and dropping the service from it would lose what the alert is about.
Six steps covering Multi-WAN monitoring, the Live View / Latency & Packet Loss jump buttons, WAN outage alerts, Starlink alerts, and the Smart Queues (SQM) check. Engine additions the tour needed: - multi-wan and starlink predicates on TourPredicateResolver. multi-wan counts enabled WANs from the console config rather than WanProfiles rows, which are an ISP Health side effect and so are both stale-prone and incomplete. - hideFromList, so the second half of a two-stop feature walks the user through it without spending a second bullet on a list capped at six. - matchText, which narrows the spotlight to the row inside the anchor whose text matches and scrolls to it. A list whose rows depend on the user's own configuration can only carry the anchor on the list itself. Absent text falls back to the anchor, so a step never fails over wording it hoped to find. Anchors: the Live View WAN selector, both jump buttons, the Active Alerts list and the Rules card.
The reset scripts clear AdminSettings.Password and restart, on the contract that startup then generates a password and prints it. Sign-in now goes through Identity, and the bootstrap only seeds AspNetUsers from the legacy hash when the admin account does not exist yet - so on any install that had already migrated, the scripts printed a password that was silently refused. There was no way back in short of setting APP_PASSWORD. Carry the freshly generated password through to the bootstrap and apply it to the account, which is what the scripts were always promising. It fires only when there was no stored password - a first run, or a deliberate reset - so a hash that was merely read back still never re-applies, and a password set in the app survives every reboot. Clear the account's lockout at the same time: someone resetting a password has usually been failing sign-ins to discover they need to. Deleting and re-seeding the admin account would have done this without touching auth code, but it is explicitly forbidden - it cannot be undone, and it would take the account's site memberships and roles with it. The generated password now always contains a digit. It is applied through Identity, which enforces RequireDigit, and drawing 16 characters uniformly from an alphabet with 8 digits in 55 leaves roughly one in twelve with none - which would have failed the reset it was performing. The redraw is bounded, and falls back to placing a digit rather than returning a password the policy rejects. Cover the two seed paths that had no test. A first run on an empty install creates the admin from the generated plaintext, which is the only path where Identity validates the password - a generated password that broke the policy would leave a fresh install with no admin account at all. And an upgrading install carries a password that predates the policy and need not satisfy it: the seed copies the transcoded hash through the overload that runs only the user validators, so the policy is never evaluated and nobody is locked out by the cutover. Pinned with a legacy password too short and digitless to survive validation, so the day that path starts validating, the test says so. Both scripts also warn when local logins are disabled or the admin has MFA on. Neither is changed automatically, but from the login page both look exactly like a wrong password, which is a bad thing to leave someone guessing at right after a reset. The queries are guarded so a pre-Identity install still resets cleanly - including on PowerShell 7.4+, where a missing table would otherwise abort the script through $ErrorActionPreference.
A first deploy runs the boot script over SSH with no timeout argument, so it inherited the 30 second default. The script installs its dependencies inline before it does anything else: it adds the Ookla packagecloud repo, which runs its own apt-get update and fetches a GPG key, then apt-get installs speedtest, bc and jq. On a cold apt cache or a slow WAN that comfortably outruns 30 seconds, and the deploy then tore down a deployment that was still working and reported that the boot script had failed. Give that one command five minutes, which clears a slow first install without leaving the page sitting on a boot script that has genuinely wedged. Re-deploys skip the whole block once the dependencies are present, which is why this only ever bit the first attempt.
tvancott42
added a commit
that referenced
this pull request
Aug 6, 2026
The scripts clear AdminSettings.Password and restart, on the contract that startup then generates a password and prints it. Since v2.5.3 sign-in reads AspNetUsers.PasswordHash, and the app only copies the legacy password across when the admin account does not exist yet - so on an install that has already migrated, the scripts have been printing a password the login page refuses, with no way back in short of setting APP_PASSWORD. Copy the freshly generated hash into the admin account ourselves, once the app has restarted and written it. The hash is copied rather than re-derived, so this needs no crypto and no plaintext in a shell script; it goes in as the old dotted PBKDF2 format, which the app still accepts and quietly upgrades on first sign-in. Clear the account's lockout with it - someone resetting a password has usually been failing sign-ins to discover they need to. The account is updated, never deleted: deleting it cannot be undone and would take the admin's site memberships and roles with it. Pre-Identity installs are unaffected. The Identity tables are probed first and the sync is skipped when they are absent, so the reset behaves exactly as it did before. On Windows the probes are wrapped, because a missing table exits sqlite3 non-zero and PowerShell 7.4+ turns that into a terminating error under $ErrorActionPreference - which would have aborted the script on the very installs it needs to keep working. Also warn when local logins are disabled or the admin has MFA on. Neither is changed automatically, but from the login page both look exactly like a wrong password, which is a bad thing to leave someone guessing at right after a reset. This is the script-side stopgap for released installs. The app-side fix, which makes a cleared password reach the account without the scripts having to touch Identity at all, is in PR #1104.
The scripts clear AdminSettings.Password and restart, on the contract that startup then generates a password and prints it. Since v2.5.3 sign-in reads AspNetUsers.PasswordHash, and the app only copies the legacy password across when the admin account does not exist yet - so on an install that has already migrated, the scripts have been printing a password the login page refuses, with no way back in short of setting APP_PASSWORD. Copy the freshly generated hash into the admin account ourselves, once the app has restarted and written it. The hash is copied rather than re-derived, so this needs no crypto and no plaintext in a shell script; it goes in as the old dotted PBKDF2 format, which the app still accepts and quietly upgrades on first sign-in. Clear the account's lockout with it - someone resetting a password has usually been failing sign-ins to discover they need to. The account is updated, never deleted: deleting it cannot be undone and would take the admin's site memberships and roles with it. Pre-Identity installs are unaffected. The Identity tables are probed first and the sync is skipped when they are absent, so the reset behaves exactly as it did before. On Windows the probes are wrapped, because a missing table exits sqlite3 non-zero and PowerShell 7.4+ turns that into a terminating error under $ErrorActionPreference - which would have aborted the script on the very installs it needs to keep working. Also warn when local logins are disabled or the admin has MFA on. Neither is changed automatically, but from the login page both look exactly like a wrong password, which is a bad thing to leave someone guessing at right after a reset. This is the script-side stopgap for released installs. The app-side fix, which makes a cleared password reach the account without the scripts having to touch Identity at all, is in PR #1104.
Reported from a NAS run: the reset died on "database is locked" and had to be run again. The Docker path never stops the container, so every write races the app's own transactions, and sqlite fails instantly by default rather than waiting. Give it a 15 second busy timeout so it takes its turn. Set through .timeout rather than "PRAGMA busy_timeout = ...", which returns the new value as a result row - prepending 15000 to the output of every query that reads one back, which silently skipped the Identity sync entirely. Route the Docker clear step through run_sql like the other two modes. It had been left calling sqlite3 directly, so it was the one write with no timeout - which is exactly where the reported failure landed. Report a failed write instead of claiming success. The sync runs as part of an || list, which switches errexit off for everything inside the function, so a failed UPDATE fell through to the success message and printed a password that would not work. That is the one outcome this script exists to avoid, and it is strictly worse than the visible error that prompted this. Checked explicitly now, on both the clear and the sync, and a busy database says so in words rather than surfacing a raw sqlite error with no indication of what to do.
tvancott42
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What needs deciding
Is the app-side auth change worth doing now, or later?
The script fix works. It clears
AdminSettings.Password, restarts, and copies the regenerated hash intoAspNetUsers.PasswordHashitself. Verified end to end on the NAS against a migrated Docker install. Nothing about a broken reset is still broken.So the app-side change here buys exactly one thing: durability. The script fix leans on
LegacyFallbackPasswordHasher, which is documented as "Removed with the session bridge after one release" (LegacyFallbackPasswordHasher.cs:14). It works because that class still accepts a dotted legacy hash. When it is removed, the scripts silently break again - they will write aPasswordHashnothing can verify, and the reset will go back to printing a password the login page refuses.That is a scheduled event we control, which is what makes deferring reasonable: the auth change can be done deliberately, at the same time the hasher is retired, instead of touching auth now for a problem that is already solved.
The coupling is the thing that must not be lost, whichever way this goes:
If this PR is abandoned, that belongs in
TODO.mdnext to the main-site agent coverage entry.What is actually unique to this branch
Most of it has already landed elsewhere, so this is smaller than it looks:
cb8400bbdev- pulled across separately874e652edevandmainas93b29a60e2782e19run_sqlrouting3c1d225aThere are no unrelated fixes or enhancements riding along. Total branch content is 8 files, ~500 lines.
Rescue regardless of the decision:
e2782e19Script-only, no auth risk, and it fixes a live bug in the scripts now on
main:database is lockedon a busy install (reported from a real NAS run). The Docker path never stops the container, so writes race the app; the clear step was also still callingsqlite3directly instead of throughrun_sql, making it the one write with no timeout - exactly where it failed.[OK]and a password that would not work. The sync runs in an||list, which switcheserrexitoff inside the function, so a failedUPDATEfell through to the success message. That silent version is worse than the visible error that prompted it.Cherry-picks to
maincleanly, same route as the stopgap.Worth lifting even if the rest is dropped (~60 lines)
Two of the four tests characterize existing behavior and pass on
devwithout any change here:LegacyPasswordThatBreaksThePolicy_StillMigratesAndSignsIn- pins that an upgrading install's password bypasses the account policy (the seed copies the transcoded hash through the overload that runs only the user validators). If that ever starts validating, short legacy passwords stop migrating and people are locked out by an upgrade.StoredLegacyHash_DoesNotOverwriteALaterPasswordOnReboot.The other two depend on the new plumbing. The digit fix in
GenerateSecurePasswordis not independently useful - it only matters because the app-side change puts the generated password through Identity's validators; without that, it never reaches one.Before this can be merged or properly reviewed: rebase
devwas reset tomainafter the v2.6.0 release, which rewrote hashes. This branch was cut fromc39b7194, which no longer exists ondev. The merge base has slid back to800baa88, with 63 commits ondevsince.So the branch is stale in the way
CLAUDE.mdwarns about after a release reset. It needsgit rebase origin/devbefore it is merged or tested, or the three-way merge is computed against a base that predates the release.The app-side change, for reference
The reset scripts clear
AdminSettings.Passwordand restart, on the contract that startup then generates a password and prints it. Sign-in goes through Identity, and the bootstrap only seedsAspNetUsersfrom the legacy hash when the admin account does not exist yet - so on a migrated install the scripts printed a password that was silently refused. Shipped in v2.5.3, so it affected released installs.AdminAuthServicehands the password it just generated to the Identity bootstrap, which applies it toadmin. Fires only when there was no stored password (a first run, or a deliberate reset), so a hash that was merely read back never re-applies and a password set in the app survives every reboot.RequireDigit. Drawing 16 characters from an alphabet with 8 digits in 55 left roughly one in twelve with none, which would have failed the reset it was performing.Deleting and re-seeding the admin account would have avoided touching auth code, but it is explicitly forbidden - it cannot be undone and would take the account's site memberships and roles with it.
The
APP_PASSWORDpath is unchanged.Testing status