Half of genuine installs report their subgen as unreachable

[coaxk]

Surfaced by the #202 telemetry correction (coaxk/subarr-telemetry#48). The inflated install counts had been hiding this completely.

The number

/v1/stats/subgen-mix, latest ping per install, last 30 days:

subgen_kind	was published	genuine	share of genuine
subarr-subgen	15,151	91	42.5%
unreachable	2,173	108	50.5%
vanilla	93	15	7.0%

The raw figures said 87% healthy and 12.5% unreachable, which read as a healthy fleet with a small tail. The genuine figures invert it: more installs cannot reach their subgen than can.

Why the raw number lied in this specific direction

Installs that churn ids are dominated by a few heavy restarters whose setup works. Each restart re-emitted a working subgen_kind under a fresh id, thousands of times, drowning out the genuine population where the failure actually lives. The same mechanism put tautulli and plex at 98.8% adoption when the real figures are 28.5% and 33.6%.

What is not yet known

This is a report of a measurement, not a diagnosis. Before anything is built:

• unreachable is one bucket covering several causes. Wrong URL, container not running, network isolation between compose stacks, a subgen that is up but failing its probe. The telemetry cannot currently distinguish them, and the fix is different for each.
• How many of these are mid-onboarding? 44% of genuine installs never finish the wizard, and an unconfigured subgen would report unreachable. The overlap between "never finished onboarding" and "subgen unreachable" needs measuring before this is read as a broken-integration problem rather than an unfinished-setup one. That single cross-tab probably decides whether this is an onboarding issue or a connectivity issue.
• Is it persistent or transient? A one-shot probe failure at ping time is not the same as an install that has never once reached its subgen.

Suggested first step

Run the cross-tab against onboarding_complete before designing anything. If unreachable is concentrated in installs that never finished onboarding, this folds into the onboarding gap and is not a separate problem. If it is spread evenly across finishers too, it is a real integration failure affecting roughly half the userbase and deserves its own diagnosis surface.

Related: #473 (the id churn that hid this), #202 (the dive that found it).

[coaxk]

Ran the cross-tab I suggested, since it was one query. It is not purely an onboarding artefact.

Latest ping per install, genuine only, last 30 days:

	subarr-subgen	unreachable	vanilla	total
finished onboarding	62 (54.4%)	43 (37.7%)	9 (7.9%)	114
did not finish	26 (27.4%)	63 (66.3%)	6 (6.3%)	95

Both readings are true at once, and they point at different work:

1. Much of it is unfinished setup. Unreachable runs 66.3% among installs that never finished the wizard, against 37.7% among those that did. An install that stopped halfway through onboarding has no configured subgen, so this is expected and folds into the onboarding gap rather than being a bug.

2. But 43 installs completed onboarding and still cannot reach their subgen. That is 37.7% of fully-onboarded installs, and it is the real finding. These are people who went through the whole wizard, told us where their subgen is, and it does not answer. Roughly one in three successful setups is degraded in a way we currently do nothing about.

So this issue stands on its own rather than folding into the onboarding gap. The scope narrows usefully though: the population to diagnose is the 43 finishers, not all 108 unreachable.

The remaining unknowns from the original post still hold for that cohort, and they are what a fix depends on:

• unreachable still collapses several distinct causes (wrong URL, container down, cross-stack network isolation, up-but-failing-probe) and the telemetry cannot separate them.
• Persistent vs transient is still unmeasured. A probe that failed once at ping time is not the same as never having connected.

Splitting unreachable into cause buckets is probably the next concrete step, because every candidate fix differs by cause and right now we would be guessing which one to build.

[coaxk]

Investigated. The headline number is real but it is not 108 broken connections, and the metric cannot currently tell us which it is.

Where unreachable comes from

SubgenCapabilities.unreachable() is returned from two structurally different conditions in probe_capabilities(), and the difference is discarded at the moment it is known:

1. httpx.HTTPError on GET /status. A transport failure. Covers DNS failure, connection refused, connect timeout, read timeout and TLS error, which are four different diagnoses.
2. r.status_code != 200. Something IS listening and answered; it is just not a healthy subgen. A reverse-proxy 502, an auth wall, a wrong port hitting a different service, or subgen itself erroring.

Both collapse to the same sentinel, so the telemetry cannot distinguish "nothing there" from "something there that is not subgen".

Existing telemetry cannot answer it, checked rather than assumed

I looked for the answer in data we already have before proposing any new field. error_counts_30d carries {ExceptionClass: count} and would have been ideal.

104 of the 108 unreachable installs report error_counts_json = {}. Only 3 carry any SubgenUnavailable at all. The probe logs a warning and returns the sentinel; it never records an error, so nothing reaches the store.

That emptiness is itself a signal. An install genuinely using a broken subgen would accumulate SubgenUnavailable from real job submissions. 104 installs with a dead subgen and zero errors in 30 days are, for the most part, not trying to use it.

The likely explanation, and why it is a defect in the metric

SUBGEN_URL defaults to http://subgen:9000, a hostname that only resolves if you happen to run a container named subgen on the same docker network.

Worse, subgen_url goes through _env_or, so setting SUBGEN_URL= to disable it falls back to the default anyway. There is no way to express "I do not use subgen". Every such install probes a hostname that does not exist and reports unreachable, indistinguishably from someone whose real subgen just went down.

⚠️ This is a hypothesis with strong supporting evidence, not a proven fact. It is exactly what one boolean would settle.

Scoping it properly

Restricting to installs that finished onboarding and cross-tabbing against actual use:

subgen	walking, real library	never walked	total
working	34 (+6 small lib)	23	63
unreachable	12 (+1 small lib)	30	43

The confirmed-broken-and-active population is about 12 installs, not 108 and not 43. Those are walking a real library with a dead transcription path, which is unambiguously the bug this issue is about.

Of the other 30, none walked and none logged an error. That is consistent with never having set subgen up.

⚠️ Reachability correlates strongly with actually using the product (40 of 63 working installs walk, against 13 of 43 unreachable). The direction of causation is genuinely unknown and I am not going to assert one: a dead subgen may stop people using subarr, or people who drifted away may simply never have fixed it.

Proposed next step: measure, do not build yet

Two small, non-fingerprintable additions settle the whole question:

• subgen_url_is_default (bool). Splits "never configured" from "configured and broken" in one field. Transmits no URL.
• The failure class rather than the sentinel: transport-vs-HTTP-status, and for transport the httpx exception class. Splits "nothing there" from "something answered that was not subgen". No hostname, no URL.

That turns one ambiguous bucket into an answerable one, costs almost nothing, and tells us how large the real problem is before anyone designs a fix for it.

The product question behind it, which the data should inform rather than precede: subgen is optional in practice but mandatory in the config model. If a meaningful share of these are deliberate non-users, they need a way to say so, and subarr should stop reporting them as broken.

[coaxk]

Measurement shipped. Receiver side is coaxk/subarr-telemetry#50, merged and deployed; client side is #482.

What now exists

subgen_probe_failure carries one token from a closed vocabulary: dns, refused, connect_timeout, read_timeout, timeout, tls, transport, not_probed, http_<code>. subgen_target_is_default says whether the install is still on the shipped SUBGEN_URL. Neither transmits a hostname, URL or port.

Verified with real probes on Linux, which is what runs in production: the shipped default http://subgen:9000 classifies as dns, and a closed port as refused.

Two things that changed my reading of this issue while building it

The default URL is very likely the whole story for most of the 108. SUBGEN_URL defaults to http://subgen:9000, and _env_or maps an empty SUBGEN_URL back to that default, so there is no way to express "I do not use subgen". Such an install probes a hostname that does not exist and reports identically to one whose real subgen died. Combined with 104 of 108 logging zero errors in 30 days, the never-configured explanation is strong. The boolean will now confirm or kill it rather than leaving it as inference.

The confirmed-broken-and-active population is about 12, not 108. Restricting to installs that finished onboarding AND are walking a real library, only 12 have a dead subgen. That is the number a fix should be sized against, and it is very different from the headline.

When this is answerable

Not immediately. The fields only arrive from installs running the new client, and telemetry is daily, so this needs an upgrade cycle. Current versions do dominate the genuine userbase (194 of 214 are on 2.5.x), so coverage should build reasonably quickly rather than waiting on the 1.5.2 cohort, which is exactly the population that cannot upgrade cleanly anyway.

The query to run once data accumulates is a cross-tab of subgen_probe_failure against subgen_target_is_default, restricted to genuine installs. dns plus still-on-default is never-configured. dns plus configured is a wrong hostname. refused or an http_* code is a real connection problem regardless.

Deliberately not done

probe_failure now reaches /api/integrations/health via SubgenCapabilities.to_dict(), so the UI could show the user why their subgen is unreachable. That is the #458 lesson applied here: detection being fixed is not the same as the user being able to act. I have not built it, because the wording depends on what the data says the common cause actually is, and guessing now risks telling most users to check a hostname when their real answer is "you never set this up and that is fine".

The product question stands and is now measurable rather than speculative: subgen is optional in practice but mandatory in the config model. If the boolean shows most of these never configured it, they need a supported way to say so, and subarr should stop reporting them as broken.

[coaxk]

First real data, and it corrects the hypothesis I was working from. n=55 genuine installs carrying the new fields, latest ping each.

Cause breakdown

probe result	on default URL	installs
reachable	configured	18
reachable	default	18
dns	default	8
refused	configured	4
dns	configured	3
refused	default	3
transport	configured	1

My leading hypothesis was wrong

I argued the shipped default http://subgen:9000 was probably most of the story, on the reasoning that a hostname resolving only inside one compose topology would fail for nearly everyone using it.

18 installs are reachable while on that default. The intended topology works, and is widely used. "On the default" does not mean "never configured", which is exactly what the boolean was added to find out and it answered the opposite of what I expected.

The never-configured group is real but small: 8 installs are dns plus default, about 15%.

The numbers, corrected again

• Reachable is 65%, not the ~42% the pre-install_id does not persist, which inflates every install count ~100x and loses user history on restart #473 data implied.
• Configured but broken: 8 installs (4 refused, 3 dns, 1 transport). These people set a URL that does not work. This is the genuinely actionable group.
• 3 more are on the default with refused, meaning a subgen host resolves but nothing answers on the port. Container down, or wrong port.

Sized against actual use

Cross-tabbing against whether the install does anything:

state	walking, real library	never walked	total
reachable	18 (+1 small lib)	17	36
unreachable	3	16	19

The confirmed-broken-and-active population is 3 installs. Sixteen of the nineteen unreachable ones have never walked at all, which is consistent with setup that was started and abandoned rather than a connection that broke.

That number has come down every time the measurement improved: 108, then about 12, now 3. Each drop was a measurement artefact being removed, not the problem being fixed.

What I am not going to do

Design anything on n=55. This sample is only installs on 2.6.0 or a late :latest, so it is early upgraders, who are not a random slice of the userbase. The shape may move as adoption spreads.

Re-measuring in about a week is the right next step. If configured but broken stays around 15% of installs, a connection test in the setup wizard is the obvious cheap intervention, since it catches a wrong URL at the moment the user can still fix it. If it collapses the way the headline number has three times now, there is nothing here worth building.

The one thing worth saying plainly: this issue opened as "half of installs cannot reach their subgen". That was never true. It was an artefact of counting churned install ids, and every improvement in measurement has made the real problem smaller.