Skip to content

feat(cloud): explain connection rejections and retry recoverable ones slowly - #92

Merged
marvinvr merged 2 commits into
mainfrom
issue-39
Sep 30, 2026
Merged

marvinvr merged 2 commits into
mainfrom
issue-39

Conversation

@marvinvr

Copy link
Copy Markdown
Owner

When DockTail Cloud refuses the agent's connection, the agent used to log one line with the raw reason code, for example cloud: hello rejected (terminal) — stopping, and then stop for the life of the process. The log gave no remediation and no hint that a restart was needed.

What changes

  • Every rejection gets a sentence that says what to do. Each reason code maps to one or two sentences naming the fix and the dashboard page where you apply it:

    • invalid key → /settings/agent-keys
    • blocked host → /hosts
    • host limit → /settings/billing
    • closed enrollment window → /settings/agent-keys

    A protocol mismatch also logs the agent's version and protocol version and says to pull a newer image. The classification lives in the new cloud/reject.go.

  • The explanation repeats. The full hint is logged on the first rejection and then every 30 minutes, including while the collector is stopped, so it stays near the end of docker logs. Retries in between log one short line.

  • Retry behaviour now depends on the reason:

    Reason Behaviour
    http_401, invalid_key Still stops. The key comes from the environment, so fixing it always means recreating the container.
    blocked, protocol_mismatch, http_403 Re-checks about every 15 minutes (±20%) for up to 24 hours, then stops. Each can clear without touching the agent: an unblock in the dashboard, a server-side fix, or a proxy/WAF rule fixed.
    over_cap Retries every 30–60 s, the same pace as enrollment_closed.
    unknown or legacy codes Ordinary backoff.
  • Docs: docs/06-cloud.md gains a "Connection Problems" table covering each reason, what it means, what the agent does and how to fix it.

The wire protocol is unchanged.

Not included

The mismatch message does not show the protocol version the server expects, because hello_ack does not carry it and adding it would need a protocol change.

Refs marvinvr/docktail-cloud#39

… slowly

Map every rejection to a sentence naming the fix and the dashboard page
that applies it, include the agent's protocol version on a mismatch, and
repeat the explanation every 30 minutes, including while the collector
is stopped.

A revoked or unknown key (HTTP 401 / invalid_key) still stops until the
container is recreated with a new key. A blocked host, a protocol
mismatch and an HTTP 403 from something in front of Cloud can all clear
without touching the agent, so they now re-check about every 15 minutes
for up to 24 hours before stopping. Over-cap rejections retry at the same
calm 30-60 s pace as a closed enrollment window.

Refs marvinvr/docktail-cloud#39
@marvinvr
marvinvr merged commit cf882ae into main Sep 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant