feat(rhodibot): deploy behind a Cloudflare Tunnel, loopback origin

bots/rhodibot/src/main.rs

@@ -36,6 +36,16 @@ struct Cli {
     #[arg(short, long, env = "PORT", default_value = "3000")]
     port: u16,
 
+    /// Address to bind.
+    ///
+    /// Loopback by default: the intended deployment fronts this process with a
+    /// Cloudflare Tunnel (or another reverse proxy) on the same host, so the
+    /// webhook port has no business being reachable from the network. Set
+    /// `--bind 0.0.0.0` (or `BIND_ADDR=0.0.0.0`) to serve directly, which is
+    /// what earlier versions did unconditionally.
+    #[arg(long, env = "BIND_ADDR", default_value = "127.0.0.1")]
+    bind: String,
+
     /// GitHub App ID
     #[arg(long, env = "GITHUB_APP_ID")]
     app_id: Option<u64>,
@@ -141,7 +151,7 @@ async fn main() -> Result<()> {
         .with_state(state);
 
     // Start server
-    let addr = format!("0.0.0.0:{}", cli.port);
+    let addr = format!("{}:{}", cli.bind, cli.port);
     let listener = TcpListener::bind(&addr).await?;
     info!("Listening on {}", addr);
 

deploy/GITHUB-APP-REGISTRATION.adoc

@@ -0,0 +1,100 @@
+= Registering the rhodibot GitHub App
+:toc:
+
+This sheet is filled in by hand in the browser: GitHub has no API to create an
+App (the GraphQL schema has no `createGitHubApp` mutation, and the REST API can
+only *list* and *read* Apps). Everything else about the deployment is
+reproducible from the repository; this page is the one manual step, so it is
+written down as a form to fill in rather than a memory to rely on.
+
+Record the values as you go. Two of them -- the App ID and the private key --
+cannot be recovered in this form later.
+
+== 1. Where
+
+https://github.com/settings/apps/new -- a personal App, which is what an owner
+of this estate needs. An organisation App is created at
+`https://github.com/organizations/<org>/settings/apps/new` and behaves the same
+otherwise, except that the App's owner controls it.
+
+== 2. The form
+
+[cols="1,1,2"]
+|===
+| Field | Value | Why
+| GitHub App name | `rhodibot` | Must be globally unique across GitHub; if taken, `rhodibot-<owner>`
+| Homepage URL | `https://github.com/hyperpolymath/gitbot-fleet` | Required by the form; unused by the bot
+| Webhook URL | `https://rhodibot.<your-domain>/webhook` | The tunnel ingress rule, `/webhook` exactly
+| Webhook secret | 32+ random bytes, e.g. `openssl rand -hex 32` | Goes in `rhodibot.env` as `GITHUB_WEBHOOK_SECRET`; without it the origin accepts unsigned deliveries
+|===
+
+Then, under *Repository permissions*:
+
+[cols="1,2,2"]
+|===
+| Permission | Access | Used for
+| Metadata | Read-only | Mandatory once any other permission is set
+| Checks | Read and write | `createCheckRun` / `updateCheckRun` -- the compliance result on a commit
+| Issues | Read and write | `createIssue` for drift that is not tied to a commit
+| Contents | Read-only | Reading repository trees and file contents
+| Pull requests | Read-only | Reviewing the PRs the fixers open
+|===
+
+Nothing else. Every unused permission is a permission that can be abused by a
+mistake in this codebase, and the bot has no use for administration, actions,
+secrets, or organisation-level access. If a future rule needs more, add it then,
+in its own commit, with the reason.
+
+Under *Subscribe to events*: `push`, `pull_request`, `repository`,
+`installation`, `installation_repositories`. The handler ignores everything else
+by name (the match in `bots/rhodibot/src/main.rs` is exhaustive in intent: an
+unknown event is logged and dropped).
+
+== 3. The two values you cannot get back
+
+*App ID* (top of the App's settings page, "About"):
+
+[source]
+----
+GITHUB_APP_ID = ______________
+----
+
+*Private key*: "Generate a private key" downloads a `.pem` exactly once. GitHub
+stores only the public half; a lost key means generating a new one and
+redeploying, which is survivable but avoidable.
+
+[source]
+----
+downloaded at: ______________
+fingerprint:   openssl rsa -in rhodibot.<date>.private-key.pem -pubout | sha256sum
+stored to /etc/fleet/rhodibot-app.pem, 0600 root:root
+----
+
+The PEM is in PKCS#1 form (`BEGIN RSA PRIVATE KEY`). `AppAuth` also accepts
+PKCS#8, so converting it first is optional, not required.
+
+== 4. Install
+
+"Install App" -> the account -> *Only select repositories*. Start with one
+repository that has no drift, so that the first deliveries produce boring
+results and a wrongly-scoped installation is visible immediately.
+
+Do not choose "All repositories" to begin with: it is the largest possible
+blast radius for a bot whose checks have never run against a real installation.
+
+== 5. Confirm the service picked it up
+
+[source,bash]
+----
+systemctl restart rhodibot
+journalctl -u rhodibot -n 20 --no-pager      # "GitHub credentials: GitHub App installation tokens"
+curl -sS https://rhodibot.<your-domain>/health
+----
+
+== 6. Clean up
+
+* The key is installed at `/etc/fleet/rhodibot-app.pem`; delete the copy in
+  `~/Downloads` once the password-manager copy is confirmed.
+* Keep the webhook secret in the password manager too: it is shown again in the
+  App's settings, so it is recoverable, but only to someone with account access.
+* `GITHUB_APP_ID` is not secret and belongs in the deployment notes.

deploy/RHODIBOT-DEPLOYMENT.adoc

@@ -0,0 +1,139 @@
+= Deploying rhodibot behind a Cloudflare Tunnel
+:toc:
+
+Rhodibot needs a public HTTPS endpoint because that is where GitHub delivers
+webhooks, but the host it runs on does not need to be reachable. A Cloudflare
+Tunnel gives us the first without the second: `cloudflared` dials out to
+Cloudflare's edge and forwards requests back down that connection to a loopback
+origin. No inbound port, no certificate to renew on this host.
+
+The pieces:
+
+[cols="1,2"]
+|===
+| `deploy/systemd/rhodibot.service` | the bot itself, bound to `127.0.0.1:3000`
+| `deploy/systemd/cloudflared-rhodibot.service` | the tunnel
+| `deploy/cloudflared/rhodibot.yml` | ingress: `/webhook` and `/health`, everything else 404
+| `deploy/rhodibot.env.example` | the App credentials, as environment variables
+| `deploy/verify-rhodibot.sh` | proof of life, from the outside in
+|===
+
+== 1. Register the GitHub App first
+
+The App ID, private key and webhook secret all feed the service, and GitHub will
+start delivering webhooks as soon as the App exists. Do
+xref:GITHUB-APP-REGISTRATION.adoc[the registration sheet] before this, or accept
+a few minutes of failed deliveries while you catch up -- they are redeliverable
+from the App's "Advanced" tab, which is only mildly annoying at this scale.
+
+== 2. Build and install the binary
+
+[source,bash]
+----
+cargo build --locked --release --manifest-path bots/rhodibot/Cargo.toml
+install -D -m 0755 bots/rhodibot/target/release/rhodibot /opt/gitbot-fleet/bots/rhodibot/rhodibot
+install -d -m 0755 /opt/gitbot-fleet/bots/rhodibot
+----
+
+== 3. Place the credentials
+
+[source,bash]
+----
+install -d -m 0750 /etc/fleet
+install -m 0600 -o root -g root ~/Downloads/rhodibot.<date>.private-key.pem /etc/fleet/rhodibot-app.pem
+install -m 0640 -o root -g fleet /dev/null /etc/fleet/rhodibot.env
+$EDITOR /etc/fleet/rhodibot.env      # from deploy/rhodibot.env.example
+----
+
+The private key is downloadable exactly once. Put it in the password manager
+before it is deleted from the download directory.
+
+Note: `ProtectHome=true` in the unit means the key cannot live under `/home`,
+and `ProtectSystem=strict` makes the rest of the filesystem read-only, so
+`/etc/fleet` is the only sensible home for it.
+
+== 4. Create the tunnel
+
+[source,bash]
+----
+cloudflared tunnel login                       # browser step, picks the zone
+cloudflared tunnel create rhodibot             # prints the UUID and writes the credentials file
+install -D -m 0600 -o cloudflared -g cloudflared \
+  ~/.cloudflared/<TUNNEL-UUID>.json /etc/cloudflared/rhodibot.json
+install -D -m 0644 \
+  deploy/cloudflared/rhodibot.yml /etc/cloudflared/rhodibot.yml
+$EDITOR /etc/cloudflared/rhodibot.yml          # replace <TUNNEL-UUID> and <your-domain>
+cloudflared tunnel ingress validate --config /etc/cloudflared/rhodibot.yml
+----
+
+Then point DNS at the tunnel and check the routing table before starting
+anything:
+
+[source,bash]
+----
+cloudflared tunnel route dns rhodibot rhodibot.<your-domain>
+cloudflared tunnel ingress rule https://rhodibot.<your-domain>/health
+cloudflared tunnel ingress rule https://rhodibot.<your-domain>/api/check/x/y   # must be http_status:404
+----
+
+== 5. Start, in order
+
+[source,bash]
+----
+install -m 0644 deploy/systemd/rhodibot.service deploy/systemd/cloudflared-rhodibot.service /etc/systemd/system/
+systemctl daemon-reload
+systemctl enable --now rhodibot.service
+journalctl -u rhodibot -n 20 --no-pager    # expect: "GitHub credentials: GitHub App installation tokens"
+systemctl enable --now cloudflared-rhodibot.service
+----
+
+If the credential line says `misconfigured`, the App ID and key do not agree --
+the service exits rather than serving unauthenticated traffic, and `systemd`
+stops retrying after five attempts. Fix `rhodibot.env` and restart.
+
+== 6. Verify from the outside
+
+[source,bash]
+----
+RHODIBOT_HOST=rhodibot.<your-domain> ./deploy/verify-rhodibot.sh
+----
+
+Every line must read `PASS`; `SKIP` means the check could not be made, not that
+it passed. The check that matters most is the unsigned `POST /webhook`, which
+proves both that the tunnel reaches the origin and that the webhook secret is
+set -- without a secret the handler accepts anything, and the tunnel would
+forward it to GitHub's own write path.
+
+== 7. Prove a delivery end to end
+
+In the App's settings, "Advanced" -> recent deliveries -> pick one -> *Redeliver*.
+Then:
+
+[source,bash]
+----
+journalctl -u rhodibot -f
+----
+
+A `push` delivery should log `Received webhook event: push`. Under ten seconds
+of silence means the tunnel is not reaching the origin; see below.
+
+== Troubleshooting
+
+[cols="1,2"]
+|===
+| 502 from Cloudflare | `rhodibot.service` is not running, or is not on the port the ingress rule names |
+| 401 on every delivery | webhook secret in the App and in `rhodibot.env` differ |
+| 404 on a valid delivery | the ingress rule does not match the path GitHub used |
+| `misconfigured` in the journal | App ID and private key do not belong together; the key must be the PEM as downloaded |
+| App requests 403 or 404 | the installation is missing the repository, or the App's permissions were changed and not approved |
+| rate limits hit during a sweep | an installation token is being minted per request; check `start-up` logs for the credential mode, and see the caching notes in `bots/rhodibot/src/app_auth.rs` |
+|===
+
+== Notes
+
+* The origin is loopback-only (`--bind 127.0.0.1`). To serve without a tunnel,
+  pass `--bind 0.0.0.0`, and put the webhook secret and a firewall in front of it.
+* `/api/check/{owner}/{repo}` is deliberately not published: it spends GitHub
+  API quota and reports on repositories, and neither is a public service.
+* The bot holds no state -- no database, no writable paths -- so backups are
+  limited to `/etc/fleet` and the App registration.

deploy/cloudflared/rhodibot.yml

@@ -0,0 +1,51 @@
+# SPDX-License-Identifier: MPL-2.0
+#
+# Cloudflare Tunnel ingress for rhodibot.
+#
+# The tunnel is the *origin* for GitHub's webhook delivery: GitHub must reach a
+# public HTTPS endpoint, and this host has no inbound exposure. cloudflared dials
+# out to Cloudflare's edge and reverses the flow, so no port is opened on the
+# firewall and no certificate is managed here.
+#
+# Replace these two placeholders before installing (see RHODIBOT-DEPLOYMENT.adoc):
+#   <TUNNEL-UUID>   from `cloudflared tunnel create rhodibot`
+#   <your-domain>   a zone you control on Cloudflare
+#
+# Verify with:
+#   cloudflared tunnel ingress validate --config deploy/cloudflared/rhodibot.yml
+#   cloudflared tunnel ingress rule https://rhodibot.<your-domain>/health
+
+tunnel: <TUNNEL-UUID>
+credentials-file: /etc/cloudflared/rhodibot.json
+
+# rhodibot binds loopback only (see --bind in bots/rhodibot/src/main.rs): the
+# origin is a local process, not a public interface.
+originRequest:
+  connectTimeout: 30s
+  # Webhook deliveries carry a small JSON body; anything larger is not a
+  # delivery GitHub would send, and there is no reason to buffer it.
+  noHappyEyeballs: false
+
+ingress:
+  # Only the two routes the service needs. GitHub delivers webhooks to
+  # POST /webhook and nothing else; /health is here so the tunnel can be
+  # checked without shell access to the host.
+  - hostname: rhodibot.<your-domain>
+    path: ^/webhook$
+    service: http://127.0.0.1:3000
+
+  - hostname: rhodibot.<your-domain>
+    path: ^/health$
+    service: http://127.0.0.1:3000
+
+  # Deliberately *not* exposed: GET /api/check/{owner}/{repo}, which spends
+  # GitHub API quota and reports on repositories. It is an operator endpoint,
+  # reachable over an SSH tunnel or from the host, and it stays that way.
+  # Everything else on this hostname is a 404, including anything added to the
+  # router later without a matching rule here -- the default is denial.
+  - hostname: rhodibot.<your-domain>
+    service: http_status:404
+
+  # Any other hostname pointing at this tunnel (a stale DNS record, say) also
+  # gets nothing.
+  - service: http_status:404

deploy/rhodibot.env.example

@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: MPL-2.0
+#
+# Copy to /etc/fleet/rhodibot.env, fill in, then:
+#   chown root:fleet /etc/fleet/rhodibot.env && chmod 640 /etc/fleet/rhodibot.env
+#
+# systemd reads this file for rhodibot.service. Nothing in it should be
+# world-readable: the webhook secret authenticates deliveries, and the path
+# below points at the App's private key.
+
+# GitHub App ID. Visible on the App's settings page; not a secret.
+GITHUB_APP_ID=
+
+# Path to the App private key, downloaded once when the key is generated.
+# The file is write-only-from-GitHub: it cannot be re-downloaded, so keep a
+# copy in the password manager before anything else touches the original.
+GITHUB_PRIVATE_KEY_PATH=/etc/fleet/rhodibot-app.pem
+
+# The secret entered under "Webhook secret" in the App's settings. Deliveries
+# are signed with it (HMAC-SHA256); without it the service accepts unsigned
+# POSTs to /webhook, which the public tunnel would happily forward.
+GITHUB_WEBHOOK_SECRET=
+
+# GitHub Enterprise only. Leave unset for github.com.
+# GITHUB_API_URL=https://api.github.com
+
+# Where the process listens. Loopback, because cloudflared fronts it.
+PORT=3000
+BIND_ADDR=127.0.0.1
+
+RUST_LOG=rhodibot=info,tower_http=info