Anton-Babaskin · Sep 30, 2026 · Sep 30, 2026 · Sep 30, 2026 · Sep 30, 2026
@@ -5,6 +5,12 @@ Preview tags are immutable and are not silently promoted to stable.
 
 ## Unreleased
 
+- Private staging monitor: bounded disk/inode, authenticated readiness, queue,
+  failure and credential-cleanup checks; installed/served TLS expiry checks;
+  redacted local alerts with deduplication and recovery, without new listeners.
+- Docker lab launcher selects an unused subnet rather than randomly colliding
+  with retained stopped labs; ENOSPC setup errors expose only operation labels.
+
 - Staging updater rejects active jobs/envelopes before service stop; pilot
   authority preflight reads nested API snapshot status correctly.
 

@@ -136,6 +136,10 @@ separate reviewed plan; other Docker projects and host rules must be preserved.
 
 ## Still required before public use
 
+For the lightweight private operator timer, thresholds and local alert reports,
+see [staging monitoring](../../docs/STAGING-MONITORING.md). Outbound notification
+delivery and an independent external uptime check still require configuration.
+
 DNS/NAT ownership, trusted HTTPS, restricted pilot admission, off-site encrypted
 backup and restore, retention/alerts, full reboot acceptance, host updates and
 SSH hardening with verified fallback access. No production launch, payment flow,

@@ -0,0 +1,37 @@
+[Unit]
+Description=MoveMailbox private operator diagnostics
+After=docker.service movemailbox-staging.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/python3 /opt/movemailbox/scripts/staging_monitor.py
+User=root
+UMask=0077
+StateDirectory=movemailbox-monitor
+StateDirectoryMode=0700
+TimeoutStartSec=75
+MemoryMax=128M
+CPUQuota=25%
+TasksMax=16
+NoNewPrivileges=yes
+ProtectSystem=strict
+ProtectHome=yes
+PrivateTmp=yes
+PrivateDevices=yes
+ProtectKernelTunables=yes
+ProtectKernelModules=yes
+ProtectKernelLogs=yes
+ProtectControlGroups=yes
+RestrictSUIDSGID=yes
+RestrictRealtime=yes
+LockPersonality=yes
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+CapabilityBoundingSet=CAP_DAC_READ_SEARCH
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=movemailbox-monitor
+
+# Root reads private metadata and invokes read-only Docker inspect commands.
+# Docker socket access is still privileged; this is not a Docker authorization boundary.
+# Read-only SQLite may need to create/map SHM when the writer is unavailable.
+# Deny filesystem writes to live volumes: inability to read becomes an alert.
@@ -0,0 +1,11 @@
+[Unit]
+Description=Check private MoveMailbox stage every five minutes
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+AccuracySec=15s
+Unit=movemailbox-monitor.service
+
+[Install]
+WantedBy=timers.target
@@ -1,5 +1,59 @@
 # Engineering handoff — 2026-09-30
 
+## VM capacity and operator monitoring (2026-09-30, technical scope)
+
+- Owner approved stage 1 and VM-only root expansion. Revalidated 50 GiB disk,
+  ext4 root `/dev/ubuntu-vg/ubuntu-lv` and 24 GiB free in its VG. Online
+  `lvextend --resizefs` expanded root from about 24 to 48 GiB. Free root space
+  increased from 5.4 to about 29 GiB (38% used). No partition, virtualization
+  host, firewall, VPN, credentials or runtime image change. API/worker healthy;
+  readiness 200 and container start times unchanged.
+- Branch `ops/staging-monitor`: private five-minute systemd diagnostics for
+  root space/inodes, selected container state, authenticated API/worker
+  readiness, aggregate queue delay/stalled progress/recent failures/envelopes,
+  installed certificate expiry and actual loopback served-certificate trust/name
+  and copy match. No credential/event/snapshot payloads or job IDs in reports;
+  SQLite query-only reads include committed WAL. No automatic restart/replay.
+- Added [monitoring runbook](STAGING-MONITORING.md). Alerts and recoveries are
+  deduplicated into local journal plus atomic root-private state; no outbound
+  channel or independent external heartbeat yet. Root/Docker access remains
+  privileged, despite the read-only commands and systemd filesystem hardening.
+- Local Linux/WSL Python: 74 tests ran with one missing-age integration skip;
+  12 new monitor tests cover fault thresholds, committed WAL, redaction,
+  expiry/mismatch, queue/error counts, dedup/recovery and corrupt/unsafe state.
+  Go race tests and vet passed; Go formatting clean; docs check passed.
+- Initial PR #59 CI at a497899 stopped during old ENOSPC lab setup, before
+  active application testing; the latest main CI also failed at that setup.
+  Original Docker stderr was withheld, so the exact old failure is not proven.
+  Found/reproduced a real launcher flaw: hashed subnets could collide with
+  retained stopped labs. Launcher now inspects IPAM and selects a free /28;
+  no existing networks removed, concurrent allocation still fails closed.
+  A real WSL Docker occupied-candidate test passed; only its two newly created
+  empty networks were removed. Four additional offline tests cover collisions,
+  full/invalid inventory and error redaction. ENOSPC errors now include a fixed
+  operation label, not Docker arguments/environment/output.
+- Read-only VM candidate probe under a transient systemd filesystem sandbox
+  passed in about 0.6 s: no alerts, API/worker ready, two completed jobs per
+  store, zero envelopes/active work, served certificate matches and about
+  74 days remaining.
+- All seven CI checks passed at 2abe88718ea01196d3907726e8d0ecaef1fe50bd
+  (run 36711089681), including ENOSPC/crash tests with the corrected launcher.
+  Installed the three monitor files from that reviewed Git archive on the VM;
+  verified checksum, inventory and root-controlled previously absent targets.
+  Permanent timer enabled at 11:58 UTC; first actual hardened unit run succeeded,
+  no alerts; root-private state 0700 and JSON/lock 0600. Next automatic run was
+  scheduled five minutes later. API/worker/proxy image IDs and start times were
+  compared before/after and unchanged. Final evidence-only commit is in PR #59;
+  merge requires green checks for that exact final head too.
+- Runtime stays v0.5.0-rc.1, image
+  `sha256:c3b4994e0692a0c1d34a65066f00296e524cccbbbdf606bbc4429b3f99a4f7ca`.
+  No new mail/provider load test needed for this operator-only stage.
+- Next two concrete steps: (1) real native Windows IMAP acceptance using a
+  compatible imapsync plus owner-approved license choice before stable;
+  (2) choose/test outbound alert delivery and an independent HTTPS check, then
+  owner-controlled certificate renewal automation. Off-site backups stay
+  deferred by owner; paid identity/billing and public launch remain separate.
+
 ## v0.5.0-rc.1 published and promoted (2026-09-30, technical scope)
 
 - PR #56 merged after all seven CI checks passed for full source commit

@@ -128,6 +128,12 @@ execute the same lease, and quota decisions remain consistent under load.
 
 ## 5. Operations and commercial readiness
 
+Private staging: a bounded five-minute operator monitor covers disk/inodes,
+container and authenticated worker readiness, queue/failures/envelope counts,
+and installed/served TLS certificates. See [monitoring runbook](STAGING-MONITORING.md).
+Alerts currently remain in the local journal/private report; owner delivery,
+independent external uptime monitoring and automatic renewal are not configured.
+
 - structured redacted logs, metrics, alerts and a status page;
 - privacy policy, terms, refund rules, subprocessors and retention schedule;
 - final MoveMailbox license/distribution decision and third-party notices;

@@ -0,0 +1,112 @@
+# Private staging monitoring
+
+This is a small single-VM operator monitor, not a public metrics endpoint or a
+host intrusion-detection system. No new listener, external request, migration,
+restart, credential read, firewall change or automatic cancellation is performed.
+The owner remains responsible for responding to alerts.
+
+## Checks and thresholds
+
+| Check | Warning | Critical |
+| --- | --- | --- |
+| Root filesystem available space | Below 10% **or** 2 GiB | Below 5% **or** 1 GiB |
+| Root filesystem available inodes | Below 10% | Below 5% |
+| API/worker container health, proxy running | — | Missing/not healthy/not running |
+| API readiness | — | Not ready/invalid response/timeout |
+| Worker queue | At least 8 waiting, or oldest waiting over 15 min | Cannot read queue |
+| Worker progress | Running record unchanged over 15 min | — |
+| API/worker failures | At least one failed job in the last hour | Cannot read metadata |
+| Credential envelopes | — | More envelopes than active worker jobs |
+| Gateway certificate lifetime | Less than 30 days | Less than 7 days/expired/unreadable |
+| Actual gateway TLS | Certificate differs from installed copy | Trust/name/handshake failure |
+
+API readiness includes authenticated worker readiness and API persistence
+health, not just an open port. The proxy has no Docker healthcheck: its running
+state and a real, certificate-verified TLS handshake on VM loopback are checked.
+The TLS probe is not an external DNS/NAT/BasicAuth test; it cannot detect a broken
+public forwarding rule. Normal external HTTPS acceptance remains a separate gate.
+
+All subprocesses/network calls have five-second timeouts; SQLite reads have a
+one-second busy timeout and two-second query progress deadline. Each probe fails
+independently. The unit has a 75-second overall bound, 128 MiB memory cap and 25%
+CPU quota. Five-minute sampling is not real-time or an audit ledger: a failure
+purged between samples can be missed. Long IMAP APPENDs can legitimately look
+like stalled progress. Investigate; do not automatically rerun/mirror/delete.
+
+The root filesystem currently holds Docker's data volumes too. If Docker or
+metadata is moved to a separate filesystem, add a separate space/inode probe
+before deployment. Cross-store counts are not an atomic distributed snapshot;
+each individual worker read is consistent, including committed WAL data.
+
+## Installation on the dedicated staging VM
+
+Install only reviewed files from an exact Git commit, **not** a dirty checkout.
+Preflight the fixed targets and their parents: root-owned, not writable by other
+users, not symbolic links. Refuse unexpected existing files; preserve their
+contents and compare before an explicit reviewed update. Do not run this on the
+virtualization host.
+
+- Install `scripts/staging_monitor.py` as root-owned mode 0644 at
+  `/opt/movemailbox/scripts/staging_monitor.py`.
+- Install the two matching files from `deploy/staging/` as root-owned mode 0644
+  at `/etc/systemd/system/movemailbox-monitor.service` and
+  `/etc/systemd/system/movemailbox-monitor.timer`.
+- Run `systemd-analyze verify` on both units, `systemctl daemon-reload`, then
+  `systemctl enable --now movemailbox-monitor.timer`.
+- Run `systemctl start movemailbox-monitor.service` for the first sample.
+  Existing API/worker/proxy must not be restarted.
+
+The service manages `/var/lib/movemailbox-monitor` (root 0700), with a lock and
+atomic `latest.json` (0600). It reads live databases with SQLite `mode=ro` and
+`query_only`; its filesystem sandbox also denies writes to live volumes. SQLite
+may need SHM creation if a writer is unavailable; a denied read becomes an alert,
+not a reason to allow volume writes or change journal mode. The Docker socket is
+privileged even though commands only inspect selected fields. This systemd
+hardening is not a Docker authorization boundary or protection against root.
+
+Read-only operator commands:
+
+```sh
+sudo systemctl list-timers movemailbox-monitor.timer
+sudo systemctl status movemailbox-monitor.service --no-pager
+sudo journalctl -u movemailbox-monitor.service --since '1 hour ago' --no-pager
+sudo python3 -m json.tool /var/lib/movemailbox-monitor/latest.json
+```
+
+An alert exits 1 (unit failed until a healthy sample); internal monitor errors
+exit 2 and log only `monitor.internal`, with exception text withheld. Repeated
+incidents are deduplicated; severity changes and recoveries emit new events.
+Every completed run emits a summary. Corrupt/insecure state fails closed rather
+than silently resetting the baseline. Stop the timer before an intentional
+maintenance window and restart it afterwards; do not hide genuine failures by
+deleting the state file. Review any recovery before considering the incident closed.
+
+## Alert delivery and certificate renewal
+
+Current delivery is **local journal + root-private report only**. No Telegram,
+email, webhook, off-host heartbeat or public status page is configured. If the
+VM or timer dies, it cannot notify the operator itself. Before a public launch,
+choose an owner-controlled outbound channel and an independent external
+availability check; store webhook/SMTP credentials outside Git and test both
+alert and recovery delivery with synthetic faults.
+
+For a small pilot, a Telegram bot/channel is easy to operate but depends on that
+service and sends operational metadata off-host. Email fits existing admin
+workflows but needs authenticated SMTP and can be delayed/filtered. Do not send
+addresses, job IDs, logs, envelope data or passwords through either channel.
+
+Certificate issuance currently uses manual DNS-01. Monitoring does **not** renew
+it. At the warning threshold schedule DNS verification/renewal, validate the new
+certificate and key privately, then update the gateway copy and reload nginx
+through the reviewed HTTPS runbook. A renewed source certificate without a
+gateway reload is not a completed renewal; the served-copy comparison catches it.
+
+## Focused verification
+
+`python3 -m unittest discover -s scripts -p test_staging_monitor.py -v` runs on
+Linux/WSL. It covers thresholds, expiry, queue delay, credential cleanup counts,
+read-only committed WAL reads, recent versus old errors, independent failures,
+TLS mismatch, redaction, dedup/escalation/recovery and unsafe/corrupt state.
+Tests use local synthetic databases; do not fill the VM disk, stop a real worker
+or load-test a mail provider just to exercise monitoring. Validate the installed
+unit's actual healthy report and subsequent timer execution on the VM too.
@@ -246,5 +246,11 @@ def run(args, temporary):
             run(args, temporary)
     except Exception as exc:
         frame = traceback.extract_tb(exc.__traceback__)[-1]
-        print("FAIL:", type(exc).__name__, getattr(exc, "sqlite_errorname", ""), "line", frame.lineno, "(sensitive details withheld)", flush=True)
+        print("FAIL:", type(exc).__name__, getattr(exc, "sqlite_errorname", ""),
+              Path(frame.filename).name, "line", frame.lineno, "(sensitive details withheld)", flush=True)
+        # Constant operation labels help distinguish setup from application faults.
+        # Never expose the exception string, Docker stderr, environment or args.
+        operation = getattr(exc, 'operation', None)
+        if operation in ('run', 'network', 'volume', 'image', 'inspect', 'exec', 'stop', 'pause', 'unpause', 'kill', 'start', 'logs'):
+            print('Docker operation:', operation, flush=True)
         raise SystemExit(1)