Monitoring: watch your sites and services, not just this server

ROADMAP.md

@@ -10,6 +10,8 @@ This document outlines the development roadmap for ServerKit. Features are organ
 
 ### Recently Completed (v1.7.x)
 
+- **Uptime Monitors** - Monitoring is no longer only about the machine the panel runs on. A monitor watches anything you point it at — a website, an API endpoint, a database port, a mail server or a managed WordPress site — with HTTP, keyword, port, ping, DNS and SMTP checks, per-monitor intervals, expected status ranges, redirect and TLS-verification switches, and a retry threshold before anything is declared down. A scheduler polls them continuously (previously checks only ran when someone pressed a button), outages open and close an incident on their own, and each monitor gets a detail page with a response-time chart, a 90-day uptime strip, a live check log and TLS certificate expiry. Host CPU/memory/disk alerting is unchanged and now shares one Incidents timeline with monitor outages, so "something is wrong" is a single place. Status pages publish monitors you already have rather than defining their own duplicate probes.
+- **Monitoring Navigation** - Monitors, Incidents, Events and Jobs are tabs of one Monitoring group instead of scattered top-level pages, and the events stream was rebuilt on the same search-and-filter layout as the rest of the panel.
 - **Themes & Theme Studio** - A gallery of color themes in Settings → Appearance (Paper, Nord Deep, Gruvbox, Phosphor, High Contrast, and the stock look), applied instantly and previewed live, kept as each user's personal choice on top of the dark/light toggle. Themes are data, not code — a small map of color tokens — so a built-in Theme Studio lets anyone edit colors over the live panel and export a shareable file, admins can set a panel-wide default or import a theme, and a public community registry installs themes with one click.
 - **Deploy Console & Live Run Logs** - Every install and deploy lands on one full-page live console: streaming build output, a step timeline with per-step durations and a live elapsed timer, and failure cards showing the real error tail with plain-language hints and one-click retry. Backed by a unified batched run-log layer (database persistence + WebSocket push with a polling fallback).
 - **New Service Wizard & Template Catalog** - Three-step service creation (Source / Connect / Review) backed by a server-side template catalog with one-click compose templates and Git-repo templates.

VERSION

@@ -1 +1 @@
-1.7.74
+1.7.75

backend/app/__init__.py

@@ -302,6 +302,11 @@ def create_app(config_name=None):
     from app.api.uptime import uptime_bp
     app.register_blueprint(uptime_bp, url_prefix='/api/v1/uptime')
 
+    # Register blueprints - Monitors (synthetic checks + their incidents).
+    # Core: watching a site must not depend on the status-page extension.
+    from app.api.monitors import monitors_bp
+    app.register_blueprint(monitors_bp, url_prefix='/api/v1/monitors')
+
     # Register blueprints - Environment Variables
     from app.api.env_vars import env_vars_bp
     app.register_blueprint(env_vars_bp, url_prefix='/api/v1/apps')

backend/app/api/monitors.py

@@ -0,0 +1,179 @@
+"""
+Monitors API
+
+Uptime/synthetic checks against URLs, hosts and managed sites. Core, not part of
+the serverkit-status extension — watching a site must not require publishing a
+status page. See app/services/monitor_service.py.
+"""
+
+from flask import Blueprint, request, jsonify
+from flask_jwt_extended import jwt_required
+
+from app.services.monitor_service import MonitorService
+
+monitors_bp = Blueprint('monitors', __name__)
+
+
+@monitors_bp.route('', methods=['GET'])
+@monitors_bp.route('/', methods=['GET'])
+@jwt_required()
+def list_monitors():
+    """List monitors.
+
+    Query params: status, type, q, page_id.
+    """
+    page_id = request.args.get('page_id', type=int)
+    monitors = MonitorService.list_monitors(
+        status=request.args.get('status') or None,
+        check_type=request.args.get('type') or None,
+        q=request.args.get('q') or None,
+        page_id=page_id,
+    )
+    # One extra query for every row's sparkline, not one per row.
+    sparks = MonitorService.recent_response_times([m.id for m in monitors])
+    payload = []
+    for monitor in monitors:
+        item = monitor.to_dict()
+        item['spark'] = sparks.get(monitor.id, [])
+        payload.append(item)
+    return jsonify({'monitors': payload})
+
+
+@monitors_bp.route('/stats', methods=['GET'])
+@jwt_required()
+def get_stats():
+    """KPI-band counts for the Monitors tab."""
+    return jsonify({'stats': MonitorService.stats()})
+
+
+@monitors_bp.route('', methods=['POST'])
+@monitors_bp.route('/', methods=['POST'])
+@jwt_required()
+def create_monitor():
+    data = request.get_json() or {}
+    try:
+        monitor = MonitorService.create(data)
+    except ValueError as e:
+        return {'error': str(e)}, 400
+    return jsonify(monitor.to_dict()), 201
+
+
+@monitors_bp.route('/<int:monitor_id>', methods=['GET'])
+@jwt_required()
+def get_monitor(monitor_id):
+    monitor = MonitorService.get(monitor_id)
+    if not monitor:
+        return {'error': 'Monitor not found'}, 404
+    return jsonify(monitor.to_dict())
+
+
+@monitors_bp.route('/<int:monitor_id>', methods=['PATCH', 'PUT'])
+@jwt_required()
+def update_monitor(monitor_id):
+    data = request.get_json() or {}
+    try:
+        monitor = MonitorService.update(monitor_id, data)
+    except ValueError as e:
+        return {'error': str(e)}, 400
+    if not monitor:
+        return {'error': 'Monitor not found'}, 404
+    return jsonify(monitor.to_dict())
+
+
+@monitors_bp.route('/<int:monitor_id>', methods=['DELETE'])
+@jwt_required()
+def delete_monitor(monitor_id):
+    if not MonitorService.delete(monitor_id):
+        return {'error': 'Monitor not found'}, 404
+    return jsonify({'deleted': True})
+
+
+@monitors_bp.route('/<int:monitor_id>/check', methods=['POST'])
+@jwt_required()
+def run_check(monitor_id):
+    """Probe now, out of band from the scheduler."""
+    hc = MonitorService.run_check(monitor_id)
+    if not hc:
+        return {'error': 'Monitor not found'}, 404
+    monitor = MonitorService.get(monitor_id)
+    return jsonify({'check': hc.to_dict(), 'monitor': monitor.to_dict()})
+
+
+@monitors_bp.route('/<int:monitor_id>/pause', methods=['POST'])
+@jwt_required()
+def set_paused(monitor_id):
+    data = request.get_json() or {}
+    monitor = MonitorService.set_paused(monitor_id, data.get('paused', True))
+    if not monitor:
+        return {'error': 'Monitor not found'}, 404
+    return jsonify(monitor.to_dict())
+
+
+@monitors_bp.route('/<int:monitor_id>/history', methods=['GET'])
+@jwt_required()
+def get_history(monitor_id):
+    """Recent check results. Query params: hours (default 24), limit."""
+    if not MonitorService.get(monitor_id):
+        return {'error': 'Monitor not found'}, 404
+    hours = request.args.get('hours', 24, type=int)
+    limit = request.args.get('limit', 200, type=int)
+    checks = MonitorService.get_check_history(monitor_id, hours=hours, limit=limit)
+    return jsonify({'checks': [c.to_dict() for c in checks]})
+
+
+@monitors_bp.route('/<int:monitor_id>/uptime', methods=['GET'])
+@jwt_required()
+def get_uptime(monitor_id):
+    """Per-day uptime buckets for the 90-day bar strip. Query param: days."""
+    monitor = MonitorService.get(monitor_id)
+    if not monitor:
+        return {'error': 'Monitor not found'}, 404
+    days = min(request.args.get('days', 90, type=int), 365)
+    return jsonify({
+        'days': MonitorService.uptime_days(monitor_id, days=days),
+        'uptime_24h': monitor.uptime_24h,
+        'uptime_7d': monitor.uptime_7d,
+        'uptime_30d': monitor.uptime_30d,
+        'uptime_90d': monitor.uptime_90d,
+    })
+
+
+# --- Incidents -------------------------------------------------------------
+# Incidents are core because the scheduler opens them automatically, with or
+# without the status-page extension installed.
+
+@monitors_bp.route('/incidents', methods=['GET'])
+@jwt_required()
+def list_incidents():
+    """Query params: state (active|resolved|all), limit."""
+    state = request.args.get('state', 'all')
+    limit = request.args.get('limit', 100, type=int)
+    incidents = MonitorService.list_incidents(state=state, limit=limit)
+    return jsonify({'incidents': [i.to_dict() for i in incidents]})
+
+
+@monitors_bp.route('/incidents', methods=['POST'])
+@jwt_required()
+def create_incident():
+    data = request.get_json() or {}
+    if not data.get('title'):
+        return {'error': 'Incident title is required'}, 400
+    incident = MonitorService.create_incident(data.get('page_id'), data)
+    return jsonify(incident.to_dict()), 201
+
+
+@monitors_bp.route('/incidents/<int:incident_id>', methods=['PATCH', 'PUT'])
+@jwt_required()
+def update_incident(incident_id):
+    incident = MonitorService.update_incident(incident_id, request.get_json() or {})
+    if not incident:
+        return {'error': 'Incident not found'}, 404
+    return jsonify(incident.to_dict())
+
+
+@monitors_bp.route('/incidents/<int:incident_id>', methods=['DELETE'])
+@jwt_required()
+def delete_incident(incident_id):
+    if not MonitorService.delete_incident(incident_id):
+        return {'error': 'Incident not found'}, 404
+    return jsonify({'deleted': True})

backend/app/jobs/builtin_handlers.py

@@ -84,22 +84,19 @@ def check_workflow_schedules():
 
 def run_health_checks():
     """Run a health check for every managed (production) WordPress site and sync
-    any status-page components bound to it. Per-site try/except so one hung site
-    never stalls the whole sweep."""
+    any monitors bound to it. Per-site try/except so one hung site never stalls
+    the whole sweep."""
     from app import db
     from app.models.wordpress_site import WordPressSite
     from app.models.status_page import StatusComponent
     from app.services.environment_health_service import EnvironmentHealthService
-    from app.services.plugin_service import get_installed_extension_attr
+    from app.services.monitor_service import MonitorService
 
     _prune_old_health_checks()
 
-    # Status-page component sync lives in the serverkit-status extension (plan 47);
-    # reach it only when installed. StatusComponent rows can't exist without it,
-    # so a lean panel simply runs the health checks without the sync.
-    StatusPageService = get_installed_extension_attr(
-        'serverkit-status', 'status_page_service', 'StatusPageService')
-
+    # The check engine is core now (it used to live in the serverkit-status
+    # extension and was reached through get_installed_extension_attr), so a lean
+    # panel with no status pages still keeps its site-bound monitors up to date.
     sites = WordPressSite.query.filter_by(is_production=True).all()
     for site in sites:
         try:
@@ -111,10 +108,11 @@ def run_health_checks():
             overall = result.get('overall_status')
             if not overall:
                 continue
-            if StatusPageService is not None:
-                components = StatusComponent.query.filter_by(wordpress_site_id=site.id).all()
-                for comp in components:
-                    StatusPageService.sync_component_from_health(comp, overall)
+            monitors = StatusComponent.query.filter_by(wordpress_site_id=site.id).all()
+            for monitor in monitors:
+                if monitor.is_paused:
+                    continue
+                MonitorService.sync_component_from_health(monitor, overall)
         except Exception as e:
             logger.error(f'Health check failed for site {site.id}: {e}')
             try:
@@ -123,6 +121,34 @@ def run_health_checks():
                 pass
 
 
+def run_monitor_checks():
+    """Poll every monitor whose interval has elapsed.
+
+    This is what actually makes monitoring happen: before it existed, checks only
+    ran when someone pressed "Check now", so a configured monitor never noticed an
+    outage on its own. Per-monitor try/except mirrors run_health_checks — one
+    unreachable target must not stall the sweep.
+    """
+    from app import db
+    from app.services.monitor_service import MonitorService
+
+    due = MonitorService.due_monitors()
+    if not due:
+        return
+    checked = 0
+    for monitor in due:
+        try:
+            MonitorService.run_check(monitor.id)
+            checked += 1
+        except Exception as e:
+            logger.error(f'Monitor check failed for monitor {monitor.id}: {e}')
+            try:
+                db.session.rollback()
+            except Exception:
+                pass
+    logger.debug('Monitor sweep polled %s of %s due monitors', checked, len(due))
+
+
 def _prune_old_health_checks():
     """Delete health-check samples older than the retention window, at most once
     per day, so the continuous poller doesn't grow the health_checks table without
@@ -315,6 +341,9 @@ def run_extension_update_check():
     ('builtin.snapshot_retention',  run_snapshot_retention,    'snapshot-retention', 3600,  120),
     ('builtin.workflow_schedules',  check_workflow_schedules,  'workflow-schedules', 60,    60),
     ('builtin.health_check',        run_health_checks,         'health-check',       300,   30),
+    # 30s tick, not the monitor interval: the sweep only polls monitors whose own
+    # interval has elapsed, so this bounds how late a 30s check can fire.
+    ('builtin.monitor_check',       run_monitor_checks,        'monitor-check',      30,    20),
     ('builtin.wp_update',           check_update_schedules,    'wp-update',          60,    105),
     ('builtin.api_background',      run_api_background,        'api-background',     3600,  3600),
     ('builtin.pairing_prune',       run_pairing_prune,         'pairing-prune',      3600,  60),