RFC: Expand Prometheus Metrics to Include ZFS Pools and Workload (issue #298)

[Starosdev]

This discussion is a request for feedback on the proposed scope for issue #298 -- expanding the existing Prometheus metrics exporter to cover ZFS pool health and workload insights.

#298

---

Background

Scrutiny already exports Prometheus metrics for SMART devices under /api/metrics (when web.metrics.enabled: true). The current implementation, in webapp/backend/pkg/metrics/collector.go, exposes:

• scrutiny_device_info -- device identity labels
• scrutiny_device_status -- pass/fail status (0 or 1)
• scrutiny_device_capacity_bytes
• scrutiny_smart_temperature_celsius
• scrutiny_smart_power_on_hours
• scrutiny_smart_power_cycle_count
• scrutiny_smart_collector_timestamp
• Per-attribute SMART metrics (dynamic, one gauge per numeric attribute per device)
• scrutiny_devices_total and scrutiny_devices_by_protocol

The collector implements the standard prometheus.Collector interface. Adding new metric families follows the same pattern: store the latest values in memory, emit them from Collect().

The ZFS pool and workload data already exists in Scrutiny -- it is stored in SQLite and InfluxDB, surfaced through the dashboard, and returned by the API. Prometheus export would make the current values available for external alerting and visualization without requiring a direct InfluxDB connection.

---

Proposed Scope

Rather than mirroring everything, the goal is to export the most actionable current values. The comment thread on the issue raised a fair concern about scope -- we are not trying to expose hundreds of metrics. The following is a focused, practical set.

ZFS Pool Metrics

scrutiny_zfs_pool_status
  Labels: guid, name, host_id
  Value:  0=ONLINE, 1=DEGRADED, 2=FAULTED, 3=OFFLINE, 4=REMOVED, 5=UNAVAIL

scrutiny_zfs_pool_size_bytes
  Labels: guid, name, host_id

scrutiny_zfs_pool_allocated_bytes
  Labels: guid, name, host_id

scrutiny_zfs_pool_free_bytes
  Labels: guid, name, host_id

scrutiny_zfs_pool_fragmentation_percent
  Labels: guid, name, host_id

scrutiny_zfs_pool_capacity_percent
  Labels: guid, name, host_id

scrutiny_zfs_pools_total
  (no labels -- count of monitored pools)

The pool GUID is the stable identifier used throughout the codebase. Name and host_id are included as labels for human-readable Grafana panels.

Workload Metrics

scrutiny_workload_daily_read_bytes_per_sec
  Labels: device_id, wwn, device_name, host_id

scrutiny_workload_daily_write_bytes_per_sec
  Labels: device_id, wwn, device_name, host_id

scrutiny_workload_intensity
  Labels: device_id, wwn, device_name, host_id, intensity
  Value:  1 (the intensity level is carried in the label: light/moderate/heavy/unknown)

Workload values are computed from InfluxDB over the selected duration window. For Prometheus export, we would use the "forever" window as the baseline, since Prometheus is polling for current state rather than a historical slice.

---

Implementation Approach

The Collector struct in collector.go would be extended:

1. Add a pools map (keyed by GUID) alongside the existing devices map
2. Add an UpdateZFSPoolMetrics(pool *models.ZFSPool) method, called from the ZFS pool upload handler
3. Add collectZFSPools(ch) and collectWorkload(ch) methods called from Collect()
4. Populate initial ZFS data in LoadInitialData() alongside the existing SMART data load
5. Workload values are computed on demand from the database; cache the latest computed result and refresh on each collector update

No new API endpoints are needed. The existing /api/metrics endpoint serves all Prometheus data.

---

Relationship to InfluxDB / Grafana

As noted in the issue thread: InfluxDB and Prometheus serve different purposes. InfluxDB already stores the full historical time-series for ZFS and workload data, and Grafana can query it directly via the InfluxDB data source. Prometheus export is useful for users who:

• Already have a Prometheus/Alertmanager/Grafana stack and do not want to configure a separate InfluxDB data source
• Want to trigger alerts on pool degradation without polling the Scrutiny API directly
• Prefer the Prometheus pull model for consistency with other system metrics

Both paths remain valid. This feature does not change the InfluxDB integration.

---

Questions for the Community

• Is the proposed metric set useful for your monitoring setup, or are there specific values you would prioritize differently?
• For workload intensity, is encoding it as a label on a constant-1 gauge the right approach, or would you prefer a numeric encoding (e.g., 0=unknown, 1=light, 2=moderate, 3=heavy)?
• Are there ZFS metrics not listed here that would be particularly useful for alerting? (For example: scrub status, last scrub errors, or vdev-level breakdown.)
• For users already using Grafana with InfluxDB: would Prometheus export change anything about how you use Scrutiny, or is this mainly useful for Prometheus-native setups?

All feedback is welcome. Once the scope is settled we will move forward with implementation.