Describe the bug
When the influxdb service on node cn02 experiences an anomaly, attempting to recover it using check_repair or deep repair fails. Additionally, systemctl is unable to stop or restart the influxdb service.
Concurrently, the Node List appears completely blank. This issue has occurred consistently across multiple environment reboots (specifically, 3 times in the National Tsing Hua University (NTHU) environment). Although the node list eventually displayed normally after manual engineer intervention in previous instances, the root cause remains unresolved.
The underlying failure stems from a cluster disconnection (crm_mon connection loss), which corrupts or empties the low-level command outputs. The API logic lacks adequate defense-in-depth and inappropriately treats nil, empty strings, or contaminated standard outputs as valid responses, causing JSON parsing failures rather than structured error handling.
Version
COS 3.1.0
To Reproduce
Steps to reproduce the behavior:
-
Get into NTHU environment by the permissions information on 1Password.
-
Encounter or trigger an anomaly in the influxdb service on node cn02 such that the cluster connection is lost.
-
Attempt to run check_repair or deep repair (both will fail to repair).
-
Try to stop or restart the influxdb service using systemctl stop influxdb or systemctl restart influxdb (the service will hang or fail to respond).
-
Navigate to the Node Management interface / Dashboard.
-
See that the Node List is completely blank.
Expected behavior
-
Robust Error Handling & Fallbacks: If the cluster is disconnected or a downstream command fails, the UI should gracefully display an error state, a specific warning notification, or a partially cached list, rather than a completely blank Node List.
-
Service Process Lifecycle Control: systemctl operations on influxdb should either succeed, timeout safely, or terminate gracefully without locking up or preventing service recovery.
-
Defensive API Deserialization: The API backend should validate response payloads (checking status codes, checking for empty strings, and filtering warning text/garbage output) before attempting to unmarshal them as JSON.
Screenshots
.
Add any other context about the problem here.
- Hardware [e.g. CPU, RAM, Disk Controller, Disks]
- Supporting logs
Reference:
- Slack threads:
- Hotspot tickets:
Describe the bug
When the influxdb service on node cn02 experiences an anomaly, attempting to recover it using check_repair or deep repair fails. Additionally, systemctl is unable to stop or restart the influxdb service.
Concurrently, the Node List appears completely blank. This issue has occurred consistently across multiple environment reboots (specifically, 3 times in the National Tsing Hua University (NTHU) environment). Although the node list eventually displayed normally after manual engineer intervention in previous instances, the root cause remains unresolved.
The underlying failure stems from a cluster disconnection (crm_mon connection loss), which corrupts or empties the low-level command outputs. The API logic lacks adequate defense-in-depth and inappropriately treats nil, empty strings, or contaminated standard outputs as valid responses, causing JSON parsing failures rather than structured error handling.
Version
COS 3.1.0
To Reproduce
Steps to reproduce the behavior:
Get into NTHU environment by the permissions information on 1Password.
Encounter or trigger an anomaly in the influxdb service on node cn02 such that the cluster connection is lost.
Attempt to run check_repair or deep repair (both will fail to repair).
Try to stop or restart the influxdb service using systemctl stop influxdb or systemctl restart influxdb (the service will hang or fail to respond).
Navigate to the Node Management interface / Dashboard.
See that the Node List is completely blank.
Expected behavior
Robust Error Handling & Fallbacks: If the cluster is disconnected or a downstream command fails, the UI should gracefully display an error state, a specific warning notification, or a partially cached list, rather than a completely blank Node List.
Service Process Lifecycle Control: systemctl operations on influxdb should either succeed, timeout safely, or terminate gracefully without locking up or preventing service recovery.
Defensive API Deserialization: The API backend should validate response payloads (checking status codes, checking for empty strings, and filtering warning text/garbage output) before attempting to unmarshal them as JSON.
Screenshots
Add any other context about the problem here.
Reference: