frontend: alert user if telemetry latency increases past a threshold

[ES-Alexander]

Current behaviour

We've had some reports of growing latency during operation (from @vshie), but it's difficult to verify when and the extent to which this is happening.

Expected or desired behaviour

Regardless of solutions to the current problem, it seems worthwhile to have a latency metric to identify similar issues in future (and potentially help debug / confirm resolution of the current one).

It would be helpful if Cockpit stored a time-stamped SYSTEM_TIME MAVLink message from soon after it connects to an autopilot, then occasionally checks the delta from the latest SYSTEM_TIME message and its received time to measure latency since startup. Exposing that through the data lake could allow users to track it in a plotter widget, and/or generate an alert if it gets too high.

Something along the lines of (pseudocode):

startup_time_received, message = mavlink.receive('SYSTEM_TIME')
startup_time = message.time_boot_ms
...
def regular_check():
    latest_time_received, message = mavlink.receive('SYSTEM_TIME')
    latest_time_reported = message.time_boot_ms
    time_elapsed = current_time_received - startup_time_received
    reported_time_elapsed = latest_time_reported - startup_time
    latency = time_elapsed - reported_time_elapsed
    # push latency to data lake

Prerequisites

• I have checked to make sure that a similar request has not already been filed or fixed.

[ES-Alexander]

#2211 added a data-lake variable to track the latency :-)

Now we just need to add an alert for it.