dashboard: add a daily usage + uptime report cron - #276
Open
JadenFiotto-Kaufman wants to merge 2 commits into
Open
dashboard: add a daily usage + uptime report cron#276JadenFiotto-Kaufman wants to merge 2 commits into
JadenFiotto-Kaufman wants to merge 2 commits into
Conversation
The dashboard already had a monitor cron, but it is an alerting job: it fires on transitions (down / back up / models failed) and says nothing while things are fine. There was no way to answer "what happened on NDIF today?" without opening Grafana. `jobs/report.py` is that digest. Once a day it posts one Discord message with uptime, request volume and outcomes, active users, models used, execution percentiles and bytes moved. Two sources, deliberately: - **Uptime** from the `connected_*.log` datapoints the monitor cron already writes. The report therefore probes nothing itself and cannot disagree with the alerts that were already sent. An outage is a run of consecutive non-ok ticks measured first-bad to next-good, so it does not assume the monitor interval. - **Usage** from InfluxDB, the durable metrics store. Loki is not consulted: every headline number is already a metric, and log retention is usually shorter than the window a daily report wants. Two accounting decisions worth stating, because the obvious versions are wrong: - Failures are counted against *received*, not against executed. A request that dies before reaching a replica never emits an `execution_time` point, so an executed-based rate hides it — on the test data that was 146 received against 139 executed, with the 7 missing requests invisible in the headline. - Requests with no `email` tag are reported as `unattributed` rather than dropped. That is every request when auth is off, and the per-user counts otherwise fail to add up to the received count (117 of 146 in the same run). Fails open: if InfluxDB is unreachable you still get an uptime-only report, which is exactly the moment you want one. Configurable via `NDIF_DASHBOARD_REPORT_CRON` (default `0 0 * * *`), `NDIF_DASHBOARD_REPORT_WINDOW_HOURS` / `--window-hours` (24), `report.top_n` and `messages.daily_report` in `config.json`, and the `NDIF_INFLUX_*` vars — which compose now passes to the dashboard service, since it previously had no telemetry access at all. `--dry-run` prints instead of posting. `build_report()` returns a plain dict, so an endpoint can serve the same data to the UI later without a rewrite. Verified end to end on the compose stack: real traffic through the API (25 completed / 4 errored) lands in the report exactly, the numbers reconcile (160 completed + 15 failed = 175 received; 117 attributed + 58 unattributed = 175), the crontab installs with the Influx env, a real webhook POST carries the right payload with a custom template and `top_n` honoured, and the empty-window, Influx-down and no-connectivity-log paths all degrade gracefully. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_make_client passed aws_access_key_id/aws_secret_access_key on every call, so boto3 never reached its own credential chain and a deployment on AWS had to carry a long-lived IAM user's key. Pass the pair only when both are set; otherwise hand boto3 nothing and let it find the environment, shared config, container role or instance role. Empty strings would not have done this on their own — botocore reads an empty string as a credential rather than an absence, and signing fails. The MinIO defaults are untouched, so a local stack behaves as before. ndif-aws drops its object-store-access-key / object-store-secret-key SSM parameters alongside this and sets both variables empty, leaving the instance role its network stack already grants read/write on the results bucket. That removes the per-environment IAM user those keys needed, which nothing validated against the bucket it was meant to reach: a second environment handed the first's key authenticates fine and fails every upload with AccessDenied. Also track nnsight's 0.8 branch tip rather than a commit, and say so where the file claimed the opposite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The dashboard has a monitor cron, but it is an alerting job — it fires on transitions (down / back up / models failed) and says nothing while things are fine. There was no way to answer "what happened on NDIF today?" without opening Grafana.
jobs/report.pyis that digest. Once a day it posts a single Discord message:Two sources, deliberately
connected_*.logdatapoints the monitor cron already writes. The report probes nothing itself, so it cannot disagree with alerts that were already sent. An outage is a run of consecutive non-ok ticks measured first-bad to next-good, so it does not assume the monitor interval.Two accounting decisions
Both because the obvious version is wrong, and both caught by running against real data:
received, notexecuted. A request that dies before reaching a replica never emits anexecution_timepoint, so an executed-based rate hides it entirely — in testing that was 146 received against 139 executed, with the 7 missing requests invisible in the headline.emailtag are reported asunattributedrather than dropped. That is every request when auth is off, and otherwise the per-user counts silently fail to add up to the received count (117 of 146 in the same run).It fails open: if InfluxDB is unreachable you still get an uptime-only report, which is exactly the moment you want one.
Configuration
NDIF_DASHBOARD_REPORT_CRON0 0 * * *NDIF_DASHBOARD_REPORT_WINDOW_HOURS/--window-hoursreport.top_ninconfig.jsonmessages.daily_reportinconfig.jsonconfig.example.jsonNDIF_INFLUX_*Compose passes
NDIF_INFLUX_*to thedashboardservice, which previously had no telemetry access at all.--dry-runprints instead of posting.build_report()returns a plain dict, so an endpoint could serve the same data to the UI later without a rewrite.Testing
Verified end to end on the compose stack:
longest 40m.top_nhonoured.Note for review
The uptime numbers only ever come from the monitor cron's own log, so a deployment that runs
reportwithoutmonitorwill correctly sayno connectivity data (is the monitor cron running?)rather than claiming 100%.🤖 Generated with Claude Code