Skip to content

dashboard: add a daily usage + uptime report cron - #276

Open
JadenFiotto-Kaufman wants to merge 2 commits into
0.8from
dashboard/daily-report
Open

dashboard: add a daily usage + uptime report cron#276
JadenFiotto-Kaufman wants to merge 2 commits into
0.8from
dashboard/daily-report

Conversation

@JadenFiotto-Kaufman

Copy link
Copy Markdown
Member

What

The dashboard has a monitor cron, but it is an alerting job — it fires on transitions (down / back up / models failed) and says nothing while things are fine. There was no way to answer "what happened on NDIF today?" without opening Grafana.

jobs/report.py is that digest. Once a day it posts a single Discord message:

📊 **NDIF report** — 2026-08-21 01:20 → 01:50 UTC
**Uptime** 100.0% (2/2 checks)
**Requests** 175 received · 160 completed · 15 failed (8.6%)
**Users** 4 active · alice.chen@mit.edu (60), bob.rivera@stanford.edu (30), … · 58 unattributed
**Models** 2 used · openai-community/gpt2 (136), meta-llama/Llama-3.1-8B (39) · 1 HOT now
**Latency** p50 754 ms · p95 2414 ms
**Data** 49.1 MB in · 250.6 MB out

Two sources, deliberately

  • Uptime from the connected_*.log datapoints the monitor cron already writes. The report probes nothing itself, so it cannot disagree with alerts that were already sent. An outage is a run of consecutive non-ok ticks measured first-bad to next-good, so it does not assume the monitor interval.
  • Usage from InfluxDB, the durable metrics store. Loki is not consulted: every headline number is already a metric, and log retention is usually shorter than the window a daily report wants.

Two accounting decisions

Both because the obvious version is wrong, and both caught by running against real data:

  • Failures are counted against received, not executed. A request that dies before reaching a replica never emits an execution_time point, so an executed-based rate hides it entirely — in testing that was 146 received against 139 executed, with the 7 missing requests invisible in the headline.
  • Requests with no email tag are reported as unattributed rather than dropped. That is every request when auth is off, and otherwise the per-user counts silently fail to add up to the received count (117 of 146 in the same run).

It fails open: if InfluxDB is unreachable you still get an uptime-only report, which is exactly the moment you want one.

Configuration

Knob Where Default
schedule NDIF_DASHBOARD_REPORT_CRON 0 0 * * *
window NDIF_DASHBOARD_REPORT_WINDOW_HOURS / --window-hours 24
rows per top-list report.top_n in config.json 3
message messages.daily_report in config.json see config.example.json
metrics source NDIF_INFLUX_* compose now sets these

Compose passes NDIF_INFLUX_* to the dashboard service, which previously had no telemetry access at all. --dry-run prints instead of posting.

build_report() returns a plain dict, so an endpoint could serve the same data to the UI later without a rewrite.

Testing

Verified end to end on the compose stack:

  • Real traffic through the API (25 completed / 4 errored) lands in the report exactly, and the numbers reconcile: 160 completed + 15 failed = 175 received; 117 attributed + 58 unattributed = 175.
  • Uptime validated against a synthetic day with a known 40-minute outage → reported 97.22% and longest 40m.
  • Crontab installs after a rebuild with the Influx env passed through.
  • A real webhook POST carries the right payload, with a custom template and top_n honoured.
  • Empty-window, InfluxDB-down and no-connectivity-log paths all degrade gracefully.

Note for review

The uptime numbers only ever come from the monitor cron's own log, so a deployment that runs report without monitor will correctly say no connectivity data (is the monitor cron running?) rather than claiming 100%.

🤖 Generated with Claude Code

JadenFiotto-Kaufman and others added 2 commits August 20, 2026 21:50
The dashboard already had a monitor cron, but it is an alerting job: it fires on
transitions (down / back up / models failed) and says nothing while things are
fine. There was no way to answer "what happened on NDIF today?" without opening
Grafana.

`jobs/report.py` is that digest. Once a day it posts one Discord message with
uptime, request volume and outcomes, active users, models used, execution
percentiles and bytes moved.

Two sources, deliberately:

- **Uptime** from the `connected_*.log` datapoints the monitor cron already
  writes. The report therefore probes nothing itself and cannot disagree with
  the alerts that were already sent. An outage is a run of consecutive non-ok
  ticks measured first-bad to next-good, so it does not assume the monitor
  interval.
- **Usage** from InfluxDB, the durable metrics store. Loki is not consulted:
  every headline number is already a metric, and log retention is usually
  shorter than the window a daily report wants.

Two accounting decisions worth stating, because the obvious versions are wrong:

- Failures are counted against *received*, not against executed. A request that
  dies before reaching a replica never emits an `execution_time` point, so an
  executed-based rate hides it — on the test data that was 146 received against
  139 executed, with the 7 missing requests invisible in the headline.
- Requests with no `email` tag are reported as `unattributed` rather than
  dropped. That is every request when auth is off, and the per-user counts
  otherwise fail to add up to the received count (117 of 146 in the same run).

Fails open: if InfluxDB is unreachable you still get an uptime-only report,
which is exactly the moment you want one.

Configurable via `NDIF_DASHBOARD_REPORT_CRON` (default `0 0 * * *`),
`NDIF_DASHBOARD_REPORT_WINDOW_HOURS` / `--window-hours` (24), `report.top_n` and
`messages.daily_report` in `config.json`, and the `NDIF_INFLUX_*` vars — which
compose now passes to the dashboard service, since it previously had no
telemetry access at all. `--dry-run` prints instead of posting.

`build_report()` returns a plain dict, so an endpoint can serve the same data to
the UI later without a rewrite.

Verified end to end on the compose stack: real traffic through the API
(25 completed / 4 errored) lands in the report exactly, the numbers reconcile
(160 completed + 15 failed = 175 received; 117 attributed + 58 unattributed =
175), the crontab installs with the Influx env, a real webhook POST carries the
right payload with a custom template and `top_n` honoured, and the empty-window,
Influx-down and no-connectivity-log paths all degrade gracefully.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_make_client passed aws_access_key_id/aws_secret_access_key on every call, so
boto3 never reached its own credential chain and a deployment on AWS had to
carry a long-lived IAM user's key. Pass the pair only when both are set;
otherwise hand boto3 nothing and let it find the environment, shared config,
container role or instance role. Empty strings would not have done this on
their own — botocore reads an empty string as a credential rather than an
absence, and signing fails.

The MinIO defaults are untouched, so a local stack behaves as before.

ndif-aws drops its object-store-access-key / object-store-secret-key SSM
parameters alongside this and sets both variables empty, leaving the instance
role its network stack already grants read/write on the results bucket. That
removes the per-environment IAM user those keys needed, which nothing validated
against the bucket it was meant to reach: a second environment handed the
first's key authenticates fine and fails every upload with AccessDenied.

Also track nnsight's 0.8 branch tip rather than a commit, and say so where the
file claimed the opposite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant