Skip to content

Gate notification checks on a Redis next-due cache to let Neon suspend - #468

Merged
jlunder00 merged 1 commit into
devfrom
feature/neon-cron-gating
Jul 4, 2026
Merged

Gate notification checks on a Redis next-due cache to let Neon suspend#468
jlunder00 merged 1 commit into
devfrom
feature/neon-cron-gating

Conversation

@jlunder00

Copy link
Copy Markdown
Owner

Summary

  • _run_notification_check (api/routes/internal.py) and run_polling's inline anchor/followup block previously hit Postgres unconditionally on every invocation, regardless of whether anything was actually due — defeating managed-Postgres (Neon) auto-suspend even when the app is fully idle.
  • Adds shared/notify_due.py: a Redis ZSET/HASH cache of each user's next anchor-boundary and next-followup-ping time. Both entry points now check this cache first and skip Postgres entirely when nothing is due.
  • check_followups and _check_anchor_transitions now return their own next-due estimate (computed from data they already fetched — zero extra Postgres cost), so the cache self-perpetuates with no separate background refresh job.
  • Anchor create/update/delete (api/routes/anchors.py) refresh the cache immediately so edits take effect without waiting for the safety TTL.
  • Everything fails open on any Redis error/unavailability (unconfigured REDIS_URL, connection errors, malformed anchor data) — gating can only ever save a round-trip, never cause a real notification to be silently skipped.
  • Meeting events are push-based (in-process WS-listener queue owned by tether-premium), not time-based, so drain_meeting_events is deliberately exempt from this gate in run_polling and always called — it already self-gates for free on an empty queue. Companion premium-side PR: tether-premium#74.

Design doc / context

Full design discussion in team coordination — TL;DR: gate ONCE per user (not per sub-check), since anchor-transition can create followup rows that check_followups needs to see in the same pass; a per-function gate risks skipping just-created rows via a stale cache entry.

Test plan

  • 16 unit tests for shared/notify_due.py (combined min-of-components scoring, fail-open on unconfigured Redis, fail-open on actual Redis exceptions, next_anchor_boundary pure function incl. malformed-row hardening)
  • 8 unit tests for classify_followup_row (pure followup due-classification extracted from check_followups)
  • 7 integration tests for _run_notification_check / _check_anchor_transitions gating (mocked pool/Redis — asserts Postgres is never touched when nothing due, fail-open fallback, per-user recompute-after-run)
  • 4 unit tests for api/routes/anchors._refresh_anchor_due_cache (incl. critical fail-open case: a Postgres error refreshing the cache must never turn an already-committed anchor mutation into a 500)
  • Full tests/api/, tests/bot/, tests/shared/ suite run locally: 396 passed, 11 pre-existing failures unrelated to this change (missing claude_agent_sdk dev dependency + one pre-existing async-cancellation flake in test_redis_pubsub.py, a file untouched by this PR)
  • Addressed a silent-failure-hunter review pass: fixed an unguarded get_anchors call that could turn a successful anchor mutation into a 500, hardened next_anchor_boundary against malformed anchor rows, bumped log severity on two fail-open paths, added Redis-exception (not just unconfigured) fail-open test coverage

Known follow-ups (out of scope for this PR)

  • No metric/counter yet for "gating has been failing open for N consecutive checks" — currently only an ERROR-level log line; real observability (e.g. a Prometheus counter, mirroring the pattern already used by agent_pool_manager) would make chronic Redis unavailability more visible than log-grepping.
  • The pre-existing duplication between _check_anchor_transitions (api/routes/internal.py) and run_polling's inline anchor-transition block was intentionally left alone to keep this PR minimal — a separate cleanup.

_run_notification_check and run_polling's inline anchor/followup block
were hitting Postgres unconditionally on every invocation, regardless of
whether anything was actually due, defeating managed-Postgres (Neon)
auto-suspend even when the app is fully idle.

Add shared/notify_due.py: a Redis ZSET/HASH cache of each user's next
anchor-boundary and next-followup-ping time. _run_notification_check and
run_polling now check this cache first and skip Postgres entirely when
nothing is due; check_followups and _check_anchor_transitions return
their own next-due estimate (computed from data they already fetched) so
the cache self-perpetuates with no separate background job. Anchor
create/update/delete refresh the cache immediately. Everything fails
open on any Redis error, so gating can only save a round-trip, never
skip a real notification.

Meeting events are push-based (in-process WS-listener queue owned by
tether-premium), not time-based, so drain_meeting_events is deliberately
exempt from this gate in run_polling and always called — it already
self-gates for free on an empty queue.
@jlunder00
jlunder00 merged commit 8933281 into dev Jul 4, 2026
7 checks passed
jlunder00 added a commit that referenced this pull request Jul 4, 2026
Add REDIS_URL to [program:bot] to un-brick Neon idle gating (#468)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant