prepared dashboard

CHANGELOG.md

@@ -1,5 +1,112 @@
 # Changelog
 
+## Unreleased
+
+### Dashboard SSE hardening
+
+- Make SSE the default dashboard transport. `:polling` remains an explicit compatibility fallback.
+- Replace one `PRAGMA data_version` loop **per browser connection** with one shared `EventHub` watcher per Rack app process while at least one dashboard tab is connected. It fans complete overview snapshots to every SSE body through latest-value mailboxes, so slow tabs do not accumulate unbounded event queues.
+- Use complete snapshots on connect and after a change rather than a replay log: reconnecting EventSource clients cannot miss the current queue state.
+- Coalesce client-side list refreshes after a burst of changes, cancel stale list fetches on tab switches, and retain previous pages when `Load more` is used.
+- Add SSE retry control, 25-second heartbeat frames, `no-cache, no-transform`, and `X-Accel-Buffering: no`. Remove the hop-by-hop `Connection` response header.
+- Fingerprint asset URLs and cache immutable assets by digest, so a dashboard deploy cannot leave a browser on incompatible HTML/JS/CSS.
+- Add coverage for event fan-out, latest-value coalescing, clean stream shutdown, forced overview reads, and the SSE configuration constraints.
+
+## 1.1.0
+
+Server-Sent Events transport for the dashboard. Replaces HTTP polling as the recommended transport.
+
+### Added
+
+- **SSE transport for the dashboard.** Set `c.transport = :sse` in `Async::Background::Web.configure` and the dashboard now uses a single long-lived `text/event-stream` connection per browser tab instead of polling `/api/overview` every 2 seconds. The browser opens `EventSource(mount_path + '/api/stream')` once; the server pushes an `overview` event whenever `PRAGMA data_version` changes, and a `:keepalive` comment frame every 30 seconds. Result: 1 HTTP connection per dashboard tab regardless of how long it stays open, instead of 30 req/min per tab.
+
+  - New module `Async::Background::Web::Stream` implements the event loop as a Rack streaming body (responds to `#each`, yields SSE frames). Holds no state across requests.
+  - New route `GET /api/stream` returns `200 text/event-stream` when `transport == :sse`, `404` otherwise. Subject to the same auth gate as every other endpoint.
+  - New `Response.sse(body)` helper sets the correct headers including `x-accel-buffering: no` (disables nginx buffering for the streaming response).
+  - JS client (`assets.rb`) detects `state.config.transport === 'sse'` at boot and chooses `EventSource` over `setInterval(tick, ...)`. Both transports share the same `applyOverview()` and `refreshActiveList()` handlers, so the UI behaves identically.
+
+- **`Configuration#transport`** with default `:polling` (backward compatible) and accepted values `:polling | :sse`. Validation rejects anything else with `ConfigurationError`. The chosen transport is exposed at `/api/config` so the client knows which path to take.
+
+### Migration
+
+Existing deployments keep working unchanged. To opt into SSE:
+
+```ruby
+Async::Background::Web.configure do |c|
+  c.queue_path = ...
+  c.auth = ->(env) { ... }
+  c.transport = :sse
+end
+```
+
+### Server compatibility note
+
+SSE holds the request thread/fiber open for the lifetime of the dashboard tab. **Recommended for Falcon**, which handles long-lived connections natively via fibers. **Puma works** but each open dashboard tab holds one worker thread for its lifetime — fine for an admin dashboard with a handful of operators, problematic if many concurrent operators would starve the worker pool. **Unicorn does not work** for SSE since its blocking worker model can't hold long-lived connections without timeouts; stay on `:polling` there.
+
+### Backend-side polling
+
+The server still polls `PRAGMA data_version` every 500ms inside the snapshot connection to detect changes. This is a connection-local PRAGMA call, microseconds, never hits a rate limiter. Client-facing transport is push.
+
+### Tests
+
+- New `spec/async/background/web/stream_spec.rb` — covers overview event on data_version change, heartbeat after idle, graceful exit on `EPIPE`/`IOError`, error frame on `ClosedError`/`UnavailableError`.
+- Extended `spec/async/background/web/app_spec.rb` — `/api/stream` returns 404 on polling default, 200 text/event-stream on `:sse`, 401 without auth.
+- Extended `spec/async/background/web/configuration_spec.rb` — accepts `:sse`, rejects unknown transports.
+
+## 1.0.0
+
+First stable release. The queue execution contract from 0.7.2 (claim-token CAS, lifecycle columns, barrier-based shutdown drain, per-status partial indexes, versioned migrations) is now considered the public API.
+
+### Features
+
+- **Web dashboard.** Rack-mountable read-only UI under `require 'async/background/web'`. Vanilla HTML/CSS/JS, no JS framework, no npm.
+  - Endpoints: `GET /`, `GET /assets/app.css`, `GET /assets/app.js`, `GET /api/overview`, `GET /api/executing`, `GET /api/claimed`, `GET /api/pending`, `GET /api/done`, `GET /api/failed`, `GET /api/metrics`, `GET /api/config`.
+  - Default transport is JSON polling (`poll_interval_ms`, default 2000). SSE adapter for Falcon is intentionally deferred to a later release; the dashboard already coalesces work via a shared overview cache, so adding SSE later is a backward-compatible change.
+  - Read path runs through `Async::Background::Web::Snapshot`, which opens SQLite with `file:?mode=ro`, wraps a `Mutex` around a single shared connection, and uses one read transaction per endpoint and caches each overview as one consistent snapshot.
+  - Distinguishes `Executing` (`status='running' AND started_at IS NOT NULL`) from `Claimed` (`status='running' AND started_at IS NULL`).
+  - Overview snapshot cache for `counts_cache_ttl` seconds (default 3.0) so a busy queue does not turn the dashboard into a hot reader.
+  - Cursor pagination for `done`/`failed`/`pending` using `(finished_at, id)` / `(run_at, id)` tuples. Stable on ties.
+  - Args hidden by default (`expose_args: false`); when enabled, content runs through `redact_args`. All user content rendered through `textContent`, never `innerHTML`.
+  - Auth hook is **mandatory**. `Configuration#validate!` rejects an unconfigured `auth`. There is no permissive default.
+
+  - Add the optional Rack dashboard for the SQLite queue.
+  - Make sqlite3 an explicit runtime dependency for queue/dashboard installs.
+
+### Configuration
+
+```ruby
+require 'async/background/web'
+
+Async::Background::Queue::Store.prepare_dashboard!(path: '/var/lib/app/queue.db')
+
+Async::Background::Web.configure do |c|
+  c.queue_path        = '/var/lib/app/queue.db'
+  c.auth              = ->(env) { env['warden'].user&.admin? }
+  c.expose_args       = false
+  c.metrics_path      = '/run/app/async-background.shm'
+  c.total_workers     = 4
+  c.counts_cache_ttl  = 3.0
+  c.poll_interval_ms  = 2000
+  c.list_limit        = 50
+  c.mount_path        = '/admin/background'
+  c.title             = 'My App background jobs'
+end
+
+run Async::Background::Web.app
+```
+
+### Dependencies
+
+- `rack` is an optional dependency. Required only when `require 'async/background/web'` is loaded. Core gem and worker processes do not require it.
+
+### Breaking changes from 0.7.x
+
+None beyond what 0.7.2 already shipped. The 1.0 line locks the existing contract:
+
+- `Queue::Store#fetch` returns `claim_token` in the result hash.
+- All terminal `Queue::Store` methods (`complete`, `fail`, `retry_or_fail`) require the `claim_token:` kwarg and return CAS success boolean / `:retried` / `:failed` / `nil`.
+- Schema is versioned via `PRAGMA user_version`. Use `Queue::Store.migrate!(path:)` to upgrade. Use `Queue::Store.prepare_dashboard!(path:)` from the dashboard process to lazily create dashboard-only indexes (per-status partial indexes for `done` / `failed`, plus separate `executing` and `claimed` indexes).
+
 ## 0.7.2
 
 - Harden queue execution, retries, shutdown, and metrics.
@@ -102,7 +209,7 @@
   - Proper job distribution validation across worker pool
 - **Test fixtures** — dedicated `ci/fixtures/jobs.rb` and `ci/fixtures/schedule.yml` for scenario testing
 
-### Bug Fixes  
+### Bug Fixes
 - **SQLite busy timeout** — added `PRAGMA busy_timeout = 5000` to `Queue::Store` to prevent `SQLITE_BUSY` errors under concurrent multi-process database access
 - **Enhanced Queue::Notifier error handling** — restructured IO error handling with clearer categorization:
   - `WRITE_DROPPED` for write failures (`IO::WaitWritable`, `Errno::EAGAIN`, `IOError`, `Errno::EPIPE`) — all non-fatal as job is already in store

README.md

@@ -12,15 +12,15 @@ A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https:/
 
 - Ruby >= 3.3
 - `async ~> 2.0`, `fugit ~> 1.0`
-- `sqlite3 ~> 2.0` (optional, for the job queue)
+- `sqlite3 ~> 2.0` (optional, storage)
 - `async-utilization >= 0.3, < 0.5` (optional, for metrics)
 
 ## Install
 
 ```ruby
 # Gemfile
 gem "async-background"
-gem "sqlite3", "~> 2.0"            # optional
+gem "sqlite3", "~> 2.0"  # optional
 gem "async-utilization", ">= 0.3", "< 0.5"  # optional
 ```
 
@@ -126,6 +126,12 @@ The dynamic queue runs alongside it:
 
 Jobs are persisted in SQLite, so a missed wake-up is never a lost job — workers also poll every 5 seconds as a safety net.
 
+### Queue-only workers
+
+Recurring schedules are optional. A worker that serves only dynamic jobs starts with
+`config_path: nil` and a `queue_socket_dir`; it does not need a placeholder schedule file.
+A supplied schedule path stays strict and raises when the file is missing or empty.
+
 ### Schema migration during deploy
 
 Run queue migrations once in the release/pre-deploy step, before starting new web or worker
@@ -140,18 +146,18 @@ A fresh database still self-initializes on first use for local development, but
 migration is the production path. For an existing queue, finish or stop 0.7.1 producers/workers,
 run the migration once, then start 0.7.2 processes.
 
-### Future dashboard indexes
+### Dashboard indexes
 
 The queue does **not** install dashboard indexes by default. They slow every enqueue even though
-pending rows never enter terminal or in-flight read-model indexes. When the 1.0 dashboard module
-is enabled, its installer will run this once in the same release step:
+pending rows never enter terminal or in-flight read-model indexes. Enable them once before
+mounting the dashboard:
 
 ```ruby
 Async::Background::Queue.prepare_dashboard!(path: ENV.fetch("QUEUE_DB_PATH"))
 ```
 
-It adds three compact indexes: one each for cursor-sorted done and failed jobs, plus one for
-the bounded in-flight list. It does not change queue behavior or rerun the core migration.
+It adds four compact indexes: cursor-sorted `done` / `failed` history plus separate
+`executing` / `claimed` in-flight lists. It does not change queue behavior or rerun the core migration.
 
 ## Metrics
 

async-background.gemspec

@@ -33,12 +33,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'async',   '~> 2.0'
   spec.add_dependency 'console', '~> 1.0'
   spec.add_dependency 'fugit',   '~> 1.0'
+  spec.add_dependency 'base64',  '~> 0.2'
 
   # Optional: add to your own Gemfile if you need these features
-  #   gem 'sqlite3',          '~> 2.0'  # dynamic job queue
+  #   gem 'sqlite3', '~> 2.0'
   #   gem 'async-utilization', '>= 0.3', '< 0.5' # shared-memory worker metrics
 
   spec.add_development_dependency 'rake',  '~> 13.0'
   spec.add_development_dependency 'rspec', '~> 3.12'
+  spec.add_development_dependency 'rack',  '~> 3.0'
   spec.add_development_dependency 'async-utilization', '>= 0.3', '< 0.5'
 end

docs/GET_STARTED.md

@@ -172,8 +172,8 @@ Async::Background::Queue.migrate!(path: ENV.fetch("QUEUE_DB_PATH"))
 ```
 
 For an existing queue, stop or drain 0.7.1 processes first, run the migration, then start 0.7.2.
-The base queue keeps only its pending-job index. A future dashboard installer can later call
-`Async::Background::Queue.prepare_dashboard!(path: DB_PATH)` once to add its three read-model
+The base queue keeps only its pending-job index. Before mounting the dashboard, call
+`Async::Background::Queue.prepare_dashboard!(path: DB_PATH)` once to add its four read-model
 indexes without slowing normal enqueue-only deployments.
 
 That's the full config — both web and scheduler share the same SQLite file and notify each other through Unix domain sockets. Web controllers can now enqueue:
@@ -190,6 +190,25 @@ end
 
 > **How wake-up works.** When any process (web or scheduler) enqueues a job, `SocketNotifier` sends one byte to a Unix domain socket. The chosen background worker wakes in ~30–80 µs and reads from SQLite — no polling delay.
 
+### Queue-only worker
+
+A recurring schedule is optional. For applications that use only `perform_async`,
+`perform_in`, and `perform_at`, pass `config_path: nil`. The worker keeps listening
+to the queue until it receives `SIGTERM` / `SIGINT`; no placeholder YAML file is needed.
+
+```ruby
+Async::Background::Runner.new(
+  config_path: nil,
+  worker_index: 1,
+  total_workers: 1,
+  queue_db_path: Rails.root.join("storage/async-background.sqlite3").to_s,
+  queue_socket_dir: "/tmp"
+).run
+```
+
+A non-`nil` `config_path` remains strict: a missing or empty schedule file raises
+`Async::Background::ConfigError` rather than silently disabling recurring jobs.
+
 ### Environment variables
 
 | Variable | Default | Description |
@@ -217,6 +236,128 @@ is `false` and `Async::Background::Metrics.read_all(...)` returns `[]`. When web
 background run in separate containers, point `ASYNC_BACKGROUND_METRICS_PATH` at a file under
 a shared volume (for example `/app/tmp/queue/async-background.shm`).
 
+---
+
+## Step 2.5 — Mount the optional dashboard
+
+The dashboard is a separate, read-only Rack app over the same SQLite file. It never enqueues,
+retries, deletes, or otherwise mutates jobs. Before mounting it, add its read-model indexes once
+in the same release step as the queue migration:
+
+```ruby
+# bin/migrate_async_background
+require "async/background/queue/client"
+
+queue_path = ENV.fetch("QUEUE_DB_PATH")
+Async::Background::Queue.migrate!(path: queue_path)
+Async::Background::Queue.prepare_dashboard!(path: queue_path)
+```
+
+`prepare_dashboard!` is idempotent. It installs four dashboard-only indexes for done, failed,
+claimed, and executing lists; the core pending index already exists. Normal queue-only deployments
+do not pay their write cost.
+
+### Rack / Falcon
+
+Put this in the Rack app that serves the dashboard (for example, a dedicated `config.ru`):
+
+```ruby
+# frozen_string_literal: true
+
+require "async/background/web"
+
+Async::Background::Web.configure do |config|
+  config.queue_path = ENV.fetch("QUEUE_DB_PATH", "/var/lib/app/queue.db")
+  config.auth = ->(env) { env["warden"]&.user&.admin? }
+
+  # Optional: requires async-utilization and a shared path visible to workers.
+  config.metrics_path = ENV["ASYNC_BACKGROUND_METRICS_PATH"]
+  config.total_workers = ENV.fetch("BACKGROUND_FORKS", 1).to_i
+
+  # Must match the Rack/Rails mount point below.
+  config.mount_path = "/admin/background"
+
+  # Default transport. One SSE connection per open tab; no browser polling.
+  config.transport = :sse
+  config.stream_poll_seconds = 0.5       # one SQLite change check per Rack process
+  config.stream_heartbeat_seconds = 25.0 # keeps proxies from idling out the stream
+  config.stream_retry_ms = 5_000
+end
+
+run Async::Background::Web.app
+```
+
+Add `rack` to the application bundle when it is not already present:
+
+```ruby
+gem "rack", "~> 3.0"
+```
+
+When metrics are not needed, omit both `metrics_path` and `total_workers`.
+
+### Rails
+
+Configure the dashboard once during boot:
+
+```ruby
+# config/initializers/async_background_dashboard.rb
+require "async/background/web"
+
+Async::Background::Web.configure do |config|
+  config.queue_path = ENV.fetch("QUEUE_DB_PATH", Rails.root.join("tmp/queue/background.db").to_s)
+  config.auth = ->(env) { env["warden"]&.user&.admin? }
+  config.metrics_path = ENV["ASYNC_BACKGROUND_METRICS_PATH"]
+  config.total_workers = ENV.fetch("BACKGROUND_FORKS", 1).to_i
+  config.mount_path = "/admin/background"
+  config.transport = :sse
+end
+```
+
+Then mount the Rack app:
+
+```ruby
+# config/routes.rb
+mount Async::Background::Web.app => "/admin/background"
+```
+
+Use an application-specific authorization predicate. The gem intentionally has no permissive
+default: a missing or falsey `auth` result returns `401`. Do not expose the dashboard publicly
+without an authentication layer in front of it.
+
+### Live updates and rate limits
+
+SSE is the default transport. A dashboard tab opens one authenticated `GET /api/stream` request;
+the server sends a complete overview snapshot after connect and after the queue changes. The browser
+performs ordinary JSON requests only for the initial page and when it needs to redraw the *active*
+list. It does **not** poll on a timer.
+
+Internally, each Rack process with at least one connected dashboard uses one long-lived SQLite read
+connection and compares `PRAGMA data_version` every `stream_poll_seconds` (default `0.5`). This is a
+single local database read per process, not per browser tab. SQLite documents `data_version` for
+exactly this interactive-cache invalidation use case. The stream ships a heartbeat every 25 seconds,
+uses a server-supplied 5-second reconnect delay, and each reconnect begins from a full current
+snapshot; no event log or Redis is required.
+
+When the host application applies a generic Rack::Attack throttle to all `/admin` requests, exempt
+**authenticated** dashboard reads or put the dashboard behind a separate admin throttle. Do not let
+a long-lived stream and its initial list request count as abuse:
+
+```ruby
+# config/initializers/rack_attack.rb
+Rack::Attack.safelist("authenticated async-background dashboard") do |request|
+  request.path.start_with?("/admin/background") &&
+    request.env["warden"]&.user(:admin_user).present?
+end
+```
+
+The dashboard's own `config.auth` still runs for every request; this only prevents a generic rate
+limit from treating an authenticated operator's live dashboard as a burst. Adapt the Warden scope to
+your application. If the reverse proxy buffers streaming responses, disable buffering for
+`/admin/background/api/stream`; the response already includes `X-Accel-Buffering: no` for nginx.
+
+Use `config.transport = :polling` only for a server that cannot keep an SSE response open. It is a
+compatibility fallback, not the recommended production mode.
+
  
 
 ## Step 3 — Docker setup

lib/async/background/metrics.rb

@@ -191,10 +191,12 @@ def validate_worker!(worker_index, total_workers)
 
       def ensure_shm!(total_workers, path)
         required_size = self.class.segment_size * total_workers
+        page_size = IO::Buffer::PAGE_SIZE
+        mapped_size = ((required_size + page_size - 1) / page_size) * page_size
 
         File.open(path, File::CREAT | File::RDWR, 0o644) do |file|
           file.flock(File::LOCK_EX)
-          file.truncate(required_size) if file.size < required_size
+          file.truncate(mapped_size) if file.size < mapped_size
         ensure
           file.flock(File::LOCK_UN) rescue nil
         end

lib/async/background/queue/schema.rb

@@ -13,7 +13,12 @@ module Schema
         VERSION = 1
         MIGRATION_BUSY_TIMEOUT_MS = 30_000
         CORE_INDEXES = %w[idx_jobs_pending].freeze
-        DASHBOARD_INDEXES = %w[idx_jobs_done_finished_at idx_jobs_failed_finished_at idx_jobs_running].freeze
+        DASHBOARD_INDEXES = %w[
+          idx_jobs_done_finished_at
+          idx_jobs_failed_finished_at
+          idx_jobs_executing_started_at
+          idx_jobs_claimed_locked_at
+        ].freeze
         REQUIRED_INDEXES = CORE_INDEXES
 
         module_function