Kestrel — Technical Design & Stack Selection

Executive Summary

Kestrel is a ground-up, Rust-native rewrite of an open-source BI/analytics platform in the incumbent category. It is a single statically-linked binary with role flags (api / worker / all-in-one), a gRPC-first API surface consumable from browsers, a from-scratch Rust reimplementation of the dbt Semantic Layer (MetricFlow) as the sole source of semantic truth, and a DuckDB-over-Parquet serving tier running as stateless, scale-to-zero pods on Kubernetes. Postgres is the only stateful store (metadata AND job queue); object storage holds Parquet rollups, cached results, and rendered chart images; Arrow moves data end-to-end.

This document takes firm positions. The headline recommendations:

RPC layer: adopt tonic (gRPC + gRPC-Web via tonic-web) as the primary transport, multiplexed with axum on one port via tower/hyper. Do NOT bet the browser path on a Rust Connect-RPC implementation — the Connect-in-Rust crates are immature. Instead serve gRPC-Web to the Next.js app with @connectrpc/connect-web on the client, which speaks gRPC-Web natively.
Serving engine: DuckDB via duckdb-rs, with Apache DataFusion as a strategic hedge. DataFusion is now the fastest engine in the ClickBench raw-Parquet category and is pure-Rust/Arrow-native, but DuckDB's richer SQL surface and operational maturity win for v1. Design the serving tier behind a trait so DataFusion can be swapped in per-workload later.
MetricFlow-in-Rust is the single largest build risk and the core IP. Consume dbt's semantic_manifest.json, build a semantic graph, plan through a dataflow DAG, and emit dialect-specific SQL. Reuse datafusion-sqlparser-rs AST types for emission; do NOT try to force MetricFlow semantics into DataFusion's logical plan.
Job queue: hand-rolled Postgres SKIP LOCKED queue, or apalis with its Postgres backend. Avoid heavy external brokers — the single-Postgres constraint is a feature.
Chart rendering without Chromium: ECharts SSR in a minimal Node/Bun sidecar producing SVG strings, rasterized to PNG with the Rust resvg crate. This guarantees browser/email parity because the browser also uses ECharts.
Warehouse connectivity: per-warehouse strategy. ADBC Snowflake driver (Go-based, via the Rust driver manager) is production-grade; gcp-bigquery-client with the Storage Read API (Arrow) for BigQuery; tokio-postgres for Postgres/Redshift; ADBC/Flight SQL or REST for Databricks.
Authz: embedded Cedar (cedar-policy). Do NOT introduce OpenFGA/SpiceDB — an external Zanzibar service breaks the single-binary goal.
MCP: official rmcp SDK with streamable HTTP. Mature enough to ship.

System Overview (ASCII)

                          ┌──────────────────────────────────────────────┐
                          │            Next.js / TypeScript SPA            │
                          │  @connectrpc/connect-web (gRPC-Web) + Query    │
                          │  ECharts (interactive) · Arrow-JS · TanStack   │
                          └───────────────┬────────────────────────────────┘
                                          │ gRPC-Web / gRPC / JSON  (HTTP/2 + HTTP/1.1, one port)
                                          ▼
        ┌───────────────────────────────────────────────────────────────────────┐
        │            KESTREL single binary  (role = api | worker | all-in-one)    │
        │                                                                         │
        │  tonic + tonic-web + axum (tower middleware, one hyper listener)        │
        │  ┌──────────────┐  ┌───────────────────┐  ┌──────────────────────────┐  │
        │  │ AuthN: OIDC  │  │ MetricFlow planner│  │ MCP server (rmcp,        │  │
        │  │ AuthZ: Cedar │  │ manifest→DAG→SQL  │  │ streamable HTTP)         │  │
        │  └──────────────┘  └─────────┬─────────┘  └──────────────────────────┘  │
        │                              │                                          │
        │   ┌──────────────────────────┼───────────────────────────────────┐     │
        │   │ Serving trait: DuckDB (duckdb-rs)  ⇄  [DataFusion hedge]      │     │
        │   └──────────────────────────┬───────────────────────────────────┘     │
        └───────────────┬──────────────┼───────────────────────┬─────────────────┘
                        │              │                       │
              Arrow IPC │       Arrow  │ object_store          │ Go ADBC / gRPC / pg
                        ▼              ▼                       ▼
     ┌────────────────────────┐  ┌───────────────────┐  ┌───────────────────────────┐
     │ Postgres               │  │ Object storage    │  │ Customer warehouse         │
     │  metadata + job queue  │  │ S3/GCS: Parquet    │  │  Snowflake / BigQuery /    │
     │  (SKIP LOCKED)         │  │  rollups, result   │  │  Postgres·Redshift /       │
     │                        │  │  cache, chart PNGs │  │  Databricks (materialize)  │
     └────────────────────────┘  └───────────────────┘  └───────────────────────────┘

  DuckDB serving pods (K8s Deployment, KEDA scale-to-zero, spot):
     stateless · httpfs→S3 · NVMe/emptyDir byte-range cache · Arrow out

Layer 1 — Rust Web / RPC Layer

Decision: Serve gRPC + gRPC-Web + JSON on a single port using tonic 0.14.x (gRPC over HTTP/2), tonic-web 0.14.x (gRPC-Web for browsers, no proxy), and axum 0.8 for plain-HTTP/JSON endpoints (health, OIDC callbacks, chart image GETs), all composed over one hyper listener with tower middleware. The Next.js client uses @connectrpc/connect-web, which speaks gRPC-Web to tonic-web directly.

Chosen libraries + maintenance evidence:

tonic 0.14.6 (released 2026-05-07; regular cadence: 0.14.0 2025-07-28 → 0.14.6 2026-05-07). Now maintained under the grpc/grpc-rust org (moved from hyperium). router feature is axum-based, so axum composition is first-class.
tonic-web 0.14.6 enables tonic servers to accept gRPC-Web from browsers with accept_http1(true) + GrpcWebLayer, no external proxy (Envoy) needed.
tower / tower-http 0.6 for middleware (auth, tracing, compression, timeouts, CORS).
Content-type based multiplexing: gRPC (application/grpc), gRPC-Web (application/grpc-web), and REST/JSON are routed by a Steer/content-type split or by merging tonic services into an axum Router. accept_http1(true) lets HTTP/1.1 (gRPC-Web/JSON) and HTTP/2 (gRPC) coexist on one port.

Protobuf tooling: prost for message codegen, tonic-build for services, buf CLI for linting, breaking-change detection, and managing the schema; optionally the Buf Schema Registry to publish generated TS. On the client, generate with @bufbuild/protoc-gen-es + @connectrpc/connect-web.

Runner-ups considered and why rejected:

Connect-RPC in Rust (axum-connect, connectrpc-axum, connectrpc). The Connect protocol is attractive (unary JSON that's trivially curl-able, one wire spec for browser + server). But the Rust server implementations are not production-ready: axum-connect (Alec Thilenius) is pinned at 0.5.3 (last release ~9 months before writing) and self-describes as focused/experimental; the newer connectrpc-axum explicitly states "Under active development. Not recommended for production use yet." A community connectrpc tower-service crate exists but is young. Rejected as the primary transport — we would be depending on a single-maintainer, pre-1.0 crate for our entire API surface. Revisit at P2 if a Connect-in-Rust crate reaches 1.0.
Pure Connect protocol via axum (hand-rolled). Viable but reinvents streaming, error mapping, and content negotiation that tonic+tonic-web already give us. Rejected for v1.
actix-web + tonic. Works but adds a second HTTP stack; axum shares tower/hyper with tonic, so it is the lower-friction choice.

Why gRPC-Web to the browser (not Connect): @connectrpc/connect-web can speak either the Connect protocol or gRPC-Web against the same generated client, selectable by transport. By choosing createGrpcWebTransport, the browser gets Connect-ES's excellent DX and TanStack Query integration while the server runs mature tonic-web. We get Connect's client ergonomics without Connect's immature Rust server.

Risks: gRPC-Web does not support client-streaming or bidi-streaming from the browser (server-streaming is fine). For Kestrel's request/response + server-push (query progress) patterns this is acceptable. If full-duplex is ever needed in-browser, fall back to WebTransport/WebSocket side-channels.

Layer 2 — SQL Generation / Query Planning (the MetricFlow-in-Rust core)

Decision: Build a bespoke MetricFlow-equivalent planner in Rust that (1) deserializes dbt's semantic_manifest.json, (2) constructs a semantic graph, (3) plans a query through an internal dataflow DAG, and (4) emits dialect-specific SQL using the AST types from datafusion-sqlparser-rs as the emission substrate. Do not build the semantic planner on DataFusion's LogicalPlan, and do not route metric queries through Substrait for v1.

Why not DataFusion LogicalPlan as the planner substrate: DataFusion's logical plan is an execution plan for DataFusion's own engine over Arrow — it is not a portable, dialect-targeting SQL generator. MetricFlow's job is to emit SQL text that runs on Snowflake/BigQuery/Databricks/Postgres/DuckDB, each with dialect quirks (date spines, QUALIFY, window semantics, APPROX_*). MetricFlow's own architecture separates the dataflow plan (semantic) from the SQL plan (rendered per engine). We mirror that split. DataFusion's planner is the right tool for the DuckDB/DataFusion serving tier, not for warehouse-targeted metric SQL.

Why sqlparser-rs AST for emission: datafusion-sqlparser-rs (Apache) is the most mature Rust SQL AST, used by DataFusion, Polars, ParadeDB, GreptimeDB, and many others. It has a Dialect abstraction and supports round-tripping AST→SQL text. We construct Statement/Query/Expr nodes and render per dialect. sea-query (dialect-aware query builder) is a runner-up but is oriented toward ORM-style CRUD, not the deeply nested analytical SQL (CTE ladders, window frames, time spines) MetricFlow produces — rejected. A hand-rolled string emitter is rejected as unmaintainable for 6 dialects.

Substrait (substrait-rs): Substrait is a cross-engine plan serialization format. It is compelling as an internal IR later — e.g., to hand a plan to DataFusion vs. compile to warehouse SQL from one representation. But MetricFlow semantics (fan-out/chasm-join protection, cumulative windows, conversion metrics) are richer than a naive relational plan, and Substrait's producer/consumer coverage per warehouse dialect is incomplete. P2 investigation, not v1.

What the semantic manifest contains (must be reimplemented):

semantic_models[]: each has name, node_relation ({alias, schema_name, database, relation_name} — the fully-qualified physical table), entities[], measures[], dimensions[], and defaults (e.g. agg time dimension).
entities[]: real-world keys (primary/foreign/unique/natural) that form the join edges of the semantic graph; their type drives MetricFlow join logic.
dimensions[]: categorical or time dimensions (with granularity); the new spec defines granularity at the column level.
measures[]: aggregations (sum, avg, count, count_distinct, min, max, sum_boolean, median, percentile) over a column expr, with an agg_time_dimension.
metrics[]: name, type ∈ {simple, ratio, derived, cumulative, conversion}, and type_params (measure, numerator/denominator, expr for derived, window/grain_to_date for cumulative, entity/window for conversion).
project_configuration (time spine table config), plus saved queries.
semantic_manifest.json is a trimmed sibling of manifest.json in target/; it exists because dbt-core and MetricFlow use different (de)serialization libraries and MetricFlow needs only semantic details. Note: MetricFlow was open-sourced under Apache-2.0 as part of the Open Semantic Interchange (OSI) initiative, and dbt now also parses Apache "Ossie" JSON semantic documents into the same artifacts — Kestrel should target the artifact schema, not the YAML authoring format, for stability.

Planning stages (mirror MetricFlow):

Manifest parse + lookups — build SemanticManifestLookup-equivalent indexes (metric→measures, measure→semantic model, entity→models).
Query parsing/spec — resolve requested metrics + group-by items (dimensions, entities, time grains) + filters into a typed query spec; validate group-by items are reachable.
Source-node building — for each measure, identify the source semantic model and required columns.
Dataflow plan building — build a DAG of nodes: read → filter → aggregate-measures → join-to-dimensions (with fan-out/chasm-join protection) → compute-metrics (ratio/derived/cumulative/conversion) → order/limit. Cumulative metrics require time-spine joins; conversion metrics require windowed self-joins on an entity.
Dataflow-plan optimization — predicate pushdown, column pruning, redundant-join elimination, CTE reuse.
Dataflow→SQL conversion — lower DAG nodes to a SQL plan of nested CTEs.
SQL rendering — per-dialect emission via sqlparser-rs AST + a Dialect trait.

Dialect emission strategy: a Dialect trait with methods for identifier quoting, date-truncation/date-spine syntax, QUALIFY availability, percentile/approx functions, string agg, and window-frame syntax. Implement for DuckDB (serving tier), Snowflake, BigQuery, Postgres/Redshift, Databricks.

Conformance testing approach: MetricFlow (Apache-2.0) ships an extensive snapshot test suite of (query spec → rendered SQL) per engine. Kestrel should build a differential/golden test harness: for a corpus of semantic manifests + query specs, assert Kestrel's rendered SQL is semantically equivalent to MetricFlow's — ideally by executing both against DuckDB and comparing result sets (Arrow record-batch equality), not just string diffing. This is the single most important correctness investment.

Existing Rust semantic-layer projects: none mature. There is no established Rust MetricFlow reimplementation as of 2026 — this is genuinely greenfield and is Kestrel's core IP and core risk.

Risks: MetricFlow semantics are a moving target (OSI-driven spec evolution). Cumulative + conversion metrics and multi-hop joins are where correctness bugs hide. Mitigate with the differential harness and by pinning a manifest schema version.

Layer 3 — DuckDB Serving/Acceleration Tier

Decision: Use DuckDB via duckdb-rs as the v1 serving engine, reading Parquet rollups from S3/GCS via the httpfs extension, returning Arrow to the Rust process, running as stateless pods. Keep Apache DataFusion as an architecturally-supported hedge behind a serving trait.

Chosen libraries + maintenance evidence:

duckdb crate (duckdb-rs) — actively maintained by DuckDB Labs + contributors; new versioning scheme encodes the bundled DuckDB version in the second semver component (e.g. crate 1.10502.0 bundles DuckDB v1.5.x). Recent releases upgraded to Arrow 58, bundled DuckDB v1.5.1, moved to Rust edition 2024, added full rust_decimal support. Feature flags matter: bundled (compile DuckDB from source — simplest for reproducible K8s images), vtab-arrow/appender-arrow (zero-copy Arrow ⇄ DuckDB), modern-full (chrono, serde_json, r2d2 pooling, uuid, polars), parquet, json, httpfs via runtime install.
Arrow interop: the vtab-arrow feature converts between Arrow RecordBatch and DuckDB data chunks; DuckDB can also stream Arrow IPC in/out (the arrow extension added read_arrow/Arrow IPC support in 2025). This keeps the hot path Arrow end-to-end.
Connection pooling: DuckDB is in-process; use an r2d2 pool (modern-full) of connections per pod, each with its own memory budget. DuckDB connections are cheap; the constraint is total pod memory.
httpfs config for K8s: create an S3 SECRET with credential_chain (IRSA/workload identity), set memory_limit per pod (e.g. 75% of container limit), set temp_directory to node-local NVMe for spill-to-disk, and enable enable_http_metadata_cache.

The critical httpfs caveat (must design around): DuckDB's httpfs can generate a huge number of small HTTP range requests against S3. In DuckDB's own tracker (duckdb/duckdb-httpfs Issue #172 / Discussion #12458, reproduced on DuckDB 1.4.1), reading a single 17.4M-row, 282-column, 533-row-group part-0.parquet file triggered roughly 150,000 S3 requests taking over 19 minutes — approximately (number of variables × number of row groups) — versus an Arrow-based reader that "generates around 60 requests to the S3 backend and finishes in about 1m20s." Separately, a v1.5.0 regression saw a QUALIFY ROW_NUMBER() query on hive-partitioned S3 Parquet jump from ~80 GET requests to over 4,200 with wall-clock time nearly tripling. Root cause: a separate range request per column chunk per row group, amplified by prefetch setting DIRECT_IO and bypassing buffering. Design implications: (1) write rollups with few, large row groups and sorted/clustered layout; (2) enable HTTP metadata caching; (3) put a node-local byte-range cache on NVMe in front of S3 (emptyDir on an NVMe-backed instance, or the community cache_httpfs extension); (4) prune columns aggressively in emitted SQL. Historically, enabling enable_http_metadata_cache took one workload "from unusable to usable" (~10× faster).

DataFusion vs DuckDB — honest comparison (2025-2026):

ClickBench raw-Parquet: On 2024-11-18, Andrew Lamb (Staff Engineer, InfluxData) announced on the Apache DataFusion / InfluxData blog that DataFusion 43.0.0 is the fastest engine for querying Apache Parquet files in ClickBench — verbatim: "I am extremely excited to announce that Apache DataFusion 43.0.0 is the fastest engine for querying Apache Parquet files in ClickBench. It is faster than both DuckDB and chDB/Clickhouse using the same hardware… the first time a Rust based engine holds the top spot." The benchmark ran on a c6a.4xlarge (16 CPU/32GB) over a partitioned 14GB Parquet dataset (100 files ~140MB). This is the DataFusion project's own post and refers specifically to the unmodified-Parquet category; both engines remain in the same performance tier. The same post notes "ClickBench performance improved over 30% between DataFusion 34 (released Dec 2023) and DataFusion 43 (released Nov 2024)."
Arrow-native zero-copy: DataFusion operates on Arrow throughout, so there is zero serialization cost when data is already Arrow (from Flight, caches, streaming). For Kestrel, where the hot path is Arrow, this is a real structural advantage.
DuckDB strengths: richer SQL surface (window functions, QUALIFY, PIVOT, list/struct ops), a more mature/tuned Parquet reader with zone maps, dictionary pushdown, late materialization; better out-of-the-box single-node file-scan performance for complex analytical SQL.
Real-world migration signal: Bauplan Labs publicly moved from DuckDB to DataFusion, citing DataFusion's extensibility ("DuckDB is more of an open-source product than an open-source project") — while acknowledging DuckDB's S3 performance and extension support improved substantially since 2023.
Recommendation: DuckDB for v1 (SQL completeness, mature Parquet reader, DuckLake option, matches the team's existing DuckDB-on-K8s operational expertise). Abstract the serving engine behind a trait (ServingEngine::execute(sql/plan) -> Arrow stream) so DataFusion can be introduced for specific Arrow-native or federation workloads, or swapped wholesale if DuckDB's object-store behavior becomes a bottleneck. This is a change from a naive "DuckDB-only" plan: the research shows DataFusion is now competitive-to-faster on pure Parquet scans and is a genuine option, so the architecture should not hard-code DuckDB.

DuckLake option: DuckLake reached v1.0 (production-ready), announced 13 April 2026 on ducklake.select — verbatim: "We are happy to release DuckLake v1.0, a production-ready lakehouse format specification built on SQL. Its reference implementation, the ducklake DuckDB extension, is available as of today in DuckDB v1.5.2." (Per InfoQ's May 2026 coverage the 1.0 release "merges 108 PRs since late 2025 — 68 focused on reliability and correctness alone.") It stores lakehouse metadata in a SQL catalog (Postgres/SQLite/DuckDB) with Parquet data files. Since Kestrel already runs Postgres, DuckLake-on-Postgres is a natural fit for the rollup/materialization tier: ACID snapshots, time travel, schema evolution, and statistics-driven file pruning without a separate catalog service. Recommend evaluating DuckLake for the pre-aggregate store at P1 (it directly reduces the small-file/small-request problem via its metadata-driven pruning and compaction).

Risks: DuckDB is C++ (bundled build increases image size and compile time; bundled mitigates dependency drift). Memory pressure on large aggregations — enforce memory_limit + NVMe spill. httpfs request amplification — mitigations above are mandatory, not optional.

Layer 4 — Warehouse Connectivity (materialization only)

The warehouse is used only for scheduled pre-aggregate materialization (worker role), not interactive serving. Per-warehouse strategy:

Snowflake → ADBC (Go driver via Rust driver manager). The ADBC Snowflake driver is Status: Stable (written on the Snowflake Go connector, returns Arrow natively). It is usable from Rust via the driver manager; the adbc_snowflake crate wraps the Go driver (link at build or runtime). This is the recommended path: Arrow-native bulk export, no row-by-row transposition. Runner-up: snowflake-api/snowflake-connector-rs pure-Rust crates exist but are less complete; prefer ADBC.
BigQuery → gcp-bigquery-client + BigQuery Storage Read API (Arrow). The gcp-bigquery-client crate is an ergonomic async client supporting all endpoints plus the Storage Read API, which serves Arrow record batches (the Storage Read API supports Arrow or Avro). Use service-account or workload-identity auth (yup-oauth2). This gives high-throughput Arrow reads for materialization. (The smaller bigquery-storage crate also outputs Arrow RecordBatch but is narrower.)
Postgres / Redshift → tokio-postgres. Mature, async, well-maintained. Redshift speaks the Postgres wire protocol. Row-oriented, so batch into Arrow on the Rust side; acceptable because this is batch materialization, not the hot path.
Databricks → ADBC (Flight SQL / Databricks driver) or REST/Thrift. Databricks SQL warehouses expose Arrow-based result fetching; ADBC or the Databricks SQL connector path is preferred. If ADBC coverage is thin, fall back to the Databricks SQL Statement Execution REST API (which can return Arrow). Treat as the least-mature connector and isolate behind the connector trait.

Arrow Flight SQL per warehouse: availability is uneven — it is not a universal warehouse protocol in 2026. ADBC (which can wrap Flight SQL where available and Go drivers elsewhere) is the better abstraction than betting on Flight SQL everywhere.

Connector trait: define WarehouseConnector::materialize(query) -> Arrow stream with per-warehouse implementations, so the ADBC FFI (Go) dependency is contained to Snowflake/Databricks and does not contaminate the pure-Rust build for BigQuery/Postgres.

Risks: ADBC Go drivers pull a cgo/Go runtime into the binary via FFI — this complicates the "pure Rust single binary" story. Mitigation: gate ADBC behind a build feature and/or run warehouse materialization only in the worker role image, keeping the api image pure-Rust.

Layer 5 — Object Storage + Parquet

Decision: object_store crate (Apache arrow-rs) for S3/GCS/Azure/local, and the parquet crate (arrow-rs) AsyncArrowWriter / ParquetRecordBatchStreamBuilder for async Parquet I/O.

Evidence + maintenance: object_store is a focused, high-performance async crate originally from InfluxData, donated to Apache Arrow, now in its own arrow-rs-object-store repo; the same binary runs against S3/GCS/Azure/local via config. The parquet crate's ParquetObjectReader integrates with object_store to optimize IO based on predicates/projections; ParquetObjectWriter implements the async writer over any ObjectStore. Multipart uploads are supported via put_multipart / BufWriter.

Multipart uploads for large rollups (put_multipart).
Conditional PUTs (If-Match / If-None-Match) — object_store exposes PutOptions/preconditions; use these to make the cache index / rollup manifest updates atomic (compare-and-swap on an index object) so concurrent workers don't corrupt the cache catalog. This is the integrity primitive for the result cache and Parquet-rollup index.
S3 Express One Zone relevance: for the node-local-adjacent hot cache tier, S3 Express One Zone offers single-digit-ms latency and could reduce the httpfs small-request penalty, but it is single-AZ (durability/AZ-affinity tradeoffs). Recommend NVMe emptyDir as the first-line hot cache and treat S3 Express as an optional acceleration for shared cache, not a requirement.

Runner-up: opendal. Apache OpenDAL is a superb, broader storage abstraction (more backends). For Kestrel, object_store is the better fit because it is co-developed with the parquet/arrow crates and integrates natively with the Parquet reader/writer. Choose object_store; keep OpenDAL in mind only if exotic backends are needed.

Risks: arrow-rs recently refactored the object_store ↔ parquet integration (feature reorganization); pin compatible versions of arrow, parquet, and object_store and upgrade together.

Layer 6 — Postgres Metadata + Job Queue

Decision: sqlx (async, compile-time-checked queries) for the metadata store, and a Postgres SKIP LOCKED job queue — either hand-rolled on sqlx or via apalis with its Postgres backend (which uses LISTEN/NOTIFY + SKIP LOCKED). Postgres is the only stateful dependency; this is deliberate.

Why sqlx over diesel/sea-orm: sqlx is async-native (matches tokio/tonic), supports compile-time query verification against a live schema, and has no heavy ORM layer — appropriate for a system that writes deliberate SQL. diesel is synchronous-first (diesel-async exists but is less ergonomic) and ORM-centric; sea-orm is a full async ORM built on sqlx but adds abstraction we don't need for a control-plane schema. Recommend sqlx.

Job queue options evaluated:

Hand-rolled SELECT ... FOR UPDATE SKIP LOCKED on sqlx: minimal, transparent, transactional (enqueue in the same tx as metadata writes — true exactly-once for DB-side effects), and it's the pattern every mature Postgres queue uses. Recommended default for maximum control and zero extra deps.
apalis + apalis-postgres/apalis-sql: mature worker framework; Postgres storage "uses NOTIFY and SKIP LOCKED", supports polling and NOTIFY-driven (low-latency) storages, heartbeats, orphaned-job re-enqueue, middleware, and an apalis-board UI. Good if you want batteries-included workers rather than hand-rolling retries/heartbeats.
graphile_worker_rs: a Rust rewrite of Graphile Worker — SKIP LOCKED claiming, LISTEN/NOTIFY wakeups, cron, priorities, retries, job keys; schema-compatible with the Node version. Strong option if you like Graphile's model.
sqlxmq: sqlx-based Postgres MQ with transactional enqueue/checkpointing. Functional but lower recent activity; verify maintenance before adopting.
pgmq (Tembo, Postgres extension) via apalis-pgmq: requires installing the pgmq extension in Postgres — acceptable if you control the Postgres image, but adds a server-side dependency.

Recommendation: Start with a hand-rolled SKIP LOCKED queue on sqlx (few hundred lines, full control, transactional with metadata). Adopt apalis if/when you want its worker lifecycle/observability without building it. Avoid the pgmq extension unless you already ship a custom Postgres image.

Cron scheduling: tokio-cron-scheduler (async, tokio-native, persistent-store options) for scheduled materializations; croner as the cron-expression parser if you build scheduling into the queue directly. Recommend tokio-cron-scheduler for the materialization scheduler, with schedules stored in Postgres so any worker pod can own them via leader-election or a scheduled-jobs table with SKIP LOCKED.

Risks: Postgres-as-queue latency is higher than a dedicated broker, but the single-Postgres simplicity is worth it at Kestrel's scale. Use LISTEN/NOTIFY to cut polling latency. Watch connection counts — use a pool and, if needed, PgBouncer.

Layer 7 — Server-Side Chart Rendering (no Chromium)

Decision: Render charts with ECharts SSR producing SVG strings, then rasterize to PNG with the Rust resvg crate. Two viable placements for the ECharts step; recommend a minimal Node/Bun sidecar for v1, with an embedded-JS-engine path as an optimization.

Why this guarantees parity: the interactive browser layer uses ECharts; if deliveries (Slack/email) also render the same ECharts option JSON server-side, the emailed chart is pixel-consistent with what the user sees. Per the Apache ECharts Handbook, ECharts introduced a zero-dependency, string-based SVG SSR mode in v5.3.0 (echarts.init(null, null, {renderer:'svg', ssr:true, width, height}) → chart.renderToSVGString()), and a lightweight client runtime in v5.5.0 that allows some interaction without loading full ECharts on the client. No headless browser required.

Pipeline:

Server builds the ECharts option JSON (same structure the frontend uses).
ECharts SSR renders option → SVG string.
resvg rasterizes SVG → PNG (pure Rust, no Chromium, no cairo/GNOME deps; static SVG subset — perfect for charts). PNG is cached in object storage and attached to Slack/email.

Where the ECharts step runs:

Recommended v1: a tiny Node or Bun sidecar exposing an option JSON → SVG string endpoint. Simple, uses upstream ECharts unmodified, trivially kept in version-sync with the frontend's ECharts. Operationally it's one small container, not a Chromium pool.
Embedded-in-Rust option: charming crate with the ssr feature, which embeds a deno_core (V8) engine to run ECharts' JS inside the Rust process and can emit SVG (and, with ssr-raster, PNG). This removes the sidecar but pulls a V8 engine into the binary (build size/complexity) and couples you to charming's ECharts option coverage. Alternatives for embedding the JS are boa/quickjs (lighter than V8 but ECharts is heavy — V8/deno_core is the realistic choice). Recommend deferring the embedded path until the sidecar proves a bottleneck.

Runner-ups considered:

plotters (pure Rust charting): excellent and dependency-light, but it is a different rendering engine than the browser's ECharts, so email charts would not match interactive charts. Rejected for the parity requirement; acceptable only for internal/system charts.
Vega/Vega-Lite server-side: capable, but again a different engine from ECharts and pulls a JS runtime anyway. Rejected unless the interactive layer were Vega too.
Headless Chromium pool: the thing we're explicitly avoiding — heavy memory, slow cold starts, security surface, and operational pain at scale.

Operational-simplicity quantification: A headless-Chromium renderer typically needs hundreds of MB of RAM per instance and hundreds-of-ms-to-seconds cold starts, plus a browser pool to manage concurrency. The ECharts-SSR→resvg pipeline renders SVG in a Node/Bun process (tens of MB) and rasterizes in-process in Rust in milliseconds, with no browser lifecycle to manage. This is the decisive operational win.

Risks: ECharts SSR does not run interaction/animation JS (fine for static images). Fonts: resvg needs fonts loaded (bundle a known font set into the image for deterministic text rendering). Keep ECharts versions pinned identically between frontend and the SSR sidecar.

Layer 8 — Frontend Stack

Decision: Next.js (App Router) + TypeScript, data layer via @connectrpc/connect-web + @connectrpc/connect-query (TanStack Query integration), ECharts for interactive charts, apache-arrow (JS) for zero-copy table ingestion, TanStack Table + TanStack Virtual for virtualized grids, and a react-grid-layout-style dashboard grid.

Details + evidence:

connect-query wraps TanStack Query with generated, type-safe hooks per RPC; query keys are structured [service, method, input, transport]. Codegen via @bufbuild/protoc-gen-es (messages) + @connectrpc/protoc-gen-connect-query (hooks). Transport is createGrpcWebTransport pointing at tonic-web. Connect-Query is a mature, thoroughly-tested TanStack Query expansion pack maintained by Buf.
Arrow-JS (apache-arrow npm): decode Arrow IPC/Flight payloads to columnar Tables for near-zero-copy rendering into virtualized grids — no row-by-row JSON parsing in the hot path, matching the Arrow-end-to-end pillar.
Virtualized tables: TanStack Table (headless) + TanStack Virtual for windowed rendering of large result sets; feed directly from Arrow columns.
Dashboard grid: react-grid-layout is the incumbent (verify maintenance/React 19 compat before committing); gridstack is the primary alternative and is actively maintained with a React wrapper. Recommend evaluating both against React 19 / App Router; lean gridstack if react-grid-layout's maintenance has stalled.
Explore-state as a protobuf message: model the explore/query builder state as a proto message (the same type the API consumes). This makes URLs shareable (serialize the message), makes undo/redo trivial (immutable snapshots), and eliminates client/server drift. Keep transient UI state in React/Zustand; keep the query definition in the proto message.

Runner-ups: plain grpc-web JS client (works, but connect-web has better DX and the connect-query/TanStack integration); Redux (heavier than needed — TanStack Query owns server state, Zustand owns local UI state).

Risks: App Router + gRPC-Web transport must run client-side (browser fetch/streaming); ensure RPC calls are in client components or route handlers, not RSC where the transport isn't available. gRPC-Web needs the server (tonic-web) to expose the right CORS + grpc-web content types.

Layer 9 — AuthN / AuthZ

Decision: OIDC via the openidconnect crate for authentication, jsonwebtoken for JWT validation/issuance of Kestrel session tokens, and embedded cedar-policy (Cedar) for authorization (RBAC + attribute-based row-level security). Do not introduce an external Zanzibar service.

Evidence + maturity:

openidconnect 4.0.1 — the de-facto standard Rust OIDC crate (David Ramos / ramosbugs), strongly-typed, ~9M+ downloads, fully documented; 4.0.0 notes state no further breaking changes expected until the next major. Production-ready. Depends on oauth2 ^5.
jsonwebtoken 10.4.0 (2026-05-11) — extremely widely used (~87M+ downloads), modern crypto backends (aws-lc-rs, ed25519-dalek 2, rsa 0.9, p256/p384), auto-validates exp/nbf. Production standard.
cedar-policy — AWS's open-source policy language/engine, purpose-built for RBAC and ABAC, embeddable directly as a Rust crate (Authorizer::is_authorized), with a schema validator for policy typechecking and fast, indexable evaluation. Policies live outside code, can be authored/audited independently.

Why Cedar (embedded) over OpenFGA/SpiceDB/Oso/casbin:

OpenFGA / SpiceDB are excellent Zanzibar-style ReBAC systems but are external services — running one directly conflicts with Kestrel's single-binary, single-stateful-store (Postgres) goal. Rejected for v1. (If deeply nested relationship graphs ever become central, revisit — but Cedar covers RBAC+ABAC, which is what BI row-level security needs.)
Oso open-source library direction has been deprioritized/uncertain — avoid depending on it.
casbin-rs is a solid embedded option but Cedar's schema validation, ABAC ergonomics, and formal-analysis tooling are a better fit for expressing row-level security as attribute conditions.

Row-level security pattern: model the semantic query as a Cedar Action on a Resource (metric/dimension/dataset) with Context attributes (tenant, team, PII scope). On authorize, Cedar returns allow/deny and the policy set determines attribute-based filters to inject. Kestrel then injects those filters into the MetricFlow query spec (as additional where/dimension constraints) before SQL emission — RLS is enforced in the generated SQL, not post-filtered. This is the secure design: the warehouse/DuckDB never sees rows the user can't access.

Secrets / warehouse credentials — envelope encryption: store warehouse credentials in Postgres encrypted with a data key, where the data key is itself encrypted by a KMS master key (AWS KMS / GCP KMS) — classic envelope encryption. At runtime, the pod (with IRSA/workload-identity) calls KMS to decrypt the data key, decrypts the credential in memory, and never persists plaintext. Rotate data keys without re-encrypting everything by versioning them. Prefer cloud-native workload identity (IRSA / GKE Workload Identity) over static keys wherever the warehouse supports it (e.g., Snowflake external OAuth, BigQuery workload identity).

Risks: Cedar entity/attribute modeling for RLS is non-trivial; invest in a clear schema and a policy test suite. JWT/OIDC clock-skew and key-rotation (JWKS caching) must be handled.

Layer 10 — Observability + Kubernetes Ops

Observability decision: Instrument with tracing and bridge to OpenTelemetry via tracing-opentelemetry; export metrics via the metrics facade + metrics-exporter-prometheus (or the OTel metrics pipeline).

Maturity reality (mid-2026): OpenTelemetry-Rust has stable Logs and Metrics APIs+SDKs (0.32 line) but Traces are still Beta, and every first-party OTel crate is pre-1.0 — breaking changes land in minor releases (0.28 was a big one), and the crates version in lockstep. Pin exact versions and upgrade the whole family together. Notably this is the reverse of Go/Java, where traces stabilized first — do not assume the maturity order transfers. tracing-opentelemetry 0.33 pairs with OpenTelemetry 0.32. The metrics crate (0.24.x, ~87M downloads) is mature but also pre-1.0. Recommendation: use tracing as the primary instrumentation API (the whole axum/tower/tonic/sqlx/reqwest ecosystem emits tracing spans), bridge to OTel for traces, and use metrics + Prometheus for metrics since that's the stable, K8s-native path. Structured errors via thiserror (library) + anyhow (bin) with error codes mapped to gRPC Status.

Kubernetes serving-tier design (DuckDB pods):

Deployment, not StatefulSet, for query pods. The pods are stateless (cache is disposable); a Deployment scales/reschedules more freely and is spot-friendly. Use a StatefulSet only if you need stable per-pod identity for sharded cache affinity (see below) — for v1, Deployment.
Scale-to-zero via KEDA. HPA cannot go below 1 replica; KEDA patches replicas to 0 when idle and back up on demand. Drive scaling on a queue-depth / in-flight-query custom metric (KEDA's Postgres scaler on a "pending queries" query, or a Prometheus scaler on an in-flight-queries gauge) rather than CPU — CPU can't trigger scale-from-zero (no pods, no metric). This directly serves the cost goal: idle → 0 pods.
KEDA vs prometheus-adapter: KEDA is the better fit because it owns scale-to-zero and has 60+ scalers (Postgres, Prometheus, queue). Use prometheus-adapter only if you're standardizing all custom-metric HPAs on it and don't need scale-to-zero. Recommend KEDA.
Spot tolerance: run serving pods on spot/preemptible nodes with tolerations + graceful-shutdown handling (drain in-flight queries on SIGTERM, short terminationGracePeriodSeconds). Because pods are stateless and queries are re-runnable, spot interruptions are cheap. Keep the api role on on-demand nodes for stable ingress.
NVMe cache: emptyDir on NVMe-backed instances, not a PersistentVolume. emptyDir (medium: NVMe local disk) is disposable, fast, and spot-compatible; a local PV/StatefulSet couples a pod to a node and fights scale-to-zero. Point DuckDB temp_directory (spill) and the httpfs byte-range cache at this emptyDir.
Cold starts are the scale-to-zero tradeoff: mitigate with lean images (Rust static binary → tiny image; the DuckDB-bundled binary is larger but still fast to start), tuned readiness probes, and optionally minReplicaCount: 1 during business hours via KEDA cron scaling. A compiled Rust binary starts in tens of milliseconds with a small RSS, which is exactly why scale-to-zero is viable here (vs. a JVM/Node service).
Pod routing / consistent-hashing cache affinity: routing identical queries to the same pod improves node-local cache hit rate. But it adds a stateful router, complicates scale events (rebalancing), and fights spot churn. Recommendation: skip consistent-hashing for v1. Rely on the shared object-storage result cache + per-node NVMe warm-up. Revisit consistent-hashing (e.g., via a hashing ingress or a query-router) only if cache-hit metrics show node-local misses dominating latency — that's the threshold that would change this decision.

Risks: KEDA scale-from-zero latency (cold start) hits the first query after idle — acceptable for a cost-optimized tier, but set expectations / offer a "keep-warm" schedule for latency-sensitive tenants.

Layer 11 — MCP Server (semantic layer for AI agents)

Decision: Use the official rmcp crate (modelcontextprotocol/rust-sdk) with the streamable HTTP transport, mounted on the same axum stack. Expose the semantic layer (list metrics/dimensions/entities, run metric queries, describe the semantic graph) as MCP tools.

Evidence + maturity: rmcp is the official Rust MCP SDK; it implements current spec revisions (2025-11-25 and 2026-07-28) and ships the Streamable HTTP transport (the successor to HTTP+SSE; SSE parsing is an implementation detail behind the streamable transport, not a separate transport). It integrates cleanly with axum (StreamableHttpService + a session manager), and #[tool_router]/#[tool] macros make tool definitions straightforward. As of the 2026-07-28 spec it supports stateless serving and standardized routing headers. A community rust-mcp-sdk also exists (claims 100% conformance, Axum/Actix integrations) as a fallback, but prefer the official rmcp.

Auth for MCP: MCP's current spec supports OAuth-based auth; reuse Kestrel's OIDC/JWT layer to authenticate MCP clients, and run every MCP-issued query through the same Cedar authorization + RLS filter injection as the human API. The agent must never bypass row-level security. Rate-limit and audit MCP tool calls.

Risks: MCP spec is evolving quickly (multiple revisions per year); pin rmcp and track spec changes. Ensure agent queries are sandboxed to the semantic layer (no raw SQL passthrough).

Layer 12 — Arrow Flight SQL (P2 feasibility note)

Exposing Kestrel's serving tier as an Arrow Flight SQL endpoint (so external BI/notebook tools connect over Flight SQL) is feasible with the arrow-flight crate (arrow-rs), which provides Flight and Flight SQL building blocks. This is a P2 item: it would let third-party tools pull Arrow directly from Kestrel at wire speed. The main work is mapping Flight SQL's prepared-statement/catalog RPCs onto Kestrel's MetricFlow planner and DuckDB execution. Defer until the core product ships; the crate maturity is sufficient that this is low-risk when scheduled.

Performance Targets & Evidence

Targets below are engineering goals; each is annotated with the supporting evidence and its strength.

API tail latency (non-query RPCs): p99 < 15 ms in-process. Evidence: the independent grpc_bench suite (i9-13900KF) measured tonic at ~181,577 req/s, p99 6.61 ms, ~18.9 MiB RSS — within ~8% of the fastest engine while using a fraction of the memory of JVM/.NET/grpcio stacks (88–182 MiB). Caveat: synthetic unary RPC; Go can match/beat raw throughput, and tonic requires proper tokio runtime tuning (a known pitfall: a mis-tuned tonic server once self-capped at ~10% CPU).
Data movement: Arrow end-to-end, no row-by-row JSON in hot paths. Evidence: peer-reviewed benchmarking of Apache Arrow Flight (arXiv:2204.03032) reports up to ~6000 MB/s (DoGet) / ~4800 MB/s (DoPut), utilization up to 95% of available bandwidth, and Dremio-on-Flight performing ~20–30× better than turbodbc/ODBC; the paper notes >80% of data-access time is typically serialization/deserialization in row-oriented paths. Treat vendor "20×/60–90%" claims (Dremio/MotherDuck) as directional marketing; the arXiv figures are the citable ones. (Avoid the IJIRSET "96×/340×" figures — non-reputable, uncorroborated.)
Cold start for scale-to-zero: first-query wake < ~1 s (pod schedule + Rust start + first S3 read). Evidence: a compiled Rust service cold-started in ~20.9 ms using ~15–16 MB vs Node.js ~132.7 ms / ~64 MB in an equivalent Lambda benchmark; Rust needs no JVM-style warmup. This is precisely why scale-to-zero is viable for Kestrel's serving tier. Caveat: single-author benchmark, but consistent with the no-GC/static-binary expectation; the dominant cold-start cost in K8s will be pod scheduling + image pull, not the binary.
Parquet-on-S3 query latency: single-digit-second interactive queries on rollups, sub-second on cached/NVMe-warm rollups. Evidence + hard caveat: DuckDB httpfs can amplify to thousands of small range requests (documented 150,000-request/19-minute and 80→4,200-request cases in DuckDB's own issues), which dominates S3 latency; mitigations (large row groups, metadata cache, NVMe byte-range cache, column pruning) are mandatory and have shown ~10× improvements. DataFusion is now top of the ClickBench raw-Parquet category (DataFusion 43.0.0, Nov 2024), giving Kestrel a credible alternative engine if DuckDB's object-store behavior underperforms.
Chart render: < ~100 ms per chart (ECharts SVG + resvg PNG), vs hundreds of MB / cold-start seconds for a Chromium pool. Evidence: ECharts zero-dependency SVG SSR (v5.3.0+) + pure-Rust resvg rasterization; no browser lifecycle. (Quantitative rasterization benchmarks for resvg are workload-dependent; the operational argument — tens of MB vs a browser pool — is the strong claim.)

Honesty note: the thinnest evidence areas are (1) a clean, current DuckDB-vs-DataFusion wall-clock head-to-head for 2025–2026 (rely on the Nov-2024 DataFusion 43.0.0 ClickBench announcement + the live ClickBench leaderboard), and (2) independently-published tonic-vs-Go production benchmarks (grpc_bench is synthetic). Treat the corresponding targets as provisional pending Kestrel's own benchmarks.

Changes From the Earlier Draft

This section flags where research contradicted or improved a naive initial plan:

Connect-RPC in Rust is NOT ready — do not build the API on it. Both axum-connect (0.5.3, ~9 months stale) and connectrpc-axum ("not recommended for production use yet") are immature. Change: use mature tonic + tonic-web on the server and get Connect's DX only on the client via @connectrpc/connect-web's gRPC-Web transport. This is a material change from any "Connect everywhere" assumption.
DataFusion is a real contender for the serving tier, not just DuckDB. DataFusion 43.0.0 became the fastest ClickBench raw-Parquet engine (Nov 2024) and is Arrow-native/zero-copy. Change: put the serving engine behind a trait and treat DataFusion as a first-class hedge/alternative rather than hard-committing to DuckDB. DuckDB still wins v1 on SQL completeness and operational familiarity.
DuckDB httpfs request amplification is a first-order design constraint, not a footnote. Documented 150,000-request/19-minute and 80→4,200-request pathologies mean the NVMe byte-range cache, large row groups, metadata caching, and column pruning are mandatory from day one — and strengthen the case for DuckLake (v1.0, production-ready, April 2026) as the rollup store because its DB-catalog metadata enables aggressive file pruning and compaction.
ADBC pulls a Go runtime via FFI — this dents the "pure Rust single binary" ideal. Change: confine ADBC (Snowflake/Databricks) to the worker role image behind a build feature, keeping the api image pure-Rust; use pure-Rust gcp-bigquery-client and tokio-postgres where possible.
OpenTelemetry-Rust traces are still Beta and all OTel crates are pre-1.0. Change: lead with tracing + Prometheus metrics (stable, K8s-native) and treat OTel tracing as bridged-but-versioned-carefully; pin the whole OTel crate family and upgrade together.
External authz services (OpenFGA/SpiceDB) are ruled out as violating the single-binary/single-stateful-store constraint. Change: embed Cedar and enforce RLS by injecting attribute filters into the MetricFlow query spec before SQL emission.
No mature Rust MetricFlow reimplementation exists — this is confirmed greenfield and the dominant build risk. Change: elevate the differential/golden conformance harness (execute Kestrel SQL vs MetricFlow SQL on DuckDB, compare Arrow results) to a P0 workstream.

Build-Order Implications

Foundations (P0): single-binary skeleton with role flags; tonic+tonic-web+axum on one port; Postgres schema + sqlx + hand-rolled SKIP LOCKED queue; object_store + parquet I/O; OIDC/JWT.
Semantic core (P0, highest risk — start immediately, in parallel): semantic_manifest.json parser → semantic graph → dataflow DAG → sqlparser-rs dialect emission, beginning with DuckDB + Postgres dialects; stand up the differential conformance harness against MetricFlow on DuckDB from the first metric type.
Serving tier (P0): DuckDB via duckdb-rs behind the ServingEngine trait; httpfs + NVMe byte-range cache; KEDA scale-to-zero on a queue-depth metric; Arrow end-to-end.
Materialization (P1): worker-role connectors — BigQuery (Storage Read API) and Postgres first (pure Rust), then Snowflake/Databricks via ADBC (feature-gated); scheduled rollups; evaluate DuckLake-on-Postgres as the rollup store.
Frontend (P1): Next.js App Router + connect-query over gRPC-Web; Arrow-JS grids; ECharts; explore-state proto.
Deliveries (P1): ECharts-SSR sidecar + resvg PNG pipeline; Slack/email.
AI + interop (P2): rmcp MCP server; Cedar policy authoring UX; Arrow Flight SQL endpoint.

Risk Register

#	Risk	Likelihood	Impact	Mitigation
1	MetricFlow reimplementation correctness (cumulative/conversion/multi-hop joins)	High	Critical	Differential harness executing Kestrel vs MetricFlow SQL on DuckDB, Arrow result equality; pin manifest schema version; start P0
2	DuckDB httpfs request amplification on S3	High	High	Large row groups, metadata cache, NVMe byte-range cache, column pruning; consider DuckLake; DataFusion hedge
3	ADBC Go-FFI dependency pollutes the "pure Rust binary"	Medium	Medium	Feature-gate ADBC into `worker` image only; pure-Rust BigQuery/Postgres paths
4	OTel Rust pre-1.0 churn / traces Beta	High	Low–Medium	Lead with tracing + Prometheus; pin OTel family; upgrade together
5	Connect-in-Rust immaturity (if adopted)	—	—	Avoided: tonic-web server + connect-web client
6	Scale-to-zero cold-start latency on first query	Medium	Medium	Lean images, readiness tuning, optional KEDA cron warm-up for latency-sensitive tenants
7	Cedar RLS modeling errors leaking rows	Medium	Critical	Filter injection into query spec (not post-filter); policy test suite; schema validation
8	DuckDB C++ bundled build size / compile time	Medium	Low	`bundled` feature for reproducibility; multi-stage image; cache builds
9	Spot interruptions on serving pods	Medium	Low	Stateless pods + graceful drain; queries re-runnable; api role on on-demand
10	Chart font/version drift breaking parity	Low	Medium	Pin identical ECharts versions frontend/sidecar; bundle fonts into image
11	MetricFlow/OSI spec evolution	Medium	Medium	Target the artifact schema (not YAML), pin versions, track OSI

Caveats

Several performance figures derive from vendor blogs or single-author benchmarks; they are directional. The peer-reviewed Arrow Flight numbers, DuckDB's own issue tracker (httpfs amplification), the reproducible grpc_bench suite, and the Apache DataFusion 43.0.0 ClickBench announcement are the strongest citations. Kestrel must publish its own benchmarks before hardening the targets.
The MetricFlow-in-Rust planner has no mature open-source precedent in Rust; effort and schedule risk here dominate the project.
Library versions cited are current as of the research window (2025–2026) and move fast — re-verify tonic, duckdb, arrow-rs crates, rmcp, cedar-policy, and the OTel family at implementation time and pin them.
Warehouse connector maturity is uneven; Databricks is the least-certain connector and should be prototyped early if it's a launch requirement.
This document deliberately refers to prior-art products only as "the incumbent category" and names no incumbent BI product.

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md

Navigation Menu

Repository files navigation

Kestrel — Technical Design & Stack Selection

Executive Summary

System Overview (ASCII)

Layer 1 — Rust Web / RPC Layer

Layer 2 — SQL Generation / Query Planning (the MetricFlow-in-Rust core)

Layer 3 — DuckDB Serving/Acceleration Tier

Layer 4 — Warehouse Connectivity (materialization only)

Layer 5 — Object Storage + Parquet

Layer 6 — Postgres Metadata + Job Queue

Layer 7 — Server-Side Chart Rendering (no Chromium)

Layer 8 — Frontend Stack

Layer 9 — AuthN / AuthZ

Layer 10 — Observability + Kubernetes Ops

Layer 11 — MCP Server (semantic layer for AI agents)

Layer 12 — Arrow Flight SQL (P2 feasibility note)

Performance Targets & Evidence

Changes From the Earlier Draft

Build-Order Implications

Risk Register

Caveats

About

Releases

Packages

Contributors

Latest commit

History

Folders and files

Repository files navigation

Kestrel — Technical Design & Stack Selection

Executive Summary

System Overview (ASCII)

Layer 1 — Rust Web / RPC Layer

Layer 2 — SQL Generation / Query Planning (the MetricFlow-in-Rust core)

Layer 3 — DuckDB Serving/Acceleration Tier

Layer 4 — Warehouse Connectivity (materialization only)

Layer 5 — Object Storage + Parquet

Layer 6 — Postgres Metadata + Job Queue

Layer 7 — Server-Side Chart Rendering (no Chromium)

Layer 8 — Frontend Stack

Layer 9 — AuthN / AuthZ

Layer 10 — Observability + Kubernetes Ops

Layer 11 — MCP Server (semantic layer for AI agents)

Layer 12 — Arrow Flight SQL (P2 feasibility note)

Performance Targets & Evidence

Changes From the Earlier Draft

Build-Order Implications

Risk Register

Caveats

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors