diff --git a/CLAUDE.md b/CLAUDE.md index 6445cc77..88b06e7e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -241,7 +241,7 @@ When adding a new Docker stack, **all locations must be updated**: 3. **Pin Docker image versions:** - **CRITICAL:** Always use specific version tags, NOT `latest` - - **Exception:** Only use `latest` for non-critical standalone tools (drawio, it-tools, wetty, code-server, jupyter, marimo, adminer, excalidraw) + - **Exception:** Only use `latest` for non-critical standalone tools (drawio, it-tools, wetty, code-server, jupyter, marimo, adminer, excalidraw, evidence). Common property: presentation-layer or dev-tool, no persistent state beyond cache; upstream publishes no semver tags (only `:latest`), so digest pinning would still require manual update on every refresh and offers no real predictability win. - **Pin versions for:** - All data storage services (databases, object storage, data lakes) - Services with persistent state or databases diff --git a/README.md b/README.md index e87554c9..eb06ef13 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ After deployment you'll have: ![Quick Start Flow](docs/assets/architecture-quickstart.svg) -## Available Stacks (72) +## Available Stacks (73) [![AKHQ](https://img.shields.io/badge/AKHQ-000000?logo=apachekafka&logoColor=white)](https://akhq.io) [![Adminer](https://img.shields.io/badge/Adminer-34567C?logo=adminer&logoColor=white)](https://www.adminer.org) @@ -88,6 +88,7 @@ After deployment you'll have: [![Dozzle](https://img.shields.io/badge/Dozzle-7B16FF?logo=docker&logoColor=white)](https://dozzle.dev) [![Draw.io](https://img.shields.io/badge/Draw.io-F08705?logo=diagramsdotnet&logoColor=white)](https://www.diagrams.net) [![Excalidraw](https://img.shields.io/badge/Excalidraw-6965DB?logo=excalidraw&logoColor=white)](https://excalidraw.com) +[![Evidence](https://img.shields.io/badge/Evidence-7B61FF?logo=markdown&logoColor=white)](https://evidence.dev) [![Filestash](https://img.shields.io/badge/Filestash-2B3A67?logo=files&logoColor=white)](https://www.filestash.app) [![Flink](https://img.shields.io/badge/Apache_Flink-E6526F?logo=apacheflink&logoColor=white)](https://flink.apache.org) [![Garage](https://img.shields.io/badge/Garage-59C6A6?logo=amazons3&logoColor=white)](https://garagehq.deuxfleurs.fr) @@ -163,6 +164,7 @@ After deployment you'll have: | **Dozzle** | Realtime Docker logs in the browser — tail every container without SSH | [dozzle.dev](https://dozzle.dev) | | **Draw.io** | Flowchart and diagramming tool for technical diagrams | [diagrams.net](https://www.diagrams.net) | | **Excalidraw** | Virtual whiteboard for sketching hand-drawn diagrams | [excalidraw.com](https://excalidraw.com) | +| **Evidence** | SQL + markdown BI for analytics engineers — pages diff as plain text, charts render inline, ships with a sample project | [evidence.dev](https://evidence.dev) | | **Filestash** | Web-based file manager with S3/FTP/SFTP/WebDAV backend support | [filestash.app](https://www.filestash.app) | | **Flink** | Distributed stream and batch processing engine (JobManager + TaskManager cluster) | [flink.apache.org](https://flink.apache.org) | | **Garage** | Lightweight S3-compatible object storage for self-hosting | [garagehq.deuxfleurs.fr](https://garagehq.deuxfleurs.fr) | diff --git a/docs/stacks/README.md b/docs/stacks/README.md index f4fac128..7ec5d208 100644 --- a/docs/stacks/README.md +++ b/docs/stacks/README.md @@ -47,6 +47,7 @@ Images are pinned to **major versions** where supported for automatic security p | IT-Tools | `corentinth/it-tools` | `latest` | Latest ² | | Jupyter PySpark | `quay.io/jupyter/pyspark-notebook` | `python-3.13` | Minor | | Excalidraw | `excalidraw/excalidraw` | `latest` | Latest ² | +| Evidence | `evidencedev/devenv` | `latest` | Latest ² | | Filestash | `machines/filestash` | `latest` | Latest ² | | Flink JobManager | `flink` (custom build) | `1.20.1` | Exact ³ | | Flink TaskManager | `flink` (custom build) | `1.20.1` | Exact ³ | @@ -147,6 +148,7 @@ Images are pinned to **major versions** where supported for automatic security p | **Dinky** | Flink SQL IDE with web editor | [dinky.md](dinky.md) | | **Dozzle** | Realtime Docker logs in the browser | [dozzle.md](dozzle.md) | | **Draw.io** | Flowchart and diagramming tool | [drawio.md](drawio.md) | +| **Evidence** | SQL + markdown BI for analytics engineers | [evidence.md](evidence.md) | | **Excalidraw** | Virtual whiteboard for diagrams | [excalidraw.md](excalidraw.md) | | **Filestash** | Web-based file manager | [filestash.md](filestash.md) | | **Apache Flink** | Distributed stream and batch processing | [flink.md](flink.md) | diff --git a/docs/stacks/evidence.md b/docs/stacks/evidence.md new file mode 100644 index 00000000..c55fcc61 --- /dev/null +++ b/docs/stacks/evidence.md @@ -0,0 +1,78 @@ +--- +title: "Evidence" +--- + +## Evidence + +![Evidence](https://img.shields.io/badge/Evidence-7B61FF?logo=markdown&logoColor=white) + +**SQL + markdown BI for analytics engineers** + +[Evidence](https://evidence.dev) is an open-source "BI as code" framework: each page is a Markdown file with embedded SQL blocks that render to charts, tables, and inline values. Projects are plain text — version them in Git, edit them in your normal tools, and the dev server reloads on save. + +This stack ships the Evidence `devenv` runtime preloaded with a sample project that queries the in-stack Postgres. Extend the `sources/` directory to connect ClickHouse, Trino, DuckDB, Iceberg/Lakekeeper, or any external warehouse. + +| Setting | Value | +|---------|-------| +| Host Port | `3007` (container internal port is Evidence's default `3000`; 3000–3006 are already taken by Metabase/Uptime-Kuma/Wetty/Hoppscotch/Dagster/Wiki.js/big-AGI) | +| Suggested Subdomain | `evidence` | +| Public Access | No (Cloudflare Access via email OTP) | +| Website | [evidence.dev](https://evidence.dev) | +| Source | [GitHub](https://github.com/evidence-dev/evidence) | +| Docker image | [`evidencedev/devenv`](https://hub.docker.com/r/evidencedev/devenv) | +| Project root | `/opt/docker-server/stacks/evidence/project/` on the server (mounted at `/evidence-workspace` inside the container) | + +### Why Evidence + +Most BI tools assume a GUI workflow: drag dimensions onto a canvas, save the chart as a binary artifact, hope the underlying SQL stays in sync. Evidence inverts that: SQL is the source of truth, charts are derived, the whole project diffs as plain text. That makes it a natural fit alongside the existing **code-server**, **gitea**, and **woodpecker-ci** stacks — you edit pages like you edit any other code, push to a feature branch, and review the rendered diff before merging to main. + +Compared to the other BI tools in this stack: + +| Tool | Best for | Auth | +|---|---|---| +| **Metabase** | Self-service exploration by non-technical users | Built-in user management | +| **Superset** | Dashboards with rich GUI editing + drilldowns | Built-in user management | +| **Evidence** | Code-first, Git-reviewed analytics pages | Cloudflare Access at the edge | + +### Usage + +1. Enable **Evidence** in the Control Plane → Spin Up. +2. Open `https://evidence.YOUR_DOMAIN` → CF Access email OTP → landing page. +3. Edit the sample page at `/opt/docker-server/stacks/evidence/project/pages/index.md` on the server (the project root is bind-mounted into the container at `/evidence-workspace`, so changes apply on save). +4. Add new pages as `.md` files in `/opt/docker-server/stacks/evidence/project/pages/` — each one renders at `https://evidence.YOUR_DOMAIN/`. + +### Adding data sources + +Each source lives in its own directory under `project/sources//`: + +``` +project/sources/ +├── nexus_postgres/ # shipped with the stack +│ ├── connection.yaml # connection config (env-var interpolated) +│ └── database_overview.sql +└── my_clickhouse/ # operator adds this + ├── connection.yaml + └── ... +``` + +`connection.yaml` supports `${VAR}` interpolation against the container's environment, so the recommended pattern is: + +1. Add the credentials to Infisical under a folder of your choice. +2. Reference them from `stacks/evidence/.env` (the deploy pipeline renders this from Infisical on every spin-up). +3. Use `${VAR}` in `connection.yaml` to reference them. + +For ClickHouse, Trino, MySQL, BigQuery, Snowflake, and others, see the [Evidence connector docs](https://docs.evidence.dev/core-concepts/data-sources/). Add the matching `@evidence-dev/` package to `stacks/evidence/project/package.json` and run `docker compose restart evidence` to pull it in. + +### Building a static site + +For a production hand-off, the devenv runtime can build a static HTML export: + +```bash +ssh nexus 'docker exec evidence npm run sources && docker exec evidence npm run build' +``` + +Output lands in `/opt/docker-server/stacks/evidence/project/build/` on the server. Copy it into any of the file-store stacks (MinIO, Garage, SeaweedFS, RustFS) and serve as static HTML — or commit it to a GitHub Pages / Cloudflare Pages repo for a fully decoupled deploy. + +### Secrets + +No Tofu-managed secrets specific to Evidence. The bundled sample project reads the in-stack Postgres credentials (`POSTGRES_PASSWORD`) which are already managed via the **postgres** stack and Infisical. Operator-added data sources reference whatever credentials the operator wires into `stacks/evidence/.env` — no double-managing. diff --git a/services.yaml b/services.yaml index 2a10b43f..a526ad75 100644 --- a/services.yaml +++ b/services.yaml @@ -237,6 +237,16 @@ services: long_description: "Excalidraw is a virtual whiteboard that produces hand-drawn style diagrams. Perfect for architecture sketches, brainstorming, wireframes, and technical illustrations. Supports real-time collaboration, a library of reusable components, and export to PNG/SVG." image: "excalidraw/excalidraw:latest" + evidence: + subdomain: "evidence" + port: 3007 + public: false + category: "analytics" + website: "https://evidence.dev" + description: "SQL + markdown BI framework — write queries inside Markdown, render charts inline." + long_description: "Evidence is an open-source 'BI as code' framework: each page is a Markdown file containing SQL blocks that render into charts, tables, and value components. Projects live in plain text — version them in Git, edit with your normal tools, and reload on save. This stack ships the devenv runtime preloaded with a sample project that queries the in-stack Postgres; extend the sources/ directory to connect ClickHouse, Trino, DuckDB, Iceberg/Lakekeeper, and more." + image: "evidencedev/devenv:latest" + filestash: subdomain: "filestash" port: 8334 diff --git a/src/nexus_deploy/service_env.py b/src/nexus_deploy/service_env.py index 845bb914..10c88345 100644 --- a/src/nexus_deploy/service_env.py +++ b/src/nexus_deploy/service_env.py @@ -501,6 +501,29 @@ def _render_lakekeeper(c: NexusConfig, e: BootstrapEnv) -> RenderedEnv: ) +def _render_evidence(c: NexusConfig, e: BootstrapEnv) -> RenderedEnv: + """Evidence: SQL+markdown BI runtime. The bundled sample project + queries the in-stack Postgres via env-var interpolation, so we + pipe through the existing ``postgres_password`` field (no + dedicated Evidence secret to manage) plus the absolute public + URL Evidence uses for OG tags + canonical links. + + No fail-fast guard: Evidence renders pages even without a working + data source (it just shows query errors inline), and the operator + may legitimately be wiring an external warehouse instead of the + in-stack Postgres. Leaving the password empty produces an + "auth failed" message on the affected query rather than a crashed + container. + """ + domain_host = service_host("evidence", e.domain or "", e.subdomain_separator) + return RenderedEnv( + env_vars={ + "POSTGRES_PASSWORD": c.postgres_password or "", + "EVIDENCE_BASE_URL": f"https://{domain_host}", + }, + ) + + def _render_litellm( c: NexusConfig, e: BootstrapEnv, *, litellm_config_template: str | None = None ) -> RenderedEnv: @@ -1303,6 +1326,7 @@ def _placeholder_jupyter(c: NexusConfig, e: BootstrapEnv) -> RenderedEnv: EnvSpec("hedgedoc", _is_enabled("hedgedoc"), _render_hedgedoc), EnvSpec("litellm", _is_enabled("litellm"), _render_litellm), EnvSpec("lakekeeper", _is_enabled("lakekeeper"), _render_lakekeeper), + EnvSpec("evidence", _is_enabled("evidence"), _render_evidence), EnvSpec("mage", _is_enabled("mage"), _render_mage), EnvSpec("minio", _is_enabled("minio"), _render_minio), EnvSpec("sftpgo", _is_enabled("sftpgo"), _render_sftpgo), diff --git a/stacks/evidence/docker-compose.yml b/stacks/evidence/docker-compose.yml new file mode 100644 index 00000000..e6501a9e --- /dev/null +++ b/stacks/evidence/docker-compose.yml @@ -0,0 +1,82 @@ +# ============================================================================= +# Evidence - SQL + markdown BI for analytics engineers +# ============================================================================= +# Evidence is a "BI as code" framework: write SQL queries inside +# .md files and the framework renders charts, tables, and value +# components inline. The project directory is plain markdown + +# YAML — version it in Git, edit with your normal tools. +# +# This stack ships the `devenv` runtime — the container watches the +# mounted project directory and re-renders pages on save. For a +# static production export run: +# docker exec evidence npm run sources && docker exec evidence npm run build +# and serve the `build/` output from any of the file-store stacks. +# +# Data sources are configured per-project in `project/sources/`. +# Connection strings reference env vars (Evidence supports +# ${VAR} interpolation in sources/*/connection.yaml), so the +# operator points them at the Infisical-managed Postgres / +# ClickHouse / Trino / DuckDB credentials that already exist +# elsewhere in the stack. +# +# Access: https://evidence.YOUR_DOMAIN (behind Cloudflare Access +# unless you flip `public: true` in services.yaml). +# ============================================================================= + +services: + evidence: + # devenv:latest per CLAUDE.md exception for non-critical + # standalone tools (Evidence is a presentation layer with no + # persistent state beyond cache). The devenv image is the + # supported dev-runtime entrypoint; Evidence does not publish + # versioned devenv tags. + image: ${IMAGE_EVIDENCE:-evidencedev/devenv:latest} + container_name: evidence + # env_file picks up the renderer-populated POSTGRES_PASSWORD / + # EVIDENCE_BASE_URL AND any operator-added variables in + # stacks/evidence/.env — Evidence sources//connection.yaml + # supports ${VAR} interpolation, so additional data-source + # credentials reach the container without having to extend the + # explicit `environment:` list below. + env_file: + - .env + restart: unless-stopped + ports: + # Host 3007 → container 3000 (Evidence's SvelteKit dev port). + # 3000-3006 are already taken by metabase/uptime-kuma/wetty/ + # hoppscotch/dagster/wikijs/big-agi. + - "3007:3000" + environment: + # Bind the dev server to all interfaces so the published port + # is reachable from the host's network namespace (and through + # the Cloudflare Tunnel). + HOST: 0.0.0.0 + PORT: "3000" + # EVIDENCE_BASE_URL (used by Evidence for absolute links + OG tags) + # is provided by env_file above — no explicit entry needed. + # Point the user-supplied sources/*/connection.yaml files at + # the in-stack Postgres for the bundled sample query. Operators + # extend the sources/ folder with additional connections. + POSTGRES_HOST: postgres + POSTGRES_PORT: "5432" + POSTGRES_DATABASE: postgres + POSTGRES_USER: nexus-postgres + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + volumes: + # The sample project shipped under stacks/evidence/project/ is + # synced to /opt/docker-server/stacks/evidence/project/ by the + # deploy pipeline and mounted read-write so users can edit it + # in-place via code-server / a Git checkout. + - ./project:/evidence-workspace + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://localhost:3000/ >/dev/null 2>&1 || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 90s + networks: + - app-network + +networks: + app-network: + external: true diff --git a/stacks/evidence/project/.gitignore b/stacks/evidence/project/.gitignore new file mode 100644 index 00000000..0dbdec8d --- /dev/null +++ b/stacks/evidence/project/.gitignore @@ -0,0 +1,4 @@ +node_modules/ +.evidence/ +build/ +static/data/ diff --git a/stacks/evidence/project/evidence.config.yaml b/stacks/evidence/project/evidence.config.yaml new file mode 100644 index 00000000..4ff9c4b5 --- /dev/null +++ b/stacks/evidence/project/evidence.config.yaml @@ -0,0 +1,6 @@ +deployment: + basePath: "" + +appearance: + default: system + switcher: true diff --git a/stacks/evidence/project/package.json b/stacks/evidence/project/package.json new file mode 100644 index 00000000..9d45c02a --- /dev/null +++ b/stacks/evidence/project/package.json @@ -0,0 +1,21 @@ +{ + "name": "nexus-evidence-project", + "version": "0.0.1", + "private": true, + "scripts": { + "build": "evidence build", + "build:strict": "evidence build:strict", + "dev": "evidence dev --host 0.0.0.0 --port 3000", + "sources": "evidence sources" + }, + "dependencies": { + "@evidence-dev/core-components": "^5.0.0", + "@evidence-dev/evidence": "^40.1.8", + "@evidence-dev/postgres": "^2.0.0" + }, + "type": "module", + "engines": { + "npm": ">=7.0.0", + "node": ">=18.0.0" + } +} diff --git a/stacks/evidence/project/pages/index.md b/stacks/evidence/project/pages/index.md new file mode 100644 index 00000000..09e34dab --- /dev/null +++ b/stacks/evidence/project/pages/index.md @@ -0,0 +1,47 @@ +--- +title: Nexus-Stack on Evidence +--- + +Welcome to Evidence. This file is `pages/index.md` in the project mounted at +`/evidence-workspace`. Edit it from the host (or via the `code-server` / +`gitea` stacks) and the dev server reloads on save. + +## Postgres source + +The bundled `sources/nexus_postgres/` reads the in-stack Postgres credentials +through the env vars that `docker-compose` populates from Infisical. The +sample query below lists the largest tables in the `public` schema — if you +have not yet loaded data, the result will be empty, which is also a healthy +signal that the connection is wired up. + +```sql database_overview +select * from nexus_postgres.database_overview +``` + + + +## Adding more sources + +Drop a sibling directory under `project/sources/` with its own +`connection.yaml` and Evidence will pick it up on the next `npm run sources`. +Connection strings can reference environment variables via `${VAR}` syntax, +so the recommended pattern is to add the relevant credentials to the +`stacks/evidence/.env` file (which the deploy pipeline renders from +Infisical) and reference them here. + +For ClickHouse, Trino, DuckDB, Iceberg/Lakekeeper and other backends, see +the Evidence connector docs and add the matching `@evidence-dev/` +package to `package.json`. + +## Building a static export + +For a production hand-off, run the two commands below inside the +running container: + +```bash +docker exec evidence npm run sources +docker exec evidence npm run build +``` + +The output lands in `project/build/`; copy it into any of the file-store +stacks (MinIO/Garage/SeaweedFS/RustFS) and serve it as static HTML. diff --git a/stacks/evidence/project/sources/nexus_postgres/connection.yaml b/stacks/evidence/project/sources/nexus_postgres/connection.yaml new file mode 100644 index 00000000..8a070592 --- /dev/null +++ b/stacks/evidence/project/sources/nexus_postgres/connection.yaml @@ -0,0 +1,18 @@ +# Nexus-Stack default Postgres source. +# Credentials come from environment variables that the +# docker-compose env_file populates from Infisical. +# Add more sources by creating sibling directories under +# stacks/evidence/project/sources// with their own +# connection.yaml that uses ${VAR} interpolation for any +# credentials. Place .sql query files alongside it; Evidence +# discovers both automatically on `npm run sources`. +name: nexus_postgres +type: postgres +options: + host: ${POSTGRES_HOST} + port: ${POSTGRES_PORT} + database: ${POSTGRES_DATABASE} + user: ${POSTGRES_USER} + password: ${POSTGRES_PASSWORD} + schema: public + ssl: false diff --git a/stacks/evidence/project/sources/nexus_postgres/database_overview.sql b/stacks/evidence/project/sources/nexus_postgres/database_overview.sql new file mode 100644 index 00000000..bc036f23 --- /dev/null +++ b/stacks/evidence/project/sources/nexus_postgres/database_overview.sql @@ -0,0 +1,10 @@ +-- Snapshot of the public schema: table list + estimated row counts. +-- Edit or replace with queries that match the data the operator has +-- loaded into the nexus-postgres instance. +SELECT + relname AS table_name, + n_live_tup AS estimated_rows +FROM pg_stat_user_tables +ORDER BY n_live_tup DESC NULLS LAST, + relname +LIMIT 50; diff --git a/tests/unit/test_service_env.py b/tests/unit/test_service_env.py index 10acb2ba..c97fa204 100644 --- a/tests/unit/test_service_env.py +++ b/tests/unit/test_service_env.py @@ -701,6 +701,59 @@ def test_lakekeeper_domain_respects_subdomain_separator( assert rendered.env_vars["LAKEKEEPER_DOMAIN"] == "lakekeeper-example.com" +# --------------------------------------------------------------------------- +# Evidence — pipes through postgres_password + domain composition +# --------------------------------------------------------------------------- + + +def test_evidence_renders_postgres_password_and_domain( + full_config: NexusConfig, full_env: BootstrapEnv +) -> None: + """Evidence's bundled sample project queries the in-stack Postgres + via env-var interpolation, so the renderer pipes through the + existing postgres_password (no dedicated Evidence secret) plus the + absolute public URL Evidence bakes into OG tags + canonical links. + """ + from nexus_deploy.service_env import _render_evidence + + rendered = _render_evidence(full_config, full_env) + assert rendered.env_vars["POSTGRES_PASSWORD"] == full_config.postgres_password + assert rendered.env_vars["EVIDENCE_BASE_URL"] == "https://evidence.example.com" + + +def test_evidence_domain_respects_subdomain_separator( + full_config: NexusConfig, full_env: BootstrapEnv +) -> None: + """Multi-tenant forks set subdomain_separator='-' for flat + subdomains (evidence-user1.example.com). Renderer must use + service_host so EVIDENCE_BASE_URL tracks that override.""" + from nexus_deploy.service_env import _render_evidence + + env = BootstrapEnv( + **{ + **{k: getattr(full_env, k) for k in full_env.__dataclass_fields__}, + "domain": "user1.example.com", + "subdomain_separator": "-", + } + ) + rendered = _render_evidence(full_config, env) + assert rendered.env_vars["EVIDENCE_BASE_URL"] == "https://evidence-user1.example.com" + + +def test_evidence_does_not_raise_on_empty_postgres_password( + full_config: NexusConfig, full_env: BootstrapEnv +) -> None: + """Evidence has no fail-fast guard: empty postgres_password + surfaces as a per-query auth-failed message inline, not a crashed + container. The operator may also be wiring an external warehouse + instead of the in-stack Postgres.""" + from nexus_deploy.service_env import _render_evidence + + config = full_config.model_copy(update={"postgres_password": ""}) + rendered = _render_evidence(config, full_env) + assert rendered.env_vars["POSTGRES_PASSWORD"] == "" + + # --------------------------------------------------------------------------- # SFTPGo — fail-fast guard # ---------------------------------------------------------------------------