feat: central reusable backup engine (BackupHelper v1) - #1
Merged
Merged
Conversation
Set up the central backup engine as a modern src-layout Python package so plugin discovery via entry points works cleanly — the primary extension point for consuming repos. * pyproject.toml declares the package, runtime deps (pydantic, boto3, apscheduler, tenacity, typer, rich, PyYAML) and a `backuphelper.sources` entry-point group that repos extend with their own Source plugins * pytest configured with import-mode=importlib so per-package test modules can share basenames (e.g. sources/test_base.py, notify/test_base.py) * .gitignore / .dockerignore exclude the venv, caches, /data and secrets
Configuration is the migration-critical surface: it must accept the whole
multi-job config inline in compose (no host file) like the fleet's init.json
containers, while staying backward compatible with discrete env vars.
* config loader merges four layers by precedence: discrete BACKUP_..__ env
overrides > BACKUP_CONFIG_JSON(_BASE64) inline > BACKUP_CONFIG_FILE >
model defaults, with recursive ${VAR} interpolation resolved from an
injectable env mapping so secrets never live in the JSON literal
* pydantic models: RootConfig/Job with open SourceSpec (plugin types keep
their extra fields) and closed DestinationSpec (local | s3 only)
* streamed sha256 hashing (constant memory), a schema-versioned
self-describing manifest (embedded + sidecar) with an open `kind`
vocabulary + metadata contribution hook, and a byte-deterministic
tar.gz bundler (sorted members, gzip mtime=0) with traversal-safe extract
The core knows HOW to move bytes; sources declare WHAT to capture. Every source stages files and returns components; the engine hashes and bundles them. App-specific sources (n8n, NocoDB) plug in via the entry-point registry without touching engine code. Sources (each with backup + restore, password via subprocess env never argv): * postgres — pg_dump custom/plain + pg_restore/psql, PG client-version pinnable * mariadb / mysql — mariadb-dump (mysqldump fallback), multi-DB, utf8mb4 * s3 — full-bucket mirror PRESERVING per-object metadata/content-type/tags, faithfully re-applied on restore (a capability no fleet tool had) * filesystem — named path-groups (uploads/content/…) with exclude globs, deterministic tar, independent restore selectors * env — whitelist env-var snapshot Destinations are S3 or local only: local is the working store, S3 the off-site target when configured. S3 uses a hand-rolled equal-chunk multipart (abort-on-failure + post-upload size verify) for MinIO/Ceph compatibility — deliberately not boto3 upload_file. Retention combines count / age / GFS / smart-last (never prune the sole backup), applied independently per destination. Plus optional age/gpg client-side encryption, a tenacity retry helper (429-aware) and the plugin/hook extension API (source discovery + opt-in lifecycle hooks).
Wires the building blocks into a working engine and container entrypoint. * runner: one job end to end — produce → hash → embedded manifest → deterministic bundle → optional encrypt → sidecar manifest (archive_sha256) → put to every destination → retention → tri-state notify; plus restore_snapshot (decrypt → extract → per-source restore) and lifecycle hooks. Staging lives outside the destination listing so it never pollutes retention * notify: severity-gated fan-out with per-channel fault isolation across email, HMAC-SHA256 webhook (X-Signature-256), Teams (Adaptive Card + MessageCard fallback), Slack, Discord, ntfy/Gotify and a healthchecks.io dead-man's-switch * scheduler: APScheduler cron/interval accepting raw-cron or field-based input, coalesce + max_instances=1 + misfire grace, on-startup, SIGTERM drain * CLI (Typer): create/list/show/verify/restore/prune/download/config/ healthcheck, plus a --now one-shot and the daemon default * logging with a secret-redacting filter (key=value, DSN and JSON forms) and a functional healthcheck reflecting last-backup staleness
The production image cannot be assembled unless the pytest stage passes
(COPY --from=test), baking the quality gate into the build itself.
* multi-stage Dockerfile (builder → test-gate → prod) on python:3.14-alpine,
non-root backup user (uid/gid 1000), tini PID 1, mariadb-client +
postgresql${PG_CLIENT_VERSION}-client + gnupg + age, and a functional
HEALTHCHECK; PG client major pinnable via build-arg
* docker-compose.yml demonstrates the inline BACKUP_CONFIG_JSON multi-source
config with ${VAR} secret references — no host config file needed
* .env.example documents discrete-env, inline-JSON and base64 config paths
* README covers config layers, sources, CLI and the meta-Dockerfile pattern
consuming repos use to adopt the image
Follows the BAUER GROUP convention: reusable automation-templates workflows, semantic-release, dual publish to GHCR and Docker Hub. * docker-release.yml runs pytest for fast feedback, then semantic-release and a multi-arch (amd64/arm64) build gated on a created release; PRs get a no-push build validation * explicit permissions, timeouts and secrets: inherit throughout * dependabot keeps pip, github-actions and docker deps current with chore(deps)/chore(ci) commit prefixes
Two correctness improvements surfaced while writing the docs. * keep_local (Job, default true): when false, the local copy is deleted after a successful off-site S3 upload — the archive then lives only off-site while local stays the working store. The local copy is retained if the upload had errors, so a failed off-site push never leaves you with nothing. * the `prune` CLI now parses the real timestamp from each snapshot id (the same helper the scheduler uses) instead of stamping "now", so age- and GFS-based retention behave identically whether pruning runs automatically after a backup or manually via the CLI.
Replaced the minimal release workflow with the full standard CI/CD stack used across the fleet (see CS-Outline), so BackupHelper is maintained the same way as every other image. * docker-release.yml — validate-compose + validate-scripts + pytest gate → semantic-release → multi-arch GHCR/Docker Hub build (release + PR variants), with SBOM, Docker Hub README sync and Dockerfile version write-back * check-base-images.yml — daily base-image digest monitor that triggers a release when python:3.14-alpine moves (config in .github/config/…) * docker-maintenance.yml — auto-merges Dependabot base-image PRs * ai-issue-summary.yml + teams-notifications.yml — triage + notifications * semantic-release config, expanded dependabot (actions/pip/docker/compose) and CODEOWNERS
A full docs/ tree plus copy-and-adapt example configs for every common use case, so the image is self-documenting. * docs/: configuration (layers, secrets, full schema), sources, destinations, retention, notifications (incl. webhook HMAC verification), encryption, cli, restore (disaster-recovery walkthrough), deployment (meta-Dockerfile pattern), plugins (extension API), migration (fleet adoption plan) * examples/config/: 10 ready-to-adapt configs (postgres, mariadb+files, mysql, s3 mirror, multi-source bundle, multi-job, encrypted, GFS, all-notifications) — all validated against the config schema * README reworked into a docs hub with a feature overview and quick start
Brought the deployment examples up to the standard used across the stack
(header, x-logging anchor, healthchecks, resource limits, profiles).
* docker-compose.yml — complete standalone example (Postgres + uploads →
local + off-site S3) driven by an inline BACKUP_CONFIG_JSON; secrets kept out
of the rendered file via doubled $${VAR} placeholders
* docker-compose.sidecar.yml — attach-to-an-existing-stack example that feeds
the config through a Compose configs: block mounted as BACKUP_CONFIG_FILE
* .env.example expanded to cover identity, schedule, retention, S3 target,
encryption, notifications, logging and resource limits
Both compose files validate with `docker compose config` and their rendered
inline JSON parses against the config schema.
This is a Linux-container project (image, shell, Python, config all run on Linux), so LF is mandatory. `.gitattributes` with `eol=lf` makes Git store AND check out LF regardless of a contributor's core.autocrlf, which also removes the "LF will be replaced by CRLF" warnings on Windows. A matching `.editorconfig` reinforces LF + UTF-8 + final-newline at the editor level.
The image name did not follow the fleet convention (bauer-group/CS-<Repo>/ <component> for GHCR, as in cs-iamstack/database-backup). Corrected across the workflow and every reference. * docker-release.yml: ghcr-image-name → bauer-group/CS-BackupHelper/backuphelper (docker-image-name stays bauergroup/backuphelper) * pull references (compose, .env.example, docs, examples) → ghcr.io/bauer-group/cs-backuphelper/backuphelper * Dockerfile image.source label → github.com/bauer-group/CS-BackupHelper
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 34616905 | Triggered | Generic Password | 0d7f88c | tests/config/test_interpolation.py | View secret |
| 34616905 | Triggered | Generic Password | e0d5f12 | tests/sources/test_postgres.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Restore matched a snapshot's components to job sources by name, computing the
expected name as "<name> or <database> or 'database'" — but the postgres source
named its component just "<name> or 'database'", ignoring the database value.
So a job like {"database": "app"} produced a component named "database" that
restore looked for under "app" and skipped ("no source config for component").
Aligned postgres with the mariadb/mysql sources: the component now defaults to
the database name (e.g. "app.dump"), matching how restore resolves it. Caught
by an end-to-end restore roundtrip against a live PostgreSQL.
docker-compose.development.yml stands up a self-contained stack for local development and end-to-end testing: PostgreSQL (the source), the BAUER GROUP MinIO S3 server as the off-site target, minio-init provisioning a `backups` bucket + a scoped service account from an inline JSON config (no host file), and BackupHelper built from ./Dockerfile snapshotting the DB + an uploads dir to local /data AND the MinIO bucket. Verified end to end: backup → archive + sidecar land in both local and the MinIO bucket (scoped service account), verify OK, and a drop-table → restore roundtrip brings the row back. Everything has dev defaults so it runs with no .env file.
The Validate Docker Compose job failed with "Required service 'backup' not found". The shared modules-validate-compose workflow runs `docker compose config`, which by design omits services gated behind a Compose profile. Both compose files declare the backup service as profiles:[backup] (on-demand), so it never appears in the rendered config and could never satisfy validate-services: [database, backup]. * Validate only docker-compose.sidecar.yml — the shipped deliverable of this repo; the standalone example's database is test-only. * Dropped the merge of the two independent example files (they are alternatives, not overlays: the sidecar uses an external network and a configs: block, so `-f a -f b` produced a nonsensical merge). * validate-services set to [] — a render/syntax check, since the profile-gated backup service cannot be asserted until the shared workflow activates profiles. Unblocks the release + docker-build pipeline on this branch.
A DB restore fed the client the *compressed* dump: passing a gzip file object as subprocess stdin (`run(stdin=gzip.open(...))`) hands the child the raw file descriptor, so it reads the gzip magic — not the decompressed SQL — and errors with `ASCII '\0' appeared in the statement` / a garbled query. pg_restore's custom-format path dodged it, which is why only mariadb/mysql (and the postgres plain-SQL path) were affected. Now the .sql.gz is streamed to a real temp file first, whose fd carries the decompressed SQL. A regression guard asserts the client is fed a real file, not a GzipFile. Found by the end-to-end restore roundtrips.
docker-compose.e2e.yml + scripts/e2e.sh run a real backup -> (local + MinIO S3) -> restore roundtrip against every source engine (postgres, mariadb, mysql, filesystem, s3-bucket-source), seeding data, destroying it, restoring and asserting — 16/16 green. MinIO is provisioned from inline JSON (bucket + scoped service account), matching the fleet's minio-init pattern. The matrix surfaced a real limitation now documented in docs/sources.md: the Alpine mariadb-client ships no caching_sha2_password client plugin, so it cannot authenticate to a default MySQL 8/9 — a mysql_native_password backup user or the Oracle mysql-client (meta-layer) is required. MariaDB is unaffected. It also asserts finished:success so an errored snapshot can no longer masquerade as a passing backup.
The unit-test placeholder password "s3cret" tripped GitGuardian's generic password detector (a false positive — it is a throwaway test fixture, never a real credential). Replaced it with the canonical "changeme" placeholder so the secret scanner stops flagging it.
Contributor
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces BackupHelper — one GHCR image that replaces the fleet of
individually-maintained backup sidecars. It is a strict superset of the existing
containers: pluggable sources → S3/local destinations, with a sha256 manifest,
retention, notifications, optional encryption and a full restore CLI.
Design principle: the core knows how to move bytes safely; each consuming
repo keeps a ~20-line meta-Dockerfile and owns its app-specific logic via a
Source plugin or lifecycle hook — the engine never imports an app SDK.
What's included
S3 buckets (mirrored with per-object metadata/tags/content-type), filesystem
path-groups, env whitelist — combinable into one atomic snapshot.
equal-chunk multipart uploader (MinIO/Ceph/R2/B2/Wasabi/Garage). Policy: S3
when configured, else local; `keep_local` drops the local copy after a clean
off-site upload.
Discord, ntfy/Gotify, healthchecks.io — severity-gated, per-channel fault
isolation.
or a mounted file — with `${VAR}` secret interpolation. Multi-job.
secret-redacting logs, plugin/hook extension API.
Quality gates
assembled unless the suite passes. Verified: the image builds and a container
smoke test (backup → list → verify → restore, non-root) passes.
validate with `docker compose config`.
CI/CD
Adopts the standard BAUER GROUP `automation-templates` pipeline (as in
CS-Outline): validate-compose + pytest + semantic-release + multi-arch
GHCR/Docker Hub build, daily base-image monitor, Dependabot auto-merge,
AI issue triage and Teams notifications. Image:
`ghcr.io/bauer-group/cs-backuphelper/backuphelper`.
Docs
Comprehensive `docs/` (configuration, sources, destinations, retention,
notifications, encryption, cli, restore, deployment, plugins, migration),
10 ready-to-adapt example configs, and two professional compose examples.
`docs/migration.md` contains the per-repo fleet migration plan.
Not in this PR (follow-ups)
against real Postgres/MariaDB before production restores.
Phase 3.