Ingest a Google Takeout Gmail mbox export into Postgres for permanent local archival, search, and export, with a local web UI for browsing.
Roughly twenty years of mail — expect a few hundred thousand messages and tens
of gigabytes. Raw message bytes live in a content-addressed blob store on disk,
not in Postgres; the database holds derived metadata and the search index, so a
pg_dump stays small enough to be useful.
Status: early. Nothing here is finished. See "Project status" below for what actually works today.
This is a personal archival tool published in the open because there's no reason not to. It is not a product. There is no support, no release process, and no commitment to backwards compatibility — the schema, the CLI, and the storage layout may all change without migration paths. If you find it useful, fork it.
Everything runs in Docker, and everything can be exercised against generated fixture data — you do not need a real mbox export to run this.
git clone https://github.com/evanwtf/gmail-archive.git
cd gmail-archive
cp .env.example .env # then edit POSTGRES_PASSWORD
docker compose up -d
curl localhost:8000/healthz
curl localhost:8000/versionFor local development outside Docker:
uv sync
uv run pre-commit install # required: the hooks are what keep real mail out of git
uv run pytest
uv run gmail-archive versionTo produce input without a real export — this is what makes the rest of the project exercisable:
uv run gmail-archive gen-fixture /tmp/fixture.mbox --count 500 --seed 1
uv run gmail-archive gen-fixture /tmp/menu.mbox --pathologies list
uv run gmail-archive gen-fixture /tmp/nasty.mbox --count 40 \
--pathologies deep-nesting,body-nul,charset-nonexistent,bare-fromWith no --pathologies, the generator emits a realistic mix at the defect rates
measured against a real 20-year export. Naming pathologies instead produces each
one at least once, including several that no real export contains. --seed
is byte-reproducible.
Every generated address is confined to an RFC 2606 reserved domain
(example.com, .invalid, .test) by construction, and a test asserts that
against the generated bytes rather than against the call sites.
uv run pre-commit install is not optional housekeeping. This repository is
public, and the hooks reject staged .mbox files, anything under blobs/, and
oversized files. Without them there is nothing between a stray git add . and a
permanent public commit of real mail.
Work is tracked in three places, split by what each is good at:
| Where | What | Why there |
|---|---|---|
| GitHub issues | Live status — one issue per build phase, closed at its gate | open/closed is a real signal. A hand-edited status line goes stale silently |
| docs/plan.md | The scoped specification, all phases | Reference material, versioned with the code it describes and readable with no network. Splitting one coherent spec across nine issues would fragment it |
| docs/progress.md | What was built, how to verify it, and findings worth keeping | Archaeology outlives a tracker |
So: issues say where we are, plan.md says what we are building, progress.md
says what we learned. A phase issue links to its plan.md section and lists
acceptance criteria only — it never restates the spec, so there is one place to
change when the design moves. Phases are grouped into two milestones: Prototype
(Phases 2–5), specified seriously, and Directional (Phases 6–10), where the
shape is right but the details are guesses.
Completed phases are also tagged (git tag -n), so a tag checkout gives a
reviewed working state.
Working today:
- Docker image (Debian slim pinned to an exact Python patch release, runs as a non-root user) and a compose stack with a dedicated Postgres 18
/healthz,/readyz,/version;gmail-archive version,gmail-archive servegmail-archive gen-fixture— the synthetic mbox generator, with 26 individually selectable pathologies and a default mix weighted to measured rates
Not built yet: the parser, the schema and migrations, the ingest pipeline, search, export, and the web UI.
MIT — see LICENSE.