Amazon Notify

Amazon Notify watches Gmail for Amazon.co.jp delivery emails and posts them to Discord. It is designed for recovery-safe processing rather than maximum speed. A message is considered processed only after the notification succeeds, and catch-up always relies on Gmail inbox state. Target platform: Linux single-host deployment, systemd-first operations.

Two operating modes are supported:

• polling for single-host setups
• Gmail Watch + Pub/Sub StreamingPull for near-real-time triggering

Japanese README: README.ja.md

Version Track

Track	Description
main branch	Latest implementation (may be ahead of a release tag)
Latest release	Last tagged release on GitHub
Next intended release	0.6.0 (see CHANGELOG.md Unreleased)

Behavior Example

1. Message A is notified successfully -> checkpoint advances to A.
2. Message B notification fails -> run stops at B.
3. Checkpoint remains at A (no forward hole).
4. Next run resumes from B and continues oldest-first.

Highlights

Note: the main branch may be ahead of the latest GitHub Release.

• Ordered-frontier processing (oldest-first, stop on midstream failure)
• Checkpoint source of truth in events.jsonl, with state.json compatibility snapshots and runs.jsonl audit logs
• Rebuildable index snapshots (events.jsonl.checkpoint.index.json, runs.jsonl.summary.index.json) to keep startup/runtime-status reads fast on long-lived files
• Retry and recovery handling for transient Gmail/Discord failures
• Transient-failure alert boundary with persistence threshold and cooldown
• Unhandled guard-path exceptions are normalized into persisted RunResult/source_failed records
• Discord dedupe lock is fail-fast on non-fcntl platforms (also exposed via --health-check as dedupe_lock_supported)
• Realtime mode with Pub/Sub StreamingPull
• In-process StreamingPull self-healing:
  • trigger failure backoff/circuit-breaker
  • stream session reconnect backoff (before systemd restart)
• Hybrid HA mode:
  • Main: StreamingPull service
  • Fallback: timer-based polling with watchdog (systemd + heartbeat)
• systemd restart storm protection and OnFailure alert hook

Guarantees

• A message is treated as processed only after the notification path succeeds.
• Ordered frontier consistency is preserved (oldest-first, stop on midstream failure).
• Pub/Sub is treated as a trigger path; Gmail inbox state remains the catch-up source.

Non-goals

• Multi-instance distributed processing.
• Per-Pub/Sub-message durable workflow tracking.
• Generic mail forwarding platform.

Paths and working directory

Runtime paths (state_file, events_file, runs_file, log_file, and similar) are not tied to where you cloned the repository. They are resolved relative to the directory that contains config.json, and you can point at any config file with amazon-notify --config /path/to/config.json. Use absolute paths in config.json when you need to pin a location explicitly.

Quick Start

python3 -m venv .venv
source .venv/bin/activate
pip install .
cp config.example.json config.json
# use config.full.example.json if you need Pub/Sub / advanced retry knobs

1. Set discord_webhook_url in config.json
2. Place credentials.json next to config.json
3. Run amazon-notify --reauth and complete the browser OAuth flow when prompted.
4. Run one-shot verification:

amazon-notify --reauth
amazon-notify --once

Common Commands

# loop polling
amazon-notify

# single run
amazon-notify --once

# dry run (no Discord post, no checkpoint commit)
amazon-notify --once --dry-run

# realtime streaming pull
amazon-notify --streaming-pull --pubsub-subscription projects/PROJECT/subscriptions/SUB

# setup Gmail watch
amazon-notify --setup-watch --pubsub-topic projects/PROJECT/topics/TOPIC

# fallback watchdog single run
amazon-notify --once --fallback-watchdog

# rebuild index snapshots from current events/runs files
amazon-notify --rebuild-indexes

# one-shot operator summary (frontier/incident/failure/consistency)
amazon-notify --status

# detailed integrity diagnostics as JSON
amazon-notify --doctor

# audit append-only/state/index consistency (JSON)
amazon-notify --verify-state

# minimal operational metrics
amazon-notify --metrics
amazon-notify --metrics-plain --metrics-window 50

# archive / restore / restore drill
amazon-notify --archive-runtime --archive-label 20260412-000000
amazon-notify --restore-runtime --restore-label 20260412-000000
amazon-notify --restore-drill

# fault-injection scenario harness
amazon-notify --scenario-harness
amazon-notify --scenario-harness --scenario-names gmail_transient_failure,discord_429_retry

Runtime Artifacts

Source of truth:

• events.jsonl: authoritative checkpoint history (checkpoint_advanced events). Check this first when confirming frontier state.

Derived / compatibility:

• state.json: compatibility snapshot of the latest checkpoint boundary.
• runs.jsonl: per-run summary log for failure kind, counters, and before/after checkpoint.

Rebuildable cache:

• events.jsonl.checkpoint.index.json: cached pointer for fast checkpoint reads.
• runs.jsonl.summary.index.json: cached pointer for fast latest-run summary reads.
• If these index files are stale/corrupted, rebuild with amazon-notify --rebuild-indexes.

Coordination / lock:

• .state.json.lock: lock file used during state updates.
• .discord_dedupe_state.json: Discord notification dedupe coordination state.
• .discord_dedupe_state.lock: lock file for Discord dedupe coordination.

Logs:

• logs/: runtime logs (default file: logs/amazon_mail_notifier.log).

Try With Minimal Docker

docker build -t amazon-notify:slim .
docker run --rm amazon-notify:slim --help
docker run --rm -v "$(pwd):/work" amazon-notify:slim --config /work/config.json --validate-config
docker run --rm -v "$(pwd):/work" amazon-notify:slim --config /work/config.json --once --dry-run

Positioning:

• Primary production path: Linux single-host + systemd-first operations.
• Docker path: local evaluation and reproducible testing of the CLI/runtime boundary.

Scope note: this minimal image is for CLI bring-up only. It does not include systemd, hybrid HA/watchdog orchestration, multi-container operations, or production secret/monitoring design.

Health Check Notes

• amazon-notify --health-check includes dedupe_lock_supported.
• On platforms without fcntl, this check becomes false and dedupe lock paths are treated as unsupported (fail-fast).

Optional Dependencies

Pub/Sub mode:

pip install -e .[pubsub]

Development:

pip install -e .[dev]

Documentation

Run it:

• Hybrid quickstart (English): docs/HYBRID_QUICKSTART.en.md
• Hybrid quickstart (Japanese): docs/HYBRID_QUICKSTART_JA.md

Operate it:

• Operations runbook (English): docs/OPERATIONS.en.md
• Operations runbook (Japanese): docs/OPERATIONS.md
• Portability parameters (English): docs/PORTABILITY_PARAMS.en.md
• Portability parameters (Japanese): docs/PORTABILITY_PARAMS_JA.md

Understand design:

• Engineering decisions and design rationale (English): docs/engineering-decisions.en.md
• Engineering decisions and design rationale (Japanese): docs/engineering-decisions.md
• Implementation rationale (English): docs/IMPLEMENTATION_RATIONALE.en.md
• Implementation rationale (Japanese): docs/IMPLEMENTATION_RATIONALE_JA.md
• Hybrid architecture and failover design (English): docs/HYBRID_ARCHITECTURE.en.md
• Hybrid architecture and failover design (Japanese): docs/HYBRID_ARCHITECTURE_JA.md

Docker:

• Minimal Docker guide (English): docs/DOCKER.en.md

• Minimal Docker guide (Japanese): docs/DOCKER.md

• Japanese full README: README.ja.md

• Language policy: operations, Docker, and engineering-decision docs are maintained in both English (*.en.md) and Japanese (*.md). This README prioritizes English links and also includes Japanese counterparts.

• Optional structured logging (structured_logging=true) is supported.

Security

Do not commit local runtime files:

• credentials.json
• token.json
• config.json
• state.json
• events.jsonl
• events.jsonl.checkpoint.index.json
• runs.jsonl
• runs.jsonl.summary.index.json
• .state.json.lock
• .discord_dedupe_state.json
• .discord_dedupe_state.lock
• logs/

License

MIT License. See LICENSE.