Replace Redis with SQLite for storage and job queue#202
Merged
Conversation
Owner
dir01
commented
Feb 28, 2026
dir01
commented
- Storage: new storage/sqlite.go (NewSQLiteStorage) replaces Redis key-value store
- Queue: new service/jobs_queue/sqlq_queue.go using github.com/dir01/sqlq replaces taylorchu/work
- Uses modernc.org/sqlite (pure Go) to avoid CGO symbol conflicts with torrent library
- Env var SQLITE_DB_PATH replaces REDIS_URL (default: ./mediary.db)
- Single shared *sql.DB opened with WAL + busy_timeout pragmas
- Tests now use in-memory SQLite — no Docker/testcontainers needed for queue or storage tests
- Lint: local golangci-lint replaces Docker-based runner in Makefile and CI uses golangci-lint-action
- Storage: new storage/sqlite.go (NewSQLiteStorage) replaces Redis key-value store - Queue: new service/jobs_queue/sqlq_queue.go using github.com/dir01/sqlq replaces taylorchu/work - Uses modernc.org/sqlite (pure Go) to avoid CGO symbol conflicts with torrent library - Env var SQLITE_DB_PATH replaces REDIS_URL (default: ./mediary.db) - Single shared *sql.DB opened with WAL + busy_timeout pragmas - Tests now use in-memory SQLite — no Docker/testcontainers needed for queue or storage tests - Lint: local golangci-lint replaces Docker-based runner in Makefile and CI uses golangci-lint-action Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace do_lint_test.yml with do_lint.yml and do_test.yml so they run in parallel and report independently on PRs and master pushes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Propagate per-job context from sqlq through Subscribe handler so spans and timeouts from the queue flow into job processing - Change Subscribe handler signature to include context.Context - Fix Start() to call Run() before Subscribe() so the schema is initialized before consumers begin polling - Set 2-hour default job timeout (matching old taylorchu/work behavior) - Regenerate mocks after interface change - Update golden test fixtures to match current magnetURL Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of relying on external torrents from the internet, the tests now spin up a local anacrolix torrent client as a seeder and wire it into the downloader via AddBootstrapPeer. This makes metadata tests fast, reliable, and deterministic — no DHT or internet access required. - downloader/torrent: add AddBootstrapPeer + addMagnet helper that calls AddClientPeer for each registered seeder after AddMagnet - tests/local_seeder.go: SetupLocalSeeder helper creates a temp dir with provided files, builds metainfo, seeds it, and returns the canonical magnet URL (infohash only, no dn/tr params) - tests/e2e_test.go: replace the external magnetURL constant with a local seeder; use an unreachable infohash for the "timeout" subtest; move S3 setup to top-level with panic recovery so metadata subtests run without Docker - Update golden files to match new local test torrent Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dir01
force-pushed
the
feature/no-redis
branch
2 times, most recently
from
b55fc65 to
a982d40
Compare
- Add SetupLocalSeeder helper to seed fake torrent data locally in tests, eliminating reliance on external torrent availability - Add MakeMinimalMP3 helper (uses ffmpeg) to generate valid MP3 bytes for job creation tests; reduces job test timeout from 50m to 2m - Wire local seeders into the torrent downloader via AddBootstrapPeer for direct peer exchange (no DHT or trackers required) - Use a temp-file SQLite DB with WAL mode in e2e tests instead of in-memory (in-memory silently ignores WAL, causing write contention under sqlq's read transaction) - Fix concatenate_flow.go: use per-call statusCtx instead of a shared 1-second context that expires before later SaveJob calls complete - Handle Docker-unavailable panic from testcontainers at top level with recover(); S3-dependent subtests skip gracefully when s3Client is nil Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.