This repository was archived by the owner on Jun 23, 2026. It is now read-only.
Introduces Background Worker for Async Processing - #44
Merged
Merged
Conversation
Defines commands for the web server and background worker. Enables deployment and process management on compatible platforms.
Renames the primary application entry point to `web/main.go` to clarify its role as the web server. Adds a new `worker/main.go` to house background and asynchronous tasks, enabling a more robust and scalable architecture. Updates environment variable loading in the web component to directly use `os.Getenv`, reducing coupling with the Gin framework's mode detection.
Updates environment variable loading and retrieval functions to terminate the application with a fatal error if a variable is missing or loading fails. This change eliminates repetitive error handling in consuming code, ensuring critical configuration is present at application start-up.
Expands the list of HTML elements removed during content processing. This ensures a more focused and cleaner textual output by stripping out interactive, media, and code-related tags.
Updates module requirements by moving `rivershared` from a direct dependency to an indirect one. This reflects that the module is no longer directly imported but is still a transitive requirement.
Centralizes the `EPUBWorker` implementation in a dedicated package for improved reusability. Updates the River client initialization to accept an external `Workers` object, providing more flexible worker management. Changes EPUB file processing to read the entire file into a byte slice before enqueueing, simplifying data handling for workers. Introduces a 32MB file size limit for uploads. Adjusts River queue configurations for improved resilience and resource management, including setting max attempts and worker limits. Renames command directories for cleaner project structure.
Increases the timeout duration for individual batch operations to accommodate larger uploads. Ensures immediate context cancellation upon encountering errors during batch processing, improving resource cleanup. Adjusts inter-batch sleep duration and logic to reduce pressure on the Firestore API and optimize final batch completion.
Provides comprehensive documentation for the backend API endpoints, categorized into Client, Manage, and User groups. Clarifies setup and run instructions, emphasizing the need to run both web and worker processes. Updates technology stack details, including database changes.
Optimizes Firestore queries by consolidating the initial page load into a single request. Improves the accuracy of `nextCursor` determination and the chapter slicing process to ensure correct pagination. Streamlines error handling for better readability.
Ensures the River worker can shut down cleanly upon receiving termination signals. Listens for SIGINT and SIGTERM to initiate a controlled shutdown of the River client, allowing in-flight jobs to complete within a timeout period and preventing abrupt process termination.
Simplifies local development and testing by preventing CORS issues. Sets `AllowOrigins` to `*` when in debug mode or if the `DOMAIN` environment variable is not configured, allowing requests from any origin.
Ensures the wildcard domain (`*`) is only applied when the application is in debug mode and no specific domain is configured. This prevents unintended broad domain matching in production or non-debug environments.
Configures the Gin-based HTTP server with proper read, write, and idle timeouts.
Enables the server to handle OS signals (SIGINT, SIGTERM) to initiate a controlled shutdown.
Ensures ongoing requests can complete within a timeout before the server fully exits, improving reliability during restarts or deployments.
Removes a redundant `select {}` from the worker's main function.
Web/Worker dynos and Postgres database
Updates Procfile commands to execute pre-built binaries instead of directly running Go source files. Improves startup performance and aligns with production deployment practices.
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces background job processing for EPUB uploads, implements graceful shutdown for both web and worker processes, and enhances the application's configuration and reliability.
- Adds River-based background job processing for asynchronous EPUB uploads
- Implements graceful shutdown handling for both web server and worker processes
- Refactors environment variable handling and improves CORS configuration
Reviewed Changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| go.mod | Adds River queue and PostgreSQL driver dependencies |
| api/internal/usecases/worker/args.go | Defines EPUB processing worker and job arguments |
| api/internal/usecases/collections/novels.go | Refactors EPUB processing to accept byte array instead of multipart file |
| api/internal/interfaces/rest/server.go | Implements graceful shutdown with proper signal handling |
| api/internal/interfaces/rest/routes.go | Updates CORS configuration with environment-based domain |
| api/internal/interfaces/rest/handlers/novels.go | Converts EPUB upload to async processing with River queue |
| api/internal/infrastructure/collections/chapters.go | Improves cursor pagination logic and batch upload reliability |
| api/internal/common/env.go | Simplifies environment variable handling with fatal error on missing vars |
| api/internal/common/queue.go | Adds River client initialization utility |
| api/cmd/worker/main.go | New worker process entry point with graceful shutdown |
| api/cmd/web/main.go | New web server entry point |
| README.md | Updates documentation with new run instructions and API endpoints |
| Procfile | Defines both web and worker processes for deployment |
Comments suppressed due to low confidence (1)
api/internal/interfaces/rest/routes.go:1
- Typo in header name: 'Acces-Control-Allow-Origin' should be 'Access-Control-Allow-Origin' (missing 's').
package firestore_server
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| // }) | ||
| // return | ||
| // } | ||
|
|
There was a problem hiding this comment.
Remove commented-out code. Dead code should be deleted rather than commented out to maintain code cleanliness.
Suggested change
| workers := river.NewWorkers() | ||
| river.AddWorker(workers, &worker.EPUBWorker{}) | ||
| riverClient := cmn.InitializeRiverClient(ctx, workers) | ||
| _, err = riverClient.Insert(ctx, worker.ProcessEPUBArgs{File: fileData}, nil) |
There was a problem hiding this comment.
Creating a new River client and workers on every request is inefficient. Consider initializing the River client once at application startup and reusing it.
Suggested change
| _, err = riverClient.Insert(ctx, worker.ProcessEPUBArgs{File: fileData}, nil) | |
| // Use the pre-initialized workers and river client | |
| _, err = epubRiverClient.Insert(ctx, worker.ProcessEPUBArgs{File: fileData}, nil) |
| subset := chapters[i:min(i+chunkSize, len(chapters))] | ||
|
|
||
| batchCtx, cancel := context.WithTimeout(ctx, 300*time.Second) | ||
| batchCtx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) |
There was a problem hiding this comment.
Using context.Background() instead of the parent context breaks the cancellation chain. Use the passed ctx parameter instead.
Suggested change
| batchCtx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) | |
| batchCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) |
| env_variable := os.Getenv(v) | ||
| if env_variable == "" { | ||
| return "", &Error{Err: errors.New("Environmental Variable " + v + " Not Found"), Status: http.StatusNotFound} | ||
| log.Fatal(&Error{Err: errors.New("Environmental Variable " + v + " Not Found"), Status: http.StatusNotFound}) |
There was a problem hiding this comment.
log.Fatal expects a string or values that can be formatted, but &Error{} is a struct pointer. Use log.Fatal(err) or log.Fatalf with proper formatting.
| env_variable := os.Getenv(v) | ||
| if env_variable == "" { | ||
| return "", &Error{Err: errors.New("Environmental Variable " + v + " Not Found"), Status: http.StatusNotFound} | ||
| log.Fatal(&Error{Err: errors.New("Environmental Variable " + v + " Not Found"), Status: http.StatusNotFound}) |
There was a problem hiding this comment.
log.Fatal expects a string or values that can be formatted, but &Error{} is a struct pointer. Use log.Fatal(err) or log.Fatalf with proper formatting.
Ensures the River client is initialized only once using a `sync.Once` pattern. Registers essential workers, such as the EPUB processing worker, during the initial setup. Provides a centralized `GetRiverClient` function for controlled access to the client instance. Restructures the file to `river/client.go` for improved organization.
Centralizes River queue client and worker initialization into a dedicated package. Eliminates redundant setup logic across various application components, improving resource utilization and simplifying queue interactions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces a dedicated
workerprocess to handle asynchronous tasks, significantly improving the application's responsiveness.workerprocess using Riverqueue with a PostgreSQL backend.webserver and the newworkerprocess.README.mdwith comprehensive API documentation and revised run instructions for the new multi-process setup.