"Create a job queue whose workers fetch data from a URL and store the results in a database. The job queue should expose a REST API for adding jobs and checking their status / results."
- Implements topic-based in-memory queue
- Runs a worker to perform web-scraping
- Exposes POST API to request website to be scraped
- Caches websites in DB for future lookups
- Exposes GET API endpoint to check status of request and see result
- Port and Topic configurable with
.envfile (copy.env.example)
- in-memory-queue - Queue Service
- LokiJS - Simple NoSQL DB
Install the dependencies with
npm installStart the server with
node index.js4 things will happen when you do this:
- API server will start at http://127.0.0.1:8080 by default
- The In-Memory Queue will be created
- A topic (webscrapers by default) will be created
- A single worker will spin up, subscribing to that topic
To get started, make a POST request to the root of the API at http://127.0.0.1:8080
The body should be a simple JSON object with a url property
{
"url": "http://www.google.com"
}NOTES:
- A protocol (http:// or https://) is required.
www.google.comwill return a422 Unprocessible Entitybecause that is not a valid URL and this application will not attempt to guess which protocol you meant to use.
If the URL you are requesting has not already been scraped in your session, your request will be added to a queue and processed.
In this case, your response object will look something like this:
{
"links": {
"status": "http://127.0.0.1:8080/IG6PSLt6M"
},
"data": {
"queueId": "IG6PSLt6M",
"url": "http://www.google.com/",
"processed": false,
"hasError": null,
"html": null,
"meta": {
"revision": 0,
"created": 1552974578712,
"version": 0
},
"$loki": 1
}
}If processed is false the request has not been completed. In this case, you can use the link.status URL to check the status of the request.
To check the status of a request, use the queueId property returned from your request, and make a GET request to http://127.0.0.1:8080/:queueId
NOTES:
- The LokiJS DB is not being persisted by default, so the database is wiped each time the server is started.
If the URL you are requesting has already been scraped in your session or you make a GET request to the links.status URL and the process has completed, you will see a response that looks like this:
{
"links": {
"status": "http://127.0.0.1:8080/IG6PSLt6M"
},
"data": {
"queueId": "IG6PSLt6M",
"url": "http://www.google.com/",
"processed": true,
"hasError": false,
"html": "<html><head><title>Google</title>...</html>",
"meta": {
"revision": 1,
"created": 1552974578712,
"version": 0,
"updated": 1552974579924
},
"$loki": 3queueId, url, and links.status will not change.
The following properties that will differ between an in-queue response and a processed response:
| Property | In-Queue Value | Processed Value | Description |
|---|---|---|---|
| processed | false |
true |
Has the queue been processed |
| hasError | null |
true or false |
Was there an error during the scrape |
| html | null |
string |
Raw HTML string from scrape |
- This was created in an evening, as a coding challenge. As such, I would not recommend this for any real use.
- While Redis is extremely popular for cases such as this, I chose to build a solution that did not depend on Redis.