tembi-challenge

Running from docker

1. To run the app from docker, make sure you have docker and docker-compose installed
2. Run docker-compose up --build from the root directory
3. The scraper service will run and once finished data can be accessed at http://localhost:5050/get-data

Running locally

1. Make sure you have python 3.14+ installed
2. Create and activate a virtual environment (optional but recommended)
3. Instal the required dependencies with pip install -r requirements.txt
4. Setup the database with python -m scraper.setup
5. Run the scraper with python -m scraper.main

Description

The architecture consists of a flask web server that exposes an endpoint to get the scraped data. The scraping is done using Selenium, and the data is stored in a PostgreSQL database using SQLAlchemy as the ORM.

The project structure clearly separates the different layers of the application:

• The scraper module contains the main scraping logic, including the ETL pipeline and "scheduling"
• The data_layer module contains the database setup and models, abstracting away the database interactions from the scraping logic
• The tests module should contains unit tests for the different components of the application

The scraper has been defined as a Class with methods (supposed) to generically execute subtasks during a scraping flow, i.e. navigating to the cart or detecting and closing pop ups. This way selenium details are abstracted away from the main scraping logic and future scraping flows can be implemented easier by inheriting from the base Scraper class, let's say when we want to create scrapers for certain types of websites that require specific handling.

The ETL pipeline is defined as scraping product data, cleaning the data, and storing it in the database. Each step is only concerned with its own responsibility, making the code modular and easier to maintain as well as testable. The higher modularity also allows to execute tasks in parallel in the future.

The database schema is designed to normalize the data and have it very structured. Product data and delivery options are stored in separate tables, with a many-to-many relationship between them. Thsi requires more joins when querying the data, but it ensures data integrity and reduces redundancy opposed to storing everything in a single table or document database.

The data transformation step extracts data such as currency, price, delivery providers and normalizes them into a consistent format before storing them in the database. Since there is not a lot of data being scraped I used one data transformation function, but for larger datasets or more complex transformations, it would be better to break this down into smaller, more focused functions with their own way of handling missing data, errors, etc. Those should also live separately in a data_transform module for better separation of concerns and modularity.

The docker setup includes a selenium standalone chrome container as the web driver for scraping, a postgres container for the database, and the scraper service itself. The scraper service depends on the selenium and database services, but the scraper can also be run locally against a database without the need of a selenium container, making development and testing easier.

Improvements for the future

Implement error handling and logging

Currently there is minimal error handling and no logging implemented besides the steps and data displayed by print statements to follow the progress. Logging also helps us figure out what is going on in the application, a minimal setup with sentry would be a good idea to receive error reports and statistics. ALternatively for larger scale, data dog has more capabilities for monitoring and logging.

In a production system, it is important to have robust error handling to manage unexpected situations, especially while scraping where finding and acting on the right elements and data can be hard. Implementing logging would also help in monitoring the application's behavior and diagnosing issues.

Implement an extensive test setup

Since python is a dynamically typed language, it is important to have a good test coverage to ensure code quality and catch bugs early. Setting up thresholds for code coverage and integrating tests into the CI/CD pipeline would be optimal for maintainablity.

Optimize scraping performance

Setting up parallel scraping flows for different flows/sites can significantly reduce scraping time. This can be done using multithreading or parallization. I would set it up with a worker pool to manage the scraping tasks and distribute them across multiple threads or processes. This setup even requires more logging and error handling to manage the complexity of parallel execution.

Save screenshot flows

During scraping, it would be helpful to save screenshots of the web pages at different stages of the scraping process. This can help in debugging issues with element selection or navigation, as well as providing a visual record of the scraping process.