CLIMB

A contest winning Content Localization and Intelligent Multimedia Retrieval system.

Figure 1: Competition scores. We are Group 10.

---

A content-based video retrieval system designed for searching short video segments, focusing on Known-Item Search (KIS) and Visual Question Answering (VQA) tasks inspired by the Video Browser Showdown (VBS). The system provides an intuitive graphical user interface for interactive video exploration and integrates with the Distributed Retrieval Evaluation Server (DRES) through its REST API, enabling seamless submission of retrieved video segments.

Getting Started - User

Run CLIMB for the first time

In order to run CLIMB, you will need to make sure that the following things are fulfilled:

• You will need to store all the Videos in dataset/web_ready/
• There will need to be all the keyframes in dataset/keyframes/ with a naming scheme of <video_id>_shot_<shot_num>_kf_<shot_num>.jpg
• (To see how to do extract the keyframes see the "Keyframe extraction" section under "Getting Started - Developer")
• Make sure there is a Database Dump with the embeddings stored under /dataset/climb_db.dump
• (If you don't have a db-dump you will need to embed everything yourself, for that follow the "Getting Started - Developer" Section)
• Verify that there is a .env file in the root directory having all the necessary Parameters (listed in Section " Environment Variables")

If all that is done, you can run setup_climb.sh (found in the root directory)

./setup_climb.sh

Once you are set up, simply run start_climb.sh (also found in the root directory)

./start_climb.sh

Restart CLIMB

If you already setup climb you can always restart it using the shell scripts

./start_climb.sh

If we don't support your terminal emulator, and you don't want to have all processes running in the same one, you can

./start_climb_using_conda.sh

but for that you will need to use conda, otherwise you are fine with plain python.

To setup the exact same conda environment used in this shell script, move to the video_processing folder and run

conda env create -n climb -f conda_environment.yml

How To use CLIMB

Using CLIMB is as simple as climbing a ladder ;)

When opening up the climb website, you will see a pretty empty screen. Don't be scared it's all how it's supposed to be:

If you are in a competition I can recommend 2 things:

• sabotage your opponents
• connect to DRES in order to be able to submit frames to the server. In order to connect to dres, fill out the login field in the header.

If you don't want to search for something specific, but want to just browse through all the videos select the browsing tab on the top right. We will dynamically load more videos on scrolling and cache the Videos loaded for faster retrieval.

If you click on a video you can see all the scenes. If you click one it will open up in the right bar.

Under the main video screen you cann see all keyframes to scroll through and a large submit to DRES button if you want to submit the current scene. Additionally there is an "ask VQA" field where you can ask the backend easy questions like the color of the protagonists shirt.

Going back to the search tab, you can search for video scenes including specific content.

Under the "submit to DRES" button, you can find a "find similar" button which will instead of asking the backend for scenes including your queries will search for scenes similar to the one you clicked earlier.

Well and that's it.

Environment Variables

In order for CLIMB to work correctly you will need to create a .env file in the root directory.

And example file is provided here (please change the password):

POSTGRES_DB_NAME=CLIMB_DB
POSTGRES_PASSWORD=password
SEARCH_ENGINE_URL=localhost
SEARCH_ENGINE_PORT=5000
REDIS_URL=redis://localhost:6379
VIDEOS_CACHE_TTL_SECONDS=30
DB_PORT=5432
DB_HOST=localhost
BACKEND_URL=localhost
BACKEND_PORT=8000
FRONTEND_PORT=3000
ALLOWED_ORIGIN_REGEX=^https?:\/\/(?:[a-zA-Z0-9-]+\.)*q1studios\.at(?::\d+)?$

---

Getting Started - Developer

Project Structure

.env                        # Environment variables and secrets
start_climb.sh                # Launch ClIMB
setup_climb.sh                # Setup CLIMB
start_climb_using_conda.sh    # Launch CLIMB using Conda
backend/
    openapi.yaml            # API specification
    package.json            # Backend dependencies
    server.js               # Express API server
    controller/             # Route handlers
    models/                 # Database models and queries
    routes/                 # Express routes

dataset/                    # local folder only
    climb_db_backup.dump    # Database dump
    compression.checkpoint  # Compression checkpoint metadata
    keyframes/              # Extracted keyframes
    V3C1_200/               # Source video dataset
    web_ready/              # Compressed videos for web playback

frontend/
    index.html              # Application shell
    package.json            # Frontend dependencies
    vite.config.js          # Vite dev server config
    public/                 # Static assets
    src/
        App.jsx             # Main application component
        App.css             # Styles
        main.jsx            # React entry point
        components/
            SearchBar.jsx   # Search input with history
            ResultsGrid.jsx # Thumbnail grid of results
            VideoPlayer.jsx # Video player with segment loop
            ShotBrowser.jsx # Filmstrip navigation
            VideoBrowser.jsx# Browse all videos
            VqaAnswer.jsx   # VQA text answer input
            TaskTimer.jsx   # 5-minute countdown
            SubmissionLog.jsx # Submission history log

video_processing/
    requirements.txt           # python dependencies        
    src/
        config.py              # Settings
        custom_logger.py       # Logging utilities
        dataset_compression.py # Dataset compression helpers
        db_queries.py          # Database queries
        db_setup.py            # Database setup
        embeddings_extraction.py # Feature extraction
        keyframe_extraction.py  # Keyframe extraction
        main.py                # Pipeline entry point
        search_engine.py       # Search and embedding service
        utils.py               # Utility functions
        vqa_engine.py          # VQA inference engine
        worker_http_endpoint.py # Search Engine HTTP interface 
    logs/                     # Log files (local only)
   
readme_images/              # Images displayed in readme

1. Video Processing

In order to get started you will first need to process the videos. Extract the keyframes, encode them and compress them down to decrease loading time in the frontend. To do that go into the video processing part of CLIMB by running

cd video_processing

1.1 Installation

Run

pip install -r requirements.txt

to install neccessary requirements.

In case you prefer using a conda environment run

conda env create -n climb -f conda_environment.yml

All steps can be done by running main.py with the respective options. In order to have the correct relative paths please run

cd src

to step into the src folder.

Available arguments

You can control different stages of CLIMB using the following flags:

python main.py [OPTIONS]

Flag	Long option	Description
-c	--compress	Compress the dataset videos using FFmpeg to allow for efficient video retrieval
-spc	--showshowPostgresCommand	Create and show the command to create and start the postgres database
-ek	--extractKeyframes	Extract the Keyframes to store and embed, also updates the Database
-ekndb	--extractKeyframesNoDatabase	Keyframe extraction without editing the Database
-ee	--extractEmbeddings	Embed the Images and store the vectors in the Database
-start	--startSearchEngine	Start the Webserver which embeds user Queries and answers VQA Questions.
-h	--help	Shows how to use the CLIMB-CLI and exits

python main.py --help

1.2 Data Preprocessing

Download and extract your Dataset (i.e. from: "https://www2.itec.aau.at/owncloud/index.php/s/AcA1pvZIpDrOom5"). Save it to /dataset/V3C1_200 also extract the scenes and put them under /dataset/V3C1_200/scenes_v3c1_200. If you would like to choose a different Dataset / Folder structure edit the respective parameters in /video_processing/src/config.py

In order to allow for efficient browser based retrieval the vide sizes must be small. To compress the videos run

python main.py --compress

This will initiate a FFmpeg based compression of all the videos which will by default be stored under /dataset/web_ready. Please be sure that you have FFmpeg installed under your system as CLIMB will spawn a child-process executing FFmpeg. To download FFmpeg visit: https://ffmpeg.org/download.html

1.3 Keyframe extraction

Next you will need to extract the keyframes from the dataset. Additionally you will need to save the frame rate of every video to later be able to build the millisecond payload for the dres server. For that we will need to create simple postgres-database. Because every sane people hates it when postgres runs locally on your machine we will spin up a podman container for that. The parameters for the database can be found and edited in /video_processing/src/config.py. Sensitive information should be
stored in a .env file placed in the root directory of the project (CLIMB/). To automatically generate the podman command run

python main.py --showPostgresCommand

In order to create a Podman Container running Postgres just run the command in you shell. This will automatically fetch the postgres image, build the container and start it in the background. To stop the container just run

podman stop climb

To restart the container run

podman start climb

Always start the container before running any video_processing / frontend or backend otherwise CLIMB won't function properly.

(Note: Other usefull commands include podman ps to see all running containers and podman logs climb to see the logs if you stumble upon undesired behaviour. For more details however I will recommend their excellent documentation found under https://docs.podman.io/en/latest/)

Now that you set up the Database, you can populate it. In order to do so, run

python main.py --extractKeyframes

This will extract the keyframes out of the videos, save the Screenshots locally to later calculate the embeddings and will also insert them into the postgres climb Database. If you already got your Database set up (i.e. through a provided) dumb, but deleted the keyframes folder (I don't know why you would do that but still) you can run

python main.py --extractKeyframesNoDatabase

to still extract the keyframes, without updating the Database. (Please note you will still need an active database connection to do so)

1.4 Video Embedding

In order to later to semantic video retrieval, we will need to encode the Videos (Keyframes to be more specific) into a high Dimensional Vector Space (1024dim). By doing so, we can later encode your searches into the same space, and do semantic retrieval by performing nearest neighbour searches in this space. The setup is pretty easy. Just run

python main.py --extractEmbeddings

This will scan your climb database for video shots missing embeddings, extract the their features using SigLIP2, and store the vectors in the db. For more Information about SigLIP2 see: https://arxiv.org/pdf/2502.14786

1.5 Start the Search Engine

You are all set, now you can finally start the Search Engine which will open up a connection for the backend to connect to, to encode the searches and anser VQA-Questions. Just run

python main.py --startSearchEngine

and relax. By default the search engine will run locally on port 5000 but just as everything else, this is configurable in the config file.

Since the console will not be yours anymore I guess, so start up a new one and find your way to the root directory and start if the next Section.

2. Backend

To get the backend working you need to do 3 to 4 things.

• Spin up the DB-Container
• Optionally create and spin up the caching container
• Start the AI-Embedding Endpoint
• Start the backend server itself

First of all I hope you followed step 1 and properly setup everything. If so please return to the root directory in order to align the relative paths.

2.1 Spin up the DB-Container

As previously stated, to start the container run

podman start climb

2.2 Caching

In order to reduce load times during video browsing we added some paging and caching using Redis. It's importnat to note that CLIMB will run completely fine without any caching enabled but you might find that video browsing takes longer to load. If you want to activate it just create a new podman container

podman run --name climb_caching -p 6379:6379 -d docker.io/library/redis:7

and spin it up everytime you need some performance boost.

podman start climb_caching

Redis is configurable via the following parameters in your root environment file:

REDIS_URL=<url>:<port>
VIDEOS_CACHE_TTL_SECONDS=<time>

2.3 Start the search endpoint

In order to embed the user searches start the search engine by navigating into the 'video_processing/src' folder and running

python main.py --startSearchEngine

2.4 Start the backend server itself

Now you are all set. please open a new console if necessary and step into the backend folder, install all dependencies and start the backend.

cd backend
npm install
npm start

3. Frontend

Starting the frontend is even easier. All you need to do is to open a new terminal, navigate to the frontend directory, install all dependencies and run it.

cd frontend
npm install
npm run dev

To now see the User interface open the url provided in the terminal. By default it will be http://localhost:3000.

3.1 Backend API

If you are interested in creating your own frontend or are just interested in general, you can find the API Specifications of our backend under /backend/openapi.yaml. In order to properly view it I would recommend using an openapi viewer of your choice. JetBrains products typically have one included, browser based wise I like to use "https://editor.swagger.io/", but that's completely up to you.

THE END