Skip to content
BranchesTags

Repository files navigation

VQQP-Bench

This is the code for paper: "Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation"

Abstract

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a key paradigm for grounding MLLMs with external knowledge. While query pre-processing (e.g., rewriting) is standard in text-based RAG, existing MRAG pipelines predominantly treat visual inputs as static and immutable, implicitly assuming they are noise-free. However, real-world visual queries are often ''imperfect''---suffering from geometric distortions, quality degradation, or semantic ambiguity---leading to catastrophic retrieval failures. To address this gap, we propose V-QPP-Bench, the first comprehensive benchmark dedicated to Visual Query Pre-processing (V-QPP). We formulate V-QPP as an agentic decision-making task where MLLMs must autonomously diagnose imperfections and deploy perceptual tools to refine queries. Our extensive evaluation across 46,700 imperfect queries and diverse MRAG paradigms reveals three critical insights: (1) Vulnerability---visual imperfections severely degrade both retrieval recall and end-to-end MRAG performance; (2) Restoration Potential & Bottleneck---while oracle preprocessing recovers near-perfect performance, off-the-shelf MLLMs struggle with tool selection and parameter prediction without specialized training; and (3) Training Enhancement---supervised fine-tuning enables compact models to achieve comparable or superior performance to larger proprietary models, demonstrating the benchmark's value for developing robust MRAG systems.

Environment Setup

Due to the use of diverse embedding models, VLMs (Vision-Language Models), and image processing tools—each with distinct and often incompatible environment requirements—we explicitly specify the required environment for every Python script. We also provide official installation guides for each model and reference environment files. All environment configuration files are stored in the ./requirements directory, with each TXT file named after its corresponding environment. For example, the reference environment file for GroundingDINO is located at ./requirements/GroundingDINO.txt.

Env: embedding

This environment is primarily used for computing embeddings, building the vector database, and performing retrieval operations. It is applicable to get_embedding.py and retrieval_service.py. Note that only NomicEmbedding (with model_name="nomic") and ClipEmbedding (with model_name="clip") are supported in this environment. Both models can map images and text into a shared embedding space.

Installation:

pip install -r ./requirements/embedding.txt

Env: lavis

This environment is designed for using the BLIP-2 model to generate captions for images. It is applicable to BlipCaption in get_embedding.py when model_name="blip". The model converts images into descriptive captions.

Installation:
Refer to the official installation guide at https://github.com/salesforce/LAVIS, or use the provided environment file:

pip install -r ./requirements/lavis.txt

Env: gme

This environment is dedicated to computing embeddings with the GME model, applicable to GmeEmbedding in get_embedding.py when model_name="gme". The model supports mapping both images and text into a unified embedding space, and additionally enables hybrid embedding—jointly embedding image-text pairs into the feature space.

Installation:

pip install -r ./requirements/gme.txt

Env: GroundingDINO

This environment is required for running the main pipeline in pipeline.py. It provides support for the GroundingDINO model used in visual grounding tasks.

Installation:
First, follow the official installation instructions at https://github.com/IDEA-Research/GroundingDINO. Then, install additional dependencies via:

pip install opencv-python openai tqdm

or simply:

pip install -r ./requirements/GroundingDINO.txt

To use GroundingDINO, you need to download its pre-trained weights and place them in the ./configs/ directory:

cd ./configs
wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
mv groundingdino_swint_ogc.pth groundingdino.pth
cd ..

Env: sglang

This environment is used to run the SGLang runtime for invoking various Vision-Language Models (VLMs). Please refer to the official installation guide at https://docs.sglang.io/get_started/install.html.

A reference environment file is provided at ./requirements/sglang.txt.

About the data

Our dataset is derived from the MMLongBench dataset (https://github.com/EdinburghNLP/MMLongBench) and the ViQuAE dataset (https://github.com/PaulLerner/ViQuAE). The code used to process and obtain the data will be made publicly available. Below is a description of our data structure:

infoseek

infoseek.json stores data extracted from MMLongBench. It is formatted as a list, where each item represents a single data sample:

[
  {
    "infoseek_id": "infoseek_val_00003201",
    "question": "What country does this lake belong to?",
    "answer": [
      "Confoederatio Helvetica",
      "Schweiz",
      "Swiss Confederation",
      "Svizzera",
      "Switzerland",
      "Suisse",
      "Swiss"
    ],
    "entity_id": "Q14512",
    "entity_text": "Lake Sils",
    "gold_context": [
      {
        "doc_id": "chunk_0",
        "text": "Lake Sils (German \"Silsersee\", Romansh: \"Lej da Segl\") is a lake in the Upper Engadine valley, Grisons, Switzerland. It takes its name from the village of Sils im Engadin.",
        "title": "Lake Sils"
      }
    ],
    "all_context": [...],
    "gold_index": [676]
  }
]

Field descriptions:

  • infoseek_id: Unique identifier from the InfoSeek dataset (e.g., infoseek_val_00003201)
  • question: The query text presented to the model
  • answer: A list of acceptable ground-truth answers; any entry in the list is considered correct
  • entity_id: Wikidata ID of the entity depicted in the associated image
  • entity_text: Name or description of the entity depicted in the image
  • gold_context: List of relevant context passage(s) that support the answer. Each item follows the format:
    {"doc_id": document_id, "title": document_title, "text": text_content}
  • all_context: Complete set of retrieved context passages for this sample, sharing the same format as gold_context
  • gold_index: List of indices indicating the positions of gold context passages within all_context

Due to the large file size, you can download infoseek.json from Google Drive link, then unzip and place it in the root directory of this project.

viquae

viquae.json stores data extracted from ViQuAE. It is formatted as a list, where each item represents a single data sample:

[
  {
    "kilt_id": "qw_15291",
    "question": "In 1938 he wrote, produced, and narrated a radio play adaptation of what work?",
    "answer": [
      "The War of the Worlds by H.G.Wells",
      "war of worlds by h g wells"
    ],
    "entity_id": "Q24829",
    "gold_context": [
      {
        'chunk_id': 6495500,
        'passage_id': 1005625,
        'base_id': 1,
        'title': 'Orson Welles',
        'text': 'of the Worlds\" by H. G. Wells October 30, 1938, brought Welles instant fame. The combination of the news bulletin form of the performance with the between-breaks dial spinning habits of listeners was later reported to have created widespread confusion among listeners who failed to hear the introduction, although the extent of this confusion has come into question. Panic was reportedly spread among listeners who believed the fictional news reports of a Martian invasion. The myth of the result created by the combination was reported as fact around the world and disparagingly mentioned by Adolf Hitler in a public speech.'
      },
      ...
    ],
    "gold_index": [1005625]
  }
]

Field descriptions:

  • kilt_id: Unique identifier from the ViQuAE dataset (e.g., qw_15291)
  • question: The query text presented to the model
  • answer: A list of acceptable ground-truth answers; any entry in the list is considered correct
  • entity_id: Wikidata ID of the entity depicted in the associated image
  • gold_context: List of relevant context passage(s) that support the answer. Each item follows the format:
    {"chunk_id": chunked_id, "passage_id": passage_id, "base_id": base_id, "title": document_title, "text": text_content}
  • gold_index: List of indices indicating the positions of gold context passages within the corpus

viquae_wiki

While the corpus for the InfoSeek dataset is embedded within the all_context field of infoseek.json, the ViQuAE corpus is stored separately in viquae_wiki.json. This file serves as a standalone knowledge base containing 1,495,352 entries, where each entry uniquely corresponds to an image.

Example entry:

[
  {
    "title": "A",
    "text": "A (named , plural \"As\", \"A's\", \"a\"s, \"a's\" or \"aes\") is the first letter and the first vowel of the modern English alphabet and the ISO basic Latin alphabet. It is similar to the Ancient Greek letter alpha, from which it derives. The uppercase version consists of the two slanting sides of a triangle, crossed in the middle by a horizontal bar. The lowercase version can be written in two forms: the double-storey a and single-storey ɑ. The latter is commonly used in handwriting and fonts based on it, especially fonts intended to be read by children, and is",
    "image": "512px-Latin_alphabet_Aa.svg.png"
  }
]

Field descriptions:

  • title: Title of the Wikipedia article
  • text: Text content excerpted from the Wikipedia article
  • image: Filename of the associated image (e.g., "512px-Latin_alphabet_Aa.svg.png")

Due to the large file size, you can download viquae_wiki.json from Google Drive link, then unzip and place it in the root directory of this project.

process_infomation

For each dataset, we preserve image processing parameters used during augmentation. These parameters are stored in process_information_infoseek.json and process_information_viquae.json. Each JSON file is a list where every element corresponds to the processing configuration of a single sample. Below is an example:

[
  {
    "rot": 180,
    "flip": 0,
    "lum": 0.5,
    "blur": 27,
    "gauss": 0.2,
    "crop": [
      0.5,
      [
        220,
        49,
        620,
        274
      ]
    ],
    "expand": {
      "position": "lower_right",
      "expand_image_path": [
        "infoseek/oven_05010518.jpg",
        "infoseek/oven_05067860.JPEG",
        "infoseek/oven_05015117.jpg"
      ],
      "img_size": [
        800,
        450
      ]
    },
    "over": {
      "factor": 2,
      "position": "bottom-left"
    },
    "water": {
      "factor": 1.0
    },
    "real": {
      "mode": 4,
      "text_prompt": "The image on the left screen"
    }
  }
]

Parameter descriptions:

  • rot: Rotation angle in degrees (e.g., 180°)
  • flip: Flip mode (0 = no flip, 1 = horizontal flip, 2 = vertical flip, etc.)
  • lum: Luminance adjustment coefficient; each pixel value is multiplied by this factor to modify brightness
  • blur: Kernel size for Gaussian blur operation
  • gauss: Intensity of additive Gaussian noise
  • crop: [crop_ratio, [top_left_x, top_left_y, bottom_right_x, bottom_right_y]] — defines the cropping region and ratio
  • expand:
    • position: Placement location for expanded content (e.g., "lower_right")
    • expand_image_path: List of image paths used for expansion
    • img_size: Original image dimensions [width, height]
  • over:
    • factor: Scaling factor for the overlay image size
    • position: Placement position of the overlay (e.g., "bottom-left")
  • water:
    • factor: Font size or scaling factor for watermark text
  • real:
    • mode: Realism enhancement mode identifier
    • text_prompt: Natural language description indicating the position or content of the original image within a composite layout

Images

All images are stored under the ./image/ directory with the following structure:

  • ./image/viquae_wiki/
    Contains all images used to construct the ViQuAE knowledge base. You can download the complete image collection from:
    https://huggingface.co/datasets/PaulLerner/viquae_all_images
    After downloading, extract all images into the ./image/viquae_wiki/ folder.

  • ./image/viquae/ and ./image/infoseek/
    Store the processed images for our constructed datasets. For each sample indexed by i, its images are organized under:
    ./image/infoseek/{i}/ (or ./image/viquae/{i}/)

    Within each sample folder:

    • {op}.jpg represents the image after applying a specific augmentation operation, where {op} can be one of:

      • ori: original image
      • rot: rotation
      • flip: flipping
      • lum: luminance adjustment
      • blur: Gaussian blur
      • gauss: Gaussian noise addition
      • crop: cropping
      • expand: image expansion
      • over: overlay
      • water: watermarking
      • real: realism enhancement

    • reverse_{op}.jpg represents the image restored from the corresponding augmentation operation under the oracle processing pipeline.

Due to the large file size, you can download all images for infoseek from Google Drive link, then unzip and place it in ./image/infoseek. All images for viquae are in Hugging Face, you can download then unzip and place it in ./image/viquae folder.

Examples and illustrate

get_embedding.py

This script is used to build embedding databases, compute embeddings, and perform retrieval operations. It supports multiple embedding models and various retrieval modes.

Parameters:

  • --model_name (required): The embedding model to use. Options:
    • nomic: NomicEmbedding model (supports image and text encoding)
    • clip: CLIP model (supports image and text encoding)
    • gme: GME model (supports image, text, and fusion encoding)
    • blip: BLIP-2 model (used for generating image captions)
  • --mode (required): The operation mode. Options:
    • encode_text: Build text library embeddings (only for infoseek dataset)
    • image2text: Perform image-to-text retrieval (only for infoseek dataset)
    • caption_image: Generate image captions using caption model (only for infoseek dataset)
    • text2text: Retrieve text using captions (only for infoseek dataset)
    • encode_image: Build image library embeddings (only for viquae dataset)
    • image2image: Perform image-to-image retrieval (for viquae dataset)
    • encode_fusion: Build fusion embeddings (only for viquae dataset, requires gme model)
    • fusion2fusion: Perform fusion-to-fusion retrieval (only for viquae dataset, requires gme model)
  • --dataset_name (required): Dataset name (infoseek or viquae)
  • --batch_size (optional, default=8): Batch size for processing
  • --k (optional, default=100): Number of top-k results to save
  • --image_path (optional, default=None): Custom image storage path. If None, defaults to ./image/{dataset_name}

Usage Examples:

  1. Build text embeddings for InfoSeek dataset (using nomic model):
conda activate embedding
python get_embedding.py --model_name nomic --mode encode_text --dataset_name infoseek
  1. Perform image-to-text retrieval on InfoSeek dataset:
conda activate embedding
python get_embedding.py --model_name nomic --mode image2text --dataset_name infoseek --k 100
  1. Build image embeddings for ViQuAE dataset:
conda activate embedding
python get_embedding.py --model_name nomic --mode encode_image --dataset_name viquae
  1. Perform image-to-image retrieval on ViQuAE dataset:
conda activate embedding
python get_embedding.py --model_name nomic --mode image2image --dataset_name viquae --k 100
  1. Generate image captions using BLIP-2:
conda activate lavis
python get_embedding.py --model_name blip --mode caption_image --dataset_name infoseek
  1. Perform text-to-text retrieval using captions:
conda activate embedding
python get_embedding.py --model_name nomic --mode text2text --dataset_name infoseek --k 100
  1. Build fusion embeddings for ViQuAE dataset (using GME model):
conda activate gme
python get_embedding.py --model_name gme --mode encode_fusion --dataset_name viquae
  1. Perform fusion-to-fusion retrieval on ViQuAE dataset:
conda activate gme
python get_embedding.py --model_name gme --mode fusion2fusion --dataset_name viquae --k 100

Output:

  • Embeddings are saved in ./embedding/{dataset_name}/{type}/{model_name}/ directory (for encode_text mode, saved as {i}.pkl files)
  • Retrieval results are saved in ./retrieval_results/{dataset_name}/{mode}/{model_name}.json
  • Image captions are saved in {image_path}/caption_{model_name}.json

Notes:

  • The script supports incremental updates: if an embedding file already exists, it will be skipped
  • Image operations are defined in the all_operations list in the code, including: ori, rot, flip, lum, blur, gauss, crop, expand, over, water, real, and their reverse_ variants
  • Default image storage location: ./image/{dataset_name}/{i}/{op}.jpg where {i} is the sample index and {op} is the operation name

retrieval_service.py

This script provides a Flask-based retrieval service that can be invoked by the main pipeline. It supports image-to-text retrieval using pre-computed text embeddings. The service supports both NomicEmbedding and ClipEmbedding models, with NomicEmbedding as the default.

Parameters:

  • --port (optional, default=5000): Port number for the Flask service
  • --host (optional, default='0.0.0.0'): Host address for the service
  • --dataset (required): Path to the dataset JSON file (e.g., infoseek.json)
  • --embedding_dir (required): Directory containing pre-computed text embeddings (e.g., ./embedding/infoseek/text/nomic/)
  • --device (optional, default='cuda'): Computing device (cuda or cpu)
  • --batch_size (optional, default=32): Batch size for processing

Usage Example:

  1. Start the retrieval service for InfoSeek dataset (using nomic embeddings, default):
conda activate embedding
python retrieval_service.py \
    --port 5000 \
    --dataset infoseek.json \
    --embedding_dir ./embedding/infoseek/text/nomic/ \
    --device cuda \
    --batch_size 32

API Endpoints:

  • POST /retrieve: Perform image-to-text retrieval

    • Request body (JSON):
      • image_path (optional): Path to the query image file
      • image (optional): Base64-encoded image string
      • top_k (optional, default=5): Number of top-k results to return
    • Response (JSON):
      • success: Boolean indicating success or failure
      • results: List of retrieved context strings in format "title: {title}\ntext: {text}"
      • count: Number of results returned
      • error (if failed): Error message
      • traceback (if failed): Error traceback

    Example request:

    {
  "image_path": "./image/infoseek/0/ori.jpg",
  "top_k": 5
}

    Example response:

    {
  "success": true,
  "results": [
    "title: Lake Sils\ntext: Lake Sils (German \"Silsersee\", Romansh: \"Lej da Segl\") is a lake in the Upper Engadine valley, Grisons, Switzerland...",
    ...
  ],
  "count": 5
}

  • GET /health: Health check endpoint

    • Response: {"status": "healthy"}

Notes:

  • Currently only supports Image->Text retrieval paradigm
  • The service extracts the sample index from the image path (assuming format .../idx/xxx.jpg)
  • Text embeddings must be pre-computed using get_embedding.py and stored as {idx}.pkl files in the embedding directory
  • The service supports both image file paths and base64-encoded images
  • When using base64 images, they are temporarily saved to disk and automatically deleted after processing

pipeline.py

This script implements the main Visual RAG Pipeline, which performs agentic visual query pre-processing, retrieval, and answer generation. The pipeline consists of three main steps: (1) image preprocessing decision and execution, (2) retrieval decision and execution, and (3) answer generation with context. The pipeline supports multiple operation modes for different evaluation scenarios.

Parameters:

  • --model_name (required): VLM model name. For open-source models deployed with sglang, use the model name (e.g., Qwen/Qwen3-VL-4B-Instruct). For closed-source models like GPT-4V, use the model identifier (e.g., gpt-4-vision-preview). Note: For closed-source models, you need to set the API key in the code.
  • --result_path (required): Path to store intermediate results and final outputs (e.g., pipeline_result/qwen_4B)
  • --dataset_name (optional, default='infoseek'): Dataset name (infoseek or viquae)
  • --port (optional): Port number for VLM service (required for open-source models deployed with sglang, not needed for closed-source models)
  • --retrieval_port (optional, default=5000): Port number for retrieval service. If not provided or set to None, retrieval will be skipped
  • --start_idx (optional, default=0): Starting index for testing samples, useful for parallelization
  • --end_idx (optional, default=1128): Ending index for testing samples, useful for parallelization
  • --force_run (optional, default=False): If True, force re-run even if result files exist. If False, skip samples that already have complete results
  • --mode (optional, default=""): Operation mode. Available options:
    • llm: Directly use LLM to answer questions without retrieval or image processing
    • gold: Use gold context (ground truth context) to let LLM answer questions
    • pipeline: Normal pipeline execution, including step1 (image preprocessing decision and execution), step2 (retrieval decision and execution), and step3 (answer generation)
    • no_process: Skip step1 (image preprocessing), execute step2 and step3 only. Used for testing reverse operations and no-processing performance
    • only_step_1: Only execute step1 (image preprocessing decision and execution)
    • step_2_3: Execute step2 and step3 only. This split allows using SFT model for step1 and original model for steps 2-3

Usage Example:

The pipeline requires two services to be running before execution: the SGLang service (for VLM) and the retrieval service. Here is a complete example:

Step 1: Start SGLang Service (for VLM):

conda activate sglang
clear

CUDA_VISIBLE_DEVICES=0,1 \
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-4B-Instruct \
  --tp-size 2 \
  --cuda-graph-max-bs 16 \
  --port 30000 \
  --mem-fraction-static 0.4 \
  --max-running-requests 8

Step 2: Start Retrieval Service:

conda activate embedding
clear

python retrieval_service.py \
    --port 5000 \
    --dataset infoseek.json \
    --embedding_dir ./embedding/infoseek/text/nomic/ \
    --device cuda \
    --batch_size 32

Step 3: Run the Pipeline:

conda activate GroundingDINO
clear

CUDA_VISIBLE_DEVICES=2 python pipeline.py \
    --model_name "Qwen/Qwen3-VL-4B-Instruct" \
    --result_path "pipeline_result/qwen_4B" \
    --dataset_name infoseek \
    --port 30000 \
    --retrieval_port 5000 \
    --mode pipeline \
    --start_idx 0 \
    --end_idx 1128 \
    --force_run False

Output:

  • For each sample {idx} and operation {op}, results are stored in {result_path}/{idx}/processed_{op}.json
  • Processed images (if any preprocessing was performed) are saved as {result_path}/{idx}/processed_{op}.jpg
  • The JSON file contains:
    • question: The input question
    • operations_performed: List of image preprocessing operations executed (if any)
    • retrieval_used: Boolean indicating whether retrieval was used
    • context: Retrieved context passages (if retrieval was performed)
    • answer: Final answer generated by the model
    • force_rag_ans: Answer generated with forced retrieval (always retrieved regardless of decision)

Notes:

  • The pipeline tests all operations defined in the code: ori, rot, flip, lum, blur, gauss, crop, expand, over, water, real
  • For real and expand operations, the question is automatically modified based on information in process_information_{dataset_name}.json
  • The pipeline supports incremental execution: if a result file exists and contains complete results, it will be skipped unless --force_run True is set
  • Image preprocessing uses GroundingDINO for object localization (requires model weights in ./configs/groundingdino.pth)
  • The pipeline automatically decides whether image preprocessing and retrieval are needed based on the VLM's analysis of the image and question

eval_result.py

This script evaluates the performance of the pipeline results across three different dimensions: exact match accuracy, retrieval performance, and tool invocation correctness. It supports comprehensive evaluation of all image processing operations and their reverse operations.

Parameters:

  • --dataset_name (optional, default='infoseek'): Dataset name (infoseek or viquae)
  • --storage_path (required): Path to the pipeline results directory (e.g., pipeline_result/qwen_4B)
  • --mode (required): Evaluation mode. Options:
    • EM: Sub Exact Match evaluation of final answers. Checks if the generated answer contains any of the ground-truth answers (case-insensitive substring matching)
    • retrieval: Retrieval performance evaluation. Measures whether gold context appears in the top-k retrieved results (k=1, 3, 5)
    • tool: Tool invocation correctness evaluation. Assesses whether the correct tools are called with correct parameters for image preprocessing operations

Usage Examples:

  1. Evaluate Exact Match (EM) accuracy:
conda activate GroundingDINO
python eval_result.py --dataset_name infoseek --storage_path pipeline_result/qwen_4B --mode EM
  1. Evaluate retrieval performance:
conda activate GroundingDINO
python eval_result.py --dataset_name infoseek --storage_path pipeline_result/qwen_4B --mode retrieval
  1. Evaluate tool invocation correctness:
conda activate GroundingDINO
python eval_result.py --dataset_name infoseek --storage_path pipeline_result/qwen_4B --mode tool

Output:

  • EM mode: Prints a table with columns:

    • Operation: Operation name (e.g., ori, rot, flip, etc.)
    • Result: Exact match accuracy for normal answers
    • Force_result: Exact match accuracy for forced retrieval answers (always retrieved regardless of decision)

  • retrieval mode: Prints a table with columns:

    • Operation: Operation name
    • top-1, top-3, top-5: Recall@k metrics indicating whether gold context appears in top-k retrieved results

  • tool mode: Prints a table with columns:

    • Operation: Operation name
    • Strict Result: Strict evaluation - only correct tool called, no other tools (ignores parameters)
    • Loose Result: Loose evaluation - correct tool called at least once (ignores parameters)
    • Error Parameter: Number of cases where correct tool is called but parameters are incorrect and cannot be executed
    • Parameter Right(Only have): Average parameter correctness, only considering cases where tool was correctly invoked
    • Parameter Right(All): Average parameter correctness across all samples (wrong tool invocation defaults to 0)

Evaluated Operations:

The script evaluates all operations defined in the codebase:

  • Base operations: llm, gold, ori, rot, flip, lum, blur, gauss, crop, expand, over, water, real
  • Reverse operations: reverse_rot, reverse_flip, reverse_lum, reverse_blur, reverse_gauss, reverse_crop, reverse_expand, reverse_over, reverse_water, reverse_real
  • Processed operations: processed_ori, processed_rot, processed_flip, processed_lum, processed_blur, processed_gauss, processed_crop, processed_expand, processed_over, processed_water, processed_real

Notes:

  • The script expects result files in format: {storage_path}/{sample_idx}/{operation}.json
  • For tool evaluation, the script uses GroundingDINO (via pipeline) to compute bounding boxes when locate tool is used
  • Tool evaluation includes parameter validation for rotation angles, flip directions, luminance factors, and bounding box IoU calculations
  • The script automatically loads ground truth information from process_information_{dataset_name}.json for parameter validation
  • For bounding box evaluation (expand, over, water, real operations), IoU (Intersection over Union) is calculated between predicted and ground truth bounding boxes

Structure of the files

The directory and structure of the files are as follows:

|-- configs/                          # GroundingDINO configuration files
|   |-- GroundingDINO_config.py      # Model configuration
|   |-- groundingdino.pth            # Pre-trained model weights
|
|-- requirements/                     # Environment dependency files
|   |-- embedding.txt
|   |-- sglang.txt
|   |-- gme.txt
|   |-- lavis.txt
|   |-- GroundingDINO.txt
|
|-- image/                            # All images
|   |-- viquae_wiki/                 # ViQuAE knowledge base images
|   |   |-- 512px-Latin_alphabet_Aa.svg.png
|   |   └── ...
|   |
|   |-- infoseek/                    # Processed InfoSeek samples
|   |   |-- 0/
|   |   |   |-- ori.jpg
|   |   |   |-- rot.jpg
|   |   |   └── ...
|   |   └── ...
|   |
|   └── viquae/                      # Processed ViQuAE samples
|       |-- 0/
|       |   |-- ori.jpg
|       |   |-- rot.jpg
|       |   └── ...
|       └── ...
|
|-- Readme.md                        # Project documentation
|-- infoseek.json                    # InfoSeek dataset
|-- viquae.json                      # ViQuAE dataset
|-- viquae_wiki.json                 # ViQuAE corpus (1,495,352 entries)
|-- process_information_infoseek.json  # Image processing parameters for InfoSeek
|-- process_information_viquae.json    # Image processing parameters for ViQuAE
|-- get_embedding.py                 # Build database, compute embeddings, and perform retrieval
|-- pipeline.py                      # Main inference pipeline
|-- retrieval_service.py             # Retrieval service invoked by the pipeline
└── eval_result.py                   # Evaluation scripts

License

Copyright © Michigan State University. All rights reserved. This software is made available for non-commercial research, educational, and academic purposes. Users may download, use, and modify the software for non-commercial purposes provided that this copyright notice is retained.

Commercial use, redistribution for commercial purposes, sublicensing, or incorporation into commercial products requires prior written permission from Michigan State University.

For licensing inquiries, please contact MSU Technologies at ipdocs@msu.edu. Reference TEC22026-0154.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

5 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages