Fine-Tuning LLaMA3 8B for Image Captioning with CLIP and Flickr30k

This project demonstrates how to fine-tune the LLaMA3 8B transformer model for image captioning using the Flickr30k dataset. Images are encoded with the CLIP image encoder, and the model is trained to generate captions for images.

Project Structure

• download_images.py: Download and save test images from the Flickr30k dataset.
• fine_tune_llama.py: Fine-tune the LLaMA3 8B model using CLIP-encoded images and captions.
• inference.py: Run inference to generate captions for new images using the trained model.
• inference_cached.py: Inference with cached Q, K, V tensors for efficiency.
• inference_cached_topk_sampling.py: Inference with cached tensors and top-k sampling for more diverse captions.
• requirements.txt: Python dependencies.

Setup

1. Clone the repository and navigate to the project folder.
2. Install dependencies:

   pip install -r requirements.txt

3. Set your Hugging Face token (required for LLaMA3):
   • On Windows PowerShell:

     $env:HF_TOKEN="your_hf_token"

   • On Linux/macOS:

     export HF_TOKEN=your_hf_token

Download Test Images

Run the following to download and save test images from Flickr30k:

python download_images.py

Images will be saved in the test_images/ directory.

Fine-Tune the Model

To fine-tune LLaMA3 8B on the Flickr30k dataset:

python fine_tune_llama.py

• Training uses LoRA for parameter-efficient fine-tuning.
• Weights & Biases is used for experiment tracking (set your entity/project in the script if needed).

Inference

Standard Inference

Generate captions for images using the trained model:

python inference.py

Inference with Caching

For faster inference using cached Q, K, V tensors:

python inference_cached.py

Inference with Top-k Sampling

For more diverse captions using top-k sampling:

python inference_cached_topk_sampling.py

Notes

• Make sure you have enough GPU memory for LLaMA3 8B.
• Set the HF_TOKEN environment variable before running scripts that require model download.
• The model and adapters are saved in the trained_clip_llama directory after training.

Requirements

See requirements.txt for all dependencies:

• torch
• transformers
• datasets
• Pillow
• sentencepiece
• requests
• tqdm
• scikit-learn
• wandb
• peft

License

See LICENSE for details.