L3Masking

New Masked Language Modeling for Auxiliary tasks

Overview

L3Masking is a new mlm as auxiliary task for effective multi-task fine-tuning. This project is based on the following paper:

Paper Title: L3Masking: Multi-task Fine-tuning for Language Models by Leveraging Lessons Learned from Vanilla Models
Poster: View
Presented at: Customizable NLP Workshop @ EMNLP 2024
Workshop Date: November 16, 2024
Location: Hyatt Regency Miami Hotel, Miami, Florida

Features

• Task and domain adaptation-specific masking
• Masked Token selection based on pseudo-likelihood.

LICENSE

This software includes modifications made by usk-Kimura.

Original repository: TARTAN

Modification date: 12/10/2024

Installation and Usage

1. Clone the repository:

   git clone https://github.com/usk-Kimura/L3Masking.git

2. Install the required dependencies:

   cd L3Masking 
   pip install -r requirements.txt

   CUDA 11.8, Ubuntu 20.04.4 LTS

3. Run the application:

   bash run_meta_multiple.sh