In the field of Natural Language Processing (NLP), Transformer-based models have yielded appealing results in language understanding and generation. However, these models’ extremely high number of parameters and computationally expensive are challenging to deployment on resourceconstrained devices. This motivates research in the development of effcient methods that offer an attractive balance between model size and performance. This paper proposes a novel multi-task learning approach, which relies on a small language model like TinyBERT for NLP tasks, including: sentiment analysis, paraphrase detection, and semantic textual similarity. We perform initial training through a MultiTask Learning (MTL) mechanism to capture general language features. Subsequently, this pre-trained model is fine-tuned for each specific task. We conduct experiments to evaluate the effect of this approach on the GLUE benchmark dataset. The results demonstrate the effectiveness of our method within a multi-task learning setup for diverse NLP tasks.
(Link paper will be deployed)
@inproceedings{Le2025,
title={A Multi-Task Learning Approach Based on Small Language Model for Natural Language Processing Tasks},
author={Le, Xuan-Tuyen and Le, Quang-Hung},
year={2025}
}