An annotated corpus and computational models for offensive language classification in tweets about the 2022 presidential elections.
Comparison of AutoML, BERT and LLM approaches for detecting offensive language and hate speech in tweets from the 2022 Brazilian presidential elections.
Social media has become a central space for contemporary political debate, enabling the rapid circulation of opinions, information and electoral strategies. This expansion came hand in hand with a sharp rise in offensive comments and hate speech — insults, racism, xenophobia and other forms of verbal violence that degrade public discourse and deepen political polarization.
Although incitement to hatred is a crime in Brazil, its enforcement on social platforms remains limited. Investigating the spread of online offensive language is therefore essential to understanding how interaction, power and conflict dynamics manifest in digital environments.
This project investigates the use of sentiment analysis (SA) techniques for the automatic classification of offensive language and hate speech in tweets about Brazilian politics. The guiding hypothesis is that offensive comments tend to express negative sentiments and emotions, and that the presence of negative terms and emotions can therefore support the classification of offensive language. The work was developed within the Interfaces research group (UFSCar), as part of a regular FAPESP project (2022/03090-0) aimed at the large-scale analysis of Brazilian political data.
Two classification tasks are addressed:
- Binary classification — whether a comment is offensive or not.
- Multi-label classification — which type(s) of offense a comment carries, including hate-speech categories.
The target corpus, ITED-BR, is a reference set of 3,400 tweets manually annotated for this research, drawn from the Interfaces Twitter Elections Dataset (Iasulaitis et al., 2025) — a collection of ~228 million tweets from the 2022 Brazilian presidential elections. The 3,400 tweets relate to the second round of the election (1,700 mentioning Bolsonaro and 1,700 mentioning Lula), published between October 2 and 30, 2022.
Annotation was carried out by nine researchers in two stages: first labeling each comment as offensive/non-offensive, then assigning one or more offense types to offensive comments. Inter-annotator agreement (Krippendorff's Alpha) reached 0.68 (Lula set) and 0.72 (Bolsonaro set).
Class distribution (ITED-BR). Offense-type percentages are computed over the offensive comments of each set; a comment may carry more than one type.
| Offense | Bolsonaro set | Lula set | Total |
|---|---|---|---|
| Insult | 1,188 (95.4%) | 1,036 (96.3%) | 2,224 (95.8%) |
| Ideology | 244 (19.6%) | 193 (17.9%) | 437 (18.8%) |
| Obscene | 106 (8.5%) | 70 (6.5%) | 176 (7.6%) |
| Others | 24 (1.9%) | 14 (1.3%) | 38 (1.6%) |
| Xenophobia | 12 (1.0%) | 5 (0.5%) | 17 (0.7%) |
| Sexism | 11 (0.9%) | 5 (0.5%) | 16 (0.7%) |
| LGBTQphobia | 8 (0.6%) | 0 (0.0%) | 8 (0.3%) |
| Racism | 3 (0.2%) | 1 (0.1%) | 4 (0.2%) |
| Offensive | 1,245 (73.2%) | 1,076 (63.3%) | 2,321 (68.3%) |
| Non-offensive | 455 (26.8%) | 624 (36.7%) | 1,079 (31.7%) |
Training corpora. Models are trained on public Portuguese corpora before being applied to ITED-BR. The binary models use Hate-BR (Vargas et al., 2022; 7k Instagram comments). The multi-label models use Unified-Hate, built by merging TOLD-BR, OLID-BR and Hate-BR under a common label scheme — 15,555 comments after de-duplication.
| Class | TOLD-BR | OLID-BR | Hate-BR | Total | Original classes |
|---|---|---|---|---|---|
| Insult | 4,371 | 5,589 | — | 9,864 (63.4%) | Insult |
| Obscene | 6,639 | 2,311 | — | 8,819 (56.7%) | Obscene, Profanity |
| Ideology | — | 1,221 | 496 | 1,704 (10.9%) | Ideology, Partyism |
| LGBTQphobia | 344 | 375 | 17 | 731 (4.7%) | LGBTQ+phobia, Homophobia |
| Racism | 138 | 92 | 8 | 236 (1.5%) | Racism |
| Sexism | 463 | 461 | 97 | 1,011 (6.5%) | Misogyny, Sexism |
| Xenophobia | 151 | 128 | 1 | 279 (1.8%) | Xenophobia |
Traditional machine-learning models trained with AutoGluon,
combined with sentiment-analysis features. To measure each feature's contribution, an
incremental evaluation is performed — starting from a feature-less baseline and adding
one attribute at a time, yielding six configurations:
Baseline → MOL → MOL+GE → MOL+GE+GEP → MOL+GE+GEP+Lex → MOL+GE+GEP+Lex+IP.
The features are:
- MOL — Multilingual Offensive Lexicon (Vargas et al., 2021): weighted score of offensive terms found in the text.
- GE — GoEmotions probability vector (28 emotions) from a BERTimbau model fine-tuned for emotion analysis (Hammes & Freitas, 2021).
- GEP — GoEmotions polarity: emotions above a 0.3 threshold mapped to +1/0/-1 (Assi et al., 2022) and summed.
- Lex — proportions of positive/negative/neutral terms using three sentiment lexicons (LIWC-Br, WordNetAffectBR, SentiLex-PT).
- IP — a single polarity index combining the three Lex proportions via logistic regression.
The BERTimbau Large Transformer (Souza et al., 2020), fine-tuned separately for the binary and multi-label tasks. Hyperparameters: batch size 32, 3 epochs, learning rate 2e-5, weight decay 0.01, warmup ratio 0.1, max sequence length 128, label smoothing 0.1, early-stopping patience 2.
A generative large language model trained specifically for Brazilian Portuguese (Maritaca AI). It is prompted with task descriptions; a few-shot strategy outperformed zero-shot on both tasks and was the one adopted.
The Perspective API (Jigsaw/Google), used only for the
binary task via its TOXICITY attribute. Comments scoring ≥ 0.4 (threshold defined
empirically) are classified as offensive.
Summary on the ITED-BR corpus (Macro-F). Full per-class metrics and the training-stage results are in results/README.md.
| Approach | Binary Macro-F | Multi-label Macro-F | Compute cost |
|---|---|---|---|
| BERTimbau | 0.75 | 0.28 | High (GPU fine-tuning) |
| Sabiá-3 (LLM) | 0.73 | 0.58 | API (paid inference) |
| AutoML — best config | 0.69 | 0.55 | Low (CPU) |
| AutoML — Baseline | 0.63 | 0.55 | Low (CPU) |
| Perspective API | 0.56 | — | API |
Key findings. In the binary task, the fine-tuned BERTimbau leads (0.75), followed closely by the Sabiá-3 LLM (0.73); the much simpler and cheaper AutoML models reach 0.69, trailing by only 4–6 points. In the multi-label task the LLM leads (0.58), with the AutoML Baseline right behind (0.55), while BERTimbau collapses to 0.28 — generalizing only on the majority insult class. Sentiment-analysis features helped most in the binary task (+6 points over baseline), supporting the project's initial hypothesis. Rare classes (racism, xenophobia) remained the hardest to detect across all approaches.
.
├── data/
│ ├── ited-br/ # target corpus: 3,400 annotated election tweets
│ │ ├── processed/ # cleaned ited-br.csv
│ │ └── features/ # mol, ge, gep, lex, ip feature tables
│ ├── hate-br/ # training corpus — binary task
│ ├── unified-hate/ # training corpus — multi-label task (TOLD+OLID+Hate)
│ ├── told-br/, olid-br/ # source corpora for Unified-Hate
│ └── lexicons/ # LIWC-Br, SentiLex-PT, WordNetAffectBR, MOL
├── notebooks/
│ ├── 1_analysis.ipynb # exploratory data analysis
│ ├── 2a_filtered_corpus.ipynb # filter & sample raw election tweets
│ ├── 2b_pre_processing.ipynb # clean/normalize all corpora
│ ├── 2c_unified_corpus.ipynb # merge into Unified-Hate
│ ├── 3a_feat_lex.ipynb # Lex feature
│ ├── 3b_feat_ge.ipynb # GoEmotions (GE) feature
│ ├── 3c_feat_gep.ipynb # GoEmotions polarity (GEP) feature
│ ├── 3d_feat_ip.ipynb # polarity index (IP) feature
│ ├── 3e_feat_mol.ipynb # MOL feature
│ ├── 4a_automl_binary.ipynb # train AutoML — binary (Hate-BR)
│ ├── 4a_automl_multilabel.ipynb # train AutoML — multi-label (Unified-Hate)
│ ├── 4b_bert.ipynb # fine-tune BERTimbau — both tasks
│ ├── 5a_automl_inference.ipynb # AutoML inference on ITED-BR
│ ├── 5b_bert_inference.ipynb # BERTimbau inference on ITED-BR
│ ├── 5c_sabia.ipynb # Sabiá-3 inference on ITED-BR
│ ├── 5d_perspective.ipynb # Perspective API inference on ITED-BR
│ └── others/ # auxiliary lexicon/feature ETL notebooks
├── src/
│ ├── pipeline/ # automl.py, bert.py, sabia.py, perspective.py
│ ├── models/ # automl.py (MultilabelPredictor), bert.py
│ └── utils/ # features, report, split, preprocessing, prompt, map, path
├── models/ # trained AutoGluon and BERTimbau models
├── results/ # detailed metrics (see results/README.md)
├── requirements.txt
└── README.md
git clone https://github.com/leomsfreitas/HateSpeech.git
cd HateSpeech
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtThe notebooks are numbered in execution order. A typical end-to-end run is:
- Pre-processing —
2b_pre_processing.ipynb, then2c_unified_corpus.ipynb. - Feature extraction —
3a_feat_lex→3b_feat_ge→3c_feat_gep→3d_feat_ip→3e_feat_mol. - Training —
4a_automl_binary,4a_automl_multilabel,4b_bert. - Inference on ITED-BR —
5a_automl_inference,5b_bert_inference,5c_sabia,5d_perspective.
The Sabiá-3 and Perspective API notebooks require API keys (Maritaca AI and Jigsaw/Google, respectively).
The trained models are too large to host on GitHub and are distributed separately. Only the fine-tuned BERTimbau models and the best-performing AutoML configuration per task are provided.
| Model | Task | Best config | ITED-BR Macro-F | Download |
|---|---|---|---|---|
| BERTimbau | Binary | — | 0.75 | Google Drive |
| BERTimbau | Multi-label | — | 0.28 | Google Drive |
| AutoML | Binary | MOL+GE | 0.69 | Google Drive |
| AutoML | Multi-label | Baseline | 0.55 | Google Drive |
@inproceedings{Freitas2026,
title = {An annotated corpus and computational models for offensive language classification in tweets about the 2022 presidential elections},
author = {Freitas, Leo Marques Sabino de and Seno, Eloize Rossi Marques and Iasulaitis, Sylvia},
booktitle = {Anais do XXIII Encontro Nacional de Intelig{\^e}ncia Artificial e Computacional (ENIAC 2026)},
year = {2026},
address = {Cuiab{\'a}, MT, Brasil},
publisher = {SBC},
url = {https://github.com/leomsfreitas/HateSpeech}
}This research was developed within the Interfaces research group (UFSCar) and the LALIC (Laboratório de Linguística Computacional) at UFSCar, as part of a regular FAPESP project (2022/03090-0). The author thanks PACTEC/IFSP for the research scholarship that made this work possible, and Maritaca AI for the credits granted for the use of the Sabiá-3 LLM.