Repository navigation
Reproducing results from papers #21
Description
Activity
Hey @orionw
Thank you for your interest in our work!
Could you give us more information about how are you trying to replicate the results? (the dataset you are using, are you generating new synthetic data or using the data we made available, are you fine-tuning/evaluating using TPU/GPU,....)
And regarding the learning rate, we used 3e-4 (we are going to correct it).Moreover, we are about to release a reproduction paper of InPars with further details on how to reproduce the results.
Thank you!
Thanks for the reply @lhbonifacio!
I've tried a couple datasets (SciFact, SciDocs) but can't reproduce it. I'm using GPUs and the code in
inparsnot inlegacy. I am generating new questions using huggingface models (not the available InPars v1 questions and I haven't seen the InPars v2 generated questions publicly available).I've tried several learning rates (including 3e-4) and optimizers but for both of them any amount of re-ranker fine-tuning on the synthetic docs makes the performance worse than just using
castorini/monot5-3b-msmarco-10kwithout fine-tuning (and performance is worse than reported in the paper).If you have the fine-tuning hyperparameters for any of the BEIR runs that would be great (optimizer, learning rate, scheduler, steps, etc.).
Obviously with non-determinism there will be randomness in the generated questions and in training, but I was hoping to minimize differences due to model training.
Hi @orionw , I did found out the reason behind the performance drop and proposed an effective solution in my recent NAACL paper. You can find it here - https://arxiv.org/pdf/2404.02489.pdf
Reacted by dinosReacted by dinos
Hi there! Great work - this is a very interesting line of research!
I was hoping to replicate your results on BEIR but seem to be having some trouble. For example, in both InPars v1 and v2 papers you mention using a learning rate of 1e−3, but I can't find any example scripts that use that (in legacy or otherwise, they seem to use 3e-4). When I use the hyperparameters from the papers (or the default example), I am getting much worse results.
I'm sure it's just some config that I'm missing from reading the papers/code, but if you happen to have the commands that reproduce the numbers in the paper I'd really appreciate it!
Thanks for your time!