Update to transformers 4.x

[david-waterworth]

I'd quite like to use this library to score the output from my RoBERTa model, but it's implemented with huggingface transformers version 4.x and this library requires 3.3.1 (and that also ended up installing tokenizers-0.8.1rc2 for some reason).

It would be nice if it could be upgraded to the latest verison.

[logan-markewich]

I agree. I started to try and hack in DistilRoberta, but it was more work than I thought and I don't have time lol

[ruanchaves]

You can just edit setup.py and replace transformers~=3.3.1 by transformers~=4.4.2 or simply by transformers and take the latest version. There should be no breaking changes.

[david-waterworth]

I tried that and there does appear to be breaking changes. Plus it only works for specific huggingface models. But it's not too hard to adapt to my own requirements.