Non Windows issues trying to run `run_tests.py`

[crzfub]

Non-Windows Issues

I have tried getting this to work on OSX Sonoma 14 and Linux (NixOS 24.11) with a fresh python 3.9.11 venv respectively.
And ran into two problems when running run_tests.py.

Both OS

The current requirement.txt had conflicts in itself on fresh venv's for me, so I used pip-compose to get a version combination that sort of worked.
I used this ultimately: https://github.com/crzfub/coptic-nlp/blob/dev/requirements.txt

OSX

Running run_tests.py would result in the following path issue:

python run_tests.py                                                                                   
! Foma flookup not found at ./bin/foma/
! Attempting to obtain from zip...! Foma flookup extracted to bin/foma/
o Testing configuration: -- -o pipes (tokenize into pipes)
Traceback (most recent call last):
  File "/coptic-nlp/gittest/coptic-nlp/run_tests.py", line 131, in <module>
    t.run_tests()
  File "/coptic-nlp/gittest/coptic-nlp/run_tests.py", line 86, in run_tests
    nlp_resp = nlp_coptic(all_inputs, do_tok=True,
  File "/coptic-nlp/gittest/coptic-nlp/coptic_nlp.py", line 786, in nlp_coptic
    preloaded = {"stk":None,"xrenner":None,"parser":NeuralParser.load(neural_model),"tagger":FlairTagger(),
  File "/coptic-nlp/gittest/coptic-nlp/lib/flair_pos_tagger.py", line 41, in __init__
    self.model = SequenceTagger.load(model_dir + lang + ".flair")
  File "/coptic-nlp/gittest/coptic-nlp/venv/lib/python3.9/site-packages/flair/models/sequence_tagger_model.py", line 925, in load
    return cast("SequenceTagger", super().load(model_path=model_path))
  File "/coptic-nlp/gittest/coptic-nlp/venv/lib/python3.9/site-packages/flair/nn/model.py", line 564, in load
    return cast("Classifier", super().load(model_path=model_path))
  File "/coptic-nlp/gittest/coptic-nlp/venv/lib/python3.9/site-packages/flair/nn/model.py", line 190, in load
    state = load_torch_state(model_file)
  File "/coptic-nlp/gittest/coptic-nlp/venv/lib/python3.9/site-packages/flair/file_utils.py", line 384, in load_torch_state
    return torch.load(f, map_location="cpu")
  File "/coptic-nlp/gittest/coptic-nlp/venv/lib/python3.9/site-packages/torch/serialization.py", line 1360, in load
    return _load(
  File "/coptic-nlp/gittest/coptic-nlp/venv/lib/python3.9/site-packages/torch/serialization.py", line 1848, in _load
    result = unpickler.load()
  File "/.pyenv/versions/3.9.19/lib/python3.9/pathlib.py", line 1084, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'WindowsPath' on your system

Linux

I installed foma did the compile_grammar.py but got stuck with the following exception. I also copied flookup.c to bin/foma/ however I am unsure, if that was wanted in the first place.
flookup is accessible from my CLI.

! Foma flookup not found at ./bin/foma/
! Attempting to obtain from zip...! Need to compile foma on Linux and place flookup in bin/foma/
! See bin/foma/README.md  

(That being said, I am on 0.10 instead of 0.9 of foma (as building turned out slightly painful) but given their changelog, this shouldn't be the issue at hand I guess?)

[amir-zeldes]

Thanks for reporting - I'd like to help, but it looks like both of the issues are internal to some dependencies outside of coptic-nlp proper... I'm positive coptic-nlp can and does run under Linux and Windows though, and I'm pretty sure the same is true for OSX, so there ought to be a way to get this all to work.

Regarding the first issue, I don't have a Mac to test/reproduce that error, but it seems to be an issue with some version of flair/torch, or maybe some incompatibility in model training details, rather than the coptic-nlp code itself. From the trace it looks like the system runs fine until the flair model is loaded, and then it doesn't like something about the model itself within torch/serialization.py. The fact that it works on Linux makes me wonder what's going on though... But again, I can't actually reproduce this - I think there are Mac users who have successfully used this library, so probably there is some version of torch/flair where this works?

FWIW the requirements you posted are pretty close to what models where trained under, which I believe is:

• flair==0.14.0 (same as you have)
• torch==2.2.1 (not quite the same but should be compat)
• transformers==4.38.2 (pretty close as well)

As for the second issue, you shouldn't need to compile foma on OSX since the package ships with a compiled zip that gets unzipped depending on the platform here:

https://github.com/CopticScriptorium/coptic-nlp/blob/master/coptic_nlp.py#L694

If you are using Linux, there is a chance the OSX binary will still work, depending on the flavor of Linux. For our Rocky Linux and CentOs servers I compiled this from source:

> flookup -v
flookup 1.03 (foma library version 0.9.18alpha)

So that seems to be working at least on a CentOS/RH style system. I'm not sure if 0.10 could be the issue here.

I installed foma did the compile_grammar.py but got stuck with the following exception

I wasn't able to see the exception, if you post it maybe I'll have an idea? Depending on what it is you can also try asking Mans Hulden, the developer of foma.