Changing the dataset

[neel-forwardedge]

I'm trying to tweak the dataset to use my own data for a use case, but the model keeps on pointing to the original dataset somehow. Do I have to clone the repo and upload the model to pip?

[MartinoMensio]

Hi @neel-forwardedge ,
Have you tried changing the DB_DEFAULT_PATH here?

spaCy-entity-linker/spacy_entity_linker/DatabaseConnection.py

Line 16 in d2f2473

DB_DEFAULT_PATH = os.path.abspath(os.path.join(__file__, "../../data_spacy_entity_linker/wikidb_filtered.db"))

At the moment, the download_knowledge_base method is downloading and unzipping the default file_url at

spaCy-entity-linker/spacy_entity_linker/__main__.py

Line 19 in d2f2473

file_url="https://huggingface.co/MartinoMensio/spaCy-entity-linker/resolve/main/knowledge_base.tar.gz"

You can choose to upload your model to any place that allows big files. We opted in #12 for Hugging Face LFS because it is free also for huge files. You can alternatively link to LFS in other places (e.g. GitHub) or in a online bucket somewhere else, but I think that uploading such a huge file to pypi is somewhat inconvenient.

Best,
Martino

[neel-forwardedge]

I changed the DB_DEFAULT_PATH. I set it to the new file that I created, but it's still running the old file somehow.