Invalid Dataset Format

[NiteshMethani]

Hi,
Thanks for amaking your work open source.
I was trying to explore this repo and understand XOR limitations better but I couldn't execute the evaluate.py script even for a single dataset. For example, for conll2003 dataset, there seems to be an error in conll2003.py => read_data() function. Specifically at line 107 in conll2003.py.
Each line in the original dataset has 4 columns: token, WS, POS, NER tag. But the provided code splits it into 3 columns and hence the error.

Any suggestions on the expected dataset format to reproduce the results in the paper?

[jacobvsdanniel]

In each [dataset].py file, there is an extract_data() function to create files in the format needed by evaluate.py

cross-ner/conll2003.py

Line 173 in 7428a13

# extract_data()

[NiteshMethani]

Great. The data issues are resolved.
Now I am trying to find two files required in load_embedding() function "token.npy and vector.npy". The publicly available glove embeddings are generally in .txt format so trying to figure out these two formats.
Could you provide any leads?

Follow up- Could you pen down instructions to test this repo on custom NER dataset?

Thanks in advance,
Nitesh

[NiteshMethani]

Hi,
I was able to get the embedding files from ckiptagger repo.
The script ran but I couldn't understand the logs. It ran 400 epochs very quickly and gave 0.00 for all metrics.

Here is a part of the logs printed on the screen:

<Best Epoch 224>
[train] PPL=1.00 P=100.00 R=100.00 F1=100.00
[validate] PPL=2.32 P=100.00 R=50.00 F1=66.67
[test] P=0.00 R=0.00 F1=0.00 elapsed=0s

For reference, this is the command I ran:

python evaluate.py \
    -mode train \
    -dataset conll2003 \
    -split train \
    -predict_file predictions.txt \
    -model_name first_model

Any inputs/suggestions would be much appreciated.

[jacobvsdanniel]

For embedding

• Please prepare the token.npy and vector.npy files from GloVe or other embedding sources.
• See how conll2003.py reads the files
• Note that CkipTagger embedding is for Chinese, but datasets in this repo are in English.

For data

• We only provide the first 30 lines of raw data for each dataset due to copyright concerns
• Please acquire full datasets and do not use the tiny sample .head30 files

[NiteshMethani]

Awesome, it is working now. I missed that CkipTagger is having Chinese embedding vectors. I ran the evaluate.py file and so far its giving me 71% F1-score in 7 epochs on conll2003 dataset.

I will go through the paper in more detail now while training continues.
I have few questions though about the code base. Could you kindly provide the details/definition of the following:

• what is PPL? From code, I understand ppl=2**(loss/tokens) but what is the intuition behind it?
• What are bilstm and cnn arguments? What is the semantics behind bilstm = 2-100 and cnn = 1-20-2-20-3-20?
• What is the semantics of split and predict_file flag?
• I just ran evaluate.py but from the paper I see there are couple of more models like Att-BiLSTM and Cross-BiLSTM. Which model is being by default trained by evaluate.py and how do I switch between different models?

Some of my questions might get clear once I go through the experiments section of the paper but I just wanted to pen it down here for reference.

As a minor contribution,
I could write the instructions to run this code base and you can then maybe update it in the README.md file :)

Thanks,
Nitesh