question of the coinco processed format

[gmichalo]

Hello,

Thank you for sharing the version of the two datasets that you used.

However, I have a question about the CoINCo dataset.

Whenever the target word is "." or "," it seems that you have updated the target sentence where you have duplicated the previous word
for example:
..N 14925 18 until thirty seconds ago , i didn n't believe in magic or any of that kind of ...weirdness . "
you have updated to
..N 14925 18 until thirty seconds ago , i didn n't believe in magic or any of that kind of ...weirdness weirdness "

could you let me know why you have done this update?

[orenmel]

Thanks for pointing this out. This seems to be a bug. Looking at the original coinco dataset, the origin of this seems to be in the coinco xml. See for example how it lists 'weirdness' as the wordform of '.' <token id="XXX" wordform="or" lemma="or" posMASC="XXX" posTT="CC" /> <token id="XXX" wordform="any" lemma="any" posMASC="XXX" posTT="DT" /> <token id="XXX" wordform="of" lemma="of" posMASC="XXX" posTT="IN" /> <token id="XXX" wordform="that" lemma="that" posMASC="XXX" posTT="DT" /> <token id="14924" wordform="kind" lemma="kind" posMASC="NN" posTT="NN" problematic="no" > <substitutions> <subst lemma="sort" pos="NN" freq="4" /> <subst lemma="type" pos="NN" freq="2" /> <subst lemma="variety" pos="NN" freq="1" /> </substitutions> </token> <token id="XXX" wordform="of" lemma="of" posMASC="XXX" posTT="IN" /> <token id="XXX" wordform="…weirdness" lemma="&lt;unknown&gt;" posMASC="XXX" posTT="NN" /> <token id="14925" *wordform="weirdness" lemma="."* posMASC="NNS" posTT="SENT" problematic="no" > <substitutions> <subst lemma="." pos="SENT" freq="11" /> </substitutions> </token>

…

On Mon, May 3, 2021 at 11:31 AM gmichalo ***@***.***> wrote: Hello, Thank you for sharing the version of the two datasets that you used. However, I have a question about the CoINCo dataset. Whenever the target word is "." or "," it seems that you have updated the target sentence where you have duplicated the previous word for example: ..N 14925 18 until thirty seconds ago , i didn n't believe in magic or any of that kind of ...weirdness . " you have updated to ..N 14925 18 until thirty seconds ago , i didn n't believe in magic or any of that kind of ...weirdness weirdness " could you let me know why you have done this update? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFTIJOSRK3DLHKGMEKNL5TTL26UNANCNFSM44BAUUOA> .