Identify a way of filtering data from Zenodo (based on keywords)

[dgarijo]

Related to #2, but based on additional information such as abstract (could be expanded with classifier)

[dgarijo]

According to what we discussed today, the current way of filtering is not enough. We agreed this should be the correct way (inspired by TF-IDF):

• Create a vector out of all keywords, from less frequent to more frequent. Each position will have a weight, which we can discuss.
• For each entry found, get the relevant keywords/extract keywords from abstract/description.
• For each entry, calculate how many keywords are matched in the vector of the first step. Multiply them by the keyword weight. Add all the resultant values and you will get a relevance score for the entry. This approach will match entities that have many keywords from the list, and will favor the ones that are less frequent.
• Then from the final list we will filter those elements from above a certain threshold (to be determined).

[dgarijo]

See #6 to coordinate.
We should have a baseline where we just count the keywords for every entry, and then the TF-IDF (we can see which one provides better results). @TZ199 should do 1 of the measures and @AlexRanzh the other one. Coordinate with @hardiksurana to verify results

[AlexRanzh]

Term frequency and TFIDF scores developed in /filterby_keywords/filter_TF-IDF.py with score results in filterby_keywords/kw_score_TF_TFIDF.csv.

[AlexRanzh]

• For each entry, calculate how many keywords are matched in the vector of the first step.

Do we consider keywords of multiple occurrence in calculating the relevance score for this entry.

For example, if the entry has abstract "Today is today. Tomorrow is another day. ", and keyword-weight is {"today": 300, "day": 200}, then the relevance score for this entry would be (300*2+200) or (300+200)?

[jelenamirkovic]

I would suggest the first approach.

…

On Tue, Mar 24, 2020 at 12:20 PM Alex Ran ***@***.***> wrote: - For each entry, calculate how many keywords are matched in the vector of the first step. Do we consider keywords of multiple occurrence in calculating the relevance score for this entry. For example, if the entry has abstract "Today is today. Tomorrow is another day. ", and keyword-weight is {"today": 300, "day": 200}, then the relevance score for this entry would be $300*2+200$ or $300+200$? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF4XKPLKYB245X2CIJXA4TRJEBWPANCNFSM4KVABTBQ> .

[AlexRanzh]

The relevance score is calculated for every object(artefact) in the database. The result is in json format here.

Threshold is to be determined.

[jelenamirkovic]

Can you draw a cdf of scores please?

…

On Tue, Mar 24, 2020 at 1:52 PM Alex Ran ***@***.***> wrote: The relevance score is calculated for every object(artefact) in the database. The result is in json format here <https://github.com/STEELISI/CKIDS/blob/master/filterby_keywords/filter_TFIDF_result.json> . Threshold is to be determined. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF4XKIQFQXSBHB6YAKESO3RJEMSNANCNFSM4KVABTBQ> .

[AlexRanzh]

The code CDF of relevance score has been added. The figure rlv_score_cdf.png is created.

[jelenamirkovic]

Does this mean that lots of documents have very low relevance score and a handful have high relevance? If so please prepare a sample of 20 random documents with low score so we can review them in the meeting.

…

On Tue, Mar 24, 2020 at 3:06 PM Alex Ran ***@***.***> wrote: The code CDF of relevance score has been added. The figure rlv_score_cdf.png <https://github.com/STEELISI/CKIDS/blob/master/filterby_keywords/rlv_score_cdf.png> is created. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF4XKKRKCWEHBZTRAGMRVTRJEVGDANCNFSM4KVABTBQ> .

[AlexRanzh]

Does this mean that lots of documents have very low relevance score and a handful have high relevance? If so please prepare a sample of 20 random documents with low score so we can review them in the meeting.
…
On Tue, Mar 24, 2020 at 3:06 PM Alex Ran @.***> wrote: The code CDF of relevance score has been added. The figure rlv_score_cdf.png https://github.com/STEELISI/CKIDS/blob/master/filterby_keywords/rlv_score_cdf.png is created. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF4XKKRKCWEHBZTRAGMRVTRJEVGDANCNFSM4KVABTBQ .

Please see examples here
https://github.com/STEELISI/CKIDS/blob/master/filterby_keywords/filter_by_TFIDF/samples_w_scores3.csv