Objective
Skipgrams allow patterns to become easier to spot compared with ngram, as word sequences can contain noise.
The code needs the ability to extract skipgrams as features from a news story, as the same point where ngrams are extracted.
Overview of skipgrams and their benefit for trending news: https://gist.github.com/readikus/3fa5fce829adaafcbd9f95af4242f670
Requirements:
- Add parameters to take skip config (i.e.
max_skip_size). By default, don't use it (default: 0).
- Add functionality around here
|
for n in range(1, self.options["max_n"] + 1): |
that does the same for skipgrams.
Does this need to be stored in the ngram_history? Or a new object (i.e. skipgram_history). Reason: when working with results, we want to know that the phrase is a skipgram or an n-gram, as skipgrams may seem disjointed when displayed. However, this can be a big rebuild, so maybe out of scope for now.
Objective
Skipgrams allow patterns to become easier to spot compared with ngram, as word sequences can contain noise.
The code needs the ability to extract skipgrams as features from a news story, as the same point where ngrams are extracted.
Overview of skipgrams and their benefit for trending news: https://gist.github.com/readikus/3fa5fce829adaafcbd9f95af4242f670
Requirements:
max_skip_size). By default, don't use it (default: 0).royston/royston/royston.py
Line 274 in 45e74ef
Does this need to be stored in the
ngram_history? Or a new object (i.e.skipgram_history). Reason: when working with results, we want to know that the phrase is a skipgram or an n-gram, as skipgrams may seem disjointed when displayed. However, this can be a big rebuild, so maybe out of scope for now.