SEARCH command needs a limit option

[wanghenshui]

like matt said #50 , need a stop words, or a limit output, or rank option(need INDEX support)

[f-prime]

As I just asked on #50 should stop words be specified at the time of indexing or from within the config file?

In a future update results that are returned will be ranked based on some ranking algorithm (TD-IDF most likely) and sorted by most relevant results.

a limit argument is interesting as well for search. I will consider that.

[wanghenshui]

As I just asked on #50 should stop words be specified at the time of indexing or from within the config file?

Yeah, indexing with a StopTokenFile will be easy to extend.

In a future update results that are returned will be ranked based on some ranking algorithm (TD-IDF most likely) and sorted by most relevant results.

My point to rank option is more like Redis's ZSET, score, with user's defined function handler to customize. TD-IDF is also a good first step.

also, SEARCH with limit count to avoid the Redis's KEYS problem.

[f-prime]

Interesting idea with the StopTokenFile. I like that. As I mentioned in #50 we could specify a stop token list from the config file.

Interesting idea allowing users to define their own ranking algorithm.. Maybe allow them to do that as an option? How would that work exactly though? Seems to me it'd be easier to just include a choice of ranking algorithm on Fist's end.

[wanghenshui]

Seems to me it'd be easier to just include a choice of ranking algorithm on Fist's end

Yeah, typically one dataset use one ranking algorithm, like tweets, INDEX tweets doc with some option like thumbs up number or forwarding number,or some thing mix up. then, SEARCH it will output with number order.

The option could be init at beginning, same as StopTokenFile. because in most case one dataset only need one ranking algorithm(maybe?). multiple dimension score number is seems difficult to management

well, if two more dataset mix up in server, SEARCH is going to read skewed (some doc has option and some not), so SEARCH should be specify the score/number option to filter the suitable doc.