How to do AND queries with this tokenizer?

[brendand]

I have an example of a term I would search for without this tokenizer and it would bring back a record.

If the data is:

Cat and Mouse

if I searched for:

Cat Mouse

then SQLite would return the record.

But with the character tokenizer installed, it no longer returns any records with the words Cat and Mouse in them. It MUST be queried as Cat and Mouse to get results.

Is there a workaround for that?

[haifengkao]

I never test it. The tokenizer works like this:

CatMouse
-> C a t M o u s e

You can think that the tokenizer adds space between each character,
to make each character becomes "word".

I think it should be able to support your case. But I never test it.

[brendand]

I actually realized I'm doing a phrase query with quotes around the search term. For example, I'm searching for "Cat Mouse" (with the quotes). So that explains why it's not finding anything with other words in between. However, if I remove the double-quotes I get WAY too many results. It's almost like it's doing an OR query with every letter.

[haifengkao]

That is correct. The underlying substring search is "phrase query".
I am surprised that I am the first person to figure this out.
SQLite does not support substring when this project is created. I don't know if it is supported now.

[brendand]

I found that if I just put all my search terms in double-quotes then I get the results I want. So I now match "Cat" "Mouse" and that works perfect.