Vector Search Capabilities...

[VagyokC4]

Hi @nicolasff. How have you been. Hopefully staying safe out there. I wanted to enquire about utilizing the new Vector search from Redis.

Here is the conversation I have with Redis-OM library maintainer, and here are his comments regarding the implementation of the Vector Search capabilities.

redis/redis-om-dotnet#352 (comment)

Just getting your thoughts on the best way to implement (if possible) doing vector searches of Redis via Webdis. It looks like I just need to stringify the current changes to get functionality back to where it was.

Wanted to see if we can get vector searching supported or if it can be supported.

[slorello89]

@VagyokC4 - FWIW you are going to get a byte[] from Redis.OM for Vectors. Probably the biggest challenge is going to be whether they are singles/doubles. My guess is that you are going to want to pass the vector as an array of numbers in their original encoding. The problem is that I would guess that webdis will handle all number arrays as effectively double arrays (as there's generally no differentiation between single/double in JSON). So you might have to restrict yourself to double arrays (could be interesting given most of the vectorizers are Single).

The other thing of course is if webdis will pass the parameters correctly passing arrays to Redis as a single parameter is somewhat novel - though I suspect if Webdis is just forwarding the parameters this probably won't be a problem.

[nicolasff]

Hi @VagyokC4, I'm not familiar with this feature and haven't used Redis Stack or any of the commercial versions of Redis so it's difficult for me to answer your question.

I looked at the docs for FT.SEARCH, but the command has tons of parameters and I don't know how it would be called for your use case. As long as the command can be expressed in the expected format, Webdis should be able to forward it to Redis.

The main point of concern for me is the mention of parameters being arrays:

passing arrays to Redis as a single parameter is somewhat novel

I don't see anything in the FT.SEARCH docs about a parameter being an array, or anything that would indicate that one of the values is an array. Arrays are mentioned in the reply format, but I understood "parameter" as being part of the command, not the reply. Which parameters can be arrays, then?

If you could give me an example of what the command would look like and specify the types of values being passed in and/or returned, it would really help. I also noticed byte[] being mentioned above and Webdis does have some support for binary data, but it is pretty limited (see here for an example).

The main wire format for Webdis inputs and outputs is JSON, and JSON doesn't support binary fields without some kind of encoding.

[slorello89]

Hi @nicolasff

So FT.SEARCH (as you've noted) does an awful lot of lifting. You won't see what I'm referring to in the docs for FT.SEARCH take a look at the Vector search docs instead, you'll notice this:

FT.SEARCH idx "*=>[KNN 10 @vec $BLOB]" PARAMS 2 BLOB "\x12\xa9\xf5\x6c" DIALECT 2

"\x12\xa9\xf5\x6c" is the "array" - it's a little-endian IEEE 754 binary string. Keep in mind you've already told redis the encoding (single or double precision floating point numbers) so it just reads the bytes in 4/8 byte chunks. Most clients will encode it correctly if it's sent in the right format. With redis-plus-plus I think I had to reinterpret_cast<char*>(float_array), and then use a string view to prevent the client from automatically escaping the string, but I don't remember if it's hiredis or redis-plus-plus that did the automatic escaping.