[Proposal]: Replace Meilisearch & Elasticsearch with SEAL

[stloyd]

Describe the Proposal

As we already spoke on Discord, I think we should remove Meilisearch and potentially Elasticsearch.

If we want to think about any future search engine, we should consider SEAL (I think we were speaking about it a long time ago as well...)

API Adjustments

No response

Are you intending to also work on proposed change?

Maybe

Are you interested in sponsoring this change?

None

Integration & Dependencies

No response

[norberttech]

Fully agree, SEAL look like much better approach, but we would need to research first:

• can we paginate over results
• can we index documents in batches
• does SEAL expect any type of schema
• how the dependencies of this adapter would look like
• does SEAL come with some extension points where we could hook in our flow-php/telemetry

Pretty much all the questions from the proposal template would need to be first answered before we can move forward with this in order to not hit the wall somewhere in the middle of implementation 😅

@stloyd would you be interested in doing some initial research and putting together some high level implementation plan?

ping @alexander-schranz

[stloyd]

1. Seems there is pagination support: https://php-cmsig.github.io/search/search-and-filters/index.html#pagination
2. There is some bulk system: https://php-cmsig.github.io/search/indexing/index.html#bulk-operations
3. There is a schema: https://php-cmsig.github.io/search/schema/index.html
4. There is a per adapter dependency, so it looks like if we do the "core", we would already allow developers to decide what adapter and dependencies they want, it may be tricky to test that tho,
5. Hard one, would require knowledge of SEAL internals, which at this point I don't have ;)

[norberttech]

Hard one, would require knowledge of SEAL internals, which at this point I don't have ;)

me neither 🫢 but it's necessary to even start thinking about it 😅

btw. I'm also fine with killing melisearch entierly and keeping just elasticsearch for now (just upgrading to latest supported version) as a bare minimum

[alexander-schranz]

can we paginate over results

1. Seems there is pagination support: https://php-cmsig.github.io/search/search-and-filters/index.html#pagination

👍

can we index documents in batches
2. There is some bulk system: https://php-cmsig.github.io/search/indexing/index.html#bulk-operations

👍

does SEAL expect any type of schema
3. There is a schema: https://php-cmsig.github.io/search/schema/index.html

👍 SEAL is strict about a Schema, every field need to be defined, there is no support for something like just pushing fields there which does not exists yet.

how the dependencies of this adapter would look like
4. There is a per adapter dependency, so it looks like if we do the "core", we would already allow developers to decide what adapter and dependencies they want, it may be tricky to test that tho,

The core of SEAL is dependency free: https://github.com/PHP-CMSIG/search/blob/0.12/packages/seal/composer.json

But you require atleast one adapter. The memory adapter is simple and mostly used just as fast alternative to Functional or Unit tests. The easist adapter which really persis thed ata is Loupe which is a adapter for Loupe which requires just the SQLite extension.

As every adapter is it's own composer package you only need install whatever you really want. Mostly the only dependency is to the officials php packages of the search engines. Or in case of Solr Solarium which is widely used.

See composer.json of the different adapters: https://github.com/PHP-CMSIG/search/tree/0.12/packages

does SEAL come with some extension points where we could hook in our flow-php/telemetry
5. Hard one, would require knowledge of SEAL internals, which at this point I don't have ;)

I'm not familiar with flow-php/telemetry. But SEAL has a single point how you communicate with the engines and that is the Engine class. If you want to track or trace something you could decorate it via the EngineInterface.

There are a few other ideas in mind like add some Middleware or Event specific extension points. But not sure if in your case the EngineInterface decoration would still be the best.

Let me know if you need to know more?

[norberttech]

Thanks @alexander-schranz !
It seems that both project are very much aligned when it comes to core principles, this should make integration much smoother!

The most critical part was this:

👍 SEAL is strict about a Schema, every field need to be defined, there is no support for something like just pushing fields there which does not exists yet.

Which naturally works with Flow 💪 We are going to need to create a mapping between Flow and SEAL schema but that should be pretty straightforward

I'm not familiar with flow-php/telemetry. But SEAL has a single point how you communicate with the engines and that is the Engine class. If you want to track or trace something you could decorate it via the EngineInterface.

That should be more than enough, flow-php/telemetry is pretty much a OTLP (open telemetry protocol) implementation in pure PHP (it's a standalone library that is dependency free)

Let me know if you need to know more?

Thank you!