Find a way to persist C extension types

[adrinjalali]

The way pickle persists C extension types (including objects created by cython) is to rely on __reduce__ and __reduce_ex__.

We should find a protocol that third party packages can implement, so that we could avoid relying on __reduce__.

One issue is that we can't call __new__ on C extension types generated by Cython. Cython would raise if the type tries to implement this special method. However, one can implement __cinit__ in cython.

We can certainly require those objects to implement __getstate__ and __setstate__. The remaining issue is the constructor.

Right now we call __new__ for all other objects, but we can't do that here. The question is, should we then call Object() instead? Or should we find a way to have something which acts as a low level and light constructor.

Some links:

• https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html
• https://docs.python.org/3/extending/newtypes_tutorial.html
• support __new__() in extension types cython/cython#799
• https://github.com/cython/cython/blob/fcfd16c7467c31f255287a73f36cf66b32bc096c/Cython/Compiler/Symtab.py#L2291
• result type of calling an extension type is unknown cython/cython#793
• https://cython.readthedocs.io/en/latest/src/userguide/special_methods.html

[BenjaminBossan]

Could you elaborate what C extension types exist that we should support in skops?

The question is, should we then call Object() instead?

IIUC, that would require that all of the class's arguments have default values. Can we expect that?

Or should we find a way to have something which acts as a low level and light constructor.

Could you please give a quick example of what that would look like?

[adrinjalali]

Could you elaborate what C extension types exist that we should support in skops?

We already support the ones from sklearn, but if we want things to work easily for the third party estimators, then we'd need to have a way for it to work w/o us having to hard-code what they are in skops. For instance, if there are any from XGB or lightgbm or any other complied third party library.

IIUC, that would require that all of the class's arguments have default values. Can we expect that?

My bad. I'm suggesting to use __new__ whenever available, and Object() otherwise. This would mean no python object would be constructed using its __init__ since they all have __new__, and only call Object() for other types, such as C extensions.

Could you please give a quick example of what that would look like?

Being able to call __cinit__ from python would be an example.