Consider patching polars.read_parquet to make it kind of work?

[lesteve]

First of all, thanks for allowing polars to be back as a Pyodide package 🙏! ccing @fzumstein since he is the one who apparently did most of the work in https://github.com/xlwings/polars-pyodide/.

In a scikit-learn context, we have examples that can be run in JupyterLite using the pyodide kernel. We also need polars.read_parquet in external tutorials where it is convenient to have a JupyterLite setup so that attendees don't have to install anything. See this for more context about the scikit-learn use case.

I saw that you are getting polars from https://github.com/xlwings/polars-pyodide/ whose README.md says that parquet functionality is disabled because it relies on async functionality without an easy solution in sight.

I am wondering how doable this would be to patch polars.read_parquet to make it kind of work with the work-around suggested in pola-rs/polars#20876 (comment). The idea is to use pyfetch + pyarrow.parquet + polars.from_arrow. I guess you could also have a similar approach for a local file (don't use pyfetch I guess).

Not sure whether pyodide/pyodide-recipes or xlwings/polars-pyodide would be the best place to do the patch, I guess pyodide/pyodide-recipes seems more appropriate?

I am totally fine if you tell me this is a terrible idea, I thought I would open an issue and see what happens 😉.

I don't know much about polars, there may be more functionality in polars.read_parquet that this simple work-around can not handle, but it seems like it could be useful still.

[fzumstein]

Apparently, pyemscripten support is now on its way in tokio and mio, the underlying blockers for more recent versions of Polars (see below), so I'd wait for that to happen, then see if the Polars team wants to re-enable their pyemscripten build and let them decide about read_parquet. In the meantime, your workaround is probably a good solution?

• tokio#8281
• mio#1969
• tokio#8194

[lesteve]

Great to know that there is progress on this front, even if I am not going to claim I understand the fine details 😉.

It's hard for me to judge the timeline of these developments (months I guess?) and whether it is worth it to have a work-around in pyodide-build in the meantime so I will happily defer the decision to pyodide-build maintainers.

In the scikit-learn context, using the work-around won't fly I think. We don't really want to complicate the code just for the Pyodide kernel, 99.999% of people looking at this scikit-learn example will run in a vanilla CPython kernel.

[ryanking13]

Yeah ,Guy is working on emscripten support in tokio but I would say it would take a while. I didn't look through the read_parquet implementation, but patching the function sounds like a good workaround for now.

Since we don't maintain polars in pyodide-recipes repository, putting the patch in https://github.com/xlwings/polars-pyodide/ sounds more reasonable.