Would it be useful for ryp to spawn a dedicated thread for R work?

[jobrachem]

Hi @Wainberg, first of all: Thank you very much for developing ryp and making it publicly available. We have been using it in https://github.com/liesel-devs/liesel_gam for a few months now and appreciate it very much.

One issue that we have been dealing with is the single-thread enforcement (see liesel-devs/liesel_gam#54), because our library also uses jax, which does a lot of multithreading -- so we do not have full control about which thread calls ryp out of the box. We have solved the issue by explicitly moving all R work in our library into a dedicated R thread, which is created when importing the package. We then wrap and re-export the three ryp core functions so they always run in this thread.

We were wondering: would a similar solution maybe be something that is interesting for ryp in general? Maybe as an opt-in?

[Wainberg]

Hi Johannes, thanks for raising this issue and glad you're liking ryp!

I just pushed a change that removes the single-thread enforcement entirely, by using an RLock around r(), to_py() and to_r(). I think the small additional overhead of the locking is worth it, to allow packages like JAX to work seamlessly with ryp.

So here's a PySpark example that now works when it didn't before:

from pyspark.sql import SparkSession
from concurrent.futures import ThreadPoolExecutor

def thread_test(_):
    from ryp import to_py
    with ThreadPoolExecutor(10) as pool:
        return list(pool.map(lambda _: to_py('1 + 1'), range(100)))

if __name__ == '__main__':
    spark = SparkSession.builder.master("local[1]").getOrCreate()
    rdd = spark.sparkContext.parallelize([None], 1)
    print(rdd.mapPartitions(thread_test).collect())
    spark.stop()

This change should allow you to remove the dedicated thread-management code in liesel-gam. Let me know if you run into any issues when doing that, and then I'll make the change into a new minor version of ryp and update the PyPI and conda-forge packages to match.

Have you found any other pain points besides this one when integrating ryp into your library?