Python updates

[randompersona1]

Python 3.14 includes an alternative build of the interpreter that disables the GIL PEP 703. We should support this build in the medium term.

C extension modules have to be specifically recompiled for the freethreaded interpreter. Many popular packages have done this already, like numpy, but PySide6 has not. So we have to wait for that. Since we use threading, we also need to check our python code for race conditions that were previously prevented by the GIL.

New PRs should keep this in mind and should not rely on the behaviour of the GIL.

[RumovZ]

As for race conditions, I guess we already have plenty. If any new ones arise with nogil, they probably don't matter in the grand scheme of things.

[randompersona1]

It's not about race conditions by themselves, but rather atomicity. For example, on the C level, the following is not atomic:

class Singleton:
    storedvalue = 5

Singleton.storedvalue += 1  # not atomic

At the C level, the += 1 means a value is read from memory to the CPU, then incremented, then put back into memory.

With the GIL enabled, this isn't a problem. The interpreter uses the GIL to guarantee that only 1 python bytecode instruction is executed at the same time. So while the incrementation is happening, every other thread is locked from execution.

If the GIL is removed, multiple threads can access storedvalue at the same time, before the new value is put back into memory. Which means data is lost.

[RumovZ]

Yeah, but we already have code like if song_id in cls._songs: del cls._songs[song_id]. That's a race condition with or without the GIL. If we were really serious about this kind of thing, we would need to clean up this first.

[randompersona1]

Ironically, that specific syntax is probably protected by the GIL. The lock is released only at points the interpreter considers convenient, not after just any bytecode instruction. But point taken regardless.

I still think it's a bad idea to ignore it and hope nothing happens. Eventually, an issue will occur, and it will be incredibly annoying to debug.

[RumovZ]

Ironically, that specific syntax is probably protected by the GIL.

Interesting, didn't know that. Then yes, we would need to look into that.

[randompersona1]

Actually, I just checked, and I'm wrong. The GIL is released every 5ms, no matter what is currently running. So the above is actually already a race condition.

However, without the GIL, those kinds of race conditions will occur much more frequently. So I still think we should at the check the codebase before supporting freethreaded builds.

[RumovZ]

I had experimented a bit and couldn't trigger a KeyError with the GIL, but easily without it. So it seems like the GIL is helping a lot at least.
Anyway, it's bad code and should just be cls._songs.pop(song_id), which is safe with the GIL.
In nogil land, it's probably also safe, but there's no guarantee. I find this vagueness frustrating, but given that we don't write anything critical here, I would argue for making these kinds of compromises for performance's sake.

[randompersona1]

In nogil land, it's probably also safe, but there's no guarantee.

There is an implementation "guarantee" that methods of builtin containers are safe: https://docs.python.org/3/howto/free-threading-python.html#thread-safety

So .pop(), .append(), .insert() etc. are all good with current CPython.

I agree that it seems much more tricky to see incorrect behaviour at a glance. I shouldn't be expected to guess what bytecode the interpreter generates.

In any case, acquiring and releasing a lock is fairly cheap, and if another thread has to halt at that, it did so anyways with the GIL. So I have no qualms about using them liberally when needed.

[RumovZ]

In any case, acquiring and releasing a lock is fairly cheap

Depends on your definition of cheap. It's definitely notable.

>>> import timeit
>>> timeit.timeit("with lock: d[0]", "from threading import Lock; lock = Lock(); d = {0: 0}")
0.11629969999194145
>>> timeit.timeit("d[0]", "from threading import Lock; lock = Lock(); d = {0: 0}")
0.02090219990350306

I shouldn't be expected to guess what bytecode the interpreter generates.

Amen to that. Though to be fair, that was never what Python was intended for. 🙁