Optimize multi-threading to improve unpacking speed for large NPK files (resolve unresponsiveness issue)

[3189252298]

Description

When using NeoXtractor to unpack large NPK files, the tool frequently becomes unresponsive. However, observing via Task Manager shows that CPU and memory usage remain at a low level during this process.

It is suspected that the current single-threaded (or limited-threaded) processing logic fails to fully utilize the hardware resources, leading to long waiting times and unresponsiveness.

Suggestion

1. Optimize the unpacking logic to support multi-threaded parallel processing for large NPK files.
2. Add an option to configure the number of threads (e.g., auto-detect CPU cores or manual setting) to maximize hardware utilization.
3. Fix the unresponsiveness issue caused by insufficient resource utilization during large file processing.

[YUCLing]

The loading process is done in another thread (all files are done in that one thread sequentially) instead of the main thread. So it's abnormal to see unresponsive program when loading.

I have tried to support multithreaded loading when implementing the current loading mechanism, but I failed to utilize the advantages of multithreading. Need someone familiar with Python multithreading to take a look.

[YUCLing]

Just took a brief look, it seems that Python has a Global Interpreter Lock. Likely the bottleneck of the multithreaded loading I have tried.

To workaround it, an overhaul to the loading mechanism is required and might be a little complex.

[MarcosVLl2]

Python has a GIL, aka Global Interpreter Lock, which means that it is unable to use more than one thread, unless you rely on CPython or expressively use a python without GIL, which leads to race conditions and many other problems internally.

The benefit of Python is allowing us to write simple code to make it as compatible as possible!
For now, if you want higher speed, you need better hardware or you can attempt to rewrite the whole code yourself!

[YUCLing]

unless you rely on CPython or expressively use a python without GIL, which leads to race conditions and many other problems internally

@MarcosVLl2

Actually it's not the blocker for this. multiprocessing is need for this instead of multithreading.

I have a tasking system in my mind for this, which can also be the foundation of supporting #42 .

The tasking system will be composed of the following parts:

• Task
• Dispatcher
• Executor Pool

Within this framework, we will need to decouple our actual Task processing logic, and we are quite near it after my last big refactor. Now we only need to implement proper IPC (maybe Python already handled this for us), and three parts I mentioned before, then wire everything up. It should be work well.

But, it's still a quite amount of work.

Edit: The above design is based on my assumptions to what multiprocessing provides, since I haven't read the APIs of multiprocessing. However, in general, it should be working like this.