We should be able to compress the whole directory.
This will reduce number of index files being created.
We need #40 in case the generated index file is too big.
Also, need to do this in parallel. Strategies:
- Save index of file in buffer then flush it incrementally or at once in the end
- Enabling concurrent mode in sqlite (probably need to enable WAL as well)
We should be able to compress the whole directory.
This will reduce number of index files being created.
We need #40 in case the generated index file is too big.
Also, need to do this in parallel. Strategies: