can we multithread the index construction better?

[jbellis]

observed init behavior on a large repo, 24 core / 48 ht box with a 3090 GPU:

• GPU is busy only intermittently, while it is busy CPU is minimal, single digit percents
• when GPU is not busy CPU spikes to 3-5 (ht) cores

Educated guess:

• GPU is primarily only used for embeddings up front [this is surprising given what I remember about og PLAID]
• Index construction is CPU bound and poorly parallelized
• Possibly there is also a pipelining issue where GPU sits idle while waiting for CPU (but also possibly it's just not a deep pipeline and the CPU is the bottleneck)

I do see thread counts briefly burst from ~40 to ~100 but this is not highly correlated to more %CPU reported by top

[raphaelsty]

Hi @jbellis, of course, the best way right now would be to continue to convert code as embeddings while updating the index. The index update needs to be sequential. Multiple processes can't update the index in parallel right now.

I'll draft something

[cepera-ang]

I did some profiling and experiments and got decent speed up on my machine (rtx 4060, 20 cores).

current main, init on 60k codebase (zed): 1520s
pipelined and optimized: 240s (gpu encode bottleneck: 206s)

colgrep codebase: 12s (gpu: 6s) vs main 26s

Will write a report tomorrow, small teaser:

[cepera-ang]

dropped it there #67

[cepera-ang]

@jbellis try the new release, @raphaelsty merged all that stuff already, I wonder what speed up / utilization would you get