perf: remove recursive sub-community splitting, cap Leiden iterations, batch store_communities

[dg264]

Problem

On large monorepos (100k+ files, 300k+ nodes, 1M+ edges), the Leiden community detection hangs indefinitely due to:

1. Recursive sub-community splitting — After the initial Leiden pass, every community >50 nodes triggers a second full Leiden + cohesion pass via _detect_leiden_sub. With 3k+ communities on large graphs, this compounds exponentially.
2. Uncapped Leiden iterations — The default runs until convergence, which can take unbounded time on dense code graphs.
3. Per-member UPDATE in store_communities — Each member node gets an individual UPDATE query, resulting in thousands of sequential writes.

Tested on a production monorepo: 132,353 files → 328k nodes → 1.7M edges.

Scope (per review feedback)

This PR is narrowed to contributions not covered by #183 or #184:

• Cohesion O(C×E) fix → handled by perf: batch community cohesion to stop hangs on large repos #183 (_compute_cohesion_batch) ✅ already merged
• tools/build.py batch summaries → handled by perf: batch _compute_summaries queries to stop hangs on large repos #184 (more complete + regression tests) ✅ defer to perf: batch _compute_summaries queries to stop hangs on large repos #184
• This PR → the 4 items below, unique to this branch

Changes (communities.py only)

#	Change	Impact
1	Remove _detect_leiden_sub	Eliminates the recursive second-pass Leiden on communities >50 nodes that caused exponential blow-up
2	Cap Leiden iterations (n_iterations=2)	Prevents unbounded convergence on dense graphs; 2 passes produce equivalent quality for code dependency graphs
3	Batch UPDATE in store_communities	Single WHERE qualified_name IN (...) per community instead of per-member UPDATE. Fully parameterized (# nosec B608 consistent with project pattern in graph.py)
4	Progress logging	logger.info() at each phase (node loading, igraph construction, Leiden execution, cohesion, completion) so users can monitor large builds

Performance (132k-file monorepo, Apple M3 Pro)

Phase	Before	After
Leiden clustering	>2 hours (hung)	~45 seconds
store_communities	~5 min	~15 seconds

Backward Compatibility

• All changes are internal optimizations
• Public API, database schema, and output formats are unchanged
• Existing graphs do not need migration

Test Plan

• Verified full build completes end-to-end on 132k-file monorepo
• Verified community count and cohesion values are reasonable (3,273 communities detected)
• Verified MCP tools (semantic_search, query_graph, detect_changes) work correctly after build

[dg264]

FYI:- @tirth8205 i ran this on my local machine, after this optimization it is working for me. Previously build was never ending.

[tirth8205]

Code Review

This PR makes real performance improvements but has overlap and conflict with two other PRs now in queue.

Conflict with #183 (communities.py) — now merged

PR #183 was just merged. It addresses the same cohesion O(C×E) bottleneck via _compute_cohesion_batch (single pass over all edges, community-index bucketing). Your PR solves it via _compute_cohesion_fast (pre-built adjacency dict). The branch now has a merge conflict on communities.py.

Both algorithms are correct O(E) improvements. The merged approach is slightly more memory-efficient (no extra adjacency dict). What is unique and valuable in your PR that #183 does not have:

• Removing _detect_leiden_sub — the recursive second-pass Leiden on communities >50 nodes. This is a real fix for the sub-community exponential blow-up on large graphs. perf: batch community cohesion to stop hangs on large repos #183 kept this code path.
• n_iterations=2 for Leiden — capping iteration count. perf: batch community cohesion to stop hangs on large repos #183 left this uncapped.
• Batch UPDATE nodes SET community_id ... WHERE qualified_name IN (...) in store_communities — reduces per-member UPDATE calls to one per community. Not in perf: batch community cohesion to stop hangs on large repos #183.
• Progress logging throughout detect_communities — not in perf: batch community cohesion to stop hangs on large repos #183.

Overlap with #184 (tools/build.py) — still open

Your tools/build.py changes are substantively identical to PR #184. Both batch the risk_index using GROUP BY target_qualified + DISTINCT source_qualified. However #184 is strictly more complete:

• perf: batch _compute_summaries queries to stop hangs on large repos #184 also batches community_summaries (in-memory edge count dict) and flow_snapshots (chunked IN(?) queries at 450 to respect SQLITE_MAX_VARIABLE_NUMBER).
• perf: batch _compute_summaries queries to stop hangs on large repos #184 ships 3 regression tests including a SQL trace-callback guard that will catch re-introduction of per-row queries.

Your tools/build.py changes should be dropped in favour of #184.

Action needed

1. Rebase onto current main to pick up perf: batch community cohesion to stop hangs on large repos #183.
2. In communities.py: remove _compute_cohesion_fast / _build_adjacency and instead use the already-merged _compute_cohesion_batch for the Leiden path. Keep your unique contributions: removing _detect_leiden_sub, adding n_iterations=2, and the batch UPDATE in store_communities.
3. Drop all tools/build.py changes — they are superseded by perf: batch _compute_summaries queries to stop hangs on large repos #184.

One security note on the batch UPDATE in store_communities:

f"UPDATE nodes SET community_id = ? WHERE qualified_name IN ({placeholders})",  # nosec B608
[community_id] + member_qns,

placeholders is built from "?" * len(member_qns) so this is fully parameterized — the # nosec B608 annotation is correct and consistent with the project's existing pattern in graph.py.

Please rebase, narrow scope to the three unique contributions above, and this will be mergeable.

[dg264]

@tirth8205 addressed comments