A vector index is built by a job

[WaylandYang]

Decision record 0035, implementing #512 and #514 in one cut because entities.profile_embedding takes the same mechanism as chunks.embedding.

The index is built by a job, not a migration. The column is dimensionless and the dimension follows the workspace's embedding model, so nothing is known at migration time; and CREATE INDEX CONCURRENTLY cannot run inside the transaction sqlx wraps migrations in. vector_index::request is called by the first write of a dimension (set_embeddings, update_profile) and queues one build_vector_index job unless one is queued; the job builds a partial expression index, USING hnsw ((col::vector(N)) vector_cosine_ops) WHERE vector_dims(col) = N, CONCURRENTLY IF NOT EXISTS, on a plain connection, serially, with maintenance_work_mem raised for the session. An invalid leftover from a failed build is dropped first. Dimensions above 2000 (HNSW's limit on vector) are never requested and stay on the exact path.

The two reads write the dimension as a literal, cast both sides, and run in a transaction with hnsw.iterative_scan = relaxed_order, hnsw.scan_mem_multiplier = 4 and hnsw.max_scan_tuples = 100000. A bound parameter for the dimension works on the custom plan and falls back to a seq scan on the generic one, which sqlx switches to after five executions. nearest_typed_entities reads the subject's vector first so the dimension is known, and gains a dimension guard it never had.

Type resolution gathers before it reasons (#514). nearest_typed_for_each runs a batch's neighbour queries eight at a time with buffered, so they come back in input order and the per-subject reasoning stays in that order; DescendantsMemo answers descendants_of once per coarse class per batch, and a subject with no class stays out of the memo.

Measured

On a copy of a bench base (531 real chunks, 3,269 real profiles, 1024 dims) with a synthetic base of 50,000 chunks beside them:

vector_search on the 50k base, before	seq scan and top-N sort, 378.6 ms
after	index scan, 6.0 ms
Serial build over 50,531 rows, maintenance_work_mem 64 MB (default)	4 min 20 s, "graph no longer fits after 13,930 tuples"
same at 512 MB	60.6 s, 394 MB
200 real entity queries on a base of 1,415, HNSW forced, recall@10 vs exact	0.997 by distance (0.917 by id: ties at the tenth place)
522 real chunk queries, bases each ~1% of the index, HNSW forced, defaults	recall@10 0.705, 154 lists short, 32 ms a query; 0.075 with iterative_scan off
same with scan_mem_multiplier 4	recall@10 1.0, no list short, 74 ms a query
same with only max_scan_tuples 100,000	unchanged, 0.705
A real base of 229 chunks beside the 50k, the planner choosing	chunks_kb_idx and a sort, 1.5 ms; the 50k base takes the HNSW path, 3.8 ms

So the memory multiplier is what stops an iterative scan short, the tuple ceiling is not, and the planner keeps small bases on the exact path on its own. The full EXPLAIN (ANALYZE, BUFFERS) before and after are in the record's measurements.

Recall bar proposed: recall@10 of 0.99 against the exact top ten on real embeddings, measured with the index forced, which this passes on both tables once the scan memory is raised. Random vectors (16/24 in the earlier synthetic run) are the wrong corpus for the question.

Tests

Database-gated, run with UTOPIA_TEST_REQUIRE_DB=1. Every question is asked twice, without the index and after vector_index::build, and the answers must agree:

• the_nearest_chunk_is_found_however_it_is_reached: the nearest chunk, a base of one beside a base of 2,000 filling its limit, tenancy, a 5-dim chunk among 7-dim ones, a superseded chunk, a moment on the record axis; and the index's name, USING hnsw, vector_cosine_ops and the partial predicate, with a second build a no-op.
• a_batch_gathers_its_neighbours_in_order: the batch comes back in input order at concurrency 1 and 8, the same with the entity index built; a subject is not its own neighbour, an untyped entity is never evidence, a same-document neighbour is flagged, a 3-dim subject finds only 3-dim neighbours and does not error; a two-connection pool completes sixty subjects; the memo returns what the query returns, leaves a classless subject out, and does not notice a class added mid-batch.
• a_vector_index_is_built_by_a_job: the first write queues one build, the second does not, a built index is remembered, a dimension above the limit is neither queued nor buildable and fails as Validation, which the dispatcher marks terminal.
• a_search_reads_the_base_as_it_was passes unchanged.

Not written: a_signature_is_guidance_and_not_a_gate from #514, which is the candidate-ranking logic in the server loop and unchanged here.

Closes #512. Closes #514.

🤖 Generated with Claude Code