Scope graph rebuilds per store, batched and resumable, with run records

[JeroenDeDauw]

Groundwork for background graph rebuilds: graph stores get stable identities, and the all-or-nothing rebuild script becomes a per-store, batched, resumable service with durable run records.

Store identity. Nothing could previously address one store: SPARQL entries were known only by list position, a projection can be served by several stores, and Neo4j has no projection at all — hence names. neo4j names the bundled backend (names are case-sensitive; this one reservation is not), $wgNeoWikiSparqlStores entries take an optional name defaulting to their projection, and extension plugins register under a name. A conflicting or invalid name (reserved, duplicate — later in config order loses — or over 255 bytes) drops that entry entirely: it is neither written on save nor rebuildable until renamed, warned on the NeoWiki log channel. A single-store config behaves identically to master; only configs with two entries sharing a projection must add explicit names. update.php creates the new table; nothing else migrates.

Per-store rebuild runs. RebuildGraphDatabases executes one run per store (--store <name> to scope; default all stores, one at a time — the per-store run model is what later allows concurrency). A run walks subject pages by keyset cursor in batches (--batch-size, default 200), then reconciles deletions. A page-level failure is counted, logged, and stepped over; a store-level failure (initialize, timeout, DB error) ends that store's run as failed with the cursor preserved, and the loop moves on to the next store. --resume continues a failed run past both completed and failed pages — re-run the store to retry its failures. The exit code is non-zero whenever any store ends out of sync: a failed run, page failures, or (under --resume) a store that was never built.

Run records. neowiki_rebuild_runs holds one row per run; status and cursor are the resume contract. One active run per store is enforced check-then-act (a real lock is deferred — the race window is a human double-start). A hard-killed run is released by setting its row to cancelled, which keeps the cursor so --resume works. Store-level error messages are credential-sanitized before persistence and logging. No retention policy yet; rows accrue one per run.

Deferred to the follow-up PR: job-queue execution, REST endpoints, an admin Special page, staleness display, and supersede/cancel — plus three known gaps a reviewer would otherwise flag: a store dying mid-walk is counted as page failures rather than ending the run, the deletion phase is not checkpointed, and the run-start check is not a lock.

Verification: ~150 new tests including resume-after-failure and cursor-boundary cases; mutation-checked (including two survivors an independent review pass found, now closed); the generated schema verified byte-identical across mysql/sqlite/postgres; enumeration equivalence with master proven empirically at seven batch sizes.

Decisions worth a look:

1. First own database table (vs stashing state in job params) — durable, farm-queryable history.
2. Sequential writes per store — QLever requires a single writer; intra-store parallelism stays open for Neo4j later.
3. Skip-and-warn on bad store names rather than hard config failure — mirrors the config's existing degrade behavior.

Open #1226 rewrites the same enumeration area with an incompatible design (internally-paging generator vs resumable keyset cursor); whichever merges second needs rework, not conflict resolution. Proposal: merge this first, then rebase #1226 onto the cursor lookup. Worth deciding order before undrafting.

For #1230.

_{AI-authored — Claude Code, Fable 5 (max); two-PR plan agreed with @JeroenDeDauw, spec settled in-session; diff not yet human-reviewed; independent review pass applied (blocking finding — backend credentials persisted unsanitized — fixed), mutation-checked, CI green.}

Production notes

Design and supervision by Fable 5 (max); implementation, review fixes, and text pass by Opus 5 (max) subagents. The independent review's verified-fine list includes byte-identical regenerated schema on all three DBMS, empirical enumeration equivalence with master at seven batch sizes, and maintenance-runner exit-code compatibility across MW 1.43–master.

[JeroenDeDauw]

[alistair3149]

Superseded by #1254, which is this branch rebased onto master and reworked onto the page enumeration #1226 introduced. Leaving this open as the record of the original design and its review — worth closing once #1254 is accepted.

#1226 merged first, and as the description above predicted, that means rework rather than conflict resolution: it replaced SubjectPageIdsLookup/SubjectPageRebuilder with PageIdsLookup/PageRebuilder and changed the scope from Subject-bearing pages to every page. Resolved mechanically, the tree would have carried two enumerations of the same wiki and a rebuild reconciling a subset of what every other path projects.

#1254 also carries the 13 review-fix commits that were #1250, folded in before the rework so the same conflicts were resolved once. It is a draft: 12 tests still fail, and three of them raise a question about what wasOfferedToTheStore should mean now that pages without Subjects are projected too — detail in its description.

[alistair3149]

Rewound to 83f4f8bf, its state before the #1250 commits were briefly folded in, so this PR stands as an unmodified record. The review fixes and the rework both live in #1254.

[alistair3149]

Superseded by #1254

[alistair3149]

#1254 — this PR rebased onto master, reworked onto the #1226 page enumeration, and carrying the #1250 review fixes — has been merged. Closing this record PR as superseded; its diff and discussion remain the reference for the original design and its review.