Run graph rebuilds in the background with admin, API, and auto triggers - #1236
Merged
Conversation
A rebuild ran as one uninterruptible walk driven by the maintenance script. Splitting a batch out as its own step is what lets the same code be driven a batch at a time from elsewhere, and it closes the three things the scoped rebuild left open: - The removals are checkpointed like the projections. Both phases now walk by keyset cursor, and the run records which phase that cursor belongs to, so a rebuild interrupted between them continues with the removals instead of reprojecting the whole wiki first. - Starting a run takes the store's advisory database lock across the check for an active run and the row that check guards, so two callers starting at once produce one run and one refusal. - A batch of two or more pages that fails in its entirety is read as the store having gone rather than as a wiki of unprojectable pages: the run ends with its cursor left at that batch, so resuming retries it. A batch of one is exempt, or one broken page would wedge resume in a retry loop. The status and trigger enums gain the values the surfaces in the commits after this one file runs under, and the run carries the times the record already held. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A rebuild could only be run by someone sitting at a shell for as long as it took. This files one as work instead: a job per batch, each queueing the next until the run is done, so a rebuild can be asked for from a web request and outlive it. The run record is the state, not the queue. A job carries only which run to advance, and reads that record before doing anything, so a job for a run that has since been cancelled or finished does nothing — which is what lets cancelling be a single write, with no reach into the queue, and what makes a retried job safe. Rebuilds started here and run in a maintenance script are the same runs, sharing one record per store, so cancelling reaches a rebuild wherever it is running and only one may be under way at a time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing said whether a store's contents still matched what the wiki described. Editing a Mapping page is the case that matters: it changes what every mapped page's graph should contain, and nothing reprojects those pages, so the store keeps serving the old vocabulary with no sign that it is doing so. Each store is now reported as never built, stale, or in sync, derived from the run records and the wiki rather than stored: anything stored would be one more thing to keep true, and would be wrong exactly when it mattered. Staleness is measured against when the last finished rebuild started, not when it ended, because a Mapping edited mid-rebuild leaves the pages that rebuild had already passed projected under the old rules. Only a store holding an ontology projection can go stale this way. The native projection and a backend holding no RDF have no editable definition, so once rebuilt they stay as current as the per-edit projection keeps them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A rebuild could only be asked for from a shell, and how far a store was from the wiki could only be worked out by reading run rows. Three endpoints now answer both, so a rebuild is something the wiki can be asked for rather than something only its host can do. They are gated on a new neowiki-admin right, granted to administrators. What they report and change is the installation's own machinery, so no page's permissions could stand in for it, and no OAuth grant maps to it — this is not something a delegated application should acquire by asking for a category of access. A store is described by what it holds and how far that is from the wiki, never by how it is reached: the endpoint URL and access token it is configured with stay out of every response, because being allowed to rebuild a store is not being allowed to read the credentials for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebuilding a graph store, or finding out whether one needed it, meant shell access to the server. An administrator can now see and do both from the wiki. Rendered on the server and read from the run records, so what it shows is what a rebuild has actually recorded rather than a copy of it kept in the browser. A rebuild started here runs on the job queue, so the page comes back at once and progress appears on reload; there is no auto-refresh, so watching one costs a wiki nothing. Every action posts and redirects back, which is what makes reloading to watch a rebuild advance safe: it repeats a read rather than the submission that started the rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Editing a Mapping page changes what every mapped page's graph should contain, and reprojects nothing: the store keeps serving the old vocabulary. That is the gap this whole thread exists to close, and $wgNeoWikiAutoRebuildOnMappingChange closes it without anyone having to notice. Off by default. Such a rebuild reprojects every page carrying a Subject into that store, which on a large wiki is substantial work an administrator should choose to spend; while it is off, the store shows up as stale on Special:GraphStores with the rebuild a click away. A store already rebuilding is replaced rather than left to finish: a run begun under the old Mapping has projected part of the wiki under rules that no longer apply, so finishing it would leave the store half in each vocabulary with nothing recording that it had. Ending the old run and filing the new one happen under the store's start lock, so nothing can slip a third rebuild in between. The work is deferred past the change's own transaction. A rebuild cannot be started inside one — taking the start lock flushes the connection's snapshot, which a transaction with writes pending may not do — and an edit must not wait on a lock or a queue to be saved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Covers what an administrator now has: the page and the endpoints, the job-runner caveat that decides whether a large rebuild finishes this week or next, and that rebuilds started anywhere are the same runs. Also replaces the manual SQL for releasing a killed rebuild, which cancelling on the page now does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A batch read the run, spent minutes projecting, then wrote back what it got through — carrying the status it had read. A cancellation landing in that window was written straight back over: the admin was told the rebuild had stopped and it carried on. Worse, an automatic rebuild restarting a store mid-batch left the old run resurrected alongside the new one, both projecting into the store the design exists to give one rebuild at a time. A batch's write now only lands while the records still have the run going; one ended in the meantime keeps the status that ended it, and the batch's work is dropped rather than written over it. Alongside it, from reviewing the same code: - Timestamps are read back as MediaWiki timestamps rather than in whatever format the database stores them in, so staleness is not decided by comparing two different formats as strings. - A batch is read as store death only when every page the store was actually offered failed. One page the wiki dropped between the walk and the batch used to spare a dead store its verdict. - The batch totals reach the observer as something to call rather than as numbers, so a background rebuild no longer counts the whole wiki twice per batch to tell a reporter that reports nothing. - A batch is queued after the run row it names commits, not before. - A continuation that cannot be queued ends its run, like the first batch already did, rather than leaving it recorded as going with nothing going. - Special:GraphStores checks read-only mode, catches every failure its message claims to cover, and logs the reason that message points at. - Losing the start lock is a 409 rather than a 500, and a run's error is no longer serialized into REST responses — it can quote the endpoint a backend could not reach, which is for whoever reads the records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MediaWiki removed that method after 1.46, and it was the only thing these two tests used it for. What they are really about is that a store whose last rebuild was filed, or refused, can start another — which is a thing the lock's callers can observe. Also records why the restriction is passed to the SpecialPage constructor, deprecated on 1.46, rather than declared the way that release suggests: on 1.43 the permission check reads the property the constructor sets and not getRestriction(), so overriding that would leave the page open to everyone there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A batch's conditional write reported success by reading the row back, but that read is served from the batch's own transaction snapshot — which a job runner opens before the batch, and which therefore predates the cancellation the check exists to notice. The write was correctly refused and reported as landed anyway. Whether it landed now comes from the row count the write matched, and what ended the run is read under a lock so another connection's commit is visible. Taking a queued run up was an unconditional write on top of that, so a cancellation committed before the first batch was written straight back over, resurrecting a run the admin was told had stopped. It goes through the same conditional path, and a run something else ended stops there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The store name it reports back was substituted into the outcome message as wikitext, which the message transform then ran: a link handed to an administrator executed whatever it carried, on a GET. It is now substituted as plain text, after the transform. The outcome itself decided a message key, so the query string chose which message the page showed. Only the outcomes this page redirects with are reported now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A store refusing every removal ended its run Succeeded: the removal phase had no notion of a store that had gone, so the store was reported in sync while every page the wiki deleted stayed queryable in it, out of reach of --resume. Removals now report failure the way projections do, and a whole batch of them failing ends the run with its cursor rewound to that batch. The test the heuristic never had also showed it reading two things as a dead store that are not one. A short last batch is where the walk's permanently unprojectable pages collect, and a batch almost all of whose pages the wiki has since dropped tells the store's answer from one page. Both left --resume retrying pages that will never work, so a batch now has to be full and to have reached the store with at least two pages. The pages of a batch the run ends on are no longer logged or reported as pages that failed. That batch is retried from where it began, so nothing should be left recording them as settled failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every batch handed its observer a way to count how many pages there are in total, and the one observer that wanted a denominator called it — turning a rebuild into a full-wiki count per batch. The script counts each total once, for the store it is about to rebuild, and the observer is left reporting what a batch did rather than how much there is of it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cancelling read the store's active run and then wrote it back cancelled. A run that reached the end of the wiki in between had that recorded over: the rebuild had reconciled the wiki, and the store was then reported as never built. The records now end the run in one conditional write, and say whether there was one to end. The cancelled run also keeps whatever progress the records hold rather than whatever the caller last read, so a batch that advanced it meanwhile is not rolled back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An edit to a Mapping page cancelled whatever rebuild its store had going and queued its own. With automatic rebuilds enabled, that let anyone who may edit a Mapping take away a rebuild an administrator had started and was waiting on, and start their wait over from the beginning of the wiki. An automatic rebuild now replaces only another automatic one. A run started from the script, the page or the API is left to finish, and said so on the log. The store is reported stale once it ends, so the changed Mapping is not lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deleting the Mapping page that defines a projection left every page projected under it carrying that vocabulary, with nothing reprojecting them — but the store read as in sync, because the last change was looked for on a page that no longer exists. The deletion log now says when, for a page that is gone. A projection nothing ever defined still reads as never changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both endpoints write — a run record, and the jobs that carry it — and neither asked whether the wiki was accepting writes. They answer 503 with the reason now, rather than leaving the write to fail somewhere further down with a half-filed rebuild behind it. Special:GraphStores already checked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both comments read as though every failed push were caught there. A web request's push is deferred past the response, and what runs it logs failures rather than raising them, so what these see is a job the queue refuses before the deferral plus everything pushed under the command line. A continuation the queue will not take now has the test the first push already had: it ends the run rather than leaving it recorded as going with nothing carrying it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Only the userinfo of a connection URI was removed, which is how a Bolt driver quotes credentials. A SPARQL store carries them in the query string or in an Authorization header instead, and those messages are kept just as long — on a terminal, in deployment logs, and in the rebuild run records. Access tokens, API keys, passwords and bearer tokens are removed too now, leaving what names them so the message still says how the store was being authenticated to. Special:GraphStores also declares that it writes, so MediaWiki routes it to the primary database rather than a replica. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A rebuild that ran to the end but could not reconcile every page left the maintenance script exiting non-zero and the page saying "In sync" about the same records. The store holds a copy of the wiki with holes in it, which is where a store nothing has ever rebuilt stands, so it is reported the same way. Rebuild batches are also filed as deduplicable. The queue only drops a batch matching one still waiting to be claimed, so a run's serial chain is unaffected and what is dropped is the second copy of a batch that got run twice — which is where a run would otherwise fork into two chains advancing one cursor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The refusals asserted only their status code, so a handler that refused and then went on to write would have passed; they check the records now too. The CSRF path was never driven at all, only stubbed past — the real validator refuses both mutating endpoints in a test. A batch filed for a run nothing exists under asserted the absence of a run nothing had created, which no change could break; it asserts what the batch projected and what it said instead. Stale is the one store state an operator reaches without a rebuild having failed, and neither the page nor the REST serializer had ever rendered it. Both do now, over a Mapping edited after the rebuild that read it. A resume that picks up partway through the removals also has a test: the removals already made are not made again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A page called Mapping:Native no longer sets an automatic rebuild going. The native projection is defined by NeoWiki's own code, which is already why a store holding it never reads as stale. - A rebuild batch carries only the run it advances. The store came along in the job parameters as a second copy of what the run record already says, which a job filed for one store could have used to advance another's run. - A job whose parameters are not what this version files reads as run 0 rather than fatalling on a missing key. - The log tells a rebuild started from the wiki to rebuild the store again, rather than naming a maintenance-script option whoever started it cannot reach. - A Mapping edit on a wiki that has not asked for automatic rebuilds registers no deferred update at all, and one that has cannot take the edit's deferred work down with an unresolvable backend. - Removed the stray blank line in extension.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- GraphStoreStatusLookup::getStatus() and RebuildProgress::getCursor() had no production caller. - The graph-store endpoints share an access rule and an error shape; building a serializer is not something they share, only something each of them does. - The rebuild coordinator's batch size no longer defaults on the factory, so a production caller says which size it means rather than the parameter being there for tests to override. - The start lock is a port the rebuild use case depends on, so it and its exception sit with the use case rather than in Persistence, giving the REST handler one namespace to catch its refusals from. The implementation stays. - NeoWikiExtension::getStoreProjections() is private; what a Mapping change looks at is its own accessor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Creating pages that carry a Subject, reading back what a store was given, deleting a page and flattening a log buffer were written out once per test class. They are on the shared base class now. Two suites keep their own page deletion, overriding rather than repeating it: theirs also runs the deferred updates the projection they watch happens in. The one that returns the status its test asserts on is named for that instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `--resume` continues in whichever of the rebuild's two phases it stopped in. - A store that stops answering ends the run, with the size at which that stops being recognisable. - Cancelling on the page or over the API also stops a rebuild the script is running. - A rebuild queued on a wiki whose job runner does not load NeoWiki stays queued, blocking the next one, until it is cancelled. - A deleted Mapping page makes its stores stale, as an edited one does. - The rebuild endpoints answer 503 while the wiki is read only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- The rebuild's two halves are stated once, where `--resume` needs them, rather than restating what the section opener already said. - Cut the reasoning behind a whole failed batch ending the run, and behind one rebuild per store blocking the next: both restate the contract above them. - The maintenance page links the REST reference for the graph-store endpoints rather than repeating paths nothing checks for drift. - `neowiki-admin` allows viewing rather than reporting, and a Mapping edit rather than "it" is what cannot take a hand-started rebuild away. - The stale state reads "changed at <time>", and a rebuild that could not be queued is "a" rather than "the" rebuild. - qqq: deleting a Mapping page makes its stores stale as editing one does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cancelled-outcome message promised a resume only the maintenance script offers; it now states what actually happens. The state label for a store no run ever fully reconciled read "Never built" beside a listed last rebuild; it now reads "Not reconciled", which covers both ways of getting there. Also: the Stale docblock said finished where the code compares against started, the auto-rebuild docs state what happens to a Mapping edit during a hand-started run, and the REST doc gives the rebuild endpoints' refusal order straight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alistair3149
pushed a commit
that referenced
this pull request
Aug 4, 2026
…rs (#1236) * Make a rebuild batch the unit of work, and close three gaps A rebuild ran as one uninterruptible walk driven by the maintenance script. Splitting a batch out as its own step is what lets the same code be driven a batch at a time from elsewhere, and it closes the three things the scoped rebuild left open: - The removals are checkpointed like the projections. Both phases now walk by keyset cursor, and the run records which phase that cursor belongs to, so a rebuild interrupted between them continues with the removals instead of reprojecting the whole wiki first. - Starting a run takes the store's advisory database lock across the check for an active run and the row that check guards, so two callers starting at once produce one run and one refusal. - A batch of two or more pages that fails in its entirety is read as the store having gone rather than as a wiki of unprojectable pages: the run ends with its cursor left at that batch, so resuming retries it. A batch of one is exempt, or one broken page would wedge resume in a retry loop. The status and trigger enums gain the values the surfaces in the commits after this one file runs under, and the run carries the times the record already held. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Run graph rebuilds on the job queue A rebuild could only be run by someone sitting at a shell for as long as it took. This files one as work instead: a job per batch, each queueing the next until the run is done, so a rebuild can be asked for from a web request and outlive it. The run record is the state, not the queue. A job carries only which run to advance, and reads that record before doing anything, so a job for a run that has since been cancelled or finished does nothing — which is what lets cancelling be a single write, with no reach into the queue, and what makes a retried job safe. Rebuilds started here and run in a maintenance script are the same runs, sharing one record per store, so cancelling reaches a rebuild wherever it is running and only one may be under way at a time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report how far each graph store is from the wiki Nothing said whether a store's contents still matched what the wiki described. Editing a Mapping page is the case that matters: it changes what every mapped page's graph should contain, and nothing reprojects those pages, so the store keeps serving the old vocabulary with no sign that it is doing so. Each store is now reported as never built, stale, or in sync, derived from the run records and the wiki rather than stored: anything stored would be one more thing to keep true, and would be wrong exactly when it mattered. Staleness is measured against when the last finished rebuild started, not when it ended, because a Mapping edited mid-rebuild leaves the pages that rebuild had already passed projected under the old rules. Only a store holding an ontology projection can go stale this way. The native projection and a backend holding no RDF have no editable definition, so once rebuilt they stay as current as the per-edit projection keeps them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report and rebuild graph stores over the REST API A rebuild could only be asked for from a shell, and how far a store was from the wiki could only be worked out by reading run rows. Three endpoints now answer both, so a rebuild is something the wiki can be asked for rather than something only its host can do. They are gated on a new neowiki-admin right, granted to administrators. What they report and change is the installation's own machinery, so no page's permissions could stand in for it, and no OAuth grant maps to it — this is not something a delegated application should acquire by asking for a category of access. A store is described by what it holds and how far that is from the wiki, never by how it is reached: the endpoint URL and access token it is configured with stay out of every response, because being allowed to rebuild a store is not being allowed to read the credentials for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add Special:GraphStores for reporting and rebuilding stores Rebuilding a graph store, or finding out whether one needed it, meant shell access to the server. An administrator can now see and do both from the wiki. Rendered on the server and read from the run records, so what it shows is what a rebuild has actually recorded rather than a copy of it kept in the browser. A rebuild started here runs on the job queue, so the page comes back at once and progress appears on reload; there is no auto-refresh, so watching one costs a wiki nothing. Every action posts and redirects back, which is what makes reloading to watch a rebuild advance safe: it repeats a read rather than the submission that started the rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Optionally rebuild a store when its Mapping changes Editing a Mapping page changes what every mapped page's graph should contain, and reprojects nothing: the store keeps serving the old vocabulary. That is the gap this whole thread exists to close, and $wgNeoWikiAutoRebuildOnMappingChange closes it without anyone having to notice. Off by default. Such a rebuild reprojects every page carrying a Subject into that store, which on a large wiki is substantial work an administrator should choose to spend; while it is off, the store shows up as stale on Special:GraphStores with the rebuild a click away. A store already rebuilding is replaced rather than left to finish: a run begun under the old Mapping has projected part of the wiki under rules that no longer apply, so finishing it would leave the store half in each vocabulary with nothing recording that it had. Ending the old run and filing the new one happen under the store's start lock, so nothing can slip a third rebuild in between. The work is deferred past the change's own transaction. A rebuild cannot be started inside one — taking the start lock flushes the connection's snapshot, which a transaction with writes pending may not do — and an edit must not wait on a lock or a queue to be saved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document background rebuilds and the right they need Covers what an administrator now has: the page and the endpoints, the job-runner caveat that decides whether a large rebuild finishes this week or next, and that rebuilds started anywhere are the same runs. Also replaces the manual SQL for releasing a killed rebuild, which cancelling on the page now does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop a batch writing over the run it no longer owns A batch read the run, spent minutes projecting, then wrote back what it got through — carrying the status it had read. A cancellation landing in that window was written straight back over: the admin was told the rebuild had stopped and it carried on. Worse, an automatic rebuild restarting a store mid-batch left the old run resurrected alongside the new one, both projecting into the store the design exists to give one rebuild at a time. A batch's write now only lands while the records still have the run going; one ended in the meantime keeps the status that ended it, and the batch's work is dropped rather than written over it. Alongside it, from reviewing the same code: - Timestamps are read back as MediaWiki timestamps rather than in whatever format the database stores them in, so staleness is not decided by comparing two different formats as strings. - A batch is read as store death only when every page the store was actually offered failed. One page the wiki dropped between the walk and the batch used to spare a dead store its verdict. - The batch totals reach the observer as something to call rather than as numbers, so a background rebuild no longer counts the whole wiki twice per batch to tell a reporter that reports nothing. - A batch is queued after the run row it names commits, not before. - A continuation that cannot be queued ends its run, like the first batch already did, rather than leaving it recorded as going with nothing going. - Special:GraphStores checks read-only mode, catches every failure its message claims to cover, and logs the reason that message points at. - Losing the start lock is a 409 rather than a 500, and a run's error is no longer serialized into REST responses — it can quote the endpoint a backend could not reach, which is for whoever reads the records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Assert the start lock is let go of without lockIsFree MediaWiki removed that method after 1.46, and it was the only thing these two tests used it for. What they are really about is that a store whose last rebuild was filed, or refused, can start another — which is a thing the lock's callers can observe. Also records why the restriction is passed to the SpecialPage constructor, deprecated on 1.46, rather than declared the way that release suggests: on 1.43 the permission check reads the property the constructor sets and not getRestriction(), so overriding that would leave the page open to everyone there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop a stale read resurrecting a cancelled rebuild A batch's conditional write reported success by reading the row back, but that read is served from the batch's own transaction snapshot — which a job runner opens before the batch, and which therefore predates the cancellation the check exists to notice. The write was correctly refused and reported as landed anyway. Whether it landed now comes from the row count the write matched, and what ended the run is read under a lock so another connection's commit is visible. Taking a queued run up was an unconditional write on top of that, so a cancellation committed before the first batch was written straight back over, resurrecting a run the admin was told had stopped. It goes through the same conditional path, and a run something else ended stops there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop Special:GraphStores running what its query string carries The store name it reports back was substituted into the outcome message as wikitext, which the message transform then ran: a link handed to an administrator executed whatever it carried, on a GET. It is now substituted as plain text, after the transform. The outcome itself decided a message key, so the query string chose which message the page showed. Only the outcomes this page redirects with are reported now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Recognise a dead store in both phases of a rebuild, and only there A store refusing every removal ended its run Succeeded: the removal phase had no notion of a store that had gone, so the store was reported in sync while every page the wiki deleted stayed queryable in it, out of reach of --resume. Removals now report failure the way projections do, and a whole batch of them failing ends the run with its cursor rewound to that batch. The test the heuristic never had also showed it reading two things as a dead store that are not one. A short last batch is where the walk's permanently unprojectable pages collect, and a batch almost all of whose pages the wiki has since dropped tells the store's answer from one page. Both left --resume retrying pages that will never work, so a batch now has to be full and to have reached the store with at least two pages. The pages of a batch the run ends on are no longer logged or reported as pages that failed. That batch is retried from where it began, so nothing should be left recording them as settled failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop counting the whole wiki once per rebuild batch Every batch handed its observer a way to count how many pages there are in total, and the one observer that wanted a denominator called it — turning a rebuild into a full-wiki count per batch. The script counts each total once, for the store it is about to rebuild, and the observer is left reporting what a batch did rather than how much there is of it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Cancel a rebuild without reading it first Cancelling read the store's active run and then wrote it back cancelled. A run that reached the end of the wiki in between had that recorded over: the rebuild had reconciled the wiki, and the store was then reported as never built. The records now end the run in one conditional write, and say whether there was one to end. The cancelled run also keeps whatever progress the records hold rather than whatever the caller last read, so a batch that advanced it meanwhile is not rolled back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Leave a rebuild somebody started out of the automatic one's reach An edit to a Mapping page cancelled whatever rebuild its store had going and queued its own. With automatic rebuilds enabled, that let anyone who may edit a Mapping take away a rebuild an administrator had started and was waiting on, and start their wait over from the beginning of the wiki. An automatic rebuild now replaces only another automatic one. A run started from the script, the page or the API is left to finish, and said so on the log. The store is reported stale once it ends, so the changed Mapping is not lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report a store whose Mapping was deleted as stale Deleting the Mapping page that defines a projection left every page projected under it carrying that vocabulary, with nothing reprojecting them — but the store read as in sync, because the last change was looked for on a page that no longer exists. The deletion log now says when, for a page that is gone. A projection nothing ever defined still reads as never changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refuse to start or cancel a rebuild while the wiki is read only Both endpoints write — a run record, and the jobs that carry it — and neither asked whether the wiki was accepting writes. They answer 503 with the reason now, rather than leaving the write to fail somewhere further down with a half-filed rebuild behind it. Special:GraphStores already checked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Say what the batch-push catches actually reach Both comments read as though every failed push were caught there. A web request's push is deferred past the response, and what runs it logs failures rather than raising them, so what these see is a job the queue refuses before the deferral plus everything pushed under the command line. A continuation the queue will not take now has the test the first push already had: it ends the run rather than leaving it recorded as going with nothing carrying it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Redact the credentials a store carries outside its connection URI Only the userinfo of a connection URI was removed, which is how a Bolt driver quotes credentials. A SPARQL store carries them in the query string or in an Authorization header instead, and those messages are kept just as long — on a terminal, in deployment logs, and in the rebuild run records. Access tokens, API keys, passwords and bearer tokens are removed too now, leaving what names them so the message still says how the store was being authenticated to. Special:GraphStores also declares that it writes, so MediaWiki routes it to the primary database rather than a replica. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Agree with the script on what a store left out of sync looks like A rebuild that ran to the end but could not reconcile every page left the maintenance script exiting non-zero and the page saying "In sync" about the same records. The store holds a copy of the wiki with holes in it, which is where a store nothing has ever rebuilt stands, so it is reported the same way. Rebuild batches are also filed as deduplicable. The queue only drops a batch matching one still waiting to be claimed, so a run's serial chain is unaffected and what is dropped is the second copy of a batch that got run twice — which is where a run would otherwise fork into two chains advancing one cursor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Close the test gaps the graph-store surfaces were left with The refusals asserted only their status code, so a handler that refused and then went on to write would have passed; they check the records now too. The CSRF path was never driven at all, only stubbed past — the real validator refuses both mutating endpoints in a test. A batch filed for a run nothing exists under asserted the absence of a run nothing had created, which no change could break; it asserts what the batch projected and what it said instead. Stale is the one store state an operator reaches without a rebuild having failed, and neither the page nor the REST serializer had ever rendered it. Both do now, over a Mapping edited after the rebuild that read it. A resume that picks up partway through the removals also has a test: the removals already made are not made again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tidy the edges the graph-rebuild work left - A page called Mapping:Native no longer sets an automatic rebuild going. The native projection is defined by NeoWiki's own code, which is already why a store holding it never reads as stale. - A rebuild batch carries only the run it advances. The store came along in the job parameters as a second copy of what the run record already says, which a job filed for one store could have used to advance another's run. - A job whose parameters are not what this version files reads as run 0 rather than fatalling on a missing key. - The log tells a rebuild started from the wiki to rebuild the store again, rather than naming a maintenance-script option whoever started it cannot reach. - A Mapping edit on a wiki that has not asked for automatic rebuilds registers no deferred update at all, and one that has cannot take the edit's deferred work down with an unresolvable backend. - Removed the stray blank line in extension.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Take out what the graph-rebuild work does not use - GraphStoreStatusLookup::getStatus() and RebuildProgress::getCursor() had no production caller. - The graph-store endpoints share an access rule and an error shape; building a serializer is not something they share, only something each of them does. - The rebuild coordinator's batch size no longer defaults on the factory, so a production caller says which size it means rather than the parameter being there for tests to override. - The start lock is a port the rebuild use case depends on, so it and its exception sit with the use case rather than in Persistence, giving the REST handler one namespace to catch its refusals from. The implementation stays. - NeoWikiExtension::getStoreProjections() is private; what a Mapping change looks at is its own accessor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Share the test helpers the rebuild suites were each keeping Creating pages that carry a Subject, reading back what a store was given, deleting a page and flattening a log buffer were written out once per test class. They are on the shared base class now. Two suites keep their own page deletion, overriding rather than repeating it: theirs also runs the deferred updates the projection they watch happens in. The one that returns the status its test asserts on is named for that instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Say in the docs what a rebuild now does about its own failures - `--resume` continues in whichever of the rebuild's two phases it stopped in. - A store that stops answering ends the run, with the size at which that stops being recognisable. - Cancelling on the page or over the API also stops a rebuild the script is running. - A rebuild queued on a wiki whose job runner does not load NeoWiki stays queued, blocking the next one, until it is cancelled. - A deleted Mapping page makes its stores stale, as an edited one does. - The rebuild endpoints answer 503 while the wiki is read only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tighten the prose the background-rebuild work added - The rebuild's two halves are stated once, where `--resume` needs them, rather than restating what the section opener already said. - Cut the reasoning behind a whole failed batch ending the run, and behind one rebuild per store blocking the next: both restate the contract above them. - The maintenance page links the REST reference for the graph-store endpoints rather than repeating paths nothing checks for drift. - `neowiki-admin` allows viewing rather than reporting, and a Mapping edit rather than "it" is what cannot take a hand-started rebuild away. - The stale state reads "changed at <time>", and a rebuild that could not be queued is "a" rather than "the" rebuild. - qqq: deleting a Mapping page makes its stores stale as editing one does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make the graph-store UI text match what the page can do The cancelled-outcome message promised a resume only the maintenance script offers; it now states what actually happens. The state label for a store no run ever fully reconciled read "Never built" beside a listed last rebuild; it now reads "Not reconciled", which covers both ways of getting there. Also: the Stale docblock said finished where the code compares against started, the auto-rebuild docs state what happens to a Mapping edit during a hand-started run, and the REST doc gives the rebuild endpoints' refusal order straight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
alistair3149
pushed a commit
that referenced
this pull request
Aug 5, 2026
…rs (#1236) * Make a rebuild batch the unit of work, and close three gaps A rebuild ran as one uninterruptible walk driven by the maintenance script. Splitting a batch out as its own step is what lets the same code be driven a batch at a time from elsewhere, and it closes the three things the scoped rebuild left open: - The removals are checkpointed like the projections. Both phases now walk by keyset cursor, and the run records which phase that cursor belongs to, so a rebuild interrupted between them continues with the removals instead of reprojecting the whole wiki first. - Starting a run takes the store's advisory database lock across the check for an active run and the row that check guards, so two callers starting at once produce one run and one refusal. - A batch of two or more pages that fails in its entirety is read as the store having gone rather than as a wiki of unprojectable pages: the run ends with its cursor left at that batch, so resuming retries it. A batch of one is exempt, or one broken page would wedge resume in a retry loop. The status and trigger enums gain the values the surfaces in the commits after this one file runs under, and the run carries the times the record already held. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Run graph rebuilds on the job queue A rebuild could only be run by someone sitting at a shell for as long as it took. This files one as work instead: a job per batch, each queueing the next until the run is done, so a rebuild can be asked for from a web request and outlive it. The run record is the state, not the queue. A job carries only which run to advance, and reads that record before doing anything, so a job for a run that has since been cancelled or finished does nothing — which is what lets cancelling be a single write, with no reach into the queue, and what makes a retried job safe. Rebuilds started here and run in a maintenance script are the same runs, sharing one record per store, so cancelling reaches a rebuild wherever it is running and only one may be under way at a time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report how far each graph store is from the wiki Nothing said whether a store's contents still matched what the wiki described. Editing a Mapping page is the case that matters: it changes what every mapped page's graph should contain, and nothing reprojects those pages, so the store keeps serving the old vocabulary with no sign that it is doing so. Each store is now reported as never built, stale, or in sync, derived from the run records and the wiki rather than stored: anything stored would be one more thing to keep true, and would be wrong exactly when it mattered. Staleness is measured against when the last finished rebuild started, not when it ended, because a Mapping edited mid-rebuild leaves the pages that rebuild had already passed projected under the old rules. Only a store holding an ontology projection can go stale this way. The native projection and a backend holding no RDF have no editable definition, so once rebuilt they stay as current as the per-edit projection keeps them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report and rebuild graph stores over the REST API A rebuild could only be asked for from a shell, and how far a store was from the wiki could only be worked out by reading run rows. Three endpoints now answer both, so a rebuild is something the wiki can be asked for rather than something only its host can do. They are gated on a new neowiki-admin right, granted to administrators. What they report and change is the installation's own machinery, so no page's permissions could stand in for it, and no OAuth grant maps to it — this is not something a delegated application should acquire by asking for a category of access. A store is described by what it holds and how far that is from the wiki, never by how it is reached: the endpoint URL and access token it is configured with stay out of every response, because being allowed to rebuild a store is not being allowed to read the credentials for it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add Special:GraphStores for reporting and rebuilding stores Rebuilding a graph store, or finding out whether one needed it, meant shell access to the server. An administrator can now see and do both from the wiki. Rendered on the server and read from the run records, so what it shows is what a rebuild has actually recorded rather than a copy of it kept in the browser. A rebuild started here runs on the job queue, so the page comes back at once and progress appears on reload; there is no auto-refresh, so watching one costs a wiki nothing. Every action posts and redirects back, which is what makes reloading to watch a rebuild advance safe: it repeats a read rather than the submission that started the rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Optionally rebuild a store when its Mapping changes Editing a Mapping page changes what every mapped page's graph should contain, and reprojects nothing: the store keeps serving the old vocabulary. That is the gap this whole thread exists to close, and $wgNeoWikiAutoRebuildOnMappingChange closes it without anyone having to notice. Off by default. Such a rebuild reprojects every page carrying a Subject into that store, which on a large wiki is substantial work an administrator should choose to spend; while it is off, the store shows up as stale on Special:GraphStores with the rebuild a click away. A store already rebuilding is replaced rather than left to finish: a run begun under the old Mapping has projected part of the wiki under rules that no longer apply, so finishing it would leave the store half in each vocabulary with nothing recording that it had. Ending the old run and filing the new one happen under the store's start lock, so nothing can slip a third rebuild in between. The work is deferred past the change's own transaction. A rebuild cannot be started inside one — taking the start lock flushes the connection's snapshot, which a transaction with writes pending may not do — and an edit must not wait on a lock or a queue to be saved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document background rebuilds and the right they need Covers what an administrator now has: the page and the endpoints, the job-runner caveat that decides whether a large rebuild finishes this week or next, and that rebuilds started anywhere are the same runs. Also replaces the manual SQL for releasing a killed rebuild, which cancelling on the page now does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop a batch writing over the run it no longer owns A batch read the run, spent minutes projecting, then wrote back what it got through — carrying the status it had read. A cancellation landing in that window was written straight back over: the admin was told the rebuild had stopped and it carried on. Worse, an automatic rebuild restarting a store mid-batch left the old run resurrected alongside the new one, both projecting into the store the design exists to give one rebuild at a time. A batch's write now only lands while the records still have the run going; one ended in the meantime keeps the status that ended it, and the batch's work is dropped rather than written over it. Alongside it, from reviewing the same code: - Timestamps are read back as MediaWiki timestamps rather than in whatever format the database stores them in, so staleness is not decided by comparing two different formats as strings. - A batch is read as store death only when every page the store was actually offered failed. One page the wiki dropped between the walk and the batch used to spare a dead store its verdict. - The batch totals reach the observer as something to call rather than as numbers, so a background rebuild no longer counts the whole wiki twice per batch to tell a reporter that reports nothing. - A batch is queued after the run row it names commits, not before. - A continuation that cannot be queued ends its run, like the first batch already did, rather than leaving it recorded as going with nothing going. - Special:GraphStores checks read-only mode, catches every failure its message claims to cover, and logs the reason that message points at. - Losing the start lock is a 409 rather than a 500, and a run's error is no longer serialized into REST responses — it can quote the endpoint a backend could not reach, which is for whoever reads the records. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Assert the start lock is let go of without lockIsFree MediaWiki removed that method after 1.46, and it was the only thing these two tests used it for. What they are really about is that a store whose last rebuild was filed, or refused, can start another — which is a thing the lock's callers can observe. Also records why the restriction is passed to the SpecialPage constructor, deprecated on 1.46, rather than declared the way that release suggests: on 1.43 the permission check reads the property the constructor sets and not getRestriction(), so overriding that would leave the page open to everyone there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop a stale read resurrecting a cancelled rebuild A batch's conditional write reported success by reading the row back, but that read is served from the batch's own transaction snapshot — which a job runner opens before the batch, and which therefore predates the cancellation the check exists to notice. The write was correctly refused and reported as landed anyway. Whether it landed now comes from the row count the write matched, and what ended the run is read under a lock so another connection's commit is visible. Taking a queued run up was an unconditional write on top of that, so a cancellation committed before the first batch was written straight back over, resurrecting a run the admin was told had stopped. It goes through the same conditional path, and a run something else ended stops there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop Special:GraphStores running what its query string carries The store name it reports back was substituted into the outcome message as wikitext, which the message transform then ran: a link handed to an administrator executed whatever it carried, on a GET. It is now substituted as plain text, after the transform. The outcome itself decided a message key, so the query string chose which message the page showed. Only the outcomes this page redirects with are reported now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Recognise a dead store in both phases of a rebuild, and only there A store refusing every removal ended its run Succeeded: the removal phase had no notion of a store that had gone, so the store was reported in sync while every page the wiki deleted stayed queryable in it, out of reach of --resume. Removals now report failure the way projections do, and a whole batch of them failing ends the run with its cursor rewound to that batch. The test the heuristic never had also showed it reading two things as a dead store that are not one. A short last batch is where the walk's permanently unprojectable pages collect, and a batch almost all of whose pages the wiki has since dropped tells the store's answer from one page. Both left --resume retrying pages that will never work, so a batch now has to be full and to have reached the store with at least two pages. The pages of a batch the run ends on are no longer logged or reported as pages that failed. That batch is retried from where it began, so nothing should be left recording them as settled failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stop counting the whole wiki once per rebuild batch Every batch handed its observer a way to count how many pages there are in total, and the one observer that wanted a denominator called it — turning a rebuild into a full-wiki count per batch. The script counts each total once, for the store it is about to rebuild, and the observer is left reporting what a batch did rather than how much there is of it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Cancel a rebuild without reading it first Cancelling read the store's active run and then wrote it back cancelled. A run that reached the end of the wiki in between had that recorded over: the rebuild had reconciled the wiki, and the store was then reported as never built. The records now end the run in one conditional write, and say whether there was one to end. The cancelled run also keeps whatever progress the records hold rather than whatever the caller last read, so a batch that advanced it meanwhile is not rolled back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Leave a rebuild somebody started out of the automatic one's reach An edit to a Mapping page cancelled whatever rebuild its store had going and queued its own. With automatic rebuilds enabled, that let anyone who may edit a Mapping take away a rebuild an administrator had started and was waiting on, and start their wait over from the beginning of the wiki. An automatic rebuild now replaces only another automatic one. A run started from the script, the page or the API is left to finish, and said so on the log. The store is reported stale once it ends, so the changed Mapping is not lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Report a store whose Mapping was deleted as stale Deleting the Mapping page that defines a projection left every page projected under it carrying that vocabulary, with nothing reprojecting them — but the store read as in sync, because the last change was looked for on a page that no longer exists. The deletion log now says when, for a page that is gone. A projection nothing ever defined still reads as never changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refuse to start or cancel a rebuild while the wiki is read only Both endpoints write — a run record, and the jobs that carry it — and neither asked whether the wiki was accepting writes. They answer 503 with the reason now, rather than leaving the write to fail somewhere further down with a half-filed rebuild behind it. Special:GraphStores already checked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Say what the batch-push catches actually reach Both comments read as though every failed push were caught there. A web request's push is deferred past the response, and what runs it logs failures rather than raising them, so what these see is a job the queue refuses before the deferral plus everything pushed under the command line. A continuation the queue will not take now has the test the first push already had: it ends the run rather than leaving it recorded as going with nothing carrying it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Redact the credentials a store carries outside its connection URI Only the userinfo of a connection URI was removed, which is how a Bolt driver quotes credentials. A SPARQL store carries them in the query string or in an Authorization header instead, and those messages are kept just as long — on a terminal, in deployment logs, and in the rebuild run records. Access tokens, API keys, passwords and bearer tokens are removed too now, leaving what names them so the message still says how the store was being authenticated to. Special:GraphStores also declares that it writes, so MediaWiki routes it to the primary database rather than a replica. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Agree with the script on what a store left out of sync looks like A rebuild that ran to the end but could not reconcile every page left the maintenance script exiting non-zero and the page saying "In sync" about the same records. The store holds a copy of the wiki with holes in it, which is where a store nothing has ever rebuilt stands, so it is reported the same way. Rebuild batches are also filed as deduplicable. The queue only drops a batch matching one still waiting to be claimed, so a run's serial chain is unaffected and what is dropped is the second copy of a batch that got run twice — which is where a run would otherwise fork into two chains advancing one cursor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Close the test gaps the graph-store surfaces were left with The refusals asserted only their status code, so a handler that refused and then went on to write would have passed; they check the records now too. The CSRF path was never driven at all, only stubbed past — the real validator refuses both mutating endpoints in a test. A batch filed for a run nothing exists under asserted the absence of a run nothing had created, which no change could break; it asserts what the batch projected and what it said instead. Stale is the one store state an operator reaches without a rebuild having failed, and neither the page nor the REST serializer had ever rendered it. Both do now, over a Mapping edited after the rebuild that read it. A resume that picks up partway through the removals also has a test: the removals already made are not made again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tidy the edges the graph-rebuild work left - A page called Mapping:Native no longer sets an automatic rebuild going. The native projection is defined by NeoWiki's own code, which is already why a store holding it never reads as stale. - A rebuild batch carries only the run it advances. The store came along in the job parameters as a second copy of what the run record already says, which a job filed for one store could have used to advance another's run. - A job whose parameters are not what this version files reads as run 0 rather than fatalling on a missing key. - The log tells a rebuild started from the wiki to rebuild the store again, rather than naming a maintenance-script option whoever started it cannot reach. - A Mapping edit on a wiki that has not asked for automatic rebuilds registers no deferred update at all, and one that has cannot take the edit's deferred work down with an unresolvable backend. - Removed the stray blank line in extension.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Take out what the graph-rebuild work does not use - GraphStoreStatusLookup::getStatus() and RebuildProgress::getCursor() had no production caller. - The graph-store endpoints share an access rule and an error shape; building a serializer is not something they share, only something each of them does. - The rebuild coordinator's batch size no longer defaults on the factory, so a production caller says which size it means rather than the parameter being there for tests to override. - The start lock is a port the rebuild use case depends on, so it and its exception sit with the use case rather than in Persistence, giving the REST handler one namespace to catch its refusals from. The implementation stays. - NeoWikiExtension::getStoreProjections() is private; what a Mapping change looks at is its own accessor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Share the test helpers the rebuild suites were each keeping Creating pages that carry a Subject, reading back what a store was given, deleting a page and flattening a log buffer were written out once per test class. They are on the shared base class now. Two suites keep their own page deletion, overriding rather than repeating it: theirs also runs the deferred updates the projection they watch happens in. The one that returns the status its test asserts on is named for that instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Say in the docs what a rebuild now does about its own failures - `--resume` continues in whichever of the rebuild's two phases it stopped in. - A store that stops answering ends the run, with the size at which that stops being recognisable. - Cancelling on the page or over the API also stops a rebuild the script is running. - A rebuild queued on a wiki whose job runner does not load NeoWiki stays queued, blocking the next one, until it is cancelled. - A deleted Mapping page makes its stores stale, as an edited one does. - The rebuild endpoints answer 503 while the wiki is read only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tighten the prose the background-rebuild work added - The rebuild's two halves are stated once, where `--resume` needs them, rather than restating what the section opener already said. - Cut the reasoning behind a whole failed batch ending the run, and behind one rebuild per store blocking the next: both restate the contract above them. - The maintenance page links the REST reference for the graph-store endpoints rather than repeating paths nothing checks for drift. - `neowiki-admin` allows viewing rather than reporting, and a Mapping edit rather than "it" is what cannot take a hand-started rebuild away. - The stale state reads "changed at <time>", and a rebuild that could not be queued is "a" rather than "the" rebuild. - qqq: deleting a Mapping page makes its stores stale as editing one does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make the graph-store UI text match what the page can do The cancelled-outcome message promised a resume only the maintenance script offers; it now states what actually happens. The state label for a store no run ever fully reconciled read "Never built" beside a listed last rebuild; it now reads "Not reconciled", which covers both ways of getting there. Also: the Stale docblock said finished where the code compares against started, the auto-rebuild docs state what happens to a Mapping edit during a hand-started run, and the REST doc gives the rebuild endpoints' refusal order straight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1230.
Stacked on #1233 (base
graph-rebuild-scoped); the diff shows only this PR's commits. Rebuilds become background work with admin and API triggers, and Mapping edits stop silently leaving stores stale.Execution model. A background run is a chain of
neowikiGraphRebuildjobs, one batch per execution. Between batches the run row is the only coordination state: each job re-reads it, stops if it is terminal, executes one batch through the same step the CLI loop uses, and pushes its continuation. The alternative — the job queue's own root-job/dedup machinery — was rejected because its supersede state lives in a WAN cache, while a row is durable and inspectable. Every run-row write is conditional on the run still being active, and the what-ended-it re-read is a locking read, because a plain re-read inside the job'sREPEATABLE READtransaction returns the pre-cancellation snapshot (regression-pinned; the naive version resurrected cancelled runs). Cancelling — from UI, API, or SQL — is one conditionalUPDATE; a CLI run observes it at its next batch boundary. Run creation additionally holds a per-store advisory DB lock, start-time only, so concurrent starts cannot double-create; jobs never hold it.Run semantics. New
Queuedstatus andApi/Ui/Autotriggers. Both phases checkpoint their cursor (nwrr_phasedistinguishes them), so--resumecontinues within whichever phase was recorded. A store dying mid-walk is recognized when a batch that filled its limit had every still-existing page fail (at least two) — the run ends failed with the cursor rewound to that batch, in either phase. A run that finished but left pages failed reports as never built ("no run has reconciled the whole wiki"), making the Special page agree with the CLI's non-zero exit.Surfaces, all behind the new
neowiki-adminright (sysop by default; deliberately in no OAuth grant):GET /neowiki/v0/graph-stores— per store: projection, in sync / stale / never built, active and last successful run; never endpoint URLs or tokens — plusPOST/DELETE /neowiki/v0/graph-stores/{name}/rebuild(202/409/404), for farm tooling and orchestrators; andSpecial:GraphStoresfor humans — server-rendered, CSRF-protected forms, progress on reload. Staleness is derived, not stored: an ontology-projection store is stale when itsMapping:page was edited or deleted after the last successful run started (start-anchored on purpose: pages projected before a mid-run edit used the old rules). Native and Neo4j stores can only be in sync or never built.Auto-rebuild (
$wgNeoWikiAutoRebuildOnMappingChange, default off) supersedes onlyAuto-triggered runs: a Mapping save or delete cancels-and-restarts the affected stores' auto runs, deferred past the edit's transaction; an admin's CLI/API/UI run is never stomped, and rapid edits coalesce because each restart replaces the previous auto run. Enabling it hands everyone withneowiki-mapping-edit(all logged-in users by default) the power to start a full reprojection — the same cost as a CLI rebuild of that store.Accepted trade-offs and deferrals, for review:
generationcolumn is the clean fix if this ever matters in practice.nwrr_phaseschema patch exists only because Scope graph rebuilds per store, batched and resumable, with run records #1233 is unmerged — folding the column into Scope graph rebuilds per store, batched and resumable, with run records #1233's table definition at merge time deletes the patch machinery.Production notes
Design and supervision by
Fable 5 (max); implementation, review fixes, and text pass byOpus 5 (max)subagents (commit trailers on the implementation commits name the harness-configured Fable string; the code was written by Opus). The review's live experiments: two-session InnoDB snapshot proofs for the cancellation race, and wikitext injection onSpecial:GraphStoresreaching Scribunto from a GET — both now fixed and pinned.