fix(project): map Project.encounters as a Set to stop PROJECT_ENCOUNTERS IDX race#1597
Open
JasonWildMe wants to merge 1 commit into
Open
fix(project): map Project.encounters as a Set to stop PROJECT_ENCOUNTERS IDX race#1597JasonWildMe wants to merge 1 commit into
JasonWildMe wants to merge 1 commit into
Conversation
…ERS IDX race Project.encounters was a JDO ordered List, so the PROJECT_ENCOUNTERS join table carried an IDX ordering column with PK (ID_OID, IDX). DataNucleus derives the next IDX from the (eagerly cached — jdoconfig cache.collections.lazy=false) collection snapshot, so when two transactions write membership for the same project with overlapping lifetimes, both compute the same index and the second INSERT dies with "duplicate key violates PROJECT_ENCOUNTERS_pkey". On zebra this failed bulk imports at row 0 (Project.addEncounter) with the generic "task failed to process" message. Project membership has no meaningful order, so this maps it as an unordered Set: - Project.encounters: List<Encounter> -> Set<Encounter> (LinkedHashSet); JDO then omits the IDX column and keys the join on (ID_OID, CATALOGNUMBER_EID). No package.jdo change needed — the Java field type drives List-vs-Set in DataNucleus. - addEncounter rejects null-id encounters (Encounter.hashCode() is random for a null catalogNumber) and relies on Set.add() dedup; Encounter inherits id-based equals() from Base, consistent with its id-based hashCode(), so dedup is correct and O(1). - getEncounters() keeps returning List<Encounter> (a detached copy) so callers/JSPs that assign to List or index it are unaffected; no caller mutates the result. - Add Project.containsEncounter() (O(1) Set membership) and switch the hot membership-check callers (ProjectUpdate, ProjectGet, iaResultsAnnotFeed.jsp) to it instead of allocating a full copy via getEncounters().contains(). - ProjectTest: id-based dedup, null-id rejection, remove-by-id, copy semantics, batch. Requires a one-time DB migration per instance BEFORE deploying (the old List code needs IDX; the new Set code never writes it) — see docs/plans/2026-06-01-project-encounters-list-to-set.md: stop old WAR, backup, drop PROJECT_ENCOUNTERS_pkey, drop IDX, add PK (ID_OID, CATALOGNUMBER_EID), add reverse index on CATALOGNUMBER_EID, then start the new WAR. Project.users has the identical List+join pattern and the same latent race; left as a follow-up to keep this change focused. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1597 +/- ##
=======================================
Coverage 51.17% 51.17%
=======================================
Files 308 308
Lines 12073 12073
Branches 3919 3910 -9
=======================================
Hits 6178 6178
- Misses 5586 5593 +7
+ Partials 309 302 -7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bulk imports on zebra failed at row 0 with the generic "Your task failed to process due to an error." The real cause (from the container log):
Project.encountersis a JDO orderedList, soPROJECT_ENCOUNTERScarries anIDXordering column with PK(ID_OID, IDX). DataNucleus derives the nextIDXfrom the project's collection snapshot, which is eagerly cached (jdoconfig: cache.collections.lazy = false). When two transactions write membership to the same project with overlapping lifetimes, both compute the same index and the secondINSERTcollides. The loser rolls back, leaving the table gap-free — which is exactly what production showed (COUNT=6075, MIN=0, MAX=6074, contiguous → race, not corruption).This does not require two simultaneous bulk imports: any membership writers (another import, the detection/IA results path, a manual "add to project",
StandardImport) whose transactions overlap will collide, because the index is computed from a stale cached snapshot rather than at flush time.Fix
Project membership has no meaningful order, so map it as an unordered
Set, removing theIDXcolumn and the entire class of collision.Project.encounters:List<Encounter>→Set<Encounter>(LinkedHashSet). JDO then omitsIDXand keys the join on(ID_OID, CATALOGNUMBER_EID). Nopackage.jdochange needed — the Java field type drives List-vs-Set in DataNucleus 5.2.7.addEncounterrejects null-id encounters (Encounter.hashCode()is random for a nullcatalogNumber) and relies onSet.add()dedup.Encounterinherits id-basedequals()fromBase, consistent with its id-basedhashCode(), so dedup is correct and O(1).getEncounters()still returnsList<Encounter>(a detached copy), so callers that assign toListor index it are unaffected; no caller mutates the result.Project.containsEncounter()(O(1) membership); hot membership-check callers (ProjectUpdate,ProjectGet,iaResultsAnnotFeed.jsp) switched to it instead of copying the whole collection to call.contains().ProjectTest: id-based dedup, null-id rejection, remove-by-id, copy semantics, batch dedup.The old (List) code needs
IDX; the new (Set) code never writes it — they must not run against the table at once. Stop the old WAR, migrate, then start the new WAR. Full SQL + prechecks + rollback indocs/plans/2026-06-01-project-encounters-list-to-set.md:(Prechecks: no NULL
CATALOGNUMBER_EID, no duplicate(ID_OID, CATALOGNUMBER_EID).)Testing
mvn test -Dtest=ProjectTest→ 6/6 pass (compiles + JDO enhancement).PROJECT_ENCOUNTERS_pkeyerror.Review
Design and implementation both reviewed by Codex (separate passes). Design review surfaced 5 findings (deploy-stop-first, migration prechecks/element-index/0-based rollback, null-id hashCode guard, copy-vs-allocation, ordering claim) — all incorporated. Implementation review: Go, only two LOW non-blocking notes (unit test can't prove schema/concurrency without staging; a pre-existing null-assumption that isn't a regression).
Follow-ups (out of scope)
Project.usershas the identicalList+join pattern and the same latent race.ImportTask.ERRORSso this kind of failure is visible without log-diving.🤖 Generated with Claude Code