Migration plan: PR #211 + element #65 + reorg — pre-merge fixes, schema reorganization, and DB migration

[MilagrosMarin]

Migration plan: PR #211 + element #65 + reorg PR

Coordinates merging three PRs into production via one combined database migration:

• PR Integrate tools for final analysis  #211 (workflow) — Jude's integrate
• element Add a notebook for exploring spike sorting results #65 (utah_organoids_element-array-ephys) — element schema additions
• Reorg PR — to be opened from refactor/pipeline-reorganization after PR Integrate tools for final analysis  #211 merges

Approach. PR #211 + element #65 merge code-only (no DB changes). Workers pause for affected tables until the reorg PR lands. The reorg PR ships the schema reorganization plus all remaining fixes. One maintenance window then runs the full DB migration.

---

Pipeline visualizations

Current (production main)

Final (post-reorg)

---

Phase overview

#	Phase	Owner	DB action
1	Merge PR #211 + element #65 (with §1 fixes applied)	@judewerth	None
2	Pause workers for affected tables	@MilagrosMarin	None
3	Open and merge the reorg PR (with §2 fixes applied)	@MilagrosMarin	None
4	Run the combined DB migration (§5) in one maintenance window	@MilagrosMarin	All DB changes
5	Verify and resume workers	@MilagrosMarin	None

---

§1 — Pre-merge schema-shape fixes (PR #211 + element #65) — @judewerth

These fixes change table shapes; they must land before the reorg PR is opened (preferred) or be carried into the reorg PR (fallback).

• 1.1 Merge judewerth/utah_organoids_element-array-ephys#2 into judewerth:main (lazy imports, probe-type-restricted map_channel_to_electrode, LFP single-file boundary fix, ImpedanceMeasurements logic order)
• 1.2 Drop the ephys. prefix on ImpedanceFile (-> EphysRawFile) and STTC (-> CuratedClustering) FKs in ephys_no_curation.py
• 1.3 Change FrameSession.frame_param_idx: int → -> TimeFrameParamset in src/workflow/pipeline/frame.py
• 1.4 Change BurstSession parent from EphysSession → FrameAnalysis.ActiveTimeFrames in src/workflow/pipeline/mua.py
• 1.5 Rebase integrate onto main (currently 7 commits behind)

---

§2 — Reorg PR preparation — @MilagrosMarin

• 2.1 Carry forward any §1 fixes that didn't land upstream
• 2.2 Add TimeFrameParamset → FrameParamset rename (table-level rename; FK on FrameSession updated)
• 2.3 Decide drop+repopulate vs rename per moved table — coordinate with @judewerth. Tables with very few rows (e.g. EventBasedCoupling 0, BurstDetectionParamset 1, TensorpacParamset 1, PhaseAmplitudeBands 4, TraceSession 5) are simpler to drop and repopulate than to rename. Tables with significant data (e.g. LFPSpectrogram ~241k, STTFA ~363, CouplingSession ~512) must be renamed to preserve data. Output: a partition of §5d's table list.
• 2.4 Rebase refactor/pipeline-reorganization onto main after PR Integrate tools for final analysis  #211 merges
• 2.5 Open the reorg PR
• 2.6 Resolve open questions before merging:
  • Fate of frame schema's 33 legacy dev-artifact tables (archive or drop)
  • Whether any moved tables use ~external storage → migrate corresponding ~external entries
  • FK constraint behavior post-rename — validate in staging
  • FrameParamset FK target update — confirm DataJoint re-targets automatically, or update the FK manually

---

§3 — Pre-populate STTC fixes — @judewerth or @MilagrosMarin

.make() body changes only (no schema impact). Required before populating high-firing-rate sessions; otherwise out-of-memory.

• 3.1 Replace O(n²) np.subtract.outer in STTC.make() with np.searchsorted-based binary search (drafted in element Add a notebook for exploring spike sorting results #65 inline comment)
• 3.2 Use float milliseconds for neo.SpikeTrain input (drop the .astype(int) cast)

Gate: do not populate STTC on any session with max firing rate > ~100 Hz until §3.1 and §3.2 land.

---

§4 — Worker freeze — @MilagrosMarin

Active from PR #211 merge until the reorg PR merges.

• 4.1 Pause all workers (full freeze, including ephys ingestion and spike sorting)
• 4.2 Coordinate with @judewerth to keep the freeze window short

---

§5 — Combined DB migration — @MilagrosMarin

Single maintenance window after the reorg PR merges. Detailed commands kept in internal notes (not inlined here).

5a — Backup

• Save OrganoidImplantationImage rows (image attachments + num_electrodes_inside values, pulled from the existing NumElectrodesInside lookup if it has data)
• Save FOOOFAnalysis rows (if any)
• Save schema definitions (no data) for rollback: analysis, mua, frame, culture

5b — Drop tables with shape changes (cascade handles downstreams)

• culture.OrganoidImplantationImage — parent + new column
• analysis.FOOOFAnalysis (+ .FBOSCAnalysis part) — PK change
• frame.FrameSession — FK constraint added (cascades FrameAnalysis + .ActiveTimeFrames)
• mua.BurstSession — parent change (cascades PopulationBursts)

5c — Create new schemas

DataJoint creates these automatically when the reorg PR code is imported, but creating them up-front simplifies FK handling during the rename block:

• {DB_PREFIX}lfp_analysis
• {DB_PREFIX}coupling
• {DB_PREFIX}burst
• {DB_PREFIX}selection

5d — Move tables (rename or drop+repopulate per §2.3)

Disable FK checks for this block; re-enable in §5h.

Rename (preserves data):

• All remaining analysis tables → lfp_analysis (incl. LFPSpectrogram ~241k, ~log, ~external if present)
• Coupling tables: analysis → coupling (STTFA, CouplingSession, PhaseAmplitudeCoupling + part, etc. — see §2.3 for the final list)
• mua.BurstDetectionParamset → burst.BurstDetectionParamset
• mua.TraceSession → selection.TraceSession
• frame.TimeFrameParamset → selection.FrameParamset

Drop and let repopulate (per §2.3 decision): small or empty tables — recreated empty when the reorg PR code activates, then repopulated by workers.

5e — Clean up orphans

• Drop culture.NumElectrodesInside (absorbed into OrganoidImplantationImage.num_electrodes_inside)
• Handle frame schema's 33 legacy dev-artifact tables per §2.6 decision

5f — Deploy the reorg PR code

DataJoint recreates dropped tables with their new shapes in the new locations:

• culture.OrganoidImplantationImage (new parent + column)
• lfp_analysis.FOOOFAnalysis (new PK)
• selection.FrameSession (FK to FrameParamset; ActiveTimeFrames part recreated empty on first populate())
• burst.BurstSession (parent = selection.FrameAnalysis.ActiveTimeFrames)

5g — Re-insert OrganoidImplantationImage rows

• For each organoid: map old OrganoidCulture parent → corresponding Control Experiment, restore image attachment, fill num_electrodes_inside from the §5a backup

5h — Re-enable FK checks and verify (see §7)

---

§6 — Code/config notes (ship with PR merges)

• New deps: bottleneck, neo, quantities, elephant, specparam (git: fooof-tools), tensorpac (git: EtienneCmb)
• DataJoint pin: datajoint>=0.14,<2.0
• Lookup seed rows auto-populate via contents=[...] on schema activation
• ⚠️ num_electrodes_inside must be set on OrganoidImplantationImage for every organoid before FrameAnalysis or PopulationBursts can populate (both raise ValueError if missing)

---

§7 — Verification

• All four new schemas exist; {DB_PREFIX}analysis is empty
• Renamed tables show pre-migration row counts (lfp_analysis.LFPSpectrogram ~241k, coupling.STTFA ~363, coupling.CouplingSession ~512, coupling.PhaseAmplitudeCoupling ~512)
• FK constraints intact on sentinel tables: burst.BurstSession, selection.FrameSession, lfp_analysis.Coherence
• selection.FrameAnalysis.populate() succeeds on one organoid (with num_electrodes_inside set)
• burst.PopulationBursts.populate() succeeds on one active frame
• STTC.populate() succeeds on one low-firing session (skip high-firing until §3 lands)
• Dashboard images_implantation page renders for all organoids
• No new ~jobs error entries in the 24 hours after migration
• Workers resumed; next populate cycle runs cleanly

---

§8 — Downstream updates

Dashboard — @MilagrosMarin

• Update imports: from workflow.pipeline import analysis → from workflow.pipeline import lfp_analysis, coupling, burst, selection
• Verify pages/images_implantation.py renders correctly post-migration (page already updated for the new OrganoidImplantationImage parent in commit e99732e)

Notebooks — @judewerth

• Decide fate of the four notebooks deleted in review: EXPLORE_frame_analysis, CREATE_new_culture_session, CREATE_new_frame_session, ANALYZE_single_session
• Update imports in remaining CREATE_* and EXPLORE_* notebooks for new schema locations
• Spot-check ANALYZE_single_organoid.ipynb (78k lines) for schema compatibility

[judewerth]

Hi @MilagrosMarin ,

Honestly, a lot of this went over my head but I think an important decision is regarding wether to rename the data or drop and repopulate. I think in general dropping and re-populating won't be an issue. With the exception of spike sorting, spectrograms, and the one-minute longitudinal tables, processing is very fast.

On that note, spectrograms are particularly problematic (as you know) because they take up so much room. This makes it difficult for storage and increasing the time it takes to fetch them (large fetches take hours). This can be easily helped by reducing the frequency range that we store from 1250 Hz (nyquist) to 300 Hz (end of LFP).

If we decide to drop and repopulate, there are several small changes I'd prefer to make to improve the pipeline. The most important change (attached) is how we store probe information. Currently it is set up so for each EphysSession you need to make a new EphysSessionProbe and manually insert the probe information. It makes far more sense to attach the probes to Experiments. 1) The probe doesn't change across time points within an experiment. 2) If the analyzer is different from the experimenter, we won't have to worry about wrong information being inserted. 3) Ease of use for the analyzer. 4) Less data being stored.

Please let me know your thoughts and also what you'd like from me to help this re-organization.

Best,
Jude