Curated clustering heavy load#25
Closed
Risker202 wants to merge 81 commits into
Closed
Conversation
integrate with the latest version of SpikeInterface
add get_sync_ephys_function for ephys sync & remove the irrelevant ephys modules
feat: add LFP ingestion for Trellis (ns2)
Pull from upstream - `SIExport` dedicated to `phy export`
Pull from upstream
add extractors import
fix: recording_extractor_full_dict is deprecated (SpikeInterface/spikeinterface#3153)
Update element-interface requirement to `staging` branch
pull from upstream
Pull from upstream
feat(sorting): add `File` to store result files as `filepath@store`
One-by-one file insert to avoid transaction timeout when inserting large files
…element-array-ephys into main_external-storage
…nei_nienborg_element-array-ephys into main_external-storage
Pull from `main`
…nei_nienborg_element-array-ephys into main_external-storage
For SI-processed data, Kilosort's spike_sites contains device_channel_indices (sequential 0, 1, 2...) which differ from the original channel_idx in EphysRecording.Channel (e.g., 32-63 for insertion dj-sciops#2 on MBA probes). The fix: 1. Fetches sorting_analyzer files from external storage (S3) if not local 2. Uses the sorting_analyzer's probe to correctly map channel indices to electrodes via device_channel_indices and contact_ids 3. Falls back to original behavior for non-SI processed data Follows the same pattern as CuratedClustering.make() in ephys.py. Fixes: dj-sciops/nei_nienborg#111 (Issue dj-sciops#1) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…apping Fix KeyError in ApplyOfficialCuration.make for multi-insertion probes
Pull from `main`
…ration The key passed to make() only contains the Clustering PK (subject, session_datetime, insertion_number, paramset_idx). curation_id is a dependent attribute of OfficialCuration, not part of its PK. Without this fix, ManualCuration.File & key matches ALL curations for that Clustering key, causing incorrect file resolution when multiple manual curations exist. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…key-resolution Fix ApplyOfficialCuration key resolution for multiple curations
Pop channel_idx from electrode dicts in the SI code path, consistent with how the fallback path handles it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nel-idx Fix channel_idx in electrode_map for ApplyOfficialCuration
Add file infer_map.py, which contains the main function infer_map. Future commits will implement a preprocessing method that uses this function.
Unlike other preprocessing methods, this one takes additional arguments. A future commit will alter the structure of clustering param sets to make this possible.
…xtra preprocessing parameters Clustering param sets can now contain either the SI_PREPROCESSING_METHOD field, a new SI_PREPROCESSING_PARAMS field, or both. Preprocessing method can be submitted either as SI_PREPROCESSING_METHOD or as SI_PREPROCESSING_PARAMS > preprocessing_method. Remaining entries in SI_PREPROCESSING_PARAMS are forwarded as kwargs to the preprocessing function
This should allow running MBA_infer_map preprocessing method with no kwargs
Add infer map functionality for microwire brush arrays as a preprocessing method
…many units Split CuratedClustering.make into 3-part make. Split insert transaction into groups of 64 units each.
Author
|
wrong branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Calls to CuratedClustering.make with many units (~700) raise error:
LostConnectionError: Connection was lost during a transaction.This commit splits the CuratedClustering.make call into make_fetch, make_compute, and make_insert, and also splits the make_insert transaction into several chunks.