Skip to content

chore(MUATracePlot): temporarily limited key_source for testing/validation#197

Merged
ttngu207 merged 1 commit into
dj-sciops:mainfrom
MilagrosMarin:main
Dec 11, 2025
Merged

chore(MUATracePlot): temporarily limited key_source for testing/validation#197
ttngu207 merged 1 commit into
dj-sciops:mainfrom
MilagrosMarin:main

Conversation

@MilagrosMarin
Copy link
Copy Markdown
Contributor

Hi @ttngu207 @judewerth,

Jude and I discussed the new pipeline extension proposed in #196 to combine LFP and MUA data and manage large data volumes in blocks of sessions for coherence analysis.

The next key step we agree on is to manually insert a 1-hour block of MUAEphysSession data so we can verify that the processing time for MUASpikes and the relevant metrics in Channel are feasible. We do not need to process the MUATracePlot data at this point, as it is the bottleneck in the pipeline regarding resources and time.

(1) In this PR, I modify the MUATracePlot key_source to limit plot generation, since that appears to be the current bottleneck for block analysis and data processing and storage. Once this PR is merged, I will rebuild the standard worker.

(2) After completing that step, I will inform Jude so he can manually insert the 1-hour block into MUAEphysSession (since this table is to be included in the worker flow to prevent analyzing all current data).

(3) We will review this together before moving on to larger block sizes, as it is critical not to overload the system with many jobs that could impact performance.

(4) If we agree that processing the 1-hour test is time-efficient, we can then consider running a full day of data for MUASpikes analysis to assess how the pipeline performs with larger datasets.

Based on these performance tests, we will decide how best to structure the new pipeline extension outlined in #196.

@MilagrosMarin MilagrosMarin changed the title chore(MUATracePlot): temporarily update key_source to a high limit for testing/validation chore(MUATracePlot): temporarily limited key_source for testing/validation Dec 10, 2025
Copy link
Copy Markdown
Contributor

@ttngu207 ttngu207 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR title is temporarily limited key_source for testing/validation - but the changes do not seem to have any effect on key_source

@ttngu207 ttngu207 merged commit 6aa2e6d into dj-sciops:main Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants