External project details readsets tab by nabalma · Pull Request #1181 · c3g/freezeman

nabalma · 2026-08-17T13:40:21Z

Note :

Please review PR #1180 first, if you haven't already.

This PR adds the Readsets tab. It follows the Project Submissions tab PR.

…alizer

nafiz1001 · 2026-08-18T14:51:37Z

You might have forgotten to add a dependency to package.json

ERROR in ./src/components/projectOverview/ExternalIDReadSetDashboard.tsx 17:0-44
Module not found: Error: Can't resolve '@ant-design/charts' in '/home/user/projects/freezeman/frontend/src/components/projectOverview'

Fixed with npm install @ant-design/charts

nafiz1001 · 2026-08-18T15:29:01Z

+ACTIVE_PARENT_PROJECT_READSET_FILTERS = {
+    "deleted": False,
+    "dataset__deleted": False,
+    "dataset__project__deleted": False,
+    "dataset__project__parent_project__deleted": False,
+    "dataset__experiment_run__deleted": False,
+    "dataset__experiment_run__run_type__deleted": False,
+    "dataset__experiment_run__run_type__platform__deleted": False,
+    "derived_sample__deleted": False,
+    "derived_sample__biosample__deleted": False,
+    "derived_sample__biosample__individual__deleted": False,
+    "derived_sample__library__deleted": False,
+    "derived_sample__library__library_type__deleted": False,
+    "derived_sample__derived_by_samples__deleted": False,
+    "derived_sample__derived_by_samples__sample__deleted": False,
+    "derived_sample__derived_by_samples__sample__container__deleted": False,
+}


I would call this DEFAULT_PARENT_PROJECT_READSET_FILTERS

And I also believe this should be moved inside get_parent_project_readsets_queryset

A small note : The deleted flag is typically used to save a last version in reversion before deleting an object for good. This means there should be no "deleted" objects in the DB except in the reversion tables. These filter will likely slow the query without much to offer.

Thanks. I removed the redundant deleted=False filters ... Fixed.

nafiz1001 · 2026-08-18T15:30:50Z

+PARENT_PROJECT_READSET_ORDERING = [
+    "dataset__experiment_run__start_date",
+    "dataset__experiment_run__name",
+    "dataset__lane",
+    "id",
+]
+
+PARENT_PROJECT_READSET_VALUE_FIELDS = [
+    "id",
+    "name",
+    "average_quality",
+    "pf_reads_aligned",
+    "duplicate_aligned",
+    "readset_file_paths",
+    "readset_file_sizes",
+    "barcodes",
+    "validation_status",
+]
+
+PARENT_PROJECT_READSET_VALUE_ALIASES = {
+    "readset_sample_name": F("sample_name"),
+    "external_id": F("dataset__project__parent_project__external_id"),
+    "run_name": F("dataset__experiment_run__name"),
+    "lane": F("dataset__lane"),
+    "reference_genome_id": F("derived_sample__biosample__individual__reference_genome_id"),
+    "sequencing_index_name": F("derived_sample__library__index__name"),
+    "run_start_date": F("dataset__experiment_run__start_date"),
+    "alias": F("derived_sample__biosample__alias"),
+    "cohort": F("derived_sample__biosample__individual__cohort"),
+    "library_type": F("derived_sample__library__library_type__name"),
+    "number_of_reads": F("production_data__reads"),
+}


These should be moved inside get_parent_project_readsets_queryset unless they have a use outside the function.

nafiz1001 · 2026-08-18T15:32:29Z

Having codes in backend/fms_core/queries/ is stepping outside the usual pattern for creating queryset in Freezeman.

They are normally achieved with services or viewsets or filters.py or serializers.py

Nafiz is right, but i honestly find your solution tempting ... The main downside i find to it is that these queries are likely not going to be reused outside a specific viewset and in the end it would just force code reader to move back and forth between code sections (one of the things i dislike the most about our frontend ;). I would tend to recommend we move these query within the viewset itself.

Agreed. I moved the parent project readsets queryset logic into ParentProjectViewSet as a private helper, since it is specific to this action.

nafiz1001 · 2026-08-18T15:33:44Z

Brand new submodules shouldn't be introduced without ~~master's permission~~ @UlysseFG's agreement.

Thanks ..
Addressed by removing the new queries package and moving the action-specific logic into ParentProjectViewSet.

nafiz1001 · 2026-08-18T16:01:49Z

+    const total = items.length || 1
+


It's strange to always be needing at least total of 1.

nafiz1001 · 2026-08-18T16:03:49Z

+    return {
+        complete: Math.round((complete / total) * 100),
+        incomplete: Math.round((incomplete / total) * 100),
+        completeCount: complete,
+        incompleteCount: incomplete,
+    }


Ah, that's why you wanted total of at least 1. I feel like it would be better instead to do something like

complete: Math.round((complete / Math.max(total, 1)) * 100), incomplete: Math.round((incomplete / Math.max(total, 1)) * 100),

or

complete: total == 0 ? 0 : Math.round((complete / total) * 100), incomplete: total == 0 ? 0 : Math.round((incomplete / total) * 100),

I support this approach it is less risky if we ever decide to use total elsewhere. 0 is also a very valid value when not used as a denominator... All projects will have 0 readset at some point.

Updated to keep the actual total and protect the percentage calculations when total is 0.

nafiz1001 · 2026-08-18T16:04:44Z

+        const total = readsets.length || 1
+


Similar suggestion I've made for frontend/src/components/projectOverview/ExternalIDReadSetDashboard.tsx

Also fixed ....

nafiz1001 · 2026-08-18T16:07:00Z

This is corrections that were supposed to be done on the PR that introduced those code.

nafiz1001 · 2026-08-18T16:10:01Z

+
+	const projectOverviewReadsetColumns = getProjectOverviewReadsetColumns(libraryTypeFilters)
+
+	if (isLoading) {


getProjectOverviewReadsetColumns result should be memoized.

UlysseFG · 2026-08-19T14:50:43Z

+            totalReadsets: readsets.length,
+            totalReads: readsets.reduce((sum, x) => sum + Number(x.number_of_reads || 0), 0),
+            totalRuns: new Set(readsets.map((x) => x.run_name)).size,
+            totalSamples: new Set(readsets.map((x) => x.readset_sample_name)).size,


sample name is not unique. You should not trust it for counting independant samples. You would be better to count biosample_id. It is likely to be mostly the same but there might be case where this will make a difference.

Thanks a lot ... This is fixed ....

UlysseFG · 2026-08-19T14:52:47Z

+            totalReads: readsets.reduce((sum, x) => sum + Number(x.number_of_reads || 0), 0),
+            totalRuns: new Set(readsets.map((x) => x.run_name)).size,
+            totalSamples: new Set(readsets.map((x) => x.readset_sample_name)).size,
+            totalCohorts: new Set(readsets.map((x) => x.cohort)).size,


This is a metric that will likely not be very useful. Projects usually don't have many cohorts and the number of cohort is not an indicator for much of anything. You can leave it, but keep in mind if you ever need to free some space on the UI that this is a good candidate for replacement.

UlysseFG · 2026-08-19T15:31:52Z

+                "files__file_path",
+                filter=Q(files__deleted=False),distinct=True,default=Value([]),
+            ),
+            readset_file_sizes=ArrayAgg(


It would be better to combine each file with its size in a single cell instead of having to rely on matching position in a list. I admit this would not render well in a CSV but the list are likely to break the regularity of the structure.

Updated to return and render readset files as a single list of objects containing both file_path and size,

UlysseFG · 2026-08-19T15:38:10Z

+                filter=Q(files__deleted=False),distinct=True,default=Value([]),
+            ),
+            barcodes=ArrayAgg(
+                "derived_sample__derived_by_samples__sample__container__barcode",


I think there might still be some confusion between sample barcodes and container barcodes ... I would be very surprised we have to report container barcode anywhere in the project overview unless it is run container barcodes... At the readset level i think the barcode you might have seen in the original report on Nanuq would map to the index (index is sometime called barcode by some people making this more confusing than it needs to be ;).

you have the library on derived_sample__library__index__name and the sequences is available also at derived_sample__library__index__sequences_3prime and derived_sample__library__index__sequences_5prime

Seeing the container barcodes there might actually be of some use if there is someone investigating the readset to check if there was some mixup ... It would be important though to mention that this is container_barcodes.

Updated to keep the barcode information and expose it as container_barcodes .

UlysseFG · 2026-08-19T15:53:57Z

+
+
+    @action(detail=True, methods=["get"])
+    def readsets(self, request, pk=None):


maybe call this overview_readsets to reduce the chance people might confuse what is sent back to normal readsets.

action renamed

UlysseFG · 2026-08-19T17:31:06Z

+    )
+    readset_file_paths = serializers.ListField(required=False)
+    readset_file_sizes = serializers.ListField(required=False)
+    validation_status = serializers.IntegerField(allow_null=True,)


Maybe call this run_validation_status instead to make it clear it is the flag that is assigned when validating the run. The library tab and sample will likely get a validation/qc_status flag also that will be differ from this one.

UlysseFG · 2026-08-19T17:36:27Z

+        const grouped = new Map<string, number>()
+
+        readsets.forEach((item) => {
+            const libraryType = item.library_type?.trim() || 'Non renseigné'


Might be better not to mix language in there. This is something that should essentially never happen (all sequenced sample should have a library_type... This may change but this would essentially be something wrong that would need investigating on our part.). For now you can use "unknown" or "sample without library".

UlysseFG · 2026-08-19T18:15:28Z

+            totalRuns: new Set(readsets.map((x) => x.run_name)).size,
+            totalSamples: new Set(readsets.map((x) => x.readset_sample_name)).size,
+            totalCohorts: new Set(readsets.map((x) => x.cohort)).size,
+            avgQuality: readsets.reduce((sum, x) => sum + Number(x.average_quality), 0) / total,


Averaging averages based on a variable number of bases is likely to skew the value... The correct way would be to use a weighted average using the number of bases for each readset . The number of bases is typically stored on the yield metrics field. I think some run do not provide this information you can assume an equal number of bases in this case.

Fixed. I updated the average quality calculation to weight each readset by its yield (number of bases). The calculation falls back to an unweighted average when that information is incomplete.

UlysseFG · 2026-08-19T18:28:26Z

+    "alias": F("derived_sample__biosample__alias"),
+    "cohort": F("derived_sample__biosample__individual__cohort"),
+    "library_type": F("derived_sample__library__library_type__name"),
+    "number_of_reads": F("production_data__reads"),


This is not the best place to fetch the reads ... The production_data is extracted only once a run is validated and is not live... It is aggregated daily making this a bad source unless you want the have the reads that passed validation only... In our case, the source should be the readset metric nb_reads.

Fixed. number_of_reads is now retrieved from the readset nb_reads metric instead of production_data.

UlysseFG · 2026-08-19T18:30:44Z

+            totalSamples: new Set(readsets.map((x) => x.readset_sample_name)).size,
+            totalCohorts: new Set(readsets.map((x) => x.cohort)).size,
+            avgQuality: readsets.reduce((sum, x) => sum + Number(x.average_quality), 0) / total,
+            avgAlignment:


For alignment and duplication you are averaging averages again ... ideally we would do a weighted average using the number of reads of the readset.

Fixed. I updated both the alignment and duplication calculations to use weighted averages based on each readset’s number of reads.

UlysseFG · 2026-08-19T18:32:05Z

+
+                <Col xs={24} sm={12} lg={6} xl={3}>
+                    <Card size="small" styles={{ body: { padding: '8px 12px' } }}>
+                        <Statistic title="Cohortes" value={metrics.totalCohorts} />


UlysseFG · 2026-08-19T18:38:24Z

+					size="small"
+					bordered
+					scroll={{ x: 'max-content', y: 400 }}
+					pagination={{


It would be great to have the number of hits listed in the pagination.

addressed...

UlysseFG · 2026-08-19T18:40:09Z

+		onHeaderCell: compactHeaderCell,
+		filters: libraryTypeFilters,
+		filterIcon: (filtered) => <FilterOutlined style={{ color: filtered ? '#1677ff' : undefined }} />,
+		onFilter: (value, record) => record.library_type === value,


Nice touch that filter.

UlysseFG · 2026-08-19T18:40:34Z

+		render: (name: string) => <Text strong>{name}</Text>,
+	},
+	{
+		title: 'Sample',


Would be great to have a filter on that column too.

UlysseFG · 2026-08-19T18:55:29Z

+					<ProjectOverviewExportButton data={exportButtonData} />
+				</div>
+			)}
+			{!isLoading && isActive && <ExternalIDReadSetDashboard readsets={projectOverviewReadsets} />}


I would like to test having the dashboard at the very top of the tab with the export button just above the table... This would allow for a clear filters button to be inserted also next to the export button.

Updated. The dashboard is now displayed at the top of the tab, with the export button above the table.

UlysseFG · 2026-08-19T19:08:27Z

+				.includes(String(value).toLowerCase()),
+	},
+	{
+		title: 'Run Start',


A filter here might be nice...

Date range filter added to the Run Start column.

UlysseFG · 2026-08-19T19:34:44Z

+
+	const exportButtonData: ProjectOverviewExportButtonData = {
+		exportType: 'Project Readsets',
+		exportFunction: generateCsvContent,


Same comment as for the project... we typically send a request to the backend to export only the currently select items... Your csv builder is much more efficient than the package we are using ... I tested the big project MOH-YR5_CRCHUM ... The initial load time is quite large but the frontend filtering and export are much faster that way ... I am afraid that this might lead to some projects being undisplayable because the request times out... We will be adding more metrics to the query and this will slow down the call and some projects can run for years with potentially thousands of readsets ... Maybe it would be better to consider the overview_readsets to have its own endpoint to be able to use a similar behaviour as elsewhere. I am however curious if we could not improve our csv generation on the backend side.

That makes sense. As discussed, could we address this in a follow-up ?

UlysseFG · 2026-08-19T19:39:24Z

+            <Row gutter={[16, 16]} style={{ margin: 8, borderRadius: 4 }}>
+                <Col xs={24} sm={12} lg={6} xl={3}>
+                    <Card size="small" styles={{ body: { padding: '8px 12px' } }}>
+                        <Statistic title="Readsets" value={metrics.totalReadsets} prefix={<ExperimentOutlined style={iconStyle('#1677ff', '#e6f4ff')}/>} />


Numbers displayed should not use the , to separate thousands. You can use a space instead.

Fixed ...Numeric values now use spaces instead of commas as thousands separators in both the dashboard statistics and the readsets table.

UlysseFG · 2026-08-19T19:40:11Z

+
+                <Col xs={24} sm={12} lg={6} xl={3}>
+                    <Card size="small" styles={{ body: { padding: '8px 12px' } }}>
+                        <Statistic title="Samples" value={metrics.totalSamples} prefix={<TeamOutlined style={iconStyle('#13c2c2', '#e6fffb')} />} />


This will likely go to the thousands too so you should also replace the ,.

UlysseFG · 2026-08-19T19:56:42Z

+    "external_id": F("dataset__project__parent_project__external_id"),
+    "run_name": F("dataset__experiment_run__name"),
+    "lane": F("dataset__lane"),
+    "reference_genome_id": F("derived_sample__biosample__individual__reference_genome_id"),


This is not really informative without getting the assembly_name. This would need to be fetched separately from the store on the frontend or directly sent back (derived_sample__biosample__individual__reference_genome__assembly_name).

Updated. The backend now returns the reference genome assembly name

… function scope

…iewSet

…dSetDashboard

…elated components

…er_barcodes and readset_files and sizes into json

…wsets, and frontend components

…shboard

…DReadSetDashboard

…ctReadSetsTab

nafiz1001 · 2026-08-26T19:17:57Z

When I exported on http://localhost:9000/external-projects-overview/193#readsets, the readset_files column shows up like this in the csv:

duplicate_aligned	readset_files	run_validation_status
0.133361	[object Object],[object Object]	1
0.414306	[object Object],[object Object]	1
0.136845	[object Object],[object Object]	1
0.40548	[object Object],[object Object]	1

Thanks. Fixed now ...

…s number of reads

…t button next to the table.

nabalma · 2026-09-03T12:55:42Z

Closing this PR as it has been replaced by #1185, which consolidates the entire stack into a single PR.

nabalma added 4 commits August 10, 2026 15:02

Add parent project readsets backend endpoint

2f31bed

Allow null values for reference_genome_id in ParentProjectReadsetSeri…

b47ac17

…alizer

Adding the Read Sets tab with its dashboard and export button

be33f6d

update to address PR #1179 comments

ea6b29b

nafiz1001 reviewed Aug 18, 2026

View reviewed changes

UlysseFG reviewed Aug 19, 2026

View reviewed changes

nabalma added 14 commits August 20, 2026 16:01

Remove deleted filters from readset query

50669c5

get_parent_project_readsets_queryset constants are defined within the…

1a7ddd6

… function scope

Refactor readsets query logic into a private method in ParentProjectV…

d354169

…iewSet

Remove unused ProjectLibrariesTab and ProjectSamplesTab components

f598afa

Fix division by zero in QC completeness calculations in ExternalIDRea…

37b2a4b

…dSetDashboard

Add biosample_id field to ParentProjectReadsetSerializer and update r…

df0c29f

…elated components

Update ParentProjectReadsetView and related components to use contain…

b23d97f

…er_barcodes and readset_files and sizes into json

Rename readsets action to overview_readsets in ParentProjectViewSet

3e33f5b

Rename validation_status to run_validation_status in serializers, vie…

e605ffb

…wsets, and frontend components

Update library type default value to 'Unknown' in ExternalIDReadSetDa…

71be5a9

…shboard

Use the nb_reads metric for readset read counts

3adc428

Add total count display (number of hits) for readsets in pagination

3d1928e

Fix typo in statistic title from "Cohortes" to "Cohorts" in ExternalI…

971d4c1

…DReadSetDashboard

Add search and filter functionality for readset sample names in Proje…

5dea574

…ctReadSetsTab

nabalma added 8 commits August 27, 2026 17:08

Update the average quality calculation to weight readsets by yield

9f6d61d

Update the average alignment calculation to weight each readset by it…

66cd705

…s number of reads

Update the average duplication calculation to weight readsets

233925d

Move the readset dashboard above the table controls to keep the expor…

eba2531

…t button next to the table.

Add a date range filter to the readset run start column.

cf84f07

Update numeric formatting to use spaces as thousands separators.

334b072

Add reference genome assembly names to project overview readsets.

654119d

Format readset file paths and sizes for readable CSV exports.

461b84f

nafiz1001 mentioned this pull request Aug 29, 2026

Add the first production-ready version of the external project overview #1185

Open

nabalma closed this Sep 3, 2026


		const projectOverviewReadsetColumns = getProjectOverviewReadsetColumns(libraryTypeFilters)

		if (isLoading) {



		@action(detail=True, methods=["get"])
		def readsets(self, request, pk=None):

Uh oh!

Conversation

nabalma commented Aug 17, 2026

Note :

Uh oh!

nafiz1001 commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nafiz1001 Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nafiz1001 Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nafiz1001 Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nafiz1001 Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nafiz1001 Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

UlysseFG Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nafiz1001 commented Aug 18, 2026 •

edited

Loading

nafiz1001 Aug 18, 2026 •

edited

Loading

nafiz1001 Aug 18, 2026 •

edited

Loading

nafiz1001 Aug 18, 2026 •

edited

Loading

nafiz1001 Aug 18, 2026 •

edited

Loading

nafiz1001 Aug 18, 2026 •

edited

Loading

UlysseFG Aug 19, 2026 •

edited

Loading

nabalma Aug 27, 2026 •

edited

Loading

nabalma Aug 27, 2026 •

edited

Loading