Node-free variant grid export (CSV) for an arbitrary list of variants

[davmlaw]

🤖 Written by Claude

Goal

Make it possible to export a VariantGrid analysis-style grid (CSV, with all the configured annotation columns and identical formatting) for an arbitrary list/queryset of variants, without creating an Analysis or AnalysisNode.

Today the only way to produce a fully-formatted variant grid export is through the analysis grid machinery, which is hard-bound to an AnalysisNode. To export "a dozen variants" outside an analysis (e.g. from search results, a variant-tag list, a gene page) you currently have to spin up a throwaway Analysis + AllVariantsNode, restrict its queryset, run the export, then delete it. That works but has real costs: DB writes (Analysis + AnalysisNode + NodeVersion rows per export), a required user, audit-log noise, and fragility (it relies on AllVariantsNode being queryable while unloaded).

We want a clean, node-free path that produces output byte-identical to the existing analysis CSV export for the same set of columns.

Background — why this is feasible

The coupling to Analysis/AnalysisNode is almost entirely incidental. The actual export pipeline has no intrinsic need for a node — VariantGrid just happens to read all its config off node.analysis.

The core pieces are already node-free:

• Columns + overrides: snpdb/grid_columns/custom_columns.py::get_custom_column_fields_override_and_sample_position(ccc, annotation_version, analysis_tags=...) — pure function of (CustomColumnsCollection, AnnotationVersion).
• Base queryset: annotation/annotation_version_querysets.py::get_variant_queryset_for_annotation_version(annotation_version) — installs the SQL partition transformer so variantannotation__* joins resolve to the correct partition. (This is not an ordinary FK filter — you MUST start from this queryset or annotation columns will not join correctly.)
• Extra annotations: snpdb/grid_columns/custom_columns.py::get_variantgrid_extra_annotate(user) — needs a user, not a node (provides tags_global, internally_classified, etc.).
• colmodels / server-side formatters: built by the jqgrid base class from self.fields + self._overrides — driven by data the grid already holds, not by the node.
• CSV writing: library/django_utils/jqgrid_view.py::grid_export_csv(colmodels, items) — already node-free.
• Export-only row fixups: analysis/grid_export.py::format_items_iterator(sample_ids, items, variant_tags_dict) — only needs sample_ids (which is [] when there are no samples), plus each row's tags_global and id.

Current coupling to untangle

In analysis/grids.py:

• VariantGrid.__init__(self, user, node, extra_filters=None, af_show_in_percent=None) (line ~52) reads from node.analysis: genome_build, custom_columns_collection, annotation_version, default_sort_by_column, grid_sample_label_template, plus node.get_cohorts_and_sample_visibility(), node.get_extra_columns(), node.get_extra_grid_config(), NodeCount.load_for_node(...).
• VariantGrid._get_base_queryset() (line ~83) returns self.node.get_queryset().
• VariantGrid._get_fields_and_overrides(node, af_show_in_percent) (line ~154) — already mostly delegates to get_custom_column_fields_override_and_sample_position(...); the node-specific parts are the extra columns and sample/genotype columns.
• VariantGrid.get_colmodels(...) (line ~101) injects {"analysisNode": {"visible": node.visible}} into every colmodel — analysis-only.
• VariantGrid._get_q() (line ~109) — NodeCount/extra-filters (diff-to-parent) logic — analysis-only.
• ExportVariantGrid (line ~306) — paginates contig-at-a-time via node.analysis.genome_build; otherwise generic.

In analysis/grid_export.py:

• node_grid_get_export_iterator(request, node, export_type, ...) (line ~23) hardcodes ExportVariantGrid(request.user, node, ...) and pulls node.get_sample_ids() / node.analysis.genome_build.

Classification of each coupling

Incidental (drop / default for the node-free path):

• get_colmodels analysisNode injection
• _get_q / NodeCount extra-filters
• default_sort_by_column / grid_sample_label_template (these originate from UserSettings via the analysis — read them from UserSettings.get_for_user(user) directly)
• get_extra_columns() / get_extra_grid_config() (default empty)
• node.get_queryset() DAG filtering (replaced by an explicit base queryset / pk__in)

Genuinely context-bound — but to a cohort/sample, NOT an analysis:

• Sample/genotype columns (zygosity, AD/AF/DP/GQ/PL/FT, CohortGenotype packed-data columns + per-sample visibility), built in VariantGrid.get_grid_genotype_columns_and_overrides(...). This needs a cohort context, which a variant-level export does not have.

Proposed implementation

Scope (first cut)

Variant-level columns only. No sample/genotype columns in this issue — that's the one genuinely context-bound piece and untangling it is out of scope here. Get the node-free path shipped for the clean 90%; sample support can be added later by passing cohorts into the context explicitly.

Approach

1. Introduce a small context object (e.g. VariantGridExportConfig) carrying everything the grid needs, independent of a node:

   • genome_build: GenomeBuild
   • annotation_version: AnnotationVersion (the umbrella AnnotationVersion, which has .variant_annotation_version — NOT a bare VariantAnnotationVersion)
   • custom_columns_collection: CustomColumnsCollection
   • user: User
   • base_queryset: QuerySet[Variant] (already built via get_variant_queryset_for_annotation_version, optionally pre-filtered)
   • af_show_in_percent: bool (default from settings.VARIANT_ALLELE_FREQUENCY_CLIENT_SIDE_PERCENT)
   • order_by: Optional[list[str]]
   • (later) optional cohorts / visibility for sample columns

2. Refactor a node-free base (e.g. BaseVariantGrid) that depends on the context object instead of node. Move the node-free logic (_get_fields_and_overrides minus extra/sample columns, base queryset, extra annotate kwargs, colmodels) onto it.

3. Make the existing VariantGrid a thin adapter that builds the context from a node and adds the node-specific bits (extra columns, sample/genotype columns, analysisNode colmodel injection, _get_q). The existing analysis export must be unchanged.

4. Provide a node-free entry point, e.g.:
   analysis/grid_export.py::variant_list_grid_export(variants_or_qs, annotation_version, ccc, user, export_type='csv') -> tuple[str, Iterator[str]]
   mirroring node_grid_get_export_iterator but driven by the context object. user should default to library.guardian_utils.admin_bot() (note: tag/classification columns are permission-filtered per user).

5. Parameterize node_grid_get_export_iterator / format_items_iterator off the context where they currently reach for node (e.g. sample_ids becomes [] for the variant-list case).

Reference prototype

export_analysis.py - issue comment - already contains a working fake-analysis implementation (get_annotated_variants_csv, get_annotated_variant_rows) that produces correctly-formatted output by creating a throwaway Analysis + AllVariantsNode. Use it as the behavioural reference / oracle for what correct output looks like — the new node-free path should produce identical output. It is NOT the desired end state (the whole point is to remove the fake analysis); treat it as a spec, then delete it once the real path lands.

Acceptance criteria

• A public function exists that takes a list (or queryset) of Variant, an AnnotationVersion, a CustomColumnsCollection, and a user, and returns formatted CSV lines (header first) — with no Analysis/AnalysisNode/NodeVersion rows created or deleted, and no FakeRequest/grid-handler request needed.
• For a shared variant-level column set, the output is byte-identical to the existing analysis grid CSV export over the same variants (header row + data rows + server-side formatting + the -1 → "." and tag-summarising fixups from format_items_iterator).
• The existing analysis grid export (the node path) is unchanged — verified by existing tests still passing.
• A new test asserts the node-free export matches the analysis export byte-for-byte on a shared column set, using annotation/tests/test_data_fake_genes.py / analysis/tests/utils.py::AnalysisSetupMixin style fixtures. This test is required — the two paths will silently drift without it.

Out of scope (note as follow-ups)

• Sample/genotype columns in the node-free path (needs cohort context passed explicitly).
• VCF export via the node-free path (CSV first; VCF can follow the same context pattern afterward).

Key files

• analysis/grids.py — VariantGrid, ExportVariantGrid (the refactor target)
• analysis/grid_export.py — node_grid_get_export_iterator, format_items_iterator, grid_export_csv usage
• snpdb/grids.py — AbstractVariantGrid (base get_queryset/field-name logic, ~line 499)
• snpdb/grid_columns/custom_columns.py — get_custom_column_fields_override_and_sample_position, get_variantgrid_extra_annotate
• annotation/annotation_version_querysets.py — get_variant_queryset_for_annotation_version (partition transformer)
• library/django_utils/jqgrid_view.py — grid_export_csv
• library/jqgrid/jqgrid.py — base get_items / iter_format_items / get_colmodels

[davmlaw]

Here is the export_analysis.py code:

"""
Export an arbitrary list of variants as a formatted VariantGrid analysis grid (CSV),
*outside* of an analysis - the same output you'd get from the analysis-grid CSV download.

The grid formatting (column model, server-side formatters, the -1 -> "." / tag
summarising done by format_items_iterator, and the CSV writing) is all bound to
analysis.grids.ExportVariantGrid, which requires a real AnalysisNode. So we spin up a
throwaway Analysis + AllVariantsNode (restricting the grid's base queryset to the
supplied variants), drive the real export pipeline, then delete the throwaway analysis.

This mirrors analysis.grid_export.node_grid_get_export_iterator exactly, except:
  - the grid's base queryset is filtered to the supplied variant IDs
  - there's no web request (we use a FakeRequest with admin_bot by default)

Typical use:

    av = AnnotationVersion.latest(genome_build)          # or analysis.annotation_version
    ccc = CustomColumnsCollection.get_system_default()   # or any collection
    for line in get_annotated_variants_csv(variants, av, ccc):
        sys.stdout.write(line)

Or get the formatted rows (list of dicts, keyed by grid field name) instead of CSV:

    rows = get_annotated_variant_rows(variants, av, ccc)
"""
from __future__ import annotations  # PEP 563 - so list[...]/tuple[...] annotations work on Python 3.8

from typing import Iterator, Optional

from django.contrib.auth.models import User
from django.db.models import QuerySet

from analysis.grid_export import format_items_iterator
from analysis.grids import ExportVariantGrid
from analysis.models import Analysis, AllVariantsNode, NodeStatus
from annotation.models import AnnotationVersion
from library.django_utils import FakeRequest
from library.django_utils.jqgrid_view import grid_export_csv
from library.guardian_utils import admin_bot
from snpdb.models import Variant, CustomColumnsCollection


class _VariantListExportGrid(ExportVariantGrid):
    """ ExportVariantGrid whose base queryset is restricted to a fixed list of variant IDs """

    def __init__(self, *args, variant_ids=None, **kwargs):
        self._variant_ids = variant_ids or []
        super().__init__(*args, **kwargs)

    def _get_base_queryset(self) -> QuerySet:
        return super()._get_base_queryset().filter(pk__in=self._variant_ids)


def _build_grid_colmodels_and_rows(variants: list[Variant],
                                   annotation_version: AnnotationVersion,
                                   ccc: CustomColumnsCollection,
                                   user: User) -> tuple[list[dict], list[dict]]:
    """ Creates a throwaway Analysis + AllVariantsNode, runs the real grid export pipeline
        restricted to 'variants', and returns (colmodels, formatted_rows).

        Rows are materialised (so safe for a small list - a dozen variants) and the
        throwaway analysis is deleted before returning. """

    variant_ids = [v.pk for v in variants]

    analysis = Analysis(genome_build=annotation_version.genome_build,
                        annotation_version=annotation_version,
                        custom_columns_collection=ccc,
                        user=user)
    analysis.save()
    try:
        node = AllVariantsNode(analysis=analysis)
        node.save()
        # Mark the node "ready" the way the load pipeline would (count + status)
        node.count = node.get_queryset().filter(pk__in=variant_ids).count()
        node.status = NodeStatus.READY
        node.save()

        request = FakeRequest(user=user)
        grid = _VariantListExportGrid(user, node, None, variant_ids=variant_ids)

        # get_items() applies the grid's server-side formatters (iter_format_items)
        _, _, items = grid.get_items(request)
        sample_ids = node.get_sample_ids()
        # format_items_iterator does the export-only fixups (-1 -> ".", tag summarising)
        rows = list(format_items_iterator(sample_ids, items))
        colmodels = grid.get_colmodels()
    finally:
        analysis.delete()  # cascades the throwaway node

    return colmodels, rows


def get_annotated_variant_rows(variants: list[Variant],
                               annotation_version: AnnotationVersion,
                               ccc: CustomColumnsCollection,
                               user: Optional[User] = None) -> list[dict]:
    """ Returns formatted grid rows (list of dicts keyed by grid field name, e.g.
        "variantannotation__gnomad_af") for the supplied variants - the same values
        the analysis grid CSV export would write, but as dicts.

        'annotation_version' is the umbrella AnnotationVersion (has
        .variant_annotation_version), NOT a bare VariantAnnotationVersion. """
    if user is None:
        user = admin_bot()
    _colmodels, rows = _build_grid_colmodels_and_rows(variants, annotation_version, ccc, user)
    return rows


def get_annotated_variants_csv(variants: list[Variant],
                               annotation_version: AnnotationVersion,
                               ccc: CustomColumnsCollection,
                               user: Optional[User] = None) -> Iterator[str]:
    """ Yields CSV lines (header first) for the supplied variants - byte-identical to the
        analysis grid CSV download (column headers, ordering and server-side formatting).

        'annotation_version' is the umbrella AnnotationVersion (has
        .variant_annotation_version), NOT a bare VariantAnnotationVersion. """
    if user is None:
        user = admin_bot()
    colmodels, rows = _build_grid_colmodels_and_rows(variants, annotation_version, ccc, user)
    return grid_export_csv(colmodels, rows)