Skip to content

External Annotation

Jump to bottom
Dave Lawrence edited this page Jul 7, 2026 · 2 revisions

Running VEP annotation on a separate (non-VM) machine, and reusing the annotated VCFs.

VEP is the slow, heavy step in annotation. External annotation lets you decouple it from a VariantGrid server: dump the unannotated variants + a ready-to-run Snakemake workflow, run VEP wherever you have compute, then import the annotated VCFs back. The annotated VCFs can also be reused between a database and its own clone, so you don't run VEP twice for the same annotation version.

Everything is driven by one management command: annotation_external.

See issue #1568.

When to use this

  • Offload VEP to a bigger/cheaper box than the VG server.
  • Clone reuse — the driving scenario: you upgrade sapath test (a clone of prod) which needs a full annotation rerun, then upgrade prod. Instead of running VEP again on prod, you reuse the annotated VCFs produced during the test run.

Reuse is only supported between a database and its own clone, for the same annotation version. A clone starts with identical Variant primary keys, so the origin variant ids baked into the dump line up 1:1 with the target. Reuse across unrelated databases, or across regions where the ids have diverged, is not supported — the id-alignment check rejects those by failing the import.

How it works

The normal pipeline already splits into dump → VEP → import. External annotation stops after the dump, parks the run in a new Awaiting external annotation state, and resumes at import once you supply the annotated VCF.

                dump                    (off-VM)                 import
  Variants  ─────────►  input VCF   ──► snakemake/VEP ──►  annotated VCF  ─────────►  VariantAnnotation
             + meta.json + Snakefile                                        + §6a check + upload

Each dumped run writes three things into the output dir:

  • …run{pk}…standard.vcf — the unannotated variants in range (variant_id baked into INFO).
  • ….meta.json — sidecar metadata: version identity + the range's min/max variant coordinate strings. Matching on import is driven by this file, not the filename.
  • Plus, once per dump, a Snakefile + config.yaml (see Snakemake bundle).

Avoiding the scheduler: use a NEW annotation version

The normal annotation scheduler only ever operates on the latest ACTIVE VariantAnnotationVersion. So the safe way to run an external dump is against a NEW version that the scheduler will not touch:

  1. Create the new version and leave it NEW: 
    python3 manage.py create_new_variant_annotation_version --genome-build GRCh38
  2. Run the external dump/import against it (below).
  3. Only promote it to ACTIVE once external annotation has finished importing.

(External runs also carry an external=True marker as a belt-and-braces guard, but the NEW status is the primary mechanism.)

v1 scope

  • Small variants only (STANDARD pipeline type). Structural variants stay on the normal in-VM pipeline — there are few of them and they annotate quickly. --pipeline-type defaults to S.
  • The external dump must use the target's ANNOTATION_VEP_BATCH_MIN / ANNOTATION_VEP_BATCH_MAX. Range locks are deterministic in Variant.pk order + batch size, so a clone produces identical [min,max] boundaries — which is what lets import match files back to runs. Get more parallelism from VEP --fork/concurrency across runs, not from bigger batches (that would shift every boundary).

Dump

Creates all range locks + runs for the NEW version up front, writes each input VCF + meta.json, parks them in Awaiting external annotation, and emits the Snakemake bundle:

python3 manage.py annotation_external --dump \
    --genome-build GRCh38 \
    --output-dir /data/external_annotation/grch38_vep116

Dumping already-scheduled runs

Use --dump-existing to adopt CREATED runs that already exist, mark them external and dump them — instead of --dump creating every run up front. It adopts runs the dispatcher hasn't leased yet, and --leave N keeps the N lowest runs on the in-VM pipeline so both machines work in parallel.

--vav-status selects which version's CREATED runs to adopt (default NEW):

  • NEW (default, and the safe path) — first hit Run scheduler against NEW on the annotation runs page to create the runs. The dispatcher won't touch a NEW version, so nothing races you while you dump.

    python3 manage.py annotation_external --dump-existing \
    --genome-build GRCh38 \
    --output-dir /data/external_annotation/grch38_vep116

  • ACTIVE — offload pending runs from the version the scheduler is currently annotating. --leave N keeps the low end on the in-VM pipeline so both machines chew through it in parallel:

    python3 manage.py annotation_external --dump-existing \
    --vav-status ACTIVE \
    --genome-build GRCh38 \
    --leave 20 \
    --output-dir /data/external_annotation/grch38_offload

The Snakemake bundle

The dump writes a self-contained Snakefile + config.yaml into the output dir. Copy the whole directory to the compute box, then:

  1. Edit config.yaml so the paths point at that machine's VEP install + annotation data:

    dump_dir: "."
output_dir: "annotated"

# Point these at THIS box. The tree under annotation_base_dir must mirror the
# VG server's annotation_data layout (VEP cache, plugins, fasta, plugin data).
vep_code_dir: "/data/annotation/VEP/vep_code/116/ensembl-vep"
vep_cache_dir: "/data/annotation/VEP/vep_cache"
vep_plugins_dir: "/data/annotation/VEP/vep_code/116/plugins"
annotation_base_dir: "/data/annotation"

vep_fork: 8          # tune for this box
vep_buffer_size: 1000

  2. Run it:

    cd /data/external_annotation/grch38_vep116
snakemake --cores 8

The Snakefile discovers work by globbing *.meta.json, and runs the exact VEP command VariantGrid would have run (from get_vep_command()), with {input}/{output} and every server path swapped for config.yaml values. Because the command is byte-identical, the ##VEP= header on the output matches on import. Output lands in output_dir as …standard.vep_annotated.vcf.gz.

The compute box needs the same VEP version + annotation data (cache, plugins, fasta, plugin data files) as the VG server. Only the paths to them differ, via config.yaml.

Import

Copy the annotated VCFs and their .meta.json sidecars back, then:

python3 manage.py annotation_external --import \
    --genome-build GRCh38 \
    --input-dir /data/external_annotation/grch38_vep116/annotated \
    --dry-run        # optional: list matches/skips without importing

For each annotated VCF the command:

  1. Reads the sidecar meta.json.
  2. Matches it to a local VariantAnnotationVersion (version identity) and a local run Awaiting external annotation whose range min/max coordinate strings exactly equal the meta's.
  3. Runs the variant-id alignment check.
  4. Copies the VCF into ANNOTATION_VCF_DUMP_DIR and runs the normal upload-only path (which re-checks the ##VEP= header and inserts the annotation), setting the run to Finished.

--dry-run reports what would match without importing. A file that matches nothing (e.g. produced against a diverged database) is skipped with a warning, and that run just falls back to a normal VEP run.

Safety: the variant-id alignment check

The variant_id baked into a dump's INFO is the origin database's Variant.id. On import to a clone we trust those ids only after checking they still line up. Before importing each run, the command verifies that the local variant at the range lock's min_variant_id and max_variant_id has the coordinate the annotated file recorded. Range locks are contiguous blocks of Variant.pk, so if both endpoints align the interior does too.

  • On a pre-split clone → endpoints resolve to the recorded coordinates → pass.
  • A file from a diverged region or the wrong database → at least one endpoint shifts → that run is marked Error (with an actionable message) and skipped. The rest of the import continues.

This is a per-run failure, never a whole-import abort — you re-run any skipped run normally.

Clone reuse end-to-end (the sapath scenario)

  1. test VM: annotation_external --dump --genome-build GRCh38 --output-dir DIR (against a NEW version).
  2. Copy DIR to the compute box, edit config.yaml, snakemake.
  3. Copy annotated VCFs (+ metas) back; annotation_external --import on the test VM → runs Finished. Promote the version to ACTIVE.
  4. Keep the annotated VCFs + metas.
  5. prod (a clone needing the same annotation version): copy back only the annotated files whose id range predates any divergence, and run annotation_external --import on prod. The alignment check guarantees every imported run's ids line up; anything that doesn't is skipped and falls back to a normal VEP run.

See also

Clone this wiki locally