Mashpit

Mashpit is a lightweight, local genomic epidemiology platform for rapidly comparing a query genome against representative genomes from NCBI Pathogen Detection SNP clusters.

Mashpit is designed for offline or resource-limited analysis, including deployment on small computers such as a Raspberry Pi. It combines compact genomic sketches, epidemiological metadata, and tree-guided representative selection to support fast preliminary genome screening without requiring a centralized computing service.

Mashpit 1.0 is under development. The database-building workflow described below reflects the new tree-based design and may differ from the latest packaged release.

Key features

• Rapid local queries: Compare an assembled genome against a compact database of representative pathogen genomes.
• Tree-based representative selection: Select representatives directly from NCBI Pathogen Detection SNP trees rather than relying on a single centroid for every cluster.
• Adaptive database size: Use more representatives for large or genetically diverse SNP clusters and fewer representatives for compact clusters.
• Optional radius control: Limit the maximum tree distance between cluster members and their selected representatives.
• Robust assembly retrieval: Exclude records without assembly accessions, retry failed genome downloads, and remove persistently unavailable representatives.
• Integrated metadata: Return epidemiological information such as isolation date, location, host, BioSample accession, assembly accession, and SNP-cluster identifiers.
• Local visualization: Produce a query result table and a tree showing the relationship between the query and its closest database representatives.
• Offline operation: After a database has been built, routine queries can be performed without uploading genomic data.
• Custom databases: Build a database directly from local FASTA files, with no NCBI download at all - useful for proprietary or unpublished sequence collections.
• Extensible metadata: Add your own custom columns to any existing database - taxon, accession, or custom - with a built-in, per-database change history.

Mashpit 1.0 database design

An NCBI Pathogen Detection taxon database contains many SNP clusters. A single representative is not sufficient for every cluster because cluster size and within-cluster diversity vary substantially.

Mashpit 1.0 therefore uses an adaptive tree-based strategy:

1. Download the selected NCBI Pathogen Detection metadata and SNP trees.
2. Remove records that do not have a valid assembly accession.
3. Determine the number of representatives required for each SNP cluster from its tree structure.
4. Select representative leaves that cover the cluster, optionally subject to a user-defined tree radius.
5. Download the selected assemblies.
6. Retry failed downloads for a limited number of attempts.
7. Remove representatives whose assemblies remain unavailable.
8. Build the sketch database and retain metadata linking every representative to its SNP cluster.

This approach reduces redundancy in compact clusters while preserving multiple genomic backgrounds in large or diverse clusters.

Installation

Option 1: Conda or Mamba

The latest stable release can be installed from Bioconda:

conda create -n mashpit -c conda-forge -c bioconda mashpit
conda activate mashpit

Option 2: pip

Mashpit requires the NCBI Datasets command-line tool for downloading assemblies.

For Linux:

curl -o datasets \
  'https://ftp.ncbi.nlm.nih.gov/pub/datasets/command-line/v2/linux-amd64/datasets'
chmod +x datasets
export PATH="$PATH:/path/to/datasets"

Install the stable release:

python -m pip install mashpit

Install the development version from source:

git clone https://github.com/tongzhouxu/mashpit.git
cd mashpit
python -m pip install .

Verify the installation:

mashpit --help

Database contents

A Mashpit database is stored in a directory containing the sketch database and its metadata. Depending on the database type, it includes files such as:

DATABASE_NAME/
├── DATABASE_NAME.db               # sqlite metadata and representative tables
├── DATABASE_NAME.sig              # merged sourmash signatures
├── representatives.tsv            # taxon databases only
├── cluster_summary.tsv            # taxon databases only
├── unavailable_assemblies.tsv     # taxon/accession databases: accessions that could not be downloaded
└── unavailable_sequences.tsv      # custom databases: local FASTA files that failed to sketch

The build log itself (mashpit-<timestamp>.log) is written to the current working directory alongside the database folder, not inside it.

Keep the entire database directory together when moving it to another computer.

Building a database

Mashpit supports three database types:

• Taxon database: Build from NCBI Pathogen Detection SNP clusters for a selected organism.
• Accession database: Build from a user-provided list of NCBI BioSample accessions.
• Custom database: Build directly from local FASTA files, with no NCBI download at all.

Taxon database

mashpit build taxon salmonella --species Salmonella

A specific NCBI Pathogen Detection release can be selected with --pd_version:

mashpit build taxon listeria_innocua \
  --species Listeria_innocua \
  --pd_version PDG000000091.9

Supported taxon directory names are listed in the NCBI Pathogen Detection results directory.

Accession database

Prepare a text file containing one BioSample accession per line:

SAMN00000001
SAMN00000002
SAMN00000003

Then run (an Entrez email is required to resolve BioSample accessions via NCBI):

mashpit build accession custom_database --list biosamples.txt --email you@example.com

Custom database

Build a database directly from local FASTA files - no NCBI download, Entrez lookup, or datasets CLI is required for this database type, which makes it suitable for proprietary or unpublished sequence collections. Place one genome per file, named <sample_id>.fasta (.fa/.fna/.fas/.ffn, optionally gzipped), in a directory:

my_genomes/
├── sample1.fasta
├── sample2.fasta
└── sample3.fasta

Then run:

mashpit build custom my_database --input-dir my_genomes/

Optional metadata can be supplied as a TSV with a sample_id column (matching the filenames) plus any subset of the usual metadata fields (strain, collection_date, geo_loc_name, host, etc.) and/or your own custom columns; fields not supplied, or samples not listed, are recorded as missing:

sample_id   strain    collection_date   host
sample1     StrainA   2024-03-01        Homo sapiens

mashpit build custom my_database --input-dir my_genomes/ --metadata metadata.tsv

A custom database has no SNP-cluster concept, so it behaves like an accession database at query time: no cluster-candidates table, and --radius/tree-based representative selection do not apply (every provided genome becomes a database entry directly).

Tree-radius option

Mashpit 1.0 can optionally use a tree-radius threshold during representative selection, via --radius (default 20, in the same units as the SNP tree's branch lengths). A smaller radius generally selects more representatives and gives denser coverage of within-cluster diversity. A larger radius selects fewer representatives and produces a smaller database:

mashpit build taxon salmonella --species Salmonella --radius 10

Use the radius option only after evaluating an appropriate value for the organism and intended surveillance resolution.

Assembly download handling

Before representative selection and download, Mashpit removes rows without valid assembly accessions. Selected assemblies that fail to download are retried automatically. After the configured retry limit, unavailable assemblies are removed from the representative set and recorded in the build log.

This prevents missing assemblies from producing invalid database entries.

Annotating a database

mashpit annotate adds or updates custom metadata columns on a database you've already built - taxon, accession, or custom - since all three share the same underlying metadata table. This is the place to attach your own project-specific fields (an internal lab ID, a project or batch code, an outbreak investigation code, anything not in Mashpit's built-in schema) without rebuilding the database.

To see which assembly/BioSample accessions are actually in the database - the ids your values file needs to match - print them as a starting point:

mashpit annotate my_database --list-accessions > accessions.tsv

Then annotate:

mashpit annotate my_database --values project_metadata.tsv

project_metadata.tsv needs an asm_acc or biosample_acc column to match existing rows, plus whatever columns you want to add or update:

asm_acc            project_batch   internal_lab_id
GCA_000000001.1    Batch01         LAB-001

Columns not already present in the database are added automatically. Rows in the database not mentioned in your file keep their existing values (or are recorded as missing if the column is new); ids in your file with no matching database row are reported, not treated as an error.

Every mashpit annotate run is recorded in the database itself, so you always have a record of what changed and when:

mashpit annotate my_database --history

This works on databases built with older Mashpit versions too - the history table is created automatically the first time you annotate a database that doesn't have one yet, so nothing needs to be rebuilt.

Querying a database

Run a query using an assembled genome in FASTA format:

mashpit query sample.fasta path/to/database

Common query options include:

--number                 maximum number of database matches returned
--threshold              minimum similarity used when constructing the local result tree
--annotation             metadata field used to annotate tree tips
--tie-tolerance-hashes   sketch-hash tolerance used to flag near-top SNP clusters (taxon databases only)

Example:

mashpit query sample.fasta path/to/salmonella \
  --number 200 \
  --threshold 0.85 \
  --annotation isolation_source

Query output

A query produces:

1. Representative-matches table

   A CSV file (<query>_representative_matches.csv) containing the closest representatives and associated metadata. Results include fields such as BioSample accession, assembly accession, SNP-cluster accession, similarity score, and NCBI links.

2. Cluster-candidates table (taxon databases only)

   A CSV file (<query>_cluster_candidates.csv) that groups the representative matches by SNP cluster, ranked by best similarity score. A near_top column flags clusters whose best hit is within --tie-tolerance-hashes sketch hashes of the single best hit overall, since at typical sketch sizes a small similarity gap can be within normal MinHash sampling noise rather than a meaningful difference.

3. Local tree

   Newick and image files (<query>_tree.newick, <query>_tree.png) showing the query genome together with its closest database representatives above the similarity threshold. The tree is skipped (with a log message) when the top hit falls below --threshold or fewer than two candidates qualify.

4. Log file

   A timestamped record of the query parameters and processing steps.

Mashpit results are intended for rapid screening and prioritization. They do not replace validated SNP-pipeline, cgMLST, or whole-genome phylogenetic analyses when formal outbreak confirmation is required.

Graphical interface

Start the local Mashpit interface with:

mashpit gui

This launches a Streamlit app and opens it in your browser, by default at:

http://localhost:8501

Use --port to run on a different port:

mashpit gui --port 8888

The interface lets users select a local database, upload a query assembly, run the search, and inspect the result table and tree - all without the genome or results leaving the local machine.

A pre-built Mashpit database is required.

Intended use

Mashpit is intended for fast, lightweight genomic screening, including:

• preliminary placement of newly sequenced isolates;
• identification of the most relevant NCBI Pathogen Detection SNP clusters;
• local analysis of sensitive genomic data;
• decentralized surveillance in laboratories with limited computing infrastructure;
• prioritization of isolates for higher-resolution analysis.

Mashpit should not be used as the sole evidence for declaring or excluding an outbreak relationship.

Citation

Xu T. et al. (2024). Mashpit: sketching out genomic epidemiology.
Journal of Open Source Software, 9(104), 7306.
https://doi.org/10.21105/joss.07306

Contributing

See CONTRIBUTING.md for development and contribution guidelines.

License

Mashpit is distributed under the GNU General Public License v2.0. See LICENSE.