try to bring in trackhubs - #10
Open
erikenbody wants to merge 270 commits into
Open
Conversation
rename create_intervals to create_db_intervals per #114
bump bugfix
mosdepth 0.3.4 was compiled without --d4. Fix to 0.3.3 until resolved.
fix: resource issues #114
Fix mosdepth to 0.3.3 to resolve --d4 issue
Was unable to test the environment from the docs. Change ./test/ecoli to .test/ecoli (or ./.test/ecoli), and include --use-conda
At the moment, the read group string is set by the "run" column. This means that the same library sequenced on multiple flow cells gets treated separately when duplicates are marked, even though duplicate marking happens after merging reads. The correct usage of the read group would set a library string and then the full RG string would be the same value for one library sequenced across multiple lanes. This should lead to GATK and Sentieon both marking duplicates according to the LibraryName column, while retaining the utility of the run column processing each run separately before bam merging.
update read group naming in common.smk
it could be useful to add all this info to the github index too, for lazy people who don't want to click the docs link
I suggest someone reviews my suggested changes to the table
Update index.md
Update setup.md
Currently bcftools removes mtDNA rather than the chromosomes specified by chr_ex
scaffolds_to_exclude is now required
Add scaffolds_to_exclude param in QC
Add links to Globus data collection and the manuscript in the readme.
Update README.md
update libname in read group so that it returns the string, not dictionary object
pin python to 3.10 to fix pyd4 issue
Update cov_filter.yml
Update common.smk
add some doc stuff
add ploidy options
…ormalization Bugfixes, test data update
Add standalone QC runner script
* Fix dead main workflow targets * Fix non-interval GATK and URL reference paths * Preserve sample IDs in GenomicsDB mapfiles * Validate callable-sites coverage for gVCF-only inputs * Remove dead main-workflow helpers and legacy scripts * Update QC bcftools pin to 1.23 * Fix external gVCF mapfile test config --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* Implement callable sites BED generation * add unit tests and fixtures (#294) * add unit tests and fixtures * update gitattrs * update setup env task * fix conftest * ci: run parallel unittest * add pytest-split; update actions * test: prune fixtures to minimal set * test: add sample0 raw bam fixtures * ci: update actions to split full run test * add zarr tests --------- Co-authored-by: Tim Sackton <timsackton@gmail.com> Co-authored-by: Cade Mirchandani <cademirch@gmail.com>
* fix: Add example project directory with symlinks to test fixtures Lets users run a quick test with --directory example/ without needing to set up their own data or risk overwriting CI fixtures. * cleanup: delete old .test dir
Updated documentation links and added a note about Snakemake version.
* Remove retired modules and harden QC modules * Add standalone postprocess runner * Remove retired module tests * Fix numeric QC fixture generation * Remove retired module shards from CI
* Fix ffq fallback for SRA downloads * Pin sra-tools to 3.2.1 --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* Preserve numeric-like sample identifiers * Fix QC numeric ID preservation and add regression test
Co-authored-by: Tim Sackton <timsackton@gmail.com>
Co-authored-by: Tim Sackton <timsackton@gmail.com>
* add debug info * Remove duplicate rule all from debug branch --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* Avoid sorting raw coverage BED intervals * Test direct coverage BED merge parity --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* Add complexity-aware DB interval splitting * Auto-merge pathological GenomicsDB contig shards --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* Handle callable sites for mixed gVCF samples
* Move GATK memory resources into profile * Make resource dry-run tests version tolerant --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* remove redundant tabix lines from deepvariant rule * Clean GLnexus scratch DB before joint call * Bump DeepVariant container to 1.10 * Set GLnexus memory budget from profile * Remove runtimes from variant resource defaults * Resolve GLnexus memory budget after resources * Treat PLINK half-calls as missing in QC --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
* bai -> csi migration * Use explicit CSI indexes for GATK * resolve merge conflicts * Restore explicit GATK BAM CSI index handling * Fix long-contig GATK interval gather * Sort long-contig staged GATK interval VCFs --------- Co-authored-by: Tim Sackton <timsackton@gmail.com>
…#334) * Apply GATK hard filters to GATK-lineage callers; fix filtering wiring In v2.1 the GATK hard filters (get_gatk_hard_filter_args / rule variant_filtration) were never applied in a default run: rule all targeted RAW_VCF, FILTERED_VCF was only reachable via the non-default call_variants target, and both the postprocess and qc modules consumed RAW_VCF. Their `bcftools view -f .,PASS` steps therefore ran on an all-'.' FILTER column and were silent no-ops, so no hard filtering ever occurred. This wires hard filtering back in, caller-appropriately: - Hard filtering runs in the main workflow for GATK-lineage callers only (gatk, sentieon, parabricks), which emit the GATK annotations the filters need. bcftools and DeepVariant do not emit those annotations, so their raw VCF is the final call set. common.smk gains GATK_LINEAGE_TOOLS / APPLY_HARD_FILTERS / FINAL_VCF. - results/vcfs/filtered.vcf.gz (raw + FILTER column) is produced once by the main workflow and shared by postprocess and qc (no double compute). - New variant_calling.generate_filtered_vcf (default true) makes the filtered VCF a default product alongside raw (v1-like). It requires a GATK-family caller and raises a clear config error otherwise; set it false for bcftools/deepvariant or for a raw-only default. - Postprocess is slimmed: the retained results/postprocess/filtered.vcf.gz is now the strict-filtered output, the basic filter is a temp intermediate, and filtered.TEMP is removed. New modules.postprocess.filtering.split_by_type (default true) gates the SNP/indel split; keep_basic_filter (default false) can retain the basic VCF. Index handling is now long-contig-mode-aware. - QC consumes the same shared filtered VCF, so its FILTER-based steps are meaningful for GATK-lineage callers. Tests and docs updated accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: warn+auto-disable for non-GATK; drop dead vars - generate_filtered_vcf with a non-GATK caller now logs a warning and auto-disables instead of raising, so existing bcftools/deepvariant configs keep working on upgrade (raw VCF is their final call set). Docs, schema, and config comment updated to match. - Remove unused FINAL_VCF_INDEX and POSTPROCESS_KEEP_BASIC from common.smk. - Drop the now-vestigial test helper that injected generate_filtered_vcf:false for non-GATK tool swaps (auto-disable handles it); rewrite the dedicated test to assert the warn-and-disable path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Tim Sackton <timsackton@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- outputs: BAM index is .csi (bai->csi migration, #326); add external BAM path; note VCF/gVCF index type depends on long_contig_mode - cli: correct default resources (mem_mb 8000 / mem_mb_reduced 7000) and rule names in the thread table (download_sra, bwa_mem, markdup_library, mosdepth, clam_collect, sentieon_dedup_library); note per-rule GATK memory now lives in the profile (#325) - config-schema: document generate_filtered_vcf, long_contig_mode, the new interval sharding keys (#321), qc.max_sample_missingness (#312), and postprocess split_by_type / keep_basic_filter; refresh full example - modules: add max_sample_missingness and split_by_type/keep_basic_filter - tutorials/how-to: fix stale rule names and resource values - install: recommend Snakemake 9.12
Co-authored-by: Tim Sackton <timsackton@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.