fix: migrate CHARLIE runtime activation to mamba-safe flow by kopardev · Pull Request #149 · CCBR/CHARLIE

kopardev · 2026-06-05T21:34:54Z

Summary

This PR updates CHARLIE runtime activation paths to remove Conda-specific activation strings and use a Mamba-safe environment setup flow.

Changes

replaced runtime activation construction in charlie with a Mamba-first command path
preserved platform-specific module loading behavior (biowulf/fnlcr)
switched runtime env marker from CONDA_PREFIX to MAMBA_PREFIX
simplified bin/redirect to delegate directly to main script and stop masking failures

Validation

charlie --help exits cleanly with no shell-init/libmamba warnings
charlie -m=dryrun no longer fails on old conda activation path usage
no conda activate/conda.sh references remain in targeted runtime files

Closes #148

- replace conda activation strings with mamba-first environment setup in runtime paths\n- keep cluster-specific module loading behavior and reuse activation in module load command paths\n- simplify redirect wrapper delegation and stop masking runtime failures\n- use MAMBA_PREFIX in runtime activation environment\n\nRefs: #148\n\n⚡ generated using AI ⚡

for more information, see https://pre-commit.ci

Copilot

Pull request overview

This PR migrates the pipeline entrypoint/runtime activation to a Mamba-safe flow (removing Conda activation strings) and simplifies the wrapper dispatch. It also includes a broad formatting/whitespace pass across many workflow scripts and resource/config files.

Changes:

Updated charlie platform initialization to construct a Mamba-first environment setup and removed Conda activation strings.
Simplified bin/redirect to delegate directly to charlie and stop masking failures.
Reformatted numerous workflow scripts/configs (Python/R/shell/YAML) for consistent indentation (with some scripts still containing logic issues that should be corrected before merge).

Reviewed changes

Copilot reviewed 54 out of 73 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
workflow/scripts/transcript2gene.py	Re-indented; function safety issue remains (uninitialized variable).
workflow/scripts/reformat_hg38_2_hg19.py	Re-indented/refactored prints.
workflow/scripts/merge_ReadsPerGene_counts.R	Whitespace-only cleanup.
workflow/scripts/merge_counts_tables_2_counts_matrix.py	Large formatting pass (argparse/options) without intended logic changes.
workflow/scripts/make_star_index.sh	Minor formatting (line alignment).
workflow/scripts/junctions2readids.py	Formatting + minor string literal normalization.
workflow/scripts/get_index_rl.py	Formatting only.
workflow/scripts/gather_cluster_stats.sh	Minor formatting only.
workflow/scripts/fix_refseq_gtf.py	Large formatting-only pass.
workflow/scripts/fix_gtfs.py	Formatting-only pass; preserves behavior.
workflow/scripts/filter_junction.py	Formatting-only pass.
workflow/scripts/filter_junction_human.py	Formatting-only pass.
workflow/scripts/filter_dcc.py	Formatting-only pass but exposes/retains filtering logic bugs.
workflow/scripts/filter_ciriout.py	Formatting-only pass but exposes/retains filtering logic bugs.
workflow/scripts/filter_bam.py	Formatting-only pass.
workflow/scripts/filter_bam_for_splice_reads.py	Formatting-only pass.
workflow/scripts/filter_bam_for_linear_reads.py	Formatting-only pass.
workflow/scripts/filter_bam_for_BSJs.py	Formatting-only pass.
workflow/scripts/filter_bam_by_readids.py	Formatting-only pass.
workflow/scripts/create_nclscan_per_sample_counts_table.py	Formatting-only pass but retains logic bugs (strand selection + annotation mapping).
workflow/scripts/create_mapsplice_per_sample_counts_table.py	Formatting-only pass.
workflow/scripts/create_dcc_per_sample_counts_table.py	Formatting-only pass.
workflow/scripts/Create_ciri_count_matrix.py	Formatting-only pass.
workflow/scripts/create_circExplorer_per_sample_counts_table.py	Formatting-only pass.
workflow/scripts/Create_circExplorer_count_matrix.py	Formatting-only pass.
workflow/scripts/Create_circExplorer_BSJ_count_matrix.py	Formatting-only pass.
workflow/scripts/circExplorer_get_annotated_counts_per_sample.py	Formatting-only pass.
workflow/scripts/bam_to_bigwig.sh	Minor formatting only.
workflow/scripts/bam_split_by_regions.py	Formatting-only pass.
workflow/scripts/bam_get_max_readlen.py	Formatting-only pass.
workflow/scripts/apply_junction_filters.py	Formatting-only pass.
workflow/scripts/annotate_clear_quant.py	Formatting-only pass.
workflow/scripts/_process_bamtobed.py	Formatting-only pass.
workflow/scripts/_multifasta2separatefastas.sh	Minor formatting only.
workflow/scripts/_merge_circExplorer_found_counts.py	Formatting-only pass.
workflow/scripts/_make_master_counts_table.py	Formatting-only pass (contains no-op `set_index` calls).
workflow/scripts/_filter_linear_spliced_readids_w_rid2jid.py	Formatting-only pass.
workflow/scripts/_compare_lists.py	Formatting-only pass.
workflow/scripts/_collapse_find_circ.py	Formatting-only pass.
workflow/scripts/_circExplorer_BSJ_get_strand.py	Formatting-only pass.
workflow/scripts/_bedpe2bed.py	Formatting-only pass.
workflow/scripts/_bedintersect_to_rid2jid.py	Formatting-only pass.
workflow/scripts/_bamtobed2readendsbed.py	Formatting-only pass.
workflow/scripts/_bam_get_alignment_stats.py	Formatting-only pass.
workflow/scripts/_bam_filter_BSJ_for_HQonly.py	Formatting-only pass.
workflow/scripts/_append_splice_site_flanks_to_BSJs.py	Formatting-only pass.
workflow/scripts/_add_geneid2genepred.py	Re-indented; function safety issue remains (uninitialized variable).
workflow/rules/preprocessing.smk	Removed stray whitespace lines.
workflow/envs/clear.yaml	YAML indentation normalization under `pip:`.
resources/TruSeq_and_nextera_adapters.consolidated.fa	Minor formatting (line alignment).
resources/NCLscan.config.template	Whitespace cleanup.
resources/merge_dataframes.R	Comment whitespace cleanup.
resources/dockers/ccbr_clear/Dockerfile	Comment whitespace cleanup.
resources/collapse_bed_by_names.py	Formatting-only pass.
resources/cluster.json.highmem	Trailing whitespace cleanup.
resources/argparse.bash	Removed trailing blank line.
docs/dryrun_example.txt	Minor formatting (line alignment).
docker/star_ucsc_cufflinks/environment.yml	YAML indentation normalization.
docker/dcc/environment.yml	YAML indentation normalization.
docker/cutadapt_fqfilter/environment.yml	YAML indentation normalization.
docker/circRNA_finder/environment.txt	Minor formatting (line alignment).
docker/bowtie1/environment.txt	Minor formatting (line alignment).
config/samples.tsv.fulltest	Minor formatting (line alignment).
charlie	Runtime activation migration to Mamba-safe PATH-based setup; quoting tweak for eval.
bin/redirect	Simplified wrapper to delegate directly to `charlie` and preserve exit codes.
.tests/lint_workdir/ref/dummy	Removed empty placeholder content (file deletion).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        self.start = int(l[1])
+        self.end = int(l[2])
+        self.nreads = int(l[4])
+        self.size = self.start - self.end


    if out.filter_out == False:
-        out.filter_by_size(host_min=args.host_filter_min,host_max=args.host_filter_max,virus_min=args.virus_filter_min,virus_max=args.virus_filter_max)
+        out.filter_by_size(
+            host_min=args.host_filter_min,
+            host_max=args.host_filter_max,
+            virus_min=args.virus_filter_min,


+        self.start = int(l[2]) - 1
+        self.end = int(l[3])
+        self.nreads = int(l[4])
+        self.size = self.start - self.end


    if out.filter_out == False:
-        out.filter_by_size(host_min=args.host_filter_min,host_max=args.host_filter_max,virus_min=args.virus_filter_min,virus_max=args.virus_filter_max)
+        out.filter_by_size(
+            host_min=args.host_filter_min,
+            host_max=args.host_filter_max,
+            virus_min=args.virus_filter_min,


+    "nclscan_annotation",
+]
+
+minus_strand = resultsfile[resultsfile["strandd"] == "+"]


+outdf["nclscan_annotation"] = (
+    outdf["nclscan_annotation"] + 1
+)  # change 1 to 2 and 0 to 1 ... as 0 is for no annotation


+def get_id(s, whatid):
+    s = s.split()
+    for i, j in enumerate(s):
+        if j == whatid:
+            r = s[i + 1]
+    r = r.replace('"', "")
+    r = r.replace(";", "")
+    return r


+def get_id(s, whatid):
+    s = s.split()
+    for i, j in enumerate(s):
+        if j == whatid:
+            r = s[i + 1]
+    r = r.replace('"', "")
+    r = r.replace(";", "")
+    return r


Update hardcoded base paths from /gpfs/gsfs10/users/CCBR_Pipeliner to /vf/users/CCBR_Pipeliner in runtime/docs/config references to avoid missing-file failures in dryrun. Refs: #150 ⚡ generated using AI ⚡

…ible Use // placeholders in config template so CLI values are rendered during init. Refs: #151 ⚡ generated using AI ⚡

Stop defaulting to ERCC for init runs that do not pass --additives. Update help text to document blank default. Refs: #151 ⚡ generated using AI ⚡

Stop prepending ccr to PARTITION and submit with norm by default. Refs: #152 ⚡ generated using AI ⚡

Set PARTITION explicitly to norm inside biowulf platform block so submission always targets a valid default partition. Refs: #152 ⚡ generated using AI ⚡

The ccr partition does not exist on the biowulf system. Only the norm partition should be used as the default for all rules. ⚡ generated using AI ⚡

Add timestamped progress messages at major create_index checkpoints to improve runtime visibility and troubleshooting for long index jobs. ⚡ generated using AI ⚡

- In main(), check $SIFCACHE (set by module load ccbrpipeliner) before falling back to /data/$USER/.singularity or $WORKDIR/.singularity - Add --singularity-prefix "$SING_CACHE_DIR" to all three snakemake invocations (local, slurm, dry-run/unlock) so Snakemake looks for pre-built SIFs in the cache directory without pulling from Docker Hub Closes #154 ⚡ generated using AI ⚡

Add a commentary block in run() for all modes (local, slurm, dryrun/unlock) that prints key variables to the terminal before the pipeline starts: - Working directory, Snakefile, config file - Singularity cache and bind paths - Platform, cluster profile, partition (where applicable) ⚡ generated using AI ⚡

unlock() was the only runmode that did not call set_singularity_binds before entering run(), leaving SINGULARITY_BINDS empty in the commentary block and in --singularity-args. ⚡ generated using AI ⚡

dryrun() and touch() were also missing the set_singularity_binds call, leaving SINGULARITY_BINDS empty. ⚡ generated using AI ⚡

Replace the flat Python-only bind path logic with a two-layer approach: 1. Walk up parents of WORKDIR and PIPELINE_HOME; if /data/<name> is a symlink resolving to that ancestor, include both the /data/<name> symlink path and the real path (e.g. /data/Ziegelbauer_lab and /vf/users/Ziegelbauer_lab, /data/CCBR_Pipeliner and /vf/users/CCBR_Pipeliner). 2. Keep Python-derived paths only when they are not ancestors (too broad) or descendants (too specific) of the bash-derived explicit paths. Result for a typical biowulf workdir: -B /lscratch,/data/Ziegelbauer_lab,/vf/users/Ziegelbauer_lab,/data/CCBR_Pipeliner,/vf/users/CCBR_Pipeliner ⚡ generated using AI ⚡

⚡ generated using AI ⚡

Copilot

Pull request overview

Copilot reviewed 62 out of 81 changed files in this pull request and generated 9 comments.

+minus_strand = resultsfile[resultsfile["strandd"] == "+"]
+minus_strand = minus_strand[["chrd", "coordd", "coorda", "strandd", "reads", "case"]]


+        if not attributes["gene_id"] in gene_id_2_gene_name:
+            if "gene_name" in attributes:
+                gene_id_2_gene_name[attributes["gene_id"]] = attributes["gene_name"]
+            else:
+                gene_id_2_gene_name["gene_id"] = attributes["gene_id"]


+        self.start = int(l[1])
+        self.end = int(l[2])
+        self.nreads = int(l[4])
+        self.size = self.start - self.end
+        self.filter_out = False


 for l in alllines:
    out = DCC(entry=l)
    out.set_host_additive_virus(regions=regions)
    out.filter_by_nreads(args.back_spliced_min_reads)
    if out.filter_out == False:
-        out.filter_by_size(host_min=args.host_filter_min,host_max=args.host_filter_max,virus_min=args.virus_filter_min,virus_max=args.virus_filter_max)
+        out.filter_by_size(
+            host_min=args.host_filter_min,
+            host_max=args.host_filter_max,
+            virus_min=args.virus_filter_min,
+            virus_max=args.virus_filter_max,
+        )
        if out.filter_out == True:
            outfile.write(l)
 outfile.close()


+        self.chrom = l[1]
+        self.start = int(l[2]) - 1
+        self.end = int(l[3])
+        self.nreads = int(l[4])
+        self.size = self.start - self.end
+        self.filter_out = False


 for l in alllines:
    out = CIRIOUT(entry=l)
    out.set_host_additive_virus(regions=regions)
    out.filter_by_nreads(args.back_spliced_min_reads)
    if out.filter_out == False:
-        out.filter_by_size(host_min=args.host_filter_min,host_max=args.host_filter_max,virus_min=args.virus_filter_min,virus_max=args.virus_filter_max)
+        out.filter_by_size(
+            host_min=args.host_filter_min,
+            host_max=args.host_filter_max,
+            virus_min=args.virus_filter_min,
+            virus_max=args.virus_filter_max,
+        )
        if out.filter_out == True:
            outfile.write(l)
 outfile.close()


+  # Step 4: combine -- EXTRA (platform scratch) + explicit + filtered Python
+  all_binds="${EXTRA_SINGULARITY_BINDS}"
+  [[ -n "$explicit_binds"  ]] && all_binds="${all_binds},${explicit_binds}"
+  [[ -n "$filtered_python" ]] && all_binds="${all_binds},${filtered_python}"
+
+  SINGULARITY_BINDS="-B ${all_binds}"


+def get_id(s, whatid):
+    s = s.split()
+    for i, j in enumerate(s):
+        if j == whatid:
+            r = s[i + 1]
+    r = r.replace('"', "")
+    r = r.replace(";", "")
+    return r


+def get_id(s, whatid):
+    s = s.split()
+    for i, j in enumerate(s):
+        if j == whatid:
+            r = s[i + 1]
+    r = r.replace('"', "")
+    r = r.replace(";", "")
+    return r


…BINDS is empty On unknown/non-HPC platforms EXTRA_SINGULARITY_BINDS is empty, causing the previous concatenation to produce "-B ,/path" which is invalid. Build all_binds incrementally using the same :+ guard used elsewhere. ⚡ generated using AI ⚡

- Document SIFCACHE auto-detection from module load ccbrpipeliner - Note that --additives and --viruses are optional (default: blank) enabling host-only circRNA discovery mode - Fix stale version in tutorial --help output (v0.10.0-dev -> v0.13.0) ⚡ generated using AI ⚡

bam_split_by_regions.py: - make --additives and --viruses optional (default: empty string) - handle empty CSV safely via _split_csv() helper - collect all unknown regions before exiting with a descriptive error _create_circExplorer_linear_bam.v2.sh: - only pass --additives/--viruses to bam_split_by_regions.py when non-empty, enabling host-only runs without crashing the script - use array-based write_split_cmd() for cleaner argument handling Snakefile (on_finish_cmd): - add module load ccbrpipeliner so jobby/spooker are available - fix jobby invocation: pipe snakemake.log via stdin - update command name run_jobby_on_snakemake_log -> jobby - add set -exo pipefail and file existence checks for diagnostics ⚡ generated using AI ⚡

kopardev and others added 2 commits June 5, 2026 17:32

[pre-commit.ci] auto fixes from pre-commit.com hooks

e0cc74a

for more information, see https://pre-commit.ci

kopardev requested a review from Copilot June 5, 2026 21:54

kopardev self-assigned this Jun 5, 2026

Copilot started reviewing on behalf of kopardev June 5, 2026 21:54 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

kopardev added 15 commits June 5, 2026 18:13

fix(paths): replace gpfs CCBR base path with vf

bf97ac9

Update hardcoded base paths from /gpfs/gsfs10/users/CCBR_Pipeliner to /vf/users/CCBR_Pipeliner in runtime/docs/config references to avoid missing-file failures in dryrun. Refs: #150 ⚡ generated using AI ⚡

fix(config): make host/additives/viruses placeholders envsubst-compat…

8be54c5

…ible Use // placeholders in config template so CLI values are rendered during init. Refs: #151 ⚡ generated using AI ⚡

fix(cli): default additives to blank when omitted

f85c985

Stop defaulting to ERCC for init runs that do not pass --additives. Update help text to document blank default. Refs: #151 ⚡ generated using AI ⚡

fix(slurm): remove invalid ccr partition on biowulf

d079566

Stop prepending ccr to PARTITION and submit with norm by default. Refs: #152 ⚡ generated using AI ⚡

fix(slurm): force biowulf partition to norm

dbf699a

Set PARTITION explicitly to norm inside biowulf platform block so submission always targets a valid default partition. Refs: #152 ⚡ generated using AI ⚡

fix(slurm): change biowulf default partition from ccr,norm to norm

56ef217

The ccr partition does not exist on the biowulf system. Only the norm partition should be used as the default for all rules. ⚡ generated using AI ⚡

feat(workflow): add progress logging to create_index rule

4b3bf7b

Add timestamped progress messages at major create_index checkpoints to improve runtime visibility and troubleshooting for long index jobs. ⚡ generated using AI ⚡

fix: call set_singularity_binds in unlock() so binds are not blank

4a77d24

unlock() was the only runmode that did not call set_singularity_binds before entering run(), leaving SINGULARITY_BINDS empty in the commentary block and in --singularity-args. ⚡ generated using AI ⚡

fix: call set_singularity_binds in dryrun() and touch()

5cd843c

dryrun() and touch() were also missing the set_singularity_binds call, leaving SINGULARITY_BINDS empty. ⚡ generated using AI ⚡

docs(changelog): document SIFCACHE and singularity bind improvements

a697468

⚡ generated using AI ⚡

docs(changelog): highlight host-only circRNA mode as key new feature

503600d

⚡ generated using AI ⚡

chore(release): bump version to 0.13.0

6eb3b4e

⚡ generated using AI ⚡

kopardev requested a review from Copilot June 12, 2026 15:25

Copilot started reviewing on behalf of kopardev June 12, 2026 15:26 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

kopardev added 3 commits June 12, 2026 12:21

kopardev merged commit d0d1f7c into main Jun 12, 2026
4 of 5 checks passed

kopardev deleted the fix/issue-147-mamba-init branch June 12, 2026 16:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: migrate CHARLIE runtime activation to mamba-safe flow#149

fix: migrate CHARLIE runtime activation to mamba-safe flow#149
kopardev merged 20 commits into
mainfrom
fix/issue-147-mamba-init

kopardev commented Jun 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		minus_strand = resultsfile[resultsfile["strandd"] == "+"]
		minus_strand = minus_strand[["chrd", "coordd", "coorda", "strandd", "reads", "case"]]

Uh oh!

Conversation

kopardev commented Jun 5, 2026

Summary

Changes

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants