Error when running rosace from dumpling workflow

[durr1602]

Hi, sorry if this is trivial, but I'm running into some issues when running the dumpling workflow. It appears to be related to the run_rosace script. Here is the error I get:

CalledProcessError in file /project/chlandry/users/rodur28/dumpling/workflow/rules/rosace.smk, line 18:
Command 'set -euo pipefail;  Rscript --vanilla /project/chlandry/users/rodur28/dumpling/.snakemake/scripts/tmpaxrm35zq.run_rosace.R' returned non-zero exit status 1.
[Wed Dec 11 13:11:29 2024]
Error in rule run_rosace:
    jobid: 1
    input: results/benchmark_RD/processed_counts/CN_alp_r1_F1_T0.tsv, results/benchmark_RD/processed_counts/CN_alp_r2_F1_T0.tsv, results/benchmark_RD/processed_counts/CN_alp_r1_F1_T1.tsv, results/benchmark_RD/processed_counts/CN_alp_r2_F1_T1.tsv, results/benchmark_RD/processed_counts/CN_alp_r1_F1_T2.tsv, results/benchmark_RD/processed_counts/CN_alp_r2_F1_T2.tsv
    output: results/benchmark_RD/rosace/scores_cond_A.tsv
    log: logs/benchmark_RD/rosace/benchmark_RD.rosace.log (check log file(s) for error details)

Traceback (most recent call last):
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/bin/enrich_cmd", line 8, in <module>
    sys.exit(main_cmd())
             ~~~~~~~~^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/enrich2/main.py", line 232, in main_cmd
    obj.calculate()
    ~~~~~~~~~~~~~^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/enrich2/experiment.py", line 159, in calculate
    self.calc_counts(label)
    ~~~~~~~~~~~~~~~~^^^^^^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/enrich2/experiment.py", line 254, in calc_counts
    data[(cnd.name, sel.name, "c_{}".format(tp))] = sel_data[
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/frame.py", line 4311, in __setitem__
    self._set_item(key, value)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/frame.py", line 4524, in _set_item
    value, refs = self._sanitize_column(value)
                  ~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/frame.py", line 5263, in _sanitize_column
    return _reindex_for_setitem(value, self.index)
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/frame.py", line 12692, in _reindex_for_setitem
    raise err
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/frame.py", line 12687, in _reindex_for_setitem
    reindexed_value = value.reindex(index)._values
                      ~~~~~~~~~~~~~^^^^^^^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/series.py", line 5153, in reindex
    return super().reindex(
           ~~~~~~~~~~~~~~~^
        index=index,
        ^^^^^^^^^^^^
    ...<5 lines>...
        tolerance=tolerance,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/generic.py", line 5610, in reindex
    return self._reindex_axes(
           ~~~~~~~~~~~~~~~~~~^
        axes, level, limit, tolerance, method, fill_value, copy
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ).__finalize__(self, method="reindex")
    ^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/generic.py", line 5633, in _reindex_axes
    new_index, indexer = ax.reindex(
                         ~~~~~~~~~~^
        labels, level=level, limit=limit, tolerance=tolerance, method=method
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/pandas/core/indexes/base.py", line 4429, in reindex
    raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_rep2_T1_lib.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_rep1_T2_lib.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_rep2_T2_lib.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/benchmark_RD_exp.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_R1_sel.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_R2_sel.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_rep1_T0_lib.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_rep2_T0_lib.h5
  warnings.warn(UnclosedFileWarning(msg))
/project/chlandry/users/rodur28/dumpling/.snakemake/conda/67d2d89bb9799e59a640f480438207d3_/lib/python3.13/site-packages/tables/file.py:114: UnclosedFileWarning: Closing remaining open file: results/benchmark_RD/enrich/cond_A_rep1_T1_lib.h5
  warnings.warn(UnclosedFileWarning(msg))

Most importantly, here is the log from rosace:

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

This is cmdstanr version 0.8.0
- CmdStanR documentation and vignettes: mc-stan.org/cmdstanr
- CmdStan path: /home/rodur28/.cmdstan/cmdstan-2.35.0
- CmdStan version: 2.35.0

A newer version of CmdStan is available. See ?install_cmdstan() to install it.
To disable this check set option or environment variable cmdstanr_no_ver_check=TRUE.
Rows: 8 Columns: 6
── Column specification ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (3): sample, condition, file
dbl (3): replicate, time, tile

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Warning message:
In dir.create(rosace_assayset_dir, recursive = TRUE) :
  './results/benchmark_RD/rosace/assayset' already exists
Rows: 2271 Columns: 2
── Column specification ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: "\t"
chr (1): hgvs
dbl (1): count

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Joining with `by = join_by(hgvs)`
Rows: 2271 Columns: 2
── Column specification ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: "\t"
chr (1): hgvs
dbl (1): count

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Joining with `by = join_by(hgvs)`
Rows: 2271 Columns: 2
── Column specification ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: "\t"
chr (1): hgvs
dbl (1): count

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Joining with `by = join_by(hgvs)`
Error in CreateAssayObject(counts = as.matrix(counts[2:ncol(counts)]),  :
  Length of variant names is not equal to number of rows in input matrix
In addition: Warning messages:
1: In full_join(counts, count) :
  Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 3 of `x` matches multiple rows in `y`.
ℹ Row 3 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship = "many-to-many"` to silence this warning.
2: In full_join(counts, count) :
  Detected an unexpected many-to-many relationship between `x` and `y`.
ℹ Row 3 of `x` matches multiple rows in `y`.
ℹ Row 3 of `y` matches multiple rows in `x`.
ℹ If a many-to-many relationship is expected, set `relationship = "many-to-many"` to silence this warning.
Execution halted

I though this might be related to the specification of coordinates in the config file (which I'm guessing should corroborate the list of variants) but I double checked that and it should be OK. I don't really know what's going on :/.

thanks for the help, much appreciated!

[odcambc]

Hm, the rosace environment has been finicky before, so I will definitely look at this. You are using a linux64 environment, right? Could you share your experiment config file (the csv)? You can give the samples dummy names to anonymize it if you want, that's fine.

[durr1602]

Yes! The layout is attached. I just noticed that the chariot returns are encoded as CRLF, hopefully that hasn't caused any issue ^^"

[odcambc]

Sorry for having to slog your way through this! I took the opportunity to make the r environment handling a little cleaner, as well as tidying a few issues. Could you give this branch a try and see if the issues remain?

https://github.com/odcambc/dumpling/tree/rosace_install

[durr1602]

No problem! Could there be a typo in the defined input for the run_rosace rule? .csv files are expected, but the corresponding files (generated by rule process_counts) are .tsv

[odcambc]

That's exactly right...there were some inconsistent files that I wasn't catching while testing for some reason. I just pushed another change that ought to fix that.

[durr1602]

Sorry for the late response, have been pretty busy. I've re-cloned the repo (main branch, since the rosace_install branch appeared behind) and restarted everything. It throws an error upon installation of rosace:

trying URL 'https://cloud.r-project.org/src/contrib/renv_1.0.11.tar.gz'
Content type 'application/x-gzip' length 1220458 bytes (1.2 MB)
==================================================
downloaded 1.2 MB


The downloaded source packages are in
        ‘/scratch/RtmpiFhizD/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done

Attaching package: ‘renv’

The following objects are masked from ‘package:stats’:

    embed, update

The following objects are masked from ‘package:utils’:

    history, upgrade

The following objects are masked from ‘package:base’:

    autoload, load, remove

The following package(s) will be updated:

# Bioconductor 3.20 ----------------------------------------------------------
- BiocVersion      [* -> 3.20.0]
- impute           [* -> 1.80.0]

# CRAN -----------------------------------------------------------------------
- nloptr           [1.2.2.3 -> 2.1.1]
- abind            [* -> 1.4-8]
- backports        [* -> 1.5.0]
- bayesm           [* -> 3.1-6]
- bit              [* -> 4.5.0.1]
- bit64            [* -> 4.5.2]
- boot             [* -> 1.3-31]
- broom            [* -> 1.0.7]
- car              [* -> 3.1-3]
- carData          [* -> 3.0-5]
- checkmate        [* -> 2.3.2]
- cli              [* -> 3.6.3]
- clipr            [* -> 0.8.0]
- colorspace       [* -> 2.1-1]
- compositions     [* -> 2.0-8]
- corrplot         [* -> 0.95]
- cowplot          [* -> 1.1.3]
- cpp11            [* -> 0.5.1]
- crayon           [* -> 1.5.3]
- data.table       [* -> 1.16.4]
- DEoptimR         [* -> 1.1-3-1]
- Deriv            [* -> 4.1.6]
- distributional   [* -> 0.5.0]
- doBy             [* -> 4.6.24]
- dplyr            [* -> 1.1.4]
- fansi            [* -> 1.0.6]
- farver           [* -> 2.1.2]
- Formula          [* -> 1.2-5]
- generics         [* -> 0.1.3]
- ggplot2          [* -> 3.5.1]
- ggpubr           [* -> 0.6.0]
- ggrepel          [* -> 0.9.6]
- ggsci            [* -> 3.2.0]
- ggsignif         [* -> 0.6.4]
- glue             [* -> 1.8.0]
- gridExtra        [* -> 2.3]
- gtable           [* -> 0.3.6]
- hms              [* -> 1.1.3]
- isoband          [* -> 0.2.7]
- jsonlite         [* -> 1.8.9]
- KernSmooth       [* -> 2.23-24]
- labeling         [* -> 0.4.3]
- lattice          [* -> 0.22-6]
- lifecycle        [* -> 1.0.4]
- lme4             [* -> 1.1-35.5]
- magrittr         [* -> 2.0.3]
- MASS             [* -> 7.3-61]
- Matrix           [* -> 1.7-1]
- MatrixModels     [* -> 0.5-3]
- matrixStats      [* -> 1.4.1]
- mclust           [* -> 6.1.1]
- mgcv             [* -> 1.9-1]
- microbenchmark   [* -> 1.5.0]
- minqa            [* -> 1.2.8]
- modelr           [* -> 0.1.11]
- munsell          [* -> 0.5.1]
- nlme             [* -> 3.1-166]
- nnet             [* -> 7.3-19]
- numDeriv         [* -> 2016.8-1.1]
- pbkrtest         [* -> 0.5.3]
- philentropy      [* -> 0.9.0]
- pillar           [* -> 1.9.0]
- pkgconfig        [* -> 2.0.3]
- polynom          [* -> 1.4-1]
- poorman          [* -> 0.2.7]
- posterior        [* -> 1.6.0]
- prettyunits      [* -> 1.2.0]
- processx         [* -> 3.8.4]
- progress         [* -> 1.2.3]
- ps               [* -> 1.8.1]
- purrr            [* -> 1.0.2]
- quantreg         [* -> 5.99.1]
- R6               [* -> 2.5.1]
- RColorBrewer     [* -> 1.1-3]
- Rcpp             [* -> 1.0.13-1]
- RcppArmadillo    [* -> 14.2.2-1]
- RcppEigen        [* -> 0.3.4.0.2]
- readr            [* -> 2.1.5]
- rjson            [* -> 0.2.23]
- rlang            [* -> 1.1.4]
- robustbase       [* -> 0.99-4-1]
- rstatix          [* -> 0.7.2]
- scales           [* -> 1.3.0]
- SparseM          [* -> 1.84-2]
- stringi          [* -> 1.8.4]
- stringr          [* -> 1.5.1]
- survival         [* -> 3.7-0]
- tensorA          [* -> 0.36.2.1]
- tibble           [* -> 3.2.1]
- tidyr            [* -> 1.3.1]
- tidyselect       [* -> 1.2.1]
- tzdb             [* -> 0.4.0]
- utf8             [* -> 1.2.4]
- vctrs            [* -> 0.6.5]
- viridisLite      [* -> 0.4.2]
- vroom            [* -> 1.6.5]
- withr            [* -> 3.0.2]

# GitHub ---------------------------------------------------------------------
- cmdstanr         [* -> stan-dev/cmdstanr]
- rosace           [* -> odcambc/rosace]

# Downloading packages -------------------------------------------------------
- Downloading nloptr from CRAN ...              OK [file is up to date]
- Querying repositories for available source packages ... Done!
- Downloading BiocVersion from Bioconductor 3.20 ...    ERROR [error code 22]
- Downloading BiocVersion from Bioconductor 3.20 ...    ERROR [error code 22]
- Downloading BiocVersion from Bioconductor 3.20 ...    ERROR [error code 22]
- Downloading BiocVersion from Bioconductor 3.20 ...    ERROR [error code 22]
- Downloading BiocVersion from Bioconductor 3.20 ...    ERROR [error code 22]
- Downloading BiocVersion from Bioconductor 3.20 ...    ERROR [error code 22]
Warning: failed to find source for 'BiocVersion 3.20.0' in package repositories
Warning: error downloading 'https://repo.miserver.it.umich.edu/cran/src/contrib/Archive/BiocVersion/BiocVersion_3.20.0.tar.gz' [error code 22]
Warning: error downloading 'https://bioconductor.org/packages/3.20/bioc/src/contrib/Archive/BiocVersion/BiocVersion_3.20.0.tar.gz' [error code 22]
Warning: error downloading 'https://bioconductor.org/packages/3.20/data/annotation/src/contrib/Archive/BiocVersion/BiocVersion_3.20.0.tar.gz' [error code 22]
Warning: error downloading 'https://bioconductor.org/packages/3.20/data/experiment/src/contrib/Archive/BiocVersion/BiocVersion_3.20.0.tar.gz' [error code 22]
Warning: error downloading 'https://bioconductor.org/packages/3.20/workflows/src/contrib/Archive/BiocVersion/BiocVersion_3.20.0.tar.gz' [error code 22]
Warning: error downloading 'https://bioconductor.org/packages/3.20/books/src/contrib/Archive/BiocVersion/BiocVersion_3.20.0.tar.gz' [error code 22]
Error: failed to retrieve package 'bioc::BiocVersion@3.20.0'
Traceback (most recent calls last):
9: renv::restore()
8: renv_restore_run_actions(project, diff, current, lockfile, rebuild)
7: renv_retrieve_impl(packages)
6: handler(package, renv_retrieve_impl_one(package))
5: renv_retrieve_impl_one(package)
4: renv_retrieve_bioconductor(record)
3: renv_retrieve_repos(record)
2: stopf("failed to retrieve package '%s'", remote)
1: stop(sprintf(fmt, ...), call. = call.)
Execution halted

What's bizarre is I thought that rosace was properly installed last time, since the error was upon execution.. Also I should mention that, way back, when I was trying to install dumpling, I had to install rosace in R from the home directory. I don't have sudo privileges which means I had to install a few things in the dumpling_env conda environment, but then when I was entering R, everything appeared to be installed. Now, if I changed directories to go to "dumpling", it would throw a renv error (lock file out of sync). I'm really not familiar with R so I don't know exactly if dumpling "can find rosace" or if it tries to install everything from scratch, but what I do know is that there's a fair bit of things that usually don't work when we try to install R in a conda environment (on our systems). Hopefully everything works out in the end, it might be something simple that I'm missing. Thanks again for all the help (I'm purposefully sharing more details just in case it helps with tracking down the issue)

[durr1602]

Hi there,
I've updated a bunch of stuff but most importantly, I've managed to solve the rosace_install issue I was having by specifying r-base>=4.4.2 in the yml. It now works well in regards to installing rosace.

I still encounter two separate issues:

1. Running rosace halts when trying to create the AssayObject (same error as the one mentioned in my first message on this page, second code block):

Error in CreateAssayObject(counts = as.matrix(counts[2:ncol(counts)]),  :
Length of variant names is not equal to number of rows in input matrix

2. Running Enrich2 does not output the expected file "main_identifiers_scores.tsv" (same error as the one mentioned in my first message on this page, first code block)

ValueError: cannot reindex on an axis with duplicate labels

Looks like it could be caused by the same problem related to the sample config..?