End of an Era

Notable Changes

This is the final release for version 1 of GCL-R-Scripts! After this release we are going to start developing version 2 based on a new format for .gcl objects to integrate with R 4.0.0 and the tidyverse. A static version of this release will be parked on a separate branch v1.12.0-static, for posterity.

Below are all notable changes to the body of GCL-R-Scripts on the master branch since A Cat in Your Lap v1.11.0 was released on April 1, 2020.

See the README for this repository if you have questions. Note the code for Functions.GCL has been updated to pull all functions in this repository, not just the ones with a .GCL ending.

New Scripts

No new scripts!

Function Updates & Bug Fixes

• FindAlternateSpecies
  • Bugfix, added drop=FALSE to prevent .gcl arrays from collapsing to matrices if only 1 fish in the silly
• read_project_genotypes
  • Bugfix, filter for project_name if necessary, previously this function would pull all genotypes for fish involved in a project, including genotypes from older projects
• save_sillys
  • Bugfix, report missing sillys correctly
• treeColor
  • Bugfix, colors would not plot, now they will if colors are a character vector and found in colors()
• HoFisFstTable
  • Update, added argument for path to fstatfile if already exists, rather than create a new one
• QC
  • Update, replaced lattice::levelplot with plotly::ggplotly for an interactive view of genotyping success rate
• FailureRate
  • Update, added plotly for interactive view of genotyping success rate

Tips

To avoid conflicts among packages that have the same function name, you can be more explicit in your function calls by adding package_name:: before the function name (i.e. dplyr::rename vs. rename, to avoid conflicts with reshape::rename). This makes your code more stable and less susceptible to package conflicts depending what the user has loaded in their workspace.

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

A Cat in Your Lap

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since Midsummer's Eve v1.10.0 was released on June 20, 2019.

See the README for this repository if you have questions. Note the code for Functions.GCL has been updated to pull all functions in this repository, not just the ones with a .GCL ending.

New Scripts

• QC_off_chip
  • Built off of normal QC script to deal with very rare case of off-chip QC re-runs

Function Updates & Bug Fixes

• stratified_estimator_rubias
  • Added cv argument to integrate harvest estimate error in to MSA estimates
  • Bugfix if statements, only stratify over mixvec
• CustomCombineBAYESOutput
  • Bugfix to catch auto-install for packages
• QC
  • Added project_type argument to specify explicitly whether it is a "Biomark", "uSat", or "GT-seq" project, rather than trying to infer based on the number of alleles
• load_objects
  • Added pattern argument to load only specific objects in path
• StratifiedEstimator
  • Added section to equalize mixtures run for different numbers of iterations by subsampling to the minimum number of iterations in mixvec
• ReadUSatQC
  • Bugfix to remove leading "0"s on fish_id, make character, not numeric
• GenotypeReport
  • Bugfix to read data from LOKI, filter for project_name, then spread
  • Fix spelling error in file name

Tips

To avoid conflicts among packages that have the same function name, you can be more explicit in your function calls by adding package_name:: before the function name (i.e. dplyr::rename vs. rename, to avoid conflicts with reshape::rename). This makes your code more stable and less susceptible to package conflicts depending what the user has loaded in their workspace.

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

Midsummer's Eve

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since Java yo mama v1.9.0 was released on April 15, 2019.

See the README for this repository if you have questions. Note the code for Functions.GCL has been updated to pull all functions in this repository, not just the ones with a .GCL ending.

New Scripts

• write_genepop.GCL
  • This tidy function effectively replaces gcl2Genepop.GCL
  • Works for SNPs and uSATs

Function Updates & Bug Fixes

• ReadGenepopDis.GCL
  • Update include optional loci argument, must be same as loci used to create Genepop input file, default is NULL
  • Update to include optional sillyvec argument, must be same as sillyvec used to create Genepop input file, default is NULL, this prevents population headers being labeled as SillySource of last fish in population
  • Update to tidyverse, output is now a tibble
• ReadGTseqQC.GCL
  • Treat "0/0" genotype call as NA
• StratifiedEstimator.GCL
  • Allows mixvec argument to contain the same mixture more than once, this is useful if you have to have the same mixture represent multiple strata (i.e. represent unsampled harvest with another strata)
• GenotypeReport.GCL
  • Bugfix to allow pulling genotypes by sillyvec (i.e. project_name = NULL)
• run_rubias_mixture
  • Update to pass through new arguments in rubias v0.3.0

Tips

To avoid conflicts among packages that have the same function name, you can be more explicit in your function calls by adding package_name:: before the function name (i.e. dplyr::rename vs. rename, to avoid conflicts with reshape::rename). This makes your code more stable and less susceptible to package conflicts depending what the user has loaded in their workspace.

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

Java yo mama

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since New year new updates v1.8.0 was released on January 31, 2019. In order to get anything out of LOKI, you need to do a fresh pull from the master branch.

• Crucial update to move to our R functions that interface with LOKI to Java8
  • There is a new java driver ojdbc8.jar located here: "V:/Analysis/R files/OJDBC_Jar/ojdbc8.jar"
  • You need to copy that file to your machine at either one of these locations
    • "C:/Program Files/R/RequiredLibraries/ojdbc8.jar", or
    • "~/R/ojdbc8.jar"
  • After that, you will need to restart RStudio (might as well just restart your computer) or you will get an error when trying to interface with LOKI
• Added a stratified estimator for rubias
  • Very similar to StratifiedEstimator.GCL for BAYES

See the README for this repository if you have questions.

New Scripts

• stratified_estimator_rubias
  • New script designed to calculate stratified estimates from rubias output using a vector of harvest/abundance (similar to StratifiedEstimator.GCL for BAYES).
  • Works both with bias correction and without (i.e. method = "PB")
  • Works for the same reporting groups or can "roll-up" groups
  • See function script for details and a reproducible example
• LOKI_URL.GCL
  • This function builds the URL to connect to LOKI and is called by all GCL functions that connect to LOKI via RJDBC::dbConnect
  • Having this one function allows us to update the URL for all dependent functions

Function Updates & Bug Fixes

• The following functions were all updated to set the driver to ojdbc8.jar
  • GentoypeReport
  • CreateLocusControl
  • LOKI2R
  • LOKI2R_GAPS
  • ReadProjectLOKI2R
  • read_project_genotypes
• LOKI2R_GAPS
  • Can now pull genotypes from a collection with only 1 fish (drop = FALSE)
• gcl2Genepop
  • Can now pull genotypes from a collection with only 1 fish (drop = FALSE)
• custom_combine_rubias_output
  • Cosmetic changes only to update reproducible example
• GenotypeReport.GCL
  • Added a filter argument to make sure that this only pulls fish from project_name

Tips

To avoid conflicts among packages that have the same function name, you can be more explicit in your function calls by adding package_name:: before the function name (i.e. dplyr::rename vs. rename, to avoid conflicts with reshape::rename). This makes your code more stable and less susceptible to package conflicts depending what the user has loaded in their workspace.

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

New year new updates

Notable Changes

This release is long overdue. Below are all notable changes to the body of GCL-R-Scripts on the master branch since Ready for rubias v1.7.0 was released on August 2, 2018. I highly recommend that everyone do a fresh pull from the master branch.

• Many of the updates move functions to the tidyverse.
• There is critically important bugfix to create_rubias_baseline and create_rubias_mixture.
• The QC script and its dependent scripts have all been updated. Please use these updated versions for all future QC analyses.

See the README for this repository if you have questions.

New Scripts

• GenotypeReport.GCL
  • New script designed for Heather so she can pull genotypes from LOKI. Previously this was done via OceanAK, however, it times out on large GTseq projects. This function pulls just the genotypes and creates a .csv file so that Heather can visualize failure rates.
• read_project_genotypes.GCL
  • New script designed for QC. This pulls all project genotypes based solely on project_name (i.e. "P014"). No longer need projectID or markersuite. Does not require CreateLocusControl to be run first. This function was written based off of GenotypeReport.GCL because for unknown reasons ReadProjectLOKI2R started becoming unreasonably slow. This script hits a less complicated view of LOKI so it is much faster. Note: this function is slower than GenotypeReport.GCL and can still take up to 20 minutes to run for GTseq projects, because the view in LOKI has to pair up fish genotyped in the project with the correct plateID that was run in the project (tissue table).
• ReadUSatQC.GCL
  • Removed code from QC script and made as a stand alone function.
• ReadGTseqQC.GCL
  • Reads in genotype data from GTseq .csv files and creates .gcl objects.
• gcl2tibble.GCL
  • This function converts .gcl objects into a very simple tall, tidy tibble. Only four columns: sillysource, silly, locus, genotype (there is an argument for how you want alleles separated, defaults to "/"). This function is handy if you need to share or export genotypes.

Function Updates & Bug Fixes

• QC
  • Update to tidyverse.
  • Now uses read_project_genotypes instead of ReadProjectLOKI2R.
  • New QC genotype import functions ReadUSatQC.GCL and ReadGTseqQC.GCL.
  • Conflicts now come in as tibbles.
  • Alternate species object updates (not applicable in QC).
  • Duplicate check code re-written and simplified for tidyverse.
  • QC summary tables condensced and simplified for tidyverse.
  • Switch from xlsx package to writexl package for writing output. This package is much cleaner, however, it can not append the QC summary tables to the existing QC excel file, it creates a separate excel file with "R Script" in the name.
• FailureRate
  • Update to tidyverse.
  • Automatically output heatmaps of 1) failure rate by plate and locus, and 2) failure rate by silly and locus.
• create_rubias_baseline
  • Can now handle populations with only 1 fish.
  • Major bugfix, replace "0" genotypes from scores object with NA. Otherwise, the "0" was recognized as a unique allele.
• create_rubias_mixture
  • Major bugfix, replace "0" genotypes from scores object with NA. Otherwise, the "0" was recognized as a unique allele.
• CombineConflictsWithPlateID
  • Update to tidyverse, catches mitochondrial conflicts appropriately
• CheckDupWithinSilly
  • Duplicate summary output is now a tibble (previously it was a dataframe). The nice thing about it now being a tibble is that you still get an object with nrow = 0, rather than getting a NULL when no duplicates are found. Much easier for summarizing across multiple sillys (bind_rows).
  • Bugfix "NO" = "No"
• LOKI2R_GAPS
  • Bugfix, was looking for "ojdbc6.jar" in the wrong location.
• BayesOutputCopyPaste
  • Bugfix to deal with sillyvec as a named character vector.
• custom_combine_rubias_output
  • Can now roll up reporting groups if using method = "PB". This is good incentive to always run rubias with your finest scale reporting groups and then roll up if necessary.
  • Bugfix, will not allow negative values with method = "PB".
• RemoveIndMissLoci
  • Update to only calculate proportion missing based on how many loci were genotyped per collection. This can be an issue if you have more loci in LocusControl than were actually genotyped for all collections.
• gcl2Nexus.GCL
  • Added the ability to create file with actual allele calls (letters) rather than numbers for allele 1 and 2. Update also catches "0" and "?" genotypes to indicate as missing data.

Tips

To avoid conflicts among packages that have the same function name, you can be more explicit in your function calls by adding package_name:: before the function name (i.e. dplyr::rename vs. rename, to avoid conflicts with reshape::rename). This makes your code more stable and less susceptible to package conflicts depending what the user has loaded in their workspace.

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

Ready for rubias

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since Pest Control v1.6.1 was released on February 6, 2018. Many of these scripts are to streamline use of Eric Anderson's rubias with our standard GCL objects and workflow.

Note that the README for this repository has been updated to explain:

1. Initial cloning of the repository
2. Normal usage
3. Releases
4. Branches
5. Issues
6. Remotely sourcing individual scripts (i.e. from Remote Desktop)
7. QC

New Scripts

• create_rubias_baseline
  • This function creates the baseline (aka "reference") dataframe needed for rubias from .gcl objects.
  • It also saves the baseline as a .csv file with an assumed file structure.
• create_rubias_mixture
  • This function creates the mixture dataframe needed for rubias from .gcl objects.
  • It also saves each mixture in sillyvec as its own .csv file with an assumed file structure.
• run_rubias_mixture
  • This function is a wrapper for rubias::infer_mixture.
  • It automatically sets the seed for the MCMC chain and comes with several GCL default settings (i.e. number of reps, burn_in, etc.).
  • It also saves the different types of output into wide .csv files.
• custom_combine_rubias_output
  • This function computes summary statistics from rubias output into a tibble. It can take either the output tibble or can re-create the output tibble from the output files that are saved by run_rubias_mixture.
  • It also creates a traceplot for all mixtures and reporting groups.
• save_objects
  • This function is a wrapper for dput. It takes a character vector of objects and saves them as object_name.txt in path.
• load_objects
  • This function is a wrapper for dget. It will load objects previously dput giving them the same name.
• save_sillys
  • This function is a wrapper for dput, but can take a sillyvec (i.e. doesn't need the .gcl).
• load_sillys
  • This function is a wrapper for dget, but can take a sillyvec (i.e. doesn't need the .gcl).

Function Updates & Bug Fixes

• QC
  • Update to filter the Concordance Report by only "Conflicts", just in case lab staff accidently dump out the full Concordance Report on accident.
• PoolCollections
  • Bugfix to give the correct attributes by matching with FK_FISH_ID.

Tips

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

Pest Control

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since January Blues v1.6.0 was released on January 10, 2018.

Note that the README for this repository has been updated to explain:

1. Initial cloning of the repository
2. Normal usage
3. Releases
4. Branches
5. Issues
6. Remotely sourcing individual scripts (i.e. from Remote Desktop)
7. QC

New Scripts

No new scripts in this release.

Function Updates & Bug Fixes

• gcl2Genepop
  • Bugfix so that this works as intended for uSats (replace no-calls with 000).
• ReadGenepopHWE
  • Bugfix to strsplit for "Pop : " instead of " Pop : ", due to updates in Genepop 4.7.
  • Updated to properly convert summary results from Exact test from character to numeric, due to updates in Genepop 4.7.
  • Added an optional logical argument summaryAsNumeric = TRUE that defaults to converting Exact test summary results that show up as character (i.e. "High. sign." or " < 0.155") to numeric (0.000 and 0.155, respectively). If you set this flag to FALSE, then the summary matrix will be character rather than numeric.
• Anderson_etal
  • Updated to provide a flat by population confusion matrix along with by group confusion matrix (which is flat with respect to group).
  • Updated to provide a population to group confusion matrix (default is flat by group).
  • Note this function does not work with combined markers at the moment.

Tips

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

January Blues

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since Vernal Equinox v1.5.0 was released on April 4, 2017.

Note that the README for this repository has been updated to explain:

1. Initial cloning of the repository
2. Normal usage
3. Releases
4. Branches
5. Issues
6. Remotely sourcing individual scripts (i.e. from Remote Desktop)
7. QC

New Scripts

• ConfusionMatricesLikeProfile
  • This function creates confusion matrices from the output of LeaveOneOutLikeProfile.
  • Output matches that of the confusion matrices built from LeaveOneOutDist.
• Anderson_etal
  • This function very quickly creates a group confusion matrix based on leave one out statistics from Anderson et al.
    • Overall mean group relative likelihood matrix.
    • Dataframe of individual likelihoods to each population, with population and group identifiers.
• require
  • This function is a wrapper for requiring and installing (if necessary) a vector of packages.

Function Updates & Bug Fixes

• StratifiedEstimator
  • Updated to install required packages if necessary.
  • Added a new logical argument PosteriorOutput in order to control whether the whole posterior is included, default is FALSE.
• LOKI2R_GAPS
  • Updated with code from LOKI2R and ReadLOKI_GAPS to quickly and correctly read in GAPS uSat data.
  • Updated to give a message when a silly has no fish.
• CombineConflictsWithPlateID
  • Removed package xlsx, output is now written as a .csv in order to remove dependencies (xlsx has Java issues occasionally).
• genetic_msa
  • Set minimum value for Dirichlet to avoid hard zero's and NaNs.
• ReadGenepopHWE
  • Bugfix to grep for "Pop : " instead of " Pop : ", due to updates in Genepop 4.7.
  • Added an optional arguement sillyvec that will name populations automatically rather than grabbing names from the genepop .P.txt file. This solves the issue of having pooled silly names cut off due to character limits.
• BayesOutputCopyPaste
  • Inserted a stop statement to make sure files aren't accidentally deleted if a typo in sillyvec argument causes an issue with moving files.
• LeaveOneOutDist
  • Updated to remove loci that are fixed for one allele.
• LOKI2R
  • Bugfix to properly catch sillys with no fish genotyped.
• CreateLocusControl
  • Updated with new locusnames argument so that LocusControl can be built with either markersuite or locusnames. This is really useful if you don't remember what markersuite you are using or an appropriate markersuite hasn't been built in LOKI. However, all of the loci in locusnames do have to be built in LOKI.
    • locusnames is a character vector of marker names.

Tips

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

Vernal Equinox

Important User Change Required

.gitignore was added and updated in Fall Update v1.3.0 so that Git no longer tracks .RProj. This will prevent headaches in the long run, but if you have not done it already you will need to delete .gitignore from your local machine, otherwise you will not be able to do a fresh pull.

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since QC For Everyone v1.4.0 was released on September 8, 2016.

Note that the README for this repository has been updated to explain:

1. Initial cloning of the repository
2. Normal usage
3. Releases
4. Branches
5. Issues
6. Remotely sourcing individual scripts (i.e. from Remote Desktop)
7. QC

New Scripts

• genetic_msa
  • The intent of this new function is to run the full Pella Masuda MSA model in R (i.e. BAYES, but in R).
  • This function requires mixture and baseline files written by write_data4msa.
  • This function will multi-core over the number of chains.
  • The main output that is returned is a dataframe with summary statistics for each reporting group.
  • There are two types of output file for each chain:
    • The i files indicate which population (number) each individual was assigned to at each iteration.
    • The P files indicate what proportion of the mixture is assigned to each population for each iteration.
• write_data4msa
  • This function creates and writes the baseline and mixture files (.txt) to be used with genetic_msa.
  • This function requires .gcl objects for all baseline and mixture sillys to be loaded in your workspace.

Function Updates & Bug Fixes

• QC
  • Updated to note the number of Alternate and Duplicate fish within silly without actually removing them. If removed, then might miss important QC conflicts.
  • Updated Conflict Report section to accomodate new conflict.csv that outputs " " for mitochondrial conflicts as opposed to "0". Also added some extra code to help catch sequential IDs in within silly duplicate check that may be indicative of row duplication.
  • Added simplfy = FALSE to dupcheck summary output in order to force list object.
  • Bugfix to prevent issues when nAltBySilly is not present.
• gcl2Genepop.GCL
  • Updated to add a logical switch usat to print out the scores rather than counts when set to TRUE. Default setting is FALSE.

Functions Removed

• LynnCanal_agma
• magma2012and13

Tips

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again

QC For Everyone

Important User Change Required

.gitignore was added and updated in Fall Update v1.3.0 so that Git no longer tracks .RProj. This will prevent headaches in the long run, but if you have not done it already you will need to delete .gitignore from your local machine, otherwise you will not be able to do a fresh pull.

Notable Changes

Below are all notable changes to the body of GCL-R-Scripts on the master branch since Fall Update v1.3.0 was released on August 19, 2016.

Note that the README for this repository has been updated to explain:

1. Initial cloning of the repository
2. Normal usage
3. Releases
4. Branches
5. Issues
6. Remotely sourcing individual scripts (i.e. from Remote Desktop)
7. QC

New Scripts

• QC
  • This new script is designed handle both microsatellite and SNP QC.
  • This script is intended to replace any existing QC R scripts and perform the following operations:
    • Read in Project genotypes from LOKI
    • Read in QC genotypes from Biomark.csv files (SNPs) or GeneMapper.csv files (uSATs)
    • Compute Project failure rates
    • Report conflict rates
    • Check for missing loci, duplicate, or alternate species individuals in Project fish
    • Check for missing loci individuals in QC fish
    • Perform duplicate check between QC and Project fish for individuals with a specified conflict rate
    • Create summary tables
    • Write summary tables to the Project Summary .xslx file
  • This is a new script, not a new function, it is a separate script. The idea is that when you need to perform a QC, you will do a fresh pull from master in order to have the latest version of QC.R in your remote repository (i.e. C:\Users\krshedd\Documents\R\GCL-R-Scripts). You will then copy/paste this file to the QC project folder (i.e. V:\Lab\Genotyping\SNP Projects\Sockeye\Project S161 Chignik Inseason 2016\QC), modify the Arguments, and run the script from there. You should not have to modify anything past GO!, but if the function bombs on you, you will at least know where you are.
  • Written and tested by Jim Jasper and Kyle Shedd

Function Updates & Bug Fixes

• CombineConflictsWithPlateID.GCL
  • Updated documentation for files argument to specify a vector of .csv ConcordanceReports.
• SampSizeByLocus.GCL
  • Updated to add drop = FALSE to counts.

Tips

If you run into the following error when running LOKI2R.GCL
Error in .jcall(rp, "I", "fetch", stride, block) :
java.lang.OutOfMemoryError: GC overhead limit exceeded

• Save your work
• Re-start RStudio
• Before doing anything else, run options(java.parameters = "-Xmx100g") to increase your heap space
• Try to read data again