diff --git a/vignettes/supplementing-metrics.Rmd b/vignettes/supplementing-metrics.Rmd new file mode 100644 index 0000000..0d0213e --- /dev/null +++ b/vignettes/supplementing-metrics.Rmd @@ -0,0 +1,202 @@ +--- +title: "Supplementing a Metric Repository" +output: rmarkdown::html_vignette +vignette: > + %\VignetteIndexEntry{Supplementing a Metric Repository} + %\VignetteEngine{knitr::rmarkdown} + %\VignetteEncoding{UTF-8} +--- + +```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) +``` + +```{r setup} +library(val.criterion) +library(val.meter) +``` + +A common question from organizations evaluating `val.criterion` is whether they +can mix in metrics of their *own* -- for example, an internal notion of a +"reputable" maintainer, or a bespoke security score -- alongside the metrics +published by the R Validation Hub. They can, and this vignette walks through the +three moving parts: + +1. **registering** a new metric so `val.meter` knows its name and type, +2. **publishing** its values in a supplementary `PACKAGES` file, and +3. **filtering** on a mix of criteria drawn from two separate metric + repositories. + +Throughout, we simulate everything locally with `val.meter::random_repo()` so +the example runs offline. + +## Registering a new metric + +Metrics live in `val.meter`. Filtering by a metric requires that the metric be +*registered*, because `val.criterion` uses the metric's declared type to +interpret the character values stored in a `PACKAGES` file (every value in a +repository index is stored as text). An unregistered field has no known type and +cannot be coerced reliably. + +We register metrics with `val.meter::impl_data(metric = TRUE)`. The +[*Adding a Metric*](https://pharmar.github.io/val.meter/articles/adding-metric.html) +vignette in `val.meter` covers this in depth; here we add a small +`reputable_maintainer` metric that flags packages for which the *R Validation +Hub* is a copyright holder. + +```{r} +impl_data( + "reputable_maintainer", + title = "Reputable Maintainer", + metric = TRUE, + class = S7::class_logical, + function(pkg, resource, field, ...) { + authors <- pkg$desc$get_authors() + any(vapply( + authors, + function(a) "cph" %in% a$role && + grepl("R Validation Hub", paste(a$given, a$family)), + logical(1) + )) + } +) +``` + +Once registered, the metric shows up alongside the built-in metrics: + +```{r} +"reputable_maintainer" %in% names(metrics()) +``` + +## Simulating an upstream metric repository + +A metric repository is an ordinary CRAN-style repository whose +`src/contrib/PACKAGES` file is tagged `Format: Metrics` and carries one +`Metric/@R:` entry per metric. `val.meter::random_repo()` writes one for +us -- standing in for the metrics an organization might consume from the R +Validation Hub. We'll call it `valor`. + +```{r, warning = FALSE} +set.seed(1) +valor <- random_repo(n = 8) +``` + +Reading its index shows the metric columns we can filter on: + +```{r} +valor_url <- file.path(sub("^file://", "", valor), "src", "contrib", "PACKAGES") +base_db <- read.dcf(valor_url) +base_db <- base_db[!is.na(base_db[, "Package"]), , drop = FALSE] + +grep("^Metric/", colnames(base_db), value = TRUE) +``` + +## Building a supplementary `PACKAGES` file + +Now suppose "acme" -- our organization -- wants to contribute its own +`reputable_maintainer` assessment without touching the upstream repository. We +publish a *second* metric repository containing only our metric. + +The one rule that makes the two repositories joinable is that each record must +reuse the upstream `Package`, `Version` and `MD5sum` -- those three fields are +the join key. A supplementary repository omits the package metrics it does not +provide and lists only its own. + +```{r} +acme_dir <- tempfile("acme") +dir.create(file.path(acme_dir, "src", "contrib"), recursive = TRUE) + +set.seed(42) +records <- c("Format: Metrics", "") +for (i in seq_len(nrow(base_db))) { + records <- c( + records, + paste0("Package: ", base_db[i, "Package"]), + paste0("Version: ", base_db[i, "Version"]), + paste0("MD5sum: ", base_db[i, "MD5sum"]), + # simulate acme's assessment for the demo + paste0("Metric/reputable_maintainer@R: ", sample(c("TRUE", "FALSE"), 1)), + "" + ) +} +writeLines(records, file.path(acme_dir, "src", "contrib", "PACKAGES")) +acme <- paste0("file://", acme_dir) +``` + +The result is a minimal, valid metric repository: + +```{r} +cat(readLines(file.path(acme_dir, "src", "contrib", "PACKAGES"), n = 7), sep = "\n") +``` + +## Filtering on metrics from two repositories + +Each metric repository reduces to an `available.packages()`-style matrix. A +small helper reads one, keeping only the metric columns: + +```{r} +read_metrics <- function(repo) { + url <- file.path(sub("^file://", "", repo), "src", "contrib", "PACKAGES") + fields <- grep("^Metric/", colnames(read.dcf(url)), value = TRUE) + db <- available.packages(repos = repo, fields = fields, filters = list()) + db[!is.na(db[, "Package"]), , drop = FALSE] +} +``` + +To filter on metrics from both repositories at once, join them on the +`Package`/`Version`/`MD5sum` key into a single metric matrix. This gives every +package a `downloads_total` column (from `valor`) *and* a +`reputable_maintainer` column (from `acme`): + +```{r} +combined <- merge( + read_metrics(valor), + read_metrics(acme), + by = c("Package", "Version", "MD5sum"), + all = TRUE +) + +grep("downloads_total|reputable_maintainer", colnames(combined), value = TRUE) +``` + +`package_filter()` accepts a `db` argument -- a function returning the metric +matrix to filter against -- so we hand it our combined matrix. The filter +expression then references metrics from either repository by name, as flat +variables: + +```{r} +options( + repos = valor, + available_packages_filters = package_filter( + { + # passes R CMD check *and* is maintained by a reputable org + r_cmd_check_error_count == 0 & reputable_maintainer + }, + db = function() as.matrix(combined) + ) +) +``` + +With the filter installed, base R's `available.packages()` transparently drops +packages that fail the policy -- their `Repository` is rewritten to +`""`: + +```{r} +pkgs <- available.packages() +pkgs[, c("Package", "Repository")] +``` + +`install.packages()` honours the same filter, so a package excluded by policy +cannot be installed through the normal workflow. + +## A note on joining repositories + +`val.criterion` reads the repositories in `options("val.criterion.repos")` by +row-binding their indices, which does not by itself *column-join* two +repositories that describe the same packages with different metrics. Until +native joining is available, supply the joined matrix explicitly through the +`db` argument, as shown above. The join key is always +`c("Package", "Version", "MD5sum")`.