bpmetrics

A modular Go-based tool for extracting and analyzing dataset-level metrics from Bigpicture metadata files.

Purpose

This tool processes structured metadata (e.g. image.xml, sample.xml, staining.xml, etc.) within a dataset archive and computes standardized metrics for post-upload analysis, dashboards, and validation support.

It is designed to:

• Handle thousands of datasets in a memory-efficient manner
• Output machine-readable public summaries and structured per-dataset metrics
• Enforce dataset policy constraints for type_of_dataset

⚠️ Note: This tool assumes specific XML structure as used in the current Bigpicture metadata standard. It expects entries such as STRING_ATTRIBUTE and CODE_ATTRIBUTE and does not normalize values unless explicitly configured.

---

Metric Layers

1. Public Metrics (global_summary.json)

Public metrics are defined by PublicMetrics and include only dataset-wide totals:

• number_of_datasets
• number_of_wsis
• number_of_images
• number_of_files
• number_of_observations
• number_of_cases
• number_of_specimens
• number_of_blocks
• number_of_slides
• number_of_biological_beings
• number_of_stainings
• number_of_annotations

These represent flat aggregated counts across all datasets.
No internal structures, no compound metrics, no organisations, no node affiliation.

---

2. Internal Metrics (internal_summary.json)

Internal metrics are defined by InternalMetrics and include:

Dataset–organisation aggregates

• datasets_per_organisation (PIC → dataset count)
• number_of_unique_organisations

Global label dictionaries

• anatomical_site_labels (SNOMED CT code → meaning)

Aggregated biological/specimen counts (across all datasets)

• sex_counts_of_beings
• species_counts_of_beings
• anatomical_site_counts
• age_group_counts

Normalized proportions of the above

• sex_proportions
• species_proportions
• anatomical_site_proportions
• age_group_proportions

Compound / graph-based metrics

• wsi_staining_usage_per_diagnosis
• wsi_per_species
• wsi_per_species_proportion
• wsi_per_sex
• wsi_per_sex_proportion

Node affiliation is not part of internal metrics, as that information is not available in the metadata files.

Implemented Metrics

From dataset.xml

Extracted via ExtractDatasetFileMetrics:

• Metadata Standard Version
  Read from either <METADATA_STANDARD>.
• Dataset Version
  Read from <VERSION>.

---

From image.xml

Extracted via ExtractImageMetrics:

• Number of WSIs
  Count of unique alias values under <IMAGE_OF>.
• Number of images
  Count of <IMAGE> elements.
• Number of image files
  Count of <FILE> elements inside <FILES>.

---

From observation.xml

Extracted via ExtractObservationMetrics:

• Number of observations
  Count of <OBSERVATION> elements.
• Statement types
  Frequency of <STATEMENT_TYPE> values.
• Diagnosis counts
  Based on <MEANING> inside <CODE_ATTRIBUTE> only for statements of type "Diagnosis".
• ObsDiagnosis
  Internal map (observation → diagnosis) used in compound metrics.

---

From sample.xml

Extracted via ExtractSampleMetrics:

• Number of biological beings
• Number of cases
• Number of specimens
• Number of blocks
• Number of slides
• Sex counts (per biological being)
• Species counts (per biological being)
• Anatomical sites (unique list and SNOMED CT mappings)
• Age groups (combinations of interval_start + interval_length)

Internal maps for compound metrics:

• SexOfBeing
• SpeciesOfBeing

---

From staining.xml

Extracted via ExtractStainingMetrics:

• Number of stainings
• Lists of staining methods, targets, and compounds
• SNOMED CT codes per staining (via staining_procedure)
• H&E vs Other count (based on SNOMED CT 104210008)

Note: No anomaly detection is implemented in the current code.

---

From annotation.xml

Extracted via ExtractAnnotationMetrics:

• Number of annotations

---

From organisation.xml

Extracted via ExtractOrganisationMetrics:

• Submitting organisation PIC
• Submitting organisation name

---

From policy.xml

Extracted via ExtractPolicyFileMetrics:

• Validates type_of_dataset against the supported controlled vocabulary (case/whitespace tolerant).
  Datasets with type_of_dataset = "Clinical/Anonymized" are eligible for the public aggregate (global_summary.json). Other allowed types are still processed but excluded from public metrics.

Compound Metrics (Graph-based)

Compound metrics use the alias-based reference graph extracted from:

• sample.xml
• image.xml
• observation.xml through ExtractReferences, allowing traversal across beings → specimens → blocks → slides → stainings.

---

WSI staining usage per diagnosis

Computed via CountWSIsPerDiagnosisAndStain.

• Produces:
  Diagnosis → { "Hematoxylin and eosin": N, "Other": M }
• Uses:
  • ObsDiagnosis (observation → diagnosis)
  • Reference graph (observation → slide resolution)
  • StainCodes (stainingID → SNOMED CT code)

Slides with SNOMED CT code 104210008 are counted as H&E, all others as Other.

---

WSI count per species and per sex

Computed via CountWSIsPerSpeciesAndSex.

• Produces:
  • species → WSI count
  • sex → WSI count
• Logic:
  • WSIs = all slides recorded in the reference graph.
  • For each slide: resolve block(s) → specimen(s) → being.
  • Read species/sex from SpeciesOfBeing / SexOfBeing.
  • If a slide has no species/sex or multiple values → counted as
    "Unknown species" or "Unknown sex".

---

KPI Mapping Summary

The current implementation of bpmetrics delivers the KPI extraction required for BIGPICTURE Deliverable D3.05.
All KPIs that can be derived from the existing metadata files (dataset.xml, image.xml, sample.xml, observation.xml, staining.xml, annotation.xml, organisation.xml) are fully implemented.

Limitations

Note: Landingpage entity is not considered in the current scope of bpmetrics.

Several KPIs defined in the deliverable cannot be computed because the required information is not present in the Bigpicture metadata. These include:

1. Total dataset size (GB)
2. List of Geographical Areas
3. Contributing Center Names
4. Node affiliation

---

🛠 Usage

bpmetrics /path/to/archive_root

• Outputs public metrics in global_summary.json
• Writes dataset-level IO/parse issues to dataset_errors.json when present

Optional outputs (written only when CRYPT4GH_RECIPIENT_PUBLIC_KEY is set):

• per_dataset_metrics.json.c4gh (enabled with WRITE_PER_DATASET_METRICS=true)
• refgraph_<dataset>.json.c4gh debug files (enabled with WRITE_DEBUG_REFGRAPHS=true)

Internal metrics are never written in plaintext.

If the environment variable CRYPT4GH_RECIPIENT_PUBLIC_KEY is provided, the runner encrypts the internal summary directly in memory and writes only:

internal_summary.json.c4gh

No plaintext internal_summary.json is written to disk.

The program is split across main.go (tiny entrypoint) and runner.go (logic).
Use go run . <archive_root> or the built binary.

Encryption runtime requirement

If encrypted internal metrics are enabled, the runtime environment must provide the crypt4gh CLI. The official container image includes this dependency.

---

Project layout

.
├─ main.go                 # Minimal entrypoint: initializes logging, parses args, calls Run()
├─ runner.go               # Core pipeline: dataset scan, step execution, aggregation, JSON outputs
├─ steps_definitions.go    # Registry of dataset-level and compound metric steps
├─ constants.go            # Filenames, environment defaults, node names
├─ helpers.go              # Shared helpers (normalization, merging, utilities)
├─ process_dataset.go      # Per-dataset orchestration and error handling
└─ metrics/
   ├─ dataset.go           # dataset.xml extractor
   ├─ image.go             # image.xml extractor
   ├─ observation.go       # observation.xml extractor
   ├─ sample.go            # sample.xml extractor
   ├─ staining.go          # staining.xml extractor
   ├─ annotation.go        # annotation.xml extractor
   ├─ organisation.go      # organisation.xml extractor
   ├─ policy.go            # policy.xml validation
   ├─ references.go        # Reference graph extraction (alias-based relationships)
   ├─ compound.go          # graph-based compound metrics
   ├─ types.go             # shared metric types
   └─ xmltree/             # generic XML tree loader (used by reference graph)

---

Logging & Run ID

bpmetrics emits structured logs to stderr (JSON by default). Fields:

• ts — RFC3339Nano UTC timestamp
• level — debug|info|warn|error
• msg — message
• run_id — 16‑char hex ID for the current run
• Context (when available): dataset_id, op, code, path, err

Configure

Env vars:

• LOG_LEVEL (debug|info|warn|error, default info)
• LOG_FORMAT (json default, or text)
• LOG_SOURCE (true to include file:line)
• RUN_ID (set your own 16‑char hex, else random)

Example:

RUN_ID=2025cafebabe1234 LOG_LEVEL=debug LOG_FORMAT=json ./bpmetrics /path/to/archive_root 2> bpmetrics.log

Filter logs by run

# grep (JSON lines)
grep '"run_id":"2025cafebabe1234"' bpmetrics.log

# jq
jq 'select(.run_id=="2025cafebabe1234")' bpmetrics.log

Roadmap

---

License

Apache 2.0

---

Repository

This project is part of the Bigpicture Initiative.