Skip to content

Issue 107: lead the parser examples with runnable code #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cdgaete wants to merge 3 commits into
base: prototype-2
Choose a base branch
from
+267 −109
Open

Issue 107: lead the parser examples with runnable code #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0dabe2a
doc: lead the parser examples with runnable code
cdgaete Aug 29, 2026
d8fb2f1
Merge branch 'prototype-2' into issue_107_examples_code_first
euronion Sep 2, 2026
2e2f5e9
docs: Simplify documentation for DEA storage data v10
euronion Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/worktrees/agent-ab7e7ae9f0428c62d
Open in desktop
Submodule agent-ab7e7ae9f0428c62d added at 055a76
187 changes: 133 additions & 54 deletions docs/examples/dea_storage_v10.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Danish Energy Agency Parser Documentation
# Danish Energy Agency Parser (v10)

<!--
SPDX-FileCopyrightText: technologydata contributors
Expand All @@ -7,84 +7,168 @@ SPDX-License-Identifier: MIT

-->

## Overview

!!! note
This example refers specifically to **version 10** (`v10`) of the DEA Energy Storage dataset. Details such as file names, sheet structure, and parser behaviour may differ for other versions.

The Danish Energy Agency (DEA) data parser demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/dea_energy_storage/`.
The Danish Energy Agency (DEA) maintains a dataset of techno-economic data for different energy storage technologies.
The data in its raw format is available in PDF and as an Excel file.
We parse the Excel file and extract more than 3000 individual parameters into our data schema and make them available through the package.

## Dataset Description
The dataset is published by the Danish Energy Agency via [this website](https://ens.dk/media/6589/download) and described in the [accompanying documentation](https://ens.dk/media/6588/download).
The data is licensed CC-BY-4.0.

The original dataset is available from the [Danish Energy Agency website](https://ens.dk/media/6589/download). A full description of the dataset is available in the [DEA documentation](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`.
## Quick start

The dataset is in Excel format, and it includes, under the data sheet `alldata_flat`, a flat table of technology parameters for a range of energy storage technologies. Columns include `Technology`, `ws`, `par` (parameter name), `val` (value), `unit`, `year`, `est` (case/estimate), `priceyear`, plus metadata columns such as `cat`, `ref`, `note`. Rows are individual parameter records (parameter value + unit + context) for technologies and estimation cases.
Load the parsed dataset.
The data ships with the package

## Parser description
```python
from technologydata import DataAccessor

The parser is articulated in the following steps.
# The bundled catalogues ship inside the installed package.

### Read the raw data
data_package = DataAccessor(
data_source="dea_energy_storage",
version="v10",
).load()

The script reads the raw data available at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially.
technologies = data_package.technologies
print(len(technologies.technologies))
> 136
```

### Data cleaning, validation and dealing with missing/null values
Inspect the collection as a table:

The data cleaning and validation happens with the following steps.
```python
df = technologies.to_dataframe()
print(df[["detailed_technology", "case", "year"]].head().to_string(index=False))
```

Function `_drop_invalid_rows(df)` validates whether required columns are present. It drops rows with missing/null or empty critical fields (`Technology`, `par`, `val`, `year`) and keeps rows where `year` contains a 4-digit year and `val` contains numeric characters and no comparator symbols (`<`, `>`, `≤`, `≥`).
```text
detailed_technology case year
compressed air energy storage control 2015
hydrogen storage - caverns control 2015
hydrogen storage - lohc control 2015
large-scale hot water tanks (steel) control 2015
na-nicl2 battery control 2015
```

Function `_clean_technology_string()` normalizes text fields by removing leading 3-digit numeric codes, trims whitespace and lower-cases the string for consistent matching. It is applied to the columns `Technology` and `ws`. As an example, `_clean_technology_string()` converts `151b Hydrogen Storage - LOHC` to `hydrogen storage - lohc`.
Select the data for all years of Li-ion utility scale battery in the EU:

Function `_extract_year()` extracts the first sequence of digits from the `year` column and converts it to an integer. The column contains in fact entries like `Uncertainty (2050)` (str) which are converted to `2050` (int).
```python
batteries = data.technologies.get(
name="lithium ion battery",
detailed_technology="utility-scale",
region="EU",
case="control",
# Use regex to match all years from 2000 to 2099
year="20\\d+",
)

Function `_clean_parameter_string()` removes leading hyphens, removes text inside square brackets (units/notes), collapses extra spaces and lower-cases the parameter name. It is applied to the `par` column.
len(batteries)
# 5
```

Function `_standardize_units()` is applied to columns `par` and `unit`. It completes missing units based on parameter name (e.g., `energy storage capacity for one unit` is mapped to the unit `MWh`) via a parameter-to-unit map. Moreover, it replaces known incorrect unit strings as `⁰C` -> `C` or `m2` to `meter**2`. The unit substitutions are driven by the [pint default units definition](https://github.com/hgrecco/pint/blob/master/pint/default_en.txt).
The technologies are also accessible as a list.
Get the first technology from the list and access one of its parameters (the speciic investment costs):

Function `Commons.update_unit_with_currency_year(unit, priceyear)`, if present, appends `priceyear` information to currency units. This is because `technologydata` follows the currency pattern `\b(?P<cu_iso3>[A-Z]{3})_(?P<year>\d{4})\b`, as for example `EUR_2021`.
```python
investment = batteries.technologies[0].parameters["specific investment"]
print(investment)
# 288000.0 EUR_2020 / megawatt_hour
```

## From raw data to parsed output

Each raw row carries one parameter value together with the context that identifies it.
The parser written for `technologydata` cleans and harmonises these fields, then groups the rows into `Technology` objects.
Each technology object is then created based on the group of rows identifying this technology and holds a dictionary of `Parameter` values.

```mermaid
flowchart LR
subgraph raw["DEA Excel file (alldata_flat row)"]
direction TB
r1["ws<br>Technology"]
r2["par<br>val<br>unit<br>priceyear"]
r3["year<br>est"]
end
subgraph parser["Parser"]
direction TB
s1["Drop invalid rows"]
s2["Clean names, years,<br>parameters and units"]
s3["Group by est, year,<br>ws, Technology"]
s1 --> s2 --> s3
end
subgraph out["Technology"]
direction TB
o1["name<br>detailed_technology<br>case<br>region<br>year"]
o2["parameters:<br>Parameter magnitude, units"]
o3["Source information"]
o4["Provenance information"]
end
raw --> parser --> out
```

## Deep dive: From raw to parsed data

An example for a speciic row, e.g. row 2335 of Excel file's worksheet `alldata_flat` contains the 2025 specific investment of a utility-scale lithium-ion battery.
It reaches the parsed collection as follows.

Function `_format_val_number(value, num_decimals)` parses numeric formats including comma decimal separators and scientific notation variants (e.g., `×10`) and converts them to float and rounds them to `num_decimals`.
| Raw field | Raw value | Parsed field | Parsed value |
|---|---|---|---|
| `ws` | `180 Lithium Ion Battery` | `Technology.name` | `lithium ion battery` |
| `Technology` | `Lithium-ion battery (Utility-scale)` | `Technology.detailed_technology` | `lithium-ion battery (utility-scale)` |
| `year` | `2025` | `Technology.year` | `2025` |
| `est` | `ctrl` | `Technology.case` | `control` |
| — | — | `Technology.region` | `EU`, set by the parser |
| `par` | `Specific investment [MEUR2020/MWh]` | parameter key | `specific investment` |
| `val` | `0.288` | `Parameter.magnitude` | `288000.0` |
| `unit`, `priceyear` | `MEUR/MWh`, `2020` | `Parameter.units` | `EUR_2020 / megawatt_hour` |

The parser also applies the following corrections and substitutions:
Three opinioated transformations are worth following:
The leading three-digit code `180` is stripped from `ws`, the remaining whitespace trimmed and the text lower-cased, which is why the technology is keyed as `lithium ion battery`.
The unit `MEUR/MWh` is rescaled to `EUR/MWh`, multiplying `val` by 1e6, and `priceyear` is folded into the currency, giving the `EUR_2020` form the package uses throughout.

- Convert `MEUR_2020` and `kEUR_2020`/`KEUR_2020` to `EUR_2020` and scale numeric `val` accordingly (×1e6 or ×1e3).
- Specific unit fixes (example: `mol/s/m/MPa1/2` → `mol/s/m/Pa` with value scaling).
- Certain `par` values (e.g., `energy storage capacity for one unit`, `tank volume of example`) are normalized to `capacity`.
## Parser steps in detail

Function `_clean_est_string()` normalizes the `est` column by casefolding it and by replacing `ctrl` with `control`.
**Reading.** The `alldata_flat` sheet is read with `pandas.read_excel(..., engine="calamine", dtype=str)`, so every entry starts as a string.

Function `_filter_parameters(df, filter_flag)`, if `filter_flag` is true, keeps only an allowed set of parameters (e.g., `technical lifetime`, `fixed o&m`, `specific investment`, `variable o&m`, `charge efficiency`, `discharge efficiency`, `capacity`). Otherwise returns the full set.
**Validation.** `_drop_invalid_rows()` checks that the required columns are present and drops rows whose `Technology`, `par`, `val` or `year` is missing or empty. It keeps only rows where `year` contains a four-digit year and `val` contains numeric characters without a comparator symbol (`<`, `>`, `≤`, `≥`).

### Populate and export the source and technology collections
**Cleaning.** Four functions normalise the text fields:

Function `_build_technology_collection()`:
- `_clean_technology_string()` strips leading three-digit codes, trims whitespace and lower-cases, and is applied to `Technology` and `ws`. It turns `151b Hydrogen Storage - LOHC` into `hydrogen storage - lohc`.
- `_clean_parameter_string()` removes leading hyphens and bracketed text, collapses spaces and lower-cases the parameter name.
- `_extract_year()` takes the first digit sequence from `year`, so `Uncertainty (2050)` becomes the integer `2050`.
- `_clean_est_string()` casefolds `est` and expands `ctrl` to `control`.

- if `archive_source` is set, constructs a `Source` object for the DEA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`.
- groups the cleaned DataFrame by `est`, `year`, `ws`, `Technology`.
- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `units`, `sources`, `provenance`).
- creates a `Technology` object for each group, with `name` = `ws`, `detailed_technology` = `Technology`, `year`=`year`, `region` = `EU`, `case` = `est` and collects them into a `TechnologyCollection` object.
- writes the `TechnologyCollection` object to a `technologies.json`.
- if `export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/parsers/schemas`.
**Units.** `_standardize_units()` fills in units missing from the source by parameter name, mapping for instance `energy storage capacity for one unit` to `MWh`, and replaces unit strings that pint cannot read, such as `⁰C` to `C` or `m2` to `meter**2`. `Commons.update_unit_with_currency_year()` then appends `priceyear` to currency units, producing the `EUR_2020` form matched by the package's currency pattern `\b(?P<cu_iso3>[A-Z]{3})_(?P<year>\d{4})\b`. `_format_val_number()` parses comma decimal separators and scientific notation variants such as `×10`, converts to float and rounds to `num_digits`.

## Running the parser
The parser also applies a small set of fixed corrections:

### Execution instructions
- `MEUR_2020` and `kEUR_2020`/`KEUR_2020` become `EUR_2020`, with `val` scaled by 1e6 or 1e3.
- Individual unit repairs, such as `mol/s/m/MPa1/2` to `mol/s/m/Pa`, with the value scaled to match.
- Parameter names such as `energy storage capacity for one unit` and `tank volume of example` are normalised to `capacity`.

The parser is run using the `DataAccessor` class. You need to create an instance of `DataAccessor` with the desired `data_source` and `version`, and then call the `parse()` method.
**Filtering.** `_filter_parameters()` keeps only an allowed set of parameters when `filter_params` is set — `technical lifetime`, `fixed o&m`, `specific investment`, `variable o&m`, `charge efficiency`, `discharge efficiency` and `capacity` — and otherwise returns everything.

Here is an example of how to run the parser from a Python script:
**Building the collection.** `_build_technology_collection()` groups the cleaned frame by `est`, `year`, `ws` and `Technology`. Each group becomes one `Technology`, with `name` from `ws`, `detailed_technology` from `Technology`, `region` fixed to `EU`, `case` from `est`, and a dictionary of `Parameter` objects carrying `magnitude`, `units`, `sources` and `provenance`. When `archive_source` is set it builds a `Source` for the dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise it reads an existing `sources.json`.

## Regenerate the data

The parsed files shipped with the package are produced by the same public entry point, `DataAccessor.parse()`.

!!! warning "`parse()` writes relative to the working directory"
Unlike `load()`, the output path is derived from the current working directory, as `<cwd>/src/technologydata/parsers/dea_energy_storage/v10/`, and ignores `data_path`. Run this from the root of a checkout you are willing to modify: it overwrites the files distributed with the package.

```python
from technologydata.parsers.data_accessor import DataAccessor

# Create an accessor for the version to be parsed
parser_accessor = DataAccessor(
data_source="dea_energy_storage",
version="v10"
version="v10",
)

# Run the parser with desired options
parser_accessor.parse(
input_file_name="Technology_datasheet_for_energy_storage.xlsx",
num_digits=3,
Expand All @@ -94,19 +178,14 @@ parser_accessor.parse(
)
```

The `parse` method accepts the following arguments:

- `input_file_name` (str): The name of the raw data file located in `src/technologydata/parsers/raw/`.
- `num_digits` (int, default 4): Number of decimals for rounding numeric values.
- `archive_source` (bool, default False): Whether to store the source on the Wayback Machine.
- `filter_params` (bool, default False): Whether to filter parameters.
- `export_schema` (bool, default False): Whether to export Pydantic schemas.

### Outputs
`parse()` accepts:

The parser generates the following outputs inside `src/technologydata/parsers/dea_energy_storage/v10/`:
- `input_file_name` (str): name of the raw file in `src/technologydata/parsers/raw/`.
- `num_digits` (int, default 4): number of decimals for rounding numeric values.
- `archive_source` (bool, default False): whether to store the source on the Wayback Machine.
- `filter_params` (bool, default False): whether to restrict the output to the allowed parameter set.
- `export_schema` (bool, default False): whether to export the pydantic schemas.

- `technologies.json`
- `sources.json`
## Outputs

If `export_schema` is set to `True`, the Pydantic schema files are generated and moved to `src/technologydata/parsers/schemas/`.
The parser writes `technologies.json` and `sources.json` into `<cwd>/src/technologydata/parsers/dea_energy_storage/v10/`. With `export_schema=True`, the pydantic schemas are written alongside the JSON files as `technologies.schema.json` and, when `archive_source` is set, `sources.schema.json`.
Loading
Loading