From 0dabe2af48fd1b421c31401b3b92017c0c97f0ee Mon Sep 17 00:00:00 2001 From: cdgaete Date: Sat, 29 Aug 2026 14:24:01 -0400 Subject: [PATCH 1/2] doc: lead the parser examples with runnable code Both example pages opened with narrative before showing anything runnable, and neither showed how the raw data becomes the parsed output. Restructure both to the same shape: Quick start, pipeline diagram, worked row, parser steps, regeneration and outputs. Correct four inaccurate claims, on unit resolution, currency form, schema output paths and the worked row index. --- docs/examples/dea_storage_v10.md | 180 ++++++++++++++++------- docs/examples/manual_input_usa_v0134.md | 188 +++++++++++++++++------- 2 files changed, 259 insertions(+), 109 deletions(-) diff --git a/docs/examples/dea_storage_v10.md b/docs/examples/dea_storage_v10.md index f9a00858..198cd035 100644 --- a/docs/examples/dea_storage_v10.md +++ b/docs/examples/dea_storage_v10.md @@ -1,4 +1,4 @@ -# Danish Energy Agency Parser Documentation +# Danish Energy Agency Parser (v10) -## Overview - !!! note This example refers specifically to **version 10** (`v10`) of the DEA Energy Storage dataset. Details such as file names, sheet structure, and parser behaviour may differ for other versions. -The Danish Energy Agency (DEA) data parser demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/dea_energy_storage/`. +The Danish Energy Agency (DEA) energy storage dataset is the package's main data source. In its raw form it is an Excel workbook whose `alldata_flat` sheet holds one row per parameter record, across the columns `ws`, `Technology`, `cat`, `par`, `unit`, `priceyear`, `note`, `ref`, `est`, `year` and `val`. The parser in `src/technologydata/parsers/dea_energy_storage/` turns those 3127 rows into the schema files `technologies.json` and `sources.json`. -## Dataset Description +The dataset is published by the [Danish Energy Agency](https://ens.dk/media/6589/download) and described in the [accompanying documentation](https://ens.dk/media/6588/download). A copy is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`. -The original dataset is available from the [Danish Energy Agency website](https://ens.dk/media/6589/download). A full description of the dataset is available in the [DEA documentation](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`. +## Quick start -The dataset is in Excel format, and it includes, under the data sheet `alldata_flat`, a flat table of technology parameters for a range of energy storage technologies. Columns include `Technology`, `ws`, `par` (parameter name), `val` (value), `unit`, `year`, `est` (case/estimate), `priceyear`, plus metadata columns such as `cat`, `ref`, `note`. Rows are individual parameter records (parameter value + unit + context) for technologies and estimation cases. +Load the parsed dataset. The catalogues ship inside the installed package, so `data_path` can be derived from the package location and these snippets run from any working directory. -## Parser description +```python +import pathlib -The parser is articulated in the following steps. +import technologydata +from technologydata.parsers.data_accessor import DataAccessor -### Read the raw data +# The bundled catalogues ship inside the installed package. +data = pathlib.Path(technologydata.__file__).parent / "parsers" -The script reads the raw data available at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially. +data_package = DataAccessor( + data_source="dea_energy_storage", + version="v10", + data_path=data, +).load() -### Data cleaning, validation and dealing with missing/null values +technologies = data_package.technologies +print(len(technologies.technologies)) +``` -The data cleaning and validation happens with the following steps. +```text +136 +``` -Function `_drop_invalid_rows(df)` validates whether required columns are present. It drops rows with missing/null or empty critical fields (`Technology`, `par`, `val`, `year`) and keeps rows where `year` contains a 4-digit year and `val` contains numeric characters and no comparator symbols (`<`, `>`, `≤`, `≥`). +`load()` also writes `INFO: Data source directory corresponding to version v10 found.` to standard error. -Function `_clean_technology_string()` normalizes text fields by removing leading 3-digit numeric codes, trims whitespace and lower-cases the string for consistent matching. It is applied to the columns `Technology` and `ws`. As an example, `_clean_technology_string()` converts `151b Hydrogen Storage - LOHC` to `hydrogen storage - lohc`. +Inspect the collection as a table: -Function `_extract_year()` extracts the first sequence of digits from the `year` column and converts it to an integer. The column contains in fact entries like `Uncertainty (2050)` (str) which are converted to `2050` (int). +```python +df = technologies.to_dataframe() +print(df[["detailed_technology", "case", "year"]].head().to_string(index=False)) +``` -Function `_clean_parameter_string()` removes leading hyphens, removes text inside square brackets (units/notes), collapses extra spaces and lower-cases the parameter name. It is applied to the `par` column. +```text + detailed_technology case year + compressed air energy storage control 2015 + hydrogen storage - caverns control 2015 + hydrogen storage - lohc control 2015 +large-scale hot water tanks (steel) control 2015 + na-nicl2 battery control 2015 +``` -Function `_standardize_units()` is applied to columns `par` and `unit`. It completes missing units based on parameter name (e.g., `energy storage capacity for one unit` is mapped to the unit `MWh`) via a parameter-to-unit map. Moreover, it replaces known incorrect unit strings as `⁰C` -> `C` or `m2` to `meter**2`. The unit substitutions are driven by the [pint default units definition](https://github.com/hgrecco/pint/blob/master/pint/default_en.txt). +Read one parameter of one technology: -Function `Commons.update_unit_with_currency_year(unit, priceyear)`, if present, appends `priceyear` information to currency units. This is because `technologydata` follows the currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`, as for example `EUR_2021`. +```python +battery = next( + t + for t in technologies.technologies + if t.detailed_technology == "lithium-ion battery (utility-scale)" + and t.case == "control" + and t.year == 2025 +) +investment = battery.parameters["specific investment"] +print(investment.magnitude, investment.units) +``` -Function `_format_val_number(value, num_decimals)` parses numeric formats including comma decimal separators and scientific notation variants (e.g., `×10`) and converts them to float and rounds them to `num_decimals`. +```text +288000.0 EUR_2020 / megawatt_hour +``` + +## From raw data to parsed output + +Each raw row carries one parameter value together with the context that identifies it. The parser cleans those fields, then groups the rows into `Technology` objects, so that a set of rows sharing a technology, year and estimate case becomes a single object holding a dictionary of `Parameter` values. + +```mermaid +flowchart LR + subgraph raw["alldata_flat row"] + direction TB + r1["ws
Technology"] + r2["par
val
unit
priceyear"] + r3["year
est"] + end + subgraph parser["Parser"] + direction TB + s1["Drop invalid rows"] + s2["Clean names, years,
parameters and units"] + s3["Group by est, year,
ws, Technology"] + s1 --> s2 --> s3 + end + subgraph out["Technology"] + direction TB + o1["name
detailed_technology"] + o2["parameters:
Parameter magnitude, units"] + o3["year
case
region"] + end + raw --> parser --> out +``` + +## A worked row + +Row 2333 of `alldata_flat` (the 0-based pandas index, row 2335 in Excel) records the 2025 specific investment of a utility-scale lithium-ion battery. It reaches the parsed collection as follows. + +| Raw field | Raw value | Parsed field | Parsed value | +|---|---|---|---| +| `ws` | `180 Lithium Ion Battery` | `Technology.name` | `lithium ion battery` | +| `Technology` | `Lithium-ion battery (Utility-scale)` | `Technology.detailed_technology` | `lithium-ion battery (utility-scale)` | +| `year` | `2025` | `Technology.year` | `2025` | +| `est` | `ctrl` | `Technology.case` | `control` | +| — | — | `Technology.region` | `EU`, set by the parser | +| `par` | `Specific investment [MEUR2020/MWh]` | parameter key | `specific investment` | +| `val` | `0.288` | `Parameter.magnitude` | `288000.0` | +| `unit`, `priceyear` | `MEUR/MWh`, `2020` | `Parameter.units` | `EUR_2020 / megawatt_hour` | -The parser also applies the following corrections and substitutions: +Two conversions are worth following. The leading three-digit code `180` is stripped from `ws`, the remaining whitespace trimmed and the text lower-cased, which is why the technology is keyed as `lithium ion battery`. The unit `MEUR/MWh` is rescaled to `EUR/MWh`, multiplying `val` by 1e6, and `priceyear` is folded into the currency, giving the `EUR_2020` form the package uses throughout. -- Convert `MEUR_2020` and `kEUR_2020`/`KEUR_2020` to `EUR_2020` and scale numeric `val` accordingly (×1e6 or ×1e3). -- Specific unit fixes (example: `mol/s/m/MPa1/2` → `mol/s/m/Pa` with value scaling). -- Certain `par` values (e.g., `energy storage capacity for one unit`, `tank volume of example`) are normalized to `capacity`. +## Parser steps in detail -Function `_clean_est_string()` normalizes the `est` column by casefolding it and by replacing `ctrl` with `control`. +**Reading.** The `alldata_flat` sheet is read with `pandas.read_excel(..., engine="calamine", dtype=str)`, so every entry starts as a string. -Function `_filter_parameters(df, filter_flag)`, if `filter_flag` is true, keeps only an allowed set of parameters (e.g., `technical lifetime`, `fixed o&m`, `specific investment`, `variable o&m`, `charge efficiency`, `discharge efficiency`, `capacity`). Otherwise returns the full set. +**Validation.** `_drop_invalid_rows()` checks that the required columns are present and drops rows whose `Technology`, `par`, `val` or `year` is missing or empty. It keeps only rows where `year` contains a four-digit year and `val` contains numeric characters without a comparator symbol (`<`, `>`, `≤`, `≥`). -### Populate and export the source and technology collections +**Cleaning.** Four functions normalise the text fields: -Function `_build_technology_collection()`: +- `_clean_technology_string()` strips leading three-digit codes, trims whitespace and lower-cases, and is applied to `Technology` and `ws`. It turns `151b Hydrogen Storage - LOHC` into `hydrogen storage - lohc`. +- `_clean_parameter_string()` removes leading hyphens and bracketed text, collapses spaces and lower-cases the parameter name. +- `_extract_year()` takes the first digit sequence from `year`, so `Uncertainty (2050)` becomes the integer `2050`. +- `_clean_est_string()` casefolds `est` and expands `ctrl` to `control`. -- if `archive_source` is set, constructs a `Source` object for the DEA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`. -- groups the cleaned DataFrame by `est`, `year`, `ws`, `Technology`. -- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `units`, `sources`, `provenance`). -- creates a `Technology` object for each group, with `name` = `ws`, `detailed_technology` = `Technology`, `year`=`year`, `region` = `EU`, `case` = `est` and collects them into a `TechnologyCollection` object. -- writes the `TechnologyCollection` object to a `technologies.json`. -- if `export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/parsers/schemas`. +**Units.** `_standardize_units()` fills in units missing from the source by parameter name, mapping for instance `energy storage capacity for one unit` to `MWh`, and replaces unit strings that pint cannot read, such as `⁰C` to `C` or `m2` to `meter**2`. `Commons.update_unit_with_currency_year()` then appends `priceyear` to currency units, producing the `EUR_2020` form matched by the package's currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`. `_format_val_number()` parses comma decimal separators and scientific notation variants such as `×10`, converts to float and rounds to `num_digits`. -## Running the parser +The parser also applies a small set of fixed corrections: -### Execution instructions +- `MEUR_2020` and `kEUR_2020`/`KEUR_2020` become `EUR_2020`, with `val` scaled by 1e6 or 1e3. +- Individual unit repairs, such as `mol/s/m/MPa1/2` to `mol/s/m/Pa`, with the value scaled to match. +- Parameter names such as `energy storage capacity for one unit` and `tank volume of example` are normalised to `capacity`. -The parser is run using the `DataAccessor` class. You need to create an instance of `DataAccessor` with the desired `data_source` and `version`, and then call the `parse()` method. +**Filtering.** `_filter_parameters()` keeps only an allowed set of parameters when `filter_params` is set — `technical lifetime`, `fixed o&m`, `specific investment`, `variable o&m`, `charge efficiency`, `discharge efficiency` and `capacity` — and otherwise returns everything. -Here is an example of how to run the parser from a Python script: +**Building the collection.** `_build_technology_collection()` groups the cleaned frame by `est`, `year`, `ws` and `Technology`. Each group becomes one `Technology`, with `name` from `ws`, `detailed_technology` from `Technology`, `region` fixed to `EU`, `case` from `est`, and a dictionary of `Parameter` objects carrying `magnitude`, `units`, `sources` and `provenance`. When `archive_source` is set it builds a `Source` for the dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise it reads an existing `sources.json`. + +## Regenerate the data + +The parsed files shipped with the package are produced by the same public entry point, `DataAccessor.parse()`. + +!!! warning "`parse()` writes relative to the working directory" + Unlike `load()`, the output path is derived from the current working directory, as `/src/technologydata/parsers/dea_energy_storage/v10/`, and ignores `data_path`. Run this from the root of a checkout you are willing to modify: it overwrites the files distributed with the package. ```python from technologydata.parsers.data_accessor import DataAccessor -# Create an accessor for the version to be parsed parser_accessor = DataAccessor( data_source="dea_energy_storage", - version="v10" + version="v10", ) -# Run the parser with desired options parser_accessor.parse( input_file_name="Technology_datasheet_for_energy_storage.xlsx", num_digits=3, @@ -94,19 +171,14 @@ parser_accessor.parse( ) ``` -The `parse` method accepts the following arguments: - -- `input_file_name` (str): The name of the raw data file located in `src/technologydata/parsers/raw/`. -- `num_digits` (int, default 4): Number of decimals for rounding numeric values. -- `archive_source` (bool, default False): Whether to store the source on the Wayback Machine. -- `filter_params` (bool, default False): Whether to filter parameters. -- `export_schema` (bool, default False): Whether to export Pydantic schemas. - -### Outputs +`parse()` accepts: -The parser generates the following outputs inside `src/technologydata/parsers/dea_energy_storage/v10/`: +- `input_file_name` (str): name of the raw file in `src/technologydata/parsers/raw/`. +- `num_digits` (int, default 4): number of decimals for rounding numeric values. +- `archive_source` (bool, default False): whether to store the source on the Wayback Machine. +- `filter_params` (bool, default False): whether to restrict the output to the allowed parameter set. +- `export_schema` (bool, default False): whether to export the pydantic schemas. -- `technologies.json` -- `sources.json` +## Outputs -If `export_schema` is set to `True`, the Pydantic schema files are generated and moved to `src/technologydata/parsers/schemas/`. +The parser writes `technologies.json` and `sources.json` into `/src/technologydata/parsers/dea_energy_storage/v10/`. With `export_schema=True`, the pydantic schemas are written alongside the JSON files as `technologies.schema.json` and, when `archive_source` is set, `sources.schema.json`. diff --git a/docs/examples/manual_input_usa_v0134.md b/docs/examples/manual_input_usa_v0134.md index bc938c5e..f6cdb7da 100644 --- a/docs/examples/manual_input_usa_v0134.md +++ b/docs/examples/manual_input_usa_v0134.md @@ -1,4 +1,4 @@ -# Manual Input USA Parser Documentation +# Manual Input USA Parser (v0.13.4) -## Overview - !!! note This example refers specifically to **version 0.13.4** (`v0134`) of the Manual Input USA dataset. -The Manual Input USA data parser demonstrates a data-cleaning and transformation pipeline for converting manually curated, USA-specific tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/manual_input_usa/`. +The Manual Input USA dataset is a manually curated CSV of USA-specific technology parameters. Its 286 rows each hold one parameter record, across the columns `technology`, `parameter`, `year`, `value`, `unit`, `currency_year`, `source`, `further_description`, `financial_case` and `scenario`. The parser in `src/technologydata/parsers/manual_input_usa/` turns them into the schema files `technologies.json` and `sources.json`. -## Dataset Description +The dataset originates in the [PyPSA technology-data repository](https://github.com/PyPSA/technology-data/blob/v0.13.4/inputs/US/manual_input_usa.csv). A copy is included in the repository at `src/technologydata/parsers/raw/manual_input_usa.csv`. -The original dataset is a manually curated CSV file containing USA-specific technology parameters available from the [PyPSA technology-data repository](https://github.com/PyPSA/technology-data/blob/v0.13.4/inputs/US/manual_input_usa.csv). The raw source file is included in the repository at `src/technologydata/parsers/raw/manual_input_usa.csv`. +## Quick start -The dataset is in CSV format and includes a flat table of technology parameters for various energy technologies relevant to the USA context. Columns include `technology`, `parameter`, `year`, `value`, `unit`, `currency_year`, `source`, `further_description`, `financial_case`, and `scenario`. Rows are individual parameter records (parameter value + unit + context) for technologies with different scenarios and financial cases. +Load the parsed dataset. The catalogues ship inside the installed package, so `data_path` can be derived from the package location and these snippets run from any working directory. -## Parser description +```python +import pathlib -The parser is articulated in the following steps. +import technologydata +from technologydata.parsers.data_accessor import DataAccessor -### Read the raw data +# The bundled catalogues ship inside the installed package. +data = pathlib.Path(technologydata.__file__).parent / "parsers" -The script reads the raw data available at `src/technologydata/parsers/raw/manual_input_usa.csv` in a `pandas` dataframe. It uses `pandas.read_csv(..., dtype=str, na_values="None")`. All entries are handled as strings initially except for the `value` column which is converted to float. +data_package = DataAccessor( + data_source="manual_input_usa", + version="v0.13.4", + data_path=data, +).load() -### Data cleaning, validation and dealing with missing/null values +technologies = data_package.technologies +print(len(technologies.technologies)) +``` -The data cleaning and validation happens with the following steps. +```text +85 +``` -Function `_extract_units_carriers_heating_value()` extracts standardized units, carriers, and heating values from input unit strings. This function maps complex unit representations to simplified unit, carrier, and heating value combinations using a predefined dictionary of special patterns. Examples include: +`load()` also writes `INFO: Data source directory corresponding to version v0.13.4 found.` to standard error. -- `USD_2022/MW_FT` → unit: `USD_2022/MW`, carrier: `1/FT`, heating_value: `1/LHV` -- `MWh_H2/MWh_FT` → unit: `MWh/MWh`, carrier: `H2/FT`, heating_value: `LHV` -- `MWh_el/MWh_FT` → unit: `MWh/MWh`, carrier: `el/FT`, heating_value: `LHV` -- `t_CO2/MWh_FT` → unit: `t/MWh`, carrier: `CO2/FT`, heating_value: `LHV` -- `USD_2022/kWh_H2` → unit: `USD_2022/kWh`, carrier: `1/H2`, heating_value: `LHV` -- `USD_2023/t_CO2/h` → unit: `USD_2023/t/h`, carrier: `1/CO2`, heating_value: `None` -- `MWh_el/t_CO2` → unit: `MWh/t`, carrier: `el/CO2`, heating_value: `LHV` -- `MWh_th/t_CO2` → unit: `MWh/t`, carrier: `thermal/CO2`, heating_value: `LHV` +Inspect the collection as a table: -The parser also fills missing values in the `scenario` column with `"not_available"`. +```python +df = technologies.to_dataframe() +print(df[["detailed_technology", "case", "year"]].head().to_string(index=False)) +``` -The parser applies the following unit conversions: +```text + detailed_technology case year +Alkaline electrolyzer large size Advanced - Market 2020 + PEM electrolyzer small size Advanced - Market 2020 + SOEC Advanced - Market 2020 + direct air capture Advanced - Market 2020 + battery inverter Advanced - Market 2022 +``` -- Convert `per unit` to `%` and multiply the corresponding `value` by 100.0, rounding to `num_digits` decimals. +Read one parameter of one technology: -Function `Commons.update_unit_with_currency_year(unit, currency_year)` appends `currency_year` information to currency units when present. This is because `technologydata` follows the currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`, as for example `USD_2022`. +```python +fischer_tropsch = next( + t + for t in technologies.technologies + if t.name == "Fischer-Tropsch" and t.year == 2020 +) +hydrogen_input = fischer_tropsch.parameters["hydrogen-input"] +print(hydrogen_input.magnitude, "|", hydrogen_input.units) +print(hydrogen_input.carrier, "|", hydrogen_input.heating_value) +``` -### Populate and export the source and technology collections +```text +1.43 | dimensionless +hydrogen / fischer_tropsch | lower_heating_value +``` -Function `_build_technology_collection()`: +## From raw data to parsed output + +Each raw row carries one parameter value together with the context that identifies it. The distinctive step for this dataset is the unit string: entries such as `MWh_H2/MWh_FT` encode a unit, an energy carrier and a heating value in one field, and the parser splits them into three. Rows are then grouped into `Technology` objects. + +```mermaid +flowchart LR + subgraph raw["manual_input_usa.csv row"] + direction TB + r1["technology
parameter"] + r2["value
unit
currency_year"] + r3["year
scenario
financial_case"] + end + subgraph parser["Parser"] + direction TB + s1["Fill missing scenario
convert per unit to %"] + s2["Split unit into unit,
carrier, heating value"] + s3["Group by scenario,
year, technology"] + s1 --> s2 --> s3 + end + subgraph out["Technology"] + direction TB + o1["name
detailed_technology"] + o2["parameters:
Parameter magnitude, units,
carrier, heating_value"] + o3["year
case
region"] + end + raw --> parser --> out +``` + +## A worked row + +The `Fischer-Tropsch` / `hydrogen-input` row for 2020 exercises every one of those steps: it has a compound unit, no `currency_year`, and no `scenario`. + +| Raw field | Raw value | Parsed field | Parsed value | +|---|---|---|---| +| `technology` | `Fischer-Tropsch` | `Technology.name` | `Fischer-Tropsch` | +| `technology` | `Fischer-Tropsch` | `Technology.detailed_technology` | `Fischer-Tropsch` | +| `year` | `2020` | `Technology.year` | `2020` | +| `scenario` | empty | `Technology.case` | `not_available` | +| — | — | `Technology.region` | `USA`, set by the parser | +| `parameter` | `hydrogen-input` | parameter key | `hydrogen-input` | +| `value` | `1.43` | `Parameter.magnitude` | `1.43` | +| `unit` | `MWh_H2/MWh_FT` | `Parameter.units` | `dimensionless` | +| `unit` | `MWh_H2/MWh_FT` | `Parameter.carrier` | `hydrogen / fischer_tropsch` | +| `unit` | `MWh_H2/MWh_FT` | `Parameter.heating_value` | `lower_heating_value` | +| `further_description` | `0.995 MWh_H2 per output, …` | `Parameter.note` | the same text | + +The unit is resolved in two stages, which is why the stored strings differ from the ones the parser looks up. `_extract_units_carriers_heating_value()` maps `MWh_H2/MWh_FT` to the triple `("MWh/MWh", "H2/FT", "LHV")`. Constructing the `Parameter` then resolves each of those through the package's pint registries: `MWh/MWh` cancels to `dimensionless`, while `H2` and `FT` expand to their canonical names via `src/technologydata/utils/carriers.txt`, and `LHV` via `src/technologydata/utils/heating_values.txt`. + +## Parser steps in detail -- if `archive_source` is set, constructs a `Source` object for the manual input USA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`. -- groups the cleaned DataFrame by `scenario`, `year`, `technology`. -- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `sources`, and optionally `carrier`, `heating_value`, `units`, `note`). -- captures the `financial_case` value from rows within each group to combine with `scenario`. -- creates a `case` value by combining `scenario` and `financial_case` in the format `"{scenario} - {financial_case}"` when `financial_case` is present; otherwise uses `scenario` alone. -- creates a `Technology` object for each group, with `name` = `technology`, `detailed_technology` = `technology`, `year` = `year`, `region` = `USA`, `case` = combined case value, and collects them into a `TechnologyCollection` object. -- writes the `TechnologyCollection` object to a `technologies.json`. +**Reading.** The CSV is read with `pandas.read_csv(..., dtype=str, na_values="None")`, so every entry starts as a string except `value`, which is cast to float. -## Running the parser +**Units, carriers and heating values.** `_extract_units_carriers_heating_value()` maps compound unit strings onto a `(unit, carrier, heating_value)` triple through a fixed dictionary, and returns the input unchanged with two `None`s when there is no match. The nine mapped patterns are: -### Execution instructions +| Input unit | Unit | Carrier | Heating value | +|---|---|---|---| +| `USD_2022/MW_FT` | `USD_2022/MW` | `1/FT` | `1/LHV` | +| `MWh_H2/MWh_FT` | `MWh/MWh` | `H2/FT` | `LHV` | +| `MWh_el/MWh_FT` | `MWh/MWh` | `el/FT` | `LHV` | +| `t_CO2/MWh_FT` | `t/MWh` | `CO2/FT` | `LHV` | +| `USD_2022/kWh_H2` | `USD_2022/kWh` | `1/H2` | `LHV` | +| `MWh_el/MWh_H2` | `MWh/MWh` | `el/H2` | `LHV` | +| `USD_2023/t_CO2/h` | `USD_2023/t/h` | `1/CO2` | none | +| `MWh_el/t_CO2` | `MWh/t` | `el/CO2` | `LHV` | +| `MWh_th/t_CO2` | `MWh/t` | `thermal/CO2` | `LHV` | -The parser is run using the `DataAccessor` class. You need to create an instance of `DataAccessor` with the desired `data_source` and `version`, and then call the `parse()` method. +These are the values the mapping returns, not the values that end up stored. As shown in the worked row above, `Parameter` resolves them through the pint registries in `src/technologydata/utils/`, so `H2/FT` is stored as `hydrogen / fischer_tropsch` and `LHV` as `lower_heating_value`. -Here is an example of how to run the parser from a Python script: +**Other cleaning.** Missing `scenario` entries are filled with `not_available`. Units containing `per unit` are rewritten to `%` and the corresponding `value` multiplied by 100. `Commons.update_unit_with_currency_year()` appends `currency_year` to currency units where present, producing the `USD_2022` form matched by the package's currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`. + +**Building the collection.** `_build_technology_collection()` groups the cleaned frame by `scenario`, `year` and `technology`. Each group becomes one `Technology`, with both `name` and `detailed_technology` taken from `technology`, `region` fixed to `USA`, and a dictionary of `Parameter` objects carrying `magnitude`, `sources` and, where available, `carrier`, `heating_value`, `units` and `note`. The `case` combines the two scenario fields as `"{scenario} - {financial_case}"` when a `financial_case` is present in the group, and is the `scenario` alone otherwise. When `archive_source` is set it builds a `Source` for the dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise it reads an existing `sources.json`. + +## Regenerate the data + +The parsed files shipped with the package are produced by the same public entry point, `DataAccessor.parse()`. + +!!! warning "`parse()` writes relative to the working directory" + Unlike `load()`, the output path is derived from the current working directory, as `/src/technologydata/parsers/manual_input_usa/v0.13.4/`, and ignores `data_path`. Run this from the root of a checkout you are willing to modify: it overwrites the files distributed with the package. ```python from technologydata.parsers.data_accessor import DataAccessor -# Create an accessor for the version to be parsed parser_accessor = DataAccessor( data_source="manual_input_usa", - version="v0.13.4" + version="v0.13.4", ) -# Run the parser with desired options parser_accessor.parse( input_file_name="manual_input_usa.csv", num_digits=3, @@ -89,19 +172,14 @@ parser_accessor.parse( ) ``` -The `parse` method accepts the following arguments: - -- `input_file_name` (str): The name of the raw data file located in `src/technologydata/parsers/raw/`. -- `num_digits` (int, default 4): Number of decimals for rounding numeric values. -- `archive_source` (bool, default False): Whether to store the source on the Wayback Machine. -- `filter_params` (bool, default False): Whether to filter parameters (not used by this parser). -- `export_schema` (bool, default False): Whether to export Pydantic schemas. - -### Outputs +`parse()` accepts: -The parser generates the following outputs inside `src/technologydata/parsers/manual_input_usa/v0.13.4/`: +- `input_file_name` (str): name of the raw file in `src/technologydata/parsers/raw/`. +- `num_digits` (int, default 4): number of decimals for rounding numeric values. +- `archive_source` (bool, default False): whether to store the source on the Wayback Machine. +- `filter_params` (bool, default False): accepted for interface compatibility and unused by this parser. +- `export_schema` (bool, default False): whether to export the pydantic schemas. -- `technologies.json` -- `sources.json` +## Outputs -If `export_schema` is set to `True`, the Pydantic schema files are generated and moved to `src/technologydata/parsers/schemas/`. +The parser writes `technologies.json` and `sources.json` into `/src/technologydata/parsers/manual_input_usa/v0.13.4/`. With `export_schema=True`, the pydantic schemas are written alongside the JSON files as `technologies.schema.json` and, when `archive_source` is set, `sources.schema.json`. From 2e2f5e91a9ae3ffcf2a41ff4b90ccf193ce30e9e Mon Sep 17 00:00:00 2001 From: Johannes <42553970+euronion@users.noreply.github.com> Date: Wed, 2 Sep 2026 21:31:32 +0200 Subject: [PATCH 2/2] docs: Simplify documentation for DEA storage data v10 --- .claude/worktrees/agent-ab7e7ae9f0428c62d | 1 + docs/examples/dea_storage_v10.md | 73 +++++++++++++---------- 2 files changed, 41 insertions(+), 33 deletions(-) create mode 160000 .claude/worktrees/agent-ab7e7ae9f0428c62d diff --git a/.claude/worktrees/agent-ab7e7ae9f0428c62d b/.claude/worktrees/agent-ab7e7ae9f0428c62d new file mode 160000 index 00000000..055a769a --- /dev/null +++ b/.claude/worktrees/agent-ab7e7ae9f0428c62d @@ -0,0 +1 @@ +Subproject commit 055a769ab62f077f7be38ae3978db84dfded017b diff --git a/docs/examples/dea_storage_v10.md b/docs/examples/dea_storage_v10.md index 198cd035..0ad8bcf7 100644 --- a/docs/examples/dea_storage_v10.md +++ b/docs/examples/dea_storage_v10.md @@ -10,39 +10,33 @@ SPDX-License-Identifier: MIT !!! note This example refers specifically to **version 10** (`v10`) of the DEA Energy Storage dataset. Details such as file names, sheet structure, and parser behaviour may differ for other versions. -The Danish Energy Agency (DEA) energy storage dataset is the package's main data source. In its raw form it is an Excel workbook whose `alldata_flat` sheet holds one row per parameter record, across the columns `ws`, `Technology`, `cat`, `par`, `unit`, `priceyear`, `note`, `ref`, `est`, `year` and `val`. The parser in `src/technologydata/parsers/dea_energy_storage/` turns those 3127 rows into the schema files `technologies.json` and `sources.json`. +The Danish Energy Agency (DEA) maintains a dataset of techno-economic data for different energy storage technologies. +The data in its raw format is available in PDF and as an Excel file. +We parse the Excel file and extract more than 3000 individual parameters into our data schema and make them available through the package. -The dataset is published by the [Danish Energy Agency](https://ens.dk/media/6589/download) and described in the [accompanying documentation](https://ens.dk/media/6588/download). A copy is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`. +The dataset is published by the Danish Energy Agency via [this website](https://ens.dk/media/6589/download) and described in the [accompanying documentation](https://ens.dk/media/6588/download). +The data is licensed CC-BY-4.0. ## Quick start -Load the parsed dataset. The catalogues ship inside the installed package, so `data_path` can be derived from the package location and these snippets run from any working directory. +Load the parsed dataset. +The data ships with the package ```python -import pathlib - -import technologydata -from technologydata.parsers.data_accessor import DataAccessor +from technologydata import DataAccessor # The bundled catalogues ship inside the installed package. -data = pathlib.Path(technologydata.__file__).parent / "parsers" data_package = DataAccessor( data_source="dea_energy_storage", version="v10", - data_path=data, ).load() technologies = data_package.technologies print(len(technologies.technologies)) +> 136 ``` -```text -136 -``` - -`load()` also writes `INFO: Data source directory corresponding to version v10 found.` to standard error. - Inspect the collection as a table: ```python @@ -59,31 +53,40 @@ large-scale hot water tanks (steel) control 2015 na-nicl2 battery control 2015 ``` -Read one parameter of one technology: +Select the data for all years of Li-ion utility scale battery in the EU: ```python -battery = next( - t - for t in technologies.technologies - if t.detailed_technology == "lithium-ion battery (utility-scale)" - and t.case == "control" - and t.year == 2025 +batteries = data.technologies.get( + name="lithium ion battery", + detailed_technology="utility-scale", + region="EU", + case="control", + # Use regex to match all years from 2000 to 2099 + year="20\\d+", ) -investment = battery.parameters["specific investment"] -print(investment.magnitude, investment.units) + +len(batteries) +# 5 ``` -```text -288000.0 EUR_2020 / megawatt_hour +The technologies are also accessible as a list. +Get the first technology from the list and access one of its parameters (the speciic investment costs): + +```python +investment = batteries.technologies[0].parameters["specific investment"] +print(investment) +# 288000.0 EUR_2020 / megawatt_hour ``` ## From raw data to parsed output -Each raw row carries one parameter value together with the context that identifies it. The parser cleans those fields, then groups the rows into `Technology` objects, so that a set of rows sharing a technology, year and estimate case becomes a single object holding a dictionary of `Parameter` values. +Each raw row carries one parameter value together with the context that identifies it. +The parser written for `technologydata` cleans and harmonises these fields, then groups the rows into `Technology` objects. +Each technology object is then created based on the group of rows identifying this technology and holds a dictionary of `Parameter` values. ```mermaid flowchart LR - subgraph raw["alldata_flat row"] + subgraph raw["DEA Excel file (alldata_flat row)"] direction TB r1["ws
Technology"] r2["par
val
unit
priceyear"] @@ -98,16 +101,18 @@ flowchart LR end subgraph out["Technology"] direction TB - o1["name
detailed_technology"] + o1["name
detailed_technology
case
region
year"] o2["parameters:
Parameter magnitude, units"] - o3["year
case
region"] + o3["Source information"] + o4["Provenance information"] end raw --> parser --> out ``` -## A worked row +## Deep dive: From raw to parsed data -Row 2333 of `alldata_flat` (the 0-based pandas index, row 2335 in Excel) records the 2025 specific investment of a utility-scale lithium-ion battery. It reaches the parsed collection as follows. +An example for a speciic row, e.g. row 2335 of Excel file's worksheet `alldata_flat` contains the 2025 specific investment of a utility-scale lithium-ion battery. +It reaches the parsed collection as follows. | Raw field | Raw value | Parsed field | Parsed value | |---|---|---|---| @@ -120,7 +125,9 @@ Row 2333 of `alldata_flat` (the 0-based pandas index, row 2335 in Excel) records | `val` | `0.288` | `Parameter.magnitude` | `288000.0` | | `unit`, `priceyear` | `MEUR/MWh`, `2020` | `Parameter.units` | `EUR_2020 / megawatt_hour` | -Two conversions are worth following. The leading three-digit code `180` is stripped from `ws`, the remaining whitespace trimmed and the text lower-cased, which is why the technology is keyed as `lithium ion battery`. The unit `MEUR/MWh` is rescaled to `EUR/MWh`, multiplying `val` by 1e6, and `priceyear` is folded into the currency, giving the `EUR_2020` form the package uses throughout. +Three opinioated transformations are worth following: +The leading three-digit code `180` is stripped from `ws`, the remaining whitespace trimmed and the text lower-cased, which is why the technology is keyed as `lithium ion battery`. +The unit `MEUR/MWh` is rescaled to `EUR/MWh`, multiplying `val` by 1e6, and `priceyear` is folded into the currency, giving the `EUR_2020` form the package uses throughout. ## Parser steps in detail