diff --git a/.claude/worktrees/agent-ab7e7ae9f0428c62d b/.claude/worktrees/agent-ab7e7ae9f0428c62d new file mode 160000 index 00000000..055a769a --- /dev/null +++ b/.claude/worktrees/agent-ab7e7ae9f0428c62d @@ -0,0 +1 @@ +Subproject commit 055a769ab62f077f7be38ae3978db84dfded017b diff --git a/docs/examples/dea_storage_v10.md b/docs/examples/dea_storage_v10.md index f9a00858..0ad8bcf7 100644 --- a/docs/examples/dea_storage_v10.md +++ b/docs/examples/dea_storage_v10.md @@ -1,4 +1,4 @@ -# Danish Energy Agency Parser Documentation +# Danish Energy Agency Parser (v10) -## Overview - !!! note This example refers specifically to **version 10** (`v10`) of the DEA Energy Storage dataset. Details such as file names, sheet structure, and parser behaviour may differ for other versions. -The Danish Energy Agency (DEA) data parser demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/dea_energy_storage/`. +The Danish Energy Agency (DEA) maintains a dataset of techno-economic data for different energy storage technologies. +The data in its raw format is available in PDF and as an Excel file. +We parse the Excel file and extract more than 3000 individual parameters into our data schema and make them available through the package. -## Dataset Description +The dataset is published by the Danish Energy Agency via [this website](https://ens.dk/media/6589/download) and described in the [accompanying documentation](https://ens.dk/media/6588/download). +The data is licensed CC-BY-4.0. -The original dataset is available from the [Danish Energy Agency website](https://ens.dk/media/6589/download). A full description of the dataset is available in the [DEA documentation](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`. +## Quick start -The dataset is in Excel format, and it includes, under the data sheet `alldata_flat`, a flat table of technology parameters for a range of energy storage technologies. Columns include `Technology`, `ws`, `par` (parameter name), `val` (value), `unit`, `year`, `est` (case/estimate), `priceyear`, plus metadata columns such as `cat`, `ref`, `note`. Rows are individual parameter records (parameter value + unit + context) for technologies and estimation cases. +Load the parsed dataset. +The data ships with the package -## Parser description +```python +from technologydata import DataAccessor -The parser is articulated in the following steps. +# The bundled catalogues ship inside the installed package. -### Read the raw data +data_package = DataAccessor( + data_source="dea_energy_storage", + version="v10", +).load() -The script reads the raw data available at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially. +technologies = data_package.technologies +print(len(technologies.technologies)) +> 136 +``` -### Data cleaning, validation and dealing with missing/null values +Inspect the collection as a table: -The data cleaning and validation happens with the following steps. +```python +df = technologies.to_dataframe() +print(df[["detailed_technology", "case", "year"]].head().to_string(index=False)) +``` -Function `_drop_invalid_rows(df)` validates whether required columns are present. It drops rows with missing/null or empty critical fields (`Technology`, `par`, `val`, `year`) and keeps rows where `year` contains a 4-digit year and `val` contains numeric characters and no comparator symbols (`<`, `>`, `≤`, `≥`). +```text + detailed_technology case year + compressed air energy storage control 2015 + hydrogen storage - caverns control 2015 + hydrogen storage - lohc control 2015 +large-scale hot water tanks (steel) control 2015 + na-nicl2 battery control 2015 +``` -Function `_clean_technology_string()` normalizes text fields by removing leading 3-digit numeric codes, trims whitespace and lower-cases the string for consistent matching. It is applied to the columns `Technology` and `ws`. As an example, `_clean_technology_string()` converts `151b Hydrogen Storage - LOHC` to `hydrogen storage - lohc`. +Select the data for all years of Li-ion utility scale battery in the EU: -Function `_extract_year()` extracts the first sequence of digits from the `year` column and converts it to an integer. The column contains in fact entries like `Uncertainty (2050)` (str) which are converted to `2050` (int). +```python +batteries = data.technologies.get( + name="lithium ion battery", + detailed_technology="utility-scale", + region="EU", + case="control", + # Use regex to match all years from 2000 to 2099 + year="20\\d+", +) -Function `_clean_parameter_string()` removes leading hyphens, removes text inside square brackets (units/notes), collapses extra spaces and lower-cases the parameter name. It is applied to the `par` column. +len(batteries) +# 5 +``` -Function `_standardize_units()` is applied to columns `par` and `unit`. It completes missing units based on parameter name (e.g., `energy storage capacity for one unit` is mapped to the unit `MWh`) via a parameter-to-unit map. Moreover, it replaces known incorrect unit strings as `⁰C` -> `C` or `m2` to `meter**2`. The unit substitutions are driven by the [pint default units definition](https://github.com/hgrecco/pint/blob/master/pint/default_en.txt). +The technologies are also accessible as a list. +Get the first technology from the list and access one of its parameters (the speciic investment costs): -Function `Commons.update_unit_with_currency_year(unit, priceyear)`, if present, appends `priceyear` information to currency units. This is because `technologydata` follows the currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`, as for example `EUR_2021`. +```python +investment = batteries.technologies[0].parameters["specific investment"] +print(investment) +# 288000.0 EUR_2020 / megawatt_hour +``` + +## From raw data to parsed output + +Each raw row carries one parameter value together with the context that identifies it. +The parser written for `technologydata` cleans and harmonises these fields, then groups the rows into `Technology` objects. +Each technology object is then created based on the group of rows identifying this technology and holds a dictionary of `Parameter` values. + +```mermaid +flowchart LR + subgraph raw["DEA Excel file (alldata_flat row)"] + direction TB + r1["ws
Technology"] + r2["par
val
unit
priceyear"] + r3["year
est"] + end + subgraph parser["Parser"] + direction TB + s1["Drop invalid rows"] + s2["Clean names, years,
parameters and units"] + s3["Group by est, year,
ws, Technology"] + s1 --> s2 --> s3 + end + subgraph out["Technology"] + direction TB + o1["name
detailed_technology
case
region
year"] + o2["parameters:
Parameter magnitude, units"] + o3["Source information"] + o4["Provenance information"] + end + raw --> parser --> out +``` + +## Deep dive: From raw to parsed data + +An example for a speciic row, e.g. row 2335 of Excel file's worksheet `alldata_flat` contains the 2025 specific investment of a utility-scale lithium-ion battery. +It reaches the parsed collection as follows. -Function `_format_val_number(value, num_decimals)` parses numeric formats including comma decimal separators and scientific notation variants (e.g., `×10`) and converts them to float and rounds them to `num_decimals`. +| Raw field | Raw value | Parsed field | Parsed value | +|---|---|---|---| +| `ws` | `180 Lithium Ion Battery` | `Technology.name` | `lithium ion battery` | +| `Technology` | `Lithium-ion battery (Utility-scale)` | `Technology.detailed_technology` | `lithium-ion battery (utility-scale)` | +| `year` | `2025` | `Technology.year` | `2025` | +| `est` | `ctrl` | `Technology.case` | `control` | +| — | — | `Technology.region` | `EU`, set by the parser | +| `par` | `Specific investment [MEUR2020/MWh]` | parameter key | `specific investment` | +| `val` | `0.288` | `Parameter.magnitude` | `288000.0` | +| `unit`, `priceyear` | `MEUR/MWh`, `2020` | `Parameter.units` | `EUR_2020 / megawatt_hour` | -The parser also applies the following corrections and substitutions: +Three opinioated transformations are worth following: +The leading three-digit code `180` is stripped from `ws`, the remaining whitespace trimmed and the text lower-cased, which is why the technology is keyed as `lithium ion battery`. +The unit `MEUR/MWh` is rescaled to `EUR/MWh`, multiplying `val` by 1e6, and `priceyear` is folded into the currency, giving the `EUR_2020` form the package uses throughout. -- Convert `MEUR_2020` and `kEUR_2020`/`KEUR_2020` to `EUR_2020` and scale numeric `val` accordingly (×1e6 or ×1e3). -- Specific unit fixes (example: `mol/s/m/MPa1/2` → `mol/s/m/Pa` with value scaling). -- Certain `par` values (e.g., `energy storage capacity for one unit`, `tank volume of example`) are normalized to `capacity`. +## Parser steps in detail -Function `_clean_est_string()` normalizes the `est` column by casefolding it and by replacing `ctrl` with `control`. +**Reading.** The `alldata_flat` sheet is read with `pandas.read_excel(..., engine="calamine", dtype=str)`, so every entry starts as a string. -Function `_filter_parameters(df, filter_flag)`, if `filter_flag` is true, keeps only an allowed set of parameters (e.g., `technical lifetime`, `fixed o&m`, `specific investment`, `variable o&m`, `charge efficiency`, `discharge efficiency`, `capacity`). Otherwise returns the full set. +**Validation.** `_drop_invalid_rows()` checks that the required columns are present and drops rows whose `Technology`, `par`, `val` or `year` is missing or empty. It keeps only rows where `year` contains a four-digit year and `val` contains numeric characters without a comparator symbol (`<`, `>`, `≤`, `≥`). -### Populate and export the source and technology collections +**Cleaning.** Four functions normalise the text fields: -Function `_build_technology_collection()`: +- `_clean_technology_string()` strips leading three-digit codes, trims whitespace and lower-cases, and is applied to `Technology` and `ws`. It turns `151b Hydrogen Storage - LOHC` into `hydrogen storage - lohc`. +- `_clean_parameter_string()` removes leading hyphens and bracketed text, collapses spaces and lower-cases the parameter name. +- `_extract_year()` takes the first digit sequence from `year`, so `Uncertainty (2050)` becomes the integer `2050`. +- `_clean_est_string()` casefolds `est` and expands `ctrl` to `control`. -- if `archive_source` is set, constructs a `Source` object for the DEA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`. -- groups the cleaned DataFrame by `est`, `year`, `ws`, `Technology`. -- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `units`, `sources`, `provenance`). -- creates a `Technology` object for each group, with `name` = `ws`, `detailed_technology` = `Technology`, `year`=`year`, `region` = `EU`, `case` = `est` and collects them into a `TechnologyCollection` object. -- writes the `TechnologyCollection` object to a `technologies.json`. -- if `export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/parsers/schemas`. +**Units.** `_standardize_units()` fills in units missing from the source by parameter name, mapping for instance `energy storage capacity for one unit` to `MWh`, and replaces unit strings that pint cannot read, such as `⁰C` to `C` or `m2` to `meter**2`. `Commons.update_unit_with_currency_year()` then appends `priceyear` to currency units, producing the `EUR_2020` form matched by the package's currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`. `_format_val_number()` parses comma decimal separators and scientific notation variants such as `×10`, converts to float and rounds to `num_digits`. -## Running the parser +The parser also applies a small set of fixed corrections: -### Execution instructions +- `MEUR_2020` and `kEUR_2020`/`KEUR_2020` become `EUR_2020`, with `val` scaled by 1e6 or 1e3. +- Individual unit repairs, such as `mol/s/m/MPa1/2` to `mol/s/m/Pa`, with the value scaled to match. +- Parameter names such as `energy storage capacity for one unit` and `tank volume of example` are normalised to `capacity`. -The parser is run using the `DataAccessor` class. You need to create an instance of `DataAccessor` with the desired `data_source` and `version`, and then call the `parse()` method. +**Filtering.** `_filter_parameters()` keeps only an allowed set of parameters when `filter_params` is set — `technical lifetime`, `fixed o&m`, `specific investment`, `variable o&m`, `charge efficiency`, `discharge efficiency` and `capacity` — and otherwise returns everything. -Here is an example of how to run the parser from a Python script: +**Building the collection.** `_build_technology_collection()` groups the cleaned frame by `est`, `year`, `ws` and `Technology`. Each group becomes one `Technology`, with `name` from `ws`, `detailed_technology` from `Technology`, `region` fixed to `EU`, `case` from `est`, and a dictionary of `Parameter` objects carrying `magnitude`, `units`, `sources` and `provenance`. When `archive_source` is set it builds a `Source` for the dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise it reads an existing `sources.json`. + +## Regenerate the data + +The parsed files shipped with the package are produced by the same public entry point, `DataAccessor.parse()`. + +!!! warning "`parse()` writes relative to the working directory" + Unlike `load()`, the output path is derived from the current working directory, as `/src/technologydata/parsers/dea_energy_storage/v10/`, and ignores `data_path`. Run this from the root of a checkout you are willing to modify: it overwrites the files distributed with the package. ```python from technologydata.parsers.data_accessor import DataAccessor -# Create an accessor for the version to be parsed parser_accessor = DataAccessor( data_source="dea_energy_storage", - version="v10" + version="v10", ) -# Run the parser with desired options parser_accessor.parse( input_file_name="Technology_datasheet_for_energy_storage.xlsx", num_digits=3, @@ -94,19 +178,14 @@ parser_accessor.parse( ) ``` -The `parse` method accepts the following arguments: - -- `input_file_name` (str): The name of the raw data file located in `src/technologydata/parsers/raw/`. -- `num_digits` (int, default 4): Number of decimals for rounding numeric values. -- `archive_source` (bool, default False): Whether to store the source on the Wayback Machine. -- `filter_params` (bool, default False): Whether to filter parameters. -- `export_schema` (bool, default False): Whether to export Pydantic schemas. - -### Outputs +`parse()` accepts: -The parser generates the following outputs inside `src/technologydata/parsers/dea_energy_storage/v10/`: +- `input_file_name` (str): name of the raw file in `src/technologydata/parsers/raw/`. +- `num_digits` (int, default 4): number of decimals for rounding numeric values. +- `archive_source` (bool, default False): whether to store the source on the Wayback Machine. +- `filter_params` (bool, default False): whether to restrict the output to the allowed parameter set. +- `export_schema` (bool, default False): whether to export the pydantic schemas. -- `technologies.json` -- `sources.json` +## Outputs -If `export_schema` is set to `True`, the Pydantic schema files are generated and moved to `src/technologydata/parsers/schemas/`. +The parser writes `technologies.json` and `sources.json` into `/src/technologydata/parsers/dea_energy_storage/v10/`. With `export_schema=True`, the pydantic schemas are written alongside the JSON files as `technologies.schema.json` and, when `archive_source` is set, `sources.schema.json`. diff --git a/docs/examples/manual_input_usa_v0134.md b/docs/examples/manual_input_usa_v0134.md index bc938c5e..f6cdb7da 100644 --- a/docs/examples/manual_input_usa_v0134.md +++ b/docs/examples/manual_input_usa_v0134.md @@ -1,4 +1,4 @@ -# Manual Input USA Parser Documentation +# Manual Input USA Parser (v0.13.4) -## Overview - !!! note This example refers specifically to **version 0.13.4** (`v0134`) of the Manual Input USA dataset. -The Manual Input USA data parser demonstrates a data-cleaning and transformation pipeline for converting manually curated, USA-specific tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/manual_input_usa/`. +The Manual Input USA dataset is a manually curated CSV of USA-specific technology parameters. Its 286 rows each hold one parameter record, across the columns `technology`, `parameter`, `year`, `value`, `unit`, `currency_year`, `source`, `further_description`, `financial_case` and `scenario`. The parser in `src/technologydata/parsers/manual_input_usa/` turns them into the schema files `technologies.json` and `sources.json`. -## Dataset Description +The dataset originates in the [PyPSA technology-data repository](https://github.com/PyPSA/technology-data/blob/v0.13.4/inputs/US/manual_input_usa.csv). A copy is included in the repository at `src/technologydata/parsers/raw/manual_input_usa.csv`. -The original dataset is a manually curated CSV file containing USA-specific technology parameters available from the [PyPSA technology-data repository](https://github.com/PyPSA/technology-data/blob/v0.13.4/inputs/US/manual_input_usa.csv). The raw source file is included in the repository at `src/technologydata/parsers/raw/manual_input_usa.csv`. +## Quick start -The dataset is in CSV format and includes a flat table of technology parameters for various energy technologies relevant to the USA context. Columns include `technology`, `parameter`, `year`, `value`, `unit`, `currency_year`, `source`, `further_description`, `financial_case`, and `scenario`. Rows are individual parameter records (parameter value + unit + context) for technologies with different scenarios and financial cases. +Load the parsed dataset. The catalogues ship inside the installed package, so `data_path` can be derived from the package location and these snippets run from any working directory. -## Parser description +```python +import pathlib -The parser is articulated in the following steps. +import technologydata +from technologydata.parsers.data_accessor import DataAccessor -### Read the raw data +# The bundled catalogues ship inside the installed package. +data = pathlib.Path(technologydata.__file__).parent / "parsers" -The script reads the raw data available at `src/technologydata/parsers/raw/manual_input_usa.csv` in a `pandas` dataframe. It uses `pandas.read_csv(..., dtype=str, na_values="None")`. All entries are handled as strings initially except for the `value` column which is converted to float. +data_package = DataAccessor( + data_source="manual_input_usa", + version="v0.13.4", + data_path=data, +).load() -### Data cleaning, validation and dealing with missing/null values +technologies = data_package.technologies +print(len(technologies.technologies)) +``` -The data cleaning and validation happens with the following steps. +```text +85 +``` -Function `_extract_units_carriers_heating_value()` extracts standardized units, carriers, and heating values from input unit strings. This function maps complex unit representations to simplified unit, carrier, and heating value combinations using a predefined dictionary of special patterns. Examples include: +`load()` also writes `INFO: Data source directory corresponding to version v0.13.4 found.` to standard error. -- `USD_2022/MW_FT` → unit: `USD_2022/MW`, carrier: `1/FT`, heating_value: `1/LHV` -- `MWh_H2/MWh_FT` → unit: `MWh/MWh`, carrier: `H2/FT`, heating_value: `LHV` -- `MWh_el/MWh_FT` → unit: `MWh/MWh`, carrier: `el/FT`, heating_value: `LHV` -- `t_CO2/MWh_FT` → unit: `t/MWh`, carrier: `CO2/FT`, heating_value: `LHV` -- `USD_2022/kWh_H2` → unit: `USD_2022/kWh`, carrier: `1/H2`, heating_value: `LHV` -- `USD_2023/t_CO2/h` → unit: `USD_2023/t/h`, carrier: `1/CO2`, heating_value: `None` -- `MWh_el/t_CO2` → unit: `MWh/t`, carrier: `el/CO2`, heating_value: `LHV` -- `MWh_th/t_CO2` → unit: `MWh/t`, carrier: `thermal/CO2`, heating_value: `LHV` +Inspect the collection as a table: -The parser also fills missing values in the `scenario` column with `"not_available"`. +```python +df = technologies.to_dataframe() +print(df[["detailed_technology", "case", "year"]].head().to_string(index=False)) +``` -The parser applies the following unit conversions: +```text + detailed_technology case year +Alkaline electrolyzer large size Advanced - Market 2020 + PEM electrolyzer small size Advanced - Market 2020 + SOEC Advanced - Market 2020 + direct air capture Advanced - Market 2020 + battery inverter Advanced - Market 2022 +``` -- Convert `per unit` to `%` and multiply the corresponding `value` by 100.0, rounding to `num_digits` decimals. +Read one parameter of one technology: -Function `Commons.update_unit_with_currency_year(unit, currency_year)` appends `currency_year` information to currency units when present. This is because `technologydata` follows the currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`, as for example `USD_2022`. +```python +fischer_tropsch = next( + t + for t in technologies.technologies + if t.name == "Fischer-Tropsch" and t.year == 2020 +) +hydrogen_input = fischer_tropsch.parameters["hydrogen-input"] +print(hydrogen_input.magnitude, "|", hydrogen_input.units) +print(hydrogen_input.carrier, "|", hydrogen_input.heating_value) +``` -### Populate and export the source and technology collections +```text +1.43 | dimensionless +hydrogen / fischer_tropsch | lower_heating_value +``` -Function `_build_technology_collection()`: +## From raw data to parsed output + +Each raw row carries one parameter value together with the context that identifies it. The distinctive step for this dataset is the unit string: entries such as `MWh_H2/MWh_FT` encode a unit, an energy carrier and a heating value in one field, and the parser splits them into three. Rows are then grouped into `Technology` objects. + +```mermaid +flowchart LR + subgraph raw["manual_input_usa.csv row"] + direction TB + r1["technology
parameter"] + r2["value
unit
currency_year"] + r3["year
scenario
financial_case"] + end + subgraph parser["Parser"] + direction TB + s1["Fill missing scenario
convert per unit to %"] + s2["Split unit into unit,
carrier, heating value"] + s3["Group by scenario,
year, technology"] + s1 --> s2 --> s3 + end + subgraph out["Technology"] + direction TB + o1["name
detailed_technology"] + o2["parameters:
Parameter magnitude, units,
carrier, heating_value"] + o3["year
case
region"] + end + raw --> parser --> out +``` + +## A worked row + +The `Fischer-Tropsch` / `hydrogen-input` row for 2020 exercises every one of those steps: it has a compound unit, no `currency_year`, and no `scenario`. + +| Raw field | Raw value | Parsed field | Parsed value | +|---|---|---|---| +| `technology` | `Fischer-Tropsch` | `Technology.name` | `Fischer-Tropsch` | +| `technology` | `Fischer-Tropsch` | `Technology.detailed_technology` | `Fischer-Tropsch` | +| `year` | `2020` | `Technology.year` | `2020` | +| `scenario` | empty | `Technology.case` | `not_available` | +| — | — | `Technology.region` | `USA`, set by the parser | +| `parameter` | `hydrogen-input` | parameter key | `hydrogen-input` | +| `value` | `1.43` | `Parameter.magnitude` | `1.43` | +| `unit` | `MWh_H2/MWh_FT` | `Parameter.units` | `dimensionless` | +| `unit` | `MWh_H2/MWh_FT` | `Parameter.carrier` | `hydrogen / fischer_tropsch` | +| `unit` | `MWh_H2/MWh_FT` | `Parameter.heating_value` | `lower_heating_value` | +| `further_description` | `0.995 MWh_H2 per output, …` | `Parameter.note` | the same text | + +The unit is resolved in two stages, which is why the stored strings differ from the ones the parser looks up. `_extract_units_carriers_heating_value()` maps `MWh_H2/MWh_FT` to the triple `("MWh/MWh", "H2/FT", "LHV")`. Constructing the `Parameter` then resolves each of those through the package's pint registries: `MWh/MWh` cancels to `dimensionless`, while `H2` and `FT` expand to their canonical names via `src/technologydata/utils/carriers.txt`, and `LHV` via `src/technologydata/utils/heating_values.txt`. + +## Parser steps in detail -- if `archive_source` is set, constructs a `Source` object for the manual input USA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`. -- groups the cleaned DataFrame by `scenario`, `year`, `technology`. -- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `sources`, and optionally `carrier`, `heating_value`, `units`, `note`). -- captures the `financial_case` value from rows within each group to combine with `scenario`. -- creates a `case` value by combining `scenario` and `financial_case` in the format `"{scenario} - {financial_case}"` when `financial_case` is present; otherwise uses `scenario` alone. -- creates a `Technology` object for each group, with `name` = `technology`, `detailed_technology` = `technology`, `year` = `year`, `region` = `USA`, `case` = combined case value, and collects them into a `TechnologyCollection` object. -- writes the `TechnologyCollection` object to a `technologies.json`. +**Reading.** The CSV is read with `pandas.read_csv(..., dtype=str, na_values="None")`, so every entry starts as a string except `value`, which is cast to float. -## Running the parser +**Units, carriers and heating values.** `_extract_units_carriers_heating_value()` maps compound unit strings onto a `(unit, carrier, heating_value)` triple through a fixed dictionary, and returns the input unchanged with two `None`s when there is no match. The nine mapped patterns are: -### Execution instructions +| Input unit | Unit | Carrier | Heating value | +|---|---|---|---| +| `USD_2022/MW_FT` | `USD_2022/MW` | `1/FT` | `1/LHV` | +| `MWh_H2/MWh_FT` | `MWh/MWh` | `H2/FT` | `LHV` | +| `MWh_el/MWh_FT` | `MWh/MWh` | `el/FT` | `LHV` | +| `t_CO2/MWh_FT` | `t/MWh` | `CO2/FT` | `LHV` | +| `USD_2022/kWh_H2` | `USD_2022/kWh` | `1/H2` | `LHV` | +| `MWh_el/MWh_H2` | `MWh/MWh` | `el/H2` | `LHV` | +| `USD_2023/t_CO2/h` | `USD_2023/t/h` | `1/CO2` | none | +| `MWh_el/t_CO2` | `MWh/t` | `el/CO2` | `LHV` | +| `MWh_th/t_CO2` | `MWh/t` | `thermal/CO2` | `LHV` | -The parser is run using the `DataAccessor` class. You need to create an instance of `DataAccessor` with the desired `data_source` and `version`, and then call the `parse()` method. +These are the values the mapping returns, not the values that end up stored. As shown in the worked row above, `Parameter` resolves them through the pint registries in `src/technologydata/utils/`, so `H2/FT` is stored as `hydrogen / fischer_tropsch` and `LHV` as `lower_heating_value`. -Here is an example of how to run the parser from a Python script: +**Other cleaning.** Missing `scenario` entries are filled with `not_available`. Units containing `per unit` are rewritten to `%` and the corresponding `value` multiplied by 100. `Commons.update_unit_with_currency_year()` appends `currency_year` to currency units where present, producing the `USD_2022` form matched by the package's currency pattern `\b(?P[A-Z]{3})_(?P\d{4})\b`. + +**Building the collection.** `_build_technology_collection()` groups the cleaned frame by `scenario`, `year` and `technology`. Each group becomes one `Technology`, with both `name` and `detailed_technology` taken from `technology`, `region` fixed to `USA`, and a dictionary of `Parameter` objects carrying `magnitude`, `sources` and, where available, `carrier`, `heating_value`, `units` and `note`. The `case` combines the two scenario fields as `"{scenario} - {financial_case}"` when a `financial_case` is present in the group, and is the `scenario` alone otherwise. When `archive_source` is set it builds a `Source` for the dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise it reads an existing `sources.json`. + +## Regenerate the data + +The parsed files shipped with the package are produced by the same public entry point, `DataAccessor.parse()`. + +!!! warning "`parse()` writes relative to the working directory" + Unlike `load()`, the output path is derived from the current working directory, as `/src/technologydata/parsers/manual_input_usa/v0.13.4/`, and ignores `data_path`. Run this from the root of a checkout you are willing to modify: it overwrites the files distributed with the package. ```python from technologydata.parsers.data_accessor import DataAccessor -# Create an accessor for the version to be parsed parser_accessor = DataAccessor( data_source="manual_input_usa", - version="v0.13.4" + version="v0.13.4", ) -# Run the parser with desired options parser_accessor.parse( input_file_name="manual_input_usa.csv", num_digits=3, @@ -89,19 +172,14 @@ parser_accessor.parse( ) ``` -The `parse` method accepts the following arguments: - -- `input_file_name` (str): The name of the raw data file located in `src/technologydata/parsers/raw/`. -- `num_digits` (int, default 4): Number of decimals for rounding numeric values. -- `archive_source` (bool, default False): Whether to store the source on the Wayback Machine. -- `filter_params` (bool, default False): Whether to filter parameters (not used by this parser). -- `export_schema` (bool, default False): Whether to export Pydantic schemas. - -### Outputs +`parse()` accepts: -The parser generates the following outputs inside `src/technologydata/parsers/manual_input_usa/v0.13.4/`: +- `input_file_name` (str): name of the raw file in `src/technologydata/parsers/raw/`. +- `num_digits` (int, default 4): number of decimals for rounding numeric values. +- `archive_source` (bool, default False): whether to store the source on the Wayback Machine. +- `filter_params` (bool, default False): accepted for interface compatibility and unused by this parser. +- `export_schema` (bool, default False): whether to export the pydantic schemas. -- `technologies.json` -- `sources.json` +## Outputs -If `export_schema` is set to `True`, the Pydantic schema files are generated and moved to `src/technologydata/parsers/schemas/`. +The parser writes `technologies.json` and `sources.json` into `/src/technologydata/parsers/manual_input_usa/v0.13.4/`. With `export_schema=True`, the pydantic schemas are written alongside the JSON files as `technologies.schema.json` and, when `archive_source` is set, `sources.schema.json`.