Code review

.gitignore

@@ -211,10 +211,8 @@ __marimo__/
 *.tsv
 *.parquet
 *.csv
-*.png
-*.json
-*.owl
-*.html
 
 # Results folder
-!results
+results/*
+!results/FFM
+!results/SOFTNAME

README.md

@@ -1,5 +1,8 @@
 # mdverse_entity_norm
 
+This project implements the normalization pipeline for molecular dynamics (MD) simulation metadata entities. Normalization standardizes entity mentions by mapping them to controlled vocabularies or reference databases (e.g., [ChEBI](https://www.ebi.ac.uk/chebi/), [PubChem](https://pubchem.ncbi.nlm.nih.gov/), [KEGG](https://www.genome.jp/kegg/)), ensuring consistency and interoperability across datasets.
+Normalisation is currently supported for four entity types: molecule names (MOL),simulation times (STIME) and temperatures (STEMP), software names (SOFTNAME) and force fields and models (FFM).
+
 ## Setup environment
 
 We use [uv](https://docs.astral.sh/uv/getting-started/installation/) to manage dependencies and the project environment.
@@ -19,92 +22,107 @@ uv sync
 
 ## Usage
 
-This project implements the normalisation pipeline for molecular dynamics simulation metadata entities. Normalisation is currently supported for four entity types: temperature, small molecules, simulation times, and software versions (not yet implemented). The scripts are located in `src/mdverse_entity_norm/scripts/` and can be executed independently. Output files are saved in the `results/` directory, which is created automatically if it does not exist.
+The following scripts require an `entities.tsv` file as input. The file should contain the columns `entity`, `category`, and `json_file`.
+
+### Simulation temperature (STEMP)
 
-### Normalize temperature
+To normalize simulation temperatures, run:
 
 ```sh
-uv run src/mdverse_entity_norm/scripts/normalize_temperature.py
+uv run src/mdverse_entity_norm/scripts/normalize_stemp.py --raw-entities-path data/entities.tsv --normalized-stemp-path results/STEMP/stemp_normalized.tsv
 ```
 
-This reads temperature entities from `data/entities.tsv` and writes `results/norm_temp.tsv`, a TSV file with four columns:
+This reads temperature entities from `data/entities.tsv` and writes `results/STEMP/stemp_normalized.tsv`, a TSV file with four columns:
 
-| Column | Description |
-|---|---|
-| `raw_temperature` | Original temperature string |
-| `normalised_temperature` | Numeric value after normalisation |
-| `normalised_unit` | Unit after normalisation (Kelvin) |
-| `normalized_result` | Concatenated value and unit |
+| raw_temperature | normalised_temperature | normalised_unit | normalized_result |
+| --------------- | ---------------------- | --------------- | ----------------- |
+| 315             | 315                    | K               | 315 K             |
+| 20°C           | 293,15                 | K               | 293,15 K          |
+| 310k            | 310                    | K               | 310 K             |
 
-Special cases `room temperature` and `human body temperature` are normalised to 293 K and 310 K respectively. All Celsius values are converted to Kelvin.
+> Special cases `room temperature` and `human body temperature` are normalised to 293 K and 310 K respectively. All Celsius values are converted to Kelvin.
 
-### Ground molecules
+### Simulation times (STIME)
 
-The grounding logic is illustrated below:
+The normalization of simulation times is a two-step process: first, we benchmark several candidate Large Language Models (LLMs) against a gold standard dataset to select the best performer; second, we deploy the chosen model to normalize the entire dataset.
 
-![Grounding logic](molecules_grounding_logic.png)
+> 🔑 An `OPEN_ROUTER_KEY` environment variable must be set (e.g., via a .env file) to authenticate and authorise API requests to the external LLM providers hosted on OpenRouter.
+
+#### Model evaluation:
+
+To evaluate candidate LLM models on a labelled gold standard, run:
 
 ```sh
-uv run src/mdverse_entity_norm/scripts/normalize_molecules.py
+uv run src/mdverse_entity_norm/scripts/evaluate_llm_models.py \
+  --groundtruth-path data/groundtruth/STIME.json \
+  --prompt-path data/llm_prompt.txt \
+  --runs 10 \
+  --model-evaluation-path results/STIME/model_evaluation.tsv
 ```
 
-This reads molecular entities from `data/entities.tsv`. Entities are first classified by type (PDB, UniProt, DNA, RNA, protein, or small molecule). PDB and UniProt entries are resolved via their respective APIs and saved to `results/ground_molecule/same_grounding_mol/pdb_uniprot_seq_entities.tsv`. Small molecules are grounded by consensus across ChEBI, PubChem, and KEGG, producing two output files:
-
-**`chebi_comparaison.tsv`** — ChEBI grounding results for all small molecules:
+This script benchmarks 9 models accessible via OpenRouter (including `GPT-4o`, `DeepSeek V4 Pro`, and `Claude 4.7 Opus`) against a manually annotated gold standard of 100 simulation time entities. The evaluation is repeated over the specified number of runs to ensure statistical robustness.
 
-| Column | Description |
-|---|---|
-| `Molecule` | Original molecule name |
-| `CHEBI_ID` | ID returned directly by ChEBI |
-| `CHEBI_ID_from_KEGG` | ChEBI ID resolved via KEGG |
-| `CHEBI_ID_from_PubChem` | ChEBI ID resolved via PubChem synonyms |
-| `Match` | `True` if at least two sources agree |
+The evaluation results across the tested models are detailed below:
 
-**`pubchem_comparaison_no_chebi_match.tsv`** — PubChem fallback for molecules with no ChEBI consensus:
+| model_name                         | accuracy_percentage (%) | normalisation_times_sec (s) | normalisation_cost (USD/entity) |
+| ---------------------------------- | ----------------------: | --------------------------: | ------------------------------: |
+| openai/gpt-5.5                     |                      99 |                        1.65 |                          0.0028 |
+| qwen/qwen3.6-27b                   |                      99 |                       18.80 |                          0.0012 |
+| minimax/minimax-m2.7               |                      99 |                       10.72 |                          0.0096 |
+| anthropic/claude-opus-4.7          |                      98 |                        2.85 |                          0.0013 |
+| **deepseek/deepseek-v4-pro** |            **97** |              **8.39** |                **0.0022** |
+| openai/gpt-4o                      |                      95 |                        1.26 |                          0.0016 |
+| mistralai/mistral-large-2512       |                      90 |                        4.19 |                          0.0001 |
+| moonshotai/kimi-k2.6               |                      89 |                       28.17 |                          0.0002 |
+| google/gemma-4-31b-it              |                      62 |                        2.44 |                          0.0002 |
 
-| Column | Description |
-|---|---|
-| `Molecule` | Original molecule name |
-| `PubChem_ID` | ID returned directly by PubChem |
-| `PubChem_ID_from_KEGG` | PubChem ID resolved via KEGG |
-| `Match` | `True` if both sources agree |
 
-### Normalize simulation times
+#### Entity normalization:
 
-Two scripts are involved: one evaluates candidate LLM models on a labelled gold standard, the other applies the selected model to the full dataset.
+Based on these results, **DeepSeek V4 Pro** was selected as the optimal open-weight model, offering the best balance between high accuracy (97%), reasonable latency, and cost efficiency.
 
-**Model evaluation:**
+To apply this model and normalize the entire dataset, run:
 
 ```sh
-uv run src/mdverse_entity_norm/scripts/normalize_simulation_time.py \
-  --ground_truth_file data/STIME_ground_truth.json \
-  --runs 10 \
-  --model_evaluation_file results/norm_simu_times/model_evaluation.tsv
+uv run src/mdverse_entity_norm/scripts/normalize_stime_wth_llm.py --entities-path data/entities.tsv --normalization-results-path results/STIME/stime_normalized.tsv --prompt-path data/llm_prompt.txt --model-name "deepseek/deepseek-v4-pro"
 ```
 
-This benchmarks 9 models via OpenRouter (including GPT-4o, DeepSeek V4 Pro, Claude Opus 4.7, and others) on a manually annotated gold standard of 100 simulation time entities, repeated over the specified number of runs. Results are saved to the file specified by `--model_evaluation_file`:
+This processes all raw STIME entities and outputs a three-column TSV with the standardized values and units:
+
+| STIME         | value | unit |
+| ------------- | ----: | :--: |
+| 1 μs         |   1.0 | μs |
+| 1 microsecond |   1.0 | μs |
+| 200-300ns     | 200.0 |  ns  |
+| 200-300ns     | 300.0 |  ns  |
 
-| Column | Description |
-|---|---|
-| `model_name` | Model identifier |
-| `accuracy_percentage` | Average accuracy across runs (%) |
-| `normalisation_times_sec` | Average processing time per entity (s) |
-| `normalisation_cost` | Average cost per entity (USD) |
+### Ground molecule names
 
-> An `OPEN_ROUTER_KEY` environment variable must be set (e.g. via a `.env` file) for API access.
+The grounding logic is illustrated below:
 
-**Entity normalisation:**
+![Grounding logic](plots/molecules_grounding_logic.png)
 
 ```sh
-uv run src/mdverse_entity_norm/scripts/normalize_stime_results.py \
-  --entities-file data/entities.tsv \
-  --output-file results/norm_simu_times/normalized_stime_results.tsv
+uv run src/mdverse_entity_norm/scripts/normalize_molecules.py
 ```
 
-This applies DeepSeek V4 Pro to all STIME entities in the input file and writes a TSV with three columns:
+This reads molecular entities from `data/entities.tsv`. Entities are first classified by type (PDB, UniProt, DNA, RNA, protein, or small molecule). PDB and UniProt entries are resolved via their respective APIs and saved to `results/ground_molecule/same_grounding_mol/pdb_uniprot_seq_entities.tsv`. Small molecules are grounded by consensus across ChEBI, PubChem, and KEGG, producing two output files:
+
+**`chebi_comparaison.tsv`** — ChEBI grounding results for all small molecules:
+
+| Column                    | Description                            |
+| ------------------------- | -------------------------------------- |
+| `Molecule`              | Original molecule name                 |
+| `CHEBI_ID`              | ID returned directly by ChEBI          |
+| `CHEBI_ID_from_KEGG`    | ChEBI ID resolved via KEGG             |
+| `CHEBI_ID_from_PubChem` | ChEBI ID resolved via PubChem synonyms |
+| `Match`                 | `True` if at least two sources agree |
+
+**`pubchem_comparaison_no_chebi_match.tsv`** — PubChem fallback for molecules with no ChEBI consensus:
 
-| Column | Description |
-|---|---|
-| `STIME` | Original simulation time string |
-| `LLM_value` | Normalised numeric value |
-| `LLM_unit` | Normalised unit (`ps`, `ns`, `μs`, `ms`, or `s`) |
+| Column                   | Description                     |
+| ------------------------ | ------------------------------- |
+| `Molecule`             | Original molecule name          |
+| `PubChem_ID`           | ID returned directly by PubChem |
+| `PubChem_ID_from_KEGG` | PubChem ID resolved via KEGG    |
+| `Match`                | `True` if both sources agree  |