Update scrapers new data model - #22
Draft
pierrepo wants to merge 13 commits into
Draft
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates multiple scrapers and related tooling/docs to align scraped dataset outputs with the newer Pydantic data model (notably authors: list[Person] and nested simulation: SimulationMetadata).
Changes:
- Update NOMAD, Figshare, and ATLAS scrapers to emit
authors(asPerson) and (where applicable) nestedsimulationmetadata. - Add
organismto theMoleculemodel and generate ATLAS dataset descriptions from simulation metadata. - Refresh docs/tooling: adjust the schema-drawing CLI option name, update README/docs, tweak
.gitignore, and bump a couple dependency minimums.
Reviewed changes
Copilot reviewed 9 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/mdverse/scrapers/nomad.py | Switches NOMAD dataset metadata to authors: list[Person] and nested simulation; adjusts helper return types. |
| src/mdverse/scrapers/network.py | Minor log-message punctuation change. |
| src/mdverse/scrapers/figshare.py | Emits authors: list[Person] instead of author_names; minor log-message punctuation changes. |
| src/mdverse/scrapers/atlas.py | Reworks ATLAS dataset metadata to new schema, adds generated description, adds static Person author metadata, nests simulation metadata. |
| src/mdverse/models/simulation.py | Adds organism field to Molecule. |
| src/mdverse/models/draw_pydantic_schema.py | Renames CLI option to --out-path and updates messaging. |
| README.md | Updates documentation for schema generation command; adds “Data model” section. |
| pyproject.toml | Bumps minimum versions for pyarrow and prek. |
| docs/atlas.md | Updates ATLAS docs to reflect generated descriptions. |
| .gitignore | Attempts to allow committing generated schema PNGs under data/models/. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ------- | ||
| list[Software] | None | ||
| list[Software] | ||
| A list of Software instances with `name` and `version` fields, None otherwise. |
Comment on lines
+415
to
+421
| # Add authors metadata. | ||
| authors = [ | ||
| Person(full_name=author.get("name")) | ||
| for author in dataset.get("authors", []) | ||
| ] | ||
| metadata["authors"] = authors | ||
| # Gather software metadata. |
Comment on lines
278
to
+281
| return [Software(name=name, version=version)] | ||
| except (ValueError, KeyError) as e: | ||
| logger.warning(f"Error parsing software info for entry {entry_id}: {e}") | ||
| return None | ||
| return [] |
Comment on lines
410
to
414
| "title": dataset.get("entry_name"), | ||
| "date_created": dataset.get("entry_create_time"), | ||
| "date_last_updated": dataset.get("last_processing_time"), | ||
| "number_of_files": len(dataset.get("files", [])), | ||
| "author_names": [a.get("name") for a in dataset.get("authors", [])], | ||
| "license": dataset.get("license"), |
Comment on lines
+145
to
+152
| def generate_description( | ||
| metadata: SimulationMetadata, logger: "loguru.Logger" = loguru.logger | ||
| ) -> str: | ||
| """Generate description for ATLAS dataset based on metadata. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| metadata : SimulationMetadata |
Comment on lines
+73
to
+77
| if headers is not None: | ||
| default_headers.update(headers) | ||
| headers = default_headers | ||
| else: | ||
| headers = default_headers |
Comment on lines
+418
to
+421
| authors = [ | ||
| Person(full_name=author.get("name")) | ||
| for author in dataset.get("authors", []) | ||
| ] |
Comment on lines
+248
to
+252
| "authors": [ | ||
| Person( | ||
| full_name=clean_text(author.get("full_name")), | ||
| first_name=clean_text(author.get("first_name")), | ||
| last_name=clean_text(author.get("last_name")), |
Comment on lines
224
to
+230
| logger.info("Trying connection to Zenodo...") | ||
| response = make_http_get_request_with_retries( | ||
| url="https://zenodo.org/api/deposit/depositions", | ||
| params={"access_token": token}, | ||
| headers={ | ||
| "Content-Type": "application/json", | ||
| "Authorization": f"Bearer {token}", | ||
| }, |
Comment on lines
+163
to
+166
| for molecule in metadata.molecules: | ||
| description += f"{molecule.name} " | ||
| description += f"from organism {molecule.organism} " | ||
| external_identifiers = [ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.