Overhaul database architecture and management by sheikhPHD · Pull Request #17 · MDverse/mdverse_data_catalogue

sheikhPHD · 2026-05-04T14:36:01Z

No description provided.

… a pure Python runner

pierrepo · 2026-05-05T18:01:00Z

+    dataset_id          INTEGER NOT NULL
+                            REFERENCES datasets (dataset_id)
+                            ON DELETE CASCADE,
+    name                TEXT    NOT NULL,


Could you please remove "ON DELETE CASCADE" since it's not available with duckdb?

I have removed "ON DELETE CASCADE" in the new version.

pierrepo · 2026-05-06T14:27:19Z

+    tables = {}
+    for block in re.finditer(r"CREATE TABLE\s+(?:IF NOT EXISTS\s+)?(\w+)\s*\((.*?)\);", sql, re.S | re.I):
+        table, body = block.group(1), block.group(2)
+        tables[table] = [
+            line.strip().rstrip(",").split()[0]
+            for line in body.splitlines()
+            if line.strip() and not re.match(r"(PRIMARY|FOREIGN|UNIQUE|CHECK|CONSTRAINT)", line.strip(), re.I)
+        ]
+
+    # SHOW TABLES
+    print("\nSHOW TABLES")
+    print("=" * 32)
+    for name in sorted(tables):
+        print(f"  {name}")
+    print("-" * 32)
+    print(f"  {len(tables)} table(s)\n")
+
+    # DESCRIBE <table>
+    for table, columns in sorted(tables.items()):
+        print(f"DESCRIBE {table}")
+        print("=" * 40)
+        for i, col in enumerate(columns, 1):
+            print(f"  {i:<4} {col}")
+        print(f"  {len(columns)} column(s)\n")


As we discussed, could you please remove these lines?

We don't want to parse the SQL file.

Done, removed the SQL parsing block. The script now just creates the database.

pierrepo · 2026-05-18T11:42:41Z

+    file_type_id  INTEGER PRIMARY KEY DEFAULT nextval('file_types_seq'),
+    name          TEXT    NOT NULL UNIQUE,
+    comment       TEXT
+);


Nice. Could you please ensure data types are regular duckdb data types as listed here: https://duckdb.org/docs/current/sql/data_types/overview ?
For instance, TEXT should be VARCHAR
And maybe, all dates should be of TIMESTAMP type (to be checked)

Done! Replaced all TEXT with VARCHAR, date columns changed to TIMESTAMP, and also updated boolean INTEGER columns to BOOLEAN.

- Replace all TEXT with VARCHAR - Change date columns to TIMESTAMP - Change boolean-like INTEGER columns to BOOLEAN - Add report_duckdb.py to display row and column counts per table

pierrepo · 2026-05-18T15:02:38Z

Thanks @sheikhPHD
To have a rough idea, how long does it take to ingest datasets and files from Zenodo into the database?

sheikhPHD · 2026-05-19T08:24:11Z

Thanks @sheikhPHD To have a rough idea, how long does it take to ingest datasets and files from Zenodo into the database?

Here is the estimate for ZENODO:

source type time
zenodo files 0:00:03
zenodo datasets 0:00:00

Requires DuckDB >= 1.5.3. Earlier versions had a memory corruption bug triggered when deleting large sources via the Python library after an input() call.

pierrepo · 2026-05-22T06:52:29Z

+_S_DS = "SELECT dataset_id FROM datasets WHERE data_source_id = (SELECT data_source_id FROM data_sources WHERE name = $1)"
+_S_FI = f"SELECT file_id       FROM files       WHERE dataset_id    IN ({_S_DS})"
+_S_AN = f"SELECT annotation_id FROM annotations WHERE dataset_id    IN ({_S_DS})"
+_S_MO = f"SELECT molecule_id   FROM molecules   WHERE annotation_id IN ({_S_AN})"


Please use meaningful variable names

Done — eliminated entirely. Instead of renamed subquery variables, IDs are now fetched into Python lists via collect_ids_for_dataset() and collect_ids_for_source(), then accessed as ids["dataset_ids"], ids["file_ids"], etc.

pierrepo · 2026-05-22T06:52:56Z

+
+# ── Core deletion logic ────────────────────────────────────────────────────────
+
+def _count(conn: duckdb.DuckDBPyConnection, param: list, ds: str, fi: str, an: str, mo: str) -> dict[str, int]:


Please use meaningful variable names. ds, fi, an... are not very expressive.

Done — eliminated entirely. Instead of renamed subquery variables, IDs are now fetched into Python lists via collect_ids_for_dataset() and collect_ids_for_source(), then accessed as ids["dataset_ids"], ids["file_ids"], etc.

pierrepo · 2026-05-22T06:54:14Z

+    def run(label: str, count_sql: str, delete_sql: str) -> None:
+        counts[label] = conn.execute(count_sql, param).fetchone()[0]
+        conn.execute(delete_sql, param)
+


Could you move this function outside the _delete function and rename it with a more meaningful name?

Done — moved outside and split into count_rows_by_ids() and delete_rows_by_ids().

pierrepo · 2026-05-22T06:55:48Z

+    """Remove a single dataset and all its related records."""
+    print(f"INFO  | Mode: DELETE DATASET  |  datarepo='{source_name}'  dataset='{id_in_source}'")
+    if dry_run:
+        print("WARN  | DRY-RUN — no changes will be written.")


Please use proper loger with loguru

Done — replaced all print() calls with loguru logger.info/warning/error/success.

pierrepo · 2026-05-22T06:57:45Z

+Note on transactions:
+    DuckDB v1.x enforces FK constraints per-statement even inside BEGIN/COMMIT,
+    making transactional multi-table cascades impossible with FK constraints.
+    The standard workaround is autocommit with strict child-first deletion order,
+    which guarantees no orphaned rows at any point. Each DELETE is individually
+    atomic; a mid-run failure leaves the database consistent and retryable.


Unclear to me. Could you explain a bit more?

DuckDB enforces foreign key constraints immediately after each DELETE statement, even inside a BEGIN/COMMIT block. This means we cannot wrap all deletions in a single transaction — deleting a parent row (e.g. datasets) while child rows still exist in another table (e.g. files) raises a FK violation, even if those child rows would be deleted later in the same transaction.
The workaround is autocommit with strict child-first ordering: we always delete the deepest child table first and work up to the parent. Each DELETE commits immediately, so by the time we delete a parent row, all its children are already gone. I verified this limitation is still present in DuckDB v1.5.3 with explicit tests.

pierrepo · 2026-05-22T07:01:38Z

+
+# dataset mode subqueries
+_D_DS = "SELECT $1::INTEGER"


Instead of using Duckdb subqueries, could you:

List all file_ids involved into the deletion:

if a file_id is provided: the file itself and all its related files if its a zip file

if a repo name is provided: all files from the repo and all files from zip files in this repo

then run the deletion

Done — refactored to fetch all IDs (dataset_ids, file_ids, annotation_ids, molecule_ids) into Python lists first using collect_ids_for_dataset() and collect_ids_for_source(), then delete using those lists.

…ipt)

…sions

…fetch IDs into Python, move run() helper outside _delete

…ipts

…pt names

…on handling

… connection handling

- for SQLite3 deletion - for data downloading - for parameter/topology/trajectory ingestion

…arXiv APIs

… process

+    """CLI entry point for data deletion."""
+    logger = create_logger(Path("logs/delete_data.log"))
+    start = time.perf_counter()
+    run_deletion(db_path, source_name, id_in_source, dry_run)


…nections

… the DB

… authors, and AI models

…reign key constraints

sheikhPHD added 2 commits May 4, 2026 16:35

feat: Update README

85eb853

feat: database creation script in SQL

9e4067f

pierrepo changed the title ~~feat: Update README~~ feat: Use pure SQL to handle database May 5, 2026

Move SQL instructions to database_schema.sql, make create_database.py…

c6df0d4

… a pure Python runner

pierrepo reviewed May 5, 2026

View reviewed changes

chore: Make schema and database creation DuckDB-compatible

6acf313

pierrepo reviewed May 6, 2026

View reviewed changes

sheikhPHD added 3 commits May 6, 2026 16:47

chore: remove SQL parsing, use native DB inspection

9a98e1d

feat:Add SQLite ingestion pipeline for MDverse database

584ce70

feat: Add DuckDB database implementation (schema, create, ingest)

8bae78f

pierrepo reviewed May 18, 2026

View reviewed changes

feat: update schema to canonical DuckDB types and add report script

cc96c23

- Replace all TEXT with VARCHAR - Change date columns to TIMESTAMP - Change boolean-like INTEGER columns to BOOLEAN - Add report_duckdb.py to display row and column counts per table

sheikhPHD added 2 commits May 20, 2026 14:53

feat: add DuckDB deletion script for datasets and data sources

a085fa8

feat: add DuckDB deletion script for datasets and data sources

df10bf9

Requires DuckDB >= 1.5.3. Earlier versions had a memory corruption bug triggered when deleting large sources via the Python library after an input() call.

pierrepo reviewed May 22, 2026

View reviewed changes

sheikhPHD added 8 commits May 22, 2026 11:36

chore: add duckdb>=1.5.3 dependency (fixes deletion bug in delete scr…

002d918

…ipt)

refactor: rename duckdb scripts to canonical names, remove sqlite ver…

9f6a7c7

…sions

refactor: move database_schema.sql to params/ folder

8add97c

fix: update create_database.py docstring and default schema path

14b54f9

refactor: improve delete_data.py — loguru logging, meaningful names, …

d1bbf43

…fetch IDs into Python, move run() helper outside _delete

docs: update README build database section for DuckDB and renamed scr…

68c7620

…ipts

fix: update ingest_data.py docstring and epilog to use canonical scri…

d84fca6

…pt names

fix: update report.py docstring to use canonical script name

7e889ab

Essmaw added 12 commits July 30, 2026 01:22

refactor: Simplify data ingestion logic and enhance database connecti…

e13873d

…on handling

feat: Enhance report generation with ERD diagram and improve database…

b5ab048

… connection handling

feat: Add database statistics report groupby dataset origin.

6ec59e0

feat: Add URL and comment properties for external database names

2dc2196

feat: Add SQL vocabulary setup in database creation

7d65291

chore: Remove obsolete database handling scripts

96523a8

- for SQLite3 deletion - for data downloading - for parameter/topology/trajectory ingestion

fix: Rename db-path option to db in data sources report command

62dc519

refactor: Formate SQL insert statements in database vocabularies

edc979f

feat: Implement fetch functions for paper metadata from Crossref and …

4b2d269

…arXiv APIs

feat: Implement file ingestion SQL queries and enhance data ingestion…

3d5d729

… process

refactor: Simplify data deletion logic and enhance CLI

004edf9

docs: Update README+database schema; add database-delete command

5909b0e

github-code-quality Bot found potential problems Jul 30, 2026

View reviewed changes

Comment thread src/mdverse/database/delete_data.py

"""CLI entry point for data deletion."""

logger = create_logger(Path("logs/delete_data.log"))

start = time.perf_counter()

run_deletion(db_path, source_name, id_in_source, dry_run)

Essmaw marked this pull request as draft July 31, 2026 00:03

Essmaw changed the title ~~feat: Use pure SQL to handle database~~ Overhaul database architecture and management Aug 1, 2026

Essmaw and others added 2 commits August 1, 2026 13:19

refactor(test): Update tests to use DuckDB for in-memory database con…

5f1c880

…nections

Merge branch 'main' into feature/move-to-pure-sql

4409b33

Essmaw linked an issue Aug 2, 2026 that may be closed by this pull request

Support MD AI models in dataset schema and scrapers #25

Closed

Essmaw added 5 commits August 7, 2026 02:16

chore: Clean up SQL scripts for better readability

f9ae531

refactor: Split the enrichissment of the data from the ingestion into…

82a3f60

… the DB

feat: Add SQL script for ingesting publications and linking datasets,…

f03142e

… authors, and AI models

feat: Add Pydantic models for AI model metadata

c1501db

feat: Add Hugging Face as a dataset and publication source

be14e3f

Essmaw self-assigned this Aug 19, 2026

Essmaw added the database label Aug 19, 2026

Essmaw added 5 commits August 21, 2026 14:01

refactor(report): Update table and relationship definitions

1c51c07

fix: Update database report command to use --db flag

5fbd94c

fix: Update database ingest command to use --db flag

ada7ff6

refactor(database_vocabularies): remove unused annotation entities

1cead01

refactor(annotations): rename value to annotation_value and adjust fo…

e840ba2

…reign key constraints


		# ── Core deletion logic ────────────────────────────────────────────────────────

		def _count(conn: duckdb.DuckDBPyConnection, param: list, ds: str, fi: str, an: str, mo: str) -> dict[str, int]:

Uh oh!

Conversation

sheikhPHD commented May 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrepo commented May 18, 2026

Uh oh!

sheikhPHD commented May 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants