refactor!: explicit backend api cleanup#6
Open
gkennos wants to merge 4 commits into
Open
Conversation
gkennos
changed the title
There was a problem hiding this comment.
Pull request overview
This PR refactors orm-loader to make database-specific behavior explicit via a backend abstraction, while updating the PostgreSQL path to use psycopg (v3) and expanding tests/docs accordingly.
Changes:
- Introduces a
DatabaseBackendcontract with concrete SQLite/PostgreSQL implementations plusresolve_backend()dispatch. - Routes staging-table creation, fast-path loading, merge behavior, FK toggling, and materialized view operations through the resolved backend.
- Updates packaging to add a
postgresextra (psycopg[binary]), bumps version to0.4.0, and adds/updates backend-focused test coverage and docs.
Reviewed changes
Copilot reviewed 34 out of 36 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Bumps package version and adds locked dependencies for new postgres extra (psycopg). |
| TODO.txt | Adds future-facing notes about optional malformed text repair. |
| tests/models.py | Adds an index to support index-management behavior in merge tests. |
| tests/conftest.py | Switches PostgreSQL test DSN to SQLAlchemy psycopg driver. |
| tests/backends/test_sqlite_backend.py | Adds unit tests for SQLite backend staging, FK toggling, merge SQL, and pragmas hooks. |
| tests/backends/test_postgres_backend.py | Adds unit tests for Postgres backend staging, FK toggling, merge SQL, and materialized view SQL emission. |
| tests/backends/test_base_backend.py | Adds contract tests for backend base class, capabilities, backend resolution, and import behavior without psycopg installed. |
| src/orm_loader/tables/typing.py | Tightens typing (generics, TypedDict/Unpack) and adds index_strategy/updated merge signatures to the protocol. |
| src/orm_loader/tables/serialisable_table.py | Improves type annotations for JSON/dict serialization helpers. |
| src/orm_loader/tables/orm_table.py | Tightens ORMTableBase typing and reduces cast() usage. |
| src/orm_loader/tables/loadable_table.py | Refactors staging/merge/index-management to use backend resolution and introduces index_strategy. |
| src/orm_loader/mappers/materialised_view_mixin.py | Routes MV create/refresh through backend implementations and improves typing. |
| src/orm_loader/loaders/loading_helpers.py | Updates PostgreSQL COPY helper to a psycopg3-style copy loop and introduces COPY_BLOCK_SIZE. |
| src/orm_loader/loaders/data_classes.py | Removes stale commented-out dedupe code and trailing whitespace. |
| src/orm_loader/helpers/sqlite.py | Replaces direct event-hook/pragmas logic with backend-delegated helpers for SQLite. |
| src/orm_loader/helpers/logging.py | Simplifies log-level coercion and adds typing for formatter methods. |
| src/orm_loader/helpers/discovery.py | Updates typing and changes defaulting behavior for the base parameter. |
| src/orm_loader/helpers/bulk.py | Delegates FK toggling and bulk-load context behavior to the resolved backend; delegates replica-role engine context. |
| src/orm_loader/helpers/bootstrap.py | Adds type annotations for bootstrap/schema creation helpers. |
| src/orm_loader/helpers/init.py | Re-exports new/renamed SQLite helpers. |
| src/orm_loader/backends/sqlite.py | Adds SQLite backend implementation (staging, merge SQL, FK toggling, connect hooks, FK error explanation, journal restore). |
| src/orm_loader/backends/resolve.py | Adds backend resolver selecting a concrete backend based on SQLAlchemy dialect. |
| src/orm_loader/backends/postgres.py | Adds Postgres backend implementation (unlogged staging, COPY fast-path, merge SQL, MV operations, replica-role context). |
| src/orm_loader/backends/base.py | Adds backend base contract, capabilities, shared helpers, and a generic bulk-load context manager. |
| src/orm_loader/backends/init.py | Exposes backend public API and resolver. |
| README.md | Updates project description and documents backend support and staged ingestion approach. |
| pyproject.toml | Bumps version, adds postgres optional dependency, and expands metadata/classifiers and tooling config. |
| docs/tables/mat_view.md | Updates MV docs to reflect backend resolution and PostgreSQL-only support. |
| docs/tables/loadable_table.md | Updates staged ingestion docs to reflect backend-specific behavior and index handling. |
| docs/loaders/loaders.md | Clarifies loader responsibilities vs table/backend merge behavior and updates dedupe description. |
| docs/loaders/index.md | Updates loader overview to emphasize staged loading and table/backend ownership of merge semantics. |
| docs/loaders/helpers.md | Updates helper docs, including PostgreSQL COPY behavior and fallback behavior. |
| docs/loaders/context.md | Documents quote_mode in loader context. |
| docs/index.md | Updates project overview and explicitly documents current backend support limits. |
| CHANGELOG.md | Adds 0.4.0 notes about psycopg and backend API cleanup. |
| .gitignore | Adds _temp/ ignore entry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+191
to
+200
| @contextmanager | ||
| def engine_with_replica_role(self, engine: "Engine"): | ||
| @sa.event.listens_for(engine, "connect") # type: ignore[arg-type] | ||
| def _set_replica_role( | ||
| dbapi_conn: sa.engine.interfaces.DBAPIConnection, | ||
| _, | ||
| ) -> None: | ||
| cur = dbapi_conn.cursor() | ||
| cur.execute("SET session_replication_role = replica") | ||
| cur.close() |
| if dbapi_connection.__class__.__module__.startswith("sqlite3"): | ||
| cursor = dbapi_connection.cursor() | ||
| cursor.execute(f"PRAGMA busy_timeout = {self.busy_timeout_ms}") | ||
| cursor.execute(f"PRAGMA journal_mode = {self.journal_mode}") |
Comment on lines
+196
to
+200
| def configure_dbapi_connection(self, dbapi_connection: sa.engine.interfaces.DBAPIConnection) -> None: | ||
| if dbapi_connection.__class__.__module__.startswith("sqlite3"): | ||
| cursor = dbapi_connection.cursor() | ||
| cursor.execute(f"PRAGMA busy_timeout = {self.busy_timeout_ms}") | ||
| cursor.execute(f"PRAGMA journal_mode = {self.journal_mode}") |
| dev = [ | ||
| "pytest>=9.0.3", | ||
| "mypy>=1.19.1", | ||
| "pytest>=9.0.3", |
Comment on lines
+192
to
+200
| def engine_with_replica_role(self, engine: "Engine"): | ||
| @sa.event.listens_for(engine, "connect") # type: ignore[arg-type] | ||
| def _set_replica_role( | ||
| dbapi_conn: sa.engine.interfaces.DBAPIConnection, | ||
| _, | ||
| ) -> None: | ||
| cur = dbapi_conn.cursor() | ||
| cur.execute("SET session_replication_role = replica") | ||
| cur.close() |
Comment on lines
+8
to
+9
| base: type[ModelT] = Base, | ||
| ) -> type[ModelT] | None: |
Comment on lines
+14
to
+20
| Apply the default SQLite connection settings used by orm-loader. | ||
|
|
||
| This helper is kept for compatibility with older event-hook setups. | ||
| It delegates to ``SQLiteBackend.configure_dbapi_connection()``, | ||
| which may apply more than just foreign-key settings. | ||
| """ | ||
| del connection_record |
| bind: Engine | Connection, | ||
| ) -> Iterator[Connection]: | ||
| if isinstance(bind, Engine): | ||
| with bind.connect() as conn: |
Comment on lines
+66
to
+69
| mapper: so.Mapper[Any] = sa.inspect(cls) | ||
| if not mapper: | ||
| raise TypeError(f"{cls.__name__} is not a mapped ORM class") | ||
| return cast(so.Mapper, mapper) | ||
| return mapper |
Comment on lines
+106
to
+111
| def restore_fk_check( | ||
| self, | ||
| session: so.Session, | ||
| previous_state: str | int, | ||
| ) -> None: | ||
| session.execute(text(f"PRAGMA foreign_keys = {previous_state}")) |
Comment on lines
+95
to
+96
| session.execute(sa.text(f"SET session_replication_role = '{previous_state}'")) | ||
|
|
| self, | ||
| session: so.Session, | ||
| previous_state: str | int, | ||
| ) -> None: |
Comment on lines
+90
to
+96
| def restore_fk_check( | ||
| self, | ||
| session: so.Session, | ||
| previous_state: str | int, | ||
| ) -> None: | ||
| session.execute(sa.text(f"SET session_replication_role = '{previous_state}'")) | ||
|
|
Comment on lines
+193
to
+214
| def _set_replica_role( | ||
| dbapi_conn: sa.engine.interfaces.DBAPIConnection, | ||
| _, | ||
| ) -> None: | ||
| cur = dbapi_conn.cursor() | ||
| cur.execute("SET session_replication_role = 'replica'") | ||
| cur.close() | ||
|
|
||
| sa.event.listen(engine, "connect", _set_replica_role) | ||
|
|
||
| try: | ||
| yield engine | ||
| finally: | ||
| sa.event.remove(engine, "connect", _set_replica_role) | ||
| with engine.connect() as conn: | ||
| conn = conn.execution_options(isolation_level="AUTOCOMMIT") | ||
| conn.execute(sa.text("SET session_replication_role = DEFAULT")) | ||
| role = conn.execute( | ||
| sa.text("SHOW session_replication_role") | ||
| ).scalar() | ||
| if role != "origin": | ||
| raise RuntimeError("Failed to restore session_replication_role") |
| @@ -19,19 +19,15 @@ def _coerce_log_level(level: int | str) -> int: | |||
| if isinstance(level, int): | |||
| return level | |||
|
|
|||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BREAKING CHANGE: known breaking change - removal of the dialect argument from _merge_replace() and _merge_upsert().
Opinionated backend now means that in the unlikely event that someone was using this on some other dialect, it will no longer work, but normal usage on SQLite and PostgreSQL is intended to remain functionally the same.