diff --git a/docs/memory/log/2026-08-07-ruff-defaults.md b/docs/memory/log/2026-08-07-ruff-defaults.md new file mode 100644 index 00000000..25d41688 --- /dev/null +++ b/docs/memory/log/2026-08-07-ruff-defaults.md @@ -0,0 +1,26 @@ +--- +title: Adopt ruff's default rule set via ruff.toml +summary: ruff 0.16 expanded its implicit defaults from 59 rules to 413; ruff.toml now records the rule set explicitly +created: 2026-08-07 +author: Eric Case +tags: [lint, ruff, ci, conventions] +--- + +# Adopt ruff's default rule set via ruff.toml (2026-08-07) + +The repo had no `[tool.ruff]` section anywhere, so lint policy was whatever ruff's implicit defaults happened to be. Ruff [0.16.0](https://astral.sh/blog/ruff-v0.16.0) expanded those defaults from 59 rules (`E4`, `E7`, `E9`, `F`) to 413, and the Dependabot bump surfaced 72 violations. + +The decision is to **adopt the new defaults** rather than pin back to the old set. `ruff.toml` now exists to make the rule set an explicit, reviewable choice. + +Deviations recorded in `ruff.toml`: + +- **`BLE001` ignored repo-wide.** Twelve sites in `download.py`, `content_changed.py`, `tlds.py`, and `parse/root_db_html.py` catch `Exception` at an I/O or parse boundary, log it, and degrade to a defined fallback (`"error"`, treat-as-changed, omit optional field). That is the pipeline's deliberate resilience convention: one malformed source file must not abort a run. Narrowing all twelve would mean enumerating exception types for `json.load`, `.decode("idna")`, and httpx, where guessing wrong turns a logged degradation into a crash. A thirteenth site was flagged by both `BLE001` and `S110` and is fixed below rather than ignored. + +Nothing else is ignored. The remaining 59 violations were fixed, notably: + +- **`S110` in `build/tlds.py`** was the one genuine outlier: a silent, unlogged `except Exception: pass` around the IDN punycode decode, three lines from twelve siblings that all log. Narrowed to `except UnicodeError` (covers both `UnicodeDecodeError` from bad punycode and `UnicodeEncodeError` from non-ASCII input) and given a `logger.warning`. +- **`EXE001` in `src/cli.py`**: the `#!/usr/bin/env python3` shebang was vestigial. Every caller uses `python -m src.cli` (Makefile, `bin/build`), so the shebang was removed rather than the file made executable. +- **`PLW1510`** (7 sites): tests that deliberately assert on `returncode` now pass `check=False` explicitly. +- **`TRY002`** (6 sites in `tests/utilities/test_download.py`): mock guards for unexpected URLs became `AssertionError`; simulated transport failures became `RuntimeError`. Both still exercise the same `except Exception` production paths. + +`target-version` is not set in `ruff.toml`. Ruff still infers 3.13 from `requires-python` in `pyproject.toml` even when config resolves from a standalone `ruff.toml` (verified with `ruff check --show-settings`). This matters because `UP017` (`datetime.UTC`) is 3.11+ only; a mis-inferred target would silently drop 15 of the findings. diff --git a/docs/memory/memory-index.md b/docs/memory/memory-index.md index 60a3d8e2..1da827cd 100644 --- a/docs/memory/memory-index.md +++ b/docs/memory/memory-index.md @@ -9,6 +9,7 @@ Agents: consult before suggesting layout, naming, dependencies, vendors, or conv - [Architecture](architecture.md) - current implementation: stack, layout, conventions ## Log (newest first) +- [2026-08-07 Adopt ruff default rules](log/2026-08-07-ruff-defaults.md) - ruff 0.16 grew its implicit defaults from 59 rules to 413 and broke CI; `ruff.toml` now records the rule set explicitly. Only `BLE001` is ignored (boundary handlers that log and degrade on purpose); the other 59 violations were fixed. - [2026-07-16 iptoasn AS drift (teleinfo + verisign)](log/2026-07-16-teleinfo-asn-rename.md) - one refresh window renamed CAICT's AS (teleinfo -> "CAICT-AS-AP...") and migrated Verisign's .com/.net nameservers off "VERISIGN-AS" onto VRSN-AC28/VRSN-AC50-340, stranding both asn seeds; fix is data (reseed to live string, retire departed one to aliases). Reproduce against CI's pinned update-iptoasn artifact via gh run download, NOT iptoasn.com live-latest (which drifts ahead of CI) - [2026-06-07 Nightly rebuild picks up manual curation](log/2026-06-07-nightly-manual-regen.md) - update-data.yaml always rebuilds (--all on IANA source change, else --preserve-asn) so data/manual/ edits propagate to data/generated/ nightly without ASN churn; was gated on data/source/ only. Keep local `iptoasn` fresh or local passes while CI (daily artifact) fails - [2026-06-07 Verisign per-instance ASN block](log/2026-06-07-verisign-asn-block.md) - AS36616-36632 are all Verisign, one ASN per authoritative-server instance ([letter]GTLD for gtld-servers.net letters a-m plus X/Y/Z, AROOT for the roots it runs); each is a separate opaque as_org string, folded into the one VeriSign record as it appears (HGTLD/AS36623 added; other siblings not yet in data) diff --git a/pyproject.toml b/pyproject.toml index 44345c6c..4dd767a7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,7 +22,7 @@ dev = [ "pyright>=1.1.411", "pytest>=9.1.1", "pytest-cov>=7.1.0", - "ruff>=0.15.22", + "ruff>=0.16.0", ] [tool.pytest.ini_options] diff --git a/ruff.toml b/ruff.toml new file mode 100644 index 00000000..0fa1a3bb --- /dev/null +++ b/ruff.toml @@ -0,0 +1,6 @@ +[lint] +ignore = [ + # I/O and parse boundaries deliberately catch Exception, log it, and degrade + # to a defined fallback so one bad source file cannot abort a pipeline run. + "BLE001", +] diff --git a/src/analyze/__init__.py b/src/analyze/__init__.py index 2a224e70..b862e423 100644 --- a/src/analyze/__init__.py +++ b/src/analyze/__init__.py @@ -5,8 +5,8 @@ from .tlds_txt import analyze_tlds_txt, get_tlds_analysis __all__ = [ + "analyze_rdap_json", + "analyze_root_db_html", "analyze_tlds_txt", "get_tlds_analysis", - "analyze_root_db_html", - "analyze_rdap_json", ] diff --git a/src/analyze/root_db_html.py b/src/analyze/root_db_html.py index e5520b0f..c6b22d06 100644 --- a/src/analyze/root_db_html.py +++ b/src/analyze/root_db_html.py @@ -51,23 +51,23 @@ def analyze_root_db_html(filepath: Path) -> int: delegated_idn_by_type[entry["type"]] += 1 # Count unique managers for delegated TLDs - unique_managers = set(entry["manager"] for entry in delegated_entries) + unique_managers = {entry["manager"] for entry in delegated_entries} total_unique_managers = len(unique_managers) # Count unique gTLD managers (generic types) - gtld_managers = set( + gtld_managers = { entry["manager"] for entry in delegated_entries if entry["type"] in generic_types - ) + } total_unique_gtld_managers = len(gtld_managers) # Count unique ccTLD managers (country-code) - cctld_managers = set( + cctld_managers = { entry["manager"] for entry in delegated_entries if entry["type"] == "country-code" - ) + } total_unique_cctld_managers = len(cctld_managers) # Report results diff --git a/src/build/agreements.py b/src/build/agreements.py index f9b5f81e..c63a4b4b 100644 --- a/src/build/agreements.py +++ b/src/build/agreements.py @@ -6,7 +6,7 @@ """ import logging -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from ..config import REGISTRY_AGREEMENT_TYPE_MAPPING @@ -64,7 +64,7 @@ def build_agreements_json(tlds: list[dict], output_path: Path) -> tuple[bool, st output = { "description": _DESCRIPTION, - "publication": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "publication": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), "sources": _SOURCES, "agreements": agreements, } diff --git a/src/build/cultures.py b/src/build/cultures.py index 1ddf6876..0154c522 100644 --- a/src/build/cultures.py +++ b/src/build/cultures.py @@ -5,7 +5,7 @@ """ import logging -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from ..utilities.content_changed import write_json_if_changed @@ -53,7 +53,7 @@ def build_cultures_json( output = { "description": _DESCRIPTION, - "publication": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "publication": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), "sources": _SOURCES, "cultures": cultures, } diff --git a/src/build/organizations.py b/src/build/organizations.py index 84b24b0f..2bb80fad 100644 --- a/src/build/organizations.py +++ b/src/build/organizations.py @@ -5,7 +5,7 @@ """ import logging -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from ..parse.organizations import OrgRecord, OrgResolver @@ -64,7 +64,7 @@ def build_organizations_json( output = { "description": _DESCRIPTION, - "publication": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "publication": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), "sources": _SOURCES, "orgs": out_orgs, } diff --git a/src/build/places.py b/src/build/places.py index fe397865..f91d231c 100644 --- a/src/build/places.py +++ b/src/build/places.py @@ -10,7 +10,7 @@ """ import logging -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path import pycountry @@ -89,7 +89,7 @@ def build_places_json( output = { "description": _DESCRIPTION, - "publication": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "publication": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), "sources": _SOURCES, "places": records, } diff --git a/src/build/tlds.py b/src/build/tlds.py index fbf084c7..85d44a1d 100644 --- a/src/build/tlds.py +++ b/src/build/tlds.py @@ -4,7 +4,7 @@ import json import logging from dataclasses import dataclass -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -199,7 +199,7 @@ def build_tlds_json( # Build-time stamp passed to every write; write_json_if_changed skips # unchanged artifacts, so per-file publication reads as "last changed". - publication = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + publication = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ") sources = { "iana_root_db": IANA_URLS["ROOT_ZONE_DB"], "iana_rdap": IANA_URLS["RDAP_BOOTSTRAP"], @@ -458,9 +458,8 @@ def _build_tld_entry( try: tld_unicode = tld.encode("ascii").decode("idna") entry["tld_unicode"] = tld_unicode - except Exception: - # If decoding fails, skip unicode field - pass + except UnicodeError as e: + logger.warning("Could not decode IDN %s, omitting tld_unicode: %s", tld, e) # Add tld_script for IDNs if tld in idn_script_mapping: diff --git a/src/cli.py b/src/cli.py index 4c9fae15..33c23bc3 100644 --- a/src/cli.py +++ b/src/cli.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 """Command-line interface for IANA data ETL.""" import argparse diff --git a/src/parse/__init__.py b/src/parse/__init__.py index 2d7e71ff..c8757d38 100644 --- a/src/parse/__init__.py +++ b/src/parse/__init__.py @@ -24,27 +24,27 @@ __all__ = [ "ASNLookup", "ASNRecord", - "parse_iptoasn_tsv", "GtldRecord", - "parse_gtlds_json", - "parse_manual_annotations", "OrgResolver", "build_resolver", - "parse_organizations_manual", - "parse_tlds_txt", - "tlds_txt_content_changed", - "parse_root_db_html", - "parse_root_db_tlds", - "root_db_html_content_changed", "derive_type_from_iana_tag", - "parse_rdap_json", - "rdap_json_content_changed", - "parse_supplemental_cctld_rdap", - "parse_registry_agreement_csv", - "parse_agreement_types", - "get_normalized_agreement_types", "extract_main_content", + "get_all_country_mappings", "get_country_name", + "get_normalized_agreement_types", "is_cctld", - "get_all_country_mappings", + "parse_agreement_types", + "parse_gtlds_json", + "parse_iptoasn_tsv", + "parse_manual_annotations", + "parse_organizations_manual", + "parse_rdap_json", + "parse_registry_agreement_csv", + "parse_root_db_html", + "parse_root_db_tlds", + "parse_supplemental_cctld_rdap", + "parse_tlds_txt", + "rdap_json_content_changed", + "root_db_html_content_changed", + "tlds_txt_content_changed", ] diff --git a/src/utilities/__init__.py b/src/utilities/__init__.py index 383ea259..199a763d 100644 --- a/src/utilities/__init__.py +++ b/src/utilities/__init__.py @@ -16,8 +16,8 @@ "download_iptoasn", "download_tld_pages", "get_iptoasn_path", + "is_cache_fresh", "load_metadata", - "save_metadata", "parse_cache_control_max_age", - "is_cache_fresh", + "save_metadata", ] diff --git a/src/utilities/cache.py b/src/utilities/cache.py index 35555ba6..a0cedb7b 100644 --- a/src/utilities/cache.py +++ b/src/utilities/cache.py @@ -1,7 +1,7 @@ """Cache utilities for HTTP responses.""" import re -from datetime import datetime, timezone +from datetime import UTC, datetime from typing import Any @@ -43,6 +43,6 @@ def is_cache_fresh(metadata_entry: dict[str, Any]) -> bool: max_age = int(cache_data["cache_max_age"]) # Calculate age in seconds - age = (datetime.now(timezone.utc) - download_time).total_seconds() + age = (datetime.now(UTC) - download_time).total_seconds() return age < max_age diff --git a/src/utilities/download.py b/src/utilities/download.py index 466bb87e..b2d8a1e7 100644 --- a/src/utilities/download.py +++ b/src/utilities/download.py @@ -2,8 +2,8 @@ import logging import time +from collections.abc import Callable from pathlib import Path -from typing import Callable import httpx @@ -17,8 +17,8 @@ ) from .cache import is_cache_fresh, parse_cache_control_max_age from .metadata import load_metadata, save_metadata, utc_timestamp -from .urls import get_tld_file_path, get_tld_page_url from .retry import make_request_with_retry +from .urls import get_tld_file_path, get_tld_page_url logger = logging.getLogger(__name__) diff --git a/src/utilities/metadata.py b/src/utilities/metadata.py index ca708f14..dbf2cc50 100644 --- a/src/utilities/metadata.py +++ b/src/utilities/metadata.py @@ -2,7 +2,7 @@ import json import logging -from datetime import datetime, timezone +from datetime import UTC, datetime from pathlib import Path from typing import Any, Final @@ -15,7 +15,7 @@ def utc_timestamp() -> str: Returns: Timestamp string in format: 2025-11-18T20:23:07Z """ - return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ") GENERATED_DIR: Final[str] = "data/generated" diff --git a/src/utilities/retry.py b/src/utilities/retry.py index 2ad2deda..0e1558c3 100644 --- a/src/utilities/retry.py +++ b/src/utilities/retry.py @@ -18,8 +18,6 @@ class ServerError(Exception): """Raised when server returns 5xx error, triggering retry.""" - pass - def make_request_with_retry( client: httpx.Client, diff --git a/tests/build/test_idn_script_field.py b/tests/build/test_idn_script_field.py index a1b9d693..8e73cffa 100644 --- a/tests/build/test_idn_script_field.py +++ b/tests/build/test_idn_script_field.py @@ -60,7 +60,7 @@ def test_idn_script_mapping_file_format(): assert len(mappings) > 0 # Check that all keys are IDN TLDs - for tld in mappings.keys(): + for tld in mappings: assert tld.startswith("xn--"), f"Non-IDN TLD in mapping: {tld}" # Check that all values are strings diff --git a/tests/build/test_tlds.py b/tests/build/test_tlds.py index 36f17025..467811cf 100644 --- a/tests/build/test_tlds.py +++ b/tests/build/test_tlds.py @@ -450,7 +450,7 @@ def test_build_tlds_json_gtld_no_country_name(shared_build): for gtld in test_gtlds: if gtld in gtld_tlds: - entry = [e for e in gtlds if e["tld"] == gtld][0] + entry = next(e for e in gtlds if e["tld"] == gtld) if "annotations" in entry: assert "country_name_iso" not in entry["annotations"] diff --git a/tests/parse/test_country.py b/tests/parse/test_country.py index 7d22bdea..381d3f9b 100644 --- a/tests/parse/test_country.py +++ b/tests/parse/test_country.py @@ -105,7 +105,7 @@ def test_mappings_have_country_names(self): tlds = parse_tlds_txt(fixture_path) mappings = get_all_country_mappings(tlds) - for code, name in mappings.items(): + for name in mappings.values(): assert isinstance(name, str) assert len(name) > 0 diff --git a/tests/parse/test_root_db_html.py b/tests/parse/test_root_db_html.py index 81f7dcbb..99114a6f 100644 --- a/tests/parse/test_root_db_html.py +++ b/tests/parse/test_root_db_html.py @@ -125,16 +125,16 @@ def test_parse_root_db_html_unique_managers(): delegated = [e for e in entries if e.get("delegated", True)] # Count unique managers (excluding "Not assigned") - unique_managers = set(e["manager"] for e in delegated) + unique_managers = {e["manager"] for e in delegated} assert len(unique_managers) == 28 # Count unique gTLD managers generic_types = ["generic", "sponsored", "infrastructure", "generic-restricted"] - gtld_managers = set(e["manager"] for e in delegated if e["type"] in generic_types) + gtld_managers = {e["manager"] for e in delegated if e["type"] in generic_types} assert len(gtld_managers) == 18 # Count unique ccTLD managers - cctld_managers = set(e["manager"] for e in delegated if e["type"] == "country-code") + cctld_managers = {e["manager"] for e in delegated if e["type"] == "country-code"} assert len(cctld_managers) == 10 # Verify it's counting unique managers, not total TLDs diff --git a/tests/parse/test_supplemental_cctld_rdap.py b/tests/parse/test_supplemental_cctld_rdap.py index 7ce57bb8..efdd37df 100644 --- a/tests/parse/test_supplemental_cctld_rdap.py +++ b/tests/parse/test_supplemental_cctld_rdap.py @@ -77,6 +77,6 @@ def test_parse_supplemental_cctld_rdap_default_path(): # If file exists, should have actual data if rdap_lookup: # Check structure of first entry - first_tld = list(rdap_lookup.keys())[0] + first_tld = next(iter(rdap_lookup.keys())) assert "rdap_server" in rdap_lookup[first_tld] assert "source" in rdap_lookup[first_tld] diff --git a/tests/test_circular_imports.py b/tests/test_circular_imports.py index 04cf868c..0671de13 100644 --- a/tests/test_circular_imports.py +++ b/tests/test_circular_imports.py @@ -36,6 +36,7 @@ def test_no_circular_imports_with_pydeps(): capture_output=True, text=True, timeout=30, + check=False, ) # pydeps outputs cycle information to stdout/stderr @@ -72,6 +73,7 @@ def test_pydeps_is_installed(): [sys.executable, "-m", "pydeps", "--version"], capture_output=True, text=True, + check=False, ) assert result.returncode == 0, "pydeps not installed. Run: uv add --dev pydeps" diff --git a/tests/test_cli.py b/tests/test_cli.py index 0480e649..8038ffa8 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -21,9 +21,11 @@ def test_download_all_sources_success(self): "ROOT_ZONE_DB": "downloaded", } - with patch("src.cli.download_iana_files", return_value=mock_results): - with patch.object(sys, "argv", ["cli", "--download"]): - result = main() + with ( + patch("src.cli.download_iana_files", return_value=mock_results), + patch.object(sys, "argv", ["cli", "--download"]), + ): + result = main() assert result == 0 @@ -31,9 +33,11 @@ def test_download_specific_source_success(self): """Test downloading a specific source.""" mock_results = {"TLD_LIST": "downloaded"} - with patch("src.cli.download_iana_files", return_value=mock_results): - with patch.object(sys, "argv", ["cli", "--download", "TLD_LIST"]): - result = main() + with ( + patch("src.cli.download_iana_files", return_value=mock_results), + patch.object(sys, "argv", ["cli", "--download", "TLD_LIST"]), + ): + result = main() assert result == 0 @@ -45,9 +49,11 @@ def test_download_with_error(self): "ROOT_ZONE_DB": "downloaded", } - with patch("src.cli.download_iana_files", return_value=mock_results): - with patch.object(sys, "argv", ["cli", "--download"]): - result = main() + with ( + patch("src.cli.download_iana_files", return_value=mock_results), + patch.object(sys, "argv", ["cli", "--download"]), + ): + result = main() assert result == 1 diff --git a/tests/test_fetch_place_coordinates.py b/tests/test_fetch_place_coordinates.py index 05921117..338461af 100644 --- a/tests/test_fetch_place_coordinates.py +++ b/tests/test_fetch_place_coordinates.py @@ -104,18 +104,16 @@ def test_fetch_coordinates_raises_on_http_error(): def handler(request): return httpx.Response(404, json={}) - with _client(handler) as client: - with pytest.raises(ValueError): - fpc.fetch_coordinates(client, "Nowhere") + with _client(handler) as client, pytest.raises(ValueError): + fpc.fetch_coordinates(client, "Nowhere") def test_fetch_coordinates_raises_on_missing_entity(): def handler(request): return httpx.Response(200, json={"entities": {}}) - with _client(handler) as client: - with pytest.raises(ValueError): - fpc.fetch_coordinates(client, "Nowhere") + with _client(handler) as client, pytest.raises(ValueError): + fpc.fetch_coordinates(client, "Nowhere") # --- enrich_places --- diff --git a/tests/test_production_smoke.py b/tests/test_production_smoke.py index e169c032..8ad26d9b 100644 --- a/tests/test_production_smoke.py +++ b/tests/test_production_smoke.py @@ -27,12 +27,13 @@ def test_module_can_be_imported(): Circular imports only manifest during package-level imports. """ - # This will fail if there's a circular import in __init__.py files - import src # noqa: F401 - import src.parse # noqa: F401 + # The imports are the assertion: a circular import in __init__.py raises here. + # Each statement rebinds `src`, so only the final one is what F401 sees. + import src + import src.analyze + import src.build + import src.parse import src.utilities # noqa: F401 - import src.analyze # noqa: F401 - import src.build # noqa: F401 def test_cli_module_can_run(): @@ -48,6 +49,7 @@ def test_cli_module_can_run(): capture_output=True, text=True, timeout=10, + check=False, ) assert result.returncode == 0, f"CLI failed to run: {result.stderr}" @@ -66,6 +68,7 @@ def test_cli_download_flag_works(): capture_output=True, text=True, timeout=10, + check=False, ) # Should fail gracefully with error about unknown source, not crash @@ -89,6 +92,7 @@ def test_cli_download_tld_pages_flag_works(): capture_output=True, text=True, timeout=10, + check=False, ) # Should not crash with import errors @@ -104,6 +108,7 @@ def test_cli_analyze_flag_works(): capture_output=True, text=True, timeout=10, + check=False, ) # Should not crash with import errors @@ -167,6 +172,7 @@ def test_all_cli_subcommands_have_help(): capture_output=True, text=True, timeout=10, + check=False, ) assert result.returncode == 0 diff --git a/tests/utilities/test_cache.py b/tests/utilities/test_cache.py index 35748a80..658dd6d8 100644 --- a/tests/utilities/test_cache.py +++ b/tests/utilities/test_cache.py @@ -1,6 +1,6 @@ """Tests for cache utilities.""" -from datetime import datetime, timedelta, timezone +from datetime import UTC, datetime, timedelta from src.utilities.cache import is_cache_fresh, parse_cache_control_max_age @@ -43,7 +43,7 @@ def test_parse_cache_control_max_age_empty_string(): def test_is_cache_fresh_with_fresh_cache(): """Test that recently downloaded file with max-age is considered fresh.""" # Downloaded 1 hour ago, max-age is 6 hours - download_time = datetime.now(timezone.utc) - timedelta(hours=1) + download_time = datetime.now(UTC) - timedelta(hours=1) metadata_entry = { "cache_data": { @@ -58,7 +58,7 @@ def test_is_cache_fresh_with_fresh_cache(): def test_is_cache_fresh_with_stale_cache(): """Test that old downloaded file is considered stale.""" # Downloaded 7 hours ago, max-age is 6 hours - download_time = datetime.now(timezone.utc) - timedelta(hours=7) + download_time = datetime.now(UTC) - timedelta(hours=7) metadata_entry = { "cache_data": { @@ -84,7 +84,7 @@ def test_is_cache_fresh_missing_last_downloaded(): def test_is_cache_fresh_missing_cache_data(): """Test that missing cache_data returns False.""" metadata_entry = { - "last_checked": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "last_checked": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), } assert is_cache_fresh(metadata_entry) is False @@ -94,9 +94,7 @@ def test_is_cache_fresh_missing_cache_max_age(): """Test that missing cache_max_age returns False.""" metadata_entry = { "cache_data": { - "last_downloaded": datetime.now(timezone.utc).strftime( - "%Y-%m-%dT%H:%M:%SZ" - ), + "last_downloaded": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"), "etag": "abc123", }, } @@ -108,7 +106,7 @@ def test_is_cache_fresh_boundary_condition(): """Test cache freshness at exact max-age boundary.""" # Downloaded exactly max-age seconds ago max_age = 3600 - download_time = datetime.now(timezone.utc) - timedelta(seconds=max_age) + download_time = datetime.now(UTC) - timedelta(seconds=max_age) metadata_entry = { "cache_data": { @@ -124,7 +122,7 @@ def test_is_cache_fresh_boundary_condition(): def test_is_cache_fresh_short_max_age(): """Test with very short max-age like tlds.txt (205 seconds).""" # Downloaded yesterday, max-age is 205 seconds (definitely stale) - download_time = datetime.now(timezone.utc) - timedelta(days=1) + download_time = datetime.now(UTC) - timedelta(days=1) metadata_entry = { "cache_data": { diff --git a/tests/utilities/test_download.py b/tests/utilities/test_download.py index e9e39d0e..bf0c6321 100644 --- a/tests/utilities/test_download.py +++ b/tests/utilities/test_download.py @@ -1,15 +1,16 @@ """Tests for download utilities.""" import shutil +from datetime import UTC from pathlib import Path from unittest.mock import Mock, patch import httpx from src.utilities.download import ( + _download_file_impl, download_file, download_iana_files, - _download_file_impl, ) FIXTURES_DIR = Path(__file__).parent.parent / "fixtures" @@ -111,17 +112,18 @@ def test_download_with_304_not_modified(tmp_path): def mock_get(url, headers=None): # Return 304 for RDAP (has etag/last-modified in request) - if url == "https://data.iana.org/rdap/dns.json": - if headers and ( - "If-None-Match" in headers or "If-Modified-Since" in headers - ): - response = Mock(spec=httpx.Response) - response.status_code = 304 - response.headers = {} - return response + if ( + url == "https://data.iana.org/rdap/dns.json" + and headers + and ("If-None-Match" in headers or "If-Modified-Since" in headers) + ): + response = Mock(spec=httpx.Response) + response.status_code = 304 + response.headers = {} + return response # Shouldn't get here in this test - raise Exception(f"Unexpected request to {url}") + raise AssertionError(f"Unexpected request to {url}") with ( patch("src.utilities.download.SOURCE_DIR", str(source_dir)), @@ -153,12 +155,10 @@ def test_download_with_fresh_cache(tmp_path): def mock_get(url, headers=None): # RDAP and TLD_LIST should still make requests (but get 304 responses) - if url == "https://data.iana.org/rdap/dns.json": - response = Mock(spec=httpx.Response) - response.status_code = 304 - response.headers = {} - return response - elif url == "https://data.iana.org/TLD/tlds-alpha-by-domain.txt": + if ( + url == "https://data.iana.org/rdap/dns.json" + or url == "https://data.iana.org/TLD/tlds-alpha-by-domain.txt" + ): response = Mock(spec=httpx.Response) response.status_code = 304 response.headers = {} @@ -167,16 +167,16 @@ def mock_get(url, headers=None): # ROOT_ZONE_DB should NOT make a request (cache is fresh) nonlocal root_zone_request_made root_zone_request_made = True - raise Exception( + raise AssertionError( "Should not make HTTP request for ROOT_ZONE_DB when cache is fresh" ) - raise Exception(f"Unexpected URL: {url}") + raise AssertionError(f"Unexpected URL: {url}") # Mock "now" to be 1 hour after the fixture timestamp (within 24h cache window) # Fixture has last_downloaded: 2025-11-18T16:00:00Z with max_age: 86400 - from datetime import datetime, timezone + from datetime import datetime - mock_now = datetime(2025, 11, 18, 17, 0, 0, tzinfo=timezone.utc) + mock_now = datetime(2025, 11, 18, 17, 0, 0, tzinfo=UTC) with ( patch("src.utilities.download.SOURCE_DIR", str(source_dir)), @@ -225,7 +225,7 @@ def mock_get(url, headers=None): response.content = timestamp_only_content.encode("utf-8") response.text = timestamp_only_content return response - raise Exception(f"Unexpected request to {url}") + raise AssertionError(f"Unexpected request to {url}") with ( patch("src.utilities.download.SOURCE_DIR", str(source_dir)), @@ -568,9 +568,9 @@ def test_download_file_impl_cache_fresh_initializes_metadata(tmp_path): filepath.write_text("existing cached content") # Mock datetime for cache freshness check - from datetime import datetime, timezone + from datetime import datetime - mock_now = datetime(2025, 11, 18, 17, 0, 0, tzinfo=timezone.utc) + mock_now = datetime(2025, 11, 18, 17, 0, 0, tzinfo=UTC) # Metadata with fresh cache but key not yet in metadata dict metadata: dict = {} @@ -770,7 +770,7 @@ def test_download_tld_pages_handles_exception(tmp_path): # Mock to raise exception def mock_request(client, url, headers=None): - raise Exception("Network error") + raise RuntimeError("Network error") from src.utilities.download import download_tld_pages @@ -928,7 +928,7 @@ def test_download_iptoasn_exception(tmp_path): # Mock exception def mock_request(client, url, headers=None): - raise Exception("Connection failed") + raise RuntimeError("Connection failed") with ( patch("src.utilities.download.IPTOASN_DIR", str(iptoasn_dir)), diff --git a/uv.lock b/uv.lock index 5260ba7f..21abc233 100644 --- a/uv.lock +++ b/uv.lock @@ -167,7 +167,7 @@ dev = [ { name = "pyright", specifier = ">=1.1.411" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "pytest-cov", specifier = ">=7.1.0" }, - { name = "ruff", specifier = ">=0.15.22" }, + { name = "ruff", specifier = ">=0.16.0" }, ] [[package]] @@ -299,27 +299,27 @@ wheels = [ [[package]] name = "ruff" -version = "0.15.22" +version = "0.16.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3a/06/ae069393fc66e8ff33036d4b368003833bf6e88ccf182e17e7a2f1c754fd/ruff-0.15.22.tar.gz", hash = "sha256:3f15175b1fb580126f58285a5dae6b2ea89000136d980c64499211f116b54809", size = 4785063, upload-time = "2026-07-16T15:14:13.244Z" } +sdist = { url = "https://files.pythonhosted.org/packages/4d/94/1e5e4967626faf12fa56999cd6222dff6992ceb086ad7945756baf70c7a7/ruff-0.16.0.tar.gz", hash = "sha256:e460aafd5495ec89efaa6ced2e4a9a581116451e1c88b9d37ef497e0f8e93982", size = 4790557, upload-time = "2026-07-23T19:11:30.981Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/23/18/ee54b7ae1e121be7a28ea6da4b67564ebb0530e183a54415ab7e3bcd2c4e/ruff-0.15.22-py3-none-linux_armv6l.whl", hash = "sha256:44423e73493737f5e7c5b41d475483898ff37afcdae38bc3da5085e29af1c2d8", size = 10781258, upload-time = "2026-07-16T15:13:19.452Z" }, - { url = "https://files.pythonhosted.org/packages/2f/d2/2520cb14761ddbeaf57642a76942fc36adcbdbe53b4532241995f6fc485c/ruff-0.15.22-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:b82c6482946e9eda7ff2e091d25b8bad3f718684e1916d41bd56873cee05b697", size = 10999477, upload-time = "2026-07-16T15:13:23.318Z" }, - { url = "https://files.pythonhosted.org/packages/c9/10/74e53572aa758dfaa678c2a2646b5c5515d884b7ca56be4d2ce03ca4b560/ruff-0.15.22-py3-none-macosx_11_0_arm64.whl", hash = "sha256:11c1c715af53a09f714e011106bffc419751ec8232fcb5da42173284ea3fec6f", size = 10466716, upload-time = "2026-07-16T15:13:26.162Z" }, - { url = "https://files.pythonhosted.org/packages/1e/cc/44eaaf0844e028182f2d0a8f2190d0f359159aed0a9e5ab861d892f1ae2a/ruff-0.15.22-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:742a29cf29bddb7c8327895d6a10e0e6c5b38a96dd407af9b5d0857f809c0576", size = 10892644, upload-time = "2026-07-16T15:13:29.229Z" }, - { url = "https://files.pythonhosted.org/packages/9f/21/8edf559014d2b0f82beea19cfb713993ad802ccda16868769979c6090a84/ruff-0.15.22-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:72af58b951b0ae395935ae79763dc349bc0eb706319d28f7a33ad2cfb3cfc178", size = 10576719, upload-time = "2026-07-16T15:13:32.35Z" }, - { url = "https://files.pythonhosted.org/packages/bf/1e/3a13abd392a3b50b62e5938a831f9ab6e588358cacad5c18545b716d2182/ruff-0.15.22-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:62d425005c1835eb24e2ee4161cb90e8db263415f4a71c8c72c33abaa6c0c224", size = 11376494, upload-time = "2026-07-16T15:13:35.958Z" }, - { url = "https://files.pythonhosted.org/packages/bf/3e/422d3d95bcf04dd78e1aeac22184d4f9a8fb2c01865d39d44618484a0317/ruff-0.15.22-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e8b9b3f8779a4f08c969defc3c8c35abffaa757e601ed5ae66d6d1db6519969a", size = 12208370, upload-time = "2026-07-16T15:13:39.185Z" }, - { url = "https://files.pythonhosted.org/packages/1e/91/5d065a0e0a02bf4813f5119ad278462eed081d2b832eb7c021ade0ec9e65/ruff-0.15.22-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1e0dd1b2e4d3d585f897a0d137cbf4eaf6223bef4e8ce34d6bb12556c5f9249e", size = 11581098, upload-time = "2026-07-16T15:13:42.132Z" }, - { url = "https://files.pythonhosted.org/packages/f6/f9/a0d4871d12fae702eb1f41b686caf05f1f8b124dc6db6f784f53d74918fa/ruff-0.15.22-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:365523eb91d9224e1bcb03b022fbf0facb8f9e23792a2c53d9d4b3924bdbdebb", size = 11399422, upload-time = "2026-07-16T15:13:45.2Z" }, - { url = "https://files.pythonhosted.org/packages/18/80/c843a5176cddbceb0b7e8dd41cf9993490796c1c469348d384f5a5c13c56/ruff-0.15.22-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:fabfd168afdf29fee5be98b831efa9683c94d7c5a3b58b9ce5a2e38444589a74", size = 11381683, upload-time = "2026-07-16T15:13:48.46Z" }, - { url = "https://files.pythonhosted.org/packages/d4/00/8485de0ae92239438a36cfc51350db9b9e85c9ebdfaea91b18e422706662/ruff-0.15.22-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:225dbf095a87f1d9f90f5fd7924d2613ee452a75a4308c63a8f50f761787aa7c", size = 10850295, upload-time = "2026-07-16T15:13:51.655Z" }, - { url = "https://files.pythonhosted.org/packages/fa/91/24977ec2ec72eaf15e4394ace2959fdff2dd1e14f03e005e838023407169/ruff-0.15.22-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:1877d63b9d24ed278744f1523fd11b85540566d54641f97c566d7d9dc5ca5296", size = 10579640, upload-time = "2026-07-16T15:13:54.79Z" }, - { url = "https://files.pythonhosted.org/packages/9c/47/9b51216951974df1f263ac19da550d34252e0ed7218c25f10c5ef9ed7517/ruff-0.15.22-py3-none-musllinux_1_2_i686.whl", hash = "sha256:a1606c510bd7215680d32efab38965f7cdec3ef69f5170a3f4791404ffdd5262", size = 11105077, upload-time = "2026-07-16T15:13:57.915Z" }, - { url = "https://files.pythonhosted.org/packages/c2/47/20e9d4a3b8016778acea5fc32bb50d35d207500a17ddb529ffa6996feef8/ruff-0.15.22-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:630479b18625f5ffc373f77603a22a9f8ac0acd7ff0501178b5db28ec71e9c64", size = 11490980, upload-time = "2026-07-16T15:14:01.032Z" }, - { url = "https://files.pythonhosted.org/packages/4d/76/3f72d8fc38c1cb77b38c56a70da9d0c17700cc1cc50f9649c9d3c8f5ba71/ruff-0.15.22-py3-none-win32.whl", hash = "sha256:e5ba0e4a13fd14abbed2a77b517a3911290c6c6c59ef67784328d1668fab76cf", size = 10789165, upload-time = "2026-07-16T15:14:04.16Z" }, - { url = "https://files.pythonhosted.org/packages/cb/46/4965251734c2b6fcdca1b1b187d20bcac3af0ee5b083b89c910bb961ce3a/ruff-0.15.22-py3-none-win_amd64.whl", hash = "sha256:9be63ba1eb936acd2d1342fb8337c356353706fce233b2a15a09a97037e6acde", size = 11938297, upload-time = "2026-07-16T15:14:07.316Z" }, - { url = "https://files.pythonhosted.org/packages/57/c9/e69b1ff4c8b69093ef08b8919ab767af0569666865b39c30a8795d88d3c6/ruff-0.15.22-py3-none-win_arm64.whl", hash = "sha256:e1168075b72158510839f250027659cdd78476f40507dd517892304c41318661", size = 11298172, upload-time = "2026-07-16T15:14:10.51Z" }, + { url = "https://files.pythonhosted.org/packages/4b/81/1c8818fee7ce1a04cd7d1b3172e0a8f8e4f1dc4feb7fc390e16daa8af323/ruff-0.16.0-py3-none-linux_armv6l.whl", hash = "sha256:e5115729eb08c585e5121978ba5d5b60caeae394ce21b9fb5e6cd33a1c6c9b1e", size = 10754633, upload-time = "2026-07-23T19:10:46.415Z" }, + { url = "https://files.pythonhosted.org/packages/23/df/beaf59c09d68db84304d555f188b276a77132a5d5b0b67a5c762aa143628/ruff-0.16.0-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:3c954b1d580bfa035b41654f7858cc7e71d5fc3ac5b723dd62bd9133830ed522", size = 10969164, upload-time = "2026-07-23T19:10:50.271Z" }, + { url = "https://files.pythonhosted.org/packages/42/ce/741cd197496a1abbf51352710fd15ed995d2a2be87189c1da26a450d6e83/ruff-0.16.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:e01c21d10eb1b29f47b7454e1f4056db9a3f0260c646aa88457c610291db9f81", size = 10488846, upload-time = "2026-07-23T19:10:52.639Z" }, + { url = "https://files.pythonhosted.org/packages/52/2a/a2db8e88cade358f5cdcb05674a917751074109315d014eb6352d9a893f7/ruff-0.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6e364e5ed22ed8dc05082fd78e35308618260907ac2d3c1d637b2e682415b6c9", size = 10889729, upload-time = "2026-07-23T19:10:54.89Z" }, + { url = "https://files.pythonhosted.org/packages/42/65/62a771694ebd63029dc953e27dbad40e1588bd4860ff9fe881018fddaa49/ruff-0.16.0-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d327b8fc113a1d4421a04f3839d3752057c8dd1ee320223a6f3f52d04ada462a", size = 10568275, upload-time = "2026-07-23T19:10:56.993Z" }, + { url = "https://files.pythonhosted.org/packages/3f/e2/ced249fe8af5f086c5c58cc21cc3356d50f32f7401c5df87050c999620a7/ruff-0.16.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a9b50c55e263103586b3dcf5f73d479eb8cb5fdb6098fec59a62891dab653717", size = 11385112, upload-time = "2026-07-23T19:10:59.615Z" }, + { url = "https://files.pythonhosted.org/packages/87/0b/05154977a8fd69eeb6c103271f55403bfd8711f5c0f8ed07489d95a504e7/ruff-0.16.0-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0ff4a79ce3ec0172f3241943835de1c4cb4e2dcd07f0f8c2d02603dbbbee4b17", size = 12207008, upload-time = "2026-07-23T19:11:02.154Z" }, + { url = "https://files.pythonhosted.org/packages/fb/29/98225831a3a1eab0e02f4acc6ca6559a98611dcc68b6965ff4b7234627c1/ruff-0.16.0-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e95c448fca1fb2a18372a9440926c5a6ee789639bb975c72e7ae6d0b04218ab4", size = 11650842, upload-time = "2026-07-23T19:11:04.557Z" }, + { url = "https://files.pythonhosted.org/packages/91/66/6bd3cf90500653d55dc0ffc8507aa8300bd49d0214b2e8cb4d3fef2943ba/ruff-0.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f11a8d11010301d0a398a2fdef67691feca7294da6aef55e2150e8fa2cd520b", size = 11400718, upload-time = "2026-07-23T19:11:09.233Z" }, + { url = "https://files.pythonhosted.org/packages/8e/a2/a54eb4eae05d66364050a5d3b8a9c5ef88196531b3cbe7109d873f87f819/ruff-0.16.0-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:48044c678e9cb8698246c99b14aaccfa6601dea7379eb48a6f8f73f7a6d86cd0", size = 11426177, upload-time = "2026-07-23T19:11:11.994Z" }, + { url = "https://files.pythonhosted.org/packages/1a/be/16e3eea4b2a478a496919f5e36f17c4559e54620bd3bbac5d6affa068006/ruff-0.16.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:7aa0959bad8eb8bef50340154fc9b58678dae31fa4293afa38b44b6e552c0213", size = 10856126, upload-time = "2026-07-23T19:11:14.221Z" }, + { url = "https://files.pythonhosted.org/packages/a2/84/252eb8b868a16eec7257c14f504f77537e734b2d69c762e639e588e304a3/ruff-0.16.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:28ea2b7df8ebf7f9da6b7d47b230ab48f387c0a29be3b474c4d0740e197bb9af", size = 10571208, upload-time = "2026-07-23T19:11:16.378Z" }, + { url = "https://files.pythonhosted.org/packages/21/09/817a482f542f7570cbb4554b26e896610c7114f539b1d9e2d2145bf6bef6/ruff-0.16.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:33a3dfac8c35f81498dea9181bccc2f4c4bc8f1521a1dd9406e77643e0f0fb09", size = 11063329, upload-time = "2026-07-23T19:11:19.173Z" }, + { url = "https://files.pythonhosted.org/packages/2e/23/9403c180ca1cb9b1f7335f5c3e5305c09d49ea5b345196682a36028bde4a/ruff-0.16.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:a5237a0bda500d30d81b8e07a6973a5cbc772864cbf746ae2f4e8a2e01c9f4ed", size = 11489751, upload-time = "2026-07-23T19:11:21.74Z" }, + { url = "https://files.pythonhosted.org/packages/b2/1d/1b2ef7bcde851c78d7f17f1cca13fd6dc695fc4b3d6197941e72cae5b132/ruff-0.16.0-py3-none-win32.whl", hash = "sha256:7fab76fa065c873f41ff744347c6e77bcc3dfec4bcc754dc26b63d23c0f7f5fb", size = 10785885, upload-time = "2026-07-23T19:11:23.947Z" }, + { url = "https://files.pythonhosted.org/packages/b2/a3/d5e4ef7a56be3f928ffb90b94c25ba7d3cb9c7fe0736aeaaedf361770712/ruff-0.16.0-py3-none-win_amd64.whl", hash = "sha256:429c117f022bf481fabd9d551e7a3952b24c65e6ef44337ea09d90bebef14472", size = 11923141, upload-time = "2026-07-23T19:11:26.409Z" }, + { url = "https://files.pythonhosted.org/packages/cb/9a/8415f2657cbe200f41a4531ccededf135505a92d4a012229121f885b26f9/ruff-0.16.0-py3-none-win_arm64.whl", hash = "sha256:14296fedcd2705c77ab8235439278bbb38f285cf7da5528b00b3e330c3d4872d", size = 11273407, upload-time = "2026-07-23T19:11:28.705Z" }, ] [[package]]