Add data.gov catalog harvester

[jguo144]

Description

Adds a new harvester plugin that ingests datasets from Data.gov using their catalog API. Details about their API can be found at https://resources.data.gov/catalog-api/.

Features

• DCAT-US metadata mapping: Maps Data.gov's DCAT-US format to CKAN structure (core fields, spatial, temporal, contact points, resources)
• Configurable query parameters: Supports all Data.gov API parameters (q, keyword, org_slug, spatial_geometry, spatial_within, etc.)
• Cursor-based pagination: Efficiently handles large result sets using Data.gov's after cursor
• Theme → Group mapping: Automatically maps Data.gov themes to CKAN groups with slugification
• Organization filtering: Post-harvest filtering with organizations_filter_include/organizations_filter_exclude
• Resource ID preservation: Maintains resource IDs across harvests to preserve datastore data
• Smart date trimming: Trims midnight timestamps to date-only format for cleaner display
• Composite field support: Enables mapping of converted extras to composite fields via configuration

Files Changed

• New: ckanext/custom_harvest/harvesters/datagov.py - Main harvester class
• New: ckanext/custom_harvest/tests/harvesters/mock_datagov.py - Mock server for testing
• New: ckanext/custom_harvest/tests/harvesters/test_datagov_harvester.py - Comprehensive test suite
• New: DATA_GOV_HARVESTER.md - Complete usage documentation
• Modified: ckanext/custom_harvest/converter.py - Added datagov_to_ckan() converter and helpers
• Modified: setup.py - Registered harvester entry point

[nolanpruett]

LGTM! 🚀