Skip to content

Fix wfs plugin paging functionality#559

Merged
salabi merged 3 commits into
mainfrom
fix_wfs_plugin_paging_functionality
Jun 3, 2026
Merged

Fix wfs plugin paging functionality#559
salabi merged 3 commits into
mainfrom
fix_wfs_plugin_paging_functionality

Conversation

@salabi
Copy link
Copy Markdown
Contributor

@salabi salabi commented Jun 2, 2026

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Description

Currently, we cannot download the full WFS dataset because the endpoint response is capped at around 300k records (~120mb). The complete dataset is approximately 12 million records (5.5GB).

We need to implement WFS paging so the collector can retrieve and store the full dataset. Implementation details below.

  • Extend the existing WFS plugin with optional typed parameters, following the existing ArcGIS plugin pattern.
  • Keep current WFS behaviour unchanged by default.
  • When 'page_size' is given, use GDAL 'ogr2ogr' to download the WFS layer with paging enabled and write it to a temporary GeoPackage.
  • Derive the WFS layer name from the existing 'typeName'/'typeNames' value in endpoint-url.
  • Return a 'WFSFileResource' from the plugin so the collector can save the generated file without loading it into memory.
  • Update 'Collector.save_content()' to explicitly handle WFSFileResource by hashing/copying the file in chunks, while preserving existing byte-based behaviour.
  • Add unit tests for WFS parameter handling, 'ogr2ogr' command construction, failure cleanup, and collector file-resource saving.

Related Tickets & Documents

  • Ticket Link: ticket
  • Related Issue #
  • Closes #

QA Instructions, Screenshots, Recordings

Please replace this line with instructions on how to test your changes, a note
on the devices and browsers this has been tested on, as well as any relevant
images for UI changes.

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

  • Yes
  • No, and this is why: please replace this line with details on why tests
    have not been included
  • I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?

@salabi salabi marked this pull request as ready for review June 2, 2026 10:34
@salabi salabi requested a review from tombrooks248 June 2, 2026 10:41
@salabi salabi merged commit 34161d6 into main Jun 3, 2026
5 checks passed
@salabi salabi deleted the fix_wfs_plugin_paging_functionality branch June 3, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants