Feat: add big data workflow capabilities

[RaczeQ]

Currently, the library focuses completely on the GeoPandas GeoDataFrames and requires the whole dataset from start to finish to fit on the machine. This isn't ideal, since working with bigger areas requires higher RAM usage. In this issue, we should decide which framework/library to use in the final pipeline.

Any insight from people who used those tools with any tips will be very helpful 😄

Currently available options:

• dask-geopandas - GeoPandas extension for Dask
• Apache Sedona - dedicated wrapper over Apache Spark and Flink for spatial operations
• duckdb-spatial - fast in-memory db with spatial extension
• geoarrow-python - currently developed standard for Apache Arrow for storing spatial objects
• GeoPolars - geospatial extension for Polars, written in Rust

We should also decide if the library will depend on a single framework only, or if it will be open for extensions and implement multiple backends - similar to the ibis project. Since we write our code with abstract API, we should be able to implement multiple backends, but we will have to make sure that all results are consistent (high-quality tests) and with different backends, outputs will be either different (dask-dataframe, duckdb relation, sedona object, geodataframe, geoparquet/geofeather file path) or we will have to write an abstraction around each object to make it consistent and backends-agnostic.

We could also finish each operation with a calculated geo-parquet/arrow/feather file and work on files instead of loading them into memory.

Additional tools worth mentioning:

• https://github.com/rapidsai/cuspatial - CUDA accelerated spatial operations

[RaczeQ]

dask-geopandas

Pros:

• Very similar to GeoPandas API
• Can be written in Python
• No need for external tools

Cons:

• Dataframes are lazy and have to be computed by the user, harder to interact with in Jupyter
• Have to rewrite the logic from GeoPandas and Python operations to change the flow to make it vectorized and create an expressions tree to be executed.
• Calculating spatial index for faster operations (based on Hilbert Curve) replaces original index (region_id)
• Have to learn how to partition data into chunks

[RaczeQ]

Apache Sedona

Pros:

• Has many ST_ functions and H3 related functions
• Can be written in Python

Cons:

• Dataframes are lazy and have to be computed by the user, harder to interact with in Jupyter
• You have to use Spark cluster - hard to use out of the box without setup

[RaczeQ]

Duck DB with spatial extension

Pros:

• Very fast operations
• No need for external tools, can be easily imported into Python
• Dedicated OSM PBF native reader without GDAL dependency

Cons:

• Relations can be difficult to grasp for end users, and harder to interact with in Jupyter
• Currently no spatial index, it's sometimes slower than GeoPandas in spatial operations
• Has to be rewritten in SQL for all operations

[RaczeQ]

GeoPolars

Pros:

• Fast operations - Rust
• No need for external tools, can be easily imported into Python
• Can be written in Python
• Will implement GeoArrow format in the future

Cons:

• Not many functions are implemented
• Still in development and isn't production-ready

[RaczeQ]

GeoArrow for Python

Pros:

• Can read GeoArrow format into Pandas and PyArrow
• Should be able to work on files and vectorized operations using Arrow
• Can be written in Python

Cons:

• No spatial operations are available currently (for now you have to map to Shapely object to execute a function)

[RaczeQ]

Refined idea:
Use dedicated classes with geoparquet files and write code in duckdb and polars or pyarrow for lazy loading.

Make it possible to transform data into geodataframe.

Use dudckdb geography for intersections (s2cell) and maybe use h3 extension where possible.

Make it possible to represent multiple parquet files as a single object.