Results as grid valuesv2#325
Open
timlinux wants to merge 60 commits into
Open
Conversation
- Add write_buffer_values_to_grid() for spatial join with buffer polygons - Simplify write_aggregation_to_grid() to use single SQL UPDATE - Improve write_raster_values_to_grid() with spatial filtering and batch SQL - Add grid-first approach to polyline_per_cell and polygon_per_cell workflows - Add grid-first approach to single_point_buffer and multi_buffer workflows - Add grid-first approach to classified_polygon workflow - Fix button styles to use white text on blue backgrounds - Fix unused imports and f-string placeholders - Fix shellcheck warnings in start scripts
When a factor has only one indicator child, the factor is now hidden and the indicator is displayed directly under the dimension. This reduces visual clutter in the tree. Changes: - Add get_effective_visible_children() to JsonTreeItem for child promotion - Add get_visual_parent() to JsonTreeItem for hidden parent handling - Add visible_row() to JsonTreeItem for visible position calculation - Update JsonTreeModel.rowCount() to use effective visible children - Update JsonTreeModel.index() to use effective visible children - Update JsonTreeModel.parent() to handle hidden parents - Update _findIndexByGuid() to search both visible and hidden items - Add toggle_single_child_factors_visibility() method to model - Add are_single_child_factors_hidden() method to model - Enable feature by default when loading model data
- Add indicator-vector-template.qml with [attribute] placeholder - Add add_grid_layer_to_map() function to load study_area_grid with templated styling for any column - Add "Add to map (Grid)" action to context menus for indicators, factors, and dimensions - Users can now compare raster and vector grid visualizations
The QAction.triggered signal passes a boolean 'checked' parameter which was being captured by the lambda's first keyword argument, overwriting the column_name. Added underscore parameter to properly capture and discard the signal argument.
- Log function entry with column name and working directory - Check for None working directory early with warning - Log layer URI and layer name - Verify column exists in grid before applying style - Log available columns for debugging - Append (Grid) suffix to layer name to distinguish from raster
The AnalysisAggregationWorkflow was using layer_id="geoe3" but the grid column is actually named "wee_score" as defined in grid_column_utils.get_aggregate_column_names(). This caused the analysis aggregation to fail when trying to write to a non-existent column.
The OpportunitiesByWeeScoreProcessingTask and WeeByPopulationScoreProcessor
were looking for geoe3_masked_{index}.tif but the analysis workflow now
creates wee_score_masked_{index}.tif (using layer_id as filename prefix).
Updated both processors to use the correct filename pattern.
Changed the analysis aggregation to use layer_id="geoe3" and updated
grid_column_utils to use "geoe3" and "geoe3_by_population" as the
aggregate column names.
This maintains consistency with the existing file naming convention
(geoe3_masked_{index}.tif) expected by the masked score processors.
Reduces log spam from PyQt UI loader debug messages like 'push QCheckBox', 'pop widget', 'setting property', etc.
The PyQt uic module emits verbose debug messages when loading .ui files: - "push QCheckBox", "pop widget", "setting property", etc. These are not from GeoE3 code but from PyQt's UI compiler internals. Rather than changing the overall log level to INFO, specifically suppress the PyQt5.uic and PyQt6.uic loggers to WARNING level. This keeps DEBUG level available for GeoE3 code debugging.
When adding a grid layer to the map, apply a subset filter to exclude features where the visualized column has NULL values. This prevents empty/unclassified features from being rendered. Filter is applied both for new layers and when refreshing existing ones.
When clicking a tree item (with show_layer_on_click enabled) or when a workflow completes, now adds the grid vector layer by default instead of the raster layer. The grid layer provides faster rendering and filtering of NULL values. Raster visualization is still available via context menu "Add to map".
Resolved conflict in index_score_with_ookla_workflow.py by accepting main's approach: use full clip area with score 0 when no Ookla coverage.
Modified _get_grid_layer_and_field_index to create missing columns as Real/Float type when create_if_missing=True (the default). This ensures geoe3 and other aggregate columns are created on-the-fly if the grid was created before the column names were updated. Also updated write_uniform_value_to_grid and write_raster_values_to_grid to use the helper function for consistent column creation behavior.
- on_item_clicked: Add analysis role with geoe3 column - on_workflow_completed: Add analysis role with geoe3 column - Context menu: Add "Add GeoE3 Score to Map (Grid)" option - Renamed raster option to "Add GeoE3 Score to Map (Raster)" for clarity
- Update add_masked_scores_to_map() to use add_grid_layer_to_map() with geoe3_masked and geoe3_by_population_masked columns - Add geoe3_masked and geoe3_by_population_masked to aggregate columns - Update OpportunitiesByWeeScoreProcessingTask to write masked values to geoe3_masked grid column after raster calculation - Update WEEByPopulationScoreProcessingTask to write values to geoe3_by_population_masked grid column after raster calculation - Fix AreaIterator unpacking to include 5th element (area_name)
- Update 'Add GeoE3 by Pop to Map' to use add_grid_layer_to_map() with geoe3_by_population column - Update 'Add Job Opportunities Mask to Map' to use add_grid_layer_to_map() with opportunities_mask column - Add opportunities_mask to aggregate column names in grid_column_utils.py - Update WEEByPopulationScoreProcessingTask to write to geoe3_by_population column (not geoe3_by_population_masked) - Update OpportunitiesMaskProcessor to write mask values to opportunities_mask grid column - Fix area_iterator tuple unpacking to include area_name (5th element)
Handle RuntimeError when GeoPackage file exists but required metadata tables (gpkg_spatial_ref_sys, gpkg_contents) are not yet created. This occurs during the early stages of GeoPackage initialization.
…ields, and summary methods #343
…ighttime lights data from S2S
Results as grid valuesv2
Multiple OGR write connections opening study_area.gpkg in WAL mode left uncheckpointed WAL/SHM files, causing QGIS's OGR provider to return empty CRS metadata on subsequent reads. This broke all workflows that depend on target_crs for reprojection. - Add WAL checkpoint (PRAGMA wal_checkpoint(TRUNCATE)) before closing write connections in grid_column_utils, study_area_processing_task, and features_per_cell_processor - Add _resolve_target_crs() fallback in workflow_base that reads CRS directly from gpkg metadata tables when QgsVectorLayer.crs() fails
When rerunning workflows, stale values from previous runs persisted in grid columns because they were never NULLed out before new values were written. This caused geoe3_masked and other derived columns to retain incorrect values after upstream indicators were fixed. - Add clear_grid_column call before aggregation in aggregation_workflow_base (affects all factor/dimension/analysis) - Add clear_grid_column for geoe3_masked in opportunities_by_wee_score_processor - Add clear_grid_column for geoe3_by_population in wee_by_population_score_processor - Add clear_grid_column for opportunities_mask in opportunities_mask_processor
…ling The raster algebra approach (mask * geoe3) produces 0 for cells outside the settlements mask, which then gets written to the grid as 0 instead of NULL. Replace the raster-to-grid sampling with a simple SQL copy: geoe3_masked = geoe3 WHERE opportunities_mask IS NOT NULL. This correctly leaves non-settlement cells as NULL and is more efficient.
Both workflows previously created a scored polygon, rasterized it, then sampled the raster back to the grid — an unnecessary roundtrip. Now they use write_spatial_join_to_grid to directly set the index score on grid cells that intersect the mask layer (GHSL settlements or Ookla coverage tiles), then rasterize from the grid column only for VRT visualization output. This is simpler, faster, and consistent with the SQL-first approach used by other workflows (multi-buffer, polygon-per-cell, etc.).
AreaIterator now yields 5 values (current_area, clip_area, current_bbox, progress, area_name) but population_processor.py and opportunities_by_wee_score_population_processor.py still unpacked only 4, causing 'too many values to unpack' errors that prevented population processing and geoe3_by_population from being computed.
The cleanup step previously kept all TIF files in workflow directories. Now it parses VRT files to find which TIFs they reference (the final masked rasters) and deletes all other TIFs (clipped, reclassified, aggregated, unmasked intermediates). This reduces working directory size by ~40% since only the final masked rasters needed for visualization are retained.
The intermediate file cleanup was deleting subdirectories containing child workflow outputs (e.g. factor cleanup deleted indicator dirs). This destroyed all indicator/factor VRTs and masked rasters, causing blank maps in the analysis report PDF. Now the cleanup only deletes files, never subdirectories.
…fication and S2S data handling
… checks in project panel
…ets in user guide
…dy area processing
… and unique value mapping
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.