Ch/ex dqry postprocessing by cleherny · Pull Request #49 · swiss-territorial-data-lab/object-detector

cleherny · 2025-07-18T15:14:53Z

Update of the mineral extraction site example with improved merging and filtering approach, following the one of proj-dqry.

GwenaelleSa

I reviewed the files except the new filter, because it was close to the one on the main project DQRY, so the same comments apply.
I don't agree with your addition to the data preparation, because I feel like you copy-pasted some code from elsewhere without taking into account the latest modification. I did a new branch with a PR on this branch: #51. It rollbacks some of your modifications and factorize the post-processing more.
I kept your new function merge_polygons and the post-processing in merge_adjacent_detections is a mix of your code and of the comments I left on this PR. I kept the previous version of the year control, because assigning random year to empty tiles seemed like a risky business and because I was lazy.

GwenaelleSa · 2025-09-11T10:34:00Z

+    # Prepare the tiles
+
+    ## Convert datasets shapefiles into geojson format
+    logger.info('Convert the label shapefiles into GeoJSON format (EPSG:4326)...')
+    labels_4326_gdf, written_files = prepare_labels(SHPFILE, written_files)
+    gt_labels_4326_gdf = labels_4326_gdf[['geometry', 'CATEGORY', 'SUPERCATEGORY']].copy()
+
+    # Add FP labels if it exists
+    if FP_SHPFILE:
+        logger.info('Convert the FP label shapefiles into GeoJSON format (EPSG:4326)...')
+        fp_labels_4326_gdf, written_files = prepare_labels(FP_SHPFILE, written_files, prefix='FP_')
+        labels_4326_gdf = pd.concat([labels_4326_gdf, fp_labels_4326_gdf], ignore_index=True)
+
+    # Tiling of the AoI
+    logger.info("- Get the label boundaries")  
+    boundaries_df = labels_4326_gdf.bounds
+    logger.info("- Tiling of the AoI")  
+    tiles_4326_aoi_gdf = aoi_tiling(boundaries_df)
+    tiles_4326_labels_gdf = gpd.sjoin(tiles_4326_aoi_gdf, labels_4326_gdf, how='inner', predicate='intersects')
+
+    # Tiling of the AoI from which empty tiles will be selected
+    if EPT_SHPFILE:
+        EPT_aoi_gdf = gpd.read_file(EPT_SHPFILE)
+        EPT_aoi_4326_gdf = EPT_aoi_gdf.to_crs(epsg=4326)
+        assert_year(labels_4326_gdf, EPT_aoi_4326_gdf, 'empty_tiles', EPT_YEAR)
+
+        if EPT_TYPE == 'aoi':
+            logger.info("- Get AoI boundaries")  
+            EPT_aoi_boundaries_df = EPT_aoi_4326_gdf.bounds
+
+            # Get tile coordinates and shapes
+            logger.info("- Tiling of the empty tiles AoI")  
+            empty_tiles_4326_all_gdf = aoi_tiling(EPT_aoi_boundaries_df)
+            # Delete tiles outside of the AoI limits 
+            empty_tiles_4326_aoi_gdf = gpd.sjoin(empty_tiles_4326_all_gdf, EPT_aoi_4326_gdf, how='inner', lsuffix='ept_tiles', rsuffix='ept_aoi')
+            # Attribute a year to empty tiles if necessary
+            if 'year' in labels_4326_gdf.keys():
+                if isinstance(EPT_YEAR, int):
+                    empty_tiles_4326_aoi_gdf['year'] = int(EPT_YEAR)
+                else:
+                    empty_tiles_4326_aoi_gdf['year'] = np.random.randint(low=EPT_YEAR[0], high=EPT_YEAR[1], size=(len(empty_tiles_4326_aoi_gdf)))
+        elif EPT_TYPE == 'shp':
+            if EPT_YEAR:
+                logger.warning("A shapefile of selected empty tiles are provided. The year set for the empty tiles in the configuration file will be ignored")
+                EPT_YEAR = None
+            empty_tiles_4326_aoi_gdf = EPT_aoi_4326_gdf.copy()
+
+        # Get all the tiles in one gdf 
+        logger.info("- Concatenate label tiles and empty AoI tiles") 
+        tiles_4326_all_gdf = pd.concat([tiles_4326_labels_gdf, empty_tiles_4326_aoi_gdf])
+    else: 
+        tiles_4326_all_gdf = tiles_4326_labels_gdf.copy()
+
+    # - Remove useless columns, reset feature id and redefine it according to xyz format  
+    logger.info('- Add tile IDs and reorganise the data set')
+    tiles_4326_all_gdf = tiles_4326_all_gdf[['geometry', 'title', 'year'] if 'year' in tiles_4326_all_gdf.keys() else ['geometry', 'title']].copy()
+    tiles_4326_all_gdf.reset_index(drop=True, inplace=True)
+    tiles_4326_all_gdf = tiles_4326_all_gdf.apply(add_tile_id, axis=1)
+
+    # - Remove duplicated tiles
+    tiles_4326_all_gdf.drop_duplicates(['id'], inplace=True)
+
+    nb_tiles = len(tiles_4326_all_gdf)
+    logger.info(f"There were {nb_tiles} tiles created")


Same, see function https://github.com/swiss-territorial-data-lab/object-detector/blob/ch/ex-dqry_postprocessing/helpers/functions_for_examples.py#L16

I'm not sur to what I have to look at here.

I think the link was supposed to point here: https://github.com/swiss-territorial-data-lab/object-detector/blob/ch/ex-dqry_postprocessing/helpers/functions_for_examples.py#L86

GwenaelleSa · 2025-09-11T13:41:36Z

+        detections_merge_gdf['geometry'] = detections_merge_gdf.geometry.buffer(-1, join_style='mitre')
+
+        # Merge adjacent polygons within the provided thd distance
+        detections_merge_gdf['geometry'] = detections_merge_gdf.geometry.buffer(DISTANCE, join_style='mitre')


Detections are already buffered based on ln. 104, except with 1 instead of DISTANCE. The geometry passed down is the buffered one.
Is it really what you want? Wouldn't it be simpler to always buffer with DISTANCE?

I checked, and indeed, it didn't change the input geometry much.

I had to dig a bit to remember why I did that... I think it's because I still want the adjacent tiles to be merged even if the distance threshold is below the distance to needed to merge adjacent polygons (which is unlikely to happen in this case) or if the value is set to 0 but may be we can add a condition for this case.

GwenaelleSa · 2025-09-11T13:50:09Z

+        detections_overlap_tiles_gdf = misc.merge_polygons(detections_overlap_tiles_gdf)
+
+        # Concat polygons contained within a tile and the merged ones
+        detections_merge_gdf = pd.concat([detections_overlap_tiles_gdf, detections_within_tiles_gdf], axis=0, ignore_index=True)


Since detections_within_tiles_gdf are within tiles, you don't need to include them in all the stuff about merging detections and getting the right attributes after. You could just use the initial unbuffered geometry and add them at the end (ln 156).

cleherny

Thanks Gwenaëlle for your review. I've answered to your comments and wait for #51 to make the last changes.

cleherny · 2025-09-26T16:52:55Z

+        detections_merge_gdf['geometry'] = detections_merge_gdf.geometry.buffer(-1, join_style='mitre')
+
+        # Merge adjacent polygons within the provided thd distance
+        detections_merge_gdf['geometry'] = detections_merge_gdf.geometry.buffer(DISTANCE, join_style='mitre')


I checked, and indeed, it didn't change the input geometry much.

I had to dig a bit to remember why I did that... I think it's because I still want the adjacent tiles to be merged even if the distance threshold is below the distance to needed to merge adjacent polygons (which is unlikely to happen in this case) or if the value is set to 0 but may be we can add a condition for this case.

cleherny · 2025-09-26T17:29:20Z

+    # Prepare the tiles
+
+    ## Convert datasets shapefiles into geojson format
+    logger.info('Convert the label shapefiles into GeoJSON format (EPSG:4326)...')
+    labels_4326_gdf, written_files = prepare_labels(SHPFILE, written_files)
+    gt_labels_4326_gdf = labels_4326_gdf[['geometry', 'CATEGORY', 'SUPERCATEGORY']].copy()
+
+    # Add FP labels if it exists
+    if FP_SHPFILE:
+        logger.info('Convert the FP label shapefiles into GeoJSON format (EPSG:4326)...')
+        fp_labels_4326_gdf, written_files = prepare_labels(FP_SHPFILE, written_files, prefix='FP_')
+        labels_4326_gdf = pd.concat([labels_4326_gdf, fp_labels_4326_gdf], ignore_index=True)
+
+    # Tiling of the AoI
+    logger.info("- Get the label boundaries")  
+    boundaries_df = labels_4326_gdf.bounds
+    logger.info("- Tiling of the AoI")  
+    tiles_4326_aoi_gdf = aoi_tiling(boundaries_df)
+    tiles_4326_labels_gdf = gpd.sjoin(tiles_4326_aoi_gdf, labels_4326_gdf, how='inner', predicate='intersects')
+
+    # Tiling of the AoI from which empty tiles will be selected
+    if EPT_SHPFILE:
+        EPT_aoi_gdf = gpd.read_file(EPT_SHPFILE)
+        EPT_aoi_4326_gdf = EPT_aoi_gdf.to_crs(epsg=4326)
+        assert_year(labels_4326_gdf, EPT_aoi_4326_gdf, 'empty_tiles', EPT_YEAR)
+
+        if EPT_TYPE == 'aoi':
+            logger.info("- Get AoI boundaries")  
+            EPT_aoi_boundaries_df = EPT_aoi_4326_gdf.bounds
+
+            # Get tile coordinates and shapes
+            logger.info("- Tiling of the empty tiles AoI")  
+            empty_tiles_4326_all_gdf = aoi_tiling(EPT_aoi_boundaries_df)
+            # Delete tiles outside of the AoI limits 
+            empty_tiles_4326_aoi_gdf = gpd.sjoin(empty_tiles_4326_all_gdf, EPT_aoi_4326_gdf, how='inner', lsuffix='ept_tiles', rsuffix='ept_aoi')
+            # Attribute a year to empty tiles if necessary
+            if 'year' in labels_4326_gdf.keys():
+                if isinstance(EPT_YEAR, int):
+                    empty_tiles_4326_aoi_gdf['year'] = int(EPT_YEAR)
+                else:
+                    empty_tiles_4326_aoi_gdf['year'] = np.random.randint(low=EPT_YEAR[0], high=EPT_YEAR[1], size=(len(empty_tiles_4326_aoi_gdf)))
+        elif EPT_TYPE == 'shp':
+            if EPT_YEAR:
+                logger.warning("A shapefile of selected empty tiles are provided. The year set for the empty tiles in the configuration file will be ignored")
+                EPT_YEAR = None
+            empty_tiles_4326_aoi_gdf = EPT_aoi_4326_gdf.copy()
+
+        # Get all the tiles in one gdf 
+        logger.info("- Concatenate label tiles and empty AoI tiles") 
+        tiles_4326_all_gdf = pd.concat([tiles_4326_labels_gdf, empty_tiles_4326_aoi_gdf])
+    else: 
+        tiles_4326_all_gdf = tiles_4326_labels_gdf.copy()
+
+    # - Remove useless columns, reset feature id and redefine it according to xyz format  
+    logger.info('- Add tile IDs and reorganise the data set')
+    tiles_4326_all_gdf = tiles_4326_all_gdf[['geometry', 'title', 'year'] if 'year' in tiles_4326_all_gdf.keys() else ['geometry', 'title']].copy()
+    tiles_4326_all_gdf.reset_index(drop=True, inplace=True)
+    tiles_4326_all_gdf = tiles_4326_all_gdf.apply(add_tile_id, axis=1)
+
+    # - Remove duplicated tiles
+    tiles_4326_all_gdf.drop_duplicates(['id'], inplace=True)
+
+    nb_tiles = len(tiles_4326_all_gdf)
+    logger.info(f"There were {nb_tiles} tiles created")


I'm not sur to what I have to look at here.

…x_dqry Factorize detection merge

stdl-s-sonarqube · 2025-09-30T09:41:28Z

4.72% Duplicated Lines (%) on New Code (is greater than 3.00%)

Analysis Details

2 Issues

0 Bugs
0 Vulnerabilities
2 Code Smells

Coverage and Duplications

No coverage information (0.00% Estimated after merge)
4.72% Duplicated Code (0.70% Estimated after merge)

Project ID: swiss-territorial-data-lab_object-detector_AYZ4zWIzr7JdaaSXwexc

View in SonarQube

cleherny added 16 commits July 16, 2025 15:49

Rename detectron2 config file

b2d7b60

Rename GT shapefile

585abfb

Add merge_detection and modify filter_detections

6a013c7

Update README

1f86ac8

Update version in setup.py

18853fd

Update requirements

0ccd0f2

Add fct merge_polygons and none_if_undefined in misc

dfab19a

Update prepare_data as in proj-dqry

f510a9d

Old mode to new mode

4b325a4

Remove FP list of labels folder

471a72c

Modify filter_detections

7a0406a

Add merge_detections

c796a4a

Update config files

ffd5141

Update README

1ed936a

Update get_fractional_sets

73a79a0

Correct errors in commands of the README of anthropogenic example

8ae9b82

cleherny self-assigned this Jul 18, 2025

This comment has been minimized.

Sign in to view

GwenaelleSa added 4 commits September 11, 2025 14:15

Factorize detection merge

43c7846

Re-factorize data preparation

1aed91c

Do not merge dets within tiles

07df99d

Remove unnecessary inputs

2cdad5a

GwenaelleSa requested changes Sep 12, 2025

View reviewed changes

GwenaelleSa added 2 commits September 22, 2025 08:14

Remove unnecessary else statement

a3db010

Merge branch 'master' into gs/factorize_ex_dqry

9e8eb1f

cleherny mentioned this pull request Sep 27, 2025

Factorize detection merge #51

Merged

cleherny commented Sep 27, 2025

View reviewed changes

GwenaelleSa and others added 2 commits September 29, 2025 08:40

Consider review comments

ccb1b93

Merge master

794620b

This comment has been minimized.

Sign in to view

cleherny added 2 commits September 30, 2025 10:36

Merge branch 'ch/ex-dqry_postprocessing' into gs/factorize_ex_dqry

70fe2e1

Merge pull request #51 from swiss-territorial-data-lab/gs/factorize_e…

076fb64

…x_dqry Factorize detection merge

This comment has been minimized.

Sign in to view

GwenaelleSa approved these changes Sep 30, 2025

View reviewed changes

Correct errors after merging master

9913977

cleherny merged commit ed23ba7 into master Sep 30, 2025
1 check passed

cleherny deleted the ch/ex-dqry_postprocessing branch September 30, 2025 09:44

Conversation

cleherny commented Jul 18, 2025

Uh oh!

This comment has been minimized.

GwenaelleSa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GwenaelleSa Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

cleherny Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

GwenaelleSa Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GwenaelleSa Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cleherny Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

GwenaelleSa Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cleherny left a comment

Choose a reason for hiding this comment

Uh oh!

cleherny Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cleherny Sep 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

stdl-s-sonarqube Bot commented Sep 30, 2025

Analysis Details

2 Issues

Coverage and Duplications

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GwenaelleSa left a comment •

edited

Loading

GwenaelleSa Sep 11, 2025 •

edited

Loading