Feat/dem elevation csv - #27
Closed
telespazio-tim wants to merge 31 commits into
Closed
telespazio-tim wants to merge 31 commits into
telespazio-tim wants to merge 31 commits into
Conversation
The black hook (bumped from 24.4.2 to 26.3.1 in 290b7ef) has a newer stable style, so `pre-commit run --all-files` was failing on formatting alone. No behavior changes.
Library users following the README literally (output_directory as a plain str) hit a TypeError in report generation, since some call sites join it with the `/` operator, which only works on pathlib.Path. Coerce it in __post_init__ so both str and Path inputs work consistently. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Library users following the README literally (output_directory as a plain str) hit a TypeError in report generation, since some call sites join it with the `/` operator, which only works on pathlib.Path. Coerce it in __post_init__ so both str and Path inputs work consistently. Co-authored-by: Gautier Portet <gautier.portet@telespazio.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The header banner and footer logo were referenced by relative filename and
copied next to the generated pages by _copy_assets(), so a report was only
correct as a whole directory: moving or mailing report.html alone lost its
branding.
Encode both assets as base64 data URIs instead, so each page carries its own
branding. CSS_STYLES is passed to str.format() as a *value* and is full of
literal CSS braces, so it cannot hold a {} field; the footer logo goes in via
a sentinel token replaced at generation time.
When an asset is unavailable the markup is omitted rather than emitted empty:
both src="" and url('') resolve against the page itself in browsers.
Plot and chip images keep their relative paths, so the pages are
self-contained for branding only, not fully standalone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inlining duplicates the banner into every generated page, so its size now costs ~4/3 of its bytes per page instead of once per output directory. The asset was encoded well above what a 1200x300 banner needs. Re-encoding at q85 takes it from 157,514 to 125,942 bytes (SSIM 0.981 against the previous file, visually indistinguishable), saving 126,288 bytes across the three pages. AVIF was measured as the alternative and rejected: on this image it is only ~3% smaller than webp at matched SSIM, and data-URI inlining rules out cheap <picture> fallback for viewers without AVIF support, including the HTML-to-PDF converters likely to be pointed at these reports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes to the KLT matching path, kept together because they touch the same functions and no subset leaves _match_tile in a working state. Pyramid depth is now KLTConfiguration.maxLevel instead of a source constant. It had already drifted silently during development, and it is the parameter that most affects results: on a SPOT5/BSG pair, depth 3 yields 6794 key points against 1850 at depth 5, because a deeper pyramid scales the tracking window by 2**maxLevel and erodes matching inward from every data edge. Tiles now read a margin of real neighbouring pixels, sized from that depth, so key points at a tile edge keep full window support. Tiles did not overlap, so the band was lost at every internal seam: a 300px image in 100px tiles kept 21 of the 60 seam key points it finds untiled. Synthetic padding was measured and rejected - reflected or replicated content does not move consistently between the two images, so an LK window overlapping it scores worse than a truncated one. Only real pixels help, and at the outer image edge none exist. no_values now reaches the matching mask through _valid_mask, and is excluded from the valid-pixel count. It previously only filtered surviving key points and tinted the overview plot, so a product declaring nodata 0 while filled with 1 had a third of its features detected in empty fill and reported 99.93% valid pixels where the real overlap was 66.63%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fine cv2.calcOpticalFlowPyrLK uses a window of the same pixel size at every pyramid level, so at level L it spans winSize * 2**L of the original image. Near a data edge that window hangs off the image, the coarse estimate is meaningless, and it seeds every finer level, which cannot recover. Matching erodes inward from every edge as maxLevel grows, and the deep pyramid is exactly what a large displacement needs, so the two requirements fight each other. coarse_to_fine.py descends the pyramid explicitly instead. Every level runs with maxLevel=0, keeping the window one winSize wide at that level's own resolution, and what passes down is a RANSAC-affine field fitted to the level's reliable points rather than each point's own result, so a point near an edge inherits a usable guess from its neighbours. Only the finest level's winSize/2 band is lost, and maxLevel becomes free capture range. Measured against the default matcher on four scenes - SPOT5/BSG misregistered by 13.6 px, Landsat-8/Sentinel-2, MSS over terrain, and a 10980x10980 Sentinel-2 pair - it produced 20-100% more key points with better coverage at data edges and lower dispersion, for 1.4-1.6x the matching time and no extra memory. Improving count, coverage and dispersion together is the reason to believe it: they normally trade against each other. Exposed as a runtime switch next to --enable-large-shift-detection rather than a processing-config key, so it defaults off, cannot silently change a shipped default, and can be A/B'd per pair without editing config files. It is rejected with laplacian_kernel_size "auto", whose kernel search compares full resolution Laplacians and assumes single resolution matching. A local per-cell displacement field was tried and measured as worse: cells holding few reliable points emit noisy medians that mislead the next level. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The JSON shipped 5 while the dataclass defaulted to 3, so a CLI run and a library run with a config omitting the key behaved differently. On the SPOT5/BSG pair depth 3 yields 6794 key points against 1850 at depth 5, so 3 is also the better value of the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both remove a coarse displacement before fine matching - large shift by pre-shifting the monitored image and writing a shifted raster, coarse-to-fine by seeding the tracker from a smoothed field - so enabling both corrects it twice. Validated in RuntimeConfiguration rather than the CLI's _validate_configuration so library users are covered too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The auto kernel-size search filters the tile at every candidate size, then _laplacian_track_once filtered it twice more to build the dump for the winning pair. The search now returns the arrays it already computed, removing two full-tile Laplacian passes per tile. test_match_tile_uses_auto_ksize described this behaviour correctly all along; the code had drifted from it. test_match_tile_auto_ksize_selects_best_inlier_ratio asserted a monitored kernel size its mock never distinguished: the mock keyed only on the reference ksize, so every (mon, 7) pair scored identically and the winner was whichever the product order reached first. The mock now discriminates both axes, so the expected pair is genuinely the unique best. test_api_raster_and_vector_mask_mutually_exclusive built its raster mask without a projection, so spatial_ref was None and the compatibility check raised AttributeError before reaching what the test meant to verify. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Defaulting to 3 was wrong. It was chosen from a SPOT5/BSG pair misregistered by 13.6 px, where the deeper pyramid is needed to find the displacement at all, but that pair is not representative: for an already-aligned pair a deeper pyramid only erodes the edges. On the end-to-end LS9/Sentinel-2 scene, depth 3 returned 32932 key points against 37457 at depth 1, a 12% loss. Depth 1 restores the behaviour this project shipped. --enable-coarse-to-fine is the right answer for a misregistered pair: it buys capture range without paying for it at the edges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_api_raster_and_vector_mask_mutually_exclusive built its GeoJSON from UTM coordinates without declaring a CRS, so the GeoJSON driver assumed EPSG:4326 and PROJ rejected a latitude of 4599995. It only surfaced after the previous commit gave the raster mask a projection, which let the test reach the rasterization it was always meant to exercise, and only when another end-to-end module ran first. Declaring EPSG:32631 makes it hermetic. The end-to-end reference data predated the mutual information columns added to the CSV, so it carried 8 columns against the 10 now produced and could not pass on develop either. Regenerated with the committed configuration: 99.5% of the previous key points reproduce within 1e-4 px, zncc_score is identical, and the row count moves from 37448 to 37457, the nine extra coming from the tile margin fix recovering points at seams in this 4 tile configuration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Library users following the README literally (output_directory as a plain str) hit a TypeError in report generation, since some call sites join it with the `/` operator, which only works on pathlib.Path. Coerce it in __post_init__ so both str and Path inputs work consistently. Co-authored-by: Gautier Portet <gautier.portet@telespazio.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
A float raster could yield no key points at all. The float range was never the problem: measured on a Landsat-8 / Sentinel-2 crop, native uint16 and the same data rescaled to [0,1] both returned 3983 key points. Adding a single outlier pixel, or a single +Inf, returned none. The cause is that the stretch to 8 bit took its bounds from the array's minimum and maximum, so the two most extreme pixels decided the mapping for the whole scene and everything else collapsed into a handful of DN values. np.nanmax also carries +Inf straight through. Float products routinely hold such pixels where integer sensor data does not, which is why this looked like a float bug. Floats now take their bounds from the 2nd and 98th percentile of the finite pixels. Every failing case recovers: outlier, +Inf and a -9999 fill block all return ~3900 key points instead of zero. Integer rasters keep the previous minimum/maximum stretch, which leaves every existing result and the committed end-to-end reference data byte-identical; the percentile costs about 2% of key points on integer data and that switch deserves its own reviewed change. Two hypotheses were measured and rejected rather than implemented: normalising the reference and monitored images jointly was worse (3893 against 3908), and restricting the percentile to the valid mask made no difference. The same stretch existed in four diverging copies. They now share core.radiometry.to_uint8, which is global_align's already-correct percentile version. For coarse_to_fine this is a fix too, not just deduplication, since nanpercentile drops NaN but keeps infinities; it keeps its own tighter bounds, which is what the four-scene validation was run with. Also: --no-value accepts floats, as a float raster's fill value need not be a whole number; the low dynamic range warning no longer fires for every float image, having compared a [0,1] span against a threshold of 10; and the README no longer advises multiplying float data by 100, which is measurably a no-op. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes to the KLT matching path, kept together because they touch the same functions and no subset leaves _match_tile in a working state. Pyramid depth is now KLTConfiguration.maxLevel instead of a source constant. It had already drifted silently during development, and it is the parameter that most affects results: on a SPOT5/BSG pair, depth 3 yields 6794 key points against 1850 at depth 5, because a deeper pyramid scales the tracking window by 2**maxLevel and erodes matching inward from every data edge. Tiles now read a margin of real neighbouring pixels, sized from that depth, so key points at a tile edge keep full window support. Tiles did not overlap, so the band was lost at every internal seam: a 300px image in 100px tiles kept 21 of the 60 seam key points it finds untiled. Synthetic padding was measured and rejected - reflected or replicated content does not move consistently between the two images, so an LK window overlapping it scores worse than a truncated one. Only real pixels help, and at the outer image edge none exist. no_values now reaches the matching mask through _valid_mask, and is excluded from the valid-pixel count. It previously only filtered surviving key points and tinted the overview plot, so a product declaring nodata 0 while filled with 1 had a third of its features detected in empty fill and reported 99.93% valid pixels where the real overlap was 66.63%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fine cv2.calcOpticalFlowPyrLK uses a window of the same pixel size at every pyramid level, so at level L it spans winSize * 2**L of the original image. Near a data edge that window hangs off the image, the coarse estimate is meaningless, and it seeds every finer level, which cannot recover. Matching erodes inward from every edge as maxLevel grows, and the deep pyramid is exactly what a large displacement needs, so the two requirements fight each other. coarse_to_fine.py descends the pyramid explicitly instead. Every level runs with maxLevel=0, keeping the window one winSize wide at that level's own resolution, and what passes down is a RANSAC-affine field fitted to the level's reliable points rather than each point's own result, so a point near an edge inherits a usable guess from its neighbours. Only the finest level's winSize/2 band is lost, and maxLevel becomes free capture range. Measured against the default matcher on four scenes - SPOT5/BSG misregistered by 13.6 px, Landsat-8/Sentinel-2, MSS over terrain, and a 10980x10980 Sentinel-2 pair - it produced 20-100% more key points with better coverage at data edges and lower dispersion, for 1.4-1.6x the matching time and no extra memory. Improving count, coverage and dispersion together is the reason to believe it: they normally trade against each other. Exposed as a runtime switch next to --enable-large-shift-detection rather than a processing-config key, so it defaults off, cannot silently change a shipped default, and can be A/B'd per pair without editing config files. It is rejected with laplacian_kernel_size "auto", whose kernel search compares full resolution Laplacians and assumes single resolution matching. A local per-cell displacement field was tried and measured as worse: cells holding few reliable points emit noisy medians that mislead the next level. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The JSON shipped 5 while the dataclass defaulted to 3, so a CLI run and a library run with a config omitting the key behaved differently. On the SPOT5/BSG pair depth 3 yields 6794 key points against 1850 at depth 5, so 3 is also the better value of the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both remove a coarse displacement before fine matching - large shift by pre-shifting the monitored image and writing a shifted raster, coarse-to-fine by seeding the tracker from a smoothed field - so enabling both corrects it twice. Validated in RuntimeConfiguration rather than the CLI's _validate_configuration so library users are covered too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The auto kernel-size search filters the tile at every candidate size, then _laplacian_track_once filtered it twice more to build the dump for the winning pair. The search now returns the arrays it already computed, removing two full-tile Laplacian passes per tile. test_match_tile_uses_auto_ksize described this behaviour correctly all along; the code had drifted from it. test_match_tile_auto_ksize_selects_best_inlier_ratio asserted a monitored kernel size its mock never distinguished: the mock keyed only on the reference ksize, so every (mon, 7) pair scored identically and the winner was whichever the product order reached first. The mock now discriminates both axes, so the expected pair is genuinely the unique best. test_api_raster_and_vector_mask_mutually_exclusive built its raster mask without a projection, so spatial_ref was None and the compatibility check raised AttributeError before reaching what the test meant to verify. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Defaulting to 3 was wrong. It was chosen from a SPOT5/BSG pair misregistered by 13.6 px, where the deeper pyramid is needed to find the displacement at all, but that pair is not representative: for an already-aligned pair a deeper pyramid only erodes the edges. On the end-to-end LS9/Sentinel-2 scene, depth 3 returned 32932 key points against 37457 at depth 1, a 12% loss. Depth 1 restores the behaviour this project shipped. --enable-coarse-to-fine is the right answer for a misregistered pair: it buys capture range without paying for it at the edges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_api_raster_and_vector_mask_mutually_exclusive built its GeoJSON from UTM coordinates without declaring a CRS, so the GeoJSON driver assumed EPSG:4326 and PROJ rejected a latitude of 4599995. It only surfaced after the previous commit gave the raster mask a projection, which let the test reach the rasterization it was always meant to exercise, and only when another end-to-end module ran first. Declaring EPSG:32631 makes it hermetic. The end-to-end reference data predated the mutual information columns added to the CSV, so it carried 8 columns against the 10 now produced and could not pass on develop either. Regenerated with the committed configuration: 99.5% of the previous key points reproduce within 1e-4 px, zncc_score is identical, and the row count moves from 37448 to 37457, the nine extra coming from the tile margin fix recovering points at seams in this 4 tile configuration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apply_global_alignment always rendered outputs (aligned mon, passthrough ref, ECC candidates, aligned mask) onto the reference image's pixel grid, so e.g. a 5m monitored image aligned to a 10m reference was written out at 10m, discarding its native detail. The output canvas now keeps the reference's footprint but is resampled at the monitored image's resolution instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merging develop into fix/float-input-uint8-conversion resolved klt.py, coarse_to_fine.py and api/config.py to develop's side, dropping `_stretch` and the `to_uint8` import while leaving calls to `_stretch` and `_to_uint8` in place. KLT matching raised NameError, 17 tests failed and test_klt_float_input.py could not be imported. Restore the three files from the branch tip 24e527b; none changed since the merge. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Axes.boxplot() only accepts `label` from matplotlib 3.9, while environment.yml pins 3.8, so every run with a DEM failed while generating the shift-by-altitude plots. Label the first median line instead, which works on both versions and keeps the legend entry. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When a DEM is given, the KLT_matcher_*.csv file now has an `alt` column with the DEM elevation at each key point (reference x0, y0). The DEM is loaded in match_images so each streamed chunk is written with the column; resuming from a CSV without it adds the column and rewrites the file. DEM plots reuse the column instead of sampling the DEM again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add DEM elevation in CSV output.