feat: Open Data Catalog explorer with server-side clip by prbatero · Pull Request #94 · microsoft/haste

Anthony Cintron (prbatero) · 2026-07-23T00:16:10Z

Add an Open Data Catalog side panel to the Create Image Layer page so analysts can browse open disaster imagery (Vantor/Maxar + Planet Open Data), preview it, and build an image layer without hunting for COG URLs.

Catalog UI (ui/src/Components/OpenDataCatalog/):

Dynamic discovery + merge of every event across the Vantor S3 STAC and Planet Source-Cooperative STAC catalogs; framework-free module with a discoverEvents()/fetchEventCatalog() contract (swap-to-backend seam).
Azure Maps footprints + TiTiler COG preview (works with no Azure Maps key); preview tile loading indicator.
Add a scene to pre/post in one click with source-type + capture-date auto-fill; buttons are phase-scoped (pre->Pre, post->Post).
Two-way map<->list selection: click a footprint to scroll/highlight/expand the list row; footprints stay clickable during preview.

Server-side clip AOI:

Draw a box that sets a layer-level clipBbox ([w,s,e,n] EPSG:4326); imagery prep clips the pre/post mosaics via gdalwarp -te/-te_srs (GDAL reprojects the bounds). The catalog filters scenes overlapping the AOI and badges scenes that fully cover it, so pre/post share one AOI.
Model ImageLayer.clipBbox + validate_clip_bbox; PutLayer validates/carries it; threaded through the imagery-prep config -> ImageryWorkflow -> mosaic_imagery. Verified end-to-end (output bounds match the drawn box).

Supporting fixes:

Allow data.source.coop on the imagery allowlist (UI + hastelib) and route non-S3/Blob allowlisted hosts through the generic HTTP downloader, so Planet imagery downloads at prep time.
Local dev proxy (docker/nginx.conf): TiTiler timeouts + client_max_body_size for chunk uploads.
QUICKSTART: DOCKER_GID must be 0 on macOS Docker Desktop.

Azure Maps auth is unchanged (existing anonymous/AAD Client-ID flow).

Spec: spec/features/open-data-catalog/.

Add an Open Data Catalog side panel to the Create Image Layer page so analysts can browse open disaster imagery (Vantor/Maxar + Planet Open Data), preview it, and build an image layer without hunting for COG URLs. Catalog UI (ui/src/Components/OpenDataCatalog/): - Dynamic discovery + merge of every event across the Vantor S3 STAC and Planet Source-Cooperative STAC catalogs; framework-free module with a discoverEvents()/fetchEventCatalog() contract (swap-to-backend seam). - Azure Maps footprints + TiTiler COG preview (works with no Azure Maps key); preview tile loading indicator. - Add a scene to pre/post in one click with source-type + capture-date auto-fill; buttons are phase-scoped (pre->Pre, post->Post). - Two-way map<->list selection: click a footprint to scroll/highlight/expand the list row; footprints stay clickable during preview. Server-side clip AOI: - Draw a box that sets a layer-level clipBbox ([w,s,e,n] EPSG:4326); imagery prep clips the pre/post mosaics via gdalwarp -te/-te_srs (GDAL reprojects the bounds). The catalog filters scenes overlapping the AOI and badges scenes that fully cover it, so pre/post share one AOI. - Model ImageLayer.clipBbox + validate_clip_bbox; PutLayer validates/carries it; threaded through the imagery-prep config -> ImageryWorkflow -> mosaic_imagery. Verified end-to-end (output bounds match the drawn box). Supporting fixes: - Allow data.source.coop on the imagery allowlist (UI + hastelib) and route non-S3/Blob allowlisted hosts through the generic HTTP downloader, so Planet imagery downloads at prep time. - Local dev proxy (docker/nginx.conf): TiTiler timeouts + client_max_body_size for chunk uploads. - QUICKSTART: DOCKER_GID must be 0 on macOS Docker Desktop. Azure Maps auth is unchanged (existing anonymous/AAD Client-ID flow). Spec: spec/features/open-data-catalog/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds an in-app Open Data Catalog explorer on the Create Image Layer flow, letting analysts discover/preview open disaster imagery (Vantor/Maxar + Planet Source Cooperative) and build image layers faster, including an optional server-side clip AOI applied during imagery prep.

Changes:

Adds new Open Data Catalog UI (panel + list + Azure Maps footprints + TiTiler preview) and wires it into the Create Image Layer form (including scene-to-pre/post “add” + auto-fill).
Introduces ImageLayer.clipBbox end-to-end (UI → PutLayer validation → imagery-prep config → GDAL mosaic clip).
Expands imagery URL allowlist to include data.source.coop and routes non-S3/Blob allowlisted URLs through the generic HTTP downloader; updates local dev proxy settings/docs.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
ui/src/util/validation.js	Allowlists `data.source.coop` imagery hosts in UI URL validation.
ui/src/Components/OpenDataCatalog/SceneListItem.jsx	New scene row UI with metadata, AOI badge, and phase-scoped add buttons.
ui/src/Components/OpenDataCatalog/OpenDataCatalogPanel.jsx	New catalog side panel: discovery, filtering, list↔map selection sync, clip toolbar.
ui/src/Components/OpenDataCatalog/OpenDataCatalogMap.jsx	New Azure Maps footprint rendering + TiTiler preview + rectangle clip drawing.
ui/src/Components/OpenDataCatalog/openDataCatalog.js	Framework-free STAC discovery/normalization + TiTiler URL helpers + bbox overlap helpers.
ui/src/Components/CreateEditImageLayerHelper.js	Adds `clipBbox` to form state + helper to add catalog scenes into imagery URL arrays.
ui/src/Components/CreateEditImageLayerForm.jsx	Adds “Browse Open Data Catalog” entry point; threads `clipBbox` into PutLayer payload.
spec/features/open-data-catalog/README.md	Feature spec overview and decision log for catalog + server-side clip.
spec/features/open-data-catalog/design.md	Detailed design: discovery/preview/clip flow and API contract changes.
spec/features/open-data-catalog/data-model.md	Documents `clipBbox` schema change and allowlist additions.
spec/features/open-data-catalog/user-stories.md	User stories and acceptance criteria for catalog and clip workflows.
spec/features/open-data-catalog/test-plan.md	Test strategy/coverage matrix for allowlist + clip + manual flows.
spec/features/open-data-catalog/plan.md	Implementation plan/status across core/API/UI phases.
spec/features/open-data-catalog/impact-analysis.md	Impact/risk analysis across UI/API/workflow/security considerations.
spec/features/open-data-catalog/rollout.md	Rollout plan, prerequisites, monitoring, rollback notes.
QUICKSTART.md	Documents macOS Docker Desktop `DOCKER_GID` behavior (set to 0).
hastelib/tests/core/utils/test_url_allowlist.py	Adds unit tests for `data.source.coop` allowlist + `validate_clip_bbox`.
hastelib/src/hastegeo/workflows/prepare_imagery.py	Threads `clip_bbox` into workflow and mosaic creation.
hastelib/src/hastegeo/core/utils/url_allowlist.py	Adds `source.coop` allowlist and `validate_clip_bbox` validator.
hastelib/src/hastegeo/core/utils/imagery.py	Clips mosaics during GDAL warp when `clip_bbox` is set.
hastelib/src/hastegeo/core/utils/downloader.py	Routes non-S3/Blob allowlisted imagery hosts to generic HTTP downloader.
hastelib/src/hastegeo/core/processors/imagery.py	Adds `clip_bbox` into imagery-prep config payload.
hastelib/src/hastegeo/core/models/projects.py	Adds `ImageLayer.clipBbox` field to model.
docker/nginx.conf	Increases body size and TiTiler proxy timeouts for local dev.
docker/docker-compose.yml	Adds `CLEANUP_CONTAINERS` env var to aid debugging spawned containers.
api/hastefuncapi/function_app.py	Validates `clipBbox` in PutLayer (400 on invalid bbox).

- SceneListItem.formatGsd: guard non-finite input so a non-numeric gsd renders as absent instead of "NaN m GSD". - SceneListItem.formatSize: treat only null/undefined/"" as absent so a legitimate 0-byte asset formats as "0 B" (was hidden by a falsy check). - Imagery allowlist host description (ui/util/validation.js and hastelib url_allowlist.py): say Source Cooperative "*.source.coop" to match what the validator actually allows (endsWith ".source.coop"), avoiding misleading error messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…h-img-prep

github-actions · 2026-07-24T19:58:11Z

RC artifacts ready

All branch deployment references use the same RC tag:

hastegeo_version: 1.0.27rc2
training_image_tag: 1.0.27rc2
imageprep_image_tag: 1.0.27rc2
wheel: hastegeo-1.0.27rc2-py3-none-any.whl

The Open Data Catalog fetches the Vantor (S3) and Planet (Source Cooperative) STAC catalogs + thumbnails directly from the browser. The deployed Static Web Apps CSP only allowed atlas/openstreetmap/blob hosts, so connect-src (and img-src) blocked those requests in production ("Refused to connect ... violates the document's Content Security Policy"). Add the two exact hosts to connect-src and img-src in ui/public/staticwebapp.config.json (and document them in docs/security-configuration.md): - https://vantor-opendata.s3.amazonaws.com - https://data.source.coop (Longer term, moving discovery behind a same-origin backend proxy would keep connect-src at 'self'.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Meygha Machado (mgmachado)

LGTM

github-actions · 2026-07-24T21:04:42Z

RC artifacts ready

All branch deployment references use the same RC tag:

hastegeo_version: 1.0.27rc3
training_image_tag: 1.0.27rc3
imageprep_image_tag: 1.0.27rc3
wheel: hastegeo-1.0.27rc3-py3-none-any.whl

The COG preview built tile URLs as /cog/tiles/{z}/{x}/{y}.png, a legacy TiTiler alias that works on the local titiler but 404s through the deployed TiTiler/APIM route. Switch to the canonical /cog/tiles/WebMercatorQuad/{z}/{x}/{y}?scale=1 path that the rest of HASTE already uses (labels.py, function_app.py) and that APIM serves in production. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-25T02:09:16Z

RC artifacts ready

All branch deployment references use the same RC tag:

hastegeo_version: 1.0.27rc4
training_image_tag: 1.0.27rc4
imageprep_image_tag: 1.0.27rc4
wheel: hastegeo-1.0.27rc4-py3-none-any.whl

… dump Pool ids were wired through GitHub Environment *variables*, but this repo is public and Actions logs are public with it. GitHub masks secret values in logs and does not mask variables, and `az functionapp config appsettings set` was printing the resulting settings JSON to stdout, so the values would have landed in a public log. Store them as environment secrets instead, matching how RESOURCE_PREFIX, LOCATION, RESOURCE_SUFFIX and STATIC_APP_DOMAIN are already handled, and add `--output none` so the app-settings dump is not emitted at all. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dd98b384-7e26-47a9-8c03-6f9594eb611c

github-actions · 2026-07-27T14:46:01Z

RC artifacts ready

All branch deployment references use the same RC tag:

hastegeo_version: 1.0.27rc8
training_image_tag: 1.0.27rc8
imageprep_image_tag: 1.0.27rc8
wheel: hastegeo-1.0.27rc8-py3-none-any.whl

Caleb Robinson (calebrob6) · 2026-07-28T03:54:45Z

RC artifacts ready

All branch deployment references use the same RC tag:

hastegeo_version: 1.0.27rc8

training_image_tag: 1.0.27rc8

imageprep_image_tag: 1.0.27rc8

wheel: hastegeo-1.0.27rc8-py3-none-any.whl

Meygha Machado (@mgmachado) Anthony Cintron (@prbatero) -- nice touch!!

Anthony Cintron (prbatero) requested review from Copilot and Meygha Machado (mgmachado) July 23, 2026 00:16

Anthony Cintron (prbatero) requested a review from a team as a code owner July 23, 2026 00:16

Copilot started reviewing on behalf of Anthony Cintron (prbatero) July 23, 2026 00:16 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread ui/src/Components/OpenDataCatalog/SceneListItem.jsx

Comment thread ui/src/Components/OpenDataCatalog/SceneListItem.jsx

Comment thread ui/src/util/validation.js

Comment thread hastelib/src/hastegeo/core/utils/url_allowlist.py

Copilot started work on behalf of Anthony Cintron (prbatero) July 24, 2026 16:17 View session

Copilot stopped work on behalf of Anthony Cintron (prbatero) due to an error July 24, 2026 16:18
Failed to fetch job details: HTTP 500 Internal Server Error

Anthony Cintron (prbatero) temporarily deployed to dev1 July 24, 2026 17:56 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/main' into feat-data-catalog-wit…

d3f3f79

…h-img-prep

Joaquín Rivero (jQuinRivero) temporarily deployed to dev1 July 24, 2026 19:59 — with GitHub Actions Inactive

Meygha Machado (mgmachado) previously approved these changes Jul 24, 2026

View reviewed changes

Anthony Cintron (prbatero) had a problem deploying to dev1 July 24, 2026 20:55 — with GitHub Actions Error

Anthony Cintron (prbatero) temporarily deployed to dev1 July 24, 2026 21:08 — with GitHub Actions Inactive

Anthony Cintron (prbatero) dismissed Meygha Machado (mgmachado)’s stale review via ba5d141 July 25, 2026 01:40

Anthony Cintron (prbatero) temporarily deployed to dev1 July 25, 2026 02:15 — with GitHub Actions Inactive

Anthony Cintron (prbatero) temporarily deployed to dev1 July 25, 2026 11:37 — with GitHub Actions Inactive

Joaquín Rivero (jQuinRivero) and others added 2 commits July 27, 2026 11:03

Merge branch 'main' into feat-data-catalog-with-img-prep

b45f41e

Microsoft (microsoft) deleted a comment from github-actions Bot Jul 27, 2026

Joaquín Rivero (jQuinRivero) temporarily deployed to dev1 July 27, 2026 15:08 — with GitHub Actions Inactive

Anthony Cintron (prbatero) requested a review from Joaquín Rivero (jQuinRivero) July 27, 2026 18:14

Joaquín Rivero (jQuinRivero) approved these changes Jul 27, 2026

View reviewed changes

Joaquín Rivero (jQuinRivero) mentioned this pull request Jul 28, 2026

fix(batch): scope Batch job ids to the selected pool #99

Merged

Anthony Cintron (prbatero) merged commit 05a3094 into main Jul 28, 2026
14 checks passed

Copilot finished work on behalf of Anthony Cintron (prbatero) August 5, 2026 18:29

Navigation Menu

Uh oh!

feat: Open Data Catalog explorer with server-side clip - #94

feat: Open Data Catalog explorer with server-side clip#94
Anthony Cintron (prbatero) merged 7 commits into
mainfrom
feat-data-catalog-with-img-prep

Anthony Cintron (prbatero) commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Meygha Machado (mgmachado) left a comment

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 25, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Caleb Robinson (calebrob6) commented Jul 28, 2026

RC artifacts ready

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Anthony Cintron (prbatero) commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026

RC artifacts ready

Uh oh!

Meygha Machado (mgmachado) left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 24, 2026

RC artifacts ready

Uh oh!

github-actions Bot commented Jul 25, 2026

RC artifacts ready

Uh oh!

github-actions Bot commented Jul 27, 2026

RC artifacts ready

Uh oh!

Caleb Robinson (calebrob6) commented Jul 28, 2026

RC artifacts ready

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants