Errors when POSTing to `/api/raster/collections/{id}/items/{id}/statistics` with large geojson body

[jsignell]

When you POST to /api/raster/collections/{id}/items/{id}/statistics with a small geojson containing a Polygon it works just fine:

echo '{"type": "Feature", "properties": {"ADMIN": "France", "ISO_A3": "FRA"}, "geometry": {"type": "Polygon", "coordinates": [[[-5.183429, 42.332925], [8.233998, 42.332925], [8.233998, 51.066135], [-5.183429, 51.066135], [-5.183429, 42.332925]]]}}' > simple.json
curl -H "Content-Type: application/json" --data @simple.json https://openveda.cloud/api/raster/collections/no2-monthly/items/OMI_trno2_0.10x0.10_202307_Col3_V4.nc/statistics

Response:

{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-5.183429,42.332925],[8.233998,42.332925],[8.233998,51.066135],[-5.183429,51.066135],[-5.183429,42.332925]]]},"properties":{"statistics":{"cog_default_b1":{"min":-1556922155139072.0,"max":5875976969912320.0,"mean":908950033553932.1,"count":11880.0,"sum":1.0798326398620713e+19,"std":662287275047051.9,"median":790465041399808.0,"majority":322760819605504.0,"minority":-1556922155139072.0,"unique":11879.0,"histogram":[[16,298,4477,5071,1360,478,142,37,0,1],[-1556922155139072.0,-813632229212160.0,-70342303285248.0,672947555532800.0,1416237548568576.0,2159527541604352.0,2902817266204672.0,3646107527675904.0,4389397252276224.0,5132686708441088.0,5875976969912320.0]],"valid_percent":100.0,"masked_pixels":0.0,"valid_pixels":11880.0,"percentile_2":-156033125187584.0,"percentile_98":2714252632653824.0}},"ADMIN":"France","ISO_A3":"FRA"}}

But when you use a bigger geojson in the body (this one is a MultiPolygon) it hangs:

curl https://geodata.ucdavis.edu/gadm/gadm4.1/json/gadm41_FRA_0.json | jq ".features[0]" > complex.json
curl -H "Content-Type: application/json" --data @complex.json https://openveda.cloud/api/raster/collections/no2-monthly/items/OMI_trno2_0.10x0.10_202307_Col3_V4.nc/statistics

Context

I ran into this issue while running https://github.com/NASA-IMPACT/veda-docs/blob/main/user-guide/notebooks/quickstarts/timeseries-stac-api.ipynb in the VEDA docs for NASA-IMPACT/veda-docs#235

[anayeaye]

Note: This is a pretty big geojson https://geodata.ucdavis.edu/gadm/gadm4.1/json/gadm41_FRA_0.json and might be getting blocked by the WAF? I do not see any recent blocked statistics requests.

Just in case it helps with the investigation I am dropping these links: We encountered this in the past with a very complicated Alaska geojson NASA-IMPACT/veda-ui#923 (comment). At that time the compromise was an adjustment to allow posting much larger geojson but we also reduced the size of the original problem file NASA-IMPACT/veda-ui#928 (comment).

[jsignell]

Yeah this isn't a new notebook so it did work at some point unless the geojson at that endpoint changed I guess. If it is being blocked by WAF we should try to do some kind of response though right even if it's just to say: too big! You need to simplify that shape!

[anayeaye]

We were not able to come up with a good solution for these cloudfront errors in the past but we can continue to brainstorm--our problem here is that the request is stopped at the firewall (which I suspect it is) so we cannot actually configure a too-big-for-the-firewall response in the backend STAC API. If we do find that the issue you documented here is a blocked request we can move this issue to veda-routes and explore custom cloudfront blocked request responses.

[botanical]

Thanks for surfacing @jsignell! It looks like your request was blocked by our firewall:

I can consider some changes we can make either in backend or routes to surface these blocked requests.

[jsignell]

Thanks for tracking that down @botanical! Yeah it would be great to try to figure out a way to surface an error to the user, but I get that might not be easy. Maybe this is something that can be configured directly in WAF? Or maybe WAF could send oversized requests to a different service whose only job is to say: too big!

In addition to this error handling I can imagine it being potentially useful to have a supported source of admin boundaries. For instance the large geojson in this issue represents "France" so if there could be an API where instead of POSTing a massive geojson I just post "France" then that would be a way to support this functionality. Of course I know I could also simplify the shape to get it small enough, but it seems like it would maybe not be too hard to accept admin names as an alternative. Then the geojson fetch could happen server-side or within titiler itself. Cartopy uses https://www.naturalearthdata.com/features/ for instance so we could just do that as well.

[wildintellect]

I made a suggestion in the related ticket, use https://github.com/wmgeolab/geoBoundaries/raw/9469f09/releaseData/gbOpen/FRA/ADM0/geoBoundaries-FRA-ADM0_simplified.geojson

[wildintellect]

The issue with simpler geometries is that it can really shift the results of the statistics depending on the source resolution of the raster. Caching NaturalEarth is much more complicated than it sounds and adds a lot of complexity to the API.

[jsignell]

Caching NaturalEarth is much more complicated than it sounds and adds a lot of complexity to the API.

Agreed. This would need a ton more discussion and people interested in such a feature.

I made a suggestion in the related ticket, use https://github.com/wmgeolab/geoBoundaries/raw/9469f09/releaseData/gbOpen/FRA/ADM0/geoBoundaries-FRA-ADM0_simplified.geojson

Yeah that doesn't look too good on first glance and I don't think it's small enough. Seems to be shifted:

Before I saw that comment I had been working on simplifying in the notebook. This seems like it does the trick:

france = shapely.from_geojson(json.dumps(france_aoi))
france_simplified = shapely.set_precision(
    france
        .buffer(0.1).buffer(-0.1)
        .simplify(0.01), 
    0.0001
)
france_geojson = shapely.to_geojson(france_simplified)

But it does round the coastline and ends up including some water:

[stephenkilbourn]

AWS has an article on handling oversize HTTP requests in AWS WAF:

Short description

When a web request body exceeds the body inspection size limit, the underlying host service forwards only contents that are within the limit to AWS WAF. AWS WAF doesn't detect cross-site scripting (XSS) attacks or SQL injection patterns in the web request body beyond the limit.

Note: AWS WAF doesn't log contents of the web request body whether or not it finds an attack pattern within the body inspection size limit.

To protect against attack patterns that don't appear within the body inspection size limit for oversize requests, configure a rule that blocks all oversize requests. Then, create a rule that explicitly allows legitimate oversize requests. You can either create a custom rule, or use the core rule set (CRS) managed rule group.

https://repost.aws/knowledge-center/waf-http-request-body-inspection

[sandrahoang686]

Follow-up:
Followed instructions @stephenkilbourn shared with waf-http-request-body-inspection link and created a Custom Rule (AllowLegitimateRequestsOver8KB) in UAH/dev that would allow requests to bypass SizeRestrictions_BODY if legitimate for veda-dev-webacl protection pack.

1. Updated SizeRestrictions_BODY in AWS-AWSManagedRulesCommonRuleSet from Block to Count
2. Created Custom Rule (named AllowLegitimateRequestsOver8KB) with the following:

CURL post was able to go through successfully 👍🏼

[anayeaye]

@sandrahoang686 sorry to be late to the thread here

1. AWS only inspects up to 64 KB of the body--so that's the cutoff we want to make. The MultiPolygon triggering this oversize request block is 268 KB. I think our existing WAF rules already have configuration to allow up to 64 KB but we probably should confirm that the rules are not exempting requests from inspection. (AWS Source).

2. In cases when the firewall is correctly blocking a too-large-to-inspect request, we should be informing the user why their request was blocked so that they can take another approach if extremely high resolution geometry is needed for sampling a raster. Currently, however, our CloudFront setup just unhelpfully redirects the user to the stac browser page. The redirect solved a different problem but is not an acceptable long term solution.

3. The solution to this too-large-request body error is to properly pass raster-api errors all the way back to the user--there was suggestion to adjust how we are deploying our stac-browser vue app so that we could remove the redirect and inform users of the API error message here

[sandrahoang686]

Tag-up update: Maybe looking for an ideal solution of doing this validation server side as a follow-up but possibly continue with WAF solution for now

[anayeaye]

@sandrahoang686 @botanical I just confirmed that the WAF changes work for dev (sorry I doubted that it would work!). If you want to explore that custom error message feel free to modify