SourceError: Request outside of bounds

[kylebarron]

We're getting errors of SourceError: Request outside of bounds. This is an apparent regression from #1666. It sounds like

• feat: limits reads to the maximum size of the source #1666 checks for the Content-Range header to infer the size of a request, falling back to Content-Length
• In a browser, Content-Range is only readable if the server whitelists it in CORS settings. Various sources, such as some upstream S3 buckets, do not. And I don't think we can always rely on it being able to be accessed.
• So in the browser, metadata.size gets set to the size of one chunk instead of the entire size of the file.
• SourceChunk.fetch then throws Request outside of bounds for any header-source read past byte ~1 MB

Ref developmentseed/stac-map#460 and developmentseed/deck.gl-raster#524

cc @gadomski

[kylebarron]

I implemented a simple workaround to just set size to Infinity:
developmentseed/deck.gl-raster#527

I don't think there's any actual downside to doing that, because COG tile ranges are known from the metadata

[blacha]

@kylebarron sorry for my slow response here,

Do you have a view on what is the appropriate way to fix this? Because most (all?) of the testing happens in nodejs the joys of CORS is ignored in the current test suite.

I am guessing that, if a http server responds with 206, then content-length cannot be trusted, and it must use content-range, and if content-range is not present you dont actually know the size of the file.

so a more strict approach is something like:

  const contentRange = response.headers.get('content-range');
  if (contentRange != null) {
    metadata.size = ContentRange.parseSize(contentRange);
  } else if (response.status == 200) {
    metadata.size = Number(contentLength);
    if (Number.isNaN(metadata.size)) throw new Error('Invalid content-length')
  }

[kylebarron]

I don't see why you need to check against the length of the file at all? Unless the TIF itself is malformed, then any range request that starts within a valid byte range, will always be satisfiable.

[blacha]

I dont believe it to be useful for tiff files..

It becomes more useful when using negative offsets (eg zip) then trying to block align from the start of the file, eg fetching the last 64kb may also be the first 64kb, so if you cache the last 64kb as a block aligned chunk it can be re-used when fetching data from the start of the file.

[kylebarron]

I see; I didn't realize you were using these utilities for anything other than TIFF reading.

I am guessing that, if a http server responds with 206, then content-length cannot be trusted, and it must use content-range, and if content-range is not present you dont actually know the size of the file.

That sounds reasonable