Errors importing DEMs for GCD

[ICF-Massey]

The Problem

Failed to import DEM

Reproduction steps

1. Clicked on inputs - DEM surveys
2. Selected first DEM, which generated the first exception message below (Rangitikei_lower_20210131_DEM_NZVD2016).
3. Repeated step for second DEM, which generated second exception message below (Rangitikei_corridor_2017)

Exception message 1

TIFFAppendToStrip:Write error at scanline 7808
--- Stacktrace ---
at OSGeo.GDAL.Dataset.BuildOverviews(String resampling, Int32 overviewlist, IntPtr pOverviews, GDALProgressFuncDelegate callback, String callback_data)
at OSGeo.GDAL.Dataset.BuildOverviews(String resampling, Int32[] overviewlist, GDALProgressFuncDelegate callback, String callback_data)
at GCDConsoleLib.Raster.BuildPyramids(String method) in D:\Code\gcd\gcd\GCDConsoleLib\Raster.cs:line 554
at GCDConsoleLib.RasterOperators.BuildPyramids(FileInfo rInput) in D:\Code\gcd\gcd\GCDConsoleLib\RasterOperators\Interfaces.cs:line 1229
at GCDCore.RasterPyramidManager.PerformRasterPyramids(PyramidRasterTypes ePyramidRasterType, FileInfo rasterPath)
at GCDCore.UserInterface.SurveyLibrary.frmImportRaster.ProcessRaster()
at GCDCore.UserInterface.Project.TreeNodeTypes.DEMSurveysGroup.OnAdd(Object sender, EventArgs e)
--- Exception Data ---
ArcMap = 10.8.0.12790
GCD = 7.4.1.0
Windows: Microsoft Windows NT 6.2.9200.0
Date: 15/11/2022 8:08:37 pm

Exception message 2

---

EXCEPTION
Exception has been thrown by the target of an invocation.
--- Stacktrace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at GCDConsoleLib.RasterOperators.GenericRunWithOutput(Type generic, Type innerType, String sDesc, EventHandler1 progressHandler, Object[] args) in D:\Code\gcd\gcd\GCDConsoleLib\RasterOperators\Interfaces.cs:line 1293 at GCDConsoleLib.RasterOperators.BilinearResample(Raster rInput, FileInfo sOutputRaster, ExtentRectangle outputExtent, EventHandler1 progressHandler) in D:\Code\gcd\gcd\GCDConsoleLib\RasterOperators\Interfaces.cs:line 391
at GCDCore.UserInterface.SurveyLibrary.frmImportRaster.ProcessRaster()
at GCDCore.UserInterface.Project.TreeNodeTypes.DEMSurveysGroup.OnAdd(Object sender, EventArgs e)
--- Exception Data ---
ArcMap = 10.8.0.12790
GCD = 7.4.1.0------------------------------------------------
EXCEPTION
TIFFReadDirectory:Failed to read directory at offset 8
--- Stacktrace ---
at OSGeo.GDAL.Gdal.Open(String utf8_path, Access eAccess)
at GCDConsoleLib.Raster.Open(Boolean write) in D:\Code\gcd\gcd\GCDConsoleLib\Raster.cs:line 329
at GCDConsoleLib.Raster.Write[T](Int32 xOff, Int32 yOff, Int32 xSize, Int32 ySize, T[] buffer) in D:\Code\gcd\gcd\GCDConsoleLib\Raster.cs:line 688
at GCDConsoleLib.Internal.Operators.BilinearResample1.Run() in D:\Code\gcd\gcd\GCDConsoleLib\RasterOperators\Operators\BilinearResample.cs:line 79 at GCDConsoleLib.Internal.Operators.BilinearResample1.RunWithOutput() in D:\Code\gcd\gcd\GCDConsoleLib\RasterOperators\Operators\BilinearResample.cs:line 134TIFFWriteDirectoryTagData:IO error writing tag data
Windows: Microsoft Windows NT 6.2.9200.0
Date: 16/11/2022 3:21:33 pm

Paste the exception message you are getting from the app here. It really helps us.

## Anything else?

You can attach links to datasets:
https://www.dropbox.com/scl/fo/c87g8m90z58oojji665k3/h?dl=0&rlkey=pxukevj238qkyss1jo7w9c2k8 

All help and advice appreciated!
Thanks
Ian

[philipbaileynar]

This feels like your system is having trouble accessing the DEMs you are trying to import.

Make sure the files are on your local computer and not on a cloud service (such as Dropbox) or external USB drive.

[joewheaton]

@philipbaileynar I downloaded the DEM (and hillshade and such) and they are big (5 GB from DEM). Your speculation that this is not local could be something, but I don't think so. I don't have ArcGIS anymore, so I just tested this in the standalone. I downloaded to a local directory. The DEM and hillshade add fine into QGIS. However, they are dimensionally indivisible:

For me, I tried to let it resample and it just hung at 1% progress for 15 minutes.

It is possible that it will finish some day... but this is really not the best thing to do. The best thing is to have a dimensionally divisible raster. @philipbaileynar I tried this again, and it really doesn't seem to be progressing at all and is lost in la-la land on "Resampling". You can see the TIF file it is trying to write on disk and it is not getting any bigger. So something more is fishy here. The question is why? We have not seen this kind of problem before. Also, I am not reproducing the same problem @ICF-Massey reported. So that is frustrating.

@ICF-Massey, for you for work arounds and best practices:

1. With a dataset this large, use GCD Standalone for processing, then come back to ArcGIS GCD AddIn for loading and visualizing map outputs.
2. You really want to avoid resampling the raster if at all possible. This happens when you have a width and height that are indivisible by the cell resolution. See https://gcd.riverscapes.net/Concepts/data-preparation---best-practices/. The best thing to avoid this resampling step by when the raster is being produced (e.g. a TIN to Raster or initial interpolation from point cloud and/or break lines and bndy data) to explicitly specify the output extents of the raster in nice whole numbers (i.e. evenly divisible by your 1 m resolution). This may allow you to push past the problems above and get by.
3. You may also consider troubleshooting things with a much smaller clipped area. GCD can handle bigger datasets like this, but 32 Bit ArcGIS will often choke.

Not sure that these things will help or not.

@philipbaileynar I found another odd bug I'll report separately.