In Data_Downloader.py, we currently use urllib.requesturlopen, but this seems to fail regularly on large downloads. There may be a way to implement retry or use a more reliable / resilient protocol.
We could also re-host these files on S3 (depending on licensing, etc) so we can control the hosting quality.
In
Data_Downloader.py, we currently use urllib.requesturlopen, but this seems to fail regularly on large downloads. There may be a way to implement retry or use a more reliable / resilient protocol.We could also re-host these files on S3 (depending on licensing, etc) so we can control the hosting quality.