DVC GDrive remote unavailable - moving to Hugging Face Hub

[cassiotbatista]

Apparently things gotta be managed by Google Cloud now. Need some hours to work on this. Until then, files hosted on Google Drive are inaccessible.

[cassiotbatista]

See also: official docs: https://dvc.org/doc/user-guide/data-management/remote-storage/google-drive

[cassiotbatista]

As the blobs are still available through GDrive's web interface, it is possible to manually download the files, and update the DVC remote path to a local dir. The catch is, GDrive seems to omit some of the file extensions, namely the *.wav ones, so running dvc pull on a local dir yields errors for most -- if not all -- audio files.

The trick was to copy the audio files to the same dir, but with no extension, before running dvc pull. That way I could retrieve all files!

-> % PAGER= git diff -- .dvc/
diff --git a/.dvc/config b/.dvc/config
index 087d7e5..60e6ebf 100644
--- a/.dvc/config
+++ b/.dvc/config
@@ -1,4 +1,4 @@
 ['remote "public"']
     url = gdrive://1s6fHvd9JhzU4hff98KnIA0v0_Tdd6qlT
 ['remote "align"']
-    url = gdrive://1NiH0G_Ocem3owbIquxUmbVtvGGX3kiPt
+    url = /tmp/asd

-> % cat /tmp/asd/eita.sh
for wav in $(find . -name \*.wav | sort) ; do
  cp $wav ${wav%.wav}
done

[cassiotbatista]

Some folks have reported that the process of registering an auth key on GCloud does not solve the problem fully, and some pain points remain, like re-auth being required after a few days, and every single user having to create their own credentials through a not-so-easy-as-email procedure.

That's unfortunate bc DVC is a nice tool, but as I'm already paying from my own pocket for Google Drive storage, I'm not willing to set up another paid registry back end (e.g. AWS S3).

Therefore, I made the decision to move away from DVC and host all data on Hugging Face datasets: https://huggingface.co/falabrasil/datasets

The process has just been started, so it's a bit far away from being finished. Some problems remain while I fight with the server's CI/CD to exec my scripts, so downloading only the test subset from a specific dataset remains impossible for now, for example. This repo on GitHub should be kept as a centralized reference, tho. I also should write some tutorial if people want to fetch data from the soon-legacy GDrive registry by tweaking the remote after downloading the blobs from the browser (as poorly described above).