Implement energon prepare with a remote dataset (using local temp)

[voegtlel]

Fixes #91

Implements energon prepare for remote (including media metadata preparation)

[radulescupetru]

I see ValueError: Path msc://default/path_to_s3_dataset is not local error

Preparing filesystem dataset and computing media metadata...
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/conda_env/bin/energon", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/megatron/energon/tools/prepare_media.py", line 110, in command
    stored = prepare_filesystem_dataset(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/megatron/energon/media/filesystem_prepare.py", line 44, in prepare_filesystem_dataset
    root = root_path.local_path()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/conda_env/lib/python3.12/site-packages/megatron/energon/epathlib/epath.py", line 258, in local_path
    raise ValueError(f"Path {self} is not local")

There's also a TODO in that method:

Only supporting local file system, because sqlite does not support remote file systems.

TODO: Implement remote file systems. Maybe create locally in tmp then upload?

[voegtlel]

@radulescupetru sorry for taking a while, had other priorities. Now also implemented that for filesystem:// links. Can you try again?

[philipp-fischer]

Either us or MSC team needs to fix this for local paths. This is way slower than os.walk. Maybe for now we should have os.walk here explicitly for local paths. In MSC they do os.listdir + sorting + isdir/isdile plus object construction for each file, i.e. lots of overhead.
But let's make sure we still preserve DSS URLs when using os.walk.

[philipp-fischer]

This will evaluate to True for DSS caches, because they are local. Can we maybe catch the read-only case somewhere early to get a useful error? (also for the other prepare entry points)