Compute pending worker shards on-the-fly

[philipp-fischer]

Currently, energon pre-computes all shard ranges that a worker will walk through. This list can be very large and consume memory unnecessarily.
By defining a function for compute the Nth shard for a worker on-the-fly, this precomputation becomes obsolete.

When the state is saved to disk, only the current index (number of shard ranges already processed) needs to be stored, and potentially a shuffling seed.

[philipp-fischer]

Is this done? @voegtlel

[voegtlel]

I don't think so. We're still creating the full slice-list when loading a dataset if I'm not mistaken. We'd need to replace that by a function to create the n-th slice to complete this.