Any parallelization support opportunities?

[davids91]

Hey! I was wondering if parallelization could be possible?

Broadly #66 would support that on client side, because with the knowledge of byte offsets,
anything could be built up in parallel.

But despite that I was wondering how could multiple processes support serialization or deserialization here?

[OliverUv]

The decoder takes an immutable reference to the source data, so in principle it's definitely possible. Not sure what the ideal API would look like.

So in practice you could take subslices of your source data and hand them off to different decoders, using whichever method you wish (async tasks, standard threads, etc).

Since you must already know the lengths ahead of time either way, I think the easiest way to achieve this is with decoding an index of the serialized data. Which you could either send separately, or pre-pend to the data and use partial deserialization (as we touched on in #66 ).

For this I guess instead of

fn estimated_size(&self) -> Option<usize> { None };

we'd want something like

enum SizeHint {
    Exact(usize),
    Approximate(usize),
    None,
}

fn size_hint(&self) ->  { None };

Approximate sizes could be used for pre-allocation. Exact size the same, as well as aiding in parallelization, and None would be the default (current) behavior.

[OliverUv]

I don't think it'd be a good idea for bendy to try to select parallelization primitives for you. No way for it to know about each client's context and needs.

[davids91]

Not specifically looking for that, but rather some guidelines, and a method of getting exact byte positions and sizes for the serialized data.

Parallelization itself should be the responsibility of the library user.