postcard-dyn: reserialize from arbitrary Deserializer into arbitrary Serializer

[max-heller]

Implements a postcard-dyn "reserializer" that deserializes data from an arbitrary Deserializer and feeds it into an arbitrary Serializer in a streaming fashion (without needing to construct an intermediate representation in memory) based on a postcard schema. This means postcard -> serde_json::Value (or whatever generic representation postcard-dyn ends up providing), postcard <-> json, and arbitrary other postcard <-> X transformations can be performed by the same generic implementation.

Unfortunately, Deserializer and Serializer methods require &'static strs for structs/variants/field names. To provide these, the lossless implementation allocates and leaks one copy of each unique string that must be used as &'static. I can't find a way around this that isn't lossy and this seems inevitable in order to provide a Serializeable postcard_dyn::Value. A separate, lossy transformation is also provided that does not leak memory but only supports json-like representations of structs/enums as maps.

[netlify]

✅ Deploy Preview for cute-starship-2d9c9b canceled.

Name	Link
🔨 Latest commit	41bdb42
🔍 Latest deploy log	https://app.netlify.com/sites/cute-starship-2d9c9b/deploys/67ad50f71009960008a9776a

[jamesmunns]

Ouch, yeah, that first limitiation. I don't really love that in the "always" case, do you think this would make sense as an alternate interface in this crate instead of replacing the existing machinery?

[max-heller]

Ouch, yeah, that first limitiation. I don't really love that in the "always" case, do you think this would make sense as an alternate interface in this crate instead of replacing the existing machinery?

Yes, probably, and it should be doable to provide both interfaces backed by a single implementation with a parameter selecting either "lossy, non-leaky transformation" or "lossless, leaky" transformation.

However, what do we expect users to do with a Value that doesn't involve Serializeing it in the end (which runs into this limitation)? I'm worried that most transformations on the generic form will end up running into this limitation as well. With the existing interface it isn't a problem because serde_json converts structs and enums to maps, but I'm assuming that isn't the end-goal.

[max-heller]

One other option: we could expose an unsafe interface for serializers that are known not to keep struct/field/variant names around for longer than the serializer lives (I believe this is the case for most serializers) that transmutes the lifetimes of those strings to 'static during serialization instead of leaking them.

[max-heller]

Pushed an update that avoids leaking for the from_slice_dyn() case and makes it easier to implement different workarounds for the &'static str limitation. Lots of boilerplate but hopefully reasonably straightforward

[max-heller]

Some more context on the &'static str limitation and what it may take to lift it in serde itself (TLDR: unlikely to ever happen): serde-rs/serde#2218 (comment)