missing support for #[serde(flatten)] for derive AvroSchema

[elad-yosifon]

I tried to play around with the proc-macro, and was able to make this test pass:

#[test]
    fn avro_flatten_fields() {
        #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
        struct Nested {
            a: bool,
        }


        #[derive(Debug, Serialize, Deserialize, AvroSchema, Clone, PartialEq)]
        struct Foo {
            #[serde(flatten)]
            nested: Nested,
            b: i32,
        }

        let schema = r#"
        {
            "type":"record",
            "name":"Foo",
            "fields": [
                {
                    "name":"a1",
                    "type":"boolean"
                },
                {
                    "name":"b",
                    "type":"int"
                }
            ]
        }
        "#;

        let schema = Schema::parse_str(schema).unwrap();
        let derived_schema = Foo::get_schema();
        if let Schema::Record(RecordSchema { name, fields, .. }) = &derived_schema {
            assert_eq!("Foo", name.fullname(None));
            for field in fields {
                match field.name.as_str() {
                    "a" | "b" => {
                        // expected
                    },
                    name @ _ => {
                        panic!("Unexpected field name '{name}'")
                    }
                }
            }
        } else {
            panic!(
                "Foo schema must be a record schema: {derived_schema:?}"
            )
        }
        assert_eq!(schema, derived_schema);

       // deserializing not support
       // serde_assert(Foo {
       //     nested: Nested { a: true },
       //     b: 321,
       // });
    }

but while running the other tests, it broke the serde(rename) related tests since the schema code genrator changed in order to allow nested fields to be pushed, and the tests verify the code generation.

I am having a hard time implementing this correctly mainly due to my lack of familiarity with the project.

[martin-g]

Supporting #[serde(...)] has never been the aim of avro_derive.
I think you are the first Avro user who explores this.

I am having a hard time implementing this correctly mainly due to my lack of familiarity with the project.

Same here but with the serde library :-)

My problem with serde is that it is being abandoned / not maintained by his author for the same reasons we face here - many serde attributes contradict with each other and it got out of control... Especially when one uses Rust enums with tagging!

apache/avro#1921 is another case where we faced issues with avro_derive + serde + enums.

[Sufflope]

Hello, I need serde(flatten) support too 😄 Here is my take on how to support it both for schema derivation, and de/serialization! #311

[Sufflope]

Hi, while the PR is pending, since I need this support soon, I published a sort of plugin to piggy-back as much as possible on apache-avro while supporting #[serde(flatten)]. Of course once we finish this, my crate is bound to be yanked 😄

[Theodlz]

Hey! Same issue here, I lack the flatten support for a project but also, it seems that macros like rename are not considered, is that right? For example I have a field called a in my struct with a #[serde(rename = "a_renamed")] on top. Naturally serde serialized this as a_renamed but the schema generated by apache-avro-derive calls the field a as in the sturct -> you run into an error.

[Theodlz]

Just to add to this: that problem isn't just present in apache-avro-derive, but the writer itself doesn't seem to handle flatten fields? I wrote a valid schema manually and tested it, and as soon as it encountered a struct with flatten fields, it breaks with the error: Failed to serialize value of type map using schema Union...

[martin-g]

and as soon as it encountered a struct with flatten fields

Only apache-avro-derive reads attributes via proc macros.
There is no logic in apache-avro that could possibly help you with attribute like #[serde(...)].
The above means: it is your responsibility to provide an Avro schema that could be used to serialize a given Rust struct!
This is what apache-avro-derive does - it reads the attributes (#[avro(...)] and some #[serde(...)] ones) to generate an Avro schema that could be used for your Rust struct.

[Theodlz]

Gotcha. However it's a bit unfortunate that even if I provide a valid schema - that accounts for the flattening - it can't be used because the flatten macro isn't accounted for by the writer. This means that a given struct can't necessarily be converted to an avro schema that matches the structure followed by other serde-friendly crates, i.e. the user needs to edit their structs to reach a specific output schema.

Just to explain my use case a bit:

1. I read in an avro packet with a LOT of fields as a struct
2. I compute a bunch of values from that struct's fields, and I create a new struct that contains the original one (with a serde flatten) + the new fields I computed
3. I want to put that out as an avro that looks like the original + my added fields.

Serde's flatten macro is particularly elegant when one wants to "augment" data without writing a gazillion lines of code redefining fields that already exist in another struct. Anyways I just wanted to explain why we rely on it. But if there is no plan to support that serde macro, then we'll most likely have to just redefine these large structs in multiple places to avoid the flattens (where applicable).

(@petebachant tagging you here so you know about this issue)

[martin-g]

Just to explain my use case a bit:

You could share your use case in a PR with a failing unit test(s).
I am allergic to serde.rs but someone else may pick it up!

[Kriskras99]

I'm happy to review PRs adding support for serde(flatten) in apache-avro-derive.
I think there are a few requirements:

• It should only require changes in the derive code
• It needs to work with Writer::append_ser, to_value + Writer::append, and from_value
• It needs round-trip tests for the three combinations

[Sufflope]

From what I remember in my initial digging, your first requirement is not possible. Basically the way serde flatten works is that your data suddenly takes the SerializeMap path instead of SeralizeStruct, so one must adapt your Serializer impl to be ready for when your Record schema has Map values. I tried initially doing this then gave up and tried my little attribute marker quick win 😄

I will not be able to work on this until january though, but happy to retry if it's still not resolved then.

[Theodlz]

Hey folks,

For a project where we have some tight deadlines, I started from @Sufflope 's repo and made all the quick and dirty changes I needed for it to work in my project ASAP (big thanks for making that temporary crate, it helped a ton!!!). I fixed a few things that were missing in there (support for flatten when it's not at the top level, support for the rename macro in both structs and enums). I likely won't find the time to contribute upstream before the second half of December but please do feel free to salvage whatever seems useful (here's a pointer to the PR that adds these things: https://gitlab.com/sufflope/serdavro/-/issues/3)