How is positional embedding added here ?

[bhosalems]

How exactly is spatial correspondence maintained? I see that the masks are fed through cross attention mechanism in Unet layers, but I do not see any positional embedding taken. how is this handled or am I missing something?

[marcoaversa]

Hi @bhosalems, the work has been run a long time ago were ViT visual embedders weren't already used as often as now. I agree with you ViT embedding with positional embedding would enhance the performance on the conditioning. In order to perform cross-attention, I downsampled the masks and passed them through a convolutional layer, without extracting embedding tokens per patch. It's a variation of the self-attention layer of the U-net proposed. Each mask is binary and the class information is embedded with the time, and this is why the similarity distance between QK works in that scenario from the spatial point of view. I agree with you that with latest advancements in the field, the work could be improved way further. Hope this helps in your implementation. I'd say that the most important element in this work is the random patch sampling method proposed, expanded for text-to-image generation in our latest work (https://github.com/Thanos-DB/Pixelsmith). Hope this helps.

[bhosalems]

thanks that clarifies a lot of things, my intention for comparison is not high-resolution image synthesis (although, that is more practically relevant), but conditional generation. As such without access to the private dataset used in this method it is unclear how to preprocess data and/or construct the dataset/dataloader. I think I have enough details to implement this on the publicly available dataset, I will keep you posted if I get into any trouble.