About the pretrained weights loading problem in training

[llstela]

While loading UNet2DConditionModel from pretrained SD-2, ValueError appears, as parameter weights related to image_attentions are missed.

Traceback (most recent call last):
File "train_seesr.py", line 680, in
unet = UNet2DConditionModel.from_pretrained(
File "/opt/conda/envs/seesr/lib/python3.8/site-packages/diffusers/models/modeling_utils.py", line 646, in from_pretrained
raise ValueError(
ValueError: Cannot load <class 'models.unet_2d_condition.UNet2DConditionModel'> from preset/models/stable-diffusion-2-base
because the following keys are missing:
down_blocks.1.image_attentions.0.transformer_blocks.0.ff.net.2.bias, up_blocks.3.image_attentions.0.norm.weight, down_blocks.1.image_attentions.1.transformer_blocks.0.attn1.to_out.0.weight, ...

[Zebraside]

I have the same problem. It seems that the problem arises when use_image_cross_attention=True in UNet2DConditionModel.from_pretrained. It is not used in some inference scripts.

Can someone clarify is it a mandatory setting for training?

[llstela]

I have the same problem. It seems that the problem arises when use_image_cross_attention=True in UNet2DConditionModel.from_pretrained. It is not used in some inference scripts.

Can someone clarify is it a mandatory setting for training?

I tried to overwrite the "UNet2DConditionModel.from_pretrained" function. It can work, but I'm not sure it's the right way to solve this problem.

[Renzhihan]

May I ask if you get a mismatch between the output SR image and the input LR image at the beginning of the training?
Meanwhile, I specified the use_image_cross_attention=True, --pretrained_model_name_or_path, and --unet_model_name_or_path, but I didn't catch the error.
@llstela @Zebraside

[Zebraside]

The error occurs when only --pretrained_model_name_or_path is set. (# resume from pretrained SD branch).
I'm using stable-diffusion-2-base from HuggingFace as base model.

[1343464520]

i came across the same error. what's the finnal solution? @Zebraside @Renzhihan @llstela @cswry

[1343464520]

i came across the same error. what's the finnal solution? @Zebraside @Renzhihan @llstela @cswry

set use_image_cross_attention=False，the error was solved

[JL6666JL]

set low_cpu_mem_usage=False