Training not working.

[fabvio]

Hello, thanks for your work. I tried following the tutorial, but when I download the weights using git lfs from the official huggingface repo, there is no config.json associated. I think it may be due to the diffusers version being outdated for the huggingface space.

Could you please provide more details about how to retrieve the correct weights for stable diffusion to be used in your project?

Thanks in advance.

[weijiawu]

Thank you for your attention. However, I can't recall the specific version of SD. To facilitate your usage, I have uploaded the corresponding SD weights used in my experiments to Google Drive (around 4.5GB) as following:

https://drive.google.com/file/d/12lrOexljsyvFB30-ltbYXnIpQ8oP4lrW/view?usp=sharing

I hope this helps you.

[fabvio]

Hi, you need to publicly share the file otherwise everyone will need to ask you for authorization. Thanks!

[fabvio]

It still does not train. It loads correctly the Stable Diffusion with the files you provided. Now I get this error when I try to train on Cityscapes data.

***********************   begin   **********************************
{'variance_type'} was not found in config. Values will be initialized to default values.
learning_rate: 0.0001
Epoch 0/5000
(512, 512)
  0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "tools/train_semantic_cityscapes.py", line 472, in <module>
    main()
  File "tools/train_semantic_cityscapes.py", line 396, in main
    images_here, x_t = ptp_utils.text2image(unet,vae,tokenizer,text_encoder,noise_scheduler, prompts, controller, latent=start_code, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=5, generator=g_cpu, low_resource=LOW_RESOURCE, Train=True)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 214, in text2image
    image = latent2image(vae, latents)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 90, in latent2image
    image = vae.decode(latents)['sample']
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 552, in decode
    dec = self.decoder(z)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 202, in forward
    sample = up_block(sample)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/unet_blocks.py", line 1212, in forward
    hidden_states = resnet(hidden_states, temb=None)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/resnet.py", line 336, in forward
    hidden_states = self.norm1(hidden_states.float()).type(hidden_states.dtype)
AttributeError: 'tuple' object has no attribute 'float'

Any idea of the cause? Thanks in advance.

[weimzh]

It still does not train. It loads correctly the Stable Diffusion with the files you provided. Now I get this error when I try to train on Cityscapes data.

***********************   begin   **********************************
{'variance_type'} was not found in config. Values will be initialized to default values.
learning_rate: 0.0001
Epoch 0/5000
(512, 512)
  0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "tools/train_semantic_cityscapes.py", line 472, in <module>
    main()
  File "tools/train_semantic_cityscapes.py", line 396, in main
    images_here, x_t = ptp_utils.text2image(unet,vae,tokenizer,text_encoder,noise_scheduler, prompts, controller, latent=start_code, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=5, generator=g_cpu, low_resource=LOW_RESOURCE, Train=True)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 214, in text2image
    image = latent2image(vae, latents)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/ptp_utils.py", line 90, in latent2image
    image = vae.decode(latents)['sample']
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 552, in decode
    dec = self.decoder(z)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/vae.py", line 202, in forward
    sample = up_block(sample)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/unet_blocks.py", line 1212, in forward
    hidden_states = resnet(hidden_states, temb=None)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/homes/55/fabio/miniconda3/envs/DatasetDM/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/scratch/local/projects/stephane/baselines/DatasetDM/model/diffusers/models/resnet.py", line 336, in forward
    hidden_states = self.norm1(hidden_states.float()).type(hidden_states.dtype)
AttributeError: 'tuple' object has no attribute 'float'

Any idea of the cause? Thanks in advance.

have you solved this issue? I've got the same problem here.

thanks