Questions about the datasets

[jasmine-994]

Hello, thank you for your great work. I'm interesting in reproducing your work and while I was reading your paper I couldn't find which size of the LSDIR and FFHQ dataset you used during your training. Did you use the 512x512 dataset for the LSDIR and the 128x128 dataset for FFHQ or the HR datasets?

And one other thing I would like to ask is about the paths you used in your READ.ME. Specifically the below:

"We use LSDIR (84991) and the first 10k images from FFHQ as our training data. Please write their absolute paths into a txt file. Each line in the txt file should be the absolute path of an image. Just like:

LSDIR/HR_merge/0000001.png
LSDIR/HR_merge/0000002.png
...
"

Does this mean that for the Real-World restoration training you used both datasets or only the LSDIR dataset?

Thank you in advance

[cswry]

Hi, thanks for your interest!

For training, the GT images are randomly cropped to 512×512, and the LQ images are degraded to 128×128, then upsampled to match the GT size before being fed into the network. This applies to both LSDIR and the first 10k images from FFHQ, as mentioned in the paper.

So yes, we used both LSDIR and FFHQ (10k images) for real-world isr training.

Hope this helps!

[jasmine-994]

Hello, thank you for your response!

I just want to clarify one more thing. When I try to train with the LQ datasets (meaning the 512x512 dataset from LDSIR or the 128x128 dataset from FFHQ) I get a message that says that the resolution is very low and it only accepts images of 512x512. And that because some images from the 512x512 LSDIR dataset aren't exactly 512x512. Some are a bit smaller. This is why I asked if you used the HR images as input.

export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:32
  CUDA_VISIBLE_DEVICES="0" accelerate launch train_osediff.py \
    --pretrained_model_name_or_path=/workspace-1/dipl/OSEDiff/models/stable-diffusion-2-1-base \
    --ram_path=/workspace-1/dipl/OSEDiff/models/RAM/ram_swin_large_14m.pth \
    --learning_rate 5e-5 \
    --train_batch_size=1 \
    --gradient_accumulation_steps=1 \
    --gradient_checkpointing \
    --enable_xformers_memory_efficient_attention --checkpointing_steps 500 \
    --mixed_precision 'fp16' \
    --report_to "tensorboard" \
    --seed 123 \
    --output_dir=experience/osediff \
    --dataset_txt_paths_list '/workspace-1/dipl/train_dataset_small.txt' \
    --dataset_prob_paths_list 1 \
    --neg_prompt="painting, oil painting, illustration, drawing, art, sketch, cartoon, CG Style, 3D render, unreal engine, blurring, dirty, messy, worst quality, low quality, frames, watermark, signature, jpeg artifacts, deformed, lowres, over-smooth" \
    --cfg_vsd=4 \
    --lora_rank=4 \
    --lambda_lpips=2 \
    --lambda_l2=1 \
    --lambda_vsd=1 \
    --lambda_vsd_lora=1 \
    --deg_file_path "params_realesrgan.yml" \
    --tracker_project_name "train_osediff-test" \
    --dataloader_num_workers 8

File "/venv/osediff/lib/python3.10/site-packages/torchvision/transforms/transforms.py", line 681, in forward
i, j, h, w = self.get_params(img, self.size)
File "/venv/osediff/lib/python3.10/site-packages/torchvision/transforms/transforms.py", line 640, in get_params
raise ValueError(f"Required crop size {(th, tw)} is larger than input image size {(h, w)}")
ValueError: Required crop size (512, 512) is larger than input image size (318, 618)

Cause when I feed the HR images it works properly but I had a feeling this is not what the paper proposes.