image brightness is wrong

[Manoa1911]

is it supposed to be 2x brighter with nofix ?

python test_sd2b_controlnet.py ^
--pretrained_model_name_or_path "X:\SD2-base\stable-diffusion-2-base" ^
--controlnet_model_name_or_path "S:\c-sd2\model.safetensors" ^
--image_path "X:\input4" ^
--output_dir "x4_base_door_sd2b" ^
--align_method "nofix" ^
--ram_path "X:\models\ram_swin_large_14m.pth" ^
--dape_path "X:\models\DAPE.pth" ^
--guidance_scale 3.5 ^
--num_inference_steps 50 ^
--mixed_precision "fp16" --upscale 4

[Manoa1911]

a possible solution to the problem:

from PIL import Image, ImageMath
.
.
.
        validation_image = Image.open(str(image_path)).convert("RGB")
        validation_image.load()
        rA, gA, bA = validation_image.split()
        rTmp = ImageMath.eval("int(a/((float(b)+1)/2) -1)", a=rA, b=rB).convert('L')
        gTmp = ImageMath.eval("int(a/((float(b)+1)/2) -1)", a=gA, b=gB).convert('L')
        bTmp = ImageMath.eval("int(a/((float(b)+1)/2) -1)", a=bA, b=bB).convert('L')
        validation_image = Image.merge("RGB", (rTmp, gTmp, bTmp))

this solution will not work, because the problem is in the output stage, it cannot be controlled through reduction of the input :(
most likely the solution would look like this:
WPy64-31190b5\python-3.11.9.amd64\Lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet.py
at the end of this file contains a line that looks like this:
image = self.image_processor.postprocess(image, output_type=output_type, do_denormalize=do_denormalize)
change this line to output_type="pt"
all this is only made possible, that while not explicitly documented, diffusers==0.33 still does support output_type="pt" in this line, this solves the problem of having to output a numpy array and then having to work on that array by hand in order to convert it into a (correct) torch tensor :)

then do these modifcations:

import os
os.environ["OPENCV_IO_ENABLE_OPENEXR"]="1"
import cv2
import imageio

            image_result = image_result.detach().cpu()

            if args.align_method == 'wavelet':
                output_image = wavelet_color_fix(image_result, validation_image)
            elif args.align_method == 'adain':
                output_image = adain_color_fix(image_result, validation_image)

            output_image = torch.squeeze(output_image,dim=0)
            output_image = output_image.permute(1, 2, 0)
            output_image = torch.clip(output_image, 0, 1)
            image_array = output_image.numpy().astype(np.float32)
            pil_array = image_array * 65535
            save_path = os.path.join(args.output_dir, f"{basename}.exr")
            pil_save_path = os.path.join(args.output_dir, f"{basename}.png")
            imageio.v3.imwrite(save_path, image_array)
            bgr = cv2.cvtColor(pil_array.astype(np.uint16), cv2.COLOR_RGB2BGR)
            cv2.imwrite(pil_save_path, bgr)

            #if args.align_method == 'nofix':
            #    final_image = image_result
            #else:
            #    if args.align_method == 'wavelet':
            #        final_image = wavelet_color_fix(image_result, validation_image)
            #    elif args.align_method == 'adain':
            #        final_image = adain_color_fix(image_result, validation_image)
            #    else:
            #        final_image = image_result
            
            #if resize_flag:
            #    final_image = final_image.resize((ori_width * rscale, ori_height * rscale))
            

            #out_path = os.path.join(args.output_dir, f"{basename}.png")
            #image_result.save(out_path)

it is recommended to perform color corrections in photoshop or GIMP or even davinci resolve, the python methods are unlikely to be powerful enough with the align method, saving to EXR allow lossless re-editing without degradations due to rounding errors and 16 bit PNG side-saving allows to see the image (you cannot see EXR images directly) :)

strange, in your previous releases this code was present, at least the upper part of it

it turns out this project does not support "nofix", it relies heavily on these 2 color correcting functions, what I did here is managed to increase the quality of those functions, note that the source image is still passed as PIL object not a 32-bit floating object, the important part the target which is the upscaled image does get passed as floating point object, this should preserve significant quality on output images, and when combined with 32-bit float output in EXR for additional re-processing as well as 16-bit constants for preview, this maximizes the image handling aspect of quality of this project :)

in the future it may be worth working on performing adain and wavelet simultaneously, these 2 color corrections are card-indepedent and thus would be able to run without waiting in a single thread, which slows down card occupancy :(

one point of discussion: should clipping be done before or after adain/wavelet ?

[Manoa1911]

in addition, the wavelet_color_fix.py file could use some improvements as well: these 2 function take a Image object and convert it into a tensor perform the work and then convert back to an Image object ? that's a waste. instead, combined with the above modifications causing the return from the pipeline as torch tensor, the object can be sent to these 2 functions in it's (mostly) original high-precision state, and with these modifications below allow the color correction process to be higher quality:

def adain_color_fix(target: Tensor, source: Tensor):
    # Convert images to tensors
    to_tensor = ToTensor()
    target_tensor = target.unsqueeze(0)
    source_tensor = to_tensor(source).unsqueeze(0)

    # Apply adaptive instance normalization
    result_tensor = adaptive_instance_normalization(target_tensor, source_tensor)

    # Convert tensor back to image
    #to_image = ToPILImage()
    #result_image = to_image(result_tensor.squeeze(0).clamp_(0.0, 1.0))

    return result_tensor

def wavelet_color_fix(target: Tensor, source: Tensor):
    # Convert images to tensors
    to_tensor = ToTensor()
    target_tensor = target_tensor.unsqueeze(0)
    source_tensor = to_tensor(source).unsqueeze(0)

    # Apply wavelet reconstruction
    result_tensor = wavelet_reconstruction(target_tensor, source_tensor)

    # Convert tensor back to image
    #to_image = ToPILImage()
    #result_image = to_image(result_tensor.squeeze(0).clamp_(0.0, 1.0))

    return result_tensor

I know it might be possible to convert the source into something more worthwhile, but I suspect that would have littel effect if any, because the source image does not get corrupted by various to/from PIL convertions and these functions already do what is needed :)

[Manoa1911]

one more I believe important change that should be worth to take into consideration:

#ori_width, ori_height = validation_image.size
        #resize_flag = False
        #rscale = args.upscale
        #if ori_width < args.process_size // rscale or ori_height < args.process_size // rscale:
        #    scale = (args.process_size // rscale) / min(ori_width, ori_height)
        #    tmp_image = validation_image.resize((int(scale * ori_width), int(scale * ori_height)))
        #    validation_image = tmp_image
        #    resize_flag = True

        validation_image = validation_image.resize((validation_image.size[0] * args.upscale, validation_image.size[1] * args.upscale))

        #validation_image = validation_image.resize((validation_image.size[0] // 8 * 8, validation_image.size[1] // 8 * 8))
        #width, height = validation_image.size
        #validation_image_tensor = (torch.from_numpy(np.array(validation_image)).permute(2, 0, 1).float() / 255.0) * 2 - 1
        #validation_image_tensor = (torch.from_numpy(np.array(validation_image)).permute(2, 0, 1).float() / 255.0)
        #validation_image_tensor = validation_image_tensor.unsqueeze(0)
        #resize_flag = True

this code change can help prevent adaptive decisions and upscale all images by the same factor, also the transformations prevent additional quality loss, same factor scaling is good when you are planning to run the program 4 times with a total of x2 x4 x6 and x8 factors, if you want to run only once on all images and let adaptivity here work, then removing this code is not recommended :(