When setting refiner=base_image_refiner, the refine_steps argument promises to control how many denoising steps will be performed with the refiner:

However, in practice, this is only true if you additional set prompt_strength=1.0. The reason is that:
cog-sdxl sets num_inference_steps=refiner_steps here, and
diffusers computes the init_step for refinement based both on num_inference_steps and strength here.
This results in surprising behavior, where you set refiner_steps=N only to see that it ran for fewer steps. I think you could either:
- Update your documentation to reflect this reality, or
- Automatically set
strength=1 here