I spent several days trying to make CMNET2 cooperate with my current setup, and I've got some suggestions. However, I'm afraid I have to start this comment with some context.
Context
So far, I've been using mostly my own apps (https://eriador.condak.cz/tools.php) written in C# (for the UI) and C++ (for the transformation utility). I keep color masks in separate files, which can use lower resolutions (usually half) to speed up calculations (MP4 compression drops half of the color information anyway). I also save the calculated transformation files (i.e., vector maps) so the app doesn't have to recalculate them every time I visit a different frame. To improve quality, the setup can also use depth maps created by Depth Anything 2. Additionally, to fix minor details, it is possible to create a "partial keyframe" with an alpha channel.
Unfortunately, my knowledge of Python is very limited, so I chose to use Gemini and Claude to customize your code.
The first thing I had them create was video_sequence_slide_mask.py, which works with sequences of grayscale images and separate color mask files of variable resolutions.
I wanted to be able to create new keyframe masks that would fit in with my existing ones and that I could edit. It turned out that the CMNET model could only output two channels (H+S is enough for colorization, but it makes it hard to distinguish the colors while editing), so I made it run some sections twice to get all three channels of the mask through.
Then I noticed I had to include the partial keyframes. The solution was to take the generated color mask, overlay it with the partial mask, and add the result to the set of reference images. However, in this scenario, it was necessary to prevent the original generated image from being added to the temporary set because they would clash. It was also necessary to change the behavior of the sliding window because the partial keyframes created too many reference images, causing the app to start swapping VRAM.
I managed to make it all work somehow, but the code was already quite bloated (maybe +50 kB). =}
Then I could finally start the colorization.
Findings
- I noticed that the colors of the generated frames were slightly pulsating. It wasn't much, but the slight change in global hue (maybe around 5 hue points) was noticeable, and the period of this "pulse" seemed to be 5 or 10 frames.
After some fiddling, I found out I had to change this:
'mem_every': 1, # was 5
'deep_update_every': 1, # was -1
- I am not sure how much it affected the speed (I didn't measure the exact time), but it definitely improved the color consistency of the results, and I prefer quality over speed.
-
Because of the partial keyframes, I took a closer look at their neighbors, and I noticed that the output is heavily affected by the context of the previously calculated frames. This led to the creation of the --direction forward|backward parameter. Some shots really benefited from the backward direction (especially when people come closer to the camera during the shot).
-
So, what about --direction both? Running it twice is rather easy, but the question is how to mix the results. Using a 50:50 blend or the distance to the closest keyframe could work, but I thought a better way would be using the calculated internal confidence of the prediction. I made CMNET export it separately, save it directly to the images as an alpha channel, and log the overall confidence of the frame. In the third pass, the script mixes frames from both runs based on their alpha channels. The improvement isn't huge, but it is definitely noticeable.
-
While looking at the confidence logs, I noticed that the global confidence of the keyframes was not much different from that of the other frames. It seems that all frames in the original CMNET2 are generated the same way. I suppose it might be fine if the reference files are truly just a reference or if there are only a few of them (otherwise, the only frame with exact colors could stick out a bit). However, if you want them to act as true keyframes, you have to make an exception for them and use the exact colors from the file for that specific frame. The calculated frame also should not be added to the temporary set. (Claude messed up the fix yesterday too, probably due to the combination of full and partial keyframes. I'll give it another shot later today.)
If you are interested, I could share the full code, but I'm afraid it is rather bloated because of the changes I made in the first stage... =}
I spent several days trying to make CMNET2 cooperate with my current setup, and I've got some suggestions. However, I'm afraid I have to start this comment with some context.
Context
So far, I've been using mostly my own apps (https://eriador.condak.cz/tools.php) written in C# (for the UI) and C++ (for the transformation utility). I keep color masks in separate files, which can use lower resolutions (usually half) to speed up calculations (MP4 compression drops half of the color information anyway). I also save the calculated transformation files (i.e., vector maps) so the app doesn't have to recalculate them every time I visit a different frame. To improve quality, the setup can also use depth maps created by Depth Anything 2. Additionally, to fix minor details, it is possible to create a "partial keyframe" with an alpha channel.
Unfortunately, my knowledge of Python is very limited, so I chose to use Gemini and Claude to customize your code.
The first thing I had them create was video_sequence_slide_mask.py, which works with sequences of grayscale images and separate color mask files of variable resolutions.
I wanted to be able to create new keyframe masks that would fit in with my existing ones and that I could edit. It turned out that the CMNET model could only output two channels (H+S is enough for colorization, but it makes it hard to distinguish the colors while editing), so I made it run some sections twice to get all three channels of the mask through.
Then I noticed I had to include the partial keyframes. The solution was to take the generated color mask, overlay it with the partial mask, and add the result to the set of reference images. However, in this scenario, it was necessary to prevent the original generated image from being added to the temporary set because they would clash. It was also necessary to change the behavior of the sliding window because the partial keyframes created too many reference images, causing the app to start swapping VRAM.
I managed to make it all work somehow, but the code was already quite bloated (maybe +50 kB). =}
Then I could finally start the colorization.
Findings
After some fiddling, I found out I had to change this:
Because of the partial keyframes, I took a closer look at their neighbors, and I noticed that the output is heavily affected by the context of the previously calculated frames. This led to the creation of the --direction forward|backward parameter. Some shots really benefited from the backward direction (especially when people come closer to the camera during the shot).
So, what about --direction both? Running it twice is rather easy, but the question is how to mix the results. Using a 50:50 blend or the distance to the closest keyframe could work, but I thought a better way would be using the calculated internal confidence of the prediction. I made CMNET export it separately, save it directly to the images as an alpha channel, and log the overall confidence of the frame. In the third pass, the script mixes frames from both runs based on their alpha channels. The improvement isn't huge, but it is definitely noticeable.
While looking at the confidence logs, I noticed that the global confidence of the keyframes was not much different from that of the other frames. It seems that all frames in the original CMNET2 are generated the same way. I suppose it might be fine if the reference files are truly just a reference or if there are only a few of them (otherwise, the only frame with exact colors could stick out a bit). However, if you want them to act as true keyframes, you have to make an exception for them and use the exact colors from the file for that specific frame. The calculated frame also should not be added to the temporary set. (Claude messed up the fix yesterday too, probably due to the combination of full and partial keyframes. I'll give it another shot later today.)
If you are interested, I could share the full code, but I'm afraid it is rather bloated because of the changes I made in the first stage... =}