[question] from motion model --> frame synthesis model -> video

[eisneim]

Hi @fltwr thanks for sharing such fantastic work!
test.ipynb shows how to use the

• Motion model which use VAE + U-Net diffusion to produce frame spectrum
• Frame Synthesis model which use Softmax Splatting to produce video from Optical Flow
• the missing part is to demonstrate how to convert spectrum to optical flow

i use the function spec_to_flow() form "utils.py" get a iFFT transformed optical flow, but the flow is flicking and generated frame is not moving at all, would you be kind enough to point out where i did wrong?

openCV remap() generates not so interesting results

512px-001_2015_04_19_Das_Gelb_der_Natur.jpg_ddpm100_2024-07-01T10_37_25.146594.mp4

spec_to_flow() get flickering optical flow

flicker_flow_from_spectrum.mp4

synthesis video is not moving

flicker_flow_no_moving.mp4

Thanks!

[eisneim]

Ops! my bad, looking in to utils.py there are helper functions

and this works for me

    spec_np = generate_spectrum(vae, unet, noise_scheduler, frame, num_steps=num_steps, batch_size=1)
    
    d = np.clip(np.median(np.abs(spec_np), axis=(1, 2)) - .05, a_min=0, a_max=None)
    d = np.minimum(np.abs(spec_np), d[:, None, None, :]) * np.sign(spec_np)
    _, flow_convered = transforms.deprocess_spec(torch.from_numpy(spec_np - d).permute(0, 3, 1, 2).unsqueeze(0))
    flow_convered = flow_convered[0]
    flow_convered[..., 0] *= magnification
    flow_convered[..., 1] *= magnification * frame_np.shape[0] / frame_np.shape[1]
    
    flow_rec = torch.tensor(flow_convered, dtype=torch.float32).permute(0, 3, 1, 2).to(DEVICE)
    frame_for_synth = transformsFlow.process_frame(frame_np).unsqueeze(0).to(DEVICE)
    pred = predict_tensor(frame_for_synth, flow_rec, model, transformsFlow, return_tensor=False)
    # pred = softsplat_tensor(frame_for_synth, flow_rec * 2, transformsFlow, return_tensor=False)
    
    video = mpy.ImageSequenceClip(list(pred), fps=24)
    vpath = os.path.join(out_dir, sample_name + ".mp4")
    video.write_videofile(vpath, logger=None)

is there any info about training data? what dataset, how many samples? google's paper can animate water and many other things, but this model does not, is it because of the limit of training data?

[fltwr]

Hi, I trained the model on 818 short videos. The contents of the videos are limited to plants moving in the wind. Here are the first frames of 64 randomly selected training videos:

• The model was trained on 150-frame sequences. One or more sequences were taken from each video at 30 fps. I used 7218 sequences from the 818 videos for training.
• I collected the videos from YouTube by searching plant names, places ("mountain", "meadow"...), seasons, "wind", etc.

[eisneim]

@fltwr wow that's why the motion capability is so limited, i would like to collect more training data, and share here;

my goal is to create a tool that can automaticly animate images to get cinemagraph animations, which animates water, smokes, cloth, hair; for water i need other models that can do liquid fluid simulation

here are some cinemagraph examples:

thanks again for sharing this great work

[fltwr]

That sounds great!

[eisneim]

progress update: 1600 videos, anything that can oscillate should be trainable since we can decompose it's motion into frequency domain.

i try to find footage that is shot using tripod, not sure if hand hold camera footage with slight camera movement is ok. i guess is should be ok since the camera movement adds a lowest frequency to all pixels, might have little effect on model's performance

[SimonSMH1015]

Hello, @fltwr.
I have a similar problem. As you mentioned, the current content of the videos is limited to plants moving in the wind. I'm wondering if it's possible to further use this model to animate animals.
Thank you.

[eisneim]

@SimonSMH1015 animal moving head, walking, etc. is not possible, this method only learns to create small oscillate motions; animal breath, blink eye, wind blowing fur, those animations should be work

[fyushan]

progress update: 1600 videos, anything that can oscillate should be trainable since we can decompose it's motion into frequency domain.

i try to find footage that is shot using tripod, not sure if hand hold camera footage with slight camera movement is ok. i guess is should be ok since the camera movement adds a lowest frequency to all pixels, might have little effect on model's performance

How do you generate ground truth optical flow? Will inaccurate ground truth optical flow affect performance? How can this be optimized?

[eisneim]

use pyflow to generate optical flow, it ok to be not fully accurate, since the second model softmaxSplating (in this case) will learn to translate the generated optical flow to video,

it's also possible if you use latent embedings instead of optical flow and use GAN to generate video, the key of this paper is learning motion systhesis from frequency domain

[eisneim]

here is the 2000 training video i collected, still doing the motion model training and it now able to animate more items

baiduyun: 链接: https://pan.baidu.com/s/1zAebMc7rA3bXs1TkoOq7-g?pwd=ic71 提取码: ic71

[fyushan]

here is the 2000 training video i collected, still doing the motion model training and it now able to animate more items

baiduyun: 链接: https://pan.baidu.com/s/1zAebMc7rA3bXs1TkoOq7-g?pwd=ic71 提取码: ic71

That is very generous of you, thank you for sharing.

Have you ever implemented the frequency attention layers described in the paper?
In Fig3, the operations (Spatial layer+reshape+Frequency Attentio+reshape) inside the red box are only applied to the UNet features and have nothing to do with the spectral volume S, right?

It seems that the model predicts the spectral volume for each single frequency. I don't understand how the attention mechanism is applied between multiple frequencies during training.

[eisneim]

@fyushan not yet, I'll try if the final results are still bad compare to the original paper

[whiteany]

@fyushan not yet, I'll try if the final results are still bad compare to the original paper

Hello, could you please tell me if the paper "Generative Image Dynamics" has not yet released their code? I couldn't find their code.

[fyushan]

@fyushan not yet, I'll try if the final results are still bad compare to the original paper

Hello, could you please tell me if the paper "Generative Image Dynamics" has not yet released their code? I couldn't find their code.

You are right, their code has not been released.

[Unified-Robots]

Ops! my bad, looking in to utils.py there are helper functions

and this works for me

    spec_np = generate_spectrum(vae, unet, noise_scheduler, frame, num_steps=num_steps, batch_size=1)
    
    d = np.clip(np.median(np.abs(spec_np), axis=(1, 2)) - .05, a_min=0, a_max=None)
    d = np.minimum(np.abs(spec_np), d[:, None, None, :]) * np.sign(spec_np)
    _, flow_convered = transforms.deprocess_spec(torch.from_numpy(spec_np - d).permute(0, 3, 1, 2).unsqueeze(0))
    flow_convered = flow_convered[0]
    flow_convered[..., 0] *= magnification
    flow_convered[..., 1] *= magnification * frame_np.shape[0] / frame_np.shape[1]
    
    flow_rec = torch.tensor(flow_convered, dtype=torch.float32).permute(0, 3, 1, 2).to(DEVICE)
    frame_for_synth = transformsFlow.process_frame(frame_np).unsqueeze(0).to(DEVICE)
    pred = predict_tensor(frame_for_synth, flow_rec, model, transformsFlow, return_tensor=False)
    # pred = softsplat_tensor(frame_for_synth, flow_rec * 2, transformsFlow, return_tensor=False)
    
    video = mpy.ImageSequenceClip(list(pred), fps=24)
    vpath = os.path.join(out_dir, sample_name + ".mp4")
    video.write_videofile(vpath, logger=None)

is there any info about training data? what dataset, how many samples? google's paper can animate water and many other things, but this model does not, is it because of the limit of training data?

Based on your code, I modify my inference code as below:

spec_np = generate_spectrum(vae, unet, noise_scheduler, frame, num_steps=num_steps, batch_size=1)

d = np.clip(np.median(np.abs(spec_np), axis=(1, 2)) - .05, a_min=0, a_max=None)
d = np.minimum(np.abs(spec_np), d[:, None, None, :]) * np.sign(spec_np)
_, flow_convered = transforms.deprocess_spec(torch.from_numpy(spec_np - d).permute(0, 3, 1, 2).unsqueeze(0))
flow_convered = flow_convered[0]
magnification = 5.0
flow_convered[..., 0] *= magnification
flow_convered[..., 1] *= magnification * frame_np.shape[0] / frame_np.shape[1]
flow_rec = torch.tensor(flow_convered, dtype=torch.float32).permute(0, 3, 1, 2).to(DEVICE)
frame_for_synth = transforms.process_frame(frame_np).unsqueeze(0).to(DEVICE)

model = Synthesis()
model.load_state_dict(torch.load("./models/frame_synthesis.pth")["model"])
model = model.to(DEVICE).eval()
pred = predict_tensor(frame_for_synth, flow_rec * 1.5, model, transforms, return_tensor=False)
# pred = softsplat_tensor(frame_for_synth, flow_rec * 1.5, transforms, return_tensor=False)
video = mpy.ImageSequenceClip(list(pred), fps=24)
vpath = os.path.join(out_dir, sample_name + ".mp4")
video.write_videofile(vpath, logger=None)

And I find that the moving is not realistic. How to address this issue?
https://github.com/user-attachments/assets/81da37c8-de6e-4398-b26e-09a7a8fe357b