wan2.2-ti2v-5B sample 的视频有问题

[wbfwonderful]

运行给的 example https://github.com/X-GenGroup/Flow-Factory/blob/main/examples/grpo/full/wan22_t2v.yaml ，只是改成 Wan-AI/Wan2.2-TI2V-5B-Diffusers，sample 的视频有问题

但是 Wan-AI/Wan2.2-T2V-A14B-Diffusers 和 wan2.1 就没问题，是这个模型特有的问题吗？

[Jayce-Ping]

默认配置是分辨率256x256x5, 这个低分辨率下有问题也是正常的，只是一个example。你可以根据你的显存状况修改为正常值，如512x512x121.

Wan-AI/Wan2.2-T2V-A14B-Diffusers 和 wan2.1 14B没问题，应该是模型的大小的问题，5B的小模型还是太弱了。此外，这里设置的是noise_level=0.9，你也可以尝试减小noise_level来提高视频质量。

很遗憾，我现在没有足够的计算资源去完整地测试video model😭.

[wbfwonderful]

我用更小的 wan2.1-1.3B 也是正常的。。不知道是哪里的问题

[Jayce-Ping]

看起来和这个Issue #76 是一样的问题。按理来说Wan2.2-TI2V-5B-Diffusers和Wan2.2-T2V-A14B/Wan2.2-I2V-A14B应该是一样的diffusers.pipeline，共用一套code. 你可以试试diffusers的pipeline直接推理看看，我会抽时间检查一下代码。

[Jayce-Ping]

我测试了一下，还是配置resolution太小的问题. Wan2.2-TI2V-5B这个模型应该是没有在很小的resolution上训练过，增大resolution到正常范围即可。例如修改为如下配置可以得到正常视频：

# Environment Configuration
launcher: "accelerate"  # Options: accelerate
# Use FSDP to shard the gradient and optimizer.
# If you want to further shard the model, use `FSDP full shard strategy` to train two transformers simultaneously,
# i.e., `config/accelerate_configs/fsdp_full_shard.yaml`
# NOTE: Deepspeed and FSDP2 do not support sharding two models simultaneously yet.
config_file: config/accelerate_configs/multi_gpu.yaml
num_processes: 8  # Number of processes to launch (overrides config file)
main_process_port: 29500
mixed_precision: "bf16"  # Options: no, fp16, bf16

# Data Configuration
data:
  dataset_dir: "dataset/pickscore"  # Path to dataset folder
  preprocessing_batch_size: 8  # Batch size for preprocessing
  dataloader_num_workers: 16  # Number of workers for DataLoader
  force_reprocess: true  # Force reprocessing of the dataset
  cache_dir: "~/.cache/flow_factory/datasets" # Cache directory for preprocessed datasets
  max_dataset_size: 8  # Limit the maximum number of samples in the dataset
  sampler_type: "auto"  # Options: auto, distributed_k_repeat, group_contiguous

# Model Configuration
model:
  finetune_type: 'lora' # Options: full, lora
  lora_rank : 64
  lora_alpha : 128
  target_components: 'transformer' # Options: transformer, transformer_2, or ['transformer', 'transformer_2']
  # For Wan2.2, you can specify something like ['transformer.to_k', 'transformer_2.to_q'] to train different modules for different components.
  # Even if you only want to train one transformer at the time, a trick to save memory is that
  # with `fsdp_full_shard`, set `target_components` to `['transformer', 'transformer_2']` and target_modules to `transformer.default`
  # both `transformer` and `transformer_2` will be shared among GPUs to save memory.
  # But pay attention to the `boundary_ratio` that used in Wan2.2, before/after which different transformer is used and therefore needed for backward.
  target_modules: "default"
  model_name_or_path: "Wan-AI/Wan2.2-TI2V-5B-Diffusers"  # Wan-AI/Wan2.1-T2V-14B-Diffusers / Wan-AI/Wan2.2-T2V-A14B-Diffusers
  model_type: "wan2_t2v"  # wan2_t2v, wan2_i2v, wan2_v2v
  resume_path: null # Path to load previous checkpoint/lora adapter
  resume_type: null # Options: lora, full, state. Null to auto-detect based on `finetune_type`

log:
  run_name: null  # Run name (auto: {model_type}_{finetune_type}_{trainer_type}_{timestamp})
  project: "Flow-Factory"  # Project name for logging
  logging_backend: "wandb"  # Options: wandb, swanlab, tensorboard, none
  save_dir: "saves/"  # Directory to save model checkpoints and logs
  save_freq: 20  # Save frequency in epochs (0 to disable)
  save_model_only: true  # Save only the model weights (not optimizer, scheduler, etc.)

# Training Configuration
train:
  # Trainer settings
  trainer_type: 'grpo' # Options: 'grpo', 'nft', 'awm'
  advantage_aggregation: 'gdpo' # Options: 'sum', 'gdpo'
  # Clipping
  clip_range: 1.0e-4  # PPO/GRPO clipping range
  adv_clip_range: 5.0  # Advantage clipping range
  # KL div
  kl_type: 'v-based' # Options: 'x-based', 'v-based'
  kl_beta: 0 # KL divergence coefficient. Set ~1e-2 for 'x-based' and ~1e-3 for 'v-based'.
  ref_param_device: 'cuda' # Options: cpu, cuda

  # Video settings
  resolution: [704, 1280]  # Can be int or [height, width]
  num_frames: 5 # (F - 1) mod 4 == 0
  num_inference_steps: 20  # Number of timesteps
  guidance_scale: 5.0  # Guidance scale for sampling
  guidance_scale_2: 3.0  # Guidance scale for sampling

  # Batch and sampling
  per_device_batch_size: 1  # Batch size per device
  group_size: 2  # Group size for GRPO sampling
  global_std: false  # Use global std for advantage normalization
  unique_sample_num_per_epoch: 8  # Unique samples per group
  gradient_step_per_epoch: 2  # Gradient steps per epoch
  gradient_accumulation_steps: auto  # Options: auto, or positive integer. When set, `gradient_step_per_epoch` is ignored.
  
  # Optimization
  seed: 42  # Random seed
  learning_rate: 1.0e-4  # Initial learning rate
  adam_weight_decay: 1.0e-4  # AdamW weight decay
  adam_betas: [0.9, 0.999]  # AdamW betas
  adam_epsilon: 1.0e-8  # AdamW epsilon
  max_grad_norm: 1.0  # Max gradient norm for clipping

  # EMA
  ema_decay: 0.9  # EMA decay rate (0 to disable)
  ema_update_interval: 4  # EMA update interval (in epochs)
  ema_device: "cuda"  # Device to store EMA model (options: cpu, cuda)

  # Gradient checkpointing
  enable_gradient_checkpointing: true  # Enable gradient checkpointing to save memory with extra compute


# Scheduler Configuration
scheduler:
  dynamics_type: "Flow-SDE"  # Options: Flow-SDE, Dance-SDE, CPS, ODE
  noise_level: 0.9  # Noise level for sampling
  num_sde_steps: 1  # Number of noise steps
  sde_steps: [1, 2, 3]  # Custom noise window, noise steps are randomly selected from this list during training
  seed: 42  # Scheduler seed (for noise step selection)

# Evaluation settings
eval:
  resolution: [704, 1280]  # Can be int or [height, width]
  num_frames: 5 # (F - 1) mod 4 == 0
  per_device_batch_size: 1  # Eval batch size
  guidance_scale: 5.0  # Guidance scale for sampling
  num_inference_steps: 50  # Number of eval timesteps
  eval_freq: 20  # Eval frequency in epochs (0 to disable)
  seed: 42  # Eval seed (defaults to training seed)

# Reward Model Configuration
rewards:
  - name: "pick_score"
    reward_model: "PickScore"
    weight: 1  # Weight of this reward model
    batch_size: 16
    device: "cuda"
    dtype: bfloat16

# Optional Evaluation Reward Models
# eval_rewards:
#   - name: "text_alignment"
#     reward_model: "CLIP"
#     batch_size: 16
#     device: "cuda"
#     dtype: bfloat16

[wbfwonderful]

okok，感谢大佬🙏