2024 Q4 Roadmap

[ByronHsu]

As the community grows, keeping track of issues and PRs becomes more and more challenging. This pinned issue will serve as the central place to manage the progress in 2024 Q4 (~2024/12). Here we only list the important and top level issues/PRs.

Progress Tracker

Format: [title] [link], [contributor] / [reviewer (maintainer)]

New Features: Liger to Post Training

#371

Models

• llama 3.2 vision (Monkeypatch for Llama 3.2-Vision #282), @tyler-romero / @shivam15s
• pixtral 12B ([Model] Pixtral Support #253), @AndreSlavescu / @ByronHsu
• Flux model (Request to support the Flux model (T2I diffusion transformer) #73), call for help / @qingquansong
• DeepseekV2 ([feat] support for DeepseekV2 #129), call for help / @qingquansong
• Gemma2 ([feat] FusedLinearCrossEntropy support for Gemma2 #127), call for help / @yundai424
• Qwen2-VL (Add missing Qwen2-VL monkey patch test #283), @tyler-romero / @shivam15s
• Jamba (Add support for jamba model with Liger Kernel #214), @yubofredwang / @ByronHsu

Kernels

• TVD loss, Add TVD Loss Kernel #324 / @qingquansong @lancerts
• JSD loss, Add FusedLinearJSD #300 / @qingquansong @lancerts
• GroupNorm (added group norm #225), @denti / @shivam15s
• Z Loss in cross entropy (Support Z Loss in CE #239), @Tcc0403 / @shivam15s
• Flash Attention in Triton ([Kernel] Flash attention 2 #275), @remi-or / @shivam15s
• Conv2d ([Operator] conv2d #228), @AndreSlavescu / @lancerts
• Triton mm int8 x int2(FEAT Adding experimental feature : Triton mm int8xint2  #195), @MekkCyber / @ByronHsu

Testing

• assert_verbose_allclose (Fix assert_verbose_allclose bugs #261), @Tcc0403 / @ByronHsu

Patching

• Weights are not copied for instance patching (Post-init model patching fix #280), @shimizust / @ByronHsu

Community Sync

TBD