Welcome to oMLX Discussions!

[jundot]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[HLGRL]

I saw that you already created branch with speculative decoding. I hope it will be merged some day to check how it can utilize 8x batching with draft model, if it can somehow match MoE models in PP and T/s, fingers crossed 🤞

[jundot]

yeah i did build speculative decoding on a branch. it works in theory, but if you actually run it with MoE models the acceptance rate sits around 20%, so there's basically no speedup at all. i haven't figured out yet whether that's a problem with my implementation or just an inherent limitation of MoE models with draft-based speculation.

sorry for the slow progress on it — i've been focused on adding the highlights feature (community benchmarks) the past couple days. planning to test with some other model types and merge it once it actually shows meaningful gains. if you have any ideas or suggestions feel free to share anytime.

[eokic]

I've noticed that the mlx-lm dependency is almost two weeks behind as of this writing. How often are such core libraries updated, what's the long-term plan here? How many breaking changes do you normally have when upgrading these? Could this be unrelated/independent from oMLX releases? Related #110

[jundot]

When there are dramatic improvements in performance or model support in mlx-lm, i try to apply them as fast as possible. Right now the mlx-lm version in omlx has no operational issues and there haven't been major changes in model support either, which is why i haven't updated it yet. But i'll get it updated soon!

[bbulee]

占个楼，关注大神

[eokic]

Low priority UX suggestions:

1. I noticed that the macOS menu bar app does not display which model(s) are currently loaded (pinned). Would be nice to be able to quickly read that info from the menu bar.
2. similar issue with all the model dropdowns in the admin UI. The dropdowns do not clearly communicate which model(s) are currently pinned, perhaps even an indicator on which one is set as the "default" model.