Mixture of Experts - #2
Open
dan-smith-tech wants to merge 9 commits into
Open
Conversation
dan-smith-tech
marked this pull request as draft
March 31, 2026 17:00
This reverts commit a3b4c3d.
This reverts commit 6606c03.
- Add Gate, Expert, MoE classes to network/layers/transformer.py - Update TransformerBlockModule with optional MoE FFN (use_moe param), storing per-block moe_l_aux and moe_cz_lz after each forward pass - Update PETBody in network/body/embedding.py to accept all MoE hyperparameters alongside existing use_adapter, pass them to each TransformerBlockModule, and accumulate moe_l_aux/moe_cz_lz across transformer blocks in forward - Update EveNetModel in network/evenet_model.py to pass MoE params to PETBody, track moe_l_aux_total/moe_cz_lz_total in forward, and return them as 'L_aux'/'cz_Lz' in the output dict - Add network/loss/moe.py with a loss() helper that extracts L_aux and cz_Lz from the model output dict for integration into training loops Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dan-smith-tech
marked this pull request as ready for review
March 31, 2026 17:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add MoE parameters.