Mistral 8x7B Part 1- So What is a Mixture of Experts Model?

Mistral’s newly released “8x7B” model is a Mixture of Experts (MoE) system: eight separate expert networks, each roughly the size of Mistral 7B, are combined under a gating mechanism that routes each input to the most relevant expert(s). The practical payoff is efficiency at scale—MoE architectures can activate only a subset of parameters per token—while enabling models that are effectively much larger than a single dense network. The catch is that MoE inference tends to be slower and more hardware-dependent, making local use difficult for most people.

At the core of MoE is a gating layer that decides which “expert” should handle a given prompt. Instead of sending every token through one monolithic transformer decoder, the model uses multiple experts (in this case, eight) and activates only the chosen one(s). Training therefore has two moving parts: the experts themselves must learn specialized skills, and the gating network must learn how to map inputs to the right expert. Some MoE variants can route to multiple experts and then combine their outputs, but the key theme remains specialization plus selective computation.

Interest in MoE surged again because of earlier rumors about GPT-4’s architecture. One widely discussed claim was that GPT-4 used an MoE design with eight experts, each on the order of 220B parameters, plus a gating mechanism. That kind of setup would make it easier to dedicate experts to distinct capabilities—function calling is the example that gets attention—while other experts handle complementary tasks. Similar ideas have also been speculated for Gemini, reflecting how MoE can be framed as a way to modularize model behavior.

MoE is not new. The transcript points to early work from 2014, including papers with authors such as Ilya Sutskever, and later “sparsely-gated” approaches associated with Noam Shazeer (now CEO and co-founder of CharacterAI) and collaborators including Geoffrey Hinton and Jeff Dean. Switch Transformers, another milestone, pushed training toward trillion-parameter territory using MoE-style routing, building on T5-like architectures. More recently, open-source efforts such as OpenMoE have tackled the engineering and compute hurdles required to train and run MoE models, including reliance on large-scale infrastructure.

For hands-on testing, the transcript notes several ways to try Mistral’s MoE variant: community uploads on Hugging Face (including a “Mistral-7B-8Expert” version), GPTQ weight releases from TheBloke, and online tooling such as Replicate and Vercel AI SDK. A key warning is that the MoE model is described as a base model rather than instruction-tuned, so prompts may need adjustment. Benchmarks reported on the model card include MMLU and GSM-8K results, but the scores are still far from GPT-4-level performance, and inference speed remains a limitation.

Overall, the release lands in a broader trend: using MoE to scale model capacity, then potentially distilling those larger systems into smaller, faster models later. That distillation path is presented as a likely next step for making MoE benefits more accessible.