Grok-1 Open Source: 314B Mixture-of-Experts Model by xAI | Blog post, GitHub/Source Code

xAI has open-sourced Gro-1, a 314B-parameter mixture-of-experts (MoE) model, releasing not only weights but also the model architecture and training checkpoint details. The release matters because it gives researchers and developers a full path to reproduce and study how a very large MoE system is built—while also raising practical barriers, since the published weights are about 320 GB and require substantial GPU memory to run.

The Gro-1 release page says the model is trained from scratch using xAI’s own data, with pre-training concluding in October 2023. It is described as a base (pre-trained) checkpoint rather than a chat-tuned model, implying additional fine-tuning would be needed for conversational use on xAI’s platforms. A key architectural detail is conditional computation: only a fraction of parameters activate per token. The documentation indicates that 25% of weights are active in the MoE routing setup, and later repository details specify two active experts out of eight total, yielding roughly 86B active parameters per token.

Gro-1 is positioned as an MoE system in the same family as other widely discussed sparse models, where a router selects which experts handle each token. The transcript also notes speculation that other major model families (including Google’s Gemini models) may use MoE-style designs, though Gro-1’s release provides concrete specifics. The model is licensed under Apache License 2.0, meaning downstream use, modification, and redistribution are broadly permitted.

On the implementation side, the open-source repository (hosted under xAI’s Gro-1 GitHub organization) includes JAX-based code and CUDA-related requirements, plus SentencePiece for tokenization. The repository contains example code and checkpoint runners intended to validate and run the model, but it warns that the MoE model’s sheer size makes offline testing difficult. The author estimates that running the example likely needs more than 50 GB of VRAM.

The code inspection highlights several technical choices. The main model definition appears largely self-contained in a large model.py file, with quantized weights referenced as 8-bit (8-bit quantization) to reduce memory footprint. The implementation includes a router mechanism typical of MoE layers, and the Transformer configuration uses rotary embeddings for positional encoding. The attention stack follows a multi-head Transformer pattern, with additional engineering aimed at correctness without relying on custom kernels inside the repository.

Finally, configuration details in the run.py file give a clearer picture of the model’s scale: a vocabulary size of 128,124; an 8K token sequence length; 848 attention heads; and 64 Transformer layers. The default MoE setup lists eight experts total with two active experts per token. Together, these details portray Gro-1 as a large, sparsely activated Transformer built for scalable training and inference in distributed environments—now available for developers to examine, run, and extend under an open license.