OpenLLaMA: Open-Source Reproduction of Meta AI's LLaMA for Commercial Use. Run in Google Colab.

OpenLLaMA (a 7B-parameter, open-source LLaMA-style model) can be run in Google Colab using Hugging Face Transformers, but getting usable text depends heavily on how generation is implemented. The model’s released checkpoints—trained on RedPajama—are available at least at 200B and 300B token training scales, and the walkthrough focuses on the 300B-token checkpoint. While the repository provides the weights and training details, the practical takeaway is that straightforward Transformers “generate” prompting produced garbled output on a T4-class GPU (16GB VRAM), forcing a switch to a custom decoding/sampling approach.

On the evaluation side, the OpenLLaMA release includes comparisons against Meta’s original LLaMA across common benchmarks (e.g., CB/F1 and accuracy-style metrics). The reported results show OpenLLaMA is often competitive—sometimes outperforming LLaMA on specific measures—though it can also lag on certain tasks. The broader point is that OpenLLaMA is not just a weight dump: it comes with training context (RedPajama data, long token training) and benchmark-style evidence that the model is functioning.

For local execution, the workflow starts by cloning the OpenLLaMA repository and then downloading the Hugging Face-compatible weights from the included folder structure (weights aren’t usable just by pointing at the repo root). Dependencies are installed, the model is loaded for causal language modeling, and the tokenizer is taken from the Transformers library. The run is configured for CUDA (GPU) and uses float16 loading; attempts to load in 8-bit failed in this setup, suggesting compatibility or tooling constraints.

The first generation attempt uses standard generation settings (e.g., max new tokens and temperature) with a typical prompt format. Token IDs are produced from the tokenizer, attention masks are passed to the model, and inference runs in PyTorch inference mode. Even though the model generates text, the decoded output is described as “not working as expected,” indicating that default generation/prompting choices weren’t aligned with how this model behaves.

To fix that, the walkthrough borrows a chat-oriented implementation from a separate OpenLLaMA 7B hands-on repository by Tom Misawa. The key change is custom top-k sampling (k=10): at each step, the code restricts candidate next tokens to the top 10 by probability, applies softmax over that subset, and samples from the resulting distribution. This manual token-by-token loop uses cached model states on the first pass and then feeds only the latest token on subsequent passes, stopping on the end-of-sequence token or when the max token budget is reached.

With top-k sampling, one factual prompt about the world’s tallest building returns a plausible answer (Burj Khalifa). But other prompts—like asking for a favorite phrase in a “microG scroll from the office” style format, or open-ended investment advice—produce incoherent or seemingly copied responses. The overall conclusion is pragmatic: OpenLLaMA is genuinely free and open-source to run, but reliable outputs require careful decoding (and likely better prompt formatting), not just a plug-and-play call to Transformers’ default generation.