Running Gemma using HuggingFace Transformers or Ollama

Running Gemma locally or in a notebook is straightforward, but the biggest practical takeaway is that Gemma’s chat behavior depends heavily on its prompt formatting—especially because it doesn’t rely on a separate “system prompt” the way many other instruction-tuned models do. In Hugging Face Transformers, the gated access process requires accepting terms on Kaggle to download weights, then using a Hugging Face token in Colab so the notebook can fetch the model. Once set up, the workflow centers on loading Gemma 7B instruction-tuned in 4-bit quantization (so it fits on hardware like a Tesla T4), using bitsandbytes quantization and device mapping to place the model on GPU.

The transcript then drills into how prompts must be structured. Gemma uses a chat template with explicit turn markers such as “start of turn user” and “start of turn model,” plus corresponding end-of-turn tokens that also act as stopping conditions. That means a “system prompt” has to be folded into the first user message content for Hugging Face inference, since Gemma doesn’t use system prompts directly. The tokenizer is another notable difference: Gemma’s vocabulary/tokenizer supports 256,000 tokens, compared with roughly 32,000 for Llama-family models, which the narrator suggests may affect how much usable information fits into a given token budget.

A series of prompt experiments illustrates how Gemma responds differently depending on whether step-by-step reasoning is requested. When asked for “write out your reasoning step-by-step,” the model tends to produce structured chains of thought and often stops early if max length is too low; when step-by-step is not requested, it can produce more directly usable outputs like a coherent email. The transcript also compares behavior to other models (including Mistral-style tendencies toward “first, second, third” formatting) and emphasizes that prompting strategy matters more than expecting uniform behavior across model families.

On the math and coding side, results are mixed: some GSM8K-style problems are solved correctly with clear intermediate steps, while others fail due to arithmetic details like rounding or dividing by the wrong number. However, when the problem is reformulated into an algebraic form (e.g., using variables and equations), Gemma’s reasoning improves and the correct answer emerges. The transcript also touches translation limits: Gemma can handle basic translation, but Thai translation is weak, with the model effectively improvising a “translate API” style explanation rather than performing true internet-backed translation.

Finally, the Ollama path is presented as simpler for local use. Ollama already includes Gemma models, offering both 2B and 7B variants. The Ollama model file shows the same turn-based template logic, including where system-like instructions can be inserted into the prompt text. The 2B model runs quickly but is less reliable on factual tasks, making it more suitable for tasks like rewriting, summarizing, or language polishing—potentially alongside RAG—rather than strict question answering. Overall, Gemma is positioned as a capable, locally runnable model whose best results come from correct chat templating, careful quantization choices, and prompt designs tailored to its instruction and reasoning tendencies.