Llama 4 Test with Groq: Coding, Data Extraction, Data Labelling, Summarization, RAG

Meta’s Llama 4 lineup—Scout (109B), Maverick (400B), and Behemoth (2T, still training)—arrives with headline claims built around huge context windows and strong benchmark performance, but real-world testing paints a more mixed picture. Meta says Llama 4 Scout supports a 10 million token context window, while Llama 4 Maverick supports a 1 million token context window and “beats” models like GPT-4.5 and Gemini 2.0 Flash on STEM benchmarks. The catch: these models are mixture-of-experts (MoE), meaning only a subset of parameters activates per token, yet the full model still must be loaded on the GPU—so the hardware requirements remain steep. For quantization, Meta’s guidance points to needing an H100-class GPU, which effectively keeps “local” experimentation out of reach for most users.

A key technical differentiator is Meta’s revamped post-training pipeline. After large-scale filtered pretraining, the models are tuned into chat behavior via instruction tuning and reinforcement-style optimization. Historically, human feedback drove instruction tuning; now the pipeline leans more on supervised fine-tuning and reinforcement learning using reward functions and methods such as GRPO, along with DPO (direct preference optimization). Meta’s stated motivation is that supervised fine-tuning and DPO can over-constrain the model, limiting exploration during later online RL and hurting accuracy—especially in reasoning, coding, and other demanding domains. Distillation also plays a role: Llama 4 Behemoth is positioned as a “teacher” model used to distill capability into smaller MoE variants like Scout and Maverick.

Independent evaluation and practical tests using Groq’s API access for Llama 4 Scout and Maverick show both strengths and failure modes. In coding-style prompts, Llama 4 Scout generated a synthetic dataset of wealthy people by continent (including Antarctica) and produced a Pandas workflow to compute the top five wealthiest per continent, completing in under two seconds. It also performed structured data extraction from a receipt image and from receipt text, returning correctly formatted fields such as line items and totals in markdown/JSON-like structures. Multimodal behavior looked solid: given an image of a receipt on a countertop, it described the scene and extracted key numbers quickly (around a second).

Where things became less reliable was in strict output formatting and summarization control. A tweet-labeling task that required a single JSON response triggered a “failed to generate JSON” error; the model produced partial fields but then diverged when a predicted class didn’t match the allowed label set. Summarization also showed a prompt-following gap: when asked for a concise summary, the output started with extra framing text rather than delivering only the requested summary. A follow-up attempt to generate a LinkedIn post likewise began with an unwanted preface.

Finally, retrieval-augmented generation (RAG) tests against a Meta earnings document produced grounded answers when the quote existed and returned “not enough information” when it didn’t. Table extraction worked well when values came from a single table, and remained accurate even when the task required pulling specific fields across multiple tables.

Overall, Llama 4’s MoE architecture, long-context claims, and post-training changes look promising on targeted tasks—especially coding, extraction, and multimodal OCR—but strict schema adherence and “just give me the summary” instruction following still need work. Meta’s marketing benchmark numbers may not fully match real-world performance, and the gap is likely to narrow only as more independent tests and broader availability (including smaller local variants) arrive.