Llama 405b: Full 92 page Analysis, and Uncontaminated SIMPLE Benchmark Results

Meta’s Llama 3.1 405B arrives with a 92-page technical paper and a set of benchmark claims that place the open-weight model in the same quality tier as top closed systems—while also laying out unusually detailed “how we scaled” and “how we cleaned data” specifics. The headline takeaway is that Llama 3.1 405B’s training approach—higher-quality filtered data, massive compute, and extensive verification—produced benchmark performance that is “comparable” to leading models like GPT-4, even if it doesn’t yet match their full multimodal feature set.

The paper attributes much of the jump to scale and data curation. Meta describes using higher-quality, filtered training data and a compute budget so large that regulators in the EU reportedly flagged it as a systemic risk. In a snapshot comparison across traditional benchmarks, Llama 3.1 405B is positioned as on par with GPT-4 and sometimes better, though the comparison is acknowledged as imperfect for capturing nuance. The model also benefits from being downloadable now—an important practical shift for researchers who previously had to rely on API access.

A major thread throughout the document is that “open” doesn’t mean fully reproducible. The training-data provenance is described only broadly (“a variety of data sources”), making it impossible to recreate the model even with unlimited budget. The paper also leans on a recurring strategy: using one language model to improve another. Examples include training auxiliary models to filter or improve human annotations (including multilingual expertise for non-English data) and using synthetic data generation to boost smaller models. Meta also reports that self-generated data alone wasn’t always helpful for coding, but adding execution feedback—using syntax checks and unit tests—made self-improvement work.

Reasoning receives special attention. Meta defines reasoning as multi-step computation leading to correct answers, then argues that web text often lacks reliable step-by-step “chains of thought.” To address that, the training pipeline sources prompts from humans for missing mathematical skills, uses the model to verify intermediate steps, and filters out incorrect reasoning traces. For the hardest cases, it also uses Monte Carlo-style search to generate valid reasoning traces, aiming to teach not just final answers but the intermediate steps that produce them.

Beyond Meta’s benchmarks, the transcript highlights a private “Simple” benchmark used by the analyst, where Llama 3.1 405B scores 18% versus Claude 3.5 Sonnet at 32% and other models trailing. The example used is a spatial-temporal question about ice cubes in a fire, where Llama 3.1 can often infer faint linguistic cues that many models miss. The transcript argues that language models don’t truly simulate reality, so performance depends on whether the dataset provides enough signal for inference.

The paper also tackles benchmark contamination and adversarial evaluation. Meta reports that contamination is widespread in traditional leaderboards and that cleaning can produce erratic results depending on the dataset. It also notes that adversarial phrasing can sharply reduce performance, and that safety metrics should include both violation rates and false refusals—since a model that refuses everything can look “safer” while becoming less useful.

On safety and safety testing, the transcript notes Meta’s claim of reduced violation rates and lower false refusal rates, plus admissions about prompt-injection susceptibility relative to some competitors. It also references pre-release checks for harmful chemical/biological ideation that found no significant uplift when volunteers had access to Llama 3.

Finally, Meta argues that foundation-model development is still early and that simpler architectures and training recipes can outperform more complex ones once the engineering overhead is considered. The release is framed as a push toward “open responsible development” of AGI, with the transcript’s overall message that Llama 3.1 405B is strong on text performance, but not yet the clear leader across reasoning, long-context tasks, and multimodal capabilities.