Pre-Training GPT-4.5

GPT-4.5’s biggest takeaway isn’t a new parameter count—it’s that scaling pre-training still behaves predictably enough to keep delivering smarter, more nuanced behavior, even as training shifts from being compute-limited to increasingly data-limited. After GPT-4.5 launched and users reported a noticeably different experience from GPT-4, the internal focus turned to what it actually took to get there: two years of ML-and-systems co-design, long de-risking runs, and a training process that repeatedly closed the gap between forecasts and what happened in practice.

The team described pre-training at this scale as a full-stack execution problem as much as an ML one. The work starts with ML and systems collaborating long before the main run—planning the model and the training stack, sequencing changes, and running large “de-risking” experiments to ensure improvements persist when scaled up. Even with that preparation, launch decisions often happen while issues remain unresolved. Early in a run, system behavior can be far from expectations: failure modes appear that weren’t well characterized on earlier hardware generations, and rare events can become catastrophic at scale. The practical response is to keep moving forward while tightening reliability—observing failure distributions across massive pools of accelerators and network fabric, then iterating until uptime and stability improve.

Scaling difficulty grows sharply when moving from tens of thousands of GPUs to far larger clusters. The team emphasized that problems that are merely “rare” at small scale can become “catastrophic” at frontier scale, especially when infrastructure reliability, failure rates, and network behavior vary across a large statistical sample. They also framed training time as a moving target: early estimates can be wrong, and the run’s schedule shifts as issues are found and resolved.

On the ML side, GPT-4.5 targeted a goal of “10x smarter than GPT-4” in effective compute terms, and the team said the final model met that mark. A key theme was that scaling laws still hold: test loss can be predicted to decrease in a way that correlates with intelligence-like improvements, including subtle common-sense knowledge, better context handling, and nuanced abilities that weren’t obvious from benchmarks alone. The team also highlighted what they learned when things didn’t match predictions—tracking why performance deviated from expected scaling trajectories and using those discrepancies to guide future runs.

Looking ahead, the bottleneck is shifting. Data efficiency emerged as the next major lever: as compute grows faster than data, data becomes the limiting factor, requiring algorithmic innovations to extract more learning from the same dataset. Systems needs also evolve—state management changes forced multi-cluster training, and future scaling may depend on better fault tolerance co-designed with workloads to reduce operational burden.

Finally, the discussion broadened into why unsupervised pre-training works at all. Pre-training was framed as a form of compression—learning quickly during training can turn a large model into an effective compressor via prequential compression—helping explain why next-token prediction can yield general intelligence rather than mere memorization. The team also stressed the importance of metrics and held-out test sets that aren’t contaminated by training data, arguing that evaluating “intelligence” through human-legible tests can become misleading when models can memorize or have seen near-identical content.

In short: GPT-4.5’s success is presented as evidence that scaling laws remain useful, but the next leap likely hinges on data efficiency and systems reliability at extreme scale, not just adding more compute.