From 1T Tokens to ZB Scale—How to move past the internet and scale LLMs

The central claim is that today’s “trillion-token” training regimes hit a ceiling—not because data is impossible to obtain, but because models need *useful* data, not raw volume. Moving from ~1T tokens toward 10T and beyond requires a scaling framework focused on multilingual coverage, targeted expansion of underrepresented text, and stronger automated curation—so the training set better reflects how people actually communicate worldwide.

The proposed path to 10T tokens starts with multilingual training data that shifts away from an English-dominant corpus. Instead of 80–90% English, the target is closer to 30–40% English, with substantial representation from languages such as Chinese, Hindi, French, and Spanish. That shift is paired with “focused curated expansion” of historically underrepresented sources, plus tooling upgrades: automated cleaning, duplication pipelines that work across diverse languages and formats, and more flexible tokenizers that can adapt to morphological differences. The payoff is practical: models should feel more natural in languages beyond English, because current systems often underperform in languages like Indonesian when their training data is skewed toward English.

To go from 10T to 100T tokens, the data strategy broadens from curated text snapshots into continuous ingestion. The blueprint includes real-time web streams, social media feeds, large-scale transcription of podcasts and phone calls (with permission), and multimodal inputs—especially vision tokens. The jump also implies a major infrastructure step-change: immense compute, storage, and network capacity, plus transformer architectures that can unify text, images, and video within a single scalable framework. If models can be trained on continuous feeds at that scale, the door opens to continuously updated models that stay current on world events.

Beyond 100T tokens, the transcript sketches a further ladder: quadrillion-scale training would pull in sensor logs tied to robotics and embodied experience—camera streams, tactile sensors, motor commands, and Internet of Things data—compressed into semantically meaningful tokens. That requires systems that learn from interactions and unlabeled sensor streams, plus much larger GPU/TPU clusters. The farthest stage, zettabyte-scale (10^20 tokens), shifts toward city-scale and industry-wide data: 3D scans, full audio streams, simulation logs, and standardized real-time sensor data across sectors like healthcare, manufacturing, agriculture, and autonomous vehicles.

Yet the argument doesn’t treat zettabyte scale as a prerequisite for intelligence. The transcript suggests that meaningful progress toward general intelligence may come sooner through “reasoning scale,” where test-time compute can boost capability without touching training-data volume. It also notes a practical reality: current models are already “good enough” for many enterprise tasks (e.g., product requirements documents), even if they fall short for general-purpose work.

Finally, the discussion frames the race as economically and operationally binding. Scaling data and compute is expensive, so model makers may prefer to “rent” data-center capacity (a theme tied to Azure) rather than build everything in-house. Meanwhile, distillation—turning large models into smaller, cheaper ones—lets value propagate globally, but it also means competitive advantage depends on maintaining an edge through continual training and data scaling. The transcript ends by linking later-stage scaling to hardware ecosystems, with a strong suggestion that Nvidia’s GPU and robotics push is well-positioned for the sensor-heavy future.

In short: the next leap isn’t just “more tokens,” but a structured move toward multilingual, multimodal, continuously ingested, sensor-enriched data—tempered by the possibility that reasoning compute can accelerate capability without waiting for zettabyte-scale training.