What's Next? (LLM Bootcamp)

Multimodal large language models are rapidly turning into general-purpose “brains” for both software and physical machines—especially robotics—by learning to process images and text as one unified token stream. That shift matters because it removes a long-standing bottleneck: robots no longer need bespoke perception and planning for every task. Instead, a robot body can be treated as another tool the model can reason about at a high level, enabling a looser, language-first interface where humans can request outcomes (“bring me a drink and a snack”) and the system composes actions using what it can perceive and what the robot can physically do.

The robotics case starts with how vision models work when built with Transformer architectures. Vision Transformers treat images as sequences of tokens (patches) and feed them through the same core Transformer machinery used for text, requiring little more than changes at the input stage. At sufficient scale, these models become strong foundation models for tasks far beyond classification—semantic segmentation, depth estimation, and transfer learning including zero-shot alignment with language models. A key nuance is that Vision Transformers are extremely data-hungry because they don’t bake in strong image-specific inductive biases the way convolutional networks do; they must learn those structure-and-texture preferences from data. The result is a different “bias profile”: Vision Transformers tend to show more shape bias, while convolutional networks lean more heavily on texture.

The multimodality unlock is also framed through concrete capability jumps. Public GPT-4 access is text-only, but multimodal versions are expected; meanwhile, OpenAI’s demos illustrate how combining image understanding with text generation can move from describing a design to producing working code and an interactive website. The broader point: multimodal models aren’t confined to “NLP-adjacent” tasks. They can affect domains that previously looked insulated from language-model progress.

General-purpose robotics is presented as the most exciting application. The mechanism is a language interface to embodied tools: robot skills become goal-conditioned policies (affordances) that the model can plan around. Rather than hard-coding low-level control like “drive servo X to Y,” the model uses language to interpret the user’s intent, identify relevant objects in the robot’s visual field, and select or compose actions. Language models also help with planning and mid-task interruption—reasoning from text can support a robot that adapts when a human changes their mind.

The discussion then pivots to scaling limits. The Transformer architecture remains the dominant path because it scales well with context and trains efficiently in parallel, though recurrent approaches like RWKV are emerging as potential alternatives for cheaper inference. For capability growth, compute is not the main bottleneck at institutional scale; data is. Estimates suggest high-quality language data may be exhausted between 2024 and 2026, and training performance improves predictably with compute rather than model size alone. The “Chinchilla” lesson is that models should scale with data at roughly the same pace; training too-large models on too little data wastes compute and caps achievable loss.

Finally, the transcript turns to AGI-adjacent progress and safety. Agents that can plan, call tools, and self-improve prompts (including “AutoGPT”-style projects) are portrayed as a new kind of computing metaphor—prompt updates acting like memory and execution. Security concerns are emphasized: prompt injection can override instructions, tool ecosystems can enable data exfiltration, and jailbreaks remain effective. The safety debate is unresolved, ranging from “pause training” arguments to “release and learn” approaches, with both sides acknowledging that current systems are powerful but poorly understood—more like simulators than predictable rule-followers.