AI On An Exponential? Data, Mamba, and More

AI’s next leap is less about waiting for bigger models and more about squeezing far more capability out of what already exists—especially high-quality data, smarter inference-time compute, and architectures that avoid Transformers’ quadratic bottlenecks. The through-line is that the “steep part” of the exponential curve is still ahead, and the biggest gains are increasingly coming from better inputs and better ways to use compute, not just scaling parameters.

Data quality sits at the center of that claim. Tree D. (co-author of Mamba) argues that architecture changes mostly shift performance by an offset, while the slope of scaling laws is largely driven by data quality. Even with Mamba’s buzz, the message is that better filtering and more digestible training data can dominate incremental improvements from optimization tweaks or architectural refinements. Arthur Mensch, co-founder of Mistral, reinforces the same point: researchers need new techniques for high-quality data filtering, and the frontier also includes letting models decide how much compute to allocate to different problems. Sebastian Bubeck adds a striking perspective from prior work on “effective compute”: small gains from optimization or architecture often look like marginal 2–3% improvements, but crafting training data so it’s more usable by LLMs can produce orders-of-magnitude jumps in effective learning.

That data focus arrives alongside a major architectural shift: Mamba. Transformers—behind systems from GPT-4 to earlier models—process sequences using attention where each token interacts with every other token, creating quadratic complexity. That becomes expensive at very long contexts. Mamba replaces that with a state-space approach: a compressed hidden state is updated step-by-step as new inputs arrive, using a selection mechanism to decide what to ignore and what to retain. The payoff is faster inference and the ability to handle extremely long sequences without the same attention explosion. The paper’s claims include about 5× faster inference on NVIDIA A100 hardware, and the transcript highlights tasks where long-context performance holds up—such as induction-head behavior staying accurate up to a million tokens, plus strong results on long-sequence DNA classification (human, chimpanzee, gorilla, orangutan, and more). The architecture is also described as “hardware-aware,” with parts of the computation mapped to fast GPU SRAM to reduce the cost of maintaining a rich state.

Beyond architecture, the next lever is inference-time compute—getting models to “think longer” when it matters. Lucas Kaiser (associated with OpenAI and the Transformer lineage) frames this as chains of thought that can be extended and combined with multimodal generation, where models produce sequences (text, frames, or other modalities) before answering. OpenAI’s Gnome Brown acknowledges that longer inference can be slower and costlier, but argues the tradeoff is worth it for high-stakes outcomes like drug discovery or proving hard mathematical claims. That theme connects to process-based verification (step-by-step checking), tied to a paper arguing that capabilities can improve significantly without expensive retraining—often with modest extra one-time compute costs but large “compute-equivalent” gains.

Finally, the transcript points to prompt optimization as another underappreciated accelerator: language models can learn better prompts automatically, improving performance even on existing systems. The year-ahead outlook is that multimodal output quality will keep rising—text-to-video and photorealistic generation becoming harder to distinguish from real footage—while voice imitation and audio/video synthesis improve to the point where attribution by sound alone may become unreliable.