Raven - RWKV-7B RNN's LLM Strikes Back

RWKV is a rare attempt to bring RNNs back into the large-language-model conversation by fixing the two biggest pain points that pushed transformers to dominance: slow, hard-to-parallelize training and limited context handling. Instead of treating recurrent networks as a dead end, RWKV (Receptive Weighted Key Values) uses a transformer-style training formulation that enables massive parallelization, while switching to an RNN-like stateful mechanism during inference. The result is a model that can be trained efficiently like a transformer—scaling attention linearly with the number of tokens—yet generate text with the long-horizon behavior associated with RNNs.

The core technical pitch is a “best of both worlds” design. During training, RWKV shares weights across two related formulations so the computation can be parallelized in the way transformer training benefits from. During inference, it behaves more like a conventional RNN: each step updates a state vector and produces the next token, which the transcript links to the ability to handle much longer contexts than typical transformer limits (often cited around 2,000–4,000 tokens in common setups). That long-context angle matters because it targets a practical weakness of many transformer deployments—where context windows cap how far back the model can reliably condition generation.

The project’s momentum also comes from real compute and funding. The transcript notes that training used substantial GPU resources—originally described as 64 A100s for roughly three months—and that later changes reduced the cost significantly. It also highlights that the work is driven largely by one person, with supporting documentation in blog posts and GitHub notebooks. A further practical detail: the code is Apache 2.0 open source, making it usable commercially “out of the box,” which lowers adoption friction for researchers and developers who want to experiment without licensing uncertainty.

On the experimentation side, the transcript walks through notebook runs in Google Colab using author-provided GitHub versions, including a 1.5B model and a larger 3B option that may require paid resources. The demos show that the base model can generate long, sometimes rambling outputs when prompted with instruction-like text, suggesting it isn’t fully instruction-tuned in the way many modern chat models are. A separate “chatbot” variant is described as hit-or-miss, sometimes answering factual prompts well but often drifting into completion-style repetition.

The most compelling results come from “Raven,” described as a 7B-class model fine-tuned on the Stanford Alpaca dataset. In the transcript, Raven produces coherent, fast responses on tasks like explaining what ravens are, writing a song, and interpreting a metaphor—performing comparably to other models discussed in the same timeframe. The overall takeaway is cautious but clear: RWKV and Raven offer a different direction from transformer-only scaling, even if the ecosystem’s hardware and tooling momentum still favors transformers. Still, the transcript frames RWKV as a model worth testing because it demonstrates that RNN-based language modeling can be trained and deployed in ways that address transformers’ bottlenecks—especially parallel training and longer context generation.