Vicuna - 90% of ChatGPT quality by using a new dataset?

Vicuna is being positioned as an open-source-style chat model that delivers roughly “90% of ChatGPT quality” by fine-tuning a LLaMa base model on conversation data scraped from ShareGPT—yet the dataset controversy around that source may keep the model from being usable commercially.

At the core of the claim is how Vicuna was built and how it was benchmarked. The model is essentially a fine-tuned LLaMa variant: it starts from LLaMa 13B and then trains on instruction-style conversational examples drawn from ShareGPT, a site where users post ChatGPT-like dialogue transcripts. The training set is larger and structured differently than earlier instruction-tuned efforts. Alpaca, for instance, is described as trained on 52,000 “self instruct” samples with a sequence length of 512. Vicuna, by contrast, uses about 70,000 conversation samples and expands training context from 512 up to 2048 tokens by packing multi-turn back-and-forth exchanges into a single training span.

The “90% quality” framing comes from an evaluation method that feeds outputs from multiple models—LLaMa, Alpaca, Bard, and Vicuna—into GPT-4 with a scoring prompt. In that setup, raw LLaMa scores lowest because it isn’t meaningfully instruction-tuned. Alpaca performs better, while Vicuna lands much closer to Bard in the reported comparisons. ChatGPT is treated as a near-ideal reference point because the scoring setup is designed to reward the target style.

Still, the benchmarking approach is explicitly treated as imperfect. The transcript notes that chatbot evaluation remains an open research problem: different models can excel under different prompting styles, so “best model” results can shift depending on how prompts are crafted. That uncertainty is why the comparisons also include qualitative checks—such as observing that Vicuna tends to generate longer responses than Alpaca—and why the discussion points to alternative evaluation ideas borrowed from speech research, like mean opinion score (MOS), where humans or structured judgments decide which output is more human-like.

The biggest practical constraint comes from the ShareGPT data controversy. ShareGPT reportedly removed its “explore” page after alleging that Google used the site’s data to train Bard. The dispute escalated in reporting that a Google researcher, Jacob Devlin (first author of the BERT paper), quit after raising concerns with senior leadership. Google later denied that Bard was trained on ShareGPT/ChatGPT data, but the fallout still matters for Vicuna: since Vicuna was trained on that ShareGPT-derived material, the resulting model is described as not commercially usable. Add the separate licensing limits around LLaMa itself, and the transcript paints a picture of models that are fine to test but hard to deploy.

Finally, the discussion touches on adjacent efforts at large labs, including DeepMind’s involvement in Gemini and references to DeepMind’s earlier internal system “Sparrow,” which aimed at features like citations. For now, Vicuna’s weights and training data are not fully released, though code for training/serving/evaluating is planned; the dataset itself is not expected to be shared. The result is a model that can be tried online, performs competitively in many prompts, and yet sits behind legal and data-access barriers that limit real-world adoption.