Falcon Soars to the Top - The NEW 40B LLM Rises above the rest.

Falcon has arrived as a new, from-scratch large language model family—anchored by a 40B parameter model—and it’s already topping Hugging Face’s Open LLM leaderboard. The immediate significance is practical: benchmark placement suggests strong general capability, while architectural and inference-focused optimizations (including flash attention and other speedups) aim to make that capability usable in real deployments.

Falcon comes in two main sizes: a 40 billion parameter model and a 7 billion parameter model. The 40B release is notable because models of that scale are relatively rare in the open ecosystem, with only a few comparable entries mentioned (such as LLaMA 65B). Even without a paper yet, Hugging Face benchmarking results place Falcon at the top across multiple tasks, including ARC-style reasoning and HellaSwag. On HellaSwag, the transcript cites GPT-4 reaching 95 in a few-shot setup, while GPT-3.5 lands around 85.5; Falcon’s reported score is about 85.3 in the same neighborhood—close enough to be competitive, but still enough to secure the leaderboard lead in the aggregated view.

The 7B model also performs well within its size class, with the transcript comparing it against MPT-7 and reporting Falcon 7B base scores around 78.1 versus MPT-7’s 76.1 (as presented in the cited leaderboard view). The takeaway is that Falcon’s base models look strong for their parameter budgets, though the creator repeatedly cautions that benchmarks depend on use case and that users should test directly.

Where the release gets contentious is licensing. Falcon is described as not fully free: if a company makes more than $1 million per year, a 10% royalty is expected. That has triggered debate online, and the transcript raises a strategic concern—whether businesses could restructure around the threshold by routing usage through an API provider that stays under the cap.

On Hugging Face, multiple variants are listed: Falcon 40B base, Falcon 40B instruct, Falcon 7B base, and Falcon 7B instruct. The instruct variants are central to the next story thread: early hands-on tests suggest the base models’ benchmark strength doesn’t automatically translate into better instruction-following. In the transcript’s code experiments, the 40B instruct model often produces refusal-style answers (e.g., declining to write an email to Sam Altman) and shows weak performance on certain reasoning and formatting tasks (including a miscalculation in an apples problem and difficulty with a haiku-in-a-tweet prompt). The 7B instruct model behaves differently—sometimes producing coherent, persuasive text where the 40B instruct model refuses—but it still struggles on the same kind of arithmetic reasoning.

Overall, the release lands as a mixed picture: Falcon’s base models look like serious contenders on public benchmarks, but the instruct-tuned versions may be trained on less effective data for instruction following. The transcript’s practical conclusion is that starting from Falcon 7B for further instruction fine-tuning—potentially using better curated datasets—may be the most productive path, while serving the 40B model remains hardware-intensive until quantization (like 4-bit) becomes straightforward enough for broader users.