The BEST Open Source LLM? (Falcon 40B)

TL;DR

Falcon 40B Instruct is positioned as a practical open-source alternative because it can be downloaded and run locally under Apache 2.0, enabling fine-tuning and commercial use without an API dependency.

Briefing Cornell Notes

Briefing

Falcon 40B Instruct stands out as a practical, business-friendly alternative to closed models because it can be downloaded, run locally, and fine-tuned under the Apache 2.0 license—without routing every query through an API. The core claim is that a 40B-parameter open model delivers surprisingly strong general-purpose performance “out of the box,” and that much of the gap versus top-tier systems can be narrowed further with prompting tricks, output checks, or additional fine-tuning.

The transcript breaks down what Falcon 40B is and how to choose among variants. There are two main sizes—40B (40 billion parameters) and 7B (7 billion)—plus fine-tuned versions. For pure text generation, the base variants fit best; for chat-style back-and-forth, the Instruct variant is the target. The model’s permissive Apache 2.0 licensing is framed as a major advantage for commercial use and distribution. On hardware, the 7B model is described as feasible locally at roughly 10GB of memory in 8-bit, while Falcon 40B is more demanding—about 45–55GB in 8-bit and 100+GB in 16-bit depending on context length. The setup guidance emphasizes upgrading to Torch 2.0 and using cloud instances such as Lambda’s H100 80GB for cost-effective throughput.

In quality testing, Falcon 40B Instruct is portrayed as broadly competent across knowledge and reasoning tasks. In general knowledge Q&A, it returns answers spanning topics from everyday practicalities (like water and dehumidifiers) to factual trivia (such as iPhone release dates and atomic mass of thallium). A notable example involves legal advice risk: when asked about practicing law without a law degree in the U.S., the model is said to align with the truth that it can be possible in certain states—contrasting with GPT-3.5’s “no” response and GPT-4’s more careful, less “CYA” style. The transcript also highlights math behavior: Falcon can solve problems correctly, but the results depend heavily on prompting. When asked to “show your work,” it performs better; when instructed to provide only the answer, it is more likely to fail—consistent with how large language models can struggle with stepwise algebra when generating text linearly.

Beyond factual Q&A and math, the transcript emphasizes Falcon’s ability to handle “theory of mind” style prompts—interpreting human emotions, intentions, and miscommunication. It also demonstrates programming usefulness, including generating regular expressions and producing terminal commands for an agent-like workflow. A project called “term GPT” is used as the centerpiece: Falcon 40B is close to generating runnable command sequences from a user objective, though it may make small execution mistakes (like assuming a directory exists). The speaker argues that with better pre-prompts and fine-tuning, those errors could be reduced.

Overall, the transcript positions Falcon 40B as a strong open-source baseline that can outperform GPT-3.5 in some cases, and potentially approach GPT-4-like usefulness when paired with rule-based reward models, sanity checks, and task-specific fine-tuning. It also points to an open call from the Technology Innovation Institute for compute grants to build on Falcon, suggesting a path for developers to tailor the model without being locked into a changing API environment.

Cornell Notes

Falcon 40B Instruct is presented as a high-utility open-source large language model that can be downloaded, run locally, and fine-tuned under the Apache 2.0 license. The transcript argues that its out-of-the-box performance is strong across general knowledge, law-related Q&A, math (especially when prompted to “show your work”), theory-of-mind scenarios, and programming tasks like regex generation and terminal command planning. While it may still lag behind GPT-4 in raw capability, the gap can shrink using prompting strategies, output verification, and fine-tuning. The practical takeaway is that developers can build agent-like systems (e.g., “term GPT”) while keeping control of weights and behavior, rather than relying on closed-model API heuristics. Hardware requirements are a key constraint: 7B is relatively easy to run, while 40B needs substantial GPU memory or cloud acceleration.

Why does the Apache 2.0 license matter for Falcon 40B’s real-world use?

The transcript frames Apache 2.0 as “very open and permissive” for distribution and commercial use. That means teams can download the model, run it without sending every request to a third-party API, and fine-tune it for internal or customer-facing applications. The practical implication is control: weights and behavior can be kept stable, and deployment doesn’t depend on a vendor’s changing API policies or post-processing heuristics.

How should developers choose between Falcon 7B and Falcon 40B (and between base vs Instruct variants)?

Falcon 7B is described as more comfortable for local use (about ~10GB memory at 8-bit). Falcon 40B is heavier—roughly 45–55GB at 8-bit and 100+GB at 16-bit depending on context length—so cloud GPUs may be preferable. For model behavior, base variants are positioned for pure text generation, while the Instruct variant is positioned for chatbots, Q&A, and back-and-forth correspondence. If building a conversational system, the transcript suggests fine-tuning the Instruct variant rather than the base model.

What does the transcript suggest about Falcon’s math performance and why prompting changes outcomes?

Falcon 40B is said to get math questions correct when prompted to “show your work,” but to fail more often when asked to output only the final answer. The underlying reasoning offered is that LLMs generate text linearly and can compute algebraic expressions in chunks rather than following a clean step-by-step character order. Asking for work increases the chance the model produces a structured, non-linear reasoning path (more tokens that function like more “brain power”), improving correctness.

How does Falcon 40B handle “theory of mind” style prompts compared with expectations of deterministic AI?

The transcript highlights examples inspired by Microsoft’s “Sparks of AGI” paper, where Falcon is run through multi-turn scenarios and asked to infer human emotions and intentions. In those examples, Falcon is described as correctly identifying that characters may be “talking past each other,” interpreting how one person perceives another’s actions, and suggesting ways to improve the situation. The emphasis is that the model can model emotional incongruence rather than only producing deterministic, logic-only responses.

What is “term GPT,” and what does Falcon 40B do well or poorly in that agent-like setup?

“term GPT” is described as a system that takes a general objective and outputs terminal commands that could be executed (including via os.system) to carry out tasks like writing code, installing packages, and reading/writing files. Falcon 40B is portrayed as close to the desired behavior using a one-shot pre-prompt: it generates the right command structure, but may make small execution mistakes—specifically, attempting to create a home.html file inside a templates directory without ensuring the directory exists. The transcript suggests GPT-4 may be more reliable on such details, but Falcon could improve with better pre-prompts and fine-tuning.

What’s the transcript’s explanation for why GPT-4 can outperform smaller models even when they seem similar?

The transcript argues that GPT-4 is not just a single raw model output. It suggests GPT-4 likely uses additional heuristics and possibly multiple passes: a “sanity check” or secondary model may review an initial response and request follow-up questions or revise the answer. This is contrasted with Falcon 40B, where the weights can be downloaded and the outputs are treated as more directly attributable to the base model. The implication is that closed systems may add layers of post-processing that boost reliability.

Review Questions

What hardware and memory ranges does the transcript give for running Falcon 7B vs Falcon 40B at 8-bit and 16-bit, and how does that affect deployment choices?
Give one example of a task where the transcript says Falcon 40B’s performance depends strongly on prompting. What exact prompting change improved results?
In the “term GPT” example, what specific type of mistake does Falcon 40B make, and how does the transcript propose fixing it?

Key Points

1
Falcon 40B Instruct is positioned as a practical open-source alternative because it can be downloaded and run locally under Apache 2.0, enabling fine-tuning and commercial use without an API dependency.
2
Falcon 7B is described as feasible locally at about ~10GB memory in 8-bit, while Falcon 40B typically needs far more GPU memory (roughly 45–55GB at 8-bit and 100+GB at 16-bit depending on context length).
3
Base variants fit text generation, while Instruct variants are the better starting point for chatbots, Q&A, and conversational back-and-forth.
4
Falcon’s math accuracy is presented as prompting-sensitive: asking for “show your work” improves correctness compared with requesting only the final answer.
5
In theory-of-mind scenarios, Falcon 40B is described as able to infer emotions, intentions, and miscommunication patterns rather than behaving purely deterministically.
6
For agent-style command generation (“term GPT”), Falcon 40B is close but can make small operational mistakes (like assuming directories exist), which the transcript suggests could be reduced via improved pre-prompts and fine-tuning.
7
The transcript attributes GPT-4’s reliability partly to layered heuristics and possible multi-pass verification, not just raw model size.

Highlights

Falcon 40B’s biggest practical advantage is control: Apache 2.0 licensing lets teams run and fine-tune the model themselves instead of sending every request to a closed API.

Math performance improves when the model is prompted to “show your work,” aligning with the transcript’s view that linear generation can break algebra unless structure is enforced.

Falcon 40B is portrayed as surprisingly strong at theory-of-mind prompts—inferring how people perceive actions and why conversations go wrong.

In “term GPT,” Falcon 40B can generate runnable terminal command sequences, but it may still miss small execution prerequisites like creating required directories.

The transcript argues GPT-4’s edge likely comes from extra post-processing and sanity checks layered on top of model output, not only from model scale.

Topics

Falcon 40B Instruct
Open Source LLMs
Prompting for Math
Theory of Mind
Agent Command Generation

Mentioned

LLM
API
GPU
CPU
RAM
GPT
Torch
os.system
AGI