Livecoding: Getting Started with LLMs, by Jeremy Howard

The core takeaway is that strong performance on an LLM multiple-choice science benchmark comes less from clever prompting and more from disciplined exploratory data analysis (EDA)—including learning how the evaluation metric can be “gamed” by surface patterns—and from building baselines that mirror how the model will be scored. The walkthrough uses a Kaggle science exam competition dataset generated from Wikipedia passages and GPT-3.5, then demonstrates how to inspect the data, test GPT-3.5 as a baseline, and iterate toward better scoring strategies.

The competition itself is framed around two constraints: it tests whether smaller models can answer science questions under limited compute, and it measures success using mean average precision at 3 (MAP@3)—meaning the correct option must appear somewhere in the model’s top three ranked answers, not necessarily as the single best choice. That scoring rule shifts the optimization target away from “always be correct” toward “rank the right answer highly enough.” The dataset generation process matters just as much: Wikipedia passages are used to create questions, GPT-3.5 generates the multiple-choice items and selects the correct option, and the “correct” answer may reflect GPT-3.5’s interpretation rather than Wikipedia’s factual correctness. This distinction forces a strategy that treats the benchmark as a model-mimicry problem, not a pure knowledge test.

Before writing any modeling code, the session emphasizes doing EDA by hand: manually answering sample questions to understand what kinds of information are required. The presenter finds that many questions hinge on highly specific details rather than general science knowledge. More importantly, repeated attempts reveal that simple heuristics—like choosing answers whose text overlaps with key terms shared across options, or selecting the longest option—can sometimes land the correct answer within the top three. Those observations suggest a practical “cheat surface” for models: they may learn shallow textual patterns that correlate with the correct option, even without deep understanding.

The notebook then establishes baselines by running GPT-3.5 turbo across the dataset. Because API calls are slow and costly, the approach uses concurrency (thread pool execution) and includes a rough cost estimate based on prompt length in tokens/words. The baseline scoring lands around 78 (with partial credit logic tied to whether the correct option appears in first, second, or third rank), which is described as “decent” for a naive setup.

From there, the session tests a more structured prompting approach: a two-step conversational flow where the model first reasons through options and then outputs a ranked set of A–E choices. On a resistivity example where the naive prompt fails, the more deliberate “think then choose” interaction corrects the answer. The walkthrough also warns that chain-of-thought style explanations can be unreliable—models can produce plausible rationalizations that don’t reflect the true reasoning.

Finally, the session explores data generation as an extension of EDA: generating plausible incorrect alternatives using GPT-3.5 so the training set better matches the benchmark’s multiple-choice style. The attempt highlights real-world failure modes—duplicate or malformed alternatives, inconsistent formatting, and the need for validation checks—reinforcing the broader message: LLMs accelerate data work, but the resulting data still requires careful quality control.

Overall, the session argues for a repeatable workflow: inspect the dataset deeply (especially how evaluation rewards ranking), run a baseline that matches the scoring format, mine hard errors to understand whether mistakes come from the model or the dataset, and only then invest in prompt engineering, function calling, or synthetic data generation.