State of GPT

Large language models are built through a pipeline that starts with internet-scale next-token pre-training and then progressively adds human preference signals—turning raw “document completers” into instruction-following assistants. The central takeaway is that most of the capability comes from the pre-training stage, while later steps (supervised fine-tuning and RLHF) mainly reshape behavior toward helpfulness and alignment with what people rank as better outputs.

The training recipe begins with four serial stages: pre-training, supervised fine-tuning, reward modeling, and reinforcement learning from human feedback. Pre-training is the computational heavyweight: it consumes roughly 99% of training compute and uses thousands of GPUs over months. Data is assembled from mixtures such as Common Crawl and C4, plus higher-quality sources like GitHub, Wikipedia, books, and Stack Exchange, sampled according to set proportions. Raw text is converted into token sequences via tokenization methods (e.g., byte pair encoding), then fed into a Transformer that learns by predicting the next token across extremely long contexts.

Hyperparameters and scale illustrate why parameter count alone is a weak proxy for ability. A model like Llama (about 65B parameters) can outperform a larger model (e.g., GPT-3 at 175B) when trained longer on more tokens—1.4 trillion tokens versus 300 billion in the cited comparison. The talk also emphasized that training loss curves can show sharp spikes without necessarily indicating classic overfitting; rare “bad batches” or unusually difficult sequences can produce large temporary jumps.

After pre-training, the model’s general representations enable two practical adaptation paths. One is fine-tuning: sentiment classification, for example, shifts from training a task-specific model from scratch to taking a pre-trained Transformer and adapting it with comparatively small labeled datasets. The other is prompting: around the GPT-2 era, models became effective at completing tasks from instructions and examples embedded in the prompt (few-shot prompting), without additional gradient updates.

To make assistants that reliably follow instructions, the pipeline moves beyond base models. Supervised fine-tuning trains on prompt–ideal-response pairs created by human contractors, using tens of thousands of high-quality examples and continuing language modeling on this curated data. Then reward modeling collects comparative judgments: contractors rank multiple candidate completions for the same prompt, and a reward model learns to predict those rankings. Finally, reinforcement learning uses the reward model as a fixed scorer, updating the assistant so sampled outputs receive higher predicted reward.

Why this extra RLHF step matters: humans tend to prefer RLHF-tuned outputs over base or merely prompted models. The talk offered one intuition—judging quality via comparisons is easier than generating perfect examples—so human feedback can be leveraged more efficiently.

The second half shifted to how to use these assistants effectively. Because Transformers are “token simulators” without human-style inner monologue or self-correction, prompting often needs to supply structure: show-your-work patterns (e.g., “let’s think step by step”), self-consistency via multiple samples, and techniques like tree-of-thought that combine prompt templates with search logic. For harder tasks, tool use and retrieval augmented generation help: load relevant documents into context via embeddings and vector search (e.g., Llama Index), enforce output formats through constraint prompting (e.g., JSON), and consider fine-tuning only after prompt engineering is exhausted. The practical recommendation was to start with the most capable model (notably GPT-4) for best results, then optimize cost and latency later—while keeping expectations grounded in limitations like hallucinations, knowledge cutoffs, and susceptibility to prompt injection and jailbreak attacks.