Reinforcement Fine-Tuning—12 Days of OpenAI: Day 2

OpenAI is previewing reinforcement fine-tuning for its o1 model family—an approach that lets developers and researchers adapt models to specialized tasks using reinforcement learning rather than simple “copy the examples” training. The key promise: with a small set of domain examples and a scoring system, o1 can learn to reason in ways tailored to a user’s data and objectives, producing better task performance and more useful ranked outputs.

Unlike supervised fine-tuning, which teaches a model to mimic patterns in input text or images (often to change tone, style, or formatting), reinforcement fine-tuning trains the model to improve its decision-making. During training, the model is given a problem and allowed time to think, then its final answer is graded. Correct reasoning paths are reinforced, while incorrect ones are discouraged. OpenAI frames this as a technique that builds on reinforcement learning methods used internally to train frontier models, including the o1 series.

The practical demonstration centers on computational genetics for rare diseases. Justin Rhee of Lawrence Berkeley National Laboratory describes a major challenge in the field: each rare genetic condition affects relatively few people, but collectively rare diseases impact roughly 300 million people worldwide and often require months or years to diagnose. His team works with curated case-report data extracted from hundreds of scientific publications, including patient symptoms, absent symptoms (what is not present), and the causative gene known from the case report. The task is to infer which gene could be responsible given an incomplete symptom profile.

In the live walkthrough, OpenAI uses an o1 mini starting point and trains a reinforcement fine-tuned variant using about 1,100 training examples. Each training record includes a case description (age and symptom list), an explicit list of absent symptoms to help rule out likely genes, and instructions asking the model to output a ranked list of candidate genes plus an explanation. The correct gene is used internally for grading, not shown to the model during generation.

Grading is handled by “graders” that compare the model’s ranked gene list to the known correct answer and return a score from 0 to 1, with partial credit when the correct gene appears lower in the ranking. A validation set is prepared in the same format but with no overlap in correct genes between training and validation, aiming to test generalization rather than memorization.

Results are tracked through validation reward and evaluation metrics such as Top-1 (correct gene is first), Top-5 (correct gene appears in the top five), and Top-Max (correct gene appears anywhere in the list). OpenAI reports that the base o1 mini improves from roughly 17% Top-1 to about 31% after reinforcement fine-tuning, while also showing gains across Top-5 and Top-Max. Example outputs highlight not just accuracy improvements but also the model’s reasoning—linking symptom combinations to likely causal genes and ranking alternatives when the top prediction is not the only plausible candidate.

OpenAI positions reinforcement fine-tuning as a general technique with early signs of value across legal, AI safety, healthcare, and bioinformatics. The company plans to launch it publicly in early next year, while expanding access through a “reinforcement fine-tuning research program” for universities, researchers, and enterprises working on complex, expert-driven tasks.