Andrej Karpathy on AI at Tesla (Full Stack Deep Learning - August 2018)

Deep learning for real-world autonomy is shifting the center of gravity from “clever algorithms” to “programming with data.” Andrej Karpathy, speaking from his experience building Tesla’s machine-learning stack, says a surprising amount of engineering time goes into collecting, labeling, and continuously correcting datasets—because the neural network’s weights and the training objective effectively become the program, while humans increasingly act as curators and debuggers of the data.

Karpathy frames this as a “software 2.0” stack: instead of writing C++ code that directly implements behavior, teams define an architecture and an evaluation criterion, then optimization fills in the solution via learned weights. In that view, intermediate network outputs and end-to-end training create a pipeline where some “code” is effectively produced by gradient descent. He argues this paradigm is already taking over in practice, and he points to Tesla’s perception tasks—like deciding whether a car is parked—where a neural network can use richer image context than hand-written rules.

The most concrete takeaway is how dataset work becomes a full engineering discipline. In academia, teams often start from curated benchmarks; in industrial settings, they must create their own datasets from a global fleet, and edge cases quickly force documentation to balloon into dozens of pages. Lane markings, for instance, can vary by location and rule precedence (orange over white), and small ambiguities—whether to follow a discontinuity or interpolate through it—can ripple into controller behavior. When labeling instructions change, previously labeled images may become wrong, creating “deprecated instructions” inside the dataset and requiring re-labeling and retraining.

Karpathy also highlights two forms of imbalance that rarely receive the same attention in standard benchmarks: label imbalance and data imbalance. Rare events—like blinkers being on, or orange traffic lights—can be too scarce to label efficiently if data is sampled uniformly. Tesla’s approach includes fleet-driven data sourcing, such as requesting images when vehicles transition between lanes to boost the fraction of examples with blinkers active. Environmental imbalance matters too: most data may look like clear highway driving, while wet conditions or specific cities can be underrepresented, degrading performance where it matters.

He argues that the assumptions behind classic datasets (clean labels, balanced classes, single-task supervision) are far from realistic. Real deployments involve noisy labels, safety-critical categories with few examples, and multi-task learning (often dozens of prediction heads). Even more, fleets generate massive unlabeled streams, and teams must decide which samples to label under a budget—turning labeling into an active, query-based process rather than a one-time dataset build.

Karpathy calls this ongoing loop a “data engine”: deploy a model, detect failure modes, collect and label more examples, retrain, and redeploy until the error distribution stabilizes. He illustrates the idea with Tesla’s auto wiper system, where initial training missed unusual edge cases like corn flakes and ketchup, and where the system’s behavior improved only after those cases entered the training loop.

Finally, he sketches what a “2.0 IDE” would need to look like: tools for dataset visualization and slicing, annotation-layer management, disagreement detection among labelers, automated mislabel detection using loss disagreement, and mechanisms to suggest which unlabeled images deserve labeling. The broader message is that building autonomy increasingly resembles software development—except the “source code” is data, and the debugging target is the dataset itself, not just the model.