What Textbooks Don't Tell You About Curve Fitting

Linear regression’s familiar “minimize squared vertical errors” rule isn’t just a convenient math trick—it drops out of a probabilistic assumption about noise. By treating observed outputs as a linear function of inputs plus random error, then choosing the parameters that make the observed data most likely, the optimization objective becomes least squares. The squared term appears because the noise is assumed to be Gaussian: under that assumption, maximizing the probability of the data is equivalent to minimizing the sum of squared differences between predicted and observed values.

That probabilistic framing also answers why “vertical distances” and “squares” show up in the first place. The model imagines a cloud of plausible outcomes around the predicted value for any given input, with the cloud’s shape determined by the noise distribution. When each data point is treated as an independent draw, the overall likelihood of the dataset is the product of per-point probabilities; taking logs turns that product into a sum, and the Gaussian form collapses neatly into the squared-error objective. In short: least squares is the maximum-likelihood solution for a linear model with additive Gaussian noise.

The discussion then pivots to what happens when multiple parameter settings fit the data equally well—or when “fit the data” conflicts with prior beliefs. A coin-bias example shows the limitation of pure maximum-likelihood: observing 4 heads out of 5 pushes the estimate toward 0.8, even though experience suggests most coins are near 0.5. The fix is to incorporate priors over parameters, selecting weights that jointly explain the data and align with expectations. Mathematically, this becomes maximizing the joint probability of data and parameters, which decomposes into a likelihood term (data fit) times a prior term (parameter plausibility).

From that joint-probability view, regularization stops being an arbitrary penalty and becomes a direct consequence of choosing a prior distribution over weights. Assuming weights follow a zero-centered Gaussian prior yields L2 regularization (ridge regression), which penalizes the squared magnitude of coefficients and gently shrinks them toward zero. The strength of the penalty is governed by the ratio between noise variance (how trustworthy the data are) and the prior variance (how strongly the prior expects weights to be small).

Assuming instead that weights are likely exactly zero except for a few important ones leads to a Laplace (double-exponential) prior, producing L1 regularization (lasso). Because L1 penalizes absolute values rather than squares, it more readily drives many coefficients to exactly zero, creating sparse models. The transcript links this sparsity preference to domains like genomics and neuroscience, where only a small subset of features or neurons typically matters for a given outcome.

Overall, the core takeaway is that least squares and common regularizers are not separate inventions: they are maximum-likelihood and maximum-a-posteriori solutions under specific noise and prior assumptions. Change those assumptions—Gaussian vs. non-Gaussian noise, or Gaussian vs. Laplace priors—and the resulting objectives change in predictable, principled ways. That same probabilistic logic extends beyond linear regression to much broader machine learning systems, where modeling assumptions often determine what “good behavior” looks like.