Session 45 - Hypothesis Testing Part 1 | DSMP 2023

Hypothesis testing is presented as the decision-making tool for turning sample data into probabilistic claims about a population—especially when business, finance, and economics can’t rely on “try it and see” forever. The core problem: observed results (like longer average YouTube view duration after changing a video style) can be caused by randomness, smarter timing, or other hidden factors. For high-stakes decisions—whether a new product should replace an old one, or whether a training program truly improves productivity—teams need a structured way to test whether an observed difference is strong enough to reject a “no change” assumption.

The session frames hypothesis testing around two competing statements. The null hypothesis (H0) assumes no significant effect or difference—“nothing new is happening.” The alternative hypothesis (H1) contradicts H0 and represents the effect of interest—“the new method increases/decreases the metric” or “the weight is not equal to the claimed value.” A key rule of thumb is emphasized: H0 is the status-quo baseline, and H1 is the challenge. After collecting data, the logic is not “prove H0 true,” but “reject H0 if evidence is strong enough.” Confusion is addressed directly: failing to reject H0 does not mean H0 is true; it only means the data didn’t provide sufficient evidence against it.

To operationalize the decision, the lecture introduces the “rejection region approach” as a step-by-step workflow. First, H0 and H1 are defined (e.g., average view duration is equal to 6 minutes under the old vs. new shooting technique, or a product weight equals 50 grams). Next, a significance level (α) is chosen—commonly 0.05 or 0.01—which represents the probability of rejecting H0 when H0 is actually true (linked to Type I error). Then assumptions determine which test to use (for example, a z-test when population standard deviation is known and a t-test when it isn’t). After computing a test statistic (like a t-statistic), the result is compared to critical values on a normal curve to decide whether to reject or not reject H0. The session also walks through two concrete examples: a training program productivity test (sample mean productivity rises from a baseline of 50 to 53 with n=30, σ known) and a consumer “50 grams” weight claim test (testing whether the mean weight differs from 50 using a two-sided setup with σ=4 and n=40).

The lecture then pivots to limitations of the rejection-region method: it can’t meaningfully distinguish between very close test statistics (e.g., 1.95 vs. 1.97) if both fall on the same side of the boundary. That motivates the next approach—p-value—promised for the following class. Additional foundational concepts are introduced: Type I vs. Type II errors, the trade-off controlled by α, and the difference between one-tailed and two-tailed tests based on whether the alternative hypothesis specifies “greater than,” “less than,” or “not equal to.” Finally, hypothesis testing is positioned as broadly useful across domains—evaluating interventions, comparing means and proportions, analyzing relationships, testing independence of categorical variables, and supporting machine learning tasks like model comparison, feature selection, hyperparameter tuning, and checking algorithm assumptions.