SMOTE: Synthetic Minority Over-sampling Technique

This paper addresses a central problem in supervised machine learning: how to build accurate classifiers when training data are imbalanced, meaning one class (the “minority,” e.g., abnormal cases) is much rarer than the other (the “majority,” e.g., normal cases). The authors emphasize that in many real applications the cost of false negatives (missing an abnormal case) is far higher than the cost of false positives. In such settings, conventional accuracy is misleading; a classifier can achieve very high accuracy by predicting the majority class almost always, while still performing poorly on the minority class. The research question is therefore: can a new resampling strategy—specifically, synthetic minority over-sampling combined with majority under-sampling—produce better classifier performance than existing approaches such as plain majority under-sampling, minority over-sampling by replication (with replacement), or cost/prior adjustments in standard learners?

The significance of the work is twofold. First, it reframes evaluation for imbalance using ROC analysis rather than accuracy, arguing that performance should be assessed across tradeoffs between true positive rate (TPR) and false positive rate (FPR). Second, it introduces SMOTE (Synthetic Minority Over-sampling Technique), a method that generates new minority examples by interpolation in feature space rather than duplicating existing minority points. This is motivated by the observation that naive replication can make decision regions for the minority class overly specific (and can increase tree complexity), whereas interpolation can broaden the minority region and improve generalization.

Methodologically, the paper proposes SMOTE and then evaluates it experimentally across multiple datasets and three base classifiers: C4.5 decision trees, Ripper rule learners, and Naive Bayes. The study design is comparative and uses ROC curves as the primary evaluation tool. For each dataset, they generate ROC curves by training a classifier on a sequence of modified training sets. The modification process is: (i) over-sample the minority class to a specified degree using SMOTE (or, for comparison, over-sample by replication), and then (ii) under-sample the majority class to varying degrees so that each ROC point corresponds to a consistent majority sample count relative to the under-sampling schedule. Performance is summarized using AUC (area under the ROC curve) computed via a trapezoidal rule, and also using the ROC convex hull strategy, which identifies potentially optimal classifiers independent of specific cost distributions.

The paper reports experiments using nine datasets with widely varying sizes and imbalance ratios, including Pima Indians Diabetes (768 samples; 268 minority), Phoneme (5 features; 3,818 vs 1,586), Adult (48,842; 11,687 minority), E-state (53,220; 6,351 active), Satimage (2-class collapse; 5,809 majority vs 626 minority), Forest Cover (two-class extraction; 35,754 vs 2,747), Oil (896 non-oil vs 41 oil), Mammography (11,183; 260 calcifications), and Can (443,872; 8,360 “very interesting”). For the mammography example used to illustrate the mechanism, they start with 10-fold cross-validation training sets containing 10,923 majority and 260 minority examples, and after applying SMOTE at various rates they report training-set sizes such as approximately 9,831 majority and 233 minority for the baseline training set used in cross-validation (the paper’s figures then vary the minority over-sampling degree).

Analysis techniques include: (1) ROC curve generation across resampling levels, (2) AUC computation for C4.5 with the best highlighted results in a table, and (3) ROC convex hull computation (using Graham’s algorithm) to determine which resampling configurations yield potentially optimal tradeoffs. They also compare SMOTE against approaches that directly modify learning costs: varying Ripper’s loss ratio from 0.9 down to 0.001, and varying Naive Bayes class priors for the minority class up to 50 times the original minority prior.

The key methodological contribution is SMOTE itself. For each minority instance, SMOTE finds its k nearest minority neighbors (the implementation uses k = 5) and generates synthetic points along the line segments connecting the instance to randomly chosen neighbors. If the desired over-sampling rate is N%, then the number of synthetic samples generated is $$\lfloor \frac{N}{100}\rfloor \cdot T$$ where T is the number of minority samples (with additional handling when N < 100%). Each synthetic sample is created by taking the feature difference between the minority instance and a chosen neighbor and scaling it by a random gap in $[0,1]$, then adding it back to the original instance: $$x_{\text{new}}=x_i+\text{gap}\cdot (x_{\text{nn}}-x_i).$$ This interpolation is intended to “generalize” the minority decision region rather than make it narrower.

The paper’s main findings are qualitative dominance results in ROC space and quantitative AUC improvements for C4.5. Across most datasets and configurations, the combined SMOTE + majority under-sampling approach produces ROC curves that dominate plain majority under-sampling and also yields more points on the ROC convex hull than competing methods. The authors state that for almost all ROC curves, SMOTE-based classifiers dominate, and that “out of a total of 48 experiments performed, SMOTE-classifier does not perform the best only for 4 experiments.” They also identify specific exceptions: for the Pima dataset, Naive Bayes dominates over SMOTE-C4.5 in ROC space; for the Oil dataset, Under-Ripper dominates over SMOTE-Ripper; and for the Can dataset, SMOTE-classifier and Under-classifier ROC curves overlap for most of the ROC space.

Quantitatively, Table 3 reports AUC values for C4.5 under the “Under” baseline and under SMOTE at different over-sampling degrees (50% through 500%, with best values highlighted). For example: Pima AUC improves from 7242 (under) to 7307 (SMOTE 100%); Phoneme from 8622 to 8661 (SMOTE 200%); Satimage from 8900 to 8979 (SMOTE 200%); Forest Cover from 9807 to 9849 (SMOTE 300%); Oil shows a decrease from 8524 (under) to 8368 (SMOTE 200%) and 8161 (SMOTE 300%), with best SMOTE AUC 8537 at 500% (still close to baseline); Mammography improves from 9260 (under) to 9330 (SMOTE 400%); E-state improves from 6811 to 6828 (SMOTE 200%); Can improves from 9535 to 9560 (SMOTE 100%). While the table does not provide p-values or confidence intervals, the consistent ROC dominance and convex hull behavior support the claim of improved minority-class discrimination.

The authors also provide mechanistic evidence explaining why SMOTE can outperform replication. They show (using decision tree behavior on mammography) that replication tends to create smaller, more specific minority decision regions that can lead to overfitting and larger trees, whereas SMOTE creates broader minority regions (illustrated by dashed decision boundaries in their figures) and yields better minority recognition at higher over-sampling degrees.

Limitations are not exhaustively quantified, but several are apparent from the methodology and discussion. First, SMOTE operates in continuous feature space; the paper notes that the original SMOTE does not handle all-nominal datasets and introduces SMOTE-NC as an extension for mixed nominal/continuous features. However, their SMOTE-NC experiments on Adult show worse performance than plain under-sampling, suggesting that synthetic generation in mixed spaces can be problematic. Second, SMOTE depends on the choice of k and on the geometry of the minority class in feature space; if minority points are close to majority regions, interpolation can generate synthetic points that fall into majority territory, increasing false positives (they hypothesize this for Adult and for high SMOTE degrees). Third, the evaluation uses cross-validation and ROC/AUC summaries but does not report statistical significance tests across runs.

Practical implications are clear: practitioners dealing with imbalanced classification should consider generating synthetic minority examples via interpolation and combining this with majority under-sampling, then evaluating with ROC/AUC and/or ROC convex hull rather than accuracy. The method is especially relevant for domains like fraud detection, medical diagnosis, and rare-event detection where minority recall is critical. The paper also suggests that simply tuning decision thresholds or adjusting cost/loss ratios may be less effective than altering the training distribution in a way that better reflects minority structure.

In summary, this work introduces SMOTE and demonstrates—across multiple datasets and three learning algorithms—that synthetic minority over-sampling combined with majority under-sampling often yields better ROC performance than plain under-sampling, replication-based over-sampling, or cost/prior tuning, with a small number of dataset-specific exceptions. The paper’s core contribution is both algorithmic (SMOTE) and methodological (ROC-based evaluation and convex hull analysis for imbalanced learning).