Random Oversampler in Machine Learning

The Class Imbalance Problem

Real-world datasets are almost never balanced. Fraudulent transactions might represent 0.1% of all payment activity at PhonePe. Manufacturing defects occur in 0.5% of products on a Tata Motors assembly line. Rare cancers appear in fewer than 1 in 10,000 medical scans. In each case, the class you care about most -- the minority -- is massively outnumbered.

Standard ML algorithms optimize for overall accuracy. A fraud detector that always predicts "legitimate" achieves 99.9% accuracy on a 1000:1 imbalanced dataset while catching exactly zero frauds. The model learned the optimal shortcut: ignore the minority entirely.

The Simplest Fix: Just Copy the Minority

Random oversampling is the most intuitive response to this problem. If the model ignores the minority class because there aren't enough samples, give it more. How? Pick minority samples at random (with replacement) and duplicate them until you reach the desired class ratio.

This approach predates all the sophisticated techniques. Before Chawla et al. introduced SMOTE in 2002, random oversampling was the primary data-level strategy for addressing class imbalance. He and Garcia's influential 2009 IEEE TKDE survey, Learning from Imbalanced Data, identifies random oversampling as one of the foundational resampling methods alongside random undersampling.

Why It Persists Despite Known Limitations

You might expect that SMOTE and its variants would have replaced random oversampling entirely. They haven't, for several reasons:

Simplicity: Random oversampling has zero hyperparameters in its basic form (no k-neighbors to tune) and is trivially parallelizable. In a production pipeline with 50+ components, simplicity compounds.

Speed: No k-NN search means $O(n)$ computational cost instead of $O(n^2)$ or $O(n \log n)$. For datasets with millions of minority samples, this difference matters enormously.

Universality: Random oversampling works with any data type -- tabular, text embeddings, image features, categorical, mixed. SMOTE requires a meaningful distance metric and continuous features. When your features include sparse text embeddings and categorical flags, random oversampling just works.

Competitive performance: Multiple empirical studies have shown that random oversampling performs comparably to SMOTE on many real-world benchmarks, particularly when paired with regularized models (tree ensembles, L2-regularized logistic regression) that naturally resist overfitting from duplicated samples.

Historical Note: The ROSE (Random Over-Sampling Examples) extension, proposed by Menardi and Torelli in 2014, added smoothed bootstrap to random oversampling -- creating perturbed copies rather than exact duplicates. This was integrated into imbalanced-learn as the shrinkage parameter, effectively modernizing the oldest oversampling technique.

The Core Idea: More Copies, More Attention

Think of it this way: you're training a model that processes examples one by one (or in mini-batches). If class A has 10,000 samples and class B has 100 samples, class A's patterns get reinforced 100x more often during training. The model's parameters drift heavily toward predicting class A because that's where the gradient signal overwhelmingly comes from.

Random oversampling fixes the imbalance at the data level. By duplicating class B's 100 samples until you have 10,000, each class now contributes equally to the training signal. The model can no longer take the shortcut of ignoring class B.

The Library Analogy

Imagine a library where 99 shelves contain Hindi novels and 1 shelf contains Urdu poetry. If a student randomly browses shelves, they'll almost never encounter Urdu poetry. Random oversampling is equivalent to photocopying the Urdu poetry collection and placing copies on 98 additional shelves. Now the student encounters both genres equally often.

The limitation is obvious: the student sees the same poems over and over. They might memorize specific poems rather than learning what makes Urdu poetry distinctive in general. This is the overfitting risk of naive random oversampling.

The Smoothed Bootstrap Extension

Here's where the ROSE (smoothed bootstrap) idea gets clever. Instead of making exact photocopies, imagine a slightly imperfect photocopier that introduces tiny random smudges. Each copy is slightly different from the original -- not a new poem, but not an identical reproduction either.

Mathematically, the smoothed bootstrap draws a sample from the original minority class, then adds a small Gaussian perturbation controlled by the shrinkage parameter. Higher shrinkage means more perturbation (blurrier copies), while shrinkage of 0 means exact duplication (perfect copies). This small modification can significantly reduce overfitting while maintaining the simplicity of the random oversampling framework.

What Random Oversampling Does NOT Do

This is important to internalize: random oversampling does not create new information. Whether you're duplicating exactly or perturbing slightly, the synthetic samples don't reveal patterns that weren't already present in the original data. If your 100 minority samples are noisy, mislabeled, or unrepresentative, oversampling amplifies those problems.

The technique assumes your original minority samples are reasonably representative of the true minority distribution. If they're not, no amount of duplication or perturbation will fix that -- you need better data collection.

Expert Insight: Random oversampling is equivalent to adjusting class weights in the loss function when training with SGD. Both approaches increase the effective contribution of minority samples to gradient updates. The practical difference is that oversampling changes the data, while class weights change the optimization -- but the mathematical effect on expected gradients is identical for exact duplication.

Mathematical Formulation

Let $D = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ be a training set with $y_i \in \{0, 1\}$, where class 1 (minority) has $n_{\text{min}}$ samples and class 0 (majority) has $n_{\text{maj}}$ samples, with imbalance ratio $IR = \frac{n_{\text{maj}}}{n_{\text{min}}} > 1$.

Naive Random Oversampling

To produce a balanced dataset, generate $n_{\text{maj}} - n_{\text{min}}$ additional minority samples by sampling uniformly at random with replacement from the existing minority set $D_{\text{min}} = \{\mathbf{x}_i : y_i = 1\}$:

$\mathbf{x}_{\text{new}} = \mathbf{x}_j, \quad j \sim \text{Uniform}(\{i : y_i = 1\})$

The resampled dataset is $D' = D \cup D_{\text{new}}$ where $|D_{\text{new}}| = n_{\text{maj}} - n_{\text{min}}$, yielding a balanced class distribution.

Smoothed Bootstrap (ROSE)

The smoothed variant, proposed by Menardi and Torelli (2014), adds a Gaussian perturbation to each drawn sample:

$\mathbf{x}_{\text{new}} = \mathbf{x}_j + \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, s^2 \hat{\boldsymbol{\Sigma}}_k)$

where:

• $\mathbf{x}_j$ is a randomly selected minority sample
• $s$ is the shrinkage factor (scalar controlling perturbation magnitude)
• $\hat{\boldsymbol{\Sigma}}_k$ is the estimated covariance matrix of class $k$

The covariance matrix is typically the sample covariance of the minority class, and the optimal shrinkage under multivariate normality assumptions follows the Silverman bandwidth rule:

$s_{\text{opt}} = n_{\text{min}}^{-1/(d+4)}$

where $d$ is the feature dimensionality.

Equivalence to Class Weights

For models trained with stochastic gradient descent, random oversampling with exact duplication is mathematically equivalent to applying class weights $w_k = \frac{n}{2 \cdot n_k}$ to the loss function. In both cases, the expected gradient contribution from each class is equalized:

$\mathbb{E}[\nabla \mathcal{L}_{\text{oversampled}}] = \mathbb{E}[\nabla \mathcal{L}_{\text{weighted}}]$

This equivalence breaks down for the smoothed bootstrap variant, which modifies the data distribution rather than just the sampling probability.

Sampling Strategy

The sampling_strategy parameter defines the target ratio $\rho$:

• 'auto': balance to 50-50, i.e., $|D'_{\text{min}}| = n_{\text{maj}}$
• float $\rho \in (0, 1]$: target $\frac{|D'_{\text{min}}|}{n_{\text{maj}}} = \rho$
• dict: explicit target counts per class

Computational Complexity

• Naive oversampling: $O(n_{\text{new}})$ for sampling + $O(n_{\text{new}} \cdot d)$ for copying feature vectors
• Smoothed bootstrap: $O(n_{\text{new}} \cdot d)$ for perturbation + $O(n_{\text{min}} \cdot d^2)$ for covariance estimation (one-time)

Both are dramatically cheaper than SMOTE's $O(n_{\text{min}}^2 \cdot d)$ k-NN search.

Mathematical Caveat: The smoothed bootstrap assumes the minority class distribution is approximately Gaussian around each sample. For highly non-Gaussian distributions (multi-modal, heavy-tailed), large shrinkage values can push synthetic samples into regions of feature space occupied by the majority class, degrading performance.

Random oversampling is the simplest and most universal technique for addressing class imbalance in machine learning. At its core, it duplicates minority class samples with replacement until the desired class balance is achieved. Despite its simplicity, it remains a production workhorse -- competitive with SMOTE on many real-world benchmarks, particularly when paired with regularized tree ensembles.

The technique's evolution is worth noting. The modern implementation in imbalanced-learn extends naive duplication with the shrinkage parameter (inspired by the ROSE method of Menardi and Torelli, 2014), enabling smoothed bootstrap sampling that adds controlled Gaussian perturbation to duplicated samples. This single parameter transforms random oversampling from a blunt instrument prone to overfitting into a nuanced tool with a smooth tradeoff between duplication fidelity and sample diversity.

The key decisions for practitioners are: (1) naive vs. smoothed -- use shrinkage > 0 when working with models susceptible to memorization (neural networks, k-NN) and shrinkage=None for tree ensembles; (2) oversampling vs. class weighting -- they're mathematically equivalent for SGD-trained models, so prefer class weighting when the dataset size increase from oversampling would be costly; and (3) random oversampling vs. SMOTE -- prefer random oversampling for mixed/categorical features, very small minority classes, and sparse data where SMOTE's interpolation produces invalid samples.

Random oversampling endures because it embodies a fundamental principle of engineering: the simplest solution that works is often the best solution. It's the technique you start with, the baseline you measure against, and -- more often than you'd expect -- the technique you ship to production.