SMOTE-NC in Machine Learning

The Mixed Data Problem

Vanilla SMOTE operates by finding k-nearest neighbors in feature space and generating synthetic samples via linear interpolation: $\mathbf{x}_{\text{synth}} = \mathbf{x}_i + \lambda \cdot (\mathbf{x}_j - \mathbf{x}_i)$ where $\lambda \in [0, 1]$. This works beautifully for continuous features: interpolating between salary=50,000 and salary=70,000 to get salary=62,000 is perfectly valid.

But consider a dataset with a department feature encoded as: engineering=0, marketing=1, finance=2. If SMOTE interpolates between 0 and 2, it produces 1.3 -- which maps to... nothing meaningful. It is neither engineering nor marketing nor finance. The linear interpolation assumption that underpins SMOTE breaks down entirely for categorical features.

The naive workaround -- one-hot encode all categoricals before applying SMOTE -- creates its own problems. Interpolating between [1,0,0] (engineering) and [0,0,1] (finance) produces [0.4, 0, 0.6], which is not a valid one-hot vector. You need post-processing (e.g., argmax) to recover a valid category, and the resulting distribution may not match the original data.

Chawla's Solution: Treat Feature Types Differently (2002)

In the same 2002 JAIR paper that introduced vanilla SMOTE, Chawla et al. proposed SMOTE-NC as a direct extension for mixed-type datasets. The core idea is pragmatic:

1. For continuous features: Use standard SMOTE interpolation.
2. For categorical features: Instead of interpolating, assign the most frequent category among the k-nearest neighbors of the sample being oversampled.
3. For distance computation: Use Euclidean distance on continuous features, and add a fixed penalty (the median of standard deviations of all continuous features in the minority class) for each categorical feature where the two samples disagree.

This approach preserves valid categorical values while still generating diverse synthetic samples for continuous dimensions.

Why It Matters in Practice

The need for SMOTE-NC is pervasive in real-world ML. Consider these common scenarios:

• Credit scoring at Razorpay or Lendingkart: Features include annual income (continuous), loan amount (continuous), employment type (categorical: salaried/self-employed/freelance), education level (categorical: graduate/postgraduate/diploma), and city tier (categorical: tier-1/tier-2/tier-3). The default rate is typically 3-5%, creating severe imbalance.
• Healthcare at Apollo or Narayana Health: Patient records combine blood pressure (continuous), age (continuous), diagnosis code (categorical: ICD-10), medication type (categorical), and admission type (categorical: emergency/elective/urgent). Rare conditions create extreme imbalance.
• E-commerce churn at Flipkart or Myntra: User features include monthly spend (continuous), session duration (continuous), subscription tier (categorical: free/premium/enterprise), payment method (categorical: UPI/card/wallet), and preferred category (categorical: electronics/fashion/groceries). Churn rates of 5-15% are common.

In each case, vanilla SMOTE on the raw data would produce nonsensical categorical values, and encoding-then-SMOTE introduces distribution artifacts. SMOTE-NC handles these datasets natively.

Historical Context: While SMOTE-NC was part of the original 2002 paper, it received far less attention than vanilla SMOTE for over a decade. It was not until the imbalanced-learn library added SMOTENC in 2017-2018 that mixed-type oversampling became accessible to the broader ML community. The subsequent rise of tabular ML (XGBoost, LightGBM, CatBoost) further increased demand for techniques that handle categorical features natively.

The Core Idea: Different Rules for Different Feature Types

Imagine you are a chef creating new recipes by blending existing ones. For quantitative ingredients (200g flour, 3 eggs), you can blend freely: a recipe with 200g flour and one with 300g flour might yield a valid recipe with 240g flour. But for categorical choices (oven-baked vs. pan-fried, vegetarian vs. non-vegetarian), blending makes no sense. You cannot "interpolate" between oven-baked and pan-fried to get something 40% oven-baked.

SMOTE-NC's solution is exactly this: blend the quantities, but vote on the categories. When creating a new synthetic sample, it interpolates the continuous features (like vanilla SMOTE) but for each categorical feature, it looks at the k-nearest neighbors and picks the most popular category -- a majority vote.

The Distance Problem

Before generating anything, SMOTE-NC must find the k-nearest neighbors. But how do you measure "distance" between two samples when some features are numbers and others are categories?

SMOTE-NC introduces a clever penalty system. First, it computes the standard deviation of each continuous feature across the minority class and takes the median of all those standard deviations. Call this value $M$. Now, when computing the Euclidean distance between two samples:

• For continuous features: use the standard squared difference $(x_i - x_j)^2$
• For each categorical feature where the two samples disagree: add $M^2$ to the distance
• For each categorical feature where they agree: add nothing

The intuition is elegant: $M$ represents a "typical spread" in the continuous feature space. A disagreement on a categorical feature is penalized as if the two samples were one standard deviation apart on a typical continuous feature. This puts categorical differences on a comparable scale to continuous differences without overweighting or underweighting them.

A Concrete Example

Consider two loan applicants (minority class = default):

• Applicant A: income=$50,000, credit_score=620, employment=salaried, city=tier-1
• Applicant B: income=$55,000, credit_score=640, employment=self-employed, city=tier-1

Suppose the median of standard deviations of all continuous minority features is $M = 8,000$.

Distance computation:

• income: $(50000 - 55000)^2 = 25,000,000$
• credit_score: $(620 - 640)^2 = 400$
• employment: salaried != self-employed, so add $M^2 = 64,000,000$
• city: tier-1 == tier-1, so add $0$

Total squared distance: $25,000,000 + 400 + 64,000,000 = 89,000,400$

Notice how the categorical disagreement on employment contributes more to the distance than the income difference. This reflects the intuition that switching employment types is a "bigger" change than a modest income variation.

When generating a synthetic sample from Applicant A using Applicant B as a neighbor:

• income: $50000 + 0.6 \times (55000 - 50000) = 53,000$ (interpolated)
• credit_score: $620 + 0.6 \times (640 - 620) = 632$ (interpolated)
• employment: majority vote among k neighbors (say 3 of 5 are "salaried") => salaried
• city: majority vote among k neighbors (all are "tier-1") => tier-1

Key Mental Model: Think of SMOTE-NC as two parallel systems running simultaneously: a continuous interpolation engine (identical to vanilla SMOTE) and a categorical voting engine (picks the most popular category from the neighborhood). The distance metric glues them together by making categorical disagreements contribute a fixed, calibrated penalty to the overall distance.

Mathematical Formulation

Let $D = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ be a training set with $d_c$ continuous features and $d_n$ nominal (categorical) features, where $d = d_c + d_n$. Let $S_{\text{min}} = \{\mathbf{x}_i : y_i = 1\}$ denote the minority class with $|S_{\text{min}}| = n_{\text{min}}$.

Step 1: Compute the Categorical Distance Penalty

For each continuous feature $j \in \{1, \ldots, d_c\}$, compute the standard deviation across minority class samples:

$\sigma_j = \sqrt{\frac{1}{n_{\text{min}} - 1} \sum_{\mathbf{x} \in S_{\text{min}}} (x^{(j)} - \bar{x}^{(j)})^2}$

Then compute the median of these standard deviations:

$M = \text{median}(\sigma_1, \sigma_2, \ldots, \sigma_{d_c})$

This value $M$ serves as the fixed distance penalty for each disagreeing categorical feature.

Step 2: Modified Distance Metric

The distance between two minority samples $\mathbf{x}_a$ and $\mathbf{x}_b$ is:

$d(\mathbf{x}_a, \mathbf{x}_b) = \sqrt{\sum_{j=1}^{d_c} (x_a^{(j)} - x_b^{(j)})^2 + \sum_{j=d_c+1}^{d} M^2 \cdot \mathbb{1}[x_a^{(j)} \neq x_b^{(j)}]}$

where $\mathbb{1}[\cdot]$ is the indicator function that equals 1 when the categorical values differ and 0 when they agree.

Step 3: k-Nearest Neighbor Search

For each minority sample $\mathbf{x}_i \in S_{\text{min}}$, find the $k$ nearest minority class neighbors using the modified distance metric:

$N_k(\mathbf{x}_i) = \text{k-argmin}_{\mathbf{x}_j \in S_{\text{min}} \setminus \{\mathbf{x}_i\}} d(\mathbf{x}_i, \mathbf{x}_j)$

Step 4: Synthetic Sample Generation

For each minority sample $\mathbf{x}_i$ and a randomly selected neighbor $\mathbf{x}_j \in N_k(\mathbf{x}_i)$:

Continuous features ($j \in \{1, \ldots, d_c\}$): $x_{\text{synth}}^{(j)} = x_i^{(j)} + \lambda \cdot (x_j^{(j)} - x_i^{(j)}), \quad \lambda \sim \text{Uniform}(0, 1)$

Categorical features ($j \in \{d_c+1, \ldots, d\}$): $x_{\text{synth}}^{(j)} = \text{mode}\{x_m^{(j)} : \mathbf{x}_m \in N_k(\mathbf{x}_i)\}$

where $\text{mode}$ returns the most frequent category among the k-nearest neighbors. In the case of ties, a random selection is made among the tied categories.

Complexity Analysis

• Penalty computation: $O(n_{\text{min}} \cdot d_c)$ for standard deviations + $O(d_c \log d_c)$ for median
• k-NN search: $O(n_{\text{min}}^2 \cdot d)$ for naive implementation, $O(n_{\text{min}} \cdot d \cdot \log n_{\text{min}})$ with spatial indexing
• Synthetic generation: $O(n_{\text{synth}} \cdot (d_c + k \cdot d_n))$ where $n_{\text{synth}}$ is the number of synthetic samples and the $k \cdot d_n$ term accounts for the mode computation on categorical features

Constraint

SMOTE-NC requires at least one continuous feature ($d_c \geq 1$) because the distance penalty $M$ is derived from continuous feature standard deviations. For purely categorical datasets, use SMOTEN (SMOTE for Nominal features) instead.

Mathematical Subtlety: The choice of the median (rather than the mean) of standard deviations for $M$ makes the penalty robust to outlier features with extremely high or low variance. A single continuous feature with very high variance (e.g., income in rupees vs. age in years) would inflate the mean but not the median, keeping the categorical penalty calibrated to the typical feature spread.

SMOTE-NC (Synthetic Minority Over-sampling Technique for Nominal and Continuous features) addresses a fundamental limitation of vanilla SMOTE: the inability to handle datasets containing both categorical and continuous features. Introduced alongside vanilla SMOTE in Chawla et al.'s seminal 2002 paper, SMOTE-NC provides a principled approach to generating synthetic minority samples for the mixed-type tabular data that dominates real-world ML applications -- from credit scoring with employment type and income, to healthcare with diagnosis codes and blood pressure, to churn prediction with subscription tier and monthly spend.

The algorithm's key innovations are twofold. First, it introduces a modified distance metric that bridges the gap between categorical and continuous features: the median of standard deviations of all continuous features in the minority class serves as a fixed penalty for each categorical disagreement, putting categorical differences on a comparable scale to continuous variation. Second, it uses a dual-mode generation strategy: continuous features are interpolated (like vanilla SMOTE), while categorical features are assigned the most frequent category among k-nearest neighbors (majority voting). This ensures synthetic samples always contain valid category values, unlike the nonsensical interpolations produced by applying vanilla SMOTE to encoded categoricals.

However, SMOTE-NC is not without trade-offs. The mode-based categorical assignment can homogenize synthetic samples when one category dominates within the minority class. The fixed penalty $M$ treats all categorical features equally, ignoring their varying discriminative power. The algorithm requires at least one continuous feature, fragmenting the resampling strategy for purely categorical datasets (which require SMOTEN instead). For high-cardinality categoricals, mode-based voting becomes unreliable as no single category achieves a meaningful majority among neighbors.

In production, SMOTE-NC is implemented as SMOTENC in the imbalanced-learn library with full scikit-learn pipeline compatibility. It sits in the training preprocessing stage with zero inference-time overhead. Modern alternatives like SMOTE-ENC (which uses class-conditional encoding for categoricals) and G-SMOTE-NC (which generates in geometric regions rather than line segments) address some of SMOTE-NC's limitations, but SMOTE-NC remains the most widely used and battle-tested approach for mixed-type imbalanced data. For practitioners working with tabular data in domains like fintech, healthcare, HR analytics, and telecommunications -- where mixed feature types and class imbalance are the norm rather than the exception -- SMOTE-NC is an essential tool in the data balancing arsenal.