Logistic Regression's Blind Spot: Class Imbalance and the Intercept Correction Nobody Teaches

Dive straight into logistic regression on a real dataset and one weakness shows up almost immediately: class imbalance. It's not an edge case — it's closer to the default. Fraud is rare. Default is rare. Churn, relative to retention, is rare. Cancer, relative to a healthy screening population, is rare. The label you actually care about predicting is, more often than not, the minority class.

Why Imbalance Breaks Logistic Regression

Logistic regression is fit by maximising likelihood — which, in plain terms, means it's rewarded for getting the bulk of the data right. If 98% of your training rows are "no fraud," a model that leans hard toward predicting "no fraud" every time already achieves 98% accuracy and a very respectable-looking loss, without learning anything useful about the 2% that actually matters.

The failure mode is quiet, not loud. The model doesn't crash or throw an error — it just converges to a set of coefficients that are excellent at explaining the majority class and mediocre at separating the minority class, while accuracy and even the raw log-likelihood look fine. You only see it when you check recall, precision, or a confusion matrix broken out by class.

This shows up constantly outside the obvious fraud/credit-risk cases too — rare-disease screening, insurance claims, equipment failure prediction, employee attrition. Anywhere the "interesting" outcome is scarce, plain logistic regression on the raw class ratio tends to under-serve exactly the class you built the model for.

Two Common Fixes

SMOTE

Synthetic Minority Oversampling Technique generates new minority-class rows by interpolating between a real minority example and its nearest minority neighbours in feature space — not by duplicating existing rows.

Class Weighting

Instead of touching the data, reweight the loss function so a misclassified minority case costs more than a misclassified majority case. No synthetic rows, no resampling — just a different objective.

SMOTE — and Where It Falls Short

SMOTE's appeal is that it doesn't just duplicate minority rows (which would only reinforce whatever noise those specific rows carry) — it manufactures plausible new ones along the line between existing minority points. That helps, but it comes with real limitations worth knowing before you reach for it by default:

It's still synthetic

No new information enters the dataset. SMOTE interpolates within the convex hull of examples you already have — it cannot tell you anything about minority-class behaviour your original data never captured.

It can blur the decision boundary

When minority and majority classes overlap in feature space (extremely common in fraud and health data), interpolating between minority neighbours can generate synthetic points that land inside majority territory, actively confusing the classifier.

High-dimensional and categorical data degrade it

"Nearest neighbour" gets less meaningful as dimensionality grows, and SMOTE's interpolation logic doesn't map cleanly onto categorical or mixed-type features without variants like SMOTE-NC.

It doesn't fix the intercept problem below

This is the one people forget. Whether you balance classes with SMOTE or with weights, you still need the correction covered in the next section — SMOTE is not a free pass around it.

Class Weighting — and the sklearn vs. GLM Gap

Reweighting avoids manufacturing data entirely. In scikit-learn, this is close to a one-line fix:

from sklearn.linear_model import LogisticRegression
 
# 'balanced' sets weight_j = n_samples / (n_classes * count_j)
model = LogisticRegression(class_weight='balanced')
model.fit(X_train, y_train)

class_weight='balanced' computes n_samples / (n_classes × count_of_class_j) automatically, so the rare class gets a proportionally larger weight without you having to compute anything by hand.

If you're fitting through a classical GLM interface instead — statsmodels, R's glm(), or anything built on the plain IRLS solver — there is usually no class_weight='balanced' convenience switch. You compute and pass the weights yourself:

import numpy as np
import statsmodels.api as sm
 
# Replicate sklearn's 'balanced' formula manually
classes, counts = np.unique(y_train, return_counts=True)
n_samples = len(y_train)
weight_map = {c: n_samples / (len(classes) * n) for c, n in zip(classes, counts)}
weights = y_train.map(weight_map)
 
X_design = sm.add_constant(X_train)
model = sm.GLM(y_train, X_design, family=sm.families.Binomial(), freq_weights=weights)
result = model.fit()

Either route gets you a model that no longer ignores the minority class during training. But both routes leave you with the same follow-on problem, and it's the one that's easiest to skip.

The Part Everyone Skips: Recalibrating the Intercept

Balance the training sample — whether by SMOTE, undersampling, or reweighting — and the model you fit is no longer estimating probabilities against the real world. It's estimating them against whatever ratio you engineered into the training set. The slope coefficients (the relative effect of each feature) are, under reasonable assumptions, largely unaffected by this. The intercept is not. It absorbs the artificial class ratio directly, and it will keep reporting probabilities calibrated to your balanced sample, not to production.

What This Looks Like in Practice

Train on a 50/50 balanced sample when the real-world event rate is 2%, and an "average" case — one with a perfectly middling score on every feature — gets predicted at roughly 50% probability by the raw model. The true rate for an average case is nowhere near 50%. Every predicted probability downstream inherits this bias.

This is exactly the problem King & Zeng tackled in their widely cited 2001 paper "Logistic Regression in Rare Events Data" (Political Analysis). Their original motivation was case-control sampling for rare events like civil war onset, but the correction generalises directly to any rare-event classifier trained on an artificially balanced sample — fraud, default, disease diagnosis, all of it.

The Prior Correction

The fix is a closed-form shift applied to the intercept only, after fitting. If:

$\tau$ is the true population prevalence of the positive class,
$\bar{y}$ is the prevalence of the positive class in your training sample (after balancing),
$\beta_0$ is the intercept your model estimated on that balanced sample,

then the corrected intercept is:

\beta_0^{*} = \beta_0 - \ln\!\left[\frac{1-\tau}{\tau} \times \frac{\bar{y}}{1-\bar{y}}\right]

Rearranged into a form that's easier to reason about, it's simply subtracting the sample's log-odds and adding back the population's log-odds:

\beta_0^{*} = \beta_0 + \underbrace{\ln\!\left(\frac{\tau}{1-\tau}\right)}_{\text{population log-odds}} - \underbrace{\ln\!\left(\frac{\bar{y}}{1-\bar{y}}\right)}_{\text{sample log-odds}}

Every slope coefficient is left untouched. Only the intercept moves — down, if you over-sampled the rare class (the usual case), which is exactly what should happen: the model needs to become harder to convince that a case is positive, to compensate for having been trained where positives were artificially common.

Worked Example

Say the true fraud rate is

\tau = 2\%

, and training was balanced to

\bar{y} = 50\%

via SMOTE or reweighting:

\beta_0^{*} = \beta_0 - \ln\!\left[\frac{0.98}{0.02} \times \frac{0.5}{0.5}\right] = \beta_0 - \ln(49) \approx \beta_0 - 3.89

That's not a rounding correction — it's a shift of nearly four log-odds units. Left uncorrected, an average-scoring transaction would be reported at roughly 50% fraud probability instead of something close to the true 2% base rate. Any downstream threshold, expected-loss calculation, or capital estimate built on that raw probability would be badly wrong, even though the model's ranking of risky vs. safe transactions might be perfectly reasonable.

Notice what the correction does not require: refitting the model. It's applied once, after training, directly to the intercept term. That makes it cheap enough that there's rarely a good excuse to skip it once you know it's needed.

Try It Yourself — See the Shift

Adjust the true population prevalence and how aggressively the training sample was balanced. Watch the uncorrected curve (what the model reports straight out of training) separate from the corrected curve (what King & Zeng's formula gives you back) — and see exactly how large the resulting probability error is for an average case.

Push the population prevalence down toward a genuinely rare-event regime while keeping the training sample fully balanced, and the gap between the two curves widens fast — which is exactly the regime (fraud, rare disease, catastrophic default) where getting this step wrong does the most damage.

A Practical Checklist

Diagnose the imbalance

Before touching the model, check the class ratio directly. A rule of thumb: if the minority class is under ~10–15% of rows, plain unweighted logistic regression is likely to under-serve it.

Choose a balancing strategy deliberately

Class weighting is cheaper, leaves the data untouched, and is the more defensible default. Reach for SMOTE (or a hybrid like SMOTE + Tomek links) when you specifically need more minority-class density for the algorithm you're using, and validate that it isn't blurring the boundary in overlap regions.

Apply the prior correction

Once training is done, shift the intercept using

\beta_0^{*} = \beta_0 + \ln(\tau/(1-\tau)) - \ln(\bar{y}/(1-\bar{y}))

. This is a five-line function, not a re-training exercise.

Validate on a naturally imbalanced holdout

Check calibration (a reliability diagram, or Brier score) on a held-out set that reflects the true population rate — never on a rebalanced one. That's the only way to confirm the correction actually landed.

The Takeaway

Class imbalance is close to the default condition for the problems logistic regression gets used on in practice, not an exception. SMOTE and class weighting both address the training half of that problem — but neither one, on its own, gives you probabilities that mean what they say once the model leaves the balanced sample it was trained on. The King & Zeng prior correction is the missing second half: a one-line adjustment to the intercept that costs nothing to apply and is the difference between a model that ranks cases sensibly and one whose actual probability outputs can be trusted.

Why Imbalance Breaks Logistic Regression#

Two Common Fixes#