What Is PCA?
PCA is a mathematical technique that takes a dataset with many correlated variables and rotates it so that the maximum possible variance is captured along the first new axis — and so on for each subsequent axis, orthogonally.
Imagine you have 10 different financial variables measured across 500 banks — things like loan-to-deposit ratios, net interest margins, capital adequacy ratios, and so on. Many of these variables are correlated with each other. PCA finds a new coordinate system where the axes (called principal components) are uncorrelated, ordered from most to least variance explained.
The result? You might discover that 3 components capture 85% of all variation in your 10-variable dataset. You've compressed 10 dimensions into 3 without losing much information.
Core Intuition
PCA doesn't delete variables. It rotates and projects your data cloud onto new axes that are mathematically guaranteed to capture maximum variance. Think of it as finding the "best angle" to photograph a 3D sculpture so the 2D photo shows as much detail as possible.
Each dot in the 3D cloud is an observation. The golden arrow is PC1 — the axis of maximum variance. The blue arrow is PC2 — orthogonal to PC1, capturing the next most variance. PC3 (perpendicular to both) captures whatever remains.
02 — Motivation
Why Does PCA Even Exist?
Three fundamental problems in data analysis motivated PCA's invention — and all three are still painfully real today.
Problem 1 — The Curse of Dimensionality
As dimensions grow, the volume of space grows exponentially. Data becomes sparse. Distance metrics lose meaning. Models overfit. With 50 variables, you need millions of observations to fill the space adequately. Most machine learning models quietly fall apart in high dimensions.
Problem 2 — Multicollinearity
In regression, correlated predictors make coefficient estimates unstable and uninterpretable. If deposit rate and funding cost move together 90% of the time, how do you separately estimate their effects? PCA creates uncorrelated components that can enter models cleanly — no redundant information, no inflated standard errors.
Problem 3 — Visualization
Humans can perceive at most 3 dimensions. PCA provides a principled way to project 50-dimensional data onto 2 or 3 axes that preserve as much structure as possible — so we can actually see clusters, outliers, and patterns.
A Historical Note
PCA was invented by Karl Pearson in 1901 and independently developed by Harold Hotelling in 1933. Pearson framed it geometrically — finding the line of closest fit to a set of points in multidimensional space. Hotelling re-derived it algebraically in terms of eigenvectors of the covariance matrix. Both formulations are equivalent and together gave us the modern PCA algorithm.
Where Is PCA Used Today?
The list is enormous. Some domains:
Finance & BankingGenomics & BioinformaticsImage CompressionNLP & Text AnalysisClimate ScienceRisk Factor ModelsFace RecognitionNeuroscience
In banking specifically, PCA is used to construct uncorrelated risk factors from correlated yield curve points — exactly the kind of IRRBB work where you don't want 20 correlated tenor points but 3–5 independent factors (level, slope, curvature, etc.).
03 — The Mathematics
The Formula & How PCA Works
PCA is fundamentally an eigendecomposition of the covariance matrix. Let's build this up step by step with real numbers.
Step 1 — Your Raw Data Matrix X
Say we have n = 8 observations and p = 4 variables: deposit rate, loan rate, NIM (net interest margin), and funding cost — all measured across 8 hypothetical banks.
| Bank | Deposit Rate (%) | Loan Rate (%) | NIM (%) | Funding Cost (%) |
|---|---|---|---|---|
| A | 2.1 | 5.8 | 3.7 | 2.4 |
| B | 2.8 | 6.5 | 3.7 | 3.0 |
| C | 1.5 | 4.9 | 3.4 | 1.8 |
| D | 3.2 | 7.1 | 3.9 | 3.5 |
| E | 1.9 | 5.2 | 3.3 | 2.1 |
| F | 2.5 | 6.0 | 3.5 | 2.7 |
| G | 3.5 | 7.8 | 4.3 | 3.8 |
| H | 1.3 | 4.5 | 3.2 | 1.6 |
Step 2 — Mean-Center and Standardize
We subtract the mean of each column so every variable is centered at zero. If variables are on different scales, we also divide by the standard deviation.
After this step, every variable has mean = 0 and standard deviation = 1.
Step 3 — Compute the Covariance Matrix Σ
The covariance matrix Σ is a p × p symmetric matrix that captures how each pair of variables co-varies. Each entry .
Dep.R Loan.R NIM Fund.C
Dep.R [ 0.48 0.51 0.14 0.50 ]
Loan.R [ 0.51 0.99 0.28 0.98 ]
NIM [ 0.14 0.28 0.12 0.27 ]
Fund.C [ 0.50 0.98 0.27 0.96 ]
Notice the high off-diagonal numbers — these variables are strongly correlated. PCA will find axes where these correlations vanish.
Step 4 — Eigendecomposition
This is the heart of PCA. We solve:
Where is the eigenvalue (how much variance this direction captures) and is the eigenvector (the direction itself — the principal component axis). Sorted descending: .
For our 4-variable example, the eigenvalues come out as:
| Component | Eigenvalue (λ) | Variance Explained | Cumulative |
|---|---|---|---|
| PC1 | 2.45 | 61.3% | 61.3% |
| PC2 | 0.98 | 24.5% | 85.8% |
| PC3 | 0.42 | 10.5% | 96.3% |
| PC4 | 0.15 | 3.7% | 100.0% |
Key Insight
The sum of all eigenvalues = number of standardized variables = 4. Each eigenvalue tells you: this component is responsible for λᵢ / Σλ of the total variance. PC1 alone explains 61.3%. PC1 + PC2 together explain 85.8%. We've compressed 4 variables to 2 dimensions and only lost 14.2% of information.
Step 5 — Compute Scores (Project Data)
The actual PCA "scores" — the coordinates of each observation in the new PC space — are computed by projecting the standardized data onto the eigenvectors:
Where is the standardized data matrix (n × p), is the matrix of eigenvectors (p × p), and is the scores matrix (n × p). Score for observation on PC1:
This is a simple linear combination — you're projecting each observation onto each principal axis.
04 — Preprocessing
Why We Standardize (Z-Score Scale)
This is one of the most misunderstood steps in PCA. Failing to standardize is one of the most common mistakes — and it completely distorts your results.
PCA finds directions of maximum variance. This means it is dominated by variables with large numerical scales. Consider mixing GDP (in billions) with an interest rate (0–10%). GDP will have a variance millions of times larger than the rate — so PC1 will just be "the direction of GDP" and ignore everything else.
❌ Without Standardization
GDP (billions) → σ² = 250,000
Interest Rate (%) → σ² = 0.8
PC1 captures almost nothing but GDP. The rate is invisible. PCA is just measuring units, not structure.
✓ With Standardization
GDP (z-scored) → σ² = 1.0
Interest Rate (z-scored) → σ² = 1.0
Each variable contributes equally to start. PCA now finds genuine structural patterns, not unit artifacts.
The Formula
Where is the mean of variable and is the standard deviation of variable . After this, PCA on Z is equivalent to eigendecomposition of the correlation matrix (instead of the raw covariance matrix).
When NOT to Standardize
05 — How Many Components?
If all your variables are on the same scale and you want variables with more variance to have more influence — for example, all variables are returns in %, and high-variance ones are genuinely more important — you can skip standardization and work directly with the covariance matrix. In banking rate analysis, this can make sense when all rates are in basis points.
The Eigenvalue Rule (Kaiser Criterion)
You've done PCA. Now you have p principal components. How many do you actually keep? The eigenvalue (Kaiser) rule is the most widely used starting point.
Kaiser Criterion
Keep all components with eigenvalue λ > 1.0 (when using the correlation matrix / standardized data).
Why λ > 1?
When you standardize your data, every original variable has variance = 1. So each variable "contributes 1 unit of variance." An eigenvalue of 1.0 means a principal component explains exactly as much variance as one original variable. If a component explains less than one variable, keeping it is worse than just keeping the original variable — so we discard it.
With standardized data (correlation matrix):
Σ eigenvalues = p (total number of variables)
Each original variable contributes exactly 1.0
Kaiser rule: keep PC if λᵢ > 1.0
For our 4-variable example:
λ₁ = 2.45 → KEEP (explains more than 1 variable)
λ₂ = 0.98 → borderline, discard
λ₃ = 0.42 → DISCARD
λ₄ = 0.15 → DISCARD
By the eigenvalue rule, we'd keep just PC1, which explains 61.3% of variance. This is often too conservative — which is why we also use the scree plot and proportion threshold.
Criticism of Kaiser Rule
06 — Component Selection
The λ > 1 threshold is arbitrary and can be too strict or too lenient depending on the dataset. A component with λ = 0.99 (discarded) vs λ = 1.01 (kept) is not meaningfully different. Always use this in conjunction with the scree plot and domain knowledge.
Scree Plot, Elbow, & Proportion Threshold
The scree plot is a visual tool that plots eigenvalues (or % variance explained) against component number. The "elbow" in the curve tells you where adding more components stops being useful.
The Scree Plot — Larger Example (10 Variables)
Using 10 banking variables: deposit rate, loan rate, OPR, KLIBOR, NIM, funding cost, credit spread, FX rate, CPI, and GDP growth.
The Elbow Criterion
Look for the point where the scree plot "bends" — where the drop in eigenvalue becomes much smaller. This elbow point is where adding more components gives diminishing returns.
Reading the Elbow
In the plot above, there's a sharp drop from PC1 → PC2 → PC3, then the curve flattens out. The elbow is after PC3, suggesting we keep 3 components which capture about 62% of variance in the 10-variable system.
Proportion Threshold Criterion
Keep enough components to explain at least X% of total variance. Common thresholds by field:
| Field | Common Threshold | Rationale |
|---|---|---|
| Social Sciences | 70–75% | High noise tolerance |
| Finance / Economics | 75–85% | Balance signal vs parsimony |
| Engineering / Physics | 90–95% | Precision required |
| Machine Learning (preprocessing) | 95–99% | Retain near-full information |
Decision Framework — Putting It Together
1
Apply Kaiser Rule (λ > 1) as first filter
Gives you a quick upper bound on components to consider.
2
Plot the scree and identify the elbow
Often points to fewer components than Kaiser. Usually more defensible.
3
Check the cumulative variance threshold
Does your chosen number of components meet the minimum needed for your domain?
4
Apply domain knowledge
In yield curve PCA, we know level + slope + curvature = 3 components. Theory should validate, not override, the statistics.
What Are Factor Loadings?
Factor loadings are the coefficients that tell you how much each original variable contributes to each principal component. They are the entries of the eigenvector matrix V.
Going back to our 4-variable banking example. After eigendecomposition, we get the following eigenvector matrix — each column is a principal component, each row is an original variable:
| Variable | PC1 | PC2 | PC3 | PC4 |
|---|---|---|---|---|
| Deposit Rate | 0.51 | 0.47 | −0.63 | 0.34 |
| Loan Rate | 0.55 | −0.32 | 0.21 | −0.74 |
| NIM | 0.39 | 0.82 | 0.41 | 0.02 |
| Funding Cost | 0.53 | −0.11 | 0.61 | 0.58 |
How to Read a Loading
Loading values range from −1 to +1. Think of it like a correlation:
Large positive loading (+0.5 to +1.0)
This variable moves strongly in the same direction as the component. High score on this PC → high value on this variable.
Large negative loading (−0.5 to −1.0)
This variable moves opposite to the component. High score on this PC → low value on this variable.
Near-zero loading (−0.2 to +0.2)
This variable has almost no relationship with this component. It's explained by other PCs.
The Loading Formula
Eigenvector loading (unit vector): = loading of variable on PC
Scaled loading (correlation between variable and PC):
Communality — variance of variable explained by all kept PCs:
Example — Deposit Rate with 2 PCs kept:
85.3% of Deposit Rate's variance is explained by PC1 + PC2. If communality is high (close to 1.0), the variable is well-represented in your retained components. If it's low, you're losing a lot of that variable's information in the compression.
08 — Interpretation
How to Interpret Principal Components
PCA gives you mathematical components, but interpreting what they mean economically or substantively is where the real analytical skill comes in. Let's do this step by step with our banking example.
Step 1 — Look at the Loadings
For each PC, identify which variables have loadings with absolute value > 0.4 (a common informal threshold):
PC1
The General Banking Rate Factor
Explains 61.3% of variance
High loadings on ALL four variables (0.51, 0.55, 0.39, 0.53 — all positive, all reasonably strong). This component rises when all rates rise together and falls when all rates fall. It represents the general level of the interest rate environment.
💡 A bank with a high PC1 score is operating in a high-rate environment across all dimensions. When the OPR is high, this factor is high — and it drags deposit rates, loan rates, NIM, and funding cost all up together. In yield curve terms, this is the level factor — a parallel shift in the curve.
PC2
The Profitability Squeeze Factor
Explains 24.5% of variance
High loading on NIM (+0.82), moderate negative on Loan Rate (−0.32), near-zero on Funding Cost (−0.11). This component separates banks where NIM is high relative to rates vs where it's compressed.
💡 A high PC2 score = healthy NIM despite rates. A low PC2 score = NIM being squeezed. This is the "spread vs cost" axis — directly useful for diagnosing IRRBB exposure from repricing gaps. A bank with persistently low PC2 scores warrants closer scrutiny of its asset-liability mismatch.
PC3
The Liability Structure Factor
Explains 10.5% of variance
High negative on Deposit Rate (−0.63), high positive on Funding Cost (+0.61). This picks up situations where deposit rates are suppressed but funding costs are elevated — or vice versa.
💡 This could capture the COVID-era dynamic — deposit rates stayed low (rate floor behavior, excess liquidity) while wholesale funding costs diverged. Banks that relied more on market funding would score differently on PC3 than retail-deposit-heavy banks. In IRRBB terms, this is a liability structure / funding mix signal.
Step 2 — Name Your Components
Once you've identified the loading structure, give each component a substantive name based on what it captures. These names are always provisional — they reflect your analytical judgment about the underlying economic story.
Pro Tip — What makes a good PC name?
Good names capture the contrast embedded in the loadings. "PC1 = High rates" is too vague. "PC1 = Overall Rate Level Factor" or "PC1 = Parallel Shift Factor" (in yield curve terms) communicates that all variables move together on this axis.
Step 3 — Interpret the Scores
Each observation gets a score on each PC. Here's what our 8 banks look like in PC1–PC2 space:
| Bank | PC1 Score | PC2 Score | Interpretation |
|---|---|---|---|
| A | −0.42 | +0.18 | Moderate rates, adequate NIM |
| B | +0.55 | +0.21 | Higher rates, healthy NIM |
| C | −1.21 | −0.33 | Low rate environment, squeezed NIM |
| D | +1.08 | +0.44 | High rates, strong NIM — low IRRBB risk |
| E | −0.73 | −0.55 | Low rates, NIM under pressure |
| F | +0.12 | +0.08 | Near-average on both dimensions |
| G | +1.45 | +0.67 | Highest rate environment, best NIM |
| H | −1.52 | −0.42 | Lowest rates, worst NIM — highest exposure |
The Key Insight
Bank H has PC1 = −1.52 (extreme low-rate environment) and PC2 = −0.42 (NIM squeezed). It clusters near Banks C and E. Together, these three banks share a vulnerability profile. This clustering in PC space is information that was buried across 4 correlated variables — PCA surfaced it.
A Note on Sign Indeterminacy
PCA eigenvectors have an arbitrary sign flip — the direction is real but the positive/negative orientation can be reversed depending on implementation. PC1 might show "high rate environment" or "low rate environment" with opposite signs depending on the software. Always check the loadings to establish economic directionality, and feel free to flip the sign of a component for interpretive clarity. The math is identical either way.
Reconstructing Original Data
One final powerful property — you can reconstruct an approximation of the original data from your retained components:
Where is scores on first PCs (n × k), is first eigenvectors transposed (k × p), is the diagonal matrix of standard deviations, and is the original variable means. With PCs (85.8% variance explained), the reconstruction recovers most of the original data structure. This is the basis of PCA-based image compression (eigenfaces, etc.).
09 — Putting It All Together
The Full PCA Workflow
01
Collect your data matrix X (n × p)
Ensure variables are numeric and continuous. Handle missing values first.
02
Standardize: z-score each variable
Z = (X − μ) / σ per column. Ensures PCA works on correlations, not units.
03
Compute the covariance / correlation matrix
Σ = ZᵀZ / (n−1). This is a p×p symmetric positive semi-definite matrix.
04
Eigendecompose: Σ = V Λ Vᵀ
Sort eigenvalues descending. Column i of V = direction of PC i. λᵢ = variance explained by PC i.
05
Decide k: Kaiser (λ > 1) + Scree Elbow + Variance Threshold
Use all three in combination. Apply domain knowledge to validate.
06
Compute scores: T = Z · Vₖ
Project your data into the new k-dimensional PC space.
07
Interpret: examine loadings, name components, plot biplots
Give each PC a substantive economic interpretation from its loading structure.
08
Use the scores downstream
PC scores are uncorrelated by construction — ideal as inputs to regression, clustering, risk models, or visualization.
Final Thought
PCA is not magic — it cannot create information that isn't in your data. What it does is reveal the geometric structure that was already there, hidden by correlated variables. When you see that 3 components explain 85% of a 20-variable system, you've discovered that your data actually lives in a 3-dimensional subspace — and those 3 dimensions often have a real-world interpretation waiting to be named.