为何PCA中计算特征值与特征向量的方法如此有效？

Great question—this cuts to the core of why PCA is such a powerful tool for simplifying multivariate data. Let’s break this down with both intuition and a touch of linear algebra, no jargon overload promised.

1. First, what the covariance matrix actually tells us #

Before diving into eigenvalues/eigenvectors, let’s recap what the covariance matrix represents. Suppose we have a centered dataset (mean subtracted from every variable) stored as a matrix X (rows = observations, columns = variables). The covariance matrix Σ is calculated as:

Σ = (XᵀX) / (n-1)

• The diagonal entries of Σ are the variances of each original variable (how much that variable fluctuates on its own).
• The off-diagonal entries are the covariances between pairs of variables (how much two variables move together linearly).

In short, Σ encodes the entire linear relationship structure of your data—both individual variable spread and cross-variable correlations.

2. Eigenvectors: Orthogonal directions of maximum variance #

Here’s the critical piece: the eigenvectors of Σ are orthogonal (perpendicular) directions in your data space where the variance is maximized.

By definition, an eigenvector v and corresponding eigenvalue λ satisfy:

Σv = λv

What does this mean in plain terms? If you take all your data points and project them onto the direction of v, the variance of those projected values (the "score" for this component) is exactly λ. And since Σ is a symmetric matrix (covariance matrices always are), its eigenvectors are guaranteed to be orthogonal to each other. That means projecting onto one eigenvector direction doesn’t overlap with projections onto any other—no shared linear information between them.

3. Why scores are mutually independent #

When you multiply your centered data X by the matrix of eigenvectors V (where each column is an eigenvector of Σ), you get the PCA scores matrix Z = XV.

To see why these scores are independent, let’s calculate the covariance matrix of Z:

Cov(Z) = (ZᵀZ)/(n-1) = (VᵀXᵀXV)/(n-1) = VᵀΣV

But since ΣV = VΛ (where Λ is a diagonal matrix filled with the eigenvalues), substituting that in gives:

VᵀΣV = VᵀVΛ = IΛ = Λ

Λ is a diagonal matrix—all off-diagonal entries are 0. That means the covariance between any two different PCA scores is 0. In linear terms, this makes the scores mutually uncorrelated (independent, for Gaussian data). No overlapping linear signal between them!

4. Why eigenvalues equal the scores' variance #

Looking back at Λ—the diagonal entries are exactly the eigenvalues λ₁, λ₂, ..., λₚ. Since the diagonal of a covariance matrix always represents the variance of each variable, each λₖ is the variance of the k-th PCA score.

This is why we sort eigenvalues in descending order: the first eigenvector points to the direction of largest variance in the data, so its corresponding score has the highest variance (captures the most information), the second eigenvector is orthogonal to the first and points to the next largest variance, and so on.

Wrapping it up #

At its core, PCA’s eigen decomposition of the covariance matrix is a way to reorganize the linear structure of your data into a set of orthogonal, uncorrelated components. The eigenvectors give us the directions of maximum variance, and the eigenvalues tell us how much variance each direction captures. That’s exactly why the resulting scores are independent and their variances are given by the eigenvalues—it’s not a coincidence; it’s a direct consequence of how eigenvectors and eigenvalues interact with the covariance matrix.

内容的提问来源于stack exchange，提问作者carlos_ArT