新增观测值时样本均值、协方差矩阵及逆协方差矩阵的更新方法

Alright, let's walk through updating your sample stats step by step. You mentioned the mean and covariance updates are straightforward, so I'll cover those quickly before diving into the inverse covariance update—that's the juicy part you're asking for.

1. Sample Mean Update #

This is the simplest one. If you have the original sample mean $\bar{X}n$ for $n$ observations, adding a new data point $X{n+1}$ gives you the updated mean:

$\bar{X}_{n+1} = \frac{n\bar{X}_n + X_{n+1}}{n+1}$

Think of it as a weighted average: the old sample gets weight $n/(n+1)$, and the new observation gets $1/(n+1)$. No surprises here.

2. Sample Covariance Matrix ($S$) Update #

Assuming we're using the unbiased sample covariance (denominator $n-1$ for the original sample), the updated matrix $S_{n+1}$ can be computed using your existing $S_n$ and mean. First, define the deviation vector of the new point from the original mean:

$d_n = X_{n+1} - \bar{X}_n$

Then the updated covariance matrix is:

$S_{n+1} = \frac{n-1}{n}S_n + \frac{1}{n+1}d_n d_n^T$

Breaking this down: we scale the old covariance matrix to account for the new sample size, then add the weighted outer product of the deviation vector—this captures the new variability introduced by the data point.

3. Inverse Covariance Matrix ($S^{-1}$) Update #

This is where we use the Sherman-Morrison formula, a staple for updating matrix inverses without recomputing everything from scratch. Let's start with the covariance update formula and rearrange it to fit the Sherman-Morrison form.

First, recall the relationship between the sum of squared deviations ($T_n = (n-1)S_n$) and the updated sum $T_{n+1} = nS_{n+1}$. We know:

$T_{n+1} = T_n + \frac{n}{n+1}d_n d_n^T$

Substitute $T_n = (n-1)S_n$ and rearrange to solve for $S_{n+1}$:

$S_{n+1} = \frac{n-1}{n}S_n + \frac{1}{n+1}d_n d_n^T$

Now, let's define $A = \frac{n-1}{n}S_n$ and $u = \sqrt{\frac{1}{n+1}}d_n$, so $S_{n+1} = A + uu^T$. The Sherman-Morrison formula tells us:

$(A + uu^T)^{-1} = A^{-1} - \frac{A^{-1}uu^T A^{-1}}{1 + u^T A^{-1}u}$

Plugging in our definitions for $A$ and $u$, and simplifying all the terms, we end up with the formula for $S_{n+1}^{-1}$:

$S_{n+1}^{-1} = \frac{n}{n-1}S_n^{-1} - \frac{n \cdot S_n^{-1} d_n d_n^T S_n^{-1}}{(n^2 - 1) + n \cdot d_n^T S_n^{-1} d_n}$

Key Notes: #

• $d_n = X_{n+1} - \bar{X}_n$ (you already have $\bar{X}_n$ and the new observation, so this is easy to compute)
• This formula avoids the costly $O(p^3)$ matrix inversion operation (where $p$ is the number of features) — you only need to do matrix-vector multiplications and scalar arithmetic, which is way faster for high-dimensional data.
• Make sure to check that the denominator $(n^2 -1) + n \cdot d_n^T S_n^{-1} d_n$ isn't zero (it won't be if $S_n$ was invertible and the new point isn't perfectly aligned in a way that makes the updated matrix singular, which is unlikely in practice).

内容的提问来源于stack exchange，提问作者Name LeftBlank