请求证明简单线性回归模型中帽子矩阵H的元素表达式
Hey there! Let's work through proving those expressions for the elements of the hat matrix $H$ in simple linear regression. I'll break it down step by step so it's easy to follow along.
First, let's recall some key definitions to set the stage:
- The hat matrix $H$ is defined as $H = X(X^T X){-1}XT$, where $X$ is the design matrix for our simple linear regression model.
- For simple linear regression, the design matrix $X$ is an $n \times 2$ matrix: the first column is a vector of ones (for the intercept term), and the second column holds our predictor variable values. Written out:
$$X = \begin{bmatrix} 1 & x_1 \ 1 & x_2 \ \vdots & \vdots \ 1 & x_n \end{bmatrix} = \left[ \mathbf{1}_n, \mathbf{x} \right]$$
Here, $\mathbf{1}_n$ is an $n$-dimensional vector of ones, and $\mathbf{x} = (x_1, x_2, ..., x_n)^T$. - We'll use $S_{xx} = \sum_{k=1}^n (x_k - \bar{x})^2$, where $\bar{x} = \frac{1}{n}\sum_{k=1}^n x_k$ is the mean of the predictor variable. A useful rewrite of $S_{xx}$ is $nS_{xx} = n\sum x_k^2 - (\sum x_k)^2$ (expand the squared term to verify this yourself!).
Step 1: Compute $X^T X$
First, calculate the product of $X^T$ and $X$ (a compact $2 \times 2$ matrix):
$$X^T X = \begin{bmatrix} \mathbf{1}_n^T \mathbf{1}n & \mathbf{1}n^T \mathbf{x} \ \mathbf{x}^T \mathbf{1}n & \mathbf{x}^T \mathbf{x} \end{bmatrix} = \begin{bmatrix} n & \sum{k=1}^n x_k \ \sum{k=1}^n x_k & \sum{k=1}^n x_k^2 \end{bmatrix}$$
Step 2: Compute the inverse $(X^T X)^{-1}$
For a $2 \times 2$ matrix $\begin{bmatrix} a & b \ c & d \end{bmatrix}$, the inverse is $\frac{1}{ad - bc}\begin{bmatrix} d & -b \ -c & a \end{bmatrix}$. Applying this to our $X^T X$:
- The determinant is $n\sum x_k^2 - (\sum x_k)^2 = nS_{xx}$ (from our earlier rewrite of $S_{xx}$).
- So the inverse becomes:
$$(X^T X)^{-1} = \frac{1}{nS_{xx}} \begin{bmatrix} \sum x_k^2 & -\sum x_k \ -\sum x_k & n \end{bmatrix}$$
Step 3: Derive the general element $h_{ij}$
The element $h_{ij}$ is the result of multiplying the $i$-th row of $X$, by $(X^T X)^{-1}$, by the $j$-th column of $X^T$. Formally:
$$h_{ij} = \text{Row } i \text{ of } X \times (X^T X)^{-1} \times \text{Column } j \text{ of } X^T$$
The $i$-th row of $X$ is $\begin{bmatrix} 1 & x_i \end{bmatrix}$, and the $j$-th column of $X^T$ is $\begin{bmatrix} 1 \ x_j \end{bmatrix}$. Let's compute this step by step:
Multiply $(X^T X)^{-1}$ by the $j$-th column of $X^T$:
$$\frac{1}{nS_{xx}} \begin{bmatrix} \sum x_k^2 & -\sum x_k \ -\sum x_k & n \end{bmatrix} \begin{bmatrix} 1 \ x_j \end{bmatrix} = \frac{1}{nS_{xx}} \begin{bmatrix} \sum x_k^2 - x_j \sum x_k \ -\sum x_k + n x_j \end{bmatrix}$$Multiply this result by the $i$-th row of $X$:
$$h_{ij} = \begin{bmatrix} 1 & x_i \end{bmatrix} \times \frac{1}{nS_{xx}} \begin{bmatrix} \sum x_k^2 - x_j \sum x_k \ -\sum x_k + n x_j \end{bmatrix}$$
Expanding the product gives us the numerator to simplify:
$$\sum x_k^2 - x_j \sum x_k - x_i \sum x_k + n x_i x_j = \sum x_k^2 - \sum x_k (x_i + x_j) + n x_i x_j$$
Now substitute $\sum x_k = n\bar{x}$ and $\sum x_k^2 = n\bar{x}^2 + S_{xx}$ (from $S_{xx} = \sum x_k^2 - n\bar{x}^2$):
$$\begin{align*}
\text{Numerator} &= (n\bar{x}^2 + S_{xx}) - n\bar{x}(x_i + x_j) + n x_i x_j \
&= S_{xx} + n\bar{x}^2 - n\bar{x}x_i - n\bar{x}x_j + n x_i x_j \
&= S_{xx} + n\left( \bar{x}^2 - \bar{x}x_i - \bar{x}x_j + x_i x_j \right) \
&= S_{xx} + n(\bar{x} - x_i)(\bar{x} - x_j) \
&= S_{xx} + n(x_i - \bar{x})(x_j - \bar{x})
\end{align*}$$
Divide by $nS_{xx}$ to get our final expression for $h_{ij}$:
$$h_{ij} = \frac{S_{xx} + n(x_i - \bar{x})(x_j - \bar{x})}{nS_{xx}} = \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{S_{xx}}$$
Step 4: Special case for the diagonal element $h_{ii}$
When $i = j$, we just substitute $j = i$ into our general $h_{ij}$ formula:
$$h_{ii} = \frac{1}{n} + \frac{(x_i - \bar{x})(x_i - \bar{x})}{S_{xx}} = \frac{1}{n} + \frac{(x_i - \bar{x})^2}{S_{xx}}$$
That's all there is to it! We've derived both expressions for the hat matrix elements. Feel free to ask if any step needs extra clarification.
内容的提问来源于stack exchange，提问作者Something
