求贝叶斯估计量MSE一致性的通用结论及证明指引

Sure thing! Let's dive into the general consistency results for Bayesian estimators, starting with MSE consistency since that's what you're curious about, and we'll stick to distribution-agnostic conditions and proofs as much as possible.

First, let's clarify definitions to align:

• An estimator $\hat{\theta}_n$ (based on $n$ i.i.d. samples) is MSE consistent for the true parameter $\theta_0$ if:
  $$\mathbb{E}\left[(\hat{\theta}_n - \theta_0)^2\right] \to 0 \quad \text{as } n \to \infty$$
• For Bayesian estimators, this typically refers to posterior means, medians, or other posterior summary statistics (we'll focus on posterior means here, since they're common and tie directly to MSE).

Key Distribution-Agnostic Conditions for MSE Consistency #

For a Bayesian estimator (e.g., posterior mean) to be MSE consistent, these core conditions usually suffice (no need to assume specific likelihood distributions like Gaussian or Bernoulli):

• Identifiability of the Model: For any $\theta \neq \theta_0$, the data distribution $P_\theta$ is distinct from $P_{\theta_0}$ (formally, the Kullback-Leibler divergence $KL(P_{\theta_0} || P_\theta) > 0$). This ensures data can distinguish between the true parameter and any alternative.
• Positive Prior Density at the True Parameter: The prior distribution $\pi(\theta)$ satisfies $\pi(\theta_0) > 0$. In short, the prior doesn't rule out the true parameter entirely.
• True Parameter is Interior to the Parameter Space: $\theta_0$ lies in the interior of $\Theta$ (not on a boundary). This prevents the posterior from being "pushed away" by parameter space constraints.
• Regularity of the Log-Likelihood: The log-likelihood $\log p(X|\theta)$ is well-behaved near $\theta_0$:
  • $\mathbb{E}{\theta_0}\left[\sup{\theta \in N(\theta_0)} |\log p(X|\theta)|\right] < \infty$ for some neighborhood $N(\theta_0)$ of $\theta_0$ (ensures expected log-likelihood is bounded locally).
  • The log-likelihood is continuous in $\theta$ at $\theta_0$ (with respect to the data distribution).

Proof Sketch for MSE Consistency (Posterior Mean) #

Here's a high-level, distribution-agnostic proof framework:

1. Decompose MSE: Recall that MSE splits into bias squared plus variance:
   $$\mathbb{E}\left[(\hat{\theta}_n - \theta_0)^2\right] = \left(\mathbb{E}[\hat{\theta}_n] - \theta_0\right)^2 + \text{Var}(\hat{\theta}_n)$$
   We need to show both terms tend to 0 as $n \to \infty$.

2. Posterior Concentration: Using the law of large numbers and Bayes' theorem, we can show that for any $\epsilon > 0$, the posterior probability of $\theta$ lying outside a neighborhood $(\theta_0 - \epsilon, \theta_0 + \epsilon)$ tends to 0 almost surely (this is a core result of posterior consistency, sometimes called the Bernstein-von Mises theorem's foundational step). Formally:
   $$\pi\left(|\theta - \theta_0| \geq \epsilon \mid X_1,...,X_n\right) \xrightarrow{a.s.} 0 \quad \text{as } n \to \infty$$

3. Bounding the Bias: Split the posterior mean into two parts:
   $$\mathbb{E}[\hat{\theta}n] = \mathbb{E}\left[\int{|\theta - \theta_0| < \epsilon} \theta \pi(\theta|X_n) d\theta\right] + \mathbb{E}\left[\int_{|\theta - \theta_0| \geq \epsilon} \theta \pi(\theta|X_n) d\theta\right]$$

   • The first integral is close to $\theta_0$ (since $\theta$ is within $\epsilon$ of $\theta_0$).
   • The second integral's expectation is bounded by $\sup_{\theta \in \Theta} |\theta| \times \mathbb{E}\left[\pi(|\theta - \theta_0| \geq \epsilon \mid X_n)\right]$, which tends to 0 by posterior concentration.
   • As $\epsilon$ can be made arbitrarily small, the bias $\mathbb{E}[\hat{\theta}_n] - \theta_0$ tends to 0.

4. Bounding the Variance: Similarly, split the variance into neighborhood-in and neighborhood-out components:
   $$\text{Var}(\hat{\theta}_n) = \mathbb{E}\left[\text{Var}(\theta \mid X_n)\right] = \mathbb{E}\left[\int (\theta - \hat{\theta}_n)^2 \pi(\theta|X_n) d\theta\right]$$

   • Inside the neighborhood, $(\theta - \hat{\theta}_n)^2$ is bounded by $(2\epsilon)^2$ (since $\theta$ and $\hat{\theta}_n$ are both within $\epsilon$ of $\theta_0$).
   • Outside the neighborhood, the integral is bounded by $\sup_{\theta \in \Theta} (\theta - \theta_0)^2 \times \mathbb{E}\left[\pi(|\theta - \theta_0| \geq \epsilon \mid X_n)\right]$, which tends to 0.
   • Again, choosing $\epsilon$ arbitrarily small shows the variance tends to 0.

Bonus: Beyond MSE Consistency #

MSE consistency is a strong form of convergence—it implies both convergence in probability ($\hat{\theta}_n \xrightarrow{P} \theta_0$) and almost sure convergence (under slightly stronger regularity conditions). For many Bayesian estimators, even if MSE doesn't converge, posterior consistency (convergence in probability or almost surely) often holds under the same core conditions above.

Note: Improper priors (e.g., uniform over the real line) can still yield consistent estimators, but you'll need to replace the "positive prior density" condition with a weaker requirement that the prior is "non-degenerate" near $\theta_0$ (e.g., the prior assigns positive mass to every neighborhood of $\theta_0$).

内容的提问来源于stack exchange，提问作者maxical