如何检验组间差异完全源于二项方差的原假设并计算p值？

Hey there, let's tackle your two statistical questions head-on—they're all about verifying whether group-to-group variation is exactly what we'd expect from pure binomial sampling noise, right? Let's break this down step by step.

The core idea here is to compare the observed group-level variation to the variation we'd theoretically expect if every group was just a random binomial sample from the overall population. If the observed variance matches the binomial expectation, we can't reject the null hypothesis; if it's significantly larger or smaller, we have evidence of extra (or reduced) variation from other factors.

Two common, practical methods for this:

• Variance Ratio Test (Chi-Squared Approximation)
  First, calculate the sample variance of your group sums (or group means, depending on what you're measuring). Then compare it to the theoretical binomial variance. For group sums (number of "1"s per group of size k), the theoretical variance under the null is $kp(1-p)$. For group means, it's $p(1-p)/k$.
  Construct a chi-squared statistic:

  χ² = (G-1) * (observed group variance) / (theoretical binomial variance)
  

  where G is the number of groups. Under the null hypothesis, this statistic follows a chi-squared distribution with G-1 degrees of freedom. You can then compare this to critical values or calculate a p-value (more on that below).

• Pearson's Chi-Squared Goodness-of-Fit Test
  For each group, compute the observed number of "1"s ($O_i$) and the expected number under the null ($E_i = kp$—use the overall sample proportion $\hat{p}$ if p is unknown). Then calculate:

  χ² = Σ [(O_i - E_i)² / E_i]
  

  With G groups, if you estimated p from the data, the degrees of freedom are G-2 (since we used one parameter to estimate p). This test checks if the distribution of group counts matches the binomial expectation.

Let's formalize your scenario first to avoid confusion: You have a large population split into G equal-sized groups (each size k), each member is 0 or 1, overall population mean is p. The null hypothesis is that group sums follow a binomial distribution (so group sum variance = $kp(1-p)$), and the alternative is that group variation doesn't fit this binomial pattern.

Here's how to compute the p-value:

Step 1: Estimate the overall proportion #

If p isn't known, calculate the sample proportion from all observations:

$\hat{p} = \frac{\text{Total number of 1s across all groups}}{G*k}$

Step 2: Compute observed group variance #

Calculate the sum of "1"s for each group ($S_i$), then compute the sample variance of these sums:

$s^2 = \frac{1}{G-1} \sum_{i=1}^G (S_i - k\hat{p})^2$

Step 3: Construct the test statistic #

Using the chi-squared approximation (valid for large G):

$\chi^2 = \frac{(G-1)*s^2}{k\hat{p}(1-\hat{p})}$

Under the null hypothesis, this statistic follows a $\chi^2_{G-1}$ distribution (chi-squared with G-1 degrees of freedom).

Step 4: Calculate the p-value #

The p-value depends on your alternative hypothesis:

• Two-sided alternative (group variance ≠ binomial expectation):
  p-value = 2 * min(P(χ² > observed_stat), P(χ² < observed_stat))
• One-sided (overdispersion) (group variance > binomial expectation, meaning extra group-level variation):
  p-value = P(χ² > observed_stat)
• One-sided (underdispersion) (group variance < binomial expectation, meaning less variation than binomial):
  p-value = P(χ² < observed_stat)

Alternative: Likelihood Ratio Test (LRT) #

For a more robust approach (especially with smaller sample sizes), use the LRT:

1. Compute the maximum likelihood under the null (all groups share p = $\hat{p}$)
2. Compute the maximum likelihood under the alternative (each group has its own proportion $\hat{p}_i = S_i/k$)
3. Calculate the LRT statistic:

   $LR = -2\ln\left(\frac{L_0}{L_1}\right) = 2\sum_{i=1}^G \left[S_i \ln\left(\frac{\hat{p}_i}{\hat{p}}\right) + (k - S_i)\ln\left(\frac{1-\hat{p}_i}{1-\hat{p}}\right)\right]$
   

   This statistic follows a $\chi^2_{G-1}$ distribution under the null, so you can compute the p-value the same way as above. Note: If any $S_i$ is 0 or k, add a small constant (like 0.5) to avoid undefined logarithms.

内容的提问来源于stack exchange，提问作者quarague