为何熵作为误差度量会导致过拟合？请求详解KDnuggets相关论断

Great question—this is a nuanced topic that trips up a lot of folks when diving into probabilistic modeling. Let’s break this down step by step, including the math behind why entropy and mutual information can lead to overfitting, and tie in John Langford’s key observations.

First, let’s clarify: when we talk about using entropy as an error metric, we’re almost always referring to cross-entropy loss (the standard for classification tasks). To understand the overfitting risk, we need to unpack what this loss optimizes, and how it interacts with noisy training data.

• Cross-entropy’s "greedy" optimization of certainty
  The cross-entropy loss between true labels $y$ (one-hot encoded) and model predictions $\hat{y}$ (probability distributions) is defined as:
  $$
  H(y, \hat{y}) = -\sum_{i=1}^N \sum_{k=1}^K y_{ik} \log(\hat{y}{ik})
  $$
  Here, $N$ is the number of samples, $K$ is the number of classes, $y{ik}=1$ if sample $i$ belongs to class $k$, and $\hat{y}_{ik}$ is the model’s predicted probability for that class-sample pair.

  The problem? Cross-entropy penalizes any uncertainty in predictions, even when the uncertainty comes from noisy or mislabeled training samples. For example, if a training sample is incorrectly labeled, the model will still try to drive $\hat{y}_{ik}$ as close to 1 as possible for that wrong label—learning spurious patterns (like random pixel noise in an image) just to minimize the loss. This is overfitting in action: the model memorizes training-specific noise instead of generalizable patterns.

• Sensitivity to extreme probabilities
  The $\log(\hat{y})$ term in cross-entropy becomes infinitely large (in magnitude) as $\hat{y}$ approaches 0. This means the model will go to great lengths to avoid assigning tiny probabilities to any class for any sample—even when that class should be highly unlikely. For instance, if a single training sample has a random feature that correlates with a rare class, the model will overemphasize that feature to avoid the huge loss penalty of predicting a low probability for that class. This amplifies overfitting to small, non-representative quirks in the training data.

Langford’s core point is that entropy and mutual information are distributional metrics that don’t distinguish between generalizable dependencies and training-specific noise. Let’s break this down with mutual information first, since it’s closely tied to entropy.

• Mutual Information: Maximizing all dependencies, including spurious ones
  Mutual information between features $X$ and labels $Y$ is defined as:
  $$
  I(X; Y) = H(Y) - H(Y|X)
  $$
  Where $H(Y)$ is the marginal entropy of labels, and $H(Y|X)$ is the conditional entropy of labels given features. When using mutual information as a metric (e.g., for feature selection or training a predictive model), we’re trying to maximize the dependence between $X$ and $Y$.

  The problem? This includes all dependencies—even those that exist only in the training data. For example, if your training set of cat images all have a watermark, mutual information will flag the watermark as a highly predictive feature. But in test data without the watermark, the model will fail because it learned a spurious correlation, not the true pattern of what makes a cat a cat.

• Entropy’s lack of built-in regularization
  Langford emphasizes that entropy-based metrics (like cross-entropy) don’t account for model complexity. Unlike metrics that include regularization terms (e.g., L2 penalty on model weights), cross-entropy will let the model grow arbitrarily complex to fit every single training sample—including noise.

  To formalize this, minimizing cross-entropy is equivalent to minimizing the KL divergence between the true distribution $P(Y|X)$ and the model’s predicted distribution $\hat{P}(Y|X)$:
  $$
  D_{KL}(P(Y|X) || \hat{P}(Y|X)) = H(P, \hat{P}) - H(P)
  $$
  Since $H(P)$ (the entropy of the true distribution) is a constant, minimizing cross-entropy is the same as minimizing KL divergence. But KL divergence treats every training sample as a perfect representation of the true distribution—even noisy ones. So the model will fit those outliers, leading to poor generalization.

If you’re using entropy or mutual information as metrics, you can mitigate overfitting with these strategies:

• Add explicit regularization: Use L1/L2 penalties on model weights, Dropout layers, or early stopping to limit model complexity and prevent it from memorizing noise.
• Label smoothing: Replace one-hot labels with softened values (e.g., $y_{ik} = (1-\epsilon)\delta_{ik} + \epsilon/K$ where $\epsilon$ is a small constant like 0.1). This reduces the model’s need to be 100% certain about every prediction, making it less sensitive to mislabeled samples.
• Use robust alternatives: For some tasks, hinge loss (used in SVMs) or mean squared error (for regression) can be more resistant to noise than cross-entropy.
• Bayesian regularization: Incorporate prior distributions on model parameters to encode uncertainty, which naturally penalizes overly complex models that fit noise.

内容的提问来源于stack exchange，提问作者Tim