请求证明互信息I(X;T)关于p(t|x)的导数公式

Let's break down this derivation step by step to confirm the result you've referenced. I'll start from the given mutual information definition and work through each differentiation step clearly, so you can follow along easily.

Step 1: Rewrite the Mutual Information Expression #

First, let's expand the given mutual information using logarithm properties to split the term into two separate sums—this will make differentiation much simpler:
$$
I(X;T) = \sum_{x,t} p(t|x)p(x)\log\left[\frac{p(t|x)}{p(t)}\right] = \sum_{x,t} p(x)p(t|x)\log p(t|x) - \sum_{x,t} p(x)p(t|x)\log p(t)
$$
Note that $p(t) = \sum_{x'} p(x')p(t|x')$ (the marginal distribution of $T$), which is critical for computing the derivative of the second term.

Step 2: Differentiate the First Sum #

We want to compute the partial derivative $\frac{\partial I}{\partial p(t'|x')}$, where $(x', t')$ is a specific pair of values for $X$ and $T$. For the first sum:
$$
\frac{\partial}{\partial p(t'|x')} \sum_{x,t} p(x)p(t|x)\log p(t|x)
$$
Most terms in the sum don't depend on $p(t'|x')$—only the term where $x = x'$ and $t = t'$ does. Applying the product rule to this single term:
$$
\frac{\partial}{\partial p(t'|x')} \left[ p(x')p(t'|x')\log p(t'|x') \right] = p(x')\left( \log p(t'|x') + p(t'|x') \cdot \frac{1}{p(t'|x')} \right) = p(x')\left(1 + \log p(t'|x')\right)
$$
All other terms vanish, so this is the full derivative of the first sum.

Step 3: Differentiate the Second Sum #

Now for the second sum, first rewrite it using the marginal distribution $p(t) = \sum_x p(x)p(t|x)$:
$$
\sum_{x,t} p(x)p(t|x)\log p(t) = \sum_t p(t)\log p(t)
$$
Again, we take the partial derivative with respect to $p(t'|x')$. Only the term where $t = t'$ depends on $p(t'|x')$, since $p(t')$ is a function of $p(t'|x')$. Using the chain rule:

1. First, compute the derivative of $p(t')\log p(t')$ with respect to $p(t')$: $\frac{d}{dp(t')}\left[p(t')\log p(t')\right] = 1 + \log p(t')$
2. Then multiply by the derivative of $p(t')$ with respect to $p(t'|x')$: $\frac{\partial p(t')}{\partial p(t'|x')} = p(x')$ (since $p(t') = \sum_x p(x)p(t'|x)$, only the $x=x'$ term contributes)

Putting this together, the derivative of the second sum is:
$$
\frac{\partial}{\partial p(t'|x')} \sum_t p(t)\log p(t) = p(x')\left(1 + \log p(t')\right)
$$
Since our original mutual information expression subtracts this sum, the derivative contribution from the second term is:
$$

• p(x')\left(1 + \log p(t')\right)
  $$

Step 4: Combine the Results #

Adding the derivatives from the first and second sums gives us:
$$
\frac{\partial I}{\partial p(t'|x')} = p(x')\left(1 + \log p(t'|x')\right) - p(x')\left(1 + \log p(t')\right)
$$
Dropping the primes (since $(x', t')$ was just an arbitrary pair), we get exactly the result you referenced:
$$
\frac{\partial I}{\partial p(t|x)} = p(x)\left[1+\log p(t|x)\right] - p(x)\left[1+\log p(t)\right]
$$

内容的提问来源于stack exchange，提问作者Yashar