OpenAI论文《Evolution Strategies as a Scalable Alternative to Reinforcement Learning》第3页公式推导问询

Alright, let's dive into the key formula on page 3 of Evolution Strategies as a Scalable Alternative to Reinforcement Learning—this is the math that makes ES a viable, scalable alternative to traditional RL.

Background Setup #

First, let's align on the notation we're using:

• $\theta$: The vector of model parameters we want to optimize.
• $\epsilon$: A random noise vector sampled from a Gaussian distribution $\mathcal{N}(0, \sigma^2 I)$ (zero mean, variance $\sigma^2$ across all dimensions).
• $f(\theta + \epsilon)$: The reward we get when we run our agent with parameters perturbed by $\epsilon$ (this is a single sample estimate of the expected reward $J(\theta)$).

Our end goal is to compute $\nabla_\theta J(\theta)$—the gradient of the expected reward with respect to our parameters—so we can update $\theta$ to maximize $J(\theta)$.

Step 1: Rewrite the Expected Reward #

First, we can express the expected reward $J(\theta)$ as the expectation over all possible noise perturbations (since each perturbation gives us a reward sample):
$$J(\theta) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2 I)} \left[ f(\theta + \epsilon) \right]$$

To find the gradient $\nabla_\theta J(\theta)$, we can swap the gradient and the expectation (this is valid here because $f$ is differentiable and the expectation is well-behaved):
$$\nabla_\theta J(\theta) = \mathbb{E}{\epsilon} \left[ \nabla\theta f(\theta + \epsilon) \right]$$

Step 2: Apply Stein's Lemma (The Key Trick) #

Here's where the magic happens. We use Stein's Lemma, a handy result from probability theory for Gaussian distributions. For a zero-mean Gaussian $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ and a differentiable function $g(\epsilon)$, the lemma states:
$$\mathbb{E} \left[ \epsilon \cdot g(\epsilon) \right] = \sigma^2 \mathbb{E} \left[ \nabla_\epsilon g(\epsilon) \right]$$

Let’s set $g(\epsilon) = f(\theta + \epsilon)$. Using the chain rule, $\nabla_\epsilon f(\theta + \epsilon) = \nabla_\theta f(\theta + \epsilon)$ (since the derivative of $\theta + \epsilon$ with respect to $\epsilon$ is the identity matrix, same as with respect to $\theta$). Plugging this into Stein's Lemma gives us:
$$\mathbb{E} \left[ \epsilon \cdot f(\theta + \epsilon) \right] = \sigma^2 \mathbb{E} \left[ \nabla_\theta f(\theta + \epsilon) \right]$$

Step 3: Solve for the Gradient #

Remember from Step 1 that $\nabla_\theta J(\theta)$ equals the right-hand side of the above equation. Let's rearrange to solve for our desired gradient:
$$\nabla_\theta J(\theta) = \frac{1}{\sigma^2} \mathbb{E}_{\epsilon} \left[ f(\theta + \epsilon) \cdot \epsilon \right]$$

Step 4: Add a Baseline to Reduce Variance #

To make this estimate more stable (lower variance), we introduce a baseline $b$—typically the average reward across all our noise samples. Subtracting this baseline doesn't change the expectation (since $\mathbb{E}[b \cdot \epsilon] = b \cdot \mathbb{E}[\epsilon] = 0$, because $\epsilon$ has zero mean), so we get:
$$\nabla_\theta J(\theta) = \frac{1}{\sigma^2} \mathbb{E}_{\epsilon} \left[ (f(\theta + \epsilon) - b) \cdot \epsilon \right]$$

Step 5: Monte Carlo Estimation (Practical Implementation) #

Since we can't compute the true expectation, we approximate it with a finite number of noise samples $n$. This gives us the practical update rule you'll see in the paper:
$$\hat{\nabla}\theta J(\theta) = \frac{1}{n \sigma^2} \sum{i=1}^n \left( f(\theta + \epsilon_i) - b \right) \cdot \epsilon_i$$

That's exactly the core formula from page 3! The beauty of this is that we don't need to track trajectories or compute policy gradients like in traditional RL—we just sample noise, run the agent, collect rewards, and compute this sum.

内容的提问来源于stack exchange，提问作者leonexu