何为Sample Efficiency？如何用Importance Sampling实现样本效率？

Great question—let’s break this down, especially tying it back to the context of the Sample Efficient Actor-Critic with Experience Replay paper you referenced.

In the context of reinforcement learning (RL)—and specifically actor-critic algorithms—sample efficiency refers to an agent’s ability to learn a high-performing policy using as few interactions with the environment as possible.

Think of it this way: every time an agent takes a step in the environment, that’s a "sample" of experience (state, action, reward, next state). In real-world scenarios (like robotics or autonomous driving), these interactions can be expensive, time-consuming, or even risky. Sample-efficient algorithms minimize the number of these interactions needed to reach a desired level of performance.

The paper you’re referencing addresses a common pain point: traditional actor-critic methods are often sample-inefficient because they rely solely on online, real-time experience and don’t reuse past samples. Experience replay helps, but it introduces distribution shift (old samples come from outdated policies), which is where importance sampling comes in.

Importance sampling (IS) is a statistical technique that lets us correct for distribution shifts when reusing old experience, making replay buffers viable for actor-critic methods without sacrificing accuracy. Here’s how it works in this context:

• The core problem with experience replay: When you store past samples in a replay buffer, those samples were generated by an older version of your actor policy ($\pi_{\text{old}}$). If you use them directly to update your current policy ($\pi_{\theta}$), you’re estimating expectations under the wrong distribution, leading to biased updates.
• IS weight calculation: To fix this, you assign an importance weight to each replay sample. The weight is the ratio of the current policy’s probability of taking the action in that state to the old policy’s probability:

  \rho = \frac{\pi_{\theta}(a|s)}{\pi_{\text{old}}(a|s)}
  

  This weight "reweights" the sample to align it with the current policy’s distribution, eliminating the bias from using out-of-date experience.
• Integration into actor-critic updates: In the paper’s framework, these IS weights are used to adjust both the critic’s value function estimates and the actor’s policy gradient updates. For example, when calculating the temporal difference error for the critic, you multiply it by the IS weight to ensure you’re learning from a distribution that matches your current policy.
• Balancing bias and variance: A key detail from the paper is handling high variance that can come with IS weights. If the current policy is very different from the old one, weights can become extremely large, destabilizing training. The paper likely uses techniques like weight clipping (capping weights at a maximum value) or normalization to keep variance in check, ensuring efficient sample reuse without training instability.

By using importance sampling to safely reuse old experience, the algorithm reduces the need for constant new interactions with the environment—directly boosting sample efficiency.

内容的提问来源于stack exchange，提问作者Gokul NC