NIPS 2017 Learning to Run项目：可否用一模型训练数据训练另一模型？

Absolutely, you can reuse the state-action-reward (SAR) trajectories collected from training one model (like DDPG) to train the other (like SAC)—this is actually a go-to trick when you’re crunched for time dealing with slow simulators like the Learning to Run environment. Let me walk you through how to pull this off, plus key things to keep in mind:

• Check Data Compatibility First
  Both DDPG and SAC are off-policy algorithms, meaning they don’t require data generated by their own current policy to learn. The core data you need are tuples of (state, action, reward, next_state, done)—make sure your saved dataset includes all these elements, since both algorithms rely on them for experience replay. For Learning to Run specifically, the state space (joint angles, velocities, etc.) and action space (motor commands) are identical across models, so you won’t have to reformat anything here.

• Reuse the Experience Replay Buffer
  If you already have a replay buffer from your DDPG training run, you can directly copy its contents into SAC’s replay buffer. Even though DDPG outputs deterministic actions and SAC uses stochastic ones, this doesn’t hinder off-policy learning—SAC will still learn to map states to optimal actions using the existing SAR data. If you saved trajectories to a file instead of keeping them in memory, just load them into SAC’s buffer at startup.

• Key Considerations to Avoid Pitfalls

  • Watch for Data Distribution Shift: As SAC learns, its policy will generate data that’s different from the original DDPG policy. To prevent suboptimal learning or catastrophic forgetting, don’t just rely on old data—continue collecting new trajectories with SAC and mix them with the DDPG data in the replay buffer over time.
  • Filter Low-Quality Data: If your DDPG data includes a lot of early training steps where the agent was performing poorly (e.g., falling over immediately), filtering out low-reward trajectories can speed up SAC’s learning. Keep only the top 30-50% of high-reward trajectories to give SAC a better starting point.
  • Tune SAC’s Hyperparameters Independently: Don’t reuse DDPG’s hyperparameters for SAC. SAC has unique parameters like the temperature coefficient (alpha) that balances exploration and exploitation—you’ll need to tune these separately, even with pre-collected data.

• Bonus Time-Saving Tips

  • If you have extra compute, run data collection in parallel with training. Even a single batch of pre-collected DDPG data can get SAC through its initial learning phase without waiting for the simulator to generate new data every step.
  • Look into offline RL techniques, which are built for learning from fixed datasets. You could pre-train SAC on the DDPG dataset first, then switch to online training with the simulator to refine the policy faster.

内容的提问来源于stack exchange，提问作者Kemal BEKTAŞ