实现Replay Buffer采样时触发TypeError: 'type' object is not iterable错误的问题咨询

Problem Description #

I implemented a simple replay buffer for reinforcement learning, but when I run the sampling method, I get this error: TypeError: 'type' object is not iterable. When I check the type of self.buffer[0], it shows as 'type' instead of the expected Experience namedtuple type. Here's my code:

import collections
import numpy as np

Experience = collections.namedtuple("Experience", field_names=["state", "action", "reward", "done", "next_state"])

class ReplayBuffer:
    def __init__(self, capacity):
        self.buffer = collections.deque(maxlen=capacity)

    def __len__(self):
        return len(self.buffer)

    def add_exp(self, exp: Experience):
        self.buffer.append(exp)

    def sample(self, batch_size):
        idxs = np.random.choice(len(self.buffer), batch_size, replace=False)
        states, actions, rewards, dones, next_states = zip(*[self.buffer[idx] for idx in idxs])
        return np.array(states), np.array(actions), \
               np.array(rewards, dtype=np.float32), \
               np.array(dones, dtype=np.uint8), \
               np.array(next_states)

Root Cause #

The core issue here is that you're adding the Experience class itself to the buffer instead of actual instances of it. When calling add_exp, you probably passed Experience (the type object) instead of creating a concrete instance with Experience(state=..., action=..., ...).

For example, if you had code like this by mistake:

buffer = ReplayBuffer(1000)
buffer.add_exp(Experience)  # Wrong! You're adding the class, not an instance

Every element in self.buffer would be the Experience type, not a namedtuple instance. When you try to unpack these with zip(*...), Python can't iterate over a type object—hence the TypeError.

Solution #

1. Fix experience addition logic: Ensure you create an Experience instance every time you call add_exp. Here's the correct way to do it:

   # Example of creating and adding a valid Experience instance
   current_state = np.array([1.0, 2.0])
   action = 0
   reward = 1.0
   done = False
   next_state = np.array([1.5, 2.5])
   
   buffer.add_exp(Experience(current_state, action, reward, done, next_state))
   

2. Add type validation (optional but recommended): To catch this mistake early, add a check in the add_exp method to verify the input is an actual Experience instance:

   def add_exp(self, exp: Experience):
       if not isinstance(exp, Experience):
           raise TypeError(f"Expected an instance of Experience, got {type(exp)} instead")
       self.buffer.append(exp)
   

   This will throw a clear, immediate error if you accidentally pass the class instead of an instance, making debugging much faster.

Verification #

After fixing the add_exp calls, check the type of self.buffer[0] again—it should now show as <class '__main__.Experience'> (or similar, depending on your module context). The sample method will then be able to unpack the namedtuples without errors.

内容的提问来源于stack exchange，提问作者womyat