基于Q-Learning的神经网络训练实现疑问及伪代码排查

Hey there! Let's break down your Q-Learning + neural network implementation issues and questions step by step. First, let's clean up your pseudocode for clarity:

train(int iterations) 
    buffer = empty buffer 
    for i = 0 while i < iterations: 
        move = null 
        if random(0,1) > threshold: 
            move = random_move() 
        else 
            move = network_calculate_move() 
        
        input_to_network = game.getInput() 
        output_of_network = network.calculate(input_to_network) 
        
        game.makeMove(move) 
        reward = game.getReward() 
        maximum_next_q_value = max(network.calculate(game.getInput())) 
        
        if reward is 1 or -1: // either lost or won
            output_of_network[move] = reward 
        else: 
            output_of_network[move] = reward + discount_factor * max // <-- note: max should be maximum_next_q_value
        
        buffer.add(input_to_network, output_of_network) 
        if buffer is full: 
            buffer.remove_oldest() 
            train_network() 

train_network(buffer b): 
    batch = b.extract_random_batch(batch_size) 
    for each input,output in batch: 
        network.train(input, output, learning_rate) // one forward/backward pass 

1. Is your implementation theoretically correct? #

The core idea aligns with deep Q-learning, but there are critical gaps and bugs that likely cause instability when your buffer grows beyond 200 entries:

• Target Q-value typo: You compute maximum_next_q_value but don't use it in the update line. The line output_of_network[move] = reward + discount_factor * max should reference maximum_next_q_value—this breaks the Bellman equation if left uncorrected.
• Missing target network: This is the biggest stability issue. Your code uses the same network to calculate both current Q-values and target Q-values. As the network updates, target values shift constantly, leading to training divergence, especially with larger buffers. Standard DQN uses a separate target network that's updated infrequently (e.g., every 1000 steps) to fix this.
• Ambiguous action selection: When using network_calculate_move(), it's unclear if this selects the action with the highest Q-value (the greedy choice). If it just outputs raw Q-values without picking the max, your epsilon-greedy strategy breaks.
• Inefficient replay trigger: Training only when the buffer is full wastes useful early samples. You should start training once the buffer reaches a minimum size (e.g., 5000 entries) and train every step (or every N steps) after that.
• Incorrect stored experience: Storing input_to_network and output_of_network couples targets to the network's state at transition time. Instead, store full transition tuples: (current_state, action, reward, next_state, done_flag) to recalculate targets dynamically during training.

In short, the foundation is there, but missing components (like target networks) and bugs are causing instability with larger buffers.

2. What's a reasonable ratio between batch size and buffer size? #

There's no strict rule, but here's standard practice:

• Replay buffer size: Ranges from 10,000 to 1,000,000 entries, depending on task complexity. Simple games work with 100k; complex environments may need 1M.
• Batch size: Common values are 32, 64, 128, or 256. The batch needs to be large enough to stabilize gradient estimates (too small = noisy gradients) but not so large that it drains memory or slows training.
• Ratio: Aim for a batch size that's 0.01% to 1% of the buffer size. Examples:
  • 100k buffer + 64 batch → ~0.06% ratio
  • 10k buffer + 32 batch → ~0.32% ratio

Also, don't start training until the buffer holds at least a few thousand entries—training on a tiny, non-diverse buffer leads to poor generalization.

3. How is neural network training typically done in this context? When do you stop training? Do you need to hit a specific MSE per batch? #

Let's walk through the standard DQN workflow, which is what you're building towards:

1. Experience collection: Use an epsilon-greedy policy to select actions, collect transitions (state, action, reward, next_state, done), and store them in the replay buffer.
2. Dual network setup: Initialize two identical networks:
   • A main network (for computing current Q-values and updating weights)
   • A target network (for stable target Q-value calculations)
3. Training loop:
   • Once the buffer hits a minimum size, sample a random batch of transitions.
   • For each transition:
     • If done is true (game over), target Q-value = reward
     • If not done, target = reward + discount_factor * max(target_network.predict(next_state))
   • Calculate the main network's predicted Q-values for the current state/action pair.
   • Compute MSE loss between predictions and targets, then do a backward pass to update the main network.
   • Every N steps (e.g., 1000 steps), copy the main network's weights to the target network.

Training termination & MSE notes: #

• Don't stop based on batch MSE: Low MSE doesn't equal good agent performance—it could mean overfitting to the replay buffer. Instead, track task-specific metrics:
  • For games: Win rate, average score, or steps per episode.
  • Stop when performance plateaus (no improvement for many episodes) or degrades.
• Training duration: Most RL tasks are trained for a fixed number of steps (e.g., 1 million) or episodes. Early stopping works if performance stagnates.
• Batch MSE doesn't need a target: You don't wait for a batch to hit a specific MSE—each batch gets one gradient update, and you keep iterating until the agent's performance is satisfactory.

内容的提问来源于stack exchange，提问作者Finn Eggers