Great question! Let's break this down step by step based on your specific requirements.

First off, Policy Gradient (策略梯度) methods are exactly what you're looking for:

• No pre-collected data required: It's pure online reinforcement learning—your agent learns directly by interacting with the environment in real time, no supervised learning phase needed.
• Avoids Q-learning's food-refresh issue: Unlike Q-learning, which relies on maintaining a value table/network that can become outdated when food spawns randomly, policy gradient methods optimize the policy network itself. The policy outputs actions based on the current state (including new food positions), so it adapts dynamically to environmental changes.
• Works with your reward mechanism: The core logic uses reward signals (small rewards for moving toward food, tiny step penalties) to compute gradients, which are then used via backpropagation to update the network. After each episode, the agent adjusts its policy to increase the probability of actions that led to high cumulative rewards, and decrease those that led to low rewards.

The most basic implementation is the REINFORCE algorithm (a Monte Carlo policy gradient method), which is simple to code and perfect for small-scale environments like Snake. For more stable training, you can also use a basic Actor-Critic setup (one network outputs the policy, another estimates state values to reduce reward variance)—still no pre-collected data required, fully online.

Your left/forward/right directional perception setup is extremely reasonable and efficient for Snake, here's why:

• Snake's decision-making only depends on three key factors relative to its head: whether left/forward/right have obstacles (walls/body), and whether those directions lead to food. Your input covers all critical decision points without redundant information.
• Both input formats have their merits:
  • [Object type, distance] lists: Each direction uses two values (e.g., 0 for obstacle, 1 for food, 2 for empty; distance as steps to the object). This gives the network clear semantic signals, making it easier to learn "avoid obstacles, move toward food" logic—this is the better starting point.
  • Single integer values: Using -1 for obstacles, positive numbers for food distance, and 0 for empty keeps the input dimension tiny (3 features total). It works for small networks, but you may want to normalize distances (e.g., scale to 0-1) to help the network interpret the -1 obstacle marker correctly.

This input design keeps your network small and training fast, while perfectly aligning with Snake's environmental constraints.

Absolutely! TensorFlow/Keras handles this scenario seamlessly. Here's a high-level implementation approach:

1. Build your policy network: Use a simple Sequential model with an input layer matching your perception dimension (3 or 6), 1-2 dense hidden layers, and a softmax output layer (to output probabilities for left/forward/right actions).
2. Custom loss or GradientTape: For REINFORCE, the loss function is -log(action_probability) * discounted_reward. You can collect trajectory data (states, actions, rewards) per episode, then use tf.GradientTape to record forward passes, compute gradients, and update network parameters.
3. Leverage Keras optimizers: Built-in optimizers like Adam handle parameter updates smoothly, so you don't need to write low-level backpropagation code.

Here's a minimal pseudocode snippet to illustrate:

import tensorflow as tf
from tensorflow.keras import layers

# Build policy network
model = tf.keras.Sequential([
    layers.Dense(32, activation='relu', input_shape=(3,)),  # 3 features for single-integer input
    layers.Dense(3, activation='softmax')
])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

def compute_discounted_rewards(rewards, gamma=0.99):
    # Helper to calculate discounted cumulative rewards
    discounted = []
    running_sum = 0
    for r in reversed(rewards):
        running_sum = r + gamma * running_sum
        discounted.insert(0, running_sum)
    return tf.convert_to_tensor(discounted, dtype=tf.float32)

# Training loop
for episode in range(1000):
    states, actions, rewards = [], [], []
    state = env.reset()  # Get initial left/forward/right perception
    done = False
    
    while not done:
        # Get action probabilities from the network
        action_probs = model(tf.expand_dims(state, 0), training=True)
        # Sample an action based on probabilities
        action = tf.random.categorical(tf.math.log(action_probs), num_samples=1)[0, 0].numpy()
        # Take action in the environment
        next_state, reward, done = env.step(action)
        # Store trajectory data
        states.append(state)
        actions.append(action)
        rewards.append(reward)
        state = next_state
    
    # Calculate discounted rewards for the episode
    discounted_rewards = compute_discounted_rewards(rewards)
    # Compute gradients and update network
    with tf.GradientTape() as tape:
        action_probs = model(tf.convert_to_tensor(states), training=True)
        # Get probabilities of the actions taken
        selected_probs = tf.gather(action_probs, actions, axis=1, batch_dims=1)
        # Calculate loss
        loss = -tf.math.log(selected_probs) * discounted_rewards
        loss = tf.reduce_mean(loss)
    
    # Backpropagation and parameter update
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

This code uses GradientTape to handle backpropagation automatically, and Keras takes care of the rest.

内容的提问来源于stack exchange，提问作者Jostein Dyrseth