多输出CNN训练：游戏操控任务的输出向量与数据采集咨询

Great question—building a CNN to mimic human game controls is such a fun (and surprisingly tricky) project! Let’s break down the optimal data collection pipeline and output vector structure you need to nail this.

1. Real-Time Synchronized Recording #

The biggest risk here is misalignment between game frames and input data—if your screenshot is captured 100ms after a key press, your model will learn garbage. Stick to lightweight, low-latency tools:

• Use mss or PIL.ImageGrab for fast screen capture (avoid heavy tools like OBS that add lag).
• Pair it with pyautogui to record keyboard/mouse state every frame (set your capture rate to match the game’s FPS, e.g., 30 or 60 FPS).
• Timestamp every frame and corresponding input state to enforce strict synchronization. For example:

  import mss
  import pyautogui
  from datetime import datetime
  
  with mss.mss() as sct:
      while True:
          timestamp = datetime.now().timestamp()
          # Capture game window (target a specific monitor/window to avoid desktop clutter)
          frame = sct.grab(sct.monitors[1])
          # Record input state
          keys = {
              "q": pyautogui.keyDown("q"),
              "w": pyautogui.keyDown("w"),
              # ... repeat for e/r/d/f
              "left_click": pyautogui.mouseDown(button='left'),
              "right_click": pyautogui.mouseDown(button='right')
          }
          mouse_pos = pyautogui.position()
          # Save frame + keys + mouse_pos + timestamp to your dataset
  

2. Automated "Human-in-the-Loop" Labeling #

Forget manual labeling—your best training data is just recording actual human players playing the game. Every frame’s input state is automatically labeled by the player’s real actions, which gives you the most accurate representation of how humans pair game visuals with controls. This also naturally captures long presses (e.g., holding w to move) as consecutive frames with the same key activation.

3. Diversify Your Dataset #

Your model will only be as good as the data you feed it:

• Record multiple players (different playstyles, skill levels).
• Cover all in-game scenarios: combat, exploration, menu navigation, boss fights, etc.
• Include edge cases: simultaneous multi-key presses (e.g., q + r + right click), quick mouse flicks, and even "idle" frames where no inputs are active (to avoid biasing the model toward always outputting an action).

4. Preprocessing for Efficiency & Generalization #

Raw 1080p frames will kill your training speed. Optimize:

• Resize frames to a smaller resolution (e.g., 640x360 or 320x180) — human players don’t need pixel-perfect detail to make decisions.
• Normalize pixel values to the range [0, 1] (divide by 255) to stabilize training.
• Optional: Convert to grayscale to cut input dimensions by 3x (test if RGB adds meaningful signal first—many games rely on color cues, so don’t skip this without validation).
• Add mild data augmentation: slight brightness shifts, horizontal flips (if the game isn’t left/right dependent), or small translations to make the model more robust to varying display settings.

The key here is to design a vector that supports any combination of inputs (including long presses) while being easy for your CNN to learn.

Core Design Principles #

• Keep each input type as a separate, independent dimension (no encoding multiple actions into a single value).
• Use binary values for discrete inputs (key presses, mouse clicks) and continuous values for positional inputs (mouse position).

Breakdown by Input Type #

Let’s map your required inputs to vector dimensions:

1. Keyboard Keys: 6 binary dimensions (1 = pressed, 0 = not pressed) for q, w, e, r, d, f.
2. Mouse Clicks: 2 binary dimensions for left-click and right-click activation (1 = held/clicked, 0 = not active).
3. Mouse Position: 2 normalized continuous dimensions. Divide the raw pixel coordinates by your screen’s width/height to get values in [0, 1] (e.g., if your screen is 1920x1080, a mouse at (960, 540) becomes (0.5, 0.5)). This makes your model resolution-agnostic.

Total output vector length: 10 dimensions.

Example Output Vector #

Suppose a player is holding q and w, right-clicking, and has their mouse at (1200, 800) on a 1920x1080 screen. The output vector would be:
[1, 1, 0, 0, 0, 0, 0, 1, 0.625, 0.741]
(Order: q, w, e, r, d, f, left_click, right_click, mouse_x, mouse_y)

Key Implementation Notes #

• For the binary output dimensions, use a sigmoid activation function (outputs [0,1]) and threshold at 0.5 to determine if an action is active. This lets the model learn confidence levels for each action.
• For mouse position, use a linear activation (or tanh to constrain to [-1,1], then shift to [0,1]) since it’s a regression task.
• Long presses don’t need special handling—just ensure consecutive frames have the same binary activation for the held key/click. The model will learn that sustained activation corresponds to holding the input.

• Add temporal context: Pure CNNs struggle with sequential dependencies (e.g., "press q right after w"). Pair your CNN with an LSTM or Transformer encoder to process sequences of frames (e.g., 10 consecutive frames) — this will help the model mimic human decision-making over time.
• Clean your data: Remove frames where mouse position jumps drastically (likely a recording glitch) or inputs are inconsistent. Use a sliding window to smooth mouse position if needed.
• Balance your dataset: If certain actions (e.g., pressing w) are overrepresented, use weighted loss functions or resample your data to avoid bias.

内容的提问来源于stack exchange，提问作者John Fisher