将Java CodeLabs适配Python/Flask SocketIO：音频数据格式问题排查

Hey there! Let's work through this audio format hiccup you're facing while adapting Java CodeLabs to Flask-SocketIO. It’s great your WebSocket connection is solid and you’re passing sample rate + audio data successfully—now let’s fix that misaligned audio format.

First, Double-Check Your Frontend Audio Conversion #

From what you shared, you’re using ScriptProcessorNode to handle raw audio bytes. The most common issue here is how you convert the float32 audio data (browser default) to 16-bit signed integers, and how you send it over SocketIO. Here’s a corrected, optimized version of that frontend code:

const context = new AudioContext();
const scriptNode = context.createScriptProcessor(4096, 1, 1);
const MAX_INT = 32767; // Max value for 16-bit signed integers

scriptNode.onaudioprocess = function(e) {
  const inputData = e.inputBuffer.getChannelData(0); // Float32 array (-1.0 to 1.0)
  const int16Data = new Int16Array(inputData.length);

  // Convert float32 to clamped 16-bit signed integers
  for (let i = 0; i < inputData.length; i++) {
    // Clamp values to avoid overflow beyond 16-bit range
    const clampedValue = Math.max(-1, Math.min(1, inputData[i]));
    int16Data[i] = clampedValue * MAX_INT;
  }

  // Critical: Send the raw ArrayBuffer, not the Int16Array itself
  // This avoids SocketIO serializing it into a bulky JSON array
  socket.emit('audio_data', int16Data.buffer);
};

// Hook up the script node to your audio source
yourAudioSource.connect(scriptNode);
scriptNode.connect(context.destination);

Fix the Flask-SocketIO Server Side Parsing #

On the Python end, you need to correctly decode the raw bytes into a usable audio array. Since you’re sending 16-bit signed integers, use NumPy to parse the byte stream efficiently:

from flask import Flask
from flask_socketio import SocketIO, emit
import numpy as np

app = Flask(__name__)
# Enable CORS if your frontend is on a different origin
socketio = SocketIO(app, cors_allowed_origins="*")

# Store sample rate once received (from your existing setup)
current_sample_rate = None

@socketio.on('sample_rate')
def handle_sample_rate(rate):
    global current_sample_rate
    current_sample_rate = rate
    print(f"Set sample rate to: {current_sample_rate} Hz")

@socketio.on('audio_data')
def handle_audio_data(raw_bytes):
    if not current_sample_rate:
        emit('error', 'Sample rate not received yet')
        return

    # Convert raw bytes to a 16-bit signed integer array
    # Match the endianness to your frontend (browsers use little-endian by default)
    audio_array = np.frombuffer(raw_bytes, dtype=np.int16)

    # Optional: Convert back to float32 for processing (matches browser's original format)
    float_audio = audio_array.astype(np.float32) / 32767.0

    # Debug: Verify chunk size and data range
    print(f"Received chunk: {len(audio_array)} samples | Min: {np.min(audio_array)} | Max: {np.max(audio_array)}")

    # Add your audio processing logic here (e.g., save to file, analyze, etc.)

if __name__ == '__main__':
    socketio.run(app, debug=True)

Common Pitfalls to Troubleshoot #

• Endianness Mismatch: If your audio sounds garbled, explicitly set the byte order when parsing in Python:

  audio_array = np.frombuffer(raw_bytes, dtype=np.dtype('int16').newbyteorder('<')) # Little-endian
  

• Incorrect Chunk Size: Your ScriptProcessorNode uses a 4096-sample buffer, so each audio chunk should be 4096 * 2 = 8192 bytes (since each int16 is 2 bytes). Verify on both ends:
  • Frontend: console.log(int16Data.buffer.byteLength) should output 8192
  • Python: print(len(raw_bytes)) should also output 8192
• SocketIO Serialization: Never send the Int16Array directly—always send its underlying ArrayBuffer. Sending the array itself will make SocketIO serialize it into a JSON list, which bloats data and breaks format alignment.

内容的提问来源于stack exchange，提问作者Valentin Coudert