使用ElevenLabs WebSocket向Twilio流音频仅播放白噪音问题排查

以下是你代码中存在的关键问题及对应修正：

1. 核心问题：重复发送累计音频数据 #

你当前每次向Twilio发送的是从开始到当前的所有音频chunk的base64拼接内容，而非仅当前新生成的chunk。Twilio Media Stream会将每次收到的media.payload追加到播放队列，导致前面的音频被重复播放、叠加，最终产生类似白噪音的混乱音效。

修正代码（send_to_twilio_eleven函数） #

async def send_to_twilio_eleven(websocket, message, stream_sid, interruption_event):
    print("Starting TTS streaming from ElevenLabs")
    try:
        start_time = time.time()
        print("message is: ", message)
        
        async for audio_chunk in text_to_speech_stream(message):
            print("Chunk yielded in : ",time.time()-start_time)
            
            # 仅发送当前新生成的chunk，而非累计内容
            audio_delta = {
                "event": "media",
                "streamSid": stream_sid,
                "media": {
                    "payload": audio_chunk
                }
            }

            if interruption_event.is_set():
                print("Interruption detected")
                return

            await websocket.send_json(audio_delta)
            print("received audio sent to twilio")
            await asyncio.sleep(0)

    except Exception as e:
        print(f"Error in send_to_twilio_eleven: {e}")

2. 次要问题：冗余配置与逻辑混乱 #

（1）WebSocket payload冗余字段 #

ElevenLabs WebSocket的认证密钥xi-api-key只需通过请求头传递，无需放入payload；model_id和voice_id已在URL中指定，也无需重复写入payload。

（2）重复JSON解析逻辑 #

原代码中对WebSocket消息做了两次JSON解析，逻辑冗余且可能导致错误信息无法被正确捕获。

修正代码（text_to_speech_stream函数） #

async def text_to_speech_stream(text: str):
    voice_id = "pNInz6obpgDQGcFmaJgB"
    model_id = "eleven_multilingual_v2"

    url = f"wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input?model_id={model_id}"

    headers = {
        'xi-api-key': "你的实际API密钥"  # 填入有效密钥
    }

    payload = {
        "text": " ",
        "voice_settings": {
            "stability": 0.0,
            "similarity_boost": 1.0,
            "style": 0.0,
            "use_speaker_boost": True
        },
        "output_format": "ulaw_8000",
        "flush": True
    }

    async with websockets.connect(url, extra_headers=headers) as ws:
        print("websocket connected: 11labs")
        await ws.send(json.dumps(payload))
        await ws.send(json.dumps({"text": text}))
        await ws.send(json.dumps({"text": ""}))

        start_time = time.time()
        print("waiting for receiving")

        while True:
            try:
                message = await ws.recv()
                if not isinstance(message, str):
                    print("Unknown message type received (not JSON)")
                    continue
                
                data = json.loads(message)
                if data.get("audio"):
                    print("yielded in: ", time.time()-start_time, flush=True)
                    yield data["audio"]
                elif data.get('isFinal'):
                    print("Received final signal, closing stream")
                    break
                elif data.get("warning"):
                    print(f"Warning: {data['warning']}")
                elif data.get("error"):
                    print(f"Error: {data['error']}")
                    break
                else:
                    print(f"Message from ElevenLabs: {data}")
            except websockets.exceptions.ConnectionClosedOK:
                print("Connection closed gracefully")
                break
            except json.JSONDecodeError:
                print(f"Failed to parse JSON message: {message}")
            except Exception as e:
                print(f"Error in ElevenLabs WebSocket: {e}")
                break

3. 额外验证建议 #

如果修正后仍有问题，可将ElevenLabs返回的audio字段base64解码为字节，手动添加ULAW格式的WAV头后保存为文件，验证音频本身是否正常。若保存的文件播放正常，则问题已完全解决；若仍为白噪音，需确认ElevenLabs的output_format参数是否生效（可尝试更换为pcm_8000后自行转ULAW编码）。

内容的提问来源于stack exchange，提问作者Dawny33