如何将PJSUA2来电音频流对接OpenAI API？

我用Python的PJSUA2搭建了软电话，已实现双向通话、语音互通，也配置了通话录音功能，但通过录音文件对接语音助手的方案行不通。需要找到获取实时通话音频流的方法，用来对接谷歌语音转文字服务及OpenAI API。

当前使用的Call类代码如下：

import pjsua2 as pj


class MyCall(pj.Call):
    def __init__(self, account, call_id, ep):
        super().__init__(account, call_id)
        self.account = account  # Store reference to the account
        self.recorder = None
        self.ep = ep
        self.player = None

    def onCallState(self, prm):
        ci = self.getInfo()
        if ci.state == pj.PJSIP_INV_STATE_DISCONNECTED:
            if self.recorder:
                self.recorder = None

            # Remove this call from the account's list of calls
            self.account.calls.remove(self)

    def onCallMediaState(self, prm):
        print("onCallMediaState")
        ci = self.getInfo()

        for mi in ci.media:
            if mi.type == pj.PJMEDIA_TYPE_AUDIO and mi.status == pj.PJSUA_CALL_MEDIA_ACTIVE:
                audioMedia = self.getAudioMedia(mi.index)

                audio_filename = "greeting.wav"

                # Create a player
                self.player = pj.AudioMediaPlayer()
                # Create a recorder
                self.recorder = pj.AudioMediaRecorder()

                # Capture Device Media is basically the device where the call is being made
                captureDevMed = self.ep.audDevManager().getCaptureDevMedia()  # Get capture device media
                playbackDevMed = self.ep.audDevManager().getPlaybackDevMedia()

                try:
                    # Create Player with Audio file
                    self.player.createPlayer(file_name=audio_filename, options=1)

                    # Start transmitting the media player to the call's audio media
                    self.player.startTransmit(sink=audioMedia)

                    # Create a recorder and start recording to a file
                    self.recorder.createRecorder(file_name="my_recording.wav", options=1)

                    # Start transmitting the call's audio to the recorder
                    audioMedia.startTransmit(self.recorder)

                    # Transmit the call's audio media to the capture device (for you to hear the caller)
                    audioMedia.startTransmit(sink=playbackDevMed)

                    # Transmit from the local machine's microphone to the call's audio media
                    captureDevMed.startTransmit(sink=audioMedia)

                except Exception as e:
                    print(f"Error occurred: {e}")

解决方案：自定义AudioMediaPort捕获实时音频帧 #

要实现实时音频流处理，不能依赖文件录音，而是需要通过自定义AudioMediaPort来直接获取每帧音频数据，再将数据流式传输到语音服务API。

步骤1：创建自定义AudioMediaPort子类 #

继承pj.AudioMediaPort，重写onFrameReceived方法，在这里处理接收到的音频帧：

import pjsua2 as pj

class AudioStreamPort(pj.AudioMediaPort):
    def __init__(self, ep, clock_rate=16000, channels=1, samples_per_frame=320):
        # 初始化端口，匹配语音服务要求的格式（比如16kHz单声道）
        super().__init__()
        self.clock_rate = clock_rate
        self.channels = channels
        self.samples_per_frame = samples_per_frame
        # 配置端口格式
        self.createPort(
            "AudioStreamPort",
            clock_rate,
            channels,
            samples_per_frame,
            pj.PJMEDIA_FORMAT_PCM
        )
        # 用于缓存音频数据，对接流式API
        self.audio_buffer = bytearray()

    def onFrameReceived(self, frame):
        # frame包含原始音频帧数据，类型为pj.MediaFrame
        # 将PCM数据转换为字节流（16位小端格式）
        pcm_data = frame.buffer
        # 直接将数据添加到缓存，或者实时发送到语音服务
        self.audio_buffer.extend(pcm_data)
        
        # 示例：当缓存足够时，发送到语音服务（根据API要求的 chunk 大小调整）
        if len(self.audio_buffer) >= self.samples_per_frame * 2 * self.channels:
            chunk = self.audio_buffer[:self.samples_per_frame * 2 * self.channels]
            self.audio_buffer = self.audio_buffer[self.samples_per_frame * 2 * self.channels:]
            # 这里调用谷歌/OpenAI的流式语音API
            self.send_to_voice_service(chunk)

    def send_to_voice_service(self, audio_chunk):
        # 实现对接谷歌Speech-to-Text或OpenAI Whisper的流式API逻辑
        # 示例（伪代码）：
        # google_streaming_recognizer.feed(audio_chunk)
        # 或
        # openai.Audio.transcribe("whisper-1", audio_chunk, stream=True)
        print(f"Sending {len(audio_chunk)} bytes to voice service")

步骤2：修改MyCall类，替换Recorder为自定义端口 #

在onCallMediaState中，移除Recorder的创建，改为初始化自定义的AudioStreamPort，并将通话的AudioMedia传输到该端口：

class MyCall(pj.Call):
    def __init__(self, account, call_id, ep):
        super().__init__(account, call_id)
        self.account = account
        self.audio_stream_port = None  # 替换recorder为自定义端口
        self.ep = ep
        self.player = None

    def onCallState(self, prm):
        ci = self.getInfo()
        if ci.state == pj.PJSIP_INV_STATE_DISCONNECTED:
            if self.audio_stream_port:
                self.audio_stream_port = None
            self.account.calls.remove(self)

    def onCallMediaState(self, prm):
        print("onCallMediaState")
        ci = self.getInfo()

        for mi in ci.media:
            if mi.type == pj.PJMEDIA_TYPE_AUDIO and mi.status == pj.PJSUA_CALL_MEDIA_ACTIVE:
                audioMedia = self.getAudioMedia(mi.index)

                audio_filename = "greeting.wav"

                self.player = pj.AudioMediaPlayer()
                # 初始化自定义音频流端口
                self.audio_stream_port = AudioStreamPort(self.ep)

                captureDevMed = self.ep.audDevManager().getCaptureDevMedia()
                playbackDevMed = self.ep.audDevManager().getPlaybackDevMedia()

                try:
                    self.player.createPlayer(file_name=audio_filename, options=1)
                    self.player.startTransmit(sink=audioMedia)

                    # 将通话音频传输到自定义端口（替代录音到文件）
                    audioMedia.startTransmit(sink=self.audio_stream_port)

                    audioMedia.startTransmit(sink=playbackDevMed)
                    captureDevMed.startTransmit(sink=audioMedia)

                except Exception as e:
                    print(f"Error occurred: {e}")

关键注意事项 #

1. 音频格式匹配：确保自定义端口的采样率、声道数、位深与语音服务API要求一致（比如谷歌STT默认16kHz单声道16位PCM）。如果PJSUA2默认格式不匹配，需要在创建端口时指定，或者添加格式转换逻辑（比如使用pj.MediaConverter）。
2. 流式API对接：谷歌Speech-to-Text和OpenAI Whisper都支持流式识别，需要在send_to_voice_service中实现对应的客户端逻辑，将音频chunk持续喂给API。
3. 线程安全：PJSUA2的回调方法运行在内部线程，处理音频数据时要注意线程安全，避免阻塞回调线程。

内容的提问来源于stack exchange，提问作者Jorge