谷歌云VM上Speaker Diarization方案问题及优化需求问询

一、谷歌Speech API仅返回最多2个说话人问题 #

核心原因 #

1. 关键参数缺失：未指定音频声道数、采样率，且注释了max_speaker_count，导致API自动限制识别逻辑；
2. 结果解析错误：直接将识别结果转成字符串存储，丢失了结构化的说话人分段数据，无法正确统计实际识别到的说话人数量。

修复方案 #

补全识别配置参数，并正确解析结构化结果，修改后的代码如下：

from google.cloud import speech_v1p1beta1 as speech
from google.cloud import storage
import os
import json
import sys

storage_client = storage.Client()
client = speech.SpeechClient()

if "--channel" in sys.argv:
    index = sys.argv.index("--channel") + 1
    if index < len(sys.argv):
        channel = sys.argv[index]
        print("Channel:", channel)
    else:
        print("--channel option requires a value")

audio_folder = f'audio_{channel}'
transcript_folder = f'transcript_output'

bucket = storage_client.bucket(audio_folder)
bucket2 = storage_client.bucket(transcript_folder)
wav_files = [i.name for i in bucket.list_blobs()]
# 修正JSON文件匹配逻辑，避免路径分割错误
json_files = [blob.name.split('/')[-1] for blob in bucket2.list_blobs(prefix=f'{channel}/')]

for file in wav_files:
    if not file.endswith('.wav'):
        continue
    transcript_name = file.replace('.wav', '.json')
    if transcript_name in json_files:
        continue
    gcs_uri = f"gs://{audio_folder}/{file}"
    audio = speech.RecognitionAudio(uri=gcs_uri)

    diarization_config = speech.SpeakerDiarizationConfig(
        enable_speaker_diarization=True,
        min_speaker_count=2,
        max_speaker_count=10,  # 取消注释并设置合理最大值
        use_speaker_labels=True,
    )
    # 根据实际音频设置采样率，常见为16000Hz，可通过ffprobe工具自动获取
    config = speech.RecognitionConfig(
        encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
        sample_rate_hertz=16000,
        language_code="es-ES",
        diarization_config=diarization_config,
        audio_channel_count=2,  # 单声道设为1，立体声设为2
    )

    print(f"Processing {file}...")
    operation = client.long_running_recognize(config=config, audio=audio)
    response = operation.result()

    # 正确解析说话人分段数据
    speaker_segments = []
    result = response.results[-1]
    words_info = result.alternatives[0].words
    for word_info in words_info:
        speaker_segments.append({
            "start_time": word_info.start_time.total_seconds(),
            "end_time": word_info.end_time.total_seconds(),
            "speaker": word_info.speaker_tag,
            "word": word_info.word
        })

    # 保存结构化结果
    with open(transcript_name, 'w', encoding='utf-8') as f:
        json.dump(speaker_segments, f, ensure_ascii=False, indent=2)
    
    os.system(f'gsutil cp {transcript_name} gs://transcript_output/{channel}/{transcript_name}')
    os.remove(transcript_name)
    print(f'File {file} processed. ')

二、pyannote速度过慢问题 #

速度判断 #

30分钟音频耗时3小时（6倍实时速度）在NVIDIA T4上属于偏慢，正常T4运行pyannote应达到1-2倍实时速度。

优化方案 #

1. 代码层面优化

• 减少IO开销：用gcsfuse将云存储桶挂载到VM本地，直接读写挂载路径文件，避免反复gsutil cp；
• 多进程并行处理：利用CPU多核同时处理多个音频文件；
• 升级模型版本：使用pyannote/speaker-diarization@3.1，相比2.1版本有明显速度优化；
• 强制GPU加速：确保pipeline加载到CUDA设备。

修改后的优化代码：

import os
from multiprocessing import Pool
from pyannote.audio import Pipeline
from google.cloud import storage
import sys

storage_client = storage.Client()
if "--channel" in sys.argv:
    index = sys.argv.index("--channel") + 1
    if index < len(sys.argv):
        channel = sys.argv[index]
        print("Channel:", channel)
    else:
        print("--channel option requires a value")

audio_folder = f'audio_{channel}'
transcript_folder = f'transcript_{channel}'
bucket = storage_client.bucket(audio_folder)
bucket2 = storage_client.bucket(transcript_folder)
wav_files = [i.name for i in bucket.list_blobs()]
rttm_files = [blob.name.split('/')[-1] for blob in bucket2.list_blobs()]

# 使用最新版本模型
token = "XXX"
pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization@3.1",
    use_auth_token=token
)
# 强制加载到GPU
pipeline.to("cuda")

def process_file(file):
    if not file.endswith('.wav'):
        return
    rttm_name = file.replace('.wav', '.rttm')
    if rttm_name in rttm_files:
        return
    if '2023' not in file:
        return
    
    print(f'Doing file {file}')
    # 假设已通过gcsfuse将gs://audio_{channel}挂载到/mnt/audio
    local_file_path = f'/mnt/audio/{file}'
    diarization = pipeline(local_file_path)                             
    with open(rttm_name, "w") as rttm:
        diarization.write_rttm(rttm)        
    # 上传到挂载的转录桶路径
    os.system(f'cp {rttm_name} /mnt/transcript/{rttm_name}')
    os.remove(rttm_name)

# 进程数根据VM CPU核心数设置（如4核设为4）
if __name__ == "__main__":
    with Pool(processes=4) as pool:
        pool.map(process_file, wav_files)

2. VM与环境优化

• 验证GPU环境：运行torch.cuda.is_available()确认torch已启用GPU加速，若未启用需重新安装GPU版本torch；
• 升级实例配置：若T4性能不足，可切换到A10G等更高性能GPU实例，同时增加CPU核心数和内存，避免CPU成为瓶颈；
• 清理依赖环境：用conda创建独立环境，安装最新版本的pyannote.audio、torch、torchaudio，避免依赖冲突。

3. 替代方案

若pyannote优化后仍无法满足需求，可尝试：

• WhisperX：基于OpenAI Whisper的说话人识别工具，速度与准确率平衡较好，原生支持GPU加速；
• 谷歌Cloud Speech-to-Text：解决第一个问题后，若效果符合要求，优先使用云API，无需维护本地模型。

内容的提问来源于stack exchange，提问作者Luis