Azure Kinect DK与Pupil Core双相机同步问题求助

问题描述 #

Pupil Core 录制内容比 Azure Kinect DK 延迟1秒，尝试手动添加sleep命令同步无效，当前使用 Pupil Core Network API 控制录制，核心代码如下：

import zmq
import msgpack as serializer
import subprocess
import threading
import time

def record_video():
    # Path to k4arecorder
    k4a_recorder_path = r"C:\Program Files\Azure Kinect SDK v1.4.1\tools\k4arecorder.exe"

    # Command arguments for k4arecorder
    duration = 30
    output_path = "master1.mkv"

    # Construct the command
    command = f'"{k4a_recorder_path}" --external-sync master -l {duration} {output_path}'

    # Call k4arecorder command to record video
    subprocess.call(command, shell=True)

def record_sound():
    # Path to Python interpreter and sound recording script
    python_command = r"C:\Users\scrim\AppData\Local\Programs\Python\Python311\python.exe"
    sound_recording_script = "sound_recording.py"

    # Command arguments for sound recording script
    duration = 30
    output_file = "sound_recording.wav"

    # Construct the command
    command = [python_command, sound_recording_script, "--duration", str(duration), "--output", output_file]

    # Call sound recording script
    subprocess.call(command)

if __name__ == "__main__":
    # Setup zmq context and remote helper
    ctx = zmq.Context()
    socket = zmq.Socket(ctx, zmq.REQ)
    socket.connect("tcp://127.0.0.1:50020")

    # Measure round trip delay
    t = time.time()
    socket.send_string("t")
    print(socket.recv_string())
    print("Round trip command delay:", time.time() - t)

    # Set current Pupil time to 0.0
    socket.send_string("T 0.0")
    print(socket.recv_string())

    # Start recording
    time.sleep(1)
    socket.send_string("R")
    print(socket.recv_string())

    # Create separate threads for audio and video recordings
    video_thread = threading.Thread(target=record_video)
    audio_thread = threading.Thread(target=record_sound)

    # Start the threads
    video_thread.start()
    audio_thread.start()

    # Wait for both threads to complete
    video_thread.join()
    audio_thread.join()

    # Send notification:
    def notify(notification):
        """Sends ``notification`` to Pupil Remote"""
        topic = "notify." + notification["subject"]
        payload = serializer.dumps(notification, use_bin_type=True)
        socket.send_string(topic, flags=zmq.SNDMORE)
        socket.send(payload)
        return socket.recv_string()

可行解决方向 #

1. 统一录制触发时序 #

当前代码先启动Pupil录制，再开启Azure和音频线程，时序差是延迟核心。调整逻辑为同时触发所有录制动作：

• 先初始化所有线程/进程资源，再同步发送Pupil启动指令、启动Azure和音频录制。
• 利用已测量的RTT延迟做补偿，比如在启动硬件录制后，等待RTT延迟/2再发送Pupil启动命令，抵消指令传输的时间差。

2. 绑定设备时间戳 #

• 不要固定设置Pupil时间为0.0，而是读取Azure Kinect的录制起始时间戳，将Pupil时间对齐到该值，实现时间轴统一。
• 录制开始时，向Pupil发送自定义时间标记事件，后期可通过该标记在剪辑软件中精准对齐两段素材。

3. 优化进程启动控制 #

用subprocess.Popen替代subprocess.call，可以实时获取进程启动状态，确保硬件录制真正启动后再触发Pupil录制，避免线程启动的隐性延迟。

4. 精准测量延迟值 #

做视觉触发测试：用闪屏/灯光同时被两个设备捕捉，对比两段素材中触发事件的时间差，得到精确延迟数值后，通过T <延迟值>命令给Pupil设置时间偏移（比如Pupil慢1秒，就设置T 1.0让时间轴提前1秒）。

调整后代码示例 #

if __name__ == "__main__":
    # Setup zmq context and remote helper
    ctx = zmq.Context()
    socket = zmq.Socket(ctx, zmq.REQ)
    socket.connect("tcp://127.0.0.1:50020")

    # Measure round trip delay
    t = time.time()
    socket.send_string("t")
    print(socket.recv_string())
    rtt_delay = time.time() - t
    print("Round trip command delay:", rtt_delay)

    # 提前初始化线程，不启动
    video_thread = threading.Thread(target=record_video)
    audio_thread = threading.Thread(target=record_sound)

    # 对齐Pupil时间轴
    socket.send_string("T 0.0")
    print(socket.recv_string())
    
    # 同步启动硬件录制
    video_thread.start()
    audio_thread.start()
    
    # 补偿RTT延迟后启动Pupil录制
    time.sleep(rtt_delay / 2)
    socket.send_string("R")
    print(socket.recv_string())

    # 等待所有录制完成
    video_thread.join()
    audio_thread.join()

    # 停止Pupil录制
    socket.send_string("r")
    print(socket.recv_string())

内容的提问来源于stack exchange，提问作者Junaid