Swift 4录制视频时添加音频的技术实现疑问

Hey there! Let's work through this audio-video recording and merging challenge you're dealing with. It’s super common to hit roadblocks with outdated docs or siloed tutorials, so let’s break this down step by step to fit your existing function structure.

First: Add Audio Capture to Your Setup #

You’ll want to initialize your audio device in your "prepare session" function—this keeps your setup clean and ensures audio is ready to go when recording starts. Here’s how to approach it (using Python as an example, since you mentioned functions):

• Pick a reliable audio library (like pyaudio for cross-platform capture)
• Initialize the audio stream with matching parameters to your video (e.g., if your video is 30fps, set your audio chunk size to capture ~1/30 seconds of audio per frame)

Example snippet for your prepare function:

import pyaudio
import cv2

def prepare_session():
    # Video setup
    video_cap = cv2.VideoCapture(0)  # Your video device
    fps = video_cap.get(cv2.CAP_PROP_FPS)
    frame_width = int(video_cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    frame_height = int(video_cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    # Audio setup
    audio_p = pyaudio.PyAudio()
    audio_format = pyaudio.paInt16
    channels = 1
    rate = 44100
    # Calculate chunk size to match video frame interval
    chunk = int(rate / fps)
    audio_stream = audio_p.open(format=audio_format,
                                channels=channels,
                                rate=rate,
                                input=True,
                                frames_per_buffer=chunk)

    return video_cap, audio_p, audio_stream, fps, frame_width, frame_height

Second: Capture Audio & Video Simultaneously During Recording #

In your "record session" function, you’ll need to capture both video frames and audio data in sync. Store each in temporary buffers—this ensures you don’t drop data and keeps alignment easy.

Example recording function:

def record_session(video_cap, audio_stream, fps, duration_seconds):
    video_frames = []
    audio_frames = []
    total_frames = int(fps * duration_seconds)

    for _ in range(total_frames):
        # Capture video frame
        ret, frame = video_cap.read()
        if ret:
            video_frames.append(frame)
        
        # Capture matching audio chunk
        audio_data = audio_stream.read(int(audio_stream.getframerate() / fps))
        audio_frames.append(audio_data)
    
    return video_frames, audio_frames

Third: Merge & Save the Final File #

Forget the old, clunky merge methods—use FFmpeg (either via command line or a wrapper library like moviepy) for reliable, modern merging. This handles encoding compatibility and sync automatically.

Option 1: Use FFmpeg Command Line (Simple & Reliable)

First, save your temporary video and audio files, then run this command (you can call it via subprocess in your code):

ffmpeg -i temp_video.mp4 -i temp_audio.wav -c:v copy -c:a aac -strict experimental final_output.mp4

• -c:v copy skips re-encoding the video (faster)
• -c:a aac encodes audio to a format compatible with MP4 containers

Option 2: Integrate Merge into Your Save Function

Here’s how to wrap this into your existing save function:

import subprocess
import wave
import cv2

def save_output(video_frames, audio_frames, audio_p, fps, frame_width, frame_height, output_path):
    # Save temporary video
    temp_video_path = "temp_video.mp4"
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    video_writer = cv2.VideoWriter(temp_video_path, fourcc, fps, (frame_width, frame_height))
    for frame in video_frames:
        video_writer.write(frame)
    video_writer.release()

    # Save temporary audio
    temp_audio_path = "temp_audio.wav"
    wf = wave.open(temp_audio_path, 'wb')
    wf.setnchannels(audio_p.get_default_input_device_info()['maxInputChannels'])
    wf.setsampwidth(audio_p.get_sample_size(pyaudio.paInt16))
    wf.setframerate(44100)
    wf.writeframes(b''.join(audio_frames))
    wf.close()

    # Merge with FFmpeg
    command = [
        'ffmpeg',
        '-i', temp_video_path,
        '-i', temp_audio_path,
        '-c:v', 'copy',
        '-c:a', 'aac',
        '-strict', 'experimental',
        output_path
    ]
    subprocess.run(command, check=True)

    # Clean up temp files (optional)
    import os
    os.remove(temp_video_path)
    os.remove(temp_audio_path)

Key Tips to Avoid Headaches #

• Sync is king: Always match your audio chunk size to your video’s frame interval (like we did with chunk = int(rate / fps)). This prevents drift.
• Ditch deprecated APIs: Avoid old OpenCV audio functions or custom encoding—FFmpeg is maintained and handles edge cases way better.
• Test with short recordings first: Debug sync issues with 5-second clips before moving to longer sessions.

内容的提问来源于stack exchange，提问作者Brewski