MP4容器中H.264帧类型检测与对应数据段关联的实现方法问询

Great question! To tie H.264 frame types (I/P/B) to their exact byte positions in an MP4 and apply targeted noise, you can extend your workflow using ffprobe's detailed frame metadata and a Python script that modifies specific segments. Here's a practical, step-by-step solution:

Step 1: Extract Frame Metadata with ffprobe #

First, use ffprobe to pull not just frame types, but also the byte offset and size of each frame's data in the MP4. This command outputs structured JSON for easy parsing in Python:

ffprobe -v quiet -print_format json -show_frames -show_entries frame=pict_type,pkt_pos,pkt_size,stream_index <your_file>.mp4 > frame_metadata.json

Breaking down the flags:

• -v quiet: Suppresses extra logging noise
• -print_format json: Outputs data in JSON format
• -show_entries frame=...: Filters to only get the fields we care about:
  • pict_type: Frame type (I/P/B)
  • pkt_pos: Byte offset of the frame's start in the file
  • pkt_size: Number of bytes the frame occupies
  • stream_index: Helps filter out audio frames (we only care about video)

Step 2: Parse Metadata & Map Frames to Byte Ranges #

Use Python to load the JSON metadata, filter for video frames, and calculate the start/end byte ranges for each frame type:

import json
import random

def load_frame_metadata(metadata_path):
    with open(metadata_path, 'r') as f:
        data = json.load(f)
    
    frame_ranges = []
    for frame in data['frames']:
        # Skip audio frames (adjust stream_index if your video is not index 0)
        if frame['stream_index'] != 0:
            continue
        
        frame_type = frame['pict_type']
        start_byte = int(frame['pkt_pos'])
        end_byte = start_byte + int(frame['pkt_size'])
        
        frame_ranges.append({
            'type': frame_type,
            'start': start_byte,
            'end': end_byte
        })
    return frame_ranges

Step 3: Apply Differential Noise to Frame Segments #

Now, modify your existing bit-flip script to target specific frame types with custom error rates. For example, you might want to apply less noise to I frames (since they're critical for decoding) and more to P/B frames:

def flip_bits_in_segment(segment, error_rate):
    # Convert bytes to a mutable bytearray
    mutable_segment = bytearray(segment)
    for i in range(len(mutable_segment)):
        if random.random() < error_rate:
            # Flip a random bit in the byte
            bit_pos = random.randint(0,7)
            mutable_segment[i] ^= (1 << bit_pos)
    return bytes(mutable_segment)

def apply_differential_noise(input_mp4, output_mp4, frame_ranges):
    # Define custom error rates per frame type
    error_rates = {
        'I': 0.001,  # Low noise for I frames
        'P': 0.01,   # Medium noise for P frames
        'B': 0.05    # Higher noise for B frames
    }
    
    with open(input_mp4, 'rb') as infile, open(output_mp4, 'wb') as outfile:
        # Copy the entire file first, then overwrite targeted segments
        outfile.write(infile.read())
        outfile.seek(0)
        
        for frame in frame_ranges:
            frame_type = frame['type']
            start = frame['start']
            end = frame['end']
            
            # Skip if we don't have a defined error rate for this frame type
            if frame_type not in error_rates:
                continue
            
            # Read the frame's byte segment
            outfile.seek(start)
            frame_data = outfile.read(end - start)
            
            # Apply noise based on frame type
            noisy_data = flip_bits_in_segment(frame_data, error_rates[frame_type])
            
            # Write the modified data back to the file
            outfile.seek(start)
            outfile.write(noisy_data)

# Run the workflow
if __name__ == "__main__":
    metadata = load_frame_metadata('frame_metadata.json')
    apply_differential_noise('input.mp4', 'noisy_output.mp4', metadata)

Key Notes & Caveats #

• Work on copies: Always use a copy of your original MP4 to avoid permanent corruption.
• Stream index: Adjust the stream_index filter if your video stream isn't the first one (use ffprobe -show_streams <file>.mp4 to check).
• Multi-packet frames: In rare cases, a single frame might span multiple packets. If you encounter this, group packets by frame_number or pkt_pts to get the full byte range of the frame.
• MP4 container structure: The pkt_pos values point directly to the media data (mdat box) in the MP4, so you won't accidentally modify metadata (moov box) which could break the file.

内容的提问来源于stack exchange，提问作者Alexander