Great question—this is a super common pain point when using segmented MP4 with MSE for live streaming. Let’s unpack what’s going on and how to fix it.

Does MSE support starting playback from the middle of a segmented MP4 stream? #

Yes, it does—but your current setup is missing critical pieces that let late-joining clients sync up properly. The issue right now is that when a client connects after the first few segments, you’re sending the initial ftyp/moov boxes plus the latest segment, but that latest segment likely doesn’t give the decoder a valid starting point.

Here’s why that fails:

• MP4 decoders need a keyframe to start decoding (they can’t start in the middle of a GOP, or group of pictures). If the latest segment you send starts with a non-keyframe, the decoder can’t process it.
• The moov box contains track metadata, but if your codec (like H.264) requires additional configuration data (SPS/PPS), that might only be present in the earliest segments—late clients never get that data, so the decoder can’t initialize.

How to fix your segmented MP4 setup #

Follow these steps to enable late-join playback:

1. Force regular keyframes with FFmpeg
   Configure FFmpeg to emit keyframes at fixed intervals using the -g flag (e.g., -g 30 for a keyframe every 30 frames, which is 1 second at 30fps). This ensures every Nth segment starts with a keyframe, giving late clients a valid starting point. Use this alongside segmented MP4 flags like -movflags frag_keyframe+empty_moov+default_base_moof—this makes the moov box self-contained and ensures each fragment is keyframe-aligned.

2. Send the right sequence to late clients
   When a new client connects:

   • First send the ftyp and moov boxes (same as initial clients).
   • Next, send the most recent keyframe-containing segment (not just the absolute latest segment). This gives the decoder a sync point it can actually decode.
   • After that, send subsequent segments in the order FFmpeg generates them.

3. Ensure codec config is included in the initial setup
   For codecs like H.264, make sure the SPS/PPS data is embedded in the moov box (the -movflags mentioned above helps with this). If not, you’ll need to extract this data from the first segment and send it to late clients before any segments—wrap it in an avcC box so MSE can recognize it.

4. Validate segment timestamps
   Double-check that each segment’s moof box has accurate, monotonic timestamps. MSE uses these timestamps to place segments correctly in the media source buffer; mismatched timestamps can cause playback failures or gaps.

Alternative formats for WebSocket live streaming #

If segmented MP4 is still giving you trouble, these formats are better optimized for live, late-join scenarios over WebSocket:

• WebM: A container format that’s more flexible for live streaming than MP4. It supports keyframe-based fragmenting natively, and VP9/Vorbis codecs work seamlessly with MSE. Use FFmpeg’s -f webm flag with -dash 1 to generate live-friendly fragments.
• MPEG-TS: A stream-based format designed for live transmission. It includes periodic keyframes and codec config data, so late clients can start decoding as soon as they receive a keyframe. FFmpeg outputs MPEG-TS with -f mpegts, and MSE has built-in support for TS streams via the mpegts media source extension.
• HLS/DASH (adapted for WebSocket): While these are HTTP-based protocols, you can adapt their segment formats to be sent over WebSocket. They’re purpose-built for live streaming with late-join support, handling sync and segment ordering automatically. If you can’t switch from WebSocket, you can push HLS segments or DASH fragments over the connection instead of raw MP4 boxes.

内容的提问来源于stack exchange，提问作者Elias Wagnerberger