Let's dive into your setup and tackle each of your questions with practical, real-time-friendly solutions tailored to your RTX 4090 capabilities. First, a quick recap to align on the problem context:

Your Setup & Pipeline #

• Hardware: RTX 4090, running at 25fps
• Software: Modified Linly-Talker-Stream (LiveTalk fork) with MuseTalk v1.5, using WebRTC for real-time virtual avatar lip sync
• Pipeline Logic:
  • Speaking: MuseTalk generates a new mouth region and blends it into the source frame
  • Silent: Skip MuseTalk, output the original source frame directly

Core Problem #

When speech ends, the pipeline switches instantly from a MuseTalk-generated frame to the original source frame. Since MuseTalk's generated mouth is darker and less saturated than the real footage (a known limitation: "Original facial details like lip shape and color are not well preserved"), this creates a jarring color jump viewers can easily spot.

Attempted Solutions (With Mixed Outcomes) #

You’ve already tested a few approaches, but none fully resolved the issue:

• Slowing blend weight decay (0.05 → 0.02): Made the color shift more noticeable instead of reducing it
• LAB color matching for mouth/skin regions: Fixed cheek areas but left the mouth dim; pushing correction further caused oversaturation
• Hold-then-dissolve transition: Softened the transition edge but the color mismatch remained perceptible

Answers to Your Questions #

1. How to Hide the Speech-End Transition (Real-Time Compatible with 25fps) #

The key is to eliminate the color mismatch before the transition, rather than just smoothing the switch. Here are actionable steps:

a. Dynamic Mouth Color Calibration

Instead of global color adjustments, calibrate the MuseTalk-generated mouth to match the original mouth's color stats in real-time:

• When speech starts, capture a reference frame of the original mouth (use your face box to isolate it) and compute its average LAB color values (L for lightness, A/B for color channels).
• For every MuseTalk-generated mouth frame, adjust its LAB values to match the reference's averages. Since this only modifies the mouth region, it won't affect other facial areas and runs fast enough for 25fps on RTX 4090.
• Integrate this into your paste_back_frame or blending function.

b. Combine Calibration with Your Hold-Then-Dissolve Logic

Once the cached speaking frame's mouth color matches the original, the cross-fade will be nearly invisible. You can even shorten the dissolve time to ~100ms (2-3 frames) since the color gap is eliminated. Modify your silent branch to use the calibrated cached frame:

# In the SPEAKING branch, cache a color-calibrated version
_calibrated_last_speaking = calibrate_mouth_color(combine_frame, original_mouth_ref)
_last_speaking_composite = _calibrated_last_speaking.copy()

c. Add a Short Silence Threshold

Avoid triggering the silent transition too early. Use a 50ms audio silence threshold instead of switching on the first silent frame—this prevents false triggers from brief pauses and gives the pipeline time to transition naturally.

2. Algorithm to Preserve Original Mouth Texture & Avoid Blurriness #

MuseTalk's blurriness stems from generating the entire mouth region from scratch. Instead, use a texture transfer approach that keeps the original mouth's details while modifying only the lip shape:

a. Lip Shape Warping with Original Texture

• Use a landmark detector like MediaPipe Face Mesh to extract 2D lip landmarks from both the original frame and the MuseTalk-generated frame.
• Warp the original mouth's texture to match the MuseTalk-generated lip shape using affine or thin-plate spline warping.
• Blend this warped texture into the frame instead of using the full MuseTalk-generated mouth. This preserves original color and texture while maintaining accurate lip sync.

Modify your blending function to use warped texture:

def warp_original_mouth(original_frame, lip_landmarks_original, lip_landmarks_musetalk):
    # Use OpenCV's warpPerspective or cv2.remap to warp original mouth to match MuseTalk's lip shape
    # Implement landmark-based warping logic here
    return warped_mouth

# In the SPEAKING branch
warped_mouth = warp_original_mouth(original_frame, original_lm, musetalk_lm)
# Replace MuseTalk-generated mouth with warped original texture
face[y-y_s:y1-y_s, x-x_s:x1-x_s] = warped_mouth

b. Fine-Tune MuseTalk on Your Avatar Footage

If you have a small dataset of your avatar's face, fine-tune MuseTalk to better preserve texture. With an RTX 4090, you can fine-tune on a few hundred frames in a couple of hours, which will reduce blurriness and color mismatch natively.

3. Alternative Open-Source Lip Sync Models for RTX 4090 (Real-Time, Better Quality) #

These models offer improved texture preservation and run smoothly at 25fps on RTX 4090:

• Wav2Lip 2: The updated version focuses on lip region modification (not full face synthesis), keeping original details intact while delivering accurate lip sync.
• TalkNet-2: Optimized for real-time performance, it combines audio feature extraction with facial landmark warping to preserve texture and minimize color mismatch.
• SadTalker v0.0.2: While built for full face animation, its lip sync mode preserves facial details well and runs at 25fps on RTX 4090 with optimized inference settings.

All have open-source implementations that can be adapted to your Linly-Talker-Stream pipeline with minimal changes.

Quick Color Calibration Implementation #

Add this function to your codebase to implement dynamic mouth color matching:

import cv2
import numpy as np

def calibrate_mouth_color(composite_frame, original_mouth_ref, face_box):
    x, y, x1, y1 = face_box
    # Isolate mouth regions
    composite_mouth = composite_frame[y:y1, x:x1]
    
    # Convert to LAB color space for better color adjustment
    composite_lab = cv2.cvtColor(composite_mouth, cv2.COLOR_BGR2LAB)
    original_lab = cv2.cvtColor(original_mouth_ref, cv2.COLOR_BGR2LAB)
    
    # Calculate mean color values
    orig_mean = np.mean(original_lab, axis=(0,1))
    comp_mean = np.mean(composite_lab, axis=(0,1))
    
    # Adjust composite mouth to match original mean
    composite_lab[:, :, 0] += (orig_mean[0] - comp_mean[0])
    composite_lab[:, :, 1] += (orig_mean[1] - comp_mean[1])
    composite_lab[:, :, 2] += (orig_mean[2] - comp_mean[2])
    
    # Clip values to valid color range
    composite_lab = np.clip(composite_lab, 0, 255).astype(np.uint8)
    
    # Replace mouth in composite frame
    calibrated_frame = composite_frame.copy()
    calibrated_frame[y:y1, x:x1] = cv2.cvtColor(composite_lab, cv2.COLOR_LAB2BGR)
    return calibrated_frame

内容的提问来源于stack exchange，提问作者Jimmy Fadel