In real video content, people speak and gesture with their hands, hold microphones close to their faces, take sips from drinks, and interact with objects around them. These natural behaviors are part of what makes content engaging and authentic, but they create a persistent challenge for AI lip sync technology when anything passes in front of a speaker's mouth.

Most AI dubbing solutions fail to address this challenge, causing obstructions that cover the mouth area disappear unexpectedly - The result breaks immersion and makes dubbed content feel artificial and unwatchable.

We developed an enhanced version of RealFace that solves this problem differently. Instead of erasing obstructions, our system preserves them while maintaining accurate lip synchronization underneath, keeping your content feeling authentic and professionally produced.

The Problem That Breaks AI Lip Sync

When we think of video content, it’s easy to picture clean, polished footage - faces fully visible, subjects neatly framed, actions unfolding without interruptions. But real-world video rarely behaves this way. People shift naturally, handle objects, and often block parts of their own faces - podcast hosts gesture while speaking, conference presenters hold microphones close to their faces, and content creators interact with props during filming. These unscripted moments are what make video engaging and real..

Most AI lip sync systems treat these moments as obstacles to eliminate rather than natural elements to preserve. When a hand crosses in front of a mouth or a microphone blocks part of the face, these systems typically paste an unaesthetic speaking mouth on top of the obstacles. The technical reasoning is straightforward: The obscured face is treated just as any other face, force pasting a generated mouth on the face while ignoring the obstruction.

But this approach creates jarring visual artifacts that immediately signal AI-generated content. Hands that disappear mid-gesture, microphones that fade in and out of existence, and objects that become transparent all destroy the natural flow that made the original content compelling.

RealFace's Advanced Occlusion Handling

Our enhanced RealFace system takes a different approach to these challenges. Rather than erasing obstacles, we developed technology that preserves natural interactions while maintaining accurate lip synchronization beneath them.

We added a new innovative occlusion encoder that identifies and encodes facial occlusions. These facial occlusions are fused with animated facial geometry and the facial texture, along with temporal consistency algorithms, and scene bending techniques. Altogether, our occlusion-aware lip-sync model handles obstructions gracefully - when a hand moves across a speaker's face or a microphone blocks part of their mouth, RealFace maintains the natural interaction while ensuring that lip movements remain properly synchronized with the new audio track.

This means your dubbed content retains all the authentic gestures, natural movements, and real-world interactions that made viewers connect with the original. Speakers can gesture naturally, hold objects close to their faces, and move freely without breaking the AI lip sync or creating visual artifacts.

Applications for Natural Content

This technology particularly benefits content where natural interaction is important. Podcast recordings where hosts gesture while speaking can now be dubbed without losing authentic conversational flow. Conference presentations maintain their professional appearance when speakers naturally adjust microphones or use hand gestures.

User-generated content creators who interact with props, take drinks, or gesture during filming can expand into international markets without sacrificing the authentic personality that built their audience. The dubbed versions preserve all the natural behaviors that made viewers connect with the creator.

Interview formats, educational content, and scenarios where speakers interact naturally with their environment all benefit from handling that preserves rather than eliminates real-world interactions.

See RealFace Occlusions in Action

In real-world video humans speak and gesture at same time.
Occlusions of lips and faces occur naturally.

Other lip-sync models don't know how to handle occlusions and lip sync at the same time, resulting in uncanny video that breaks immersion.

RealFace handles occlusions in a way that looks natural while also applying lip-sync to the new language, resulsting in a video that looks like it was shot in the target language.

Seamless Integration for Professional Results

RealFace's occlusion handling integrates seamlessly with our existing AI lip sync capabilities. You don't need special workflows or additional processing steps. The system automatically detects and handles obstructions while maintaining all the speaker identity preservation, timing accuracy, and visual quality that RealFace delivers.

Ready to see how RealFace handles real-world AI lip sync challenges? Try our platform and experience video dubbing that works with natural human behavior.

Hands and mics no longer break Lip Sync in AI Video Adaptation: Occlusions in RealFace 2.1