LADDER: Landmark-Only Lip-Sync Deepfake Detection Across Generators and Languages

arXiv:2604.16808 · cs.CV · Submitted 2026-04-18 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "LADDER: Landmark-Only Lip-Sync Deepfake Detection Across Generators and Languages".

Jane: Here's the gist of the paper. The gist: BioLip introduces a novel deepfake detection method that exploits biomechanical constraints in authentic lip motion, using kinematic statistics from perioral landmarks to achieve strong cross-lingual and zero-shot generalization across unseen generators.

Tom: How it works BioLip takes a video clip as input and outputs a real/fake score by processing only normalized landmark coordinates—no pixels, no audio The pipeline has three stages: extract and normalize perioral landmarks from each frame, compute kinematic features over 25-frame sliding windows, and classify with a three-branch fusion network.

Jane: Coordinate-based representations suppress appearance cues and retain only motion dynamics.

Tom: If the synthetic lip motion violates physical constraints, the violation appears in landmark kinematics.

Title and authors: Tom: So we're looking at a paper called "LADDER: Landmark-Only Lip-Sync Deepfake Detection Across Generators and Languages." The title says it focuses on using landmarks only, which is a big deal because it cuts out all the pixels and audio data.

Jane: That’s what I thought. It means they are trying to build a detector that doesn't rely on what the video looks like or how much sound is in there, but rather just where the lips are moving.

Lu: Exactly. The core idea here is that authentic lip motion has physical limits set by tissue mechanics and neuromuscular bandwidth, which generative models often ignore.

Meng: I’m curious about how they handle those constraints when they're only looking at coordinates, because if the coordinate estimation itself is off, that introduces its own kind of noise.

Lalam: The paper suggests that by looking at velocity, acceleration, and jerk statistics from these landmark coordinates over short windows, you can find a signal called temporal lip jitter.

Tom: Temporal lip jitter sounds like the key thing here because it’s described as a forgery signal that persists across different generators and languages.

Jane: So if they can capture that physical constraint violation using only coordinates, doesn't that solve the problem of zero-shot generalization?

Lu: That’s what they claim. They train only on English data and test it on five unseen generators and seven languages, which shows that the feature is independent of the training distribution.

Meng: So how robust is this thing when you look at real-world stuff? Does it break down when we introduce heavy compression, like if the video gets very low quality?

Tom: Well, they tested it against HEVC compression attacks and found that their coordinate-based features degrade much less than pixel-based ones.

Jane: That’s a huge win for practical deployment because we often deal with compressed video. So what’s the actual performance looking like when the quality drops?

The paper's summary: Jane: Okay, so let's look at the summary of this LADDER paper. It says existing detectors usually rely on pixel artifacts or audio-visual correspondence, which fails when a generator or language changes.

Tom: And LADDER takes a different approach by focusing on biomechanical constraints in lip motion instead of those surface-level details. They are computing kinematic statistics from sixty-four perioral landmarks over short sliding windows <ref:2604.16808#pg1,computing kinematic statistics from 64 perioral landmarks over short sliding windows>.

Lu: They extract four statistical features—displacement, velocity, acceleration, and jerk—for each landmark sequence over twenty-five-frame sliding windows to get a two hundred fifty-six-dimensional feature vector.

Meng: That’s a lot of math. So what does this process actually look like in practice? Is this something you can easily run at scale on a server?

Tom: The whole extraction pipeline runs entirely on the device, and only about one KB of coordinate vector leaves the hardware, which is pretty efficient for privacy <ref:2604.16808#pg1>.

Jane: It’s designed to suppress appearance cues and retain only motion dynamics, which is why it’s supposed to be more robust against texture-based attacks.

Lalam: The network architecture uses a three-branch system that processes these features in parallel: one temporal branch, one region branch, and one statistical branch.

Lu: The temporal branch looks at the full sequence of features by using three 1D convolutional blocks with channel sizes going from two hundred fifty-six down to sixty-four.

Tom: And then they combine the outputs of those three branches through a classification head to get a final score, which is a vector of size two hundred twenty-four dimensions.

Jane: So the core summary is that by using these kinematic statistics, they create a signal called temporal lip jitter that is independent of the specific generator or language used.

The paper's improvements: Tom: Now let’s talk about what they suggest improving in this work. They point out that the current method only looks at the first few derivative orders, so they want to use all four derivative orders—displacement, velocity, acceleration, and jerk.

Jane: That makes sense because they claim that all four derivative orders carry discriminative information, with Cohen’s d values between zero point two one and zero point four zero on AVLips data <ref:2604.16808#pg2,all four derivative orders carry discriminative information>.

Lu: They argue that using all four of these orders jointly means the detector doesn't depend on any single physical signature being uniquely discriminative, which strengthens the generalization across different forgery methods.

Meng: So if we only use one derivative order, say just acceleration, they suggest it might underperform at certain compression levels.

Tom: They actually found that the full multi-order model doesn't always beat a single-order variant at every compression level; for instance, at CRF=twenty-three it scores zero point zero two one points below the acceleration-only variant.

Jane: That’s a fair caveat because we need to know what the real trade-offs are when deploying this on actual videos.

Lalam: The paper also discusses video temporal consistency as another line of work they are related to, noting that optical flow methods face distribution dependency issues similar to pixel-based detectors.

Lu: LADDER differs because its temporal reference comes from biomechanical constraints on articulator dynamics rather than statistics learned over training videos, which is why it translates better across languages.

Conclusion: Jane: So to wrap up the discussion on "LADDER: Landmark-Only Lip-Sync Deepfake Detection Across Generators and Languages," the main implication is that using physical constraints in motion—velocity, acceleration, and jerk—is a way to create a forgery signal that works where previous methods fail.

Tom: It’s about exploiting that temporal lip jitter because it reflects a physical property of lip movement rather than patterns specific to one dataset or language.

Lu: The architecture, with its three-branch fusion network processing those kinematic statistics, allows it to achieve zero-shot detection across five unseen generators and seven languages.

Meng: From a deployment standpoint, the fact that the feature extraction only requires landmarks means you can run this on device with very little data leaving the hardware.

Lalam: It’s a significant step forward because it moves away from audio or language-dependent features, which are known to shift when you move between different recording conditions or speakers.

Jane: The authors flag that the most significant practical challenge right now is heavy compression, like CRF=forty because blocking artifacts can degrade the landmark localization and partially smooth out that jitter signal itself <ref:2604.16808#pg2>.

Tom: They also note a limitation: the method measures kinematic variance of MediaPipe landmark coordinates, which mixes genuine lip motion with any error in that specific tracking system.

Lu: But overall, they show substantial performance gains over prior pixel-based methods on unseen generators and languages across those compression levels.

Hao Chen, Junnan Xu

cs.CV

Submitted: 2026-04-18

Updated: 2026-10-19

Comments: 8 pages, 2 figures, 5 tables. Keywords: Deepfake detection, lip-sync, facial landmarks, domain generalization, frame rate, cross-lingual, robustness

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist: BioLip introduces a novel deepfake detection method that exploits biomechanical constraints in authentic lip motion, using kinematic statistics from perioral landmarks to achieve strong

Key concepts

Kinematic Variance: This refers to the statistical variation in lip movement features like displacement, velocity, acceleration, and jerk over time. Real biological lips have strict physical limits on how fast they can move or change direction; synthetic models do not respect these limits, causing their motion statistics to be unnaturally erratic.
Perioral Landmarks: These are specific points tracked on the face's mouth area during video analysis. BioLip extracts coordinates from these landmarks instead of raw pixels, focusing solely on the motion dynamics rather than appearance details like skin texture or lip shape.
Cross-Lingual Generalization: This means the detection method works effectively even when tested on videos generated by models in different languages or unseen generators. Because the features rely on physical motion laws rather than specific language sounds, the signal remains consistent regardless of what is being said.

Terminology

Generator — 4× in this paper · explained in 2 episodes

Summary

The gist: BioLip introduces a novel deepfake detection method that exploits biomechanical constraints in authentic lip motion, using kinematic statistics from perioral landmarks to achieve strong cross-lingual and zero-shot generalization across unseen generators.

How it works

BioLip takes a video clip as input and outputs a real/fake score by processing only normalized landmark coordinates—no pixels, no audio The pipeline has three stages: extract and normalize perioral landmarks from each frame, compute kinematic features over 25-frame sliding windows, and classify with a three-branch fusion network. Coordinate-based representations suppress appearance cues and retain only motion dynamics. If the synthetic lip motion violates physical constraints, the violation appears in landmark kinematics.

Why Kinematic Variance Is a Forgery Signal

Real lip movement is produced by a biological system with hard physical limits. Generative models do not have such a constraint. A network predicting pixel values one frame at a time has no loss term penalizing trajectory roughness. Small errors in frame t get corrected in frame t + 1 without any memory of the correction, so the trajectory jitters. We call this signal temporal lip jitter. Because it follows from the absence of biomechanical constraints in the generation process itself–rather than from any property of the training data–it persists across generators and languages.

Kinematic Feature Extraction

For each 25-frame window (stride 5, covering 1.0 s at 25 fps), we compute four kinematic statistics per landmark from the y-coordinate sequence pˆt i,y. These are displacement σ(0)i = stdt(ˆp t i,y) (displacement), velocity σ(1)i = stdt(∆ˆp t i,y) (velocity), acceleration σ(2)i = stdt(∆2 pˆ t i,y) (acceleration), and jerk σ(3)i = stdt(∆3 pˆ t i,y) (jerk). Stacking these across all 64 landmarks gives a 256-dimensional statistical feature vector xs ∈ R 256. For the temporal branch, we retain the full sequence by building Xt ∈ R T ×256 by concatenating the values per-frame y with their first three finite differences, each padded to length T by repeating the boundary value.

Network Architecture and Feature Fusion

BioLip uses a three-branch network that processes Xt, xr, and xs in parallel before combining them for classification. The total parameter count is 302K. The three branches are:

Temporal Branch: This processes Xt through three 1D convolutional blocks with channel sizes of 256→ 128→ 128→ 64, BatchNorm, and GELU activation. It then applies multi-scale pooling: global average pooling (64-d), local average pooling over four equal temporal segments (256-d), and global max pooling (64-d), concatenated to 384 dimensions.
Region Branch: This processes xr through an MLP 8→32→32(xr) ∈ R 32.
Statistical Branch: This processes xs through an MLP 256→128→64(xs) ∈ R 64.

The three branch outputs are concatenated and passed through a classification head, resulting in a final score of h = [ht∥ hr∥ hs] ∈ R 224.

Performance and Generalization

BioLip substantially outperforms prior pixel-based and audio-visual methods on the unseen generators. On LipSyncTIMIT, BioLip achieves a mean AUC of 0.9267 at CRF=23, essentially matching Xception (0.9290, a 0.0023 gap) while outperforming LipFD by 0.277 points. In the seven-language PolyGlotFake benchmark, BioLip achieves a mean AUC of 0.828 ± 0.013, 14.4 points above XRes—the strongest prior method on this benchmark. This cross-lingual performance is attributed to the fact that displacement, velocity, acceleration, and jerk describe how lips move physically, not what phoneme is being produced.

Robustness and Efficiency

BioLip's feature extraction step runs entirely on the device and only a compact coordinate vector of approximately 1 KB leaves the hardware—no raw video frames, no audio, no face images. The removal of audio has consequences beyond convenience, as audio-based detectors learn phoneme-to-viseme mappings that are language-specific and do not transfer across languages. Computational efficiency is high, with BioLip running at 0.39 ms per window on the CPU and 89× faster than Xception at 1/1892 the noise std. The full multi-order model is designed for exactly this asymmetric attack surface, where whichever derivative order a future generator regularizes first, the remaining orders carry residual signal.

Limitations

Heavy compression (CRF=40) is the most significant practical challenge because blocking artifacts degrade MediaPipe’s landmark localization and partially smooth the jitter signal itself. Furthermore, BioLip measures kinematic variance of MediaPipe landmark coordinates, which mixes genuine lip motion and MediaPipe localization error. The paper notes that temporal localization would be useful for forensic applications.

Conclusion

BioLip is built around the observation that small per-frame errors accumulate into elevated kinematic variance—in velocity, acceleration, and jerk—that real speech does not produce. Because this signal reflects a physical property of lip movement rather than anything specific to a particular generator or language, the signal holds up on generators, languages, and compression levels on which we never trained. BioLip is built around this observation. With 302K parameters and no pixels or audio, it achieves mean AUC 0.9267 at CRF=23 and 0.7585 at CRF=40 on LipSyncTIMIT, leading all baselines at heavy compression despite having 69× fewer parameters than Xception. In seven-language PolyGlotFake it achieves 0.828 ± 0.013, 14.4 points above XRes. The removal of audio has consequences beyond convenience and BioLip does not have any of these problems. There are a few things we would like to extend.

References

[1] K. R. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C.

Improvements for AI systems

Improve deepfake detection robustness against generator shifts by replacing pixel-based features with kinematic statistics; "Authentic lip motion is constrained by tissue mechanics and neuromuscular bandwidth; current generators typically do not impose these constraints, producing trajectories with elevated variance in velocity, acceleration, and jerk that real speech does not exhibit."
Enable cross-lingual deepfake detection without relying on audio or language-dependent phoneme supervision; BioLip’s features describe how lips move physically, a property that does not depend on what word is being said or what language the speaker uses.
Achieve real-time, privacy-preserving deployment by utilizing only landmark coordinates for feature extraction; The complete extraction pipeline can run on-device, supporting privacy-preserving operation.
Enhance performance under heavy compression attacks by leveraging a multi-order kinematic representation; The full multi-order model does not uniformly beat the best single-order variant at each compression level: it scores 0.021 points below acceleration-only at CRF=23 and 0.030 points below velocity-only at CRF=40.
Increase adversarial robustness against future generators by utilizing a joint kinematic feature vector; The multi-order BioLip representation is designed for exactly this asymmetric attack surface: 'whichever derivative order a future generator regularizes first, the remaining orders carry residual signal.'