MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection
MUTE extends our work on proactive voice protection to talking-head generation: systems that animate a portrait using an audio recording. A stolen voice can be used to produce a convincing video even when the attacker never trains a dedicated voice-cloning model. Our question was whether protecting the audio before release could weaken the audio-visual correspondence these systems need.
The paper was accepted at NeurIPS 2026. See the official NeurIPS poster page and the public paper at arXiv:2611.11989. This research story follows the public paper's experimental settings and results. The project page includes synchronized video comparisons and an interactive version of the user-study task.
01 · Motivation
Protecting the connection between a voice and a moving face
A talking-head model uses a portrait and speech to generate facial motion. Audio supplies timing and phonetic cues for the lips, as well as information that influences expression and head movement. The quality of the resulting video depends on a continuing correspondence between the audio and the visual sequence.
Earlier proactive defenses largely focused on a single modality. Image protection interferes with visual features; voice protection often targets the speaker representation used by a cloning system. Talking-head misuse introduces a different target: audio-visual alignment itself. Weakening identity extraction alone does not necessarily prevent a model from using speech to drive plausible lip motion.
MUTE modifies only the audio before it is released. The objective is to make generated lip motion inconsistent with the speech while retaining intelligibility and perceptual audio quality. The generated video can still exist; the measured outcome is degraded synchronization, rather than a guarantee that all forms of deepfake misuse become impossible.
The question behind MUTE
Can an audio-only perturbation disrupt the alignment mechanisms shared by different talking-head models, and remain effective after an attacker filters, denoises, or otherwise processes the recording?
02 · Threat model
Generating protection with one surrogate and testing unseen models
Let the original recording be x and the protection signal be δ. The released audio is x_adv = x + δ. In the main experiments, the waveform is in [-1, 1] and the perturbation is bounded by ‖δ‖∞ ≤ 0.004. MUTE leaves the portrait unchanged.
The defender has access to the surrogate model Hallo and its gradients. The protected audio is then evaluated with Hallo2, Audio2Head, SadTalker, and Ditto, without using those target models to generate the perturbation. This is a transfer-based black-box setting: the defender knows the surrogate, not the attacker's eventual model.
We also consider an attacker who tries to remove the signal through resampling, spectral filtering, Demucs, WavePurifier, or optimization-based purification. A separate evaluation examines TTS resynthesis, where an attacker uses the protected audio as a voice reference to generate a new waveform.
03 · Multi-level design
Disrupting alignment first, then degrading temporal representations
The audio encoder converts speech into temporal embeddings. Cross-attention then connects these audio features to visual latents during generation. MUTE acts at both points, with separate objectives for the alignment structure and the representations that feed it.
The optimization runs for 200 PGD steps. During the first 10%, the alignment objective redirects cross-attention. During the remaining 90%, the representation objective reduces the similarity between clean and protected audio embeddings. Both phases include the STFT robustness loss with weight λ = 5. The step size is α = 0.0002.
Separating the objectives helps manage conflicting gradient directions. It also reduces the cost of repeatedly optimizing attention across the full run. The phase ratio is an empirical balance between synchronization disruption on the surrogate, transfer to other architectures, and computation time.
04 · Alignment-level disruption
Redirecting cross-attention toward sensitive spatial locations
In the denoising UNet, visual latents provide queries and audio embeddings provide keys and values. We treat the resulting audio-conditioned cross-attention as a proxy for the alignment structure. Instead of imposing a uniform change everywhere, MUTE emphasizes locations where a bounded perturbation can produce a larger change.
A Taylor-based sensitivity score combines first- and second-order terms, normalized by the original attention value. A softmax converts these scores into a target distribution. The Target Attention KL Loss then encourages the adversarial attention to follow that distribution:
L_attn = KL(A_target ‖ A_adv)
The purpose is to move attention away from its original cross-modal correspondence. The appendix compares the proposed score with uniform weighting, inverse-attention weighting, and a first-order-only alternative. This component is paired with temporal representation degradation rather than used as the sole protection mechanism.
05 · Representation-level degradation
Preserving temporal detail in the optimization objective
Hallo extracts audio features with wav2vec 2.0. These representations vary across time, transformer layers, and hidden dimensions. Since facial motion is generated frame by frame, comparing only a time-averaged embedding can hide local changes that cancel during aggregation.
MUTE compares the protected and clean embeddings at every time step. The Frame-wise Embedding Cosine Loss minimizes their average cosine similarity:
L_emb = (1 / T) Σ_t cos(z_adv,t, z_t)
The temporal representations are aligned to a 25 fps video rate, giving 25 frame-wise embeddings per second. This makes local disagreement contribute throughout the sequence. In the paper's ablation, the frame-wise objective produces stronger alignment degradation than a global embedding objective.
Why use both levels?
- Alignment loss changes the cross-modal mechanism that maps audio features to visual motion.
- Representation loss degrades the temporal audio information used by that mechanism.
- Two-phase optimization separates their updates while retaining the same robustness constraint.
06 · Robustness by design
Keeping protection close to the speech signal's spectral structure
Low-energy noise is often suppressed by filtering or enhancement. MUTE uses the clean speech spectrogram to construct a mask that favors high-energy time-frequency regions. The implementation focuses on the 80–7,600 Hz speech band and uses a magnitude-based mask to reduce the contribution of low-energy and out-of-band components.
The STFT loss encourages the waveform perturbation to remain close to its masked spectral reconstruction:
L_stft = ‖δ − ISTFT(M ⊙ STFT(δ))‖²
This makes the perturbation overlap more closely with speech structure, reducing its separability as independent noise. Both optimization phases use this loss. It improves the tested resistance to purification, although it does not imply that every future removal method will fail.
TTS resynthesis is a distinct bypass route because it produces a new waveform. For that setting, an optional speaker objective reduces cosine similarity between clean and protected speaker embeddings using an ensemble of CAM++ and a ResNet-based speaker encoder. Its weight controls a trade-off between speaker-identity protection and lip-sync disruption.
07 · Evaluation protocol
Comparing six baselines under controlled audio and video settings
We use 50 TalkingHead-1KH samples following the Silencer preprocessing protocol and 50 CelebV-HQ samples selected with CLIP-IQA. CelebV-HQ audio is trimmed to five seconds. During perturbation generation, the reference image is drawn from a different identity; evaluation uses image-audio pairs of the same identity.
Portraits are resized to 512 × 512 and audio is sampled at 16 kHz. The six baselines are C&W, Psychoacoustic, Qin et al., ARO, SAGO, and CAF. Their perturbation budgets are chosen for comparable perceptual quality: ε = 0.01 for SAGO, 0.006 for Qin et al., and 0.004 for MUTE and the other methods. The main experiments use three independent random seeds on a single NVIDIA RTX 4090.
| Evaluation | Metrics | What they measure |
|---|---|---|
| Audio quality | WER ↓, STOI ↑, PESQ ↑ | Speech intelligibility and perceptual quality |
| Audio-visual alignment | Sync ↓, M-LMD ↑ | Synchronization and mouth-landmark deviation |
| Visual degradation | FID ↑, FVD ↑, E-FID ↑ | Changes in image, video, and expression distributions |
| Human perception | Most mismatched choice | Which generated clip has the most noticeable lip-sync error |
The arrows describe the desired direction for protection, except for the audio-quality measures, where the aim is to preserve usable speech. Tables below show the reported means; the paper includes standard deviations for the main comparisons.
08 · Main results
Stronger synchronization disruption with usable speech
On TalkingHead-1KH, MUTE obtains a Sync score of 1.332 and M-LMD of 1.627. On CelebV-HQ, the corresponding values are 2.152 and 1.693. These are the lowest Sync and highest M-LMD values among the evaluated protection methods on both datasets.
| Method | TalkingHead-1KH | CelebV-HQ | ||
|---|---|---|---|---|
| Sync ↓ | M-LMD ↑ | Sync ↓ | M-LMD ↑ | |
| C&W | 3.740 | 1.096 | 4.039 | 1.108 |
| Psychoacoustic | 3.787 | 1.097 | 3.926 | 1.126 |
| Qin et al. | 4.468 | 0.983 | 4.543 | 1.054 |
| ARO | 3.518 | 1.155 | 3.818 | 1.176 |
| SAGO | 3.264 | 1.200 | 3.597 | 1.226 |
| CAF | 2.173 | 1.341 | 2.570 | 1.334 |
| MUTE | 1.332 | 1.627 | 2.152 | 1.693 |
The audio-quality results are WER 11.681%, STOI 0.936, and PESQ 2.522 on TalkingHead-1KH, and 10.467%, 0.959, and 2.940 on CelebV-HQ. MUTE has the best values among the compared protection methods in this experiment. These are relative comparisons: the protected recordings still differ from clean audio.
09 · Black-box transfer
Protection transfers from Hallo to four unseen models
The perturbation is generated on Hallo once, then used with Hallo2, Audio2Head, SadTalker, and Ditto. Hallo2 is architecturally related to the surrogate, while the other targets use substantially different designs. MUTE has the lowest Sync and highest M-LMD among the tested methods on all four.
| Target model | MUTE Sync ↓ | Best baseline Sync ↓ | MUTE M-LMD ↑ | Best baseline M-LMD ↑ |
|---|---|---|---|---|
| Hallo2 | 1.397 | 2.280 (CAF) | 1.679 | 1.373 (CAF) |
| Audio2Head | 3.627 | 4.005 (CAF) | 1.299 | 1.246 (CAF) |
| SadTalker | 4.703 | 5.297 (ARO) | 0.864 | 0.658 (ARO) |
| Ditto | 3.457 | 3.803 (ARO) | 0.884 | 0.781 (ARO) |
10 · Removal attacks
Evaluating filtering, denoising, and adaptive purification
Post-processing includes a 16 → 8 → 16 kHz downsampling-upsampling transformation and adaptive spectral filtering. Demucs and WavePurifier represent learned enhancement and diffusion-based purification. We also evaluate BPDA and EOT settings that account for non-differentiable or stochastic processing when constructing adaptive attacks.
| Transformation | MUTE Sync ↓ | MUTE M-LMD ↑ | Best baseline Sync ↓ |
|---|---|---|---|
| Down/upsampling | 2.914 | 1.254 | 3.533 (CAF) |
| Spectral filtering | 2.251 | 1.410 | 3.063 (CAF) |
| Demucs | 1.534 | 1.593 | 2.566 (CAF) |
| WavePurifier | 3.494 | 1.217 | 3.940 (CAF) |
| BPDA | 4.269 | 0.655 | 4.778 (ARO) |
| EOT | 4.298 | 0.647 | 4.794 (CAF) |
MUTE retains the strongest alignment degradation among the methods in each tested condition. However, the higher Sync scores under BPDA and EOT show that adaptive attacks reduce the protection. The appendix additionally tests purification followed by transfer to Audio2Head and SadTalker, rather than evaluating those two difficulties only in isolation.
11 · TTS and multimodal protection
An optional speaker objective and complementary image protection
TTS resynthesis can replace the waveform entirely. We use protected audio as the speaker reference for CosyVoice and XTTS-v2 in white-box settings and Chatterbox in a black-box setting. An independent ECAPA-TDNN verifier measures speaker similarity. Defense success rate (DSR) is the share of generated samples with cosine similarity at or below 0.25.
With λ_spk = 0.01, DSR reaches 67.73% for CosyVoice, 99.33% for XTTS-v2, and 43.87% for Chatterbox. Lip-sync protection remains strong in the accompanying evaluation, with Sync 1.496 and M-LMD 1.541. Increasing the speaker-loss weight further improves DSR but weakens the primary alignment objective.
These speaker-verification measurements do not directly establish human-perceived speaker similarity or prove that lip-sync disruption survives every TTS-resynthesized waveform. They assess the identity-preservation part of this bypass route.
MUTE can also be combined with image protection because it modifies only the audio. In Table 2, adding MUTE lowers Sync from 3.039 to 0.902 with MIS and from 2.170 to 0.760 with Silencer. The image and audio defenses are combined, not jointly optimized; a unified multimodal objective remains future work.
12 · Human evaluation and ablations
Testing visible mismatch and the quality-protection trade-off
The user study compares MUTE, CAF, ARO, and SAGO across 10 samples. 100 participants each choose the clip with the most noticeable lip-sync mismatch, producing 1,000 responses. MUTE is selected in 78.1% ± 2.56% of responses, with a 95% confidence interval. This measures perceived mismatch among the presented alternatives, not a real-world deepfake-prevention rate.
The ablations examine the attention-phase ratio ρ, STFT weight λ, perturbation budget ε, step size, and iteration count. The selected ρ = 0.1 balances cost and transferability, while λ = 5 gives a useful balance between clean-setting disruption and robustness to purification.
The perturbation budget exposes the audio-quality trade-off particularly clearly. At ε = 0.002, the TalkingHead-1KH Sync score is 1.898 with STOI 0.948 and PESQ 2.949. At the chosen ε = 0.004, these become 1.332, 0.936, and 2.522. Increasing the budget to 0.010 lowers Sync to 1.027 but also reduces STOI to 0.885 and PESQ to 1.945. The strongest disruption is therefore not automatically the most useful operating point.
13 · Takeaways and resources
Audio-visual alignment is a useful target for proactive protection
MUTE shows that a recording can be protected against talking-head misuse by targeting both cross-modal attention and temporal audio representations. Its transfer results and purification evaluations support this approach across the tested architectures and attacks. The optional speaker objective connects the work to voice-cloning defenses, while combinations with image protection suggest a path toward broader multimodal protection.
The scope still matters. The study uses limited samples, stronger future adaptive attacks may weaken the defense, and modifying a recording before release cannot protect copies that have already leaked. Low-latency deployment and joint audio-image optimization also remain open problems.
The work was carried out with Donghyun Kim, Jin Hong, Dain Kim, Junseok Kwon, and Daeseon Choi. Donghyun Kim, Jin Hong, and Seungmin Kim contributed equally and are listed in alphabetical order.
NeurIPS 2026 · Accepted · Public preprint
MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection
NeurIPS posterPaper (arXiv)Project & demosCodeMedia coverage