NaVo: Natural Voice Protection against Voice Cloning Attacks via Generative Universal Adversarial Audio

In 2026, we studied whether a trained generator could protect new speakers without optimizing a separate perturbation for each utterance. This work became NaVo, a method for generating Universal Adversarial Audio (UAA), published at Interspeech 2026.

NaVo generates audible background sounds such as rain, babble, and music. We mix them with the original speech at a signal-to-noise ratio (SNR) of 17 dB. The generator is trained to produce plausible ambient audio that also interferes with a cloning model's extraction of speaker identity.

01 · Motivation

The cost of optimizing every recording

A voice-cloning model can extract a speaker's timbre and speaking characteristics from a few seconds of reference audio. Public videos and recordings can supply that reference for speech the person never actually said. Protecting the reference before publication offers a way to interfere with cloning before the impersonation is created.

Existing proactive defenses often optimize perturbations directly in the waveform or frequency domain. This requires repeated gradient calculations for each recording, and the output can sound like high-frequency noise or irregular distortion. The cost accumulates when protecting longer videos, broadcasts, or continuous speech.

NaVo trains a generator to produce protective background audio that can be reused across speakers and utterances. At inference, there is no per-recording gradient optimization: the system generates the audio and mixes it with the recording.

02 · Universal adversarial audio

Making background sound carry the protection

NaVo's output is intended to be heard. It follows a text prompt describing rain, babble, music, or an office environment. The premise is that recognizable ambient sound may fit a recording or social-media clip more naturally than an irregular noise pattern.

Given original speech x, text prompt p, and an AudioLDM2-based generator G, we form the protected speech as:

x_prot = x + G(z | p; Θ + ΔΘ)

Θ denotes the frozen parameters of the pretrained text-to-audio model, and ΔΘ the LoRA parameters trained for protection. We add the generator's output to x rather than repeatedly updating x itself. A trained LoRA module can be reused for speakers and sentences absent from training.

03 · NaVo framework

Adapting AudioLDM2 with LoRA

We used AudioLDM2, a latent-diffusion text-to-audio model, as the backbone. It generates acoustic latents conditioned on a prompt, then reconstructs a waveform through a VAE and vocoder. We added LoRA to the UNet's cross-attention layers to learn protective ambient audio while keeping the rest of the model fixed.

Training used both a text prompt and source speech. The predicted ambient audio was mixed with the speech and passed through a frozen synthesis model's speaker encoder. Diffusion Loss encouraged the output to retain the structure of its acoustic category. Distributional Target Loss moved the mixed speech's embeddings away from the original speaker population.

At inference, we selected the LoRA module for the speaker's gender and the desired acoustic category, generated the background audio, and mixed it with the recording. The speaker encoder and loss calculations were only needed during training.

04 · Modular LoRA

Separate adapters for gender and acoustic category

A single adapter would have to learn several speaker targets and acoustic categories at once. We separated the LoRA modules by target gender and sound category. For male source speakers, we used a module targeting the female speaker distribution; for female source speakers, we used the male distribution. Within each gender setting, separate adapters handled rain, music, babble, and related categories.

LoRA was applied only to the key and value projections of cross-attention in the UNet Transformer blocks:

W′_K = W_K + B_K A_K, W′_V = W_V + B_V A_V

Only the low-rank matrices A and B were updated. The VAE, vocoder, text encoder, and remaining UNet parameters stayed frozen. Restricting the updates to the key and value paths let us adjust how text conditions the audio while retaining the backbone's learned sound-generation behavior.

05 · Distributional target

Targeting a speaker distribution instead of one speaker

Targeted defenses often move a protected embedding toward one chosen speaker. A single target point does not capture the spread of a speaker population, and may be difficult to reuse across many source identities. NaVo instead models the target as a Gaussian distribution.

The speaker encoders' t-SNE projections showed separate regions for male and female speakers. We used this structure to set the mean and variance of female embeddings as the target for male source speech, and vice versa. From the target dataset X_target, we estimated μ_P and σ²_P. We also modeled the protected embeddings in a training batch as Q = N(μ_Q, diag(σ²_Q)).

Matching only the centroid can leave the distribution too narrow or too broad. We used Bhattacharyya distance, corresponding to α = 0.5 in the α-divergence family, to match both the mean and variance of Q and P. The objective was to shift batches of protected embeddings away from their source identities, not to reproduce a particular target person's voice.

06 · Joint objective

Keeping the sound category while disrupting speaker identity

The training objective combines Diffusion Loss and Distributional Target Loss. Diffusion Loss retains AudioLDM2's denoising objective: add noise to the clean latent of a reference ambient recording at a chosen timestep, then train the UNet to predict it. This keeps the adapted model generating recognizable sounds that match the prompt.

For Distributional Target Loss, we mixed the UNet's estimated ambient audio â with source speech s. We scaled the mixing amplitude using the two signals' RMS and kept SNR at 17 dB throughout training. This setting made the background audible while keeping the words intelligible.

The mixture passed through a frozen speaker encoder f(·). Since its parameters did not change, movement in the protected embeddings had to come from the generated audio and the LoRA updates. Gradients flowed back through the encoder and mixing operation to the UNet's LoRA parameters.

L_total = λ_diff L_diff + λ_emb I[t < t_max] D_B(Q || P)

We applied the embedding loss only at low-noise timesteps, t < t_max, where the denoised estimate was more stable. This avoided using highly uncertain audio estimates to guide speaker-target optimization.

07 · Inference-time defense

Generating the background audio and mixing it with speech

At inference, the inputs are the recording, the speaker's gender condition, and the desired ambient category. NaVo attaches the selected LoRA module to AudioLDM2, generates UAA from the prompt, scales it to 17 dB SNR, and adds it to the recording.

An optimization-based defense recalculates target losses and updates the waveform for each new utterance. NaVo performs that learning in advance, so it does not need speaker-encoder gradients for a new recording. The paper's claim of real-time applicability refers to this generation-and-mixing pipeline, without per-utterance gradient optimization. It does not report absolute end-to-end latency on a particular device.

08 · Dataset and protocol

Speech from six corpora, ambient audio from AudioSet

We used VCTK, FST, Mozilla Common Voice (MCV), CSNED, CSUKIED, and LibriSpeech. After combining the corpora, we split them by speaker: 196 for training, 42 for validation, and 42 for testing. With ten utterances per speaker, the splits contained 1,960, 420, and 420 samples. Test speakers were absent from training, allowing us to assess protection on unseen identities.

The data used to estimate target distributions was disjoint from all source splits. For ambient audio, we selected 100 AudioSet recordings per category to help retain category-consistent generation.

Defense Success Rate (DSR) is the fraction of clones whose verification score falls below the threshold for acceptance as the original speaker. Higher DSR means more successful protection. We used CLAP score to measure semantic alignment between the generated audio and its prompt. We did not use a conventional DNN MOS estimator as the main quality measure because it can penalize speech mixed with background audio regardless of how natural that background sounds.

09 · Overall performance

Higher DSR than Enkidu on unseen speakers

On the held-out test set, DSR increased from Enkidu's 25.2% to NaVo's 78.0% with Resemblyzer, from 60.6% to 83.6% with ECAPA-TDNN, and from 45.5% to 79.8% with ResNet. The difference exceeded 20 percentage points for each verifier; with Resemblyzer, NaVo's DSR was more than three times Enkidu's.

These results came from applying the trained LoRA modules directly to 42 unseen test speakers. There was no separate perturbation optimization for those identities. The comparison therefore measures how well the universal modules transfer beyond the training speakers.

10 · White-box evaluation

Strong results for rain and office sounds; lower DSR on CosyVoice

The white-box experiments targeted SV2TTS's GE2E encoder and CosyVoice's CAM++ encoder. Without UAA, CosyVoice DSR was only 0.0–2.9%, depending on the verifier. NaVo raised it to roughly 40–62%, depending on the sound category and verifier. On SV2TTS, several raindrop and office conditions reached 88–98% DSR.

The sound category made a difference. Office and raindrop produced the highest SV2TTS results, while babble was generally lower. CosyVoice, a stronger zero-shot model using supervised semantic tokens, had lower DSR than SV2TTS. Even so, every tested style improved on its near-zero No UAA baseline.

CLAP measured whether the adapted generator retained the acoustic semantics of base AudioLDM2. Scores varied by encoder and gender, but the paper interpreted the overall alignment as comparable to the backbone. This supports prompt–audio consistency; it is not a direct measure of how natural the protected speech sounds to a listener.

11 · Black-box evaluation

Transfer to Tortoise and ElevenLabs

For black-box experiments, we trained NaVo with an ensemble of white-box encoders, including GE2E and CAM++, then cloned the protected speech with Tortoise and ElevenLabs. Neither cloning model participated in optimization. Raindrop and babble retained high DSR across the three verifiers, indicating that the protection was not confined to the training encoders.

Averaged over the 18 gender–style–verifier combinations, ElevenLabs DSR was approximately 76%. Raindrop reached 92.9–97.1% in every combination, while music fell to 32.9–42.9% for male speakers. The commercial-model result therefore depends substantially on the chosen sound category.

On Tortoise, female raindrop and babble stayed at or above 88.9% across all three verifiers. Male raindrop reached 94.4% with ECAPA-TDNN and 89.5% with ResNet. No UAA results were mostly within 0–13.6%, so the increase cannot be explained simply by the cloning models failing on unprotected speech.

12 · Adaptive attacks and limits

Testing attempts to remove the background sound

An attacker who suspects the background audio is protective can apply filtering, quantization, downsampling, or spectral masking before cloning. We built these adaptive conditions using WaveGuard signal processing and DNN-based speech enhancement.

Filtering and spectral masking sometimes increased DSR relative to the unprocessed condition. The paper attributes this to purification damaging speaker cues while failing to separate the UAA cleanly from speech. An increase in DSR after attack should therefore not be read as an improvement in audio quality: retained protection and additional speech damage may both contribute.

13 · Paper and demo

Interspeech paper and listening examples

NaVo uses a distributional target and modular LoRA to generate protective ambient audio for unseen speakers. We evaluated it with SV2TTS and CosyVoice in white-box settings, Tortoise and ElevenLabs in black-box settings, and several attempts to remove the protection.

The official Interspeech 2026 paper is available through ISCA Archive. The project page includes protected and cloned speech samples, and the six-page accepted manuscript used for this write-up is also linked below. The first two authors contributed equally and are listed in the order specified by the paper's equal-contribution note.