NaVo: Natural Voice Protection against Voice Cloning Attacks via Generative Universal Adversarial Audio

Abstract

P roactive defense against voice cloning disrupts synthesis by injecting adversarial perturbations. However, prior methods rely on iterative optimization, often causing perceptual distortion and high latency that limit real-time deployment. We propose NaVo, a generative framework that produces natural-sounding universal adversarial audio. NaVo guides a latent diffusion–based text-to-audio model with fine-tuned low-rank adaptation modules tailored to gender and acoustic categories such as rain, music, and babble, enabling high-fidelity, speaker-independent adversarial audio generation. Unlike optimization-based approaches, NaVo supports real-time protection through simple audio mixing without gradient-based inference. It achieves a 76% defense success rate against a widely used commercial voice cloning system in a black-box setting. By jointly addressing universality, audio quality, and computational efficiency, NaVo offers a practical solution for proactive voice protection.

Model Overview

Model Overview Image


Overview of the NaVo framework. (Left) During training, the base text-to-audio (T2A) model is fine-tuned via LoRA using the proposed distributional target loss. (Right) During inference, an appropriate category-specific LoRA module is applied to generate UAA, which is then simply mixed with the original speech to construct the final protected audio.

White-box

RTVC Samples

Acoustic category Defense voice Text-to Speech voice
babble
office
rain

CosyVoice Samples

Acoustic category Defense voice Text-to Speech voice
babble
office
rain

Black-box

Acoustic category Defense voice Tortoise Samples
Text-to Speech voice
Elevenlabs Samples
Text-to Speech voice
babble
music
rain