RoVo: Robust Voice Protection Against Voice Cloning Attacks via Embedding-Level Adversarial Perturbations

In 2024–2025, I worked on protecting recordings before they could be used for unauthorized voice cloning. This became RoVo (Robust Voice). We released an initial arXiv version in May 2025, then expanded the datasets and attack conditions. The extended manuscript is under review at IEEE Access.

RoVo perturbs the latent representation of a neural audio codec and decodes it back into speech. The idea is to make the protection part of the reconstructed audio structure, so that an attacker cannot easily remove it with speech enhancement or purification.

01 · Research question

Protecting a recording before it is cloned

Zero-shot text-to-speech and voice-conversion systems can imitate a speaker from a short reference recording. Videos, interviews, and podcasts provide reference audio without requiring access to a private account or device. That audio can be used for impersonation, voice phishing, false statements, or attempts to bypass speaker authentication.

RoVo applies protection before the recording is published. The processed audio should remain intelligible to listeners while making it difficult for a cloning model to recover the original speaker's identity.

Earlier proactive defenses also used adversarial perturbations, usually added to the waveform or spectrogram. Those perturbations could work when the attacker used the recording directly, but weaken after speech enhancement. We therefore evaluated what happens after an attacker tries to remove the protection, changes models, or uses knowledge of the defense.

02 · Threat model

Three levels of removal attacks

We assumed the attacker could collect a limited amount of the victim's published speech and use it as reference audio for zero-shot TTS or voice conversion. We judged the cloned output with automatic speaker verification (ASV). If the verifier accepted the clone as the original speaker, the defense failed; if it rejected the clone, the defense succeeded.

We grouped removal attacks by the information and models available to the attacker. We also evaluated transfer to cloning models that were not used to generate the protection.

Even in the strong setting, the attacker only had the protected recording. If clean speech were also available, the attacker could bypass the protection by using it directly.

The defender was assumed to run RoVo locally before publication. Since the defender cannot control later processing, we tested several attack conditions rather than a single fixed cloning pipeline.

03 · Why latent space

Perturbing the codec representation

A signal-domain defense adds a perturbation δ_signal directly to the original waveform x, producing x + δ_signal. An enhancement model trained to separate speech from noise can treat that added signal as something to remove.

RoVo moves the perturbation into the codec's latent space. Encoder E maps the input to z, a trainable perturbation δ_latent is added, and decoder D reconstructs the protected waveform.

z = E(x), x_rovo = D(z + δ_latent)

The nonlinear decoder distributes the perturbation through the reconstructed speech's time-frequency structure. Because the change is tied to that structure, separating only the protective component becomes harder.

This makes audio quality after enhancement an incomplete measure of protection. For a signal-domain defense, improved quality may mean that the protective noise has been removed. In some RoVo samples, enhancement could not separate the modification from the speech and introduced further damage instead.

04 · RoVo framework

Reconstructing protected speech with Bark and EnCodec

RoVo uses Bark's neural-codec-based generation architecture. The Neural Codec Encoder compresses the input, and the Coarse and Fine Transformers process its longer structure and acoustic detail. We optimize the perturbation in this embedding representation, then reconstruct speech through the Neural Codec Decoder.

Before adding perturbations, we checked the backbone's reconstruction quality. Speech reconstructed by Bark alone had a MOS of 4.53 and cosine similarity of 0.96 to the original. This baseline helped distinguish the cost of the perturbation from errors introduced by the codec itself.

05 · Optimization

Alternating between protection and quality with PerC-AL

A stronger perturbation can move speaker identity further, but also reduce naturalness and intelligibility. Optimizing quality alone leaves the speaker encoder able to recover the original identity. RoVo uses Perceptual Alternating Loss (PerC-AL) to alternate between these objectives according to a threshold.

Target-based Loss moves the protected speaker embedding away from the original speaker and toward a chosen target. SNR Loss limits the difference between the reconstructed waveform and the original. Optimization first works toward the identity-disruption objective, then switches to quality preservation when the condition is met, repeating this process.

We selected target speakers of the opposite gender. In t-SNE projections from two speaker encoders, male and female speakers tended to form separate clusters. An opposite-gender target was therefore more likely to lie farther from the original speaker than a same-gender target, encouraging a larger embedding shift.

06 · Dataset

Selecting the threshold on VCTK, then testing other corpora

PerC-AL needs a threshold for switching between Target-based Loss and SNR Loss. Retuning it on every corpus would make it difficult to judge generalization. We selected it only on VCTK and reused it on five other English speech corpora.

For the main evaluation, we selected 10–20 speakers per corpus and ten utterances of 5–10 seconds per speaker: 70 speakers and 700 original utterances overall. We protected each recording, used it to synthesize speech with three white-box cloning models, and evaluated speaker acceptance with several verification backends.

All six corpora were in English, but differed in accent, recording conditions, and speaking style. Keeping VCTK for threshold selection and the other five for evaluation tested whether the same setting worked beyond the data used to choose it.

07 · Evaluation protocol

Separating the cloning and verification models

We treated the speaker encoder used to generate protection, the cloning model used to synthesize speech, and the verifier used to judge the result as distinct parts of the evaluation. This let us test whether protection persisted across model combinations.

Defense Success Rate (DSR) measures the fraction of clones made from protected speech that are rejected as the original speaker. Higher DSR means stronger protection. MOS assesses perceptual quality on a 1–5 scale, while word error rate (WER) measures preservation of spoken content.

Supplementary experiments used synthesis prompts involving requests for money transfers, emergencies, account verification, and suspicious transactions. These prompts brought the evaluation closer to impersonation scenarios than neutral read speech alone.

08 · Base and weak attacks

Higher DSR without retuning

On the VCTK threshold-selection set, mean DSR increased from 9.6% for unprotected (RAW) speech to 82.5% with RoVo. On the main evaluation corpora, the unchanged threshold increased mean DSR from 13.1% to 88.0%.

Mean WER rose from 5.3% on clean speech to 16.4% on protected speech. The recordings remained intelligible, but the protection had a measurable cost.

For weak attacks, we applied quantization, resampling, filtering, and mel inversion separately. The average DSR drop across all combinations was 2.2 percentage points. AVC stayed above 88.5% across all transformations and verifiers. The lowest result was 73.0% for YourTTS–ResNet after mel inversion. In some conditions, DSR increased by up to 8.8 percentage points after transformation.

09 · Moderate adaptive attacks

Retaining protection after learned enhancement

The moderate attacks used learned enhancement methods: Spectral Masking, DeepFilterNet, and the DNS and VoiceBank versions of MP-SENet. We applied the same conditions to AttackVC, AntiFake, and VoiceGuard.

RoVo retained a mean DSR of 82.8% across 36 enhancement–cloning–verification combinations, a drop of 5.2 percentage points. AttackVC, AntiFake, and VoiceGuard dropped by 30.8, 35.5, and 49.7 points, respectively. Some signal-domain baselines started with high DSR, but lost more of it after enhancement.

On AVC, RoVo retained a mean DSR of 93.8% after enhancement. Some conditions improved by up to 4.4 percentage points. That increase does not necessarily mean the protected audio became better: if enhancement damages the speech while failing to remove the perturbation, verification can become harder for that reason too.

RoVo's mean MOS was 2.53 before attack and 2.68 after enhancement. Signal-domain baselines tended to recover more quality when their perturbations were removed. RoVo's smaller recovery reflects a remaining trade-off: harder-to-remove protection came with greater perceptual cost.

10 · Transfer and strong attacks

Unseen cloning models and informed reconstruction

For black-box evaluation, we generated protected audio using an ensemble of the speaker encoders from SV2TTS, YourTTS, and AVC. We then tested it on Tortoise and CosyVoice, which were excluded from defense generation. The ensemble sought perturbation directions shared by several surrogate encoders to improve transfer.

Mean DSR across the black-box attack conditions was 81.6%: 89.0% after weak transformations and 77.3% after enhancement or purification. Ensemble-protected audio had a MOS of 2.83 ± 0.46. Protection therefore persisted to a degree on architectures outside the white-box set.

For the strong attack, the attacker knew RoVo's backbone, optimization objective, and target reference. The attacker reconstructed a more natural signal from the protected recording to try to reverse the perturbation. MOS recovered to 3.38–3.76, depending on the model, but the reconstructed speech was accepted as the original speaker only 15.3% of the time on average. Cloning from those reconstructions produced DSR of 73.2–90.7%, with a mean of 82.1%.

The supplementary evaluation added LLaSE-G1 reconstruction. This attack extracts speaker-related embeddings with WavLM, converts them into speech tokens, and reconstructs audio with a LLaMA-based language model and X-Codec decoder. Mean DSR across all combinations was 72.9%. With ECAPA-TDNN verification, DSR was 82.2% for SV2TTS, 82.7% for YourTTS, 79.9% for AVC, and 85.4% for black-box Tortoise, averaging 82.6%.

11 · Commercial verification

Checking the result with a commercial verifier

We added Microsoft Azure's speaker-verification API to test whether the results depended on the open-source verifiers and their thresholds. Azure already rejected some clones from RAW audio at relatively high rates. After RoVo, DSR exceeded 98% for all three cloning models.

After enhancement, AntiFake had a mean Azure DSR of 94.1%, down 5.6 percentage points. RoVo retained 99.0%, with a mean change of just 0.3 points. Its DSR ranged from 98.2–99.9% across the enhancement–cloning combinations.

Azure and the open-source verifiers have different decision boundaries, so their absolute scores are not directly interchangeable. The useful observation is that RoVo's protection and resistance to enhancement also appeared with an external commercial backend.

12 · Trade-off and limitations

What listeners heard, and what protection cost

We also ran an IRB-approved listening study. We recruited 100 native English speakers through Prolific, and each listened to 30 audio pairs. They rated whether the voices sounded like the same speaker on a five-point scale from Very Similar to Very Different.

The conditions paired two real recordings from the same speaker (RA_RA), recordings from different speakers (RA_RB), original and protected speech (RA_DA), and original speech with a clone made from protected audio (RA_FDA). RA_DA-SE and RA_FDA-SE added enhancement to the protected and cloned conditions.

The control pairs behaved as expected: RA_RA was mostly rated similar, and RA_RB mostly different. Protected speech in RA_DA sounded closer to the original speaker than the cloned conditions did. Most ratings for RA_FDA and RA_FDA-SE indicated a different speaker.

These results are consistent with protection that preserves some of the original voice for human listeners while disrupting its reproduction by a cloning model. The method does not fully anonymize the speaker. It also has a usability cost: enhanced protected speech, RA_DA-SE, sounded less like the original than RA_DA did.

13 · Manuscript and preprint

The extended manuscript and the public preprint

The May 2025 arXiv v1 reports the initial method and experiments from this 2024–2025 study. We subsequently extended the work with six corpora, graded adaptive attacks, PerC-AL, black-box transfer, a commercial verification API, and additional reconstruction attacks.

The extended manuscript is under review at IEEE Access. The arXiv link below points to the public 2025 v1, which does not contain all the expanded experiments described here.