RoCo: Robust Code for Fast and Effective Proactive Defense against Voice Cloning Attack

RoCo grew out of a practical limitation of RoVo: generating protected speech took too long. In this 2025–2026 study, we replaced continuous embedding perturbations with discrete perturbation codes. The aim was to reduce processing time while keeping the resistance to speech enhancement that motivated our earlier work.

The paper appeared in the ICASSP 2026 proceedings, and I gave an oral presentation in Barcelona. We combined a dedicated perturbation code, a Straight-Through Estimator (STE), and two-stage loss optimization. The evaluation covered six speech corpora, three cloning models, and three speaker verifiers, with measurements of protection, audio quality, resistance to post-processing, and generation time.

01 · From RoVo to RoCo

Moving from continuous embeddings to discrete codes

RoVo adds perturbations to the continuous embeddings of a neural audio codec. After passing through the nonlinear decoder, the perturbation becomes part of the reconstructed speech structure, making it harder for an enhancement model to separate it as ordinary noise. But optimizing a continuous perturbation for every utterance still carries a substantial cost.

RoCo puts the protection signal in a separate discrete codebook channel. It keeps the original acoustic tokens, adds a perturbation code that interferes with speaker-identity extraction, and decodes them together. The design retains RoVo's use of the codec representation while narrowing what has to be optimized.

02 · Research problem

Protection has to survive post-processing and be affordable to generate

Proactive defenses perturb speech before it is released, so that a cloning model's speaker encoder has difficulty recovering the original identity. Small waveform perturbations, however, can resemble background noise to an enhancement or denoising model. An attacker can clean the reference audio before cloning it, reducing the protection.

Processing time is another obstacle. Signal-domain methods repeatedly update many waveform samples. If protecting a 5–10-second utterance takes tens of seconds or nearly two minutes, it is difficult to use the method for video, broadcasting, or a continuous conversation.

03 · Perturbation code

A dedicated channel alongside the acoustic tokens

Neural codec models represent speech with acoustic tokens from several codebooks. A Coarse Transformer predicts the longer structure in the earlier codebooks autoregressively, and a Fine Transformer fills in acoustic details in the later ones. Let a(q,t) denote the original token at codebook q and time frame t. RoCo adds a perturbation code p(t) of the same temporal length.

ã = [a; p]

The concatenation [a; p] is along the codebook axis, not time. The original tokens continue to carry phonetic content, prosody, and timbre. The added p is optimized to interfere with speaker-identity extraction. This confines the updates to a dedicated channel instead of changing all the tokens that represent the utterance.

04 · Straight-Through Estimator

Passing gradients through discrete code selection

A perturbation code is an integer codebook index. Selecting a codebook vector through a one-hot index is not differentiable, so the Target Loss gradient cannot pass directly through that selection. We used a Straight-Through Estimator to supply a gradient path.

z_STE = q + (e − stopgrad(e))

Here, q is the discrete codebook vector selected in the forward pass, and e is a continuous representation from the embedding matrix. During the forward pass, e − stopgrad(e) evaluates to zero, leaving the quantized vector q. During backpropagation, stopgrad(e) blocks one path while the gradient through e remains. This allows gradient-based updates while the forward computation still uses a discrete code.

05 · Two-stage optimization

Establishing protection before recovering audio quality

Target Loss and SNR Loss pull in different directions. Target Loss moves the protected speaker embedding toward a chosen target speaker, making the original identity harder to extract. SNR Loss reduces the difference between the original and protected waveforms. In our experiments, a strong SNR term could suppress the perturbation before it provided enough protection.

We separated the objectives into stages. While the embedding-space magnitude of the perturbation code, ||P||₂, is below a threshold τ, the model optimizes Target Loss alone. At the threshold, it stops those updates and switches to SNR Loss.

L = L_Target if ||P||₂ < τ, otherwise L_SNR

The intention is to establish enough identity disruption first, then recover quality without largely undoing it. We selected the switching threshold on VCTK and applied it to the other five corpora without retuning.

06 · Dataset and protocol

1,200 utterances from 120 speakers across six corpora

We used VCTK, FST, MCV, CSNED, CSUKIED, and LibriSpeech. Across these corpora, we selected 120 speakers and ten utterances per speaker, for 1,200 protected recordings. VCTK supplied the threshold-selection data; the remaining corpora tested the same threshold under different recording and speaker conditions.

Defense Success Rate (DSR) is the fraction of clones made from protected audio that are rejected as the original speaker. Higher is better. We assessed audio quality with NISQA MOS on a 1–5 scale and measured the time needed to protect input recordings lasting 5–10 seconds.

07 · Base defense performance

Protection across three cloning models

Unprotected (RAW) recordings generally produced low DSR because the clones were accepted as the original speakers. RoCo increased DSR for every combination of the three cloning models and three verifiers. On AVC, its mean DSR was 81.5%, compared with 74.8% for AntiFake, 66.9% for VoiceGuard, and 60.1% for AttackVC.

AttackVC and AntiFake had higher averages on SV2TTS, while RoCo reached 84.6%. On YourTTS, RoCo's 75.3% was 1.8 percentage points below AntiFake's leading 77.1%. We assessed these results alongside post-processing resistance and generation time; the highest initial DSR was only one part of the comparison.

08 · Enhancement robustness

Smaller losses after speech enhancement

We applied Spectral Masking, DeepFilterNet, and the DNS and VoiceBank variants of MP-SENet before cloning. Across 36 combinations of enhancement, cloning, and verification models, the paper reported an average DSR drop of roughly 15 percentage points for RoCo, compared with about 38 points for the baselines.

The table below averages the nine cloning–verification combinations for each enhancement method in Table 2 of the paper. Some MP-SENet DNS combinations fell to 44.7%, while other conditions stayed near 80%. Protection persisted, but its strength varied substantially by condition.

We also tested De-AntiFake purification. On AVC, DSR remained at 77.4% with Resemblyzer, 80.3% with ECAPA, and 79.1% with ResNet. Results ranged from 63.1–79.2% on YourTTS and 49.4–70.5% on SV2TTS. The amount of remaining protection depended on the model, but purification did not remove it entirely.

09 · Speed and quality

Protecting a 5–10-second recording in 13–22 seconds

The clearest improvement was generation time. In the SV2TTS setting, RoCo took 20 seconds, compared with 113 seconds for AntiFake and 122 for AttackVC. In the YourTTS setting, the corresponding times were 22, 89, and 40 seconds; for AVC, they were 13, 105, and 59 seconds. This was still slower than real time, but substantially shorter than the 1–2 minutes required by some signal-domain configurations.

We also measured quality with NISQA MOS. RoCo's mean MOS was 2.42 before enhancement, rising to 3.23 with DeepFilterNet and 3.17 with MP-SENet DNS. These gains need to be read alongside DSR: enhancement can improve how the audio sounds while also weakening its protection.

10 · ICASSP presentation

Presenting RoCo in Barcelona

I presented RoCo at IEEE ICASSP 2026, held in Barcelona, Spain, on May 4–8, 2026. The talk introduced the codec-based defense work that led from RoVo to RoCo, then used Figure 1 to explain how perturbation codes, STE, and two-stage optimization changed generation time and resistance to post-processing.

The demo page lets readers hear what the DSR and MOS measurements describe. For AVC, SV2TTS, and YourTTS, it places the original recording, protected speech, cloned output, enhanced speech, and clones made after enhancement side by side.

11 · Publication and resources

Paper and audio samples

The study began in 2025 and was published as a five-page paper in the ICASSP 2026 proceedings, on pages 1671–1675. The IEEE Xplore DOI is 10.1109/ICASSP55912.2026.11462176.