Voice Synthesis Detection Using Language Model-Based Speech Feature Extraction

This study began in 2023 and led to a paper in May 2024. As text-to-speech and voice-conversion systems improved, we wanted to look beyond local waveform and spectral cues for evidence of synthesis. We converted speech into discrete audio tokens and used a BERT model pretrained on text to extract features from their sequence.

The question was whether the codes produced by an audio codec retain temporal relationships that distinguish real speech from synthetic speech. We used Encodec as the tokenizer and BERT-Base as the sequence feature extractor.

01 · Research question

Looking for traces of synthesis in audio-token sequences

Many synthetic-speech detectors use features taken directly from the signal: spectra, phase, filter banks, or raw waveforms. Other approaches use pretrained speech models such as HuBERT. Both can work well, though their performance may depend on cues specific to a generator or recording condition.

We represented speech as a sequence of codebook indices. Encodec produced the codes during compression, and BERT modeled their context. Our hypothesis was that relationships between codes could expose longer patterns in synthetic speech that individual code values would miss.

02 · Feature extractor

Using Encodec as an audio tokenizer for BERT

Encodec consists of an encoder, residual vector quantization (RVQ), and a decoder. The encoder maps the waveform to a continuous latent representation, which RVQ quantizes into integer indices from several codebooks. We passed these codes to BERT instead of decoding them back into speech.

Based on preliminary experiments, we used the output of the second quantization layer. The codes from each recording were arranged in a [2, 512] tensor: the first axis described the selected code configuration, and the second was the sequence length presented to BERT. These integer codes were converted to input embeddings for feature extraction.

BERT-Base had 12 Transformer encoder layers, a 768-dimensional hidden state, and 12 self-attention heads. It learned relationships between audio codes in temporal order, using the same context-modeling mechanism it applies to tokens in text.

03 · Dataset

Balancing ASVspoof 2019 and 2021 at 1:1

We evaluated on ASVspoof 2019 and ASVspoof 2021. Logical Access (LA) covers attacks based on text-to-speech and voice conversion; DeepFake (DF) focuses more directly on synthetic-speech detection. We used 2019 LA for the initial comparison and classifier selection, then tested on 2021 LA and DF under different attack conditions.

The original challenge data contained unequal numbers of bona fide and spoof samples, with one class roughly eight times larger than the other in some conditions. Overall accuracy can be misleading under this imbalance. We therefore constructed training, validation, and test splits with a 1:1 ratio of real to synthetic speech.

04 · Experiment design

Testing the features with four classifiers

We wanted to assess the Encodec-BERT representation across different classifiers. We fed the same features to an FC-layer, a Transformer, ResNet-18, and ResNet2.

• FC-layer tested whether a simple classifier could separate the two classes from the extracted features.
• Transformer added a second stage of sequence modeling.
• ResNet-18 learned patterns in the [512, 768] feature tensor through a residual network.
• ResNet2 adjusted the input and output dimensions of ResNet-18's layers for the speech features.

In stage 1, we trained and evaluated all four on ASVspoof 2019 LA and compared them with ASSD, TSSDNet, and CCT. ASSD was a useful reference because it also combined an audio codec with a Transformer. In stage 2, we took the three stronger classifiers from 2019 LA—FC-layer, ResNet-18, and ResNet2—and evaluated them on 2021 LA and DF.

05 · Metrics

Per-class accuracy and equal error rate

In the paper, Class 0 denotes real speech and Class 1 denotes synthetic speech. Per-class accuracy shows which class accounts for more errors. Balanced accuracy is the mean of the two class accuracies; overall accuracy is the fraction of all samples classified correctly.

EER is the error rate at the threshold where real speech is rejected as synthetic as often as synthetic speech is accepted as real. A lower EER indicates better separation between the classes.

06 · Results

An EER of 11.79% on 2021 DF

Stage 1: ASVspoof 2019 LA

ResNet2 gave the best results among these four configurations, with 88.08% accuracy and 15.98% EER. The FC-layer was close at 87.88% accuracy and 17.09% EER. Its performance suggested that the extracted features contained useful information even for a simple classifier.

The Transformer classifier reached only 70.43% accuracy on Class 1. Adding another sequence model after BERT did not improve detection in this setting.

Stage 2: ASVspoof 2021 LA and DF

The FC-layer performed best on 2021 LA: 84.52% Class 0 accuracy, 89.42% balanced accuracy, 88.67% overall accuracy, and 10.91% EER. On DF, ResNet2 performed best, with 91.28% Class 1 accuracy, 87.74% balanced accuracy, 87.02% overall accuracy, and 11.79% EER.

The 11.79% EER on DF was below the 15.64% reported for the compared systems using Mel-spectrogram, SincNet, or raw-audio features with CNN or ResNet classifiers. The Fbank-based reference model had an EER of 16.05%, and the CQT-based model had 18.30%. Encodec-BERT produced fewer errors under these comparison conditions.

07 · Limits

Balanced counts did not eliminate differences between classes

Class 0 and Class 1 accuracy still differed after we balanced their counts at 1:1. The splits also differed in synthesis methods and attack difficulty. Classifiers that performed well on 2019 LA did not retain the same ranking on 2021 LA and DF.

These experiments tested the representation within the attack conditions covered by ASVspoof. They do not establish generalization to new text-to-speech or voice-conversion models, other codecs, compression and retransmission, or real telephone channels. Those conditions require separate evaluation.

08 · Conclusion

What we learned from audio tokens

We used Encodec RVQ codes as BERT input tokens to extract features for synthetic-speech detection. After comparing four classifiers on 2019 LA, we evaluated the stronger three on 2021 LA and DF. The best DF result was 11.79% EER, compared with 15.64% for the reference systems discussed above.

The results support using a text-pretrained model to learn discriminative relationships in discrete audio sequences. They also show how much the final classifier and evaluation data affect the outcome. Testing on new generators and real transmission conditions remains necessary before drawing broader conclusions.

09 · Publication

Paper