Issues with audio embedding using wav2vec

Question

I am having issues with audio embedding using the wav2vec library while trying to classify emotions using audio signals from the EMODB dataset (Emotions dataset in German). I am using the following code to extract embeddings:

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained('facebook/wav2vec2-large-xlsr-53') #XLSR is for SR, not specifically Emotion Rec. 
input_audio, sample_rate = librosa.load(emodb + file,  sr=16000)
extraction = feature_extractor(input_audio, sampling_rate=16000,  return_tensors="np", padding="max_length", max_length=max_len).input_values

The embeddings and shape of the vectors are:

• (1, 143652) for wav2vec features
• (3, 162) for mfcc features

Please note I have padded them to highest value. The length of audio files is around 1 to 2 seconds.

My intended task is emotion detection. I plan to use these embeddings from audio file, along with the text, for a downstream model for emotion classification, and for this I plan to use multimodal approach, using audio and text embeddings.

So, I trained an LSTM model on these embeddings but it was constantly overfitting on the training data (~100% accuracy and ~20% on testing).

Then I decided to use wav2vec embeddings and MFCC embeddings for a simple classification task using SVM. When I use the resulting embeddings in a simple SVM classifier, I am getting random results (15-30% accuracy) for wav2vec embeddings. As a comparison, when I extract features using MFCC and use them in the same classifier, I am getting an accuracy of around 70%.

Naturally, I visualized the embeddings using TSNE to check the quality of input and, I found to be getting strange results. Specifically, when I map 7 emotions, the resulting plot forms a spiral shape. When I only map 2 emotions, the resulting plots are different and also strange. The mappings are circular again when I add more features (3+).

I am unable to understand why I am getting these results and why the embeddings are so poor. I am wondering if this is because I am using a general XLSR model without fine-tuning it for emotion recognition.

I would appreciate any suggestions on how to extract features using wav2vec in a better way, or any papers or implementations that may be helpful.

Answer 1

I am wondering if this is because I am using a general XLSR model without fine-tuning it for emotion recognition.

That might be still true but your approach contains a fundamental error you should eliminate first. You are using the class Wav2Vec2FeatureExtractor to extract the features from an audio file, but this class is not a neural network. It is a preprocessor that pads and normalizes the floating point time series from librosa. As stated by the documentation the normalization makes sure that the array has:

zero mean and unit variance

These features, will therefore only makes sense for the model that was trained with it. When you trained an SVM with it, you actually compared if an SVM can beat wav2vec2 and not if wav2vec2 is better than MFCC!

To get the actual embeddings from the wav2vec2 model, you can use the following code:

import librosa
import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2Model

input_audio, sample_rate = librosa.load("/content/bla.wav",  sr=16000)

model_name = "facebook/wav2vec2-large-xlsr-53"
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name)
model = Wav2Vec2Model.from_pretrained(model_name)

i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate)
with torch.no_grad():
  o= model(i.input_values)
print(o.keys())
print(o.last_hidden_state.shape)
print(o.extract_features.shape)

Output:

odict_keys(['last_hidden_state', 'extract_features'])
torch.Size([1, 1676, 1024])
torch.Size([1, 1676, 512])

Please refer to this StackOverflow post for the difference between last_hidden_state and extract_features. As you can see, the features are multi-dimensional for my file ([bacth_size, seq_len, hidden_size]), which means you probably want to apply some pooling (e.g. mean).

P.S.: Another point that comes to my mind when I look at your question, is if those embeddings are actually meaningful by themselves. For the pure BERT, we know that the sentence embeddings:

I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors.

I assume you will probably need to fine-tune wav2vec2 a bit. You can use huggingfaces Wav2Vec2ForSequenceClassification class for that.