I am having issues with audio embedding using the wav2vec library while trying to classify emotions using audio signals from the EMODB dataset (Emotions dataset in German). I am using the following code to extract embeddings:
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained('facebook/wav2vec2-large-xlsr-53') #XLSR is for SR, not specifically Emotion Rec.
input_audio, sample_rate = librosa.load(emodb + file, sr=16000)
extraction = feature_extractor(input_audio, sampling_rate=16000, return_tensors="np", padding="max_length", max_length=max_len).input_values
The embeddings and shape of the vectors are:
(1, 143652)
forwav2vec
features(3, 162)
formfcc
features
Please note I have padded them to highest value. The length of audio files is around 1 to 2 seconds.
My intended task is emotion detection. I plan to use these embeddings from audio file, along with the text, for a downstream model for emotion classification, and for this I plan to use multimodal approach, using audio and text embeddings.
So, I trained an LSTM model on these embeddings but it was constantly overfitting on the training data (~100% accuracy and ~20% on testing).
Then I decided to use wav2vec embeddings and MFCC embeddings for a simple classification task using SVM. When I use the resulting embeddings in a simple SVM classifier, I am getting random results (15-30% accuracy) for wav2vec embeddings. As a comparison, when I extract features using MFCC and use them in the same classifier, I am getting an accuracy of around 70%.
Naturally, I visualized the embeddings using TSNE to check the quality of input and, I found to be getting strange results. Specifically, when I map 7 emotions, the resulting plot forms a spiral shape. When I only map 2 emotions, the resulting plots are different and also strange. The mappings are circular again when I add more features (3+).
I am unable to understand why I am getting these results and why the embeddings are so poor. I am wondering if this is because I am using a general XLSR model without fine-tuning it for emotion recognition.
I would appreciate any suggestions on how to extract features using wav2vec in a better way, or any papers or implementations that may be helpful.