0
$\begingroup$

I am trying to use wav2vec embeddings from the XLSR model for emotion recognition on the EMODB dataset. How can I extract embeddings using wav2vec? I want to use the XLSR model pre-trained with wav2vec, but I am not sure how to extract embeddings from audio files to use for emotion recognition.

I have made attempt like following but they are not correct, this results in random mappings.

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained('facebook/wav2vec2-large-xlsr-53') #XLSR is for SR, not specifically Emotion Rec. 
input_audio, sample_rate = librosa.load(emodb + file,  sr=16000)
extraction = feature_extractor(input_audio, sampling_rate=16000,  return_tensors="np", padding="max_length", max_length=max_len).input_values

Are there any series of steps to follow or libraries or methods I can use to extract the embeddings? Are there any examples or tutorials that I can follow to get started?

$\endgroup$

1 Answer 1

1
$\begingroup$

As answer and explained in detail by @cronoik on this post, following is the code to get wav2vec embeddings.

import librosa
import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2Model

input_audio, sample_rate = librosa.load("/content/bla.wav",  sr=16000)

model_name = "facebook/wav2vec2-large-xlsr-53"
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name)
model = Wav2Vec2Model.from_pretrained(model_name)

i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate)
with torch.no_grad():
  o= model(i.input_values)
print(o.keys())
print(o.last_hidden_state.shape)
print(o.extract_features.shape)

OUTPUT:

odict_keys(['last_hidden_state', 'extract_features'])
torch.Size([1, 1676, 1024])
torch.Size([1, 1676, 512])

The features are multi-dimensional for sample file ([bacth_size, seq_len, hidden_size]), and probably will need some pooling (e.g. mean) to be applied.

EXPLANATION:

  • feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name) loads the Wav2Vec2FeatureExtractor component of the Wav2Vec2 architecture using the from_pretrained method from the transformers library. This component is used for normalizing the audio signals.
  • model = Wav2Vec2Model.from_pretrained(model_name) loads the Wav2Vec2Model component of the Wav2Vec2 architecture, which is used for generating representations of the audio signals.
  • i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate) normalizes the input audio signal input_audio by subtracting the mean and dividing by the standard deviation, and returns the normalized signal as a PyTorch tensor. The sampling_rate and return_tensors arguments are also passed.
  • with torch.no_grad(): is a context manager that disables gradient computation during the forward pass of the model, reducing memory usage and speeding up computation.
  • o= model(i.input_values) generates a representation of the normalized audio signal i.input_values using the Wav2Vec2Model component. The representation is returned as a dictionary o containing multiple outputs.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.