How to extract embeddings from an audio file using wav2vec along with context

Question

I am trying to use wav2vec embeddings from the XLSR model for emotion recognition on the EMODB dataset. How can I extract embeddings using wav2vec? I want to use the XLSR model pre-trained with wav2vec, but I am not sure how to extract embeddings from audio files to use for emotion recognition.

I have made attempt like following but they are not correct, this results in random mappings.

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained('facebook/wav2vec2-large-xlsr-53') #XLSR is for SR, not specifically Emotion Rec. 
input_audio, sample_rate = librosa.load(emodb + file,  sr=16000)
extraction = feature_extractor(input_audio, sampling_rate=16000,  return_tensors="np", padding="max_length", max_length=max_len).input_values

Are there any series of steps to follow or libraries or methods I can use to extract the embeddings? Are there any examples or tutorials that I can follow to get started?

Answer 1

As answer and explained in detail by @cronoik on this post, following is the code to get wav2vec embeddings.

import librosa
import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2Model

input_audio, sample_rate = librosa.load("/content/bla.wav",  sr=16000)

model_name = "facebook/wav2vec2-large-xlsr-53"
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name)
model = Wav2Vec2Model.from_pretrained(model_name)

i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate)
with torch.no_grad():
  o= model(i.input_values)
print(o.keys())
print(o.last_hidden_state.shape)
print(o.extract_features.shape)

OUTPUT:

odict_keys(['last_hidden_state', 'extract_features'])
torch.Size([1, 1676, 1024])
torch.Size([1, 1676, 512])

The features are multi-dimensional for sample file ([bacth_size, seq_len, hidden_size]), and probably will need some pooling (e.g. mean) to be applied.

EXPLANATION:

• feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name) loads the Wav2Vec2FeatureExtractor component of the Wav2Vec2 architecture using the from_pretrained method from the transformers library. This component is used for normalizing the audio signals.
• model = Wav2Vec2Model.from_pretrained(model_name) loads the Wav2Vec2Model component of the Wav2Vec2 architecture, which is used for generating representations of the audio signals.
• i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate) normalizes the input audio signal input_audio by subtracting the mean and dividing by the standard deviation, and returns the normalized signal as a PyTorch tensor. The sampling_rate and return_tensors arguments are also passed.
• with torch.no_grad(): is a context manager that disables gradient computation during the forward pass of the model, reducing memory usage and speeding up computation.
• o= model(i.input_values) generates a representation of the normalized audio signal i.input_values using the Wav2Vec2Model component. The representation is returned as a dictionary o containing multiple outputs.