As answer and explained in detail by @cronoik on this post, following is the code to get wav2vec embeddings.
import librosa
import torch
from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2Model
input_audio, sample_rate = librosa.load("/content/bla.wav", sr=16000)
model_name = "facebook/wav2vec2-large-xlsr-53"
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name)
model = Wav2Vec2Model.from_pretrained(model_name)
i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate)
with torch.no_grad():
o= model(i.input_values)
print(o.keys())
print(o.last_hidden_state.shape)
print(o.extract_features.shape)
OUTPUT:
odict_keys(['last_hidden_state', 'extract_features'])
torch.Size([1, 1676, 1024])
torch.Size([1, 1676, 512])
The features are multi-dimensional for sample file ([bacth_size, seq_len, hidden_size]), and probably will need some pooling (e.g. mean) to be applied.
EXPLANATION:
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name)
loads the
Wav2Vec2FeatureExtractor component of the Wav2Vec2 architecture using
the from_pretrained method from the transformers library. This
component is used for normalizing the audio signals.
model = Wav2Vec2Model.from_pretrained(model_name)
loads the
Wav2Vec2Model component of the Wav2Vec2 architecture, which is used
for generating representations of the audio signals.
i= feature_extractor(input_audio, return_tensors="pt", sampling_rate=sample_rate)
normalizes the input audio signal
input_audio by subtracting the mean and dividing by the standard
deviation, and returns the normalized signal as a PyTorch tensor. The
sampling_rate and return_tensors arguments are also passed.
with torch.no_grad()
: is a context manager that disables gradient
computation during the forward pass of the model, reducing memory
usage and speeding up computation.
o= model(i.input_values)
generates a representation of the normalized
audio signal i.input_values using the Wav2Vec2Model component. The
representation is returned as a dictionary o containing multiple
outputs.