0
$\begingroup$

At the moment I have this piece of code which cuts a Spectogram into fixed length tensors:

def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(l[0][0]), n):
        if(i+n < len(l[0][0])):
            yield X_sample.narrow(2, i, n)

The following piece of code

  1. downsamples the Audio
  2. Creates Mel_Spectograms and takes the log of it
  3. Applies a Cepstral Mean and Variance Normalization
  4. Then it cuts the spectogram with the code above into a fixed size of length and appends it to an array
for index, row in df.iterrows():
    #resample
    wave_form, sample_rate = torchaudio.load(row["path"], normalization=True)
    downsample_resample = torchaudio.transforms.Resample(
    sample_rate, downsample_rate, resampling_method='sinc_interpolation')
    wav = downsample_resample(wave_form)
    mel = torchaudio.transforms.MelSpectrogram(downsample_rate)(wav)
    mellog = np.log(mel + 1e-9)
    X_sample = speechpy.processing.cmvnw(mellog.squeeze(), win_size=301, variance_normalization=True)
    X_sample = torch.tensor(X_sample).unsqueeze(0)
    _min = min(np.amin(X_sample.numpy()),_min)
    _max = max(np.amax(X_sample.numpy()),_max)
    for chunked_X_sample in list(chunks(X_sample,  max_total_context)):
        print(len(chunked_X_sample[0][0]))
        if len(chunked_X_sample[0][0]) == max_total_context:
            X.append(chunked_X_sample)
            y.append(row["y"])

My question: Is this the common way to create features for deep learning? Do you have any suggestions to optimize this code? Furthermore I'm not sure if it is right to split the melspectograms instead of splitting the audio earlier.

$\endgroup$
4
  • $\begingroup$ Cutting the spectrogram instead of audio is common practice. Need to have an integer amount of spec timeframes anyway, easiest done this way. Exception is when a single spectrogram would be too large to fit in memory $\endgroup$ – Jon Nordby Oct 4 '20 at 13:35
  • $\begingroup$ Analysis windows are often created with overlap, to maximize number of instances in dataset. Your chunk splitter does not seem to allow for that $\endgroup$ – Jon Nordby Oct 4 '20 at 13:37
  • $\begingroup$ Librosa is one of the most popular libraries to compute spectrograms. But people use PyTorch or Tensorflow as well $\endgroup$ – Jon Nordby Oct 4 '20 at 13:38
  • $\begingroup$ The split into analysis windows / chunks is often done inside a data generator during the training process. This reduces duplication when overlap is used, and ties in well with on-the-fly data augmentation $\endgroup$ – Jon Nordby Oct 4 '20 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.