Working with audio data with different sample rates in Tensorflow

I am trying to implement (as a toy project) some aspects of speech recognition in Tensorflow. The audio files I want to use as training and test data have different sample rates (16, 20, 44 and 44.1 kHz).

The following list is not exhaustive, just listing a few options I could think of.

1. I can resample the whole signal (which is quite un-periodic) using scipy.signal.resample, probably after padding with zeros to achieve a signal length which is a power of 2, chopping of the equivalent amount afterwards.
2. I can try to chop the signal into short, non-overlapping windows, which I modify using scipy.signal.resample before glueing them back together and applying the rest of the chain.
3. I can chop the signal up into overlapping windows, scipy.signal.resample them and stick them together in a weighted fashion.
4. I can take the spectrogram tensor obtained using tf.contrib.signal.stft and try to stretch it, for example using rational function interpolation (reflecting tube models of the vocal tract)
5. I can assume that the model will learn to abstract away from sample rates, maybe even helping it (or hindering?) by adding convolution layers before the actual model I am interested in.

In which step of the model should I normalize sampling rates, and how should I do it?

If you have ffmpeg installed, you can just use tf.contrib.ffmpeg.decode_audio. That function takes a samples_per_second argument and does the resampling for you. The following calculates a log magnitude short-time spectrogram of your data.

waveform = tf.squeeze(
tf.contrib.ffmpeg.decode_audio(
file_format=tf.placeholder(tf.string, name="format"),
samples_per_second=p["sample_rate"],
channel_count=1))

log_mag_spectrogram = tf.log(tf.abs(tf.contrib.signal.stft(
waveform, win_length, hop_length,
n_fft, pad_end=False) + 1e-8))

• That function is now deprecated. Dec 5 '18 at 22:50

Is there a reason you want to do the re-sampling inside the tensorflow computation graph? Otherwise, there are well-tested and established tools to do re-sampling as a separate step, consider running

\$ sox unnormalized_file.wav -r 44100 normalized_file.wav


on all your data files.

• ffmpeg can do the same, but if you have ffmpeg installed, you can even use it from inside tensorflow, see my other answer. Sep 2 '18 at 21:09