Voice recognition with fourier transformation with audio input in python

Question

First of all is using the fourier transformation even a good method for recognizing different speakers? I'm not sure if it could recognize a voice if the things that are said are different. I know google and amazon have features of voice/speaker recognition in their voice assistants but what would be a good way to make that too if the fourier transformation doesn't work out?

I want to recognize voices using a neural network, to do that I need to first get a good input for the neural network but by just giving the sound recording as input I don't think it would work because it is based on frequency and time. So I found the Fourier transformation and now I'm trying to transform my audio file with Fourier and plot it.

My questions are:

How can I plot a Fourier transformation with audio input in python? And if that is working, how can I input the Fourier transformation in the neural network (I thought perhaps give every neuron a y value with the neurons as the corresponding x value)

I tried something like (a combination of things I found on the internet):

import matplotlib.pyplot as plt
from scipy.io import wavfile as wav
from scipy.fftpack import fft
import numpy as np
import wave
import sys

spf = wave.open('AAA.wav','r')

#Extract Raw Audio from Wav File
signal = spf.readframes(-1)
signal = np.fromstring(signal, 'Int16')
fs = spf.getframerate()
fft_out = fft(signal)


Time=np.linspace(0, len(signal)/fs, num=len(signal))

plt.figure(1)
plt.title('Signal Wave...')
plt.plot(Time,fft_out)
plt.show()

but considering my input in the mic was 'aaaaaa' it does not seem right.

Answer 1

The vanilla version of Fourier Transform (fft) is not the best feature extractor for audio or speech signals. This is primarily due to that FT is a global transformation, meaning that you lose all information along the time axis after the transformation.

You need to be familiar with the concept of short-time Fourier transform (STFT). Basically STFT tells you what frequency components exist in your signal at each timestamp. The result of STFT (its squared magnitude, to be precise) is called the spectrogram, which is what people usually visualize. An example of spectrogram from the link above:

You may refer to matplotlib.pyplot.specgram or scipy.signal.stft regarding how to plot a spectrogram in Python.

The most commonly used speech feature (as input for neural networks) is the Mel-Frequency Cepstral Coefficients, or MFCC, which carry the similar semantic meaning as the spectrogram. Other commonly used features include PLP, LPCC, etc which you can google for more details. But directly feeding the result of FT or STFT into a neural network is not the best practice.