1
$\begingroup$

First of all is using the fourier transformation even a good method for recognizing different speakers? I'm not sure if it could recognize a voice if the things that are said are different. I know google and amazon have features of voice/speaker recognition in their voice assistants but what would be a good way to make that too if the fourier transformation doesn't work out?

I want to recognize voices using a neural network, to do that I need to first get a good input for the neural network but by just giving the sound recording as input I don't think it would work because it is based on frequency and time. So I found the Fourier transformation and now I'm trying to transform my audio file with Fourier and plot it.

My questions are:

How can I plot a Fourier transformation with audio input in python? And if that is working, how can I input the Fourier transformation in the neural network (I thought perhaps give every neuron a y value with the neurons as the corresponding x value)

I tried something like (a combination of things I found on the internet):

import matplotlib.pyplot as plt
from scipy.io import wavfile as wav
from scipy.fftpack import fft
import numpy as np
import wave
import sys

spf = wave.open('AAA.wav','r')

#Extract Raw Audio from Wav File
signal = spf.readframes(-1)
signal = np.fromstring(signal, 'Int16')
fs = spf.getframerate()
fft_out = fft(signal)


Time=np.linspace(0, len(signal)/fs, num=len(signal))

plt.figure(1)
plt.title('Signal Wave...')
plt.plot(Time,fft_out)
plt.show()

but considering my input in the mic was 'aaaaaa' it does not seem right.

$\endgroup$

1 Answer 1

0
$\begingroup$

The vanilla version of Fourier Transform (fft) is not the best feature extractor for audio or speech signals. This is primarily due to that FT is a global transformation, meaning that you lose all information along the time axis after the transformation.

You need to be familiar with the concept of short-time Fourier transform (STFT). Basically STFT tells you what frequency components exist in your signal at each timestamp. The result of STFT (its squared magnitude, to be precise) is called the spectrogram, which is what people usually visualize. An example of spectrogram from the link above: spectrogram

You may refer to matplotlib.pyplot.specgram or scipy.signal.stft regarding how to plot a spectrogram in Python.

The most commonly used speech feature (as input for neural networks) is the Mel-Frequency Cepstral Coefficients, or MFCC, which carry the similar semantic meaning as the spectrogram. Other commonly used features include PLP, LPCC, etc which you can google for more details. But directly feeding the result of FT or STFT into a neural network is not the best practice.

$\endgroup$
2
  • $\begingroup$ Thanks very much for your reply! But if the STFT plots the frequency and the time, wouldn't that if you input it in a neural network make it train on which time the input is rather than the patterns in different voices? Or is that problem what the MFCC is for? $\endgroup$
    – Jair
    Sep 19, 2018 at 19:39
  • $\begingroup$ Features like MFCC is also local in terms of time. Depending on your NN models, you may stack the features at different timestamps (frames) into a long vector input, or using RNN which accept one feature vector (i.e. the feature vector for one frame) each step. More details: ieeexplore.ieee.org/abstract/document/7080838 $\endgroup$
    – user12075
    Sep 19, 2018 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.