How to get data back into two separate audio files after successfully applying kmeans clustering on an audio file?

Question

Okay so I wrote a very simple python code to read a wav file, get the mfcc features and then use kmeans clustering on the features. The hello.wav file has two different people saying hello at the same time, I wanted to output two separate voices

from python_speech_features import mfcc
from scipy.io import wavfile
from sklearn.cluster import KMeans
sampfreq, data = wavfile.read('hello.wav')

mfcc_feat = mfcc(data,sampfreq)

kmeans = KMeans(n_clusters=2)
kmeans.fit(mfcc_feat)
print(kmeans.cluster_centers_)
print(kmeans.labels_)

Now I got the labels and cluster centers but how do I get back my two separate sound samples using this?

Answer 1

What you are trying to do is Blind Source Separation, and unfortunately it does not works as easy as that. When you are extracting MFCC from an audio file with a mixture of voices, MFCC clusters won't represent two people. For the separation, you should end up with two sound signals with the same number of samples. The MFCC are used as the feature's space for speakaer recognition tasks, for example, but not for separation in general. For more mathematical background about BSS you can visit: https://arxiv.org/pdf/1603.03089.pdf

The most simple case is the so-called Cocktail Party Problem, in where you have at least as many microphones as persons and the mixture has to be instantaneous (no delay between signals, only attenuations due to path-losses). This very simple case can be solved applying Independent Component Analysis, with the FastICA algorithm: http://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html

For delayed mixtures you can try the DUET algorithm: https://pdfs.semanticscholar.org/1413/746141f2871e0f45056a7696e019b8f8a100.pdf

For convolved mixtures and undetermined problems (more speakers than microphones) this PARAFAC-based algorithm performs quite well: http://people.ece.umn.edu/~nikos/05233821.pdf