0
$\begingroup$

I am building a neural network to perform cry detection (i.e., binary classification of cry/non-cry situations) when capturing sound in a house environment.

To do so, I performed the following steps:

  1. Dataset building: I collected some samples in publicly available datasets, and enriched this sample set with some samples I registered with my own device (including speech, sounds from the home, sounds from the street, music...); I have a few samples (around 2500), which could be further enriched with new samples
  2. Feature extraction: I used librosa to extract MFCC descriptors from my samples
  3. Model creation and training: I built the model of my NN (using Keras, one recurrent layer, one dense layer and one final sigmoid node), and trained it on the features extracted at step 2.

Alas, although the model is performing very well on test set (achieving ~98% of accuracy), in the real world (i.e., when I put the trained model in a mobile application, stream the audio through my device and use it to classify the sounds I capture in the home environment I am at that moment) the network seems to perform very poorly, giving more or less random "cry"/"non-cry" labels over time.

I was wondering if this poor accuracy I achieve in the real world is related only to the small size of the dataset.

If this is not the case, is there some preprocessing I can do on my dataset? I learned from other questions that for speech recognition one would apply some preprocessing to remove some unwanted situations (e.g., to reduce the effect of volume change or to remove noise). Can you suggest me some preprocessing to do on the dataset, before training, either on the raw audio or on the extracted MFCC coefficients, so that this processing is adapt to my case study?

Thanks.

$\endgroup$
  • $\begingroup$ It sounds like you are overfitting on your training dataset. Did you split your dataset into train/validation/test? Are your samples realistic, i.e., comparable to what you record at deployment time? $\endgroup$ – hendrik Nov 19 '19 at 9:39
  • $\begingroup$ I have only splitted into train/test, due to the very small dataset. However, the samples are realistic (i.e., taken from the environment in which I am reusing the trained model). So does this mean that it is more convenient for me to enrich the dataset? Is normalization of MFCC coefficients of any help? $\endgroup$ – Eleanore Nov 19 '19 at 9:43
  • $\begingroup$ If you're getting 98% acc on your test set, which the network has never seen, but way less (how much less?) on your real-world test, perhaps you are accidentally processing the data differently? Do you standardize the same way etc.? There might also be a confounding issue. Do the cry/no-cry samples have comparable loudness (or are standardized somehow)? Is the training dataset balanced? ... $\endgroup$ – hendrik Nov 19 '19 at 9:48
  • $\begingroup$ I do not standardize the signal, neither raw nor by normalizing the extracted MFCC coefficients (and I recognize this could be an issue). Nevertheless, the training set is balanced (half of the dataset per class). I am also worried that, although the samples I collected are realistic, they are not all the sounds that network could hear, there could be many more voices, many more environment sounds... $\endgroup$ – Eleanore Nov 19 '19 at 9:51
  • 1
    $\begingroup$ To get rid of some loudness effects, you could try to base your MFCCs on a pre-computed log-power Mel spectrogram, which you normalize before computing the actual MFCCs. E.g. S = librosa.feature.melspectrogram(y=y, sr=sr, ...); S_norm=np.linalg.norm(S, ord=1); mfcc = librosa.feature.mfcc(S=librosa.power_to_db(S/S_norm)); I have not tested this, it's just an idea. In any case, I'd still standardize the MFCCs to zero mean and unit variance before running them through the network, and perhaps that's already sufficient. $\endgroup$ – hendrik Nov 19 '19 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.