0
$\begingroup$

I try to classify car sound samples. Using MLPClassifier from Scikit. I'm getting very different and confusing test results between 2 different test sets, and I am stuck:

  • Training is done with the first data set of 1500 samples, splitted as 70/30 train/test. Second set is isolated, I use only for the final testing, 700 samples completely unseen data.
  • First test set is stable and always around %90 test, %99 train set accuracy.
  • Second test is completely random, it changes while I change the randomState variable of the classifier. It can be %20 or %80 accuracy.

This is extremely frustrating. The difference between two sets is first one is mainly from sounds that come with UrbanSound data collection. And second set is more real world, I recorded with an iPhone. I checked that they all have the same duration, sample rate, and bit rate.

So my question is:

  • if your accuracy changes randomly by changing your randomState of your classifier, on a certain test set, what does this tell you about your data?
  • And second what would be my approach now? Totally lost.

ps: My features are first 20 of the mfcc coefficients. OR 60 band of mel spectogram. I try different things.

I am also wondering if all these code examples, academic papers about sound classification which uses the sound samples from UrbanSounds and ESD50 sets, did they ever test their accuracy with completely random real world sounds, recorded and processed with different tools ?!

Below is when I plot these 2 different sound sets(only the positive class) with:

  plt.plot(car_features_1,'.')
  plt.plot(car_features_2,'.')

They look quite different to me, they are both car sounds one is recorded by iphone other is coming from Urbansounds .etc

This is the code for 60 features(60 from mel spectogram ):

clf = MLPClassifier(activation='relu', solver='adam', alpha= 1, hidden_layer_sizes=(60, 60, 60), random_state=None, max_iter=2000)

clf.fit(X_train, y_train)

And this is how i extract features

 mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0) #128 array
 mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=20).T,axis=0) #20 array

enter image description here

enter image description here

$\endgroup$
  • $\begingroup$ Which data set did you train the model on? Is it possible that the sounds you recorded are not captured in the UrbanSound data set? Have you combined the two data sets and then trained on a sample from the combined data? The random state only impacts the training phase; so if you are training on your sounds, how many instances do you have in the data set? $\endgroup$ – Skiddles Jan 11 at 17:26
  • $\begingroup$ @Skiddles Trained with the first set. Which is splitted as 30/70 test/train. Second set I use only for testing, completely unseen data. first set is 1500 samples total(train + test), second one is 700(only for test). $\endgroup$ – Spring Jan 11 at 17:31
  • $\begingroup$ Based on your comment, I can see why you are confused. Changing the random state during testing should not change your results. The weights and biases should be static unless you are really re-initiating the training phase. Sorry if this is not helpful. $\endgroup$ – Skiddles Jan 11 at 17:38
  • $\begingroup$ @Skiddles yes I reinitiate training everytime. But its effect to 1 test set is very minimal, but to second set is huge $\endgroup$ – Spring Jan 11 at 17:41
  • $\begingroup$ Have you tried to train and test both on the second dataset (your iphone recorded dataset, with a similar 70%/30% split)? Does the accuracy fluctuate a lot with the random state? $\endgroup$ – user12075 Jan 11 at 19:19
1
$\begingroup$

Pal, you have set alpha to 1. Alpha is a L2 regularization term, its value is normally around 0.0005 ... 0.0001. By setting it to 1 you force the optimizer to make your model's weights almost zero.

P.S. After setting l2 reg term to its default value, you might get better accuracy, but if not, pay attention to the following. The ratio between the amount of data size (1500 samples, how many feature vectors?) and the number of weights (20x60x60x60xN, so more than 4M weights) is too small. It might well be that the model simply memorize the training set and does not generalize for sound classification.

Here are some suggestions that might help in training a sound classifier:

  1. Start with a simpler model. Sounds are known to be well classified with Gaussian mixture models, for instance. Unlike deep learning models, those are easier to train.

  2. Use data augmentation to increase the amount and diversity of your training data. You can mix the sounds with some light background noises.

  3. Try to reduce the amount of weights. Three layers with 60 neurons each is too much. Usually, layers become smaller with going up, something like 64->32->16. Also, convolutional layers can be very useful here, as they share the weights across their neurons. I used them quite successfully for sound classification.

  4. scikit-learn is quite unusual choice for training deep learning models. I would try Keras, it works perfectly with numpy arrays.

$\endgroup$
  • $\begingroup$ A good paper on sound classification: karol.piczak.com/papers/Piczak2015-ESC-ConvNet.pdf $\endgroup$ – Dmytro Prylipko Jan 12 at 9:42
  • $\begingroup$ Thanks a lot for your detailed answer! I tried the quick suggestions you gave but didnt help. I have %100 accuracy on one set and%50 on another, as you said its memorizing probably. I will go ahead and try the CNN now. Quick question, All my sound samples are 5 seconds, is that ok to start with CNN? $\endgroup$ – Spring Jan 12 at 11:07
  • $\begingroup$ 5 seconds is fine. It will result in several hundreds of feature vectors, so you will need kind of sliding window to feed the input into your model. Then you can average the class predictions over frames for each sample and come up with final result. $\endgroup$ – Dmytro Prylipko Jan 12 at 12:12
  • $\begingroup$ You might also find this post useful: stackoverflow.com/questions/4098279/… $\endgroup$ – Dmytro Prylipko Jan 12 at 12:13
  • 1
    $\begingroup$ Both data and hyperparams are important to train an accurate model. Having achieved that, you can trade between precision and recall using confidence threshold. Setting it higher will result in better precision and less recall. $\endgroup$ – Dmytro Prylipko Jan 15 at 10:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.