Accuracy keep changing by changing randomState of classifier

Question

I try to classify car sound samples. Using MLPClassifier from Scikit. I'm getting very different and confusing test results between 2 different test sets, and I am stuck:

• Training is done with the first data set of 1500 samples, splitted as 70/30 train/test. Second set is isolated, I use only for the final testing, 700 samples completely unseen data.
• First test set is stable and always around %90 test, %99 train set accuracy.
• Second test is completely random, it changes while I change the randomState variable of the classifier. It can be %20 or %80 accuracy.

This is extremely frustrating. The difference between two sets is first one is mainly from sounds that come with UrbanSound data collection. And second set is more real world, I recorded with an iPhone. I checked that they all have the same duration, sample rate, and bit rate.

So my question is:

• if your accuracy changes randomly by changing your randomState of your classifier, on a certain test set, what does this tell you about your data?
• And second what would be my approach now? Totally lost.

ps: My features are first 20 of the mfcc coefficients. OR 60 band of mel spectogram. I try different things.

I am also wondering if all these code examples, academic papers about sound classification which uses the sound samples from UrbanSounds and ESD50 sets, did they ever test their accuracy with completely random real world sounds, recorded and processed with different tools ?!

Below is when I plot these 2 different sound sets(only the positive class) with:

  plt.plot(car_features_1,'.')
  plt.plot(car_features_2,'.')

They look quite different to me, they are both car sounds one is recorded by iphone other is coming from Urbansounds .etc

This is the code for 60 features(60 from mel spectogram ):

clf = MLPClassifier(activation='relu', solver='adam', alpha= 1, hidden_layer_sizes=(60, 60, 60), random_state=None, max_iter=2000)

clf.fit(X_train, y_train)

And this is how i extract features

 mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0) #128 array
 mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=20).T,axis=0) #20 array

Answer 1

Pal, you have set alpha to 1. Alpha is a L2 regularization term, its value is normally around 0.0005 ... 0.0001. By setting it to 1 you force the optimizer to make your model's weights almost zero.

P.S. After setting l2 reg term to its default value, you might get better accuracy, but if not, pay attention to the following. The ratio between the amount of data size (1500 samples, how many feature vectors?) and the number of weights (20x60x60x60xN, so more than 4M weights) is too small. It might well be that the model simply memorize the training set and does not generalize for sound classification.

Here are some suggestions that might help in training a sound classifier:

1. Start with a simpler model. Sounds are known to be well classified with Gaussian mixture models, for instance. Unlike deep learning models, those are easier to train.

2. Use data augmentation to increase the amount and diversity of your training data. You can mix the sounds with some light background noises.

3. Try to reduce the amount of weights. Three layers with 60 neurons each is too much. Usually, layers become smaller with going up, something like 64->32->16. Also, convolutional layers can be very useful here, as they share the weights across their neurons. I used them quite successfully for sound classification.

4. scikit-learn is quite unusual choice for training deep learning models. I would try Keras, it works perfectly with numpy arrays.