I try to classify car sound samples. Using MLPClassifier from Scikit. I'm getting very different and confusing test results between 2 different test sets, and I am stuck:
- Training is done with the first data set of 1500 samples, splitted as 70/30 train/test. Second set is isolated, I use only for the final testing, 700 samples completely unseen data.
- First test set is stable and always around %90 test, %99 train set accuracy.
- Second test is completely random, it changes while I change the randomState variable of the classifier. It can be %20 or %80 accuracy.
This is extremely frustrating. The difference between two sets is first one is mainly from sounds that come with UrbanSound data collection. And second set is more real world, I recorded with an iPhone. I checked that they all have the same duration, sample rate, and bit rate.
So my question is:
- if your accuracy changes randomly by changing your randomState of your classifier, on a certain test set, what does this tell you about your data?
- And second what would be my approach now? Totally lost.
ps: My features are first 20 of the mfcc coefficients. OR 60 band of mel spectogram. I try different things.
I am also wondering if all these code examples, academic papers about sound classification which uses the sound samples from UrbanSounds and ESD50 sets, did they ever test their accuracy with completely random real world sounds, recorded and processed with different tools ?!
Below is when I plot these 2 different sound sets(only the positive class) with:
They look quite different to me, they are both car sounds one is recorded by iphone other is coming from Urbansounds .etc
This is the code for 60 features(60 from mel spectogram ):
clf = MLPClassifier(activation='relu', solver='adam', alpha= 1, hidden_layer_sizes=(60, 60, 60), random_state=None, max_iter=2000)
And this is how i extract features
mel = np.mean(librosa.feature.melspectrogram(X, sr=sample_rate).T,axis=0) #128 array mfccs = np.mean(librosa.feature.mfcc(y=X, sr=sample_rate, n_mfcc=20).T,axis=0) #20 array