5
$\begingroup$

I have a data set of total 8000 sound samples. These are the results of my multi layer neural network, binary classifier:

Precision: [0.95 0.96]
Recall: [0.96 0.95]
F-Score: [0.95 0.95]
S: [1217. 1254.]
Accuracy training 1.0
Accuracy Test 0.95

I am happy with the test accuracy, precision and recall. But the 1.0 accuracy bugs me since its overfitting to the training-set. Is this a bad thing even the test-set accuracy is satisfactory?

Here below I load the car sound features(1) and other types of sounds features(0) from disk and assign the labels. I double checked and I dont add labels as a feature while I process and extract features from audio. sets are cut to be equal to 4100 samples each

car_features =np.load('car_features_final.npy')
car_labels =np.ones(len(car_features),dtype=int)
shuffle(car_features, random_state=12)

other_features = np.load('other_features.npy')
other_features=list(islice(shuffle(other_features, random_state=12),4100))
other_labels = np.zeros(len(other_features),dtype=int)
shuffle(other_features, random_state=12)

all_features = np.append(car_features, other_features, axis=0)
all_labels = np.append(car_labels,other_labels, axis=0)

X_train, X_test, y_train, y_test = train_test_split(all_features, all_labels, test_size=0.30, random_state=42)

clf = MLPClassifier(activation='relu', solver='adam', alpha= 0.1, hidden_layer_sizes=(300, 300, 300, 100), random_state=1, max_iter=500)
clf.fit(X_train, y_train)

y_predicted_test = clf.predict(X_test)
y_predicted_train = clf.predict(X_train)
p,r,f,s = precision_recall_fscore_support(y_test, y_predicted_test)
$\endgroup$
  • $\begingroup$ I guess it is better to employ drop out for last layers to increase the test accuracy, two 97% are better than 100 and 95. $\endgroup$ – Media Jan 4 '19 at 11:40
  • $\begingroup$ The accuracy seems quite high. So high that I might try different splitting methods to see if the result still holds. $\endgroup$ – The Lyrist Jan 4 '19 at 17:10
  • $\begingroup$ @TheLyrist You mean the test or training data, and why you think it is so high? And can you explain what you mean by different splitting methods? tnx $\endgroup$ – Spring Jan 4 '19 at 18:10
  • $\begingroup$ Yes I mean the method to split the data into training and testing. Depending on the data you might actually achieve a model that gives you such high accuracy, but if I see mine with > 80% i would double check my data to ensure the labels are correct, I have not included feature that I shouldn’t include (like the label, etc.), double check data cleansing etc. Other due diligence. I am a skeptical person in nature, it works for and against me sometimes $\endgroup$ – The Lyrist Jan 4 '19 at 18:14
  • $\begingroup$ If you can show us some sample code we might be in a better position to help. Do you use random splitting? Did you try cross validation? Perhaps your features really are strong predictors of your result, but I should check for other possible items as due diligence. I have seen people including the label as part of the features, including row id (which in some cases are useful, but in most cases are not). I am not saying that these are the cases here, but as a skeptical person I am, I will check if I see mine so high as yours $\endgroup$ – The Lyrist Jan 4 '19 at 18:44
4
$\begingroup$

It is overfitting if you have an accuracy on training of 100%, but the test accuracy would be 5%. That's overfitting.

In your case, there is a good match between training and testing accuracy. To check for any sampling bias, do a K-fold cross validation with the same hyper-parameters.

$\endgroup$
0
$\begingroup$

A good alternative way to find out is to pick another random sample (maybe using a different selection method) from the rest of the data (so not from your first 8000 sound data), and look at the accuracy on that new (unseen) data. You will have poor performance if you have too much over-fitting.

$\endgroup$
  • $\begingroup$ OP already did that, he has 95% on testing accuracy. $\endgroup$ – Matthieu Brucher Jan 4 '19 at 11:23
  • 1
    $\begingroup$ I was mentioning an independent test sample, using a different selection method. By splitting a training set into a training and test set, it gives you a theoretical performance, which is very different to a "real world" (=real data population) performance. This is based on experience on production systems (compared to trained models), but maybe the question is not based on a real world system where more data could be collected...? $\endgroup$ – Nico Jan 4 '19 at 15:12
0
$\begingroup$

Simple answer is No. Overfit is a problem where the data reality is impossible to fit perfectly so the model is build with a consideration of something not been trained for.

On the other hand, some data are assured to be fit and accurate to its sample. For example, if you have results data of the formula y = x^3 and your model got it right (found this formula). In this case, your accuracy must be 100%!

So, in the conclusion, your model accuracy should be related to the nature of the data reality.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.