1
$\begingroup$

I have a data set with 600 data points with about 10 attributes (binary). The dataset has been normalized:

Xnormalized = StandardScaler().fit_transform(X)

The outputs are binary (2 classes) I tried different MLP architectures using 1 or 2 hidden layers with different number of nodes and performed a repeated stratified cross validation as shown below:

names=[]
classifiersmlp = []
for i in range(5, 26,5):
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd', 
                                        random_state=1, 
                                        activation='tanh', 
                                        hidden_layer_sizes=[i]))
    for j in range(5,26,5):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd', 
                                            random_state=1, 
                                            activation='tanh', 
                                            hidden_layer_sizes=[i,j]))

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=1)
scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1'}    
mlpResults = []
for name, clf in zip(names, classifiersmlp):
    print(name)
    cvresultMLP = cross_validate(clf, Xnormalized, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score']))

The results of all architectures are very similar (only 1-2% difference in all 3 evaluation measures (recall (approx 78%) and precision (approx 74%)). Is it normal that the architectures are equally good or should there be greater differences? What does it mean that the results are so similar?


After a comment from @yohanesalfredo that the scaling is outside the cv, I updated the code:

names=[]
classifiersmlp = []
for i in [5,25,50]: #range(5, 5,3)
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd', 
                                        learning_rate_init=0.01,
                                        random_state=1, 
                                        activation='tanh', 
                                        hidden_layer_sizes=(i)))
    for j in [5,20]: #range(1, 5,3):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd', 
                                            random_state=1, 
                                            learning_rate_init=0.01,
                                            activation='tanh', 
                                            hidden_layer_sizes=(i,j)))


scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1', # according to docu only score for the 1 label
          'roc_auc':'roc_auc'}    
mlpResults = []
rand_state=1
for name, clf_temp in zip(names, classifiersmlp):
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=rand_state)
    rand_state+=1        
    classifier_pipeline = make_pipeline(preprocessing.StandardScaler(), clf_temp)
    cvresultMLP = cross_validate(classifier_pipeline, X, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score'])) 
    print(np.mean(cvresultMLP['test_roc_auc']))
$\endgroup$
1
  • $\begingroup$ I examined the incorrectly classified data and found that it is not possible for the classifier to classify it correctly. for example, (0,0,0) occurs 100 times with the label 0 and 10 times with the label 1. So this is to be expected that these 10 are misclassified. I think the results are similar because all architectures are able to adapt to the data in the best possible way. Better results are probably not possible with this data. More features are needed so that the network can learn something new from it. $\endgroup$
    – methus
    Dec 17 '19 at 19:15
2
$\begingroup$

There are several things I would like to mention :

  1. I do not think that much change on your architecture will impact a lot. Try comparing 10, 20, 50, 100 or more depth. Difference will be most likely to be slightly more noticeable.
  2. You are comparing accuracy which is based on labels instead of the probabilities. Try comparing logloss or auc score which is more sensitive to small changes.

Hope this helps.

$\endgroup$
5
  • $\begingroup$ 1) I tested more architectures and thre results remain similar (as you said) 2) The ROC_AUC score is also almost the same for all architectures $\endgroup$
    – methus
    Dec 17 '19 at 19:04
  • $\begingroup$ I examined the incorrectly classified data and found that it is not possible for the classifier to classify it correctly. for example, (0,0,0) occurs 100 times with the label 0 and 10 times with the label 1. So this is to be expected that these 10 are misclassified. I think the results are similar because all architectures are able to adapt to the data in the best possible way. Better results are probably not possible with this data. More features are needed so that the network can learn something new from it. $\endgroup$
    – methus
    Dec 17 '19 at 19:15
  • $\begingroup$ First of all it 'better' result is not guaranteed. For ML model you might sometimes hit the limit where you hit the limit whereby 'expanding' the model will not improve it. If this is the case you might want to improve your methodology (e.g. add feature engineering, ensembling, change scaling, etc) instead if you want to improve your model. So for this case if I were to give my opinion I suggest you to expand the features instead. Another incorrect practice I notice is that you are performing outside cv, scaler should be in the model cv pipeline. $\endgroup$ Dec 17 '19 at 19:19
  • $\begingroup$ Thanks for the tip! Could you please see if it's correct now? $\endgroup$
    – methus
    Dec 17 '19 at 19:43
  • $\begingroup$ great, thats what I meant. $\endgroup$ Dec 20 '19 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.