Different hidden layer architectures deliver the same classification results, is that normal?

Question

I have a data set with 600 data points with about 10 attributes (binary). The dataset has been normalized:

Xnormalized = StandardScaler().fit_transform(X)

The outputs are binary (2 classes) I tried different MLP architectures using 1 or 2 hidden layers with different number of nodes and performed a repeated stratified cross validation as shown below:

names=[]
classifiersmlp = []
for i in range(5, 26,5):
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd', 
                                        random_state=1, 
                                        activation='tanh', 
                                        hidden_layer_sizes=[i]))
    for j in range(5,26,5):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd', 
                                            random_state=1, 
                                            activation='tanh', 
                                            hidden_layer_sizes=[i,j]))

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=1)
scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1'}    
mlpResults = []
for name, clf in zip(names, classifiersmlp):
    print(name)
    cvresultMLP = cross_validate(clf, Xnormalized, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score']))

The results of all architectures are very similar (only 1-2% difference in all 3 evaluation measures (recall (approx 78%) and precision (approx 74%)). Is it normal that the architectures are equally good or should there be greater differences? What does it mean that the results are so similar?

---

After a comment from @yohanesalfredo that the scaling is outside the cv, I updated the code:

names=[]
classifiersmlp = []
for i in [5,25,50]: #range(5, 5,3)
    names.append("mlp-"+str(i))
    classifiersmlp.append(MLPClassifier(solver='sgd', 
                                        learning_rate_init=0.01,
                                        random_state=1, 
                                        activation='tanh', 
                                        hidden_layer_sizes=(i)))
    for j in [5,20]: #range(1, 5,3):
        names.append("mlp-"+str(i)+"_"+str(j))
        classifiersmlp.append(MLPClassifier(solver='sgd', 
                                            random_state=1, 
                                            learning_rate_init=0.01,
                                            activation='tanh', 
                                            hidden_layer_sizes=(i,j)))


scoring = {'accuracy': 'accuracy',
           'recall': 'recall',
           'precision': 'precision',
           'f1_score':'f1', # according to docu only score for the 1 label
          'roc_auc':'roc_auc'}    
mlpResults = []
rand_state=1
for name, clf_temp in zip(names, classifiersmlp):
    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=rand_state)
    rand_state+=1        
    classifier_pipeline = make_pipeline(preprocessing.StandardScaler(), clf_temp)
    cvresultMLP = cross_validate(classifier_pipeline, X, y, cv=cv, scoring=scoring)
    mlpResults.append(cvresultMLP)
    print(np.mean(cvresultMLP['test_recall']))
    print(np.mean(cvresultMLP['test_precision']))
    print(np.mean(cvresultMLP['test_f1_score'])) 
    print(np.mean(cvresultMLP['test_roc_auc']))

Answer 1

There are several things I would like to mention :

1. I do not think that much change on your architecture will impact a lot. Try comparing 10, 20, 50, 100 or more depth. Difference will be most likely to be slightly more noticeable.
2. You are comparing accuracy which is based on labels instead of the probabilities. Try comparing logloss or auc score which is more sensitive to small changes.

Hope this helps.