# Different hidden layer architectures deliver the same classification results, is that normal?

I have a data set with 600 data points with about 10 attributes (binary). The dataset has been normalized:

Xnormalized = StandardScaler().fit_transform(X)


The outputs are binary (2 classes) I tried different MLP architectures using 1 or 2 hidden layers with different number of nodes and performed a repeated stratified cross validation as shown below:

names=[]
classifiersmlp = []
for i in range(5, 26,5):
names.append("mlp-"+str(i))
classifiersmlp.append(MLPClassifier(solver='sgd',
random_state=1,
activation='tanh',
hidden_layer_sizes=[i]))
for j in range(5,26,5):
names.append("mlp-"+str(i)+"_"+str(j))
classifiersmlp.append(MLPClassifier(solver='sgd',
random_state=1,
activation='tanh',
hidden_layer_sizes=[i,j]))

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=1)
scoring = {'accuracy': 'accuracy',
'recall': 'recall',
'precision': 'precision',
'f1_score':'f1'}
mlpResults = []
for name, clf in zip(names, classifiersmlp):
print(name)
cvresultMLP = cross_validate(clf, Xnormalized, y, cv=cv, scoring=scoring)
mlpResults.append(cvresultMLP)
print(np.mean(cvresultMLP['test_recall']))
print(np.mean(cvresultMLP['test_precision']))
print(np.mean(cvresultMLP['test_f1_score']))


The results of all architectures are very similar (only 1-2% difference in all 3 evaluation measures (recall (approx 78%) and precision (approx 74%)). Is it normal that the architectures are equally good or should there be greater differences? What does it mean that the results are so similar?

After a comment from @yohanesalfredo that the scaling is outside the cv, I updated the code:

names=[]
classifiersmlp = []
for i in [5,25,50]: #range(5, 5,3)
names.append("mlp-"+str(i))
classifiersmlp.append(MLPClassifier(solver='sgd',
learning_rate_init=0.01,
random_state=1,
activation='tanh',
hidden_layer_sizes=(i)))
for j in [5,20]: #range(1, 5,3):
names.append("mlp-"+str(i)+"_"+str(j))
classifiersmlp.append(MLPClassifier(solver='sgd',
random_state=1,
learning_rate_init=0.01,
activation='tanh',
hidden_layer_sizes=(i,j)))

scoring = {'accuracy': 'accuracy',
'recall': 'recall',
'precision': 'precision',
'f1_score':'f1', # according to docu only score for the 1 label
'roc_auc':'roc_auc'}
mlpResults = []
rand_state=1
for name, clf_temp in zip(names, classifiersmlp):
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=rand_state)
rand_state+=1
classifier_pipeline = make_pipeline(preprocessing.StandardScaler(), clf_temp)
cvresultMLP = cross_validate(classifier_pipeline, X, y, cv=cv, scoring=scoring)
mlpResults.append(cvresultMLP)
print(np.mean(cvresultMLP['test_recall']))
print(np.mean(cvresultMLP['test_precision']))
print(np.mean(cvresultMLP['test_f1_score']))
print(np.mean(cvresultMLP['test_roc_auc']))

• I examined the incorrectly classified data and found that it is not possible for the classifier to classify it correctly. for example, (0,0,0) occurs 100 times with the label 0 and 10 times with the label 1. So this is to be expected that these 10 are misclassified. I think the results are similar because all architectures are able to adapt to the data in the best possible way. Better results are probably not possible with this data. More features are needed so that the network can learn something new from it. Dec 17 '19 at 19:15