How to Improve MLP ANN accuracy

Question

I am trying to improve the accuracy of my model over the UCI Breast Cancer Dataset. There's 426 records, and it is a binary classification model.

X_train.shape
(426, 30)
X_test.shape
(143, 30)
y_train.shape
(426, 1)
y_test.shape
(143, 1)

• I have tried three types of gradient descents to compare the accuracies and it remains stagnant on 0.62.
• Batch sizes: Batch gradient descent over the whole dataset, for stochastic it is 1/1 and mini batch takes 16 batches. All of them show the same accuracy.
• The activation layers used are relu, relu, sigmoid, for 6,4,2 neurons in each layer.
• The loss function used is cross entropy.
• I have normalised the dataset using StandardScaler and Min Max Scaler from scikit learn.
• Learning rate is 0.00095, and epochs is 50. The following shows the learning rate after each epoch for stochastic gradient descent

    Cost after epoch 0: 0.690206  
    Cost after epoch 10: 0.664892  
    Cost after epoch 20: 0.661360  
    Cost after epoch 30: 0.660859  
    Cost after epoch 40: 0.660782  

Accuracy
0.6293706293706294

The Confusion Matrix gives 0 values for TN and FN, and I am aware this might be wrong, but do not understand where its going wrong.

90 0 53 0

Apart from all this, I have tried experimenting with different epoch numbers, hidden neurons, layers and learning rates, but accuracy does not improve.

Answer 1

I think two factors are working against you. First, you are working with tabular data, for which neural networks not often yield optimal results. At least when we are not talking aboult very large datasets. I would approach this problem with tree-based algorithms since they are so much simpler to handle and regularly beat neural networks in (tabular) benchmark datasets (...). Second, what I already mentioned, you are working with a comparatively small dataset. Neural networks are generally hungry for data. 569 Instances minus your test set won't give much space for learning.

I would go with RandomForest or XGBoost. In case you just want to experiment with neural networks on this dataset regardless you might be already at the possible optimum. A curcial part for training neural networks is the data preprocessing, which you did not talk about that. There might be room for improvement by applying feature scaling and general feature engineering. Although, I have to say I'm not familiar with this particular dataset.