I have a small dataset (2000 rows) and am testing different algorithms for binary classification. The data set is very small but I do not have the option of increasing the dataset.

I have tested a multilayer perceptron (deep learning), xgbooster, logistic regression and they all give an accuracy of $\pm 60$% $(59, 58, 62, 61, \text{etc.})$ no matter what I change in parameters they all give a similar accuracy.

I know the dataset is small, but I would like to understand why this is happening, and how I can potentially fix it. I am even trying to remove some features from the model, all still giving $\pm60$%

Here is the code for the deep learning model

model = Sequential()
model.add(Dense(1, input_dim=X.shape[1], activation='relu'))
model.add(Dense(4, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

history = model.fit(X_train, y_train,epochs=60, batch_size=20,validation_split=0.1)

Attaching some sample learning curves for the deep learning model

enter image description here

  • $\begingroup$ Have you tried tuning hyperparameters? Does that change at least a little bit the accuracy of your model? And another question: aren't you leaking some test data into the training dataset? $\endgroup$
    – Nicolás
    Sep 8, 2021 at 15:39
  • $\begingroup$ Yes I have tried to tune the hyperparameters however binary cross entropy, sigmoid and relu are (as expected) yielding the best results. $\endgroup$ Sep 9, 2021 at 4:19
  • $\begingroup$ The 2 curves do not seems consistent: same legends but different behaviour: could you give details? The second curve seems you let the model overfit. Have you done early stop aroud epoch n 20? $\endgroup$
    – Malo
    Sep 11, 2021 at 9:25

1 Answer 1

  • This is not exactly the same accuracy, the difference between 0.58 and 0.62 might be significant.
  • This might be perfectly normal, there's no reason to expect the accuracy to be necessarily different for different classifiers.
  • Accuracy is a very simple evaluation measure for binary classification, it's suitable only if the data is perfectly balanced. It's likely that observing precision and recall would provide some insight about the differences between classifiers.
  • Of course, results depend a lot on the data. For example it could be that around 58% of the instances are very easy to classify, around 38% are very hard, leaving only 4% in between for which classifiers predict different labels.
  • Make sure to check the proportion of the majority class. If it's close to 60%, it's possible that some classifiers just predict the majority class all the time. This would be visible with precision/recall.
  • $\begingroup$ Thank you for your answer! Does that mean that the only source of my low accuracy is the low number of data? For balance that is true, given it is just 2,000 rows even a 800/1200 split between the 2 classes might make a big difference. For that i have run a SMOTE algorithm to perfectly balance my 2 classes. Still at 62% accuracy though $\endgroup$ Sep 9, 2021 at 4:17
  • $\begingroup$ @MarcHenrySaad 2000 instances is not necessarily small, it depends on the dataset. However if the majority class is 60% the performance is very poor, since it barely beats the majority baseline. Resampling is not needed here, the imbalance is not the main problem: the main problem is to understand why the model doesn't work well. first I would look at the confusion matrix (or precision/recall) for the different classifiers Of course a lot depends on the specifics of the data and the task, in the worst case scenario the task is just not feasible from the available features. $\endgroup$
    – Erwan
    Sep 9, 2021 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.