I have built a classical ANN using keras which provides probability (using sigmoid function) of the outcomes (0 or 1). While the accuracy of the model is high when the model is fit ~90%, the outcome probability of the test set results is very poor. How can I interpret this?

Build ANN

classifier = Sequential()
classifier.add(Dense(activation="relu",input_dim=7,kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="relu",kernel_initializer="uniform", units = 4))
classifier.add(Dense(activation="sigmoid", kernel_initializer="uniform", units = 1))
classifier.compile(optimizer="adam", loss="binary_crossentropy",metrics=['accuracy'])
classifier.fit(X_train,y_train, batch_size=10,epochs=100)

Predict results:

y_pred = classifier.predict(X_test)

I have attached the test set results where result probabilities can be seen with respect to frequency. blue shows probabilities for 1 and orange for 0 test results

Is this a case of overfitting? How can I tweak an ann to avoid overfitting?


First possibility:

How well is your dataset balanced? Are there mostly 0s? If I understand your graph well, it would seem that is the case.

If you in fact have 90% 0 and 10% 1 in your dataset, then your results would indicate the model is actually just performing random selection. This means it hasn't really learny anything about the data.

Second possibility:

If your training accuracy is high, but your test accuracy is low, this generally does point to overfitting, as you suspect.

Your model is essentially memorising the training data set, making it able to simply recall the correct result quite reliably, but without learning the underlying function that describes the dataset.

Here are a few things that might be behind this:

  • training for too many epochs
  • using a model that too big and complex
  • not using enough regularisation

I would recommend first checking the class balances of your dataset. If it highly imbalanced, you could look into performing stratified sampling to create your train/test split. This ensures that each batch has roughly the same number of 0 and 1 samples.

Secondly, you might consider using a slightly larger network. In the first instance, this could be as simple as increasing the number of neurons in each of your Dense layers. I would suggest something like (100, 50, 1) to begin with. That will immediately tell you if the size of the model is restricting learning.

  • $\begingroup$ Thank you for the detailed explanation. You have correctly identified that the 0 and 1 dataset are not balanced and are in the ratio of 80%-20%. I will try out the suggestions you have made. $\endgroup$ Sep 14 '18 at 9:31
  • $\begingroup$ I tried balancing the dataset for 0 and 1. This has resulted in much better test results and seems like the it is trained on the data. I will keep on tweaking it using other methods, but this is great. A big thank you @n1k31t4 ! $\endgroup$ Sep 14 '18 at 9:52
  • $\begingroup$ @AnkurBansal - you're welcome! :-) $\endgroup$
    – n1k31t4
    Sep 14 '18 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.