Neural Network Classifier Different result

Question

I am using neural network for a binary classification problem (yes or no). My training data set is not that big (39,000 records). After using SMOTE to balance the target, I have 50 input variables that are all numerical. There are 3 hidden layers with 100, 50, 10 neurons respectively. When I train neural network, I get a very different outcome, in terms of accuracy, AUC, and the confusion matrix. Why is this and what can I do to improve my model?

UPDATE: My class(yes or no) is not unbalanced because I used smote to balance it. It was very imbalanced initially. And what I meant is every time I train, I get a different outcome.

Answer 1

So your model is not very robust. You provide not too much infomration in your question. So it is hard to tell what is going on. Are classes highly unbalanced? It is possible, that your NN learns very "local" things and fails to get the bigger picture.

First thing I would do is to establish a proper baseline model. No idea what kind of data you work with, but you could use something like Logit. Maybe NN are not adequate in your case at all because of too few learning samples. So maybe you could also look towards other methods such as Random Forest with Boosting (xgboost, lightgbm, catboost).