I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree works the best with training accuracy of 1.0 and test accuracy of 0.5. The neural networks, which I believed would always perform the best no matter what has a training accuracy of 0.92 and test accuracy of 0.42 which is 8% less than the decision tree classifier.
Could someone please explain the circumstances/cases where neural networks could have low accuracy when compared to a modelling technique like the decision tree. I had tried my neural network with different configurations like:
1 hidden layer and 1 neuron : Train Accuracy 34% Test Accuracy 42% 7 hidden layers and 5 neurons in each layer: Train Accuracy 79% Test Accuracy 42% 1 hidden layer and 100 neurons: Train Accuracy 34% and Test Accuracy 35%
but not in a single case, I found the neural network to beat the decision tree test accuracy of 50%.