I was experimenting with different modelling methods including KNN, Decision Trees, Neural Networks and SVN and trying to fit my data to see which works the best. To my surprise, the decision tree works the best with training accuracy of 1.0 and test accuracy of 0.5. The neural networks, which I believed would always perform the best no matter what has a training accuracy of 0.92 and test accuracy of 0.42 which is 8% less than the decision tree classifier.

Could someone please explain the circumstances/cases where neural networks could have low accuracy when compared to a modelling technique like the decision tree. I had tried my neural network with different configurations like:

1 hidden layer and 1 neuron : Train Accuracy 34% Test Accuracy 42%
7 hidden layers and 5 neurons in each layer: Train Accuracy 79% Test Accuracy 42%
1 hidden layer and 100 neurons: Train Accuracy 34% and Test Accuracy 35%

but not in a single case, I found the neural network to beat the decision tree test accuracy of 50%.


1 Answer 1


Neural Networks, in my experience have several hyper-parameters (number of layers, neurons per layer, activation functions, optimizers, regularizers, etc.) and are very hard in finding the best configuration for each task. In fact in most cases it's not even worth it trying to find the optimal configuration as other classifiers can outperform Neural Networks with default hyper-parameters. Furthermore, NNs require caution as they are prone to overfitting.

For most tasks where you deal with structured data, I've found tree-based algorithms (especially boosted ones) to outperform NNs.

Some NN architectures are state-of-the-art tasks where we have a lot of unstructured data (e.g. CNNs for image-related tasks).

Finally, I'd like to say that there are no absolutes (e.g. SVMs will alawys outperform DTs). There is also a theorem along these lines: No Free Lunch Theorem.

  • $\begingroup$ How do you define unstructured data? $\endgroup$ Sep 16, 2018 at 12:25
  • 2
    $\begingroup$ Structured is usually when your data is in tabular format. Think of a spreadsheet. Each column represents something specific (age, sex, height, etc.). Unstructured is when you don't have a scheme describing the data. For example in an image each pixel location could be considered a feature. But that, by itself, doesn't represent anything specific. $\endgroup$
    – JkBk
    Sep 16, 2018 at 12:38
  • $\begingroup$ Okay. What is the intuition that neural-network are outperformed by tree-based algorithms when structured data is being considered? Any document/paper you could point me to? $\endgroup$ Sep 16, 2018 at 12:51
  • 3
    $\begingroup$ Well you can take a look at kaggle competition winners. In competitions containing structured data by far the most popular algorithm is xgboost (along with other similar algorithms lightgbm, catboost, etc.). On the other hand Neural Networks are rarely used in these competitions because they are not so strong with these types of data. This is also evident by the near 20-year disappearance of neural networks, until deep learning made them relevant again. During these years trees and SVMs on top. $\endgroup$
    – JkBk
    Sep 17, 2018 at 20:04
  • $\begingroup$ Okay thank you. But it will be great to have some intuition of why they do not perform well on structured data!! $\endgroup$ Sep 18, 2018 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.