I was recently working on some classification problem where decision trees performed better than neural networks. I had tried various combinations with neural networks altering the number of neurons / hidden layers with an objective to beat the decision tree classifier accuracy on the test set. But the best accuracy I could achieve with neural networks was that of 0.42 and decision tree was at 0.50.
I had asked a question here, as what could be the case and someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image). In the comment linked to the same answer, it was pointed out that :
Well you can take a look at kaggle competition winners. In competitions containing structured data by far the most popular algorithm is xgboost (along with other similar algorithms lightgbm, catboost, etc.). On the other hand Neural Networks are rarely used in these competitions because they are not so strong with these types of data. This is also evident by the near 20-year disappearance of neural networks, until deep learning made them relevant again. During these years trees and SVMs on top.
That could generally be true but I do not have the intuition/reason as to why neural networks do not work well with the structured data? Could someone help me reason that? It will also be great if you could point me to some paper/post that explains this.
One feeling that I have is, it could be because of the less volume of data. Neural networks might not generalize well with fewer data points as compared to other classifiers like decision trees, svms etc. But then I am not really sure about this.