Why neural networks do not perform well on structured data?

Question

I was recently working on some classification problem where decision trees performed better than neural networks. I had tried various combinations with neural networks altering the number of neurons / hidden layers with an objective to beat the decision tree classifier accuracy on the test set. But the best accuracy I could achieve with neural networks was that of 0.42 and decision tree was at 0.50.

I had asked a question here, as what could be the case and someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image). In the comment linked to the same answer, it was pointed out that :

Well you can take a look at kaggle competition winners. In competitions containing structured data by far the most popular algorithm is xgboost (along with other similar algorithms lightgbm, catboost, etc.). On the other hand Neural Networks are rarely used in these competitions because they are not so strong with these types of data. This is also evident by the near 20-year disappearance of neural networks, until deep learning made them relevant again. During these years trees and SVMs on top.

That could generally be true but I do not have the intuition/reason as to why neural networks do not work well with the structured data? Could someone help me reason that? It will also be great if you could point me to some paper/post that explains this.

One feeling that I have is, it could be because of the less volume of data. Neural networks might not generalize well with fewer data points as compared to other classifiers like decision trees, svms etc. But then I am not really sure about this.

Answer 1

... someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image).

It's difficult to propose a universal analogy, but perhaps one moderately complex example which is easy to understand will suffice.

In the link you provided user JkBk mentions the "no free lunch theorem", let's add to that the "infinite monkey theorem". The fact is you can have one or more free lunches, just not as many as you like. Similarly you can not have infinite monkeys, but you can have a lot of them. Limits on each.

Take for example this data from the World Health Organization, let's only look at age versus height for boys aged 5 to 19. Let's look at a subset of the data in tabular form, I presume you understand the concept of "height-for-age".

Process this chart with your brain, what does it tell you. It tells you exactly everything averaged for a subset of the data. Can you draw any inferences from it, how about another several tables of data?

A neural network would have to process the data in all the tables and come up with a means to say "this age = this height" and "this height = this age" as an average.

Here is one of several tables:

Now let's look at the whole dataset in a chart:

See how at age 13 there is the start of a growth spurt, that continues almost to the age 15 when it starts to slow down. Between 15 and 17 it starts to level off, and from 17 to 19 there's almost no growth at all. From 5 to 11 a simple algorithm can easily solve the equation in either direction, beyond 11 the algorithm is not so simply but it still can be solved with a polynomial.

For a neural network once it gets on track and thinks it is coming up with a solution a monkey wrench gets thrown in to the works. Look at the "Alice and Bob example" (from 'no free lunch'), trying to solve the problem rationally requires knowledge and intelligence to avoid getting trapped.

Alternatively an Equation Solver can intelligently brute-force a solution, providing a relatively simple equation that can be used efficiently on large volumes of data.

Once you have a simple equation you could apply it to a huge dataset, like the Census, to check for errors such as age not matching height. A neural network would be better for discovering that age versus height was different than the average in a particular area and supply the reason (distance to inexpensive high quality food, versus earnings, and a sprinkle of education) whereas a simple polynomial wouldn't have the additional knowledge buried within it to make deductions.

A neural network doesn't comprehend (literally or figuratively) why a solution that was working later fails; it needs to continue, to learn to ignore it's success, to reach a working solution.

A non neural network solution doesn't care (literally or figuratively) whether it was on track, figured something out and learned to apply it, it only knows to minimize error and eliminate variables.

That doesn't mean that each couldn't switch roles and learn, or be programmed, to do the job of the other; but then the neural network would be pedantic about precision throughout (slow) and the solver would over analyze (calculate left or right with high precision) rather than simply weigh the outcomes with adequate precision for each step.

Answer 2

The paper "Why do tree-based models still outperform deep learning on tabular data?" by Grinsztajn, Oyallon, and Varoquaux (arXiv) attempts to tease this out. They have three empirical conclusions (my rephrasings):

1. NNs tend to favor too-smooth solutions.
2. NNs are more likely to overfit to uninformative features.
3. NNs are invariant under rotations of the training data (but problems represented by tabular data have solutions that "shouldn't" be rotation-invariant).