Why Deep Learning / Neural Networs don't achieve state of the art results in tabular data problems?

Question

Apparently, deep learning methods don't achieve state-of-the-art results on tabular data problems [1,2]. This claim appears to be known also by Kagglers. The SOTA method looks like it is the gradient boosting decision tree.

Is there any intuition on why this happens? Any relevant literature on the topic?

Do neural networks have a stronger IID assumption that inhibits learning in tabular data?

Literature:

1. Deep Neural Networks and Tabular Data: A Survey https://arxiv.org/abs/2110.01889
2. Do We Really Need Deep Learning Models for Time Series Forecasting? https://arxiv.org/abs/2110.01889

Answer 1

In my opinion, deep learning methods are best for (but not only for) representation learning on very generic and homogenous data formats: sound, images, text, videos etc. For most of these formats there are pre-trained models that achieve state-of-the-art results.

On the contrary, tabular datasets typically have more heterogeneous and messy structure often related with domain knowledge, which is outside the scope of automatic representation learning. Thus, manual feature engineering and methods like gradient boost perform better.

By the way, the most power in deep learning comes from fine-tuning models that have already been pre-trained on huge dataset e.g. BERT by Google for textual data. Then, considering how difficult or impossible it is to use pre-trained deep learning models on messy/heterogenous tabular datasets, deep learning loses its attractiveness in this scenario.

Another reason is that learning algorithms also have what we call inductive biases. If the domain knowledge that is essential for solving a tabular business problem has by its nature a tree-based/taxonomy structure, it is logical that tree-based models will have an edge. ( Because even the domain expert or label annotator will follow a tree based process)

On the other hand, if a set of images and their labels depend on spatial features that can be captured with a filter, deep learning with CNN makes better assumptions.

Finally, because deep learning models have a high number of parameters to be learned, it requires huge datasets to avoid overfitting. Thus, it would not be a good option for small tabular datasets when it is difficult/expensive to acquire more data.

Answer 2

My intuition is that it is because tabular data does not necessarily form a manifold.

There is some limited and indirect support in the literature for this hypothesis:

According to [1], the manifold hypothesis [2] states that all natural data lies in a lower-dimensional space (a manifold) which is embedded in the higher dimensional feature space and locally behaves like an Euclidean space. And deep learning models can learn these manifolds which is the reason why they actually work so well. However, the author does not explicitly define what "natural data" is but he provides some examples like human faces, MNIST digits, natural language and human voices. In line with that, [3] provides images as an example for natural data and [4] states that it has been shown that some image and video data, in fact, form a manifold. ([4] also provides examples of other, non-neural, manifold learning algorithms.)

In summary, I infer that these authors do not refer to tabular data when they speak about natural data. And, hence, neural nets might not work that well because tabular data does not form a manifold.

At least this is the case for the way we usually encode tabular data, i.e. it might be possible to present tabular data in way that does embed a manifold. But that is just speculation. (There are a few examples which go into this direction already: Some applications of CNNs operate on data visualizations/plots and another example are transformers which are able to learn arithmetic operations from natural language to a limited extend.)

Moreover, [1] raises the point that neural nets work well because the inductive biases of their architecture mirror the data (e.g. CNNs have a very special structures which works particularly well on image data). Again I am speculating, but maybe we will develop architectures going forward which provide inductive biases suitable for tabular data - or some special types of tabular data. But currently we do not have these which is another reason why neural nets lack performance on tabular data.

---

References:

[1] Chollet, Francois; "Deep Learning with Python"; Second Edition 2nd Edition; 2021

[2] https://en.wikipedia.org/wiki/Manifold_hypothesis

[3] https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

[4] Cayton, Lawrence; "Algorithms for manifold learning"; 2005; https://www.lcayton.com/resexam.pdf

Answer 3

A few years later after this post, some authors wrote a paper at NeurIPS about it.

Why do tree-based models still outperform deep learning on typical tabular data? https://proceedings.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Paper-Datasets_and_Benchmarks.pdf

The twitter thread (https://twitter.com/GaelVaroquaux/status/1549422403889106944)

A chatGPT summary of the thread (unrevised)

1. The gap in performance between deep architectures and tree models narrows on large datasets, but such datasets are exceptions for tabular data.

2. Smoothing outcomes in feature space narrows the gap, as deep architectures struggle with irregular patterns, while tree models are not affected by smoothness.

3. Uninformative features have a greater impact on MLP-like neural architectures. Removing the least informative features narrows the performance gap.

4. Data non-invariance by rotation suggests that learning procedures should be rotationally invariant. After applying random rotations to the data, deep architectures outperform tree models, which excel in rotational invariance.