8
$\begingroup$

I have been working on a problem where published results using deep learning are substantially worse than results I have obtained on the same task (using the same experimental protocol) using simple statistical methods (in this case, multinomial logistic regression). I'm wondering if this is not an uncommon event. Can anybody provide concrete examples where deep learning performs demonstrably worse than simple classifier systems?

$\endgroup$

1 Answer 1

6
$\begingroup$

Of course there are!! And this is a great question! I myself have discovered recently what follows!

Deep learning architectures are good on unstructured data, such as time series, images, audio, graphs, etc.

Decision tree models, together with their siblings (e.g., random forests, gradient boosting trees), are still dominant on structured data.

Here is a recent paper demonstrating what I said.

Why deep learning architectures are good on unstructured data? Well, it simply boils down to the exploitation of the inductive bias, such as spatial correlations in the case of Convolutional Neural Networks (CNNs) or symmetries on signals (see, e.g., Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges).

$\endgroup$
2
  • 3
    $\begingroup$ +1 I don't think I would describe XGBoost as a simple classifier system! The reference looks useful/interesting though. Interestingly the application I am working on is unstructured data (and lots of it) with strong spatial correlations (could be viewed as image classification with very small images). $\endgroup$ Aug 26, 2022 at 15:30
  • 1
    $\begingroup$ Well, with respect to a deep neural network, XGBoost is still less complex (e.g., very simple gradients, less computational power, etc.). You must agree, however, that this is certainly true for random forests if you consider gradient boosting trees non-simple classifiers and you are willing to sacrifice some accuracy (with respect to random forests). Anyway, thanks for +1, and I will follow the answers because I'm also interested. $\endgroup$
    – Eduard
    Aug 26, 2022 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.