Why use deep neural networks over methods like linear regression or SVM?

Question

This is a very broad question, but I was wondering why researchers would choose a deep neural network over linear regression or SVM? As in, what are the advantages and disadvantages of both?

Answer 1

Linear models work best on very sparse data so for instance they would work well on a dataset with only one-hot encoded categorical variables. However they tend to perform poorly on dense and correlated datasets. Usually most dataset would be a mixed between both. Given that linear models only learn linear relationship in the data, they require some feature engineering in order to learn non-linear patterns. While neural networks can learn themselves complex patterns, they require a great deal of fine-tuning that linear model do not (selecting the right architecture, training algorithm ...). Look up Wide and Deep neural networks that are an attempt at getting the best of both worlds.

SVMs work better will medium size data so when you need to scale, they might not be the best solution. Compared to linear models SVMs are very sensitive to their hyperparameters (which are data specific).

The popularity of neural networks comes from their state of the art performance on very specific problems (usually involving unstructured data) such as image classification (https://en.wikipedia.org/wiki/ImageNet). Sometimes it is misguided as people tend to think that throwing data at a complex algorithm will give them some insight (it does not). There is some hype around neural networks: for you old good structured data, a bit of feature engineering + simple model will work just fine.

My two cents is that researchers do not really overly focus on the analysis of structured data: this we know how to do and well. The challenge is to make use of all the images, videos, text to build powerful models and this is what fuels research oriented towards neural network (finding efficient architectures, how to train them better, ...). All of the "disruptive" applications of machine learning of the recent years (autonomous vehicles, face recognition, chatbots, deep fakes) involve neural networks, not SVM or linear models.

Answer 2

Perhaps the simplest answer is that researchers would choose a deep neural network over linear regression or SVMs because the deep neural network achieves better performance than linear regression and SVMs on their task. Otherwise, researchers wouldn't use them.