Is there any consensus on choosing an appropriate ML approach?

Question

I am studying data science at the moment and we are taught a dizzying variety of basic regression/classification techniques (linear, logistic, trees, splines, ANN, SVM, MARS, and so on....), along with a variety of extra tools (bootstrapping, boosting, bagging, ensemble, ridge/lasso, CV, etc etc). Sometimes the techniques are given context (eg. suitable for small/large datasets, suitable for a small/large number of predictors, etc) but for the most part, it seems like for any regression or classification problem there exist a dizzying array of options to choose from.

If I started a job in data science right now and was given a modelling problem, I don't think I could do any better than just try all the techniques I know with basic configurations, evaluate them using cross-validation and pick the best. But there must be more to it than this.

I imagine an experienced data scientist knows the catalogue of techniques well and follows some mental flowchart to decide which techniques to try, instead of mindlessly trying them all. I imagine this flowchart is a function of a) number of predictors; b) variable types; c) domain knowledge about possible relationships (linear/non-linear); d) size of the dataset; e) constraints around computation time and so on.

Is there any such agreed on, conventional flowchart to follow, to choose the techniques? Or does it really boil down to "try lots of things and see what works best on the desired measure eg. cross-validation"?

Answer 1

My data science studies started as a Masters in Applied Statistics. One of the courses was in machine learning and it had a similar approach to what you are describing. So, I can empathize a little with your current view. But, just like other things you might have learned in life, the way you do things in an academic setting and the way you do things in a business environment (i.e. for a client) are completely different. Here's what I've learned since my initial studies:

1 - Learn Python

Sure, there's other tools out there and they're fine (I used to write R code with the best of them) but Python is where the future is at, period. Plus, very few tools scale as well as Python and that's important if you want to work on some really cool stuff.

2 - It all comes down to implementation

Guess what? All those things you're learning now (confusion matrices, factor reduction, etc) don't mean a thing to your clients. They're going to just look at you and say, "What's the product? When are you going to deploy something to my phone? Where my webapp to click on?". A large part of your job will be to turn all your work into a product and you will find yourself wearing a quasi-software-developer hat. This is also another good reason to learn python.

3 - Data pipelines take time

A LOT of your work will be on data manipulation and just making sure that the data pipelines you need are there. Sure, you have a database - but how are you going to update it? What pre-processing do you need? Where are you results stored? You will spend A LOT of time figuring out this stuff. You'll miss your school days when datasets were given to you in a nice and clean fashion :)

4 - Neural networks kick ass

Once you take a bite of this apple, it's hard to go back :). Learn Keras and enjoy the ride. After a while, you'll have to remind yourself what decisions trees are :)

5 - Model searches are much easier now

To be 100% clear, the "model search" approach you are doing now is VERY valuable experience. You should definitely work hard at those classes. However, if you have the time, look at either (1) Data Robot or (2) Watson Analytics. Both of those packages do, essentially, the same thing. They will take your dataset and find you the best model for it. All of the items you described above are done for you in a matter of seconds. It's almost scary how fast they are and they're very effective in helping you reduce your work. However, be warned that these packages only support supervised data. You will still have to do it the old-fashioned way for unsupervised data (or label some and use a neural network).

6 - I still use the theory behind other models

Even if I use neural networks a lot, the other models are still useful. You'll still use linear regression or decision trees for basic problems. It's also helpful when I decide to read some research papers on archivx or whatever. So, I'll still use them for my own study and understanding, but that's about it.

Have fun!

Answer 2

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning approaches has a special application domain and that is why people usually know where to use decision trees and where to choose neural networks, e.g. in situations that all of your inputs are real-valued numbers, attempting to use decision trees is not a wise choice. I try to explain the main things that an ML practitioner usually considers.

Number of Available Features

The number of features is important due to not being able to visualise them easily in cases there are so many features. This may lead to not being able to recognize whether the data is linearly separable or not. So many features do not imply that the dataset is not linearly separable. Consequently, say you want to use neural nets for modelling the problem. You shouldn't begin with a complicated network with so many layers and neurons. You have to begin with a single neuron, equivalent to logistic regression for classification tasks, to figure out whether your data is linearly separable or not. If you observe that you don't have good performance, you can add extra neurons and layers. How? Take a look at How to set the number of neurons and layers in neural networks.

Feature Space

About choosing ML approaches, it is simple to consider the limitations of each algorithm with respect to the feature space. E.g. decision trees are not very good for problems with many features which some of them are numerical features. They may get really big. SVMs are not very good for non-linear problems with so many features due to the fact that you have to specify the kernel size. For different regions in the feature space, a single kernel size may not be valuable. To generalize, problems that have very large input space are usually handled using neural networks. If the problem has so many features but they are e.g. binary features or categorical features with a small number of choices, then the problem has smaller feature space, input space, and other ML approaches can be considered.

Size of Dataset

Depending on your problem, feature types and feature ranges, the input space may be very small or very big. Consequently, the number of possible data may differ depending upon the input space. For large input space, as mentioned, neural nets are very powerful for mappings.

Distributions and Bayes Error

For different tasks, e.g. classification, you have to do statistical analysis for knowing your available dataset better. You have to investigate if there are input patterns that there are same but their labels differ. If so, why? Is that for expert error or not. The current feature space is not valid for understanding the problem. After addressing these questions you can employ Bayes error to investigate the best approach will have what accuracy on your data.