Studying machine learning algorithms: depth of understanding vs. number of algorithms

Question

Recently I was introduced to the field of Data Science (its been 6 months approx), and Ii started the journey with Machine Learning Course by Andrew Ng and post that started working on the Data Science Specialization by JHU.

On practical application front, I have been working on building a predictive model that would predict attrition. So far I have used glm, bayesglm, rf in an effort to learn and apply these methods, but I find a lot of gap in my understanding of these algorithms.

My basic dilemma is:

Whether I should focus more on learning the intricacies of a few algorithms or should I use the approach of knowing a lot of them as and when and as much as required?

Please guide me in the right direction, maybe by suggesting books or articles or anything that you think would help.

I would be grateful if you would reply with an idea of guiding someone who has just started his career in the field of Data Science, and wants to be a person who solves practical issues for the business world.

I would read (as many as possible) resources (books,articles) suggested in this post and would provide a personal feed back on the pros and cons of the same so as to make this a helpful post for people who come across a similar question in future,and i think it would be great if people suggesting these books can do the same.

Answer 1

I would recommend limiting yourself to a few tried and trusted algorithms. I would not recommend Elements of statistical learning ( as a first book). It is too theoretical, aimed at graduate students, with exercises asking how to prove X or Y... I think ISL is more appropriate, with more practical advice ( in any case both books are free as pdf downloads).

Besides statistics, I would make sure you are comfortable with experimental design/AB tests, and with Business Intelligence/Visualisation.

Answer 2

Arguably someone calling themself a data scientist ought to know more about the intricacies of the algorithms they use—e.g. what affects the convergence rate of the Fisher scoring algorithm in GLM—than a common or garden statistician—who might be content just to know that the maximum-likelihood solution will be found (perhaps after they make a cup of coffee). In any case understanding the general concepts of statistics & machine learning is important in addition to familiarity with the methods you do use—the theory behind them, the assumptions they make, what diagnostic checks you should perform, how to interpret the results. Avoid being this parody.

You'd probably enjoy reading Hastie et al. (2009), The Elements of Statistical Learning.

Answer 3

Well, I would say knowing the intricacies of 1 or 2 algorithms in detail( like the inner workings of their parameters) is definitely better than knowing how to run a bunch of them.

I have been in the Analytics area for about 11 years and a Data Scientist for 2.5 years and I'm speaking from experience. On the other hand, you should definitely be aware of other things out there (more recent algorithms like deep learning, SVM, XGboost etc.) which might be more applicable to your problem at hand.

I think Dr. Andrew Ng's course goes into a quite a few details of some algorithms and it's a good start. As others have pointed out, http://statweb.stanford.edu/~tibs/ElemStatLearn/ is a good book and it has videos to go with it.

This is my personal opinion, the algorithms which you shouldn't miss out on are: (Know these in detail):

1) Multiple linear regression 2) Logistic regression 3) Common techniques of dimensionality reduction like PCA 4) K-means clustering 5) Non-linear regression 6) Optimization methods: gradient based search methods, linear programming and discrete optimization 7) Concepts and algorithms in feature engineering 8) Simple time-series forecasting methods

More Esoteric algorithms:

1) Random Forests 2) SVM 3) deep learning 4) Other methods of dimensionality reduction like LDA 5) Other kernel based methods 6) Genetic algorithms 7) XgBoost 8) Dynamic regression 9) GARCH/ARCH methods 10) Structural equation modeling 11) Box Jenkins methods in time-series forecasting 12) Information theory: information gain, mutual gain etc.

Answer 4

I had been in a similar situation. I started out with each and every algorithm here (and in great detail).

However, I soon found out that the academia in machine/deep learning is moving real fast, and is always coming up with faster/state-of-the-art algorithms that go a long way to outdo traditional algorithms in many real-world applications. So, it is always preferable to be updated with the latest trends. I suggest (as I normally do myself) to grab a subscription of a good newsfeed (like Medium) or an amazing, cutting-edge research journal and follow it. Many times amazing algorithms come from research papers tackling a particular problem (probably similar to yours).

The point is, to be a good data-scientist (or a ML engineer), you need a mix of both- depth and width. I personally find it useful to know a lot of algorithms on their surface (simply what they do, when are they used, pros and cons). I return to them when I feel (only feel) they might help me solve a particular problem. I read them in detail and see if they are a good fit. They might, or they might be not. But thinking about the details is essential to ensure you don't miss out on an amazing approach to your problem due to lack of insight into that approach. For instance, once I was working on something that required Object detection (very simple though). I read somewhere about R-CNN, Fast-CNN, YOLO. I immediately turned to them to see if they fit well. That day I knew them in more detail.

Whether I should focus more on learning the intricacies of a few algorithms or should I use the approach of knowing a lot of them as and when and as much as required?

Learning the intricacies is amazing. However, the world moves at a real fast pace. There might be a new algorithm that outdoes the one whom you learnt with great detail. It's time, hence, to flush out that use and see if the new one does you more good.

Learn things when needed. And when needed, learn them in detail. You should be able to apply things if you feel they can probably work. And this insight comes from knowledge.

Good luck.