A way to Identify tuning parameters and their possible range

I am a novice in Machine Learning. But when I started learning, I figure out that all the methods have some tuning parameters and those parameters take a range of possible values. By grid searching, we identify a set of these parameters that optimize some function. But is there any way to find the possible domain of the tuning parameters? This would definitely save my time and the computer's job. In addition, some methods such as xgboost have loads of tuning parameters. Is there any way to know which one to tune and which one to leave as it is. I have been using sklearn python library.

• This is where domain knowledge comes in, if you know something about your data beforehand you can use this to decrease model selection time. Sep 25 '18 at 13:23

2 Answers

Not a complete answer, but was too long for a comment.

I always first try to see how the default parameters perform. Then from the documentation or some reading, you can see what is each parameter global influence (by influence I mean maybe increasing parameter X means complexifying the model, or parameter Y means increasing the convergence speed towards a solution). Depending on the first result you get, pick up one parameter, the one that seem to have the most influence on the model, and make it vary a bit in the way that make sense from your first results. If things improve on the validation set, keep moving the value this way, if not do the opposite. Often times you get good results without tuning every single parameter.

This is a method by hand, it is not optimal. But as you precise that you are a beginner in machine learning, I believe it is the best way to learn to "feel" what usually impact the performance of an algorithm as Xgboost and what impacts less and that therefore can be overlooked for a primary coarse tuning.

https://xgboost.readthedocs.io/en/latest/parameter.html has some nice pieces of information about what parameter impacts what. Don't hesitate to ask more precise questions about some specific parameters if you need :)

I agree with the previous comment on domain knowledge, that will certainly help. As you build experience, you will also get a "feel" for what works. Some parameters work better for NLP, other parameters are more nuanced towards image processing. That's stuff that you're only going to learn after being "in the trenches" for a while.

To build that experience, you could try to build your code in such a way so that you try multiple models, each with their own unique parameters. When I am working with a new dataset, I might create multiple loops and/or threads that each build their own model and I'll compare accuracy and loss rates across all models and then narrow down which parameters I want to adjust. That creates a little more work on your part to create this approach and then track the results, but it is a good way for you to learn about what-does-what and it will help you make better decisions in the future.