Parameter tuning for machine learning algorithms

Question

When it comes to the topic of tuning parameters, most of the time you read grid search. But if you have 6 parameters, for which you want to test 10 variants, you get to 10^6 = 1000000 runs. Which in my case would be several months of processing time.

That's why I was looking for an alternative. On the Kaggle website, I found a tutorial that uses a different approach. It almost works like this:

1) Test all 6 parameter individually with the other 5 parameters as default value and plot the results

2) Change the default values for alle 6 parameters to the best value of the associated plot

3) Test all 6 parameter individually with the other 5 parameters as last best value and plot the results

4) Repeat step 2 and 3 until the results does not change anymore

This approach has the advantage of requiring much fewer runs. Is this a scientifically accepted procedure? And does this approach have a name in the literature?

Answer 1

In general, your approach will get stuck in local minima. This is why it is not scientifically accepted. (Notice that this may be different in very special cases, in particular if the performance of the algorithm is a strictly convex function of all input parameters).

To see how the approach fails, suppose your machine learning algorithm has two parameters, $x$ and $y$, which can be either $0$ or $1$. The default values are $x=1$ and $y=1.$

The performance of your machine learning algorithm is $f$ and should be as high as possible. Assume the following performance levels $f(x,y)$:

    | x=0 | x=1 
----|-----|-----
y=0 | 0.9 | 0.2
y=1 | 0.1 | 0.3

Your approach would do the following:

1. First, choose the default value $x=1$ and compute $f(x=1, y=0) = 0.2$ and $f(x=1, y=1) = 0.3$. Second, choose the default value $y=1$ and compute $f(x=0,y=1)=0.1$ and $f(x=1, y=1) = 0.3$.
2. Change the default values to the best value. In this case, this requires no change since $x=1$ and $y=1$ are the best values, respectively.
3. The result did not change. Report $(x=1, y=1)$ as the best parameter combination.

But the global performance maximum occurs at $(x=0, y=0).$