When building a model in Machine Learning, it's more than common to have several "parameters" (I'm thinking of real parameter like the step of gradient descent, or things like features) to tune. We validate these parameters on a validating set.
My question is: what is the best way of tuning these multiple parameters? For example, let say we have 3 parameters A, B and C that take 3 values each:
- A = [ A1, A2, A3 ]
- B = [ B1, B2, B3 ]
- C = [ C1, C2, C3 ]
Two methods come to my mind.
Vary all the parameters at the same time and test different combinations randomly, such as:
- Test1 = [A1,B1,C1]
- Test2 = [A2,B2,C2]
- Test3 = [A3,B3,C3]
- Test4 = [A1,B2,C3]
- Test5= [A3,B1,C2]
Fix all the parameters except one: - TestA1 = [A1,B1,C1] - TestA2 = [A2,B1,C1] - TestA3 = [A3,B1,C1] In that way, we can find the best value for parameter A, then we fix this value and use it to find the best value for B, and finally the best for C.
It seems more logical to me to use the method 2 which seems more organized. But may be we will miss a combination which can be found only in method 1 that doesn’t appear in method 2, such as [A1,B2,C3] for example.
Which method is the best? Is there another method more accurate for tuning multiple parameters?
Thanks in advance.