7
$\begingroup$

If I know the time of a given validation with set values, can I estimate the time GridSearchCV will take for n values I want to cross-validate ?

$\endgroup$
5
$\begingroup$

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through grid search.

E.g. suppose you want to use a grid search to select the hyperparameters a, b and c of your pipeline.

params = {'a': [1, 2, 3, 4, 5],
          'b': [1, 2, 3, 4],
          'c': [1, 2, 3]}

cv = GridSearchCV(pipeline, params)

By default this should run a search for a grid of $5 \cdot 4 \cdot 3 = 60$ different parameter combinations. The default cross-validation is a 3-fold cv so the above code should train your model $60 \cdot 3 = 180$ times. By default GridSearch runs parallel on your processors, so depending on your hardware you should divide the number of iterations by the number of processing units available. Let's say for example I have 4 processors available, each processor should fit the model $ 180 / 4 = 45$ times. Now, if on average my model takes $10 sec$ to train, I'm estimating around $45 \cdot 10 / 60 = 7.5min$ training time. In practice it should be closer to $8min$ due to overhead.

Finally, because some parameters heavily affect the training time of that algorithm, I would suggest using the max_iter argument whenever available so that your estimation doesn't fall far off.

$\endgroup$
  • $\begingroup$ thanks ! I'll try that out and see how long it takes. I think the average time was 0.4s that I had to run some thousands of time so it did take quite some time. I'll be able to estimate it better :) $\endgroup$ – NaturalFrequency Mar 25 '18 at 7:27
  • $\begingroup$ By default number of jobs (n_jobs) that GridSearchCV runs is 1. In case you want to use more one CPU at a time you should set n_jobs=-1 or n_jobs=<num_parallel_jobs_desired>. This may need extra memory as per documentation if the dataset is big and you may have to use pre_dispatch parameter. $\endgroup$ – smishra Aug 19 '18 at 14:12
  • $\begingroup$ I have 3 parameters with 10 levels to scan and the time for a run is about 19 seconds. Hence, 10*3*19=570/60=~10 minutes. But I definitely have to wait about 35-45 minutes. What am I doing wrong (either in time calculation or in scanning params?) $\endgroup$ – Ben Oct 11 at 14:34
  • $\begingroup$ Probably some combination of parameters severely slows down your computation. You can try setting verbose=1 so that it prints your progress. $\endgroup$ – Djib2011 Oct 11 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.