1
$\begingroup$

I am confused regarding the n_jobs parameter used in some models and for CV. I know it is used for parallel computing, where it includes the number of processors specified in n_jobs parameter. So if I set the value as -1, it will include all the cores and their threads for faster computation. But this article:-
https://machinelearningmastery.com/multi-core-machine-learning-in-python/#comment-617976

states that using all cores for training, evaluation and hyperparameter tuning is a bad idea. The crux of the article is as follows:-

1.)When using k-fold cross-validation, it is probably better to assign cores to the resampling procedure and leave model training single core.

2.)When using hyperparamter tuning, it is probably better to make the search multi-core and leave the model training and evaluation single core.

But common sense says that setting n_jobs = -1 everywhere will include all cores for faster computation and hence result in less run time. Can anyone clarify?

$\endgroup$
1
$\begingroup$

This is about distributed computing: let's say that you have 100 tasks and 10 cores available. You parallelize your tasks so that each core processes 10 of them. Now let's imagine that the task involves some subtasks and internally tries to use all the cores available: at the two levels of parallelization the processes compete for the cores, causing a loss in performance because there are more pending processes than available cores. This can usually be observed with some utilities to visualize the activity of the cores.

This is why it's much more efficient to control at which level parallelization should happen, in order to minimize competition between processes. In my example in theory the top tasks could be distributed say into 5 processes while each task could still use 2 cores. However it's usually simpler and more efficient to just decide one level of parallelization: in your case either the training or the parallelization, but not both. Imho the choice is not obvious though, in particular it depends on how intensive the training process is.

$\endgroup$
6
  • $\begingroup$ The example you gave (100 tasks, 10 cores) is for parallel computing (PC) while your first words say it is about distributed computing (DC). I thought both of them were different, PC being localized to one setup while DC being more of a cluster (multiple setups) approach. Am I missing something? $\endgroup$
    – spectre
    Jul 24 at 5:19
  • $\begingroup$ Also Can I utilize my GPU to speed up my runtime? If yes then how does one go about doing that? $\endgroup$
    – spectre
    Jul 24 at 5:39
  • 1
    $\begingroup$ @spectre sorry, I'm using PC and DC to mean the same thing here. As far as I know the GPU is used only in specific cases of matrix parallel processing with deep learning, but I'm not very knowledgeable about this. $\endgroup$
    – Erwan
    Jul 24 at 9:05
  • $\begingroup$ Ok. Since you seem to know DC, what would you suggest my approach be to DC, if I were to start. I have read about Ray, Joblib and Optuna for this but still unclear about how to use them. I am learning ML at an intermediate level. $\endgroup$
    – spectre
    Jul 24 at 10:31
  • 1
    $\begingroup$ @spectre I have access to a DC cluster where I run all my heavy computations. Sorry I don't use any particular framework, I just distribute tasks myself with scripts. Usually I don't care about a model requiring a long time to train, but I need DC to train many different models for instance. I'm not really expert in this, I just try to run whatever I need to run within a reasonable time with the resources that I have. $\endgroup$
    – Erwan
    Jul 24 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.