In most estimators on scikit-learn, there is an
n_jobs parameter in
predict methods for creating parallel jobs using
I noticed that setting it to
-1 creates just 1 Python process and maxes out the cores, causing CPU usage to hit 2500 % on top.
This is quite different from setting it to some positive integer >1, which creates multiple Python processes at ~100 % usage.
How does setting it affects CPU & core usage on a multi-CPU Linux server?
n_jobs=8 then are 8 CPUs fully locked up or do the CPUs still reserve some cores for other tasks/processes?)
Additionally, I do get
MemoryError occasionally when setting
n_jobs=-1 for large datasets.
However, the memory usage usually hovers at around 30-40 % for the single Python process.
How is the data & memory being managed/copied depending on the value of