1
$\begingroup$

I am working on bioinformatics big datasets; training set and optimisation taking huge time to execute. I check and found that training and optimisation compiling on one core of cpu and because of that reason its taking huge time.

I tried to get look for multiprocessing or parallel computing code but I didn't get any good relevant sample to code to execute on my script. I am new in multiprocessing, can anyone share the multiprocessing or parallel computing code in Python please.

Improve this question
$\endgroup$
2
  • $\begingroup$ It would be good if you could be a little more concrete with your question. Parallelism is a huge topic and is very important within Data Science and Machine Learning; so there are a lot of possible answers. You could list the tools/packages you are using, the hardware resources you have, if you might like to use cloud computing (e.g. Amazon EC2 instances), how much data you have, the type of data, and so on. $\endgroup$
    – n1k31t4
    Commented Feb 23, 2020 at 12:08
  • $\begingroup$ I am using cox regression method when script execute cph.fit(X_train, y_train) and using randomized optimation method its takes more than 7 days to execute the script. I have tried in my system Processor : Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHZ 3.50 GHz Memory (RAM) : 32.0 GB Also tried to used GPU System but didn't able get result fast. So, trying multiprocessing method to get result fast. $\endgroup$
    – DataP
    Commented Feb 23, 2020 at 15:16

2 Answers 2

Reset to default
1
$\begingroup$

If you're using scikit-learn, there is a parameter for most learners called n_jobs. This parameter can be set to -1 to utilize all processors. For more details, see scikit-learn n_jobs parameter on CPU usage & memory

Improve this answer
$\endgroup$
1
$\begingroup$

How you do this will depend a lot on the tools you are already using and the way your model trains. If you are iterating e.g. over a grid of parameters to search, it can be trivial to parallelize. Other setups can be quite difficult.

There are some generic ways to do it using Python: check out the relevant documentation.

One thing to be aware of is that you really must use multiple processes to get real parallelism in Python. Using a thread in python does not speed things up for the actual computation. Threads are typically only useful in scenarios where you think you will be waiting for responses e.g. in networking/web programming (similar to asynchronous programming in Python).

In your case, doing bioinformatics, I would first check there aren't frameworks that will already do the parallel processing for you. Maybe you could use:

  • A typical Deep Learning framework like Tensorflow or PyTorch, which do many things in parallel on the CPU and also offer GPU support, which can be even faster. You can see from all these examples using Tensorflow, that you can create many models, not just neural networks.
  • There are also other generic frameworks, such as Ray, which try to abstract the difficult parts away for you. Here is an introduction tutorial for that tool.
  • Sci-Kit Learn's tools, some of which have an options to add more workers or can be used together with joblib.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.