1
$\begingroup$

I am wondering if it is possible to enable multiprocessing in KNIME Python nodes?

This is not working:

import multiprocessing as mp

from multiprocessing import Pool

def parallelize_dataframe(df, func):
   df_split = np.array_split(df, num_partitions)
   pool = Pool(num_cores)
   df = pd.concat(pool.map(func, df_split))
   pool.close()
   pool.join()
   return df


if __name__ == '__main__':
   # the preprocess function does some stuff of string manipulation, 
   # and could be any other function
   output_table = parallelize_dataframe(input_table, preprocess)
$\endgroup$
2
  • $\begingroup$ Welcome to the site! You showed that you imported the Pool from multiprocessing, but how are you using the Pool? If you include more code in the #code ..... section, we might be able to give you a better answer. $\endgroup$ Aug 10, 2018 at 18:54
  • $\begingroup$ You are right! I included some code for better understanding of my question. The code above is working in standard python but not in Python KNIME nodes. Any solutions or workarounds available? $\endgroup$ Aug 11, 2018 at 11:35

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.