I am wondering if it is possible to enable multiprocessing in KNIME Python nodes?
This is not working:
import multiprocessing as mp
from multiprocessing import Pool
def parallelize_dataframe(df, func):
df_split = np.array_split(df, num_partitions)
pool = Pool(num_cores)
df = pd.concat(pool.map(func, df_split))
pool.close()
pool.join()
return df
if __name__ == '__main__':
# the preprocess function does some stuff of string manipulation,
# and could be any other function
output_table = parallelize_dataframe(input_table, preprocess)
Pool
frommultiprocessing
, but how are you using thePool
? If you include more code in the#code .....
section, we might be able to give you a better answer. $\endgroup$