What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

I have a dataset of approximate 1 hundred thousand records. I want to use apply method in each of the records for further data processing but it takes very long time to process (As apply method works linearly). I have tried this in Google Colab by selecting GPU settings but still it is very slow. I also try "swifter.apply" but still it is not as efficient. Is there any way to do this?

• Please give an example of the data in one of the cells and the method you want to apply - this will affect the best way to get the job done. Also, how many are "1 lakh records"? :) – n1k31t4 Apr 16 '20 at 9:57
• You can write your function with vectorization and get the output by giving the pandas columns. please tell us a little about data and function to get a better response. – Uday Apr 16 '20 at 14:28

You can try pandarallel it works very efficiently for parallel processing. You can find more information about it here.

You should not use this if your apply function is a lambda function. Now assuming you're trying to apply it on a DataFrame called df:

from pandarallel import pandarallel

pandarallel.initialize(nb_workers=n) #n is the number of worker used for parallelization, you can leave it blank and it will use all the cores

def foo(x):
return #what ever you're trying to compute

df.parallel_apply(foo, axis=1) #if you're applying to multiple columns
df[column].parallel_apply(foo) # if its just one column


Another option you can try is using the python multiprocessing library, here you will break your dataframe into smaller chunks and run them together.

import numpy as np
from multiprocessing import cpu_count, Parallel

cores = cpu_count() #Gets number of CPU cores on your machine
partitions = cores #Define number of partitions

def parallelize(df, func):
df_split = np.array_split(df, partitions)
pool = Pool(cores)
df = pd.concat(pool.map(func, df_split))
pool.close()
pool.join()
return df


Now you can run this parallelize function on your df:

df = parallelize(df, foo)


The more number of cores you have the faster this will be!

• Hi , Thankyou so much Its help me alot. – Sweety Apr 24 '20 at 7:23
• No problem @Sweety. If it worked out for you, it would be helpful if you could accept the answer so others might find it useful in the future as well! – Samarth Apr 24 '20 at 16:54
• Hi @Samarth , I wanted to know one more thing that... Wat would be more efficient to work on such multiprocessing things .. 4 cores 32GB or 8 cores 8GB as I have not so much idea about infra of setup . So on which platform should we proceed to get better results faster.(Faster I mean taking less time). – Sweety Apr 26 '20 at 6:30