Here's a slightly faster version (the idea is that if you have a huge number of dates to parse, it's more efficient to store the ones you've already parsed and look them up than to parse them anew each time):
def fast_dates_parse(df):
dates = {date: pd.to_datetime(date) for date in set(df[cols].values.ravel('K'))}
for i in df.columns:
df[i] = df[i].apply(lambda x: dates[x])
Let's create an example DataFrame with 10000 rows and 2 columns to evaluate performance on:
import pandas as pd
import numpy as np
df_raw = pd.DataFrame()
for i in range(2):
init = {'day': np.random.randint(1, 28, 10000),
'month': np.random.randint(1, 12, 10000),
'year': 2000+np.random.randint(1, 18, 10000)}
df = pd.DataFrame(init)
df_raw['date_'+str(i)] = df['day'].astype(str)+'/'+df['month'].astype(str)+'/'+df['year'].astype(str)
cols = ['date_0', 'date_1']
Let's try timing the baseline approach:
%timeit df_raw[cols].apply(pd.to_datetime)
This gives us:
4.67 s ± 270 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Next, let's try our new approach (note that you should remove .copy()
when using the function to change a DataFrame):
%timeit fast_dates_parse(df_raw[cols].copy())
This gives us:
1.58 s ± 88.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)