Ideas on speeding up code

Question

cols = ['permit_creation_date', 'current_status_date','filed_date', 'issued_date', 'completed_date', 
           'first_construction_document_date', 'permit_expiration_date']

df_raw[cols] = df_raw[cols].apply(pd.to_datetime)

I wrote this code to convert some of the columns in the dataframe to convert the datatype to a datetime. When I run the cell in jupyter notebook it takes about 1 min in runtime. Any thoughts on how I could speed up the runtime.

Answer 1

Pass the panda series directly to pandas.to_datetime() function. It works faster than using apply function and passing pandas.to_datetime as a parameter.

In[20]: %timeit -n 1000 pd.to_datetime(mdp)
96.1 µs ± 12.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In[21]: %timeit -n 1000 mdp.apply(pd.to_datetime)
5.59 ms ± 91.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Incase of dataframe, looping through the series works fine as well.

In[45]: %timeit [pd.to_datetime(df[x]) for x in df]
230 µs ± 20.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)`

In[46]: %timeit df.apply(pd.to_datetime)
1.22 ms ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Answer 2

Here's a slightly faster version (the idea is that if you have a huge number of dates to parse, it's more efficient to store the ones you've already parsed and look them up than to parse them anew each time):

def fast_dates_parse(df):
    dates = {date: pd.to_datetime(date) for date in set(df[cols].values.ravel('K'))}
    for i in df.columns:
        df[i] = df[i].apply(lambda x: dates[x])

---

Let's create an example DataFrame with 10000 rows and 2 columns to evaluate performance on:

import pandas as pd
import numpy as np

df_raw = pd.DataFrame()
for i in range(2):
    init = {'day': np.random.randint(1, 28, 10000),
            'month': np.random.randint(1, 12, 10000),
            'year': 2000+np.random.randint(1, 18, 10000)}
    df = pd.DataFrame(init)
    df_raw['date_'+str(i)] = df['day'].astype(str)+'/'+df['month'].astype(str)+'/'+df['year'].astype(str)

cols = ['date_0', 'date_1']

Let's try timing the baseline approach:

%timeit df_raw[cols].apply(pd.to_datetime)

This gives us:

4.67 s ± 270 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Next, let's try our new approach (note that you should remove .copy() when using the function to change a DataFrame):

%timeit fast_dates_parse(df_raw[cols].copy())

This gives us:

1.58 s ± 88.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)