2
$\begingroup$
cols = ['permit_creation_date', 'current_status_date','filed_date', 'issued_date', 'completed_date', 
           'first_construction_document_date', 'permit_expiration_date']

df_raw[cols] = df_raw[cols].apply(pd.to_datetime)

I wrote this code to convert some of the columns in the dataframe to convert the datatype to a datetime. When I run the cell in jupyter notebook it takes about 1 min in runtime. Any thoughts on how I could speed up the runtime.

$\endgroup$
3
  • $\begingroup$ Have you seen this: pandas.pydata.org/pandas-docs/stable/enhancingperf.html? There are lots of good ideas. I would just try a few and compare the timings. $\endgroup$ Oct 4, 2018 at 8:13
  • $\begingroup$ Why not convert them while reading the file itself? $\endgroup$
    – Aditya
    Oct 4, 2018 at 8:20
  • $\begingroup$ @Aditya maybe she/he isn't reading data from a file. $\endgroup$ Oct 4, 2018 at 8:41

2 Answers 2

1
$\begingroup$

Pass the panda series directly to pandas.to_datetime() function. It works faster than using apply function and passing pandas.to_datetime as a parameter.

In[20]: %timeit -n 1000 pd.to_datetime(mdp)
96.1 µs ± 12.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In[21]: %timeit -n 1000 mdp.apply(pd.to_datetime)
5.59 ms ± 91.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Incase of dataframe, looping through the series works fine as well.

In[45]: %timeit [pd.to_datetime(df[x]) for x in df]
230 µs ± 20.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)`

In[46]: %timeit df.apply(pd.to_datetime)
1.22 ms ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
$\endgroup$
2
  • $\begingroup$ The OP doesn't have a Series though, they have a DataFrame, so passing it directly to pd.to_datetime like that won't work $\endgroup$ Oct 4, 2018 at 9:19
  • 1
    $\begingroup$ yes. But looping through the series works fine as well.. %timeit df.apply(pd.to_datetime) 1.22 ms ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) %timeit [pd.to_datetime(df[x]) for x in df] 230 µs ± 20.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) $\endgroup$
    – chmodsss
    Oct 4, 2018 at 11:42
0
$\begingroup$

Here's a slightly faster version (the idea is that if you have a huge number of dates to parse, it's more efficient to store the ones you've already parsed and look them up than to parse them anew each time):

def fast_dates_parse(df):
    dates = {date: pd.to_datetime(date) for date in set(df[cols].values.ravel('K'))}
    for i in df.columns:
        df[i] = df[i].apply(lambda x: dates[x])

Let's create an example DataFrame with 10000 rows and 2 columns to evaluate performance on:

import pandas as pd
import numpy as np

df_raw = pd.DataFrame()
for i in range(2):
    init = {'day': np.random.randint(1, 28, 10000),
            'month': np.random.randint(1, 12, 10000),
            'year': 2000+np.random.randint(1, 18, 10000)}
    df = pd.DataFrame(init)
    df_raw['date_'+str(i)] = df['day'].astype(str)+'/'+df['month'].astype(str)+'/'+df['year'].astype(str)

cols = ['date_0', 'date_1']

Let's try timing the baseline approach:

%timeit df_raw[cols].apply(pd.to_datetime)

This gives us:

4.67 s ± 270 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Next, let's try our new approach (note that you should remove .copy() when using the function to change a DataFrame):

%timeit fast_dates_parse(df_raw[cols].copy())

This gives us:

1.58 s ± 88.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.