I am trying to remove seasonality and trends from my time series data. I found this post that said to use df_diff = df.diff().diff(12).dropna() (https://www.tobiolabode.com/blog/2020/12/30/how-to-convert-non-stationary-data-into-stationary-for-arima-model-with-python). I don't understand why we need to use diff(12) from pandas. Could someone explain this to me? Thanks!

  • $\begingroup$ Before copying any code, how do you know your data has some sort of seasonality? Can you show us? I would also recommend reading what diff does and finally Google ARIMA and in particular Autocorrelation. $\endgroup$
    – fswings
    May 23 at 22:15

Since the data is recorded every month (i.e. a data point for each month in the year) and we see a yearly seasonality trend we compare the data for the same month against previous year. This is done by taking the difference against the data from last year (.diff()), which is equal to going back 12 observations since each observation is data for a specific month.

  • $\begingroup$ So are you saying (1) df_diff_1=df.diff() takes the difference against the data from last year, then what's the purpose of doing the second step (2) df_diff_2 = df_diff_1.diff(12)? $\endgroup$
    – user118151
    May 22 at 16:31
  • $\begingroup$ The .diff() method in general takes the difference between the current row and some previous row (depending on the value provided for the periods argument). Since there is a yearly seasonality we want the difference compared to 12 months (which is 12 observations in this case) back, and therefore use the value of 12 for the periods argument and use .diff(12). $\endgroup$
    – Oxbowerce
    May 22 at 16:41
  • $\begingroup$ Does this first differencing include trends and seasonality removal? $\endgroup$
    – user118151
    May 22 at 17:09
  • $\begingroup$ First differencing is used to remove the trend, after that another difference is taken for 12 periods based on seasonality pattern. See also this page which shows the same but explicitly splits the two steps. $\endgroup$
    – Oxbowerce
    May 22 at 17:45
  • $\begingroup$ Okay. After reading the link you sent, I'm confused a little. If we want to remove the yearly seasonality, why don't we just do df.diff(12)? Why do we have to initially call diff() before diff(12)? $\endgroup$
    – user118151
    May 22 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.