5
$\begingroup$

I am attempting to improve my RNN model by making my dependent variable, a stock price, non-stationary. I am aiming to make the series stationary by removing the trend with a log transformation and then performing moving average differencing to remove noise.

I have a function that initially logs the series, to penalise the larger values and then performing rolling mean differencing on the values.

def moving_avg_differencing(col, n_roll=30, drop=False):
    log_values = np.log(col)
    moving_avg = log_values.rolling(n_roll).mean()
    ma_diff = log_values - moving_avg

My conundrum is, if I perform this differencing before my train-val-test split, I will be informing my validation and test set of mean values that precede their respective values.

If I perform the differencing after my train-val-test split, and process the transformations individually, I will have 30 NaN values before my validation and test set. This seems messy.

Is there a better approach to differencing?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I answer these questions by asking how one would use this in production. Meaning assume the model is ready and one has some series one wnats to predict, how would this data need to be transformed to be fed to the model? $\endgroup$
    – Nikos M.
    Commented Jun 20, 2021 at 12:38
  • $\begingroup$ This will answer the question. Essentially the test set is same as a dataset used in production once the model is ready. So if some model parameters need to be known to transform the input set, then so be it. That's how it will be used in production, if ever $\endgroup$
    – Nikos M.
    Commented Jun 20, 2021 at 12:40
  • $\begingroup$ This makes sense, @NikosM. I thought I may need to train-test split prior to differencing as this Is the approach I have been taught when performing Standardisation, I standardise the data sets individually. I guess, I can assume these preprocessing techniques can be approached differently. $\endgroup$
    – frantic oreo
    Commented Jun 21, 2021 at 2:23
  • 1
    $\begingroup$ @franticoreo Would it not be better to not difference the test and validation sets at all? Rather, find the predictions and then inverse transform them, and then compare the now inverted predictions to the validation/test rows. $\endgroup$
    – callmeanythingyouwant
    Commented Jul 17, 2021 at 9:39
  • $\begingroup$ @callmeanythingyouwant, if the model is trained on a differenced training split, it can be used to predict the validation split (which occurs ahead in time of the training split). So we get a prediction, on an differenced scale, corresponding to a period ahead in time of the training split. Also, the validation split will (possibly / probably) be shorter in length than the (differenced) training split. How would one use a differenced training split to inverse transform the predictions of the (shorter) validation split (without using true values of the validation split)? $\endgroup$
    – edesz
    Commented Nov 27, 2021 at 0:59

1 Answer 1

Reset to default
1
$\begingroup$

According to econometrics literature, the standard approach is to convert your data into log returns as follows: $r'(t) = log(P{t} / P_{t-1})$, where $P(t)$ is the price at timestep $t$. This improves results because it de-trends the input and is relatively stationary compared to raw prices.

There is little difference if this is performed before or after train-test split, because the log return of each row relies only on the previous row. Specifically, if you do it after split you just lose one row of data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.