1
$\begingroup$

I am using trees algorithms (decision tree, random forest and XGBoost) to forecast the sign of the returns in the stock market (classification).

I am using this article as a reference:

http://rpubs.com/raaraa/412512

When splitting the training and the test set, the author uses the following R code:

index <- sample(1:nrow(stock_indicators)) 
size=0.2*nrow(stock_indicators)
test <- stock_indicators[index, ]
train <- stock_indicators[-index, ]

Is it correct to use a random set of dates to split the test and the training set with time series financial data?

It looks like look-ahead bias to me.

This would be the alternative:

train <-head(stock_indicators,round(0.70 * nrow(stock_indicators)))
test <- tail(stock_indicators,round(0.30 * nrow(stock_indicators)))

But with this split I cannot get significant accuracies even trying with different datasets.

Do you have any suggestions?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I think what you're doing is correct, in fact it would be even more correct to introduce a gap between your test and train set, i.e.

train <-head(stock_indicators,round(0.65*nrow(stock_indicators))) 
test <- tail(stock_indicators,round(0.25*nrow(stock_indicators)))

The reason for this is time series data (usually) exhibits strong serial auto-correlation, so if you put the price for one day in your training set, and the next days price your test set they're a long way from independent and your test error is biased.

The reason you're not getting a good fit is that the data is probably not stationary, i.e. what's driving stock prices is changing over time and it's hard to forecast - if it wasn't everyone would do it perfectly.

Another way of doing the split is to split the data into blocks (say weeks) and then take 80% of the blocks for the training set and 20% for the test set. This is useful if say your data is seasonal (i.e. electricity demand), in which case using the last 20% isn't really representative of the full future you want to forecast. I'd argue stock prices aren't overly seasonal.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much David. I thought I didn't need to run any data preparation when dealing with trees. So far the only thing I'm done is labelling the difference in price with a binary variable (e.g. if P(t+30) - P(t) > 0 then y=1, otherwise y=0). For the regressors I'm using keyword search volumes for 20 words and 20 lags of these variables, for a total of 400 explanatory variables. What would you suggest to improve the performance of the model? Should I make the x's stationary (e.g. difference stationarity)? At the moment I have a pretty bad negative predicting value. $\endgroup$
    – Eaglez
    Commented Aug 8, 2019 at 0:00
  • $\begingroup$ Trees cannot extrapolate, so you probably do need to do something to make the data stationary. I've used catboost for timeseries forecasting and what worked there was to include a daily and weekly frequency sin and cos term so it can at least learn that (my data had those seasonality). For stock prices you'll probably have to difference to remove the drift as a tree cannot learn that. $\endgroup$
    – David Waterworth
    Commented Aug 8, 2019 at 2:17
1
$\begingroup$

For datetime data it is better to test the data based on the recent results you got

the learning suppose to happen through previous periods and then test it against most recent period.

Randomly selecting test data might be ok for other datasets. but it is recommended for time datasets to recent period

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.