3
$\begingroup$

enter image description here

blue: training loss
black: validation loss

If modelling gives me such plot, what does it imply? Is it overfitting? How should I make it better?

Code as below:

classifier = Sequential()
classifier.add(LSTM(
    input_dim=72,
    output_dim=50,
    return_sequences=True))
classifier.add(Dropout(0.20))

classifier.add(LSTM(
    100,
    return_sequences=False))
classifier.add(Dropout(0.20))

classifier.add(Dense(
    output_dim=1))
classifier.add(Activation('relu'))
classifier.compile(loss='mse', optimizer='rmsprop')

X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

#classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 10)
cls_return = classifier.fit(X_train, y_train, batch_size=300, nb_epoch=500, validation_split=0.05)
history = cls_return.history
plt.plot(history['loss'], label='Training Loss', color='blue')
plt.plot(history['val_loss'], label='Validation Loss', color='black')
plt.show()
$\endgroup$
8
$\begingroup$

Yes, this is overfitting. Financial time series exhibit many peculiarities, including heteroscedasticity, high tail risk, all kinds of seasonality and all kinds of momentum. Machine learning algorithms tend to pick up these patterns and overfit on them. To my knowledge, there is no machine learning algorithm that can compete with classical time series analysis when it comes to understanding stock prices.

Please also keep in mind that if someone were able to make predictions like 'tomorrow the stock price is going to rise by 5%', they could earn billions within a few weeks.

Edit: You ask if there is any chance of training a network that is better than random guessing. Yes, for two reasons.

First, if you try hundreds of parameter combinations and evaluate them all on the same test set, you become the victim of what might be called 'overfitting of the second degree.' Performance on the test set becomes part of your decision process because you discard models that perform poorly on the test set and keep models that perform well. You are essentially training on the test set and will predict it with 100% accuracy eventually. But use this seemingly perfect model on new, previously unseen test data, and you will be disappointed.

Second, heteroscedasticity, seasonality and momentum all introduce a little bit of 'real' predictability into your time series. If you get very lucky, your algorithm might identify these patterns and at the same time ignore all the weird residual noise. This is incredibly unlikely, and it is probably irrelevant. You should not compare your prediction accuracy to random guessing, but to a reasonable baseline model. For monthly prices of individual stocks, this could be the Fama-French three-factor model. (Daily or intraday stock prices will send you a bit deeper into the rabbit hole that is quantitative finance). I bet you five dollars that you will not be able to beat that model's performance on previously unseen data.

My advice to you: Don't clean the lion's cage on your first day as a zookeeper. Start with something nice and easy.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I understand Stock price prediction is challenging, I'm doing it to learn about LSTM rnn. Could you help me with second question that how should I reduce the overfitting looking at my model code? TA. $\endgroup$ – NightFurry Feb 25 '18 at 9:53
  • 1
    $\begingroup$ Also worth adding that many known sources of internal pattern have been discovered and exploited by professionals already, over a period of decades. Thus those patterns are now erased in recent data. New models based on sequence analysis only should be expected to find no exploitable predictions by default i.e. it is a reasonable expected result for a NN to find no short-term patterns, and thus a high loss (although OP should look at other metrics than learning algorithm loss) $\endgroup$ – Neil Slater Feb 25 '18 at 10:33
  • 1
    $\begingroup$ Trying to learn LSTM with stock price data will be leave you constantly frustrated. You basically have two choices: Overfit (as you did) or achieve poor accuracy on the training and the test set (you could accomplish this by setting nb_epoch=10 in your code). Here are some time series from Kaggle that might be more fun to analyze: Trending YouTube videos, Python questions from Stack Overflow, Global air pollution measurements. $\endgroup$ – Elias Strehle Feb 25 '18 at 10:59
  • $\begingroup$ @EliasStrehle Right, decreasing epoch count reduces overfitting at the cost of reduced train and test accuracy. It gets as poor as 50%. I was not targetting to improve it to 80% but I had intution to get it as good as 60+%. Is it possible on stock market data set? $\endgroup$ – NightFurry Feb 25 '18 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.