# Stock prediction through LSTM

blue: training loss
black: validation loss

If modelling gives me such plot, what does it imply? Is it overfitting? How should I make it better?

Code as below:

classifier = Sequential()
input_dim=72,
output_dim=50,
return_sequences=True))

100,
return_sequences=False))

output_dim=1))
classifier.compile(loss='mse', optimizer='rmsprop')

X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

#classifier.fit(X_train, y_train, batch_size = 10, nb_epoch = 10)
cls_return = classifier.fit(X_train, y_train, batch_size=300, nb_epoch=500, validation_split=0.05)
history = cls_return.history
plt.plot(history['loss'], label='Training Loss', color='blue')
plt.plot(history['val_loss'], label='Validation Loss', color='black')
plt.show()


Yes, this is overfitting. Financial time series exhibit many peculiarities, including heteroscedasticity, high tail risk, all kinds of seasonality and all kinds of momentum. Machine learning algorithms tend to pick up these patterns and overfit on them. To my knowledge, there is no machine learning algorithm that can compete with classical time series analysis when it comes to understanding stock prices.

Please also keep in mind that if someone were able to make predictions like 'tomorrow the stock price is going to rise by 5%', they could earn billions within a few weeks.

Edit: You ask if there is any chance of training a network that is better than random guessing. Yes, for two reasons.

First, if you try hundreds of parameter combinations and evaluate them all on the same test set, you become the victim of what might be called 'overfitting of the second degree.' Performance on the test set becomes part of your decision process because you discard models that perform poorly on the test set and keep models that perform well. You are essentially training on the test set and will predict it with 100% accuracy eventually. But use this seemingly perfect model on new, previously unseen test data, and you will be disappointed.

Second, heteroscedasticity, seasonality and momentum all introduce a little bit of 'real' predictability into your time series. If you get very lucky, your algorithm might identify these patterns and at the same time ignore all the weird residual noise. This is incredibly unlikely, and it is probably irrelevant. You should not compare your prediction accuracy to random guessing, but to a reasonable baseline model. For monthly prices of individual stocks, this could be the Fama-French three-factor model. (Daily or intraday stock prices will send you a bit deeper into the rabbit hole that is quantitative finance). I bet you five dollars that you will not be able to beat that model's performance on previously unseen data.

• Trying to learn LSTM with stock price data will be leave you constantly frustrated. You basically have two choices: Overfit (as you did) or achieve poor accuracy on the training and the test set (you could accomplish this by setting nb_epoch=10 in your code). Here are some time series from Kaggle that might be more fun to analyze: Trending YouTube videos, Python questions from Stack Overflow, Global air pollution measurements. – Elias Strehle Feb 25 '18 at 10:59