Lately I was working on a LSTM code to predict future stock prices. I did not get a good result from that. Also I saw a lot of articles talking about stock market data is a random walk and cant be predicted by machine learning. I also know overfitting is the biggest problem in prediction but a technical analyst uses some stock indicators. I also feed them to my LSTM but still did not get good result. I don't understand how can we predict based on technical analysis but we can't do it with machine learning and NN.
This is likely due to overfitting, and it won't be easy to combat. If you would ask a skilled technical analyst you would notice that he is not using the same features you are inputting into the LSTM. Also, he has the capability of generalizing. If you would be able to feed the same input into an LSTM then maybe you could have similar results, but I would not expect you to be able to generalize the features to that level.
It can be due to several reasons at once:
- Underfitting or overfitting your model. It is clear, I think.
- You try to forecast data for a long period of time. Financial data tends to change its behaviour from time to time, hence only short-term forecasting is suitable. Moreover, financial date depends not only on previous values, but on some other external factors. Using technical analysis you is able to predict trend movements, but not exact price values. There are several articles on arXiv.org about stocks forecasting that proves this point of view.