I am trying to build a neural network which takes a picture of a FOREX chart (currency exchange) and outputs an "up" or "down" prediction. I'm classifying up or down based on where the stock is after one hour, and currently I have 400 examples in each class. However, my training attempts are stuck at around 49-51% accuracy. I've come up with a few potential reasons:
1) A bad model. I don't think this is the case - I've tried transfer learning on Inception V3 and two of my own models. However, is there a different type of model that could somehow account for the randomness of markets?
2) Not enough data: I think this is the most likely explanation. From what I've read, you need thousands of images, not a few hundred, for good image classification.
3) The problem is impossible. I chose FOREX over stocks because FOREX isn't as heavily affected by headlines during the span of a few hours. Most FOREX trading in that time frame is done by technical analysis, which is doing things with charts - trend lines, moving averages etc.
I know that two is the most likely explanation, but before I go out and spend days collecting more data, I would like a bit of feedback to ensure that this is actually possible.