My task is to identify potential situation for trading and determine whether a candidate is going to succeed or not. I have a system in place to identify candidates, but there is a high rate of false positives.
To try to reduce it, I have been training a Random Forest in the hopes of pre-emptively eliminating high risk candidates. The problem is that I am using two separate sources of data: one which is a 10-year historical record, and the other which is obtained in real-time and stored in a separate database. If I split the historical data into training and testing (let's say 0.7 - 0.3 split), the results are improbably perfect: no mistakes in the classification for categories with 100's of observations. However, when I try to apply the trained model to the observations from the real-time data, the labelling accuracy is about 50%. This has presisted, despite attempts to tweak hyperparameters and introduce new features.
How could I investigate if there are substantial differences between the data from the two sources, and how could I try to go about ameliorating this situation?