1
$\begingroup$

My task is to identify potential situation for trading and determine whether a candidate is going to succeed or not. I have a system in place to identify candidates, but there is a high rate of false positives.

To try to reduce it, I have been training a Random Forest in the hopes of pre-emptively eliminating high risk candidates. The problem is that I am using two separate sources of data: one which is a 10-year historical record, and the other which is obtained in real-time and stored in a separate database. If I split the historical data into training and testing (let's say 0.7 - 0.3 split), the results are improbably perfect: no mistakes in the classification for categories with 100's of observations. However, when I try to apply the trained model to the observations from the real-time data, the labelling accuracy is about 50%. This has presisted, despite attempts to tweak hyperparameters and introduce new features.

How could I investigate if there are substantial differences between the data from the two sources, and how could I try to go about ameliorating this situation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

There are several things you can do, even though giving a detailed descripton of your datasets would help a lot:

The first thing is that you should is to thoroughly compare the two datasets:

  • Are the same features present in both datasets?

  • Is the data collected in the exact same way? As an example, if age is a feature make sure that it is not recorded differently. Let's say one dataset could use the categories (<25, 25-50, 50+) and the other the exact age.

  • How do they differ in size?

  • Plot the distribution of each feature in both dataset: do they differ significantly, and why? You could test the hypothesis that a feature is drawn from the same distribution in both datasets.

  • Look for correlated features: correlated with each other and with the target distribution

The amelioration of your algorithm will entirely depend on the answer to these questions. If you observe significant differences between the two datasets then your historical data will not be a good training set for this task.

Independently from that, your 100% accuracy is very suspicious and your algorithm is probably overfitting.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. I'm going to add an anonimized example next time I'm in the office. In short, the features are the same, and calculated in the same way. In terms of size, the historical source has data for 10 years of data collected and grouped every minute, while the real-time only started in November and gets updated several times per second (more recently whenever there's a change), so it contains about 6 months' worth of data now. Going to give the Statistical Analysis a go in the meanwhile. $\endgroup$
    – TST
    Mar 13, 2020 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.