6
$\begingroup$

The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria. Stock Screeners are then "backtested" by simulating the results of applying in over the last 10 years using Point-in-Time data. I get back the list of trades and overall graph of performance. (If that is unclear, I have an overview here and there with more details).

Now a common problem is that users create overfitted stock screeners. I would love to give them a warning when the screen is likely to be over-fitted.

Fields I have to work with

  • All trades made by the Stock Screener
    • Stock, Start Date, Start Price, End Date, End Price
  • S&P 500 performance for the same time frame
  • Market Cap, Sector, and Industry of each Stock
Improve this question
$\endgroup$
  • $\begingroup$ maybe cross validate. randomly holdout 1 month of each backtesting year. Or do 1 year(or up to today) forward testing as validation. $\endgroup$ – Xin Mar 3 '15 at 13:15
  • $\begingroup$ Unfortunately, our users got unhappy when we tried to reserve time periods for validation as they (believe) they do their own validation so need all the data :-(. Statistically speaking, is the screen more likely to be overfit if we randomly pulled months from the full set of results and checked if the screen still performed well? $\endgroup$ – Emily Crutcher Mar 3 '15 at 16:51
  • $\begingroup$ if randomly pull, the sample performance tends to have a similar mean but more volatility which considered as a worse performance in most performance measures. Don't think this has anything to do with detecting overfit. I think forward testing might be a good way to go(as Meta trader does), normally traders will do this anyway $\endgroup$ – Xin Mar 4 '15 at 11:34
  • $\begingroup$ I wish all our users were wise enough to do it that way. As you'd think telling people that the developers of the system always trade virtually before investing in a screener would be enough! :-(. I'm still hoping that someone has a few statistical tricks that will help at least a few people avoid disaster. $\endgroup$ – Emily Crutcher Mar 4 '15 at 15:40
8
$\begingroup$

Learning curves or bias-variance decomposition are the gold standard for detecting high variance, aka: overfitting. Separate your data (in your case the "back data") into 60% training data and 40% testing data. Fit the model on the training data as you usually would and see how well it is working with the test data.

Finally, when you think you have the model that you want, split each of the training and test sets into 10-100 subsets and retrain and test with incrementally larger sets. Apply your favorite performance metric and plot the results of performance vs. the number of cases used for testing and training.

The curves will never come together if the model is overfit (high variance). The curves will come together but the performance will be lower than desired if the model is underfit (high bias) and the lines will come together at an acceptable performance for a well performing model that is not overfit.

Here is an example of overfitting and underfitting with root mean square error as the performance metric: Bias-Variance decomposition via learning curves

Here is a pretty good link on the process and here is another one. Hope this helps!

Improve this answer
$\endgroup$
  • $\begingroup$ Thank you for such a clear explanation, and for the terrific links, thinking about the problem as detecting variance vs bias will definely help! $\endgroup$ – Emily Crutcher May 3 '15 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.