-3
$\begingroup$

Have you ever had a situation where your model backtested with very good with historical data, and you also felt that your model was very logical? But when put it into practice case to predict the future, the results from your model are very wrong?

For example: You create a model to predict the credit scores of your customers to make lending decisions. The backtest results with historical data are very good. You lend to a guy with a high credit score. But unfortunately his daughter had an accident and he defaulted. Also there are many other customers with good credit scores from your model. But they met many misfortunes and defaulted on their debts. Does this mean your model has problems?

The future is unpredictable and there are billions of possibilities, sometimes thing with low odd still happens, causing your model's predictions to be wrong, even your model look very good logicaly. In short, your model's prediciton result wrong because of unlucky, and your boss only care about the final result.

Does it means that Data Science depend all on luck?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Data science and statistics are all about dealing with uncertainty (a.k.a "luck"). If your model was wrong after only a few cases, there's not enough data to conclude (whether your boss understands this or not). If it was wrong over a large number of cases, then it's not "luck", there was some error in the model and/or there was a major change in circumstances which made the model invalid (e.g. economic crisis). $\endgroup$
    – Erwan
    Mar 8 at 16:10
  • $\begingroup$ "your model backtested with very good with historical data... when put it into practice case to predict the future, the results from your model are very wrong" To me it sounds that the problem is your model, aka the you factor, not else. $\endgroup$
    – lpounng
    Mar 10 at 4:25

1 Answer 1

Reset to default
3
$\begingroup$

All models are wrong, some are useful. The most useful ones are the ones constructed by people who understand the probabilities of things happening, even "black swan" events (things everyone else thinks can't happen), this is a skill based on knowledge and experience.

Its not "luck", which is an unscientific concept.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.