Have you ever had a situation where your model backtested with very good with historical data, and you also felt that your model was very logical? But when put it into practice case to predict the future, the results from your model are very wrong?
For example: You create a model to predict the credit scores of your customers to make lending decisions. The backtest results with historical data are very good. You lend to a guy with a high credit score. But unfortunately his daughter had an accident and he defaulted. Also there are many other customers with good credit scores from your model. But they met many misfortunes and defaulted on their debts. Does this mean your model has problems?
The future is unpredictable and there are billions of possibilities, sometimes thing with low odd still happens, causing your model's predictions to be wrong, even your model look very good logicaly. In short, your model's prediciton result wrong because of unlucky, and your boss only care about the final result.
Does it means that Data Science depend all on luck?