Why training model give great result but real data gives very bad result: Azure ML Studio

Question

I am using Two-Class Boosted Decision Tree to train model.

Evaluation result I'd say really good.

But when I am using real dataset - the result is very bad.

What can possibly go wrong that makes such huge difference?

Below is the screenshot of my model:

Two Class Boosted Decision Tree parameters (default):

Answer 1

Your question is not clear. There's 2 ways to understand it. Which dataset did you use to train your model?

1. You trained and tested on a premade dataset. The result is great. Then you applied this model to real dataset and the result is really bad.

If this is the case, you should retrain on your real dataset or apply some Transfer Learning techniques to your current model.

2. You trained and tested on a premade dataset. The result is great. Using the same model, you trained and tested on real dataset but the result is much worse.

I can't tell exactly the reason for this. Normally, real data is much more noisy. Did you handle missing data and do some feature engineering before training?