If I get 95+ % accuracy in normal models, should I still consider Ensemble models? Why should I choose Ensemble models over normal models?
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ It pretty much depends on your data and your goal. $\endgroup$– tagomaDec 20, 2017 at 9:12
-
2$\begingroup$ If you take part in Kaggle or any other competition, then any tiny fraction of better score is useful. So, you can choose for competition very complex model with slightly better score over simple and fast model with almost the same score. For real life it's usually not the case. Competition winner solutions not often will be useful for real business cases. $\endgroup$– CrazyElfDec 20, 2017 at 10:25
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
8
Firstly, welcome to the site!
When do we use Ensemble model?
when there are 2 models which perform moderately then we combine their results to get a model which performs better. In your scenario you already have a model which gives you good results what is the point of implementing Ensemble Models?
As @Tagoma said, it depends on your data and your goal. For example, you are trying to predict stock rates, every 0.01% matters. In such scenarios you need to use complex algorithms to maintain balance on that slim line i.e., not to over-fit, not to over-train and just predict.
Measures to check if your model is over-trained is by giving some random data and see how it performs, add some noise in the training data.
One more important thing to do is, to check for the Predictor Importance and see if there is any highly correlated feature with the target variable. For example, you are planning to predict age and if you have DOB as a feature in that yes of course you would predict with 99.99% accuracy but that is not what we use ML for.
If all of them are satisfied and achieving that accuracy then it means that your model's performance is good.
Finally, Implementation of Ensemble is dependent on your business problem and your understanding on business.
-
$\begingroup$
Generally, the threshold which we keep is 90-95%, if we are crossing that threshold, it is most likely that the model is over-trained or data is over-fitting.Don't spread the misconception if you don't understand what you are saying!! $\endgroup$– enterMLDec 21, 2017 at 5:30 -
$\begingroup$ @Nain: can you please explain what you meant? do you mean that what I said is wrong? if so do explain it, will correct myself but that doesn't meant that I'm spreading something wrong! $\endgroup$– Toros91Dec 21, 2017 at 5:38
-
$\begingroup$ There is no defined threshold. It's normal to achieve 98% accuracy on a task. It actually depends on the problem and the type of the data. 98% is no indication that the network is actually overfitting $\endgroup$– enterMLDec 21, 2017 at 9:27
-
$\begingroup$ In my case when I was working with a model and got 96% accuracy and when we re-checked it, it was because of over fitting. It is better if you check twice before deciding. So, based on my experience I was suggesting him to do so. But thank you will keep that in mind. I've removed that statement too $\endgroup$– Toros91Dec 21, 2017 at 9:33
-
$\begingroup$ That's why we always keep validation and test sets to cross validate it but that doesn't mean it's bad $\endgroup$– enterMLDec 21, 2017 at 9:34
$\begingroup$
$\endgroup$
It is a problem specific and data specific problem. For one problem, 95% accuracy may be good, for some other it might not be very good and there might still be room for improvements. Likewise, it might happen that the problem is well defined, your architecture is good but due to a lot of noise in the data, you can't do better than 95%. In that case, we will say the results are good.
Ensembles: Ensembling is a general term for combining many classifiers by averaging or voting. It is a form of meta-learning in that it focuses on how to merge results of arbitrary underlying classifiers. Generally, ensembling improves the final results but again, not necessarily. If you are ensembling two highly correlated models, your ensembled model will give almost the same result as your stand-alone models.
$\begingroup$
$\endgroup$
The answer to your question is only possible through data analysis and the task.
1. Data Analysis: If you are trying to classify data with 38 samples as A and 2 samples as B, a Majority Vote Classifier (MVC) might be good enough to get 95% accuracy too. That's where you might want to check balanced accuracy or F-1 scores. Here, the MVC may give you 95% accuracy like your normal models, but its F-1 score will be 0.
2. Task: If you are trying to predict multi-class labels and are getting 95% accuracy, your results might be good. But if your task is like something described here, the model could be over-fitting and you might be missing the key output you are looking for, i.e. accurately predict the rare example.
Suggestion
I would suggest determining the loss function might be very useful. Ensemble methods usually help if they are trying to complement each other (as you might have seen in various Kaggle competitions). Also, you can look at Gradient Boosted Trees and XGBoost which create multiple small decision trees, with each new tree trying to predict the previous tree's incorrect result.
