Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
There is a classification problem(two classes). We have train data, for which we know class labels and we have test data.
Imagine, that you have created model that with good accuracy(~95%) make predictions and we know that we are not overfitted.
If we make prediction on test data, extract objects for which we sure in class label(for example, predict_proba higher than 90%) and add this objects to train data.
Does this tactic make any sense?
This idea will most likely increase the bias in the model. Let's assume that the model has non-zero bias in the model. In this case, when it assumes its predictions to be true, without confirmation from an Oracle as in active learning, the bias of the model increases. In common terms, if the model has some amount of bias in its predictions, and it uses its predictions to learn on, the bias in the model can only increase. This issue does not arise when there is 0 bias in the model to begin with, however, in that case, there is no need to learn any further!
Note that this is a highly intuitive answer but I cannot think of an argument against the intuition :-) I will appreciate any feedback on this.
You haven't stated your reason for wanting to do this, nor what algorithm you are using, both of which will affect whether your proposed tactic "makes sense". I'll give you a couple reasons why the tactic may be a Bad Idea.
First, data are divided into training and testing data for a reason: so you have a sense for how the classifier will generalize (i.e., to estimate it's true classification accuracy). Once you start polluting your training data with the testing data, you should have less confidence in the classifier's generalized performance. On one hand, I would expect that adding high confidence testing observations to the training data would increase training accuracy. On the other hand, your testing data will now have a larger proportion of low confidence observations, which will reduce your testing accuracy. So how would you judge the ability of the classifier to generalize?
When you add the high confidence test observations to your training set, you are teaching the classifier something it already knows. Depending on your classifier, there is a good chance that by doing this your are biasing the classifier farther away from the outliers (the 5% initially misclassified) and by doing so, reducing the true (general) classification accuracy. In fact, there are algorithms designed to do just the opposite of what you are proposing: they instead place more significance on the misclassified (low probability) observations to improve the accuracy of the classifier. For an example of this, take a look at boosting.
