2
$\begingroup$

There is a classification problem(two classes). We have train data, for which we know class labels and we have test data.

Imagine, that you have created model that with good accuracy(~95%) make predictions and we know that we are not overfitted.

If we make prediction on test data, extract objects for which we sure in class label(for example, predict_proba higher than 90%) and add this objects to train data.

Does this tactic make any sense?

$\endgroup$
  • 1
    $\begingroup$ You have labeled training data, but unlabeled test data? How exactly are you testing? Is this a Kaggle competition or similar? $\endgroup$ – Neil Slater Mar 28 '16 at 20:35
  • $\begingroup$ You will have to elaborate more on this -- What is your rationale for adding records to the training data? What do you intend to do after adding these records to the training data? $\endgroup$ – Vishal Mar 29 '16 at 0:22
4
$\begingroup$

This idea will most likely increase the bias in the model. Let's assume that the model has non-zero bias in the model. In this case, when it assumes its predictions to be true, without confirmation from an Oracle as in active learning, the bias of the model increases. In common terms, if the model has some amount of bias in its predictions, and it uses its predictions to learn on, the bias in the model can only increase. This issue does not arise when there is 0 bias in the model to begin with, however, in that case, there is no need to learn any further!

Note that this is a highly intuitive answer but I cannot think of an argument against the intuition :-) I will appreciate any feedback on this.

$\endgroup$
  • 2
    $\begingroup$ Thank you. Your intuition was right:) I compete in inclass Kaggle competition and this solution perform worse because of increased bias. $\endgroup$ – Acapello Apr 17 '16 at 12:14
  • $\begingroup$ Thanks for experimenting. I will remember and cite this as an example if this topic ever comes up again :-) $\endgroup$ – Nitesh Apr 19 '16 at 19:36
0
$\begingroup$

You haven't stated your reason for wanting to do this, nor what algorithm you are using, both of which will affect whether your proposed tactic "makes sense". I'll give you a couple reasons why the tactic may be a Bad Idea.

First, data are divided into training and testing data for a reason: so you have a sense for how the classifier will generalize (i.e., to estimate it's true classification accuracy). Once you start polluting your training data with the testing data, you should have less confidence in the classifier's generalized performance. On one hand, I would expect that adding high confidence testing observations to the training data would increase training accuracy. On the other hand, your testing data will now have a larger proportion of low confidence observations, which will reduce your testing accuracy. So how would you judge the ability of the classifier to generalize?

When you add the high confidence test observations to your training set, you are teaching the classifier something it already knows. Depending on your classifier, there is a good chance that by doing this your are biasing the classifier farther away from the outliers (the 5% initially misclassified) and by doing so, reducing the true (general) classification accuracy. In fact, there are algorithms designed to do just the opposite of what you are proposing: they instead place more significance on the misclassified (low probability) observations to improve the accuracy of the classifier. For an example of this, take a look at boosting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.