2
$\begingroup$

I'm doing text classification to identify 'attacks' from Wikipedia comments using a simple bag of words model and a linear SVM classifier. Because of class imbalance, I'm using the F1 score as my error measure. I'm wondering if the tokens I have in the training data should also include words that exist only in the test data, or does it not matter? I was under the impression that it shouldn't matter since the counts for these features would be zero anyway in the training set. That should make them irrelevant to the model when training. Apparently, that's what some people on SO were saying as well (it didn't find any definitive answer though).

In order to test this, I decided to train my model both ways and see the difference: once with only features in the training data, and another with features that included test data. N folds for CV were set to 10. I got a very similar CV error for both of them, but when I generated predictions for my test data my F1 score was 0.06 higher for the model which included features from test data - 0.64 vs 0.58. Because this is a Kaggle assignment I cannot see the true labels for the test set. I'm inclined to believe that such a big difference can't simply be random. It seems like including all the features from test data did improve my model, but how could this be? Can anyone give me an explanation?

Improve this question
$\endgroup$
5
  • $\begingroup$ I wrote an answer but it's not clear to me what causes the difference in performance. What is the proportion of the majority class? Also what is the performance on the training set, if you can obtain it? $\endgroup$
    – Erwan
    May 22, 2021 at 16:22
  • $\begingroup$ Thanks for the answer Erwan. I have 80% negative class. Training set performance was very similar at about 0.63 to 0.64 (I'm assuming you meant my CV error. If you mean predictions on training I will have to compute that separately). I was thinking whether it could be due to the elevation in dimensions by adding more features? SVM handles sparse matrices pretty well but I'm not sure whether having a larger, sparser matrix for training is known to improve performance $\endgroup$
    – jonnyf
    May 23, 2021 at 12:52
  • $\begingroup$ ok thanks. I also thought about the matrix dimension but I can't think of any reason why it would have this effect. at least I can say that no, it's not known to improve performance (if anything rather the opposite in general since it makes the model more likely to overfit). Maybe on thing you could try is to train the model with a random subset of the training instances, e.g. removing a few instances randomly. The goal would be to observe whether the model is stable: normally removing a few instances should not cause a big difference, so if the perf is very different it would be... $\endgroup$
    – Erwan
    May 23, 2021 at 14:38
  • $\begingroup$ ... an indication that there's too much variance (probably due to overfitting) and that would be partial evidence that the difference with/without these test features is just due to chance. I'm not sure but it might help understand what happens. $\endgroup$
    – Erwan
    May 23, 2021 at 14:40
  • $\begingroup$ Can you provide (some of) your code, and a link to the kaggle dataset? $\endgroup$
    – Ben Reiniger ♦
    Apr 2, 2022 at 0:11

1 Answer 1

Reset to default
0
$\begingroup$

This is the problem of out of vocabulary (OOV) words.

As a rule, the training should not use anything from the test set for several reasons:

  • The risk of data leakage, which would cause an overestimated performance on the test set.
  • During training the model cannot use these words to distinguish between classes anyway, since they are not present. So it would be pointless to include them.
  • In principle, the model is meant to be used with any new input text, not only the documents in the current test set. Using the test set words as features in the model actually restricts the model, since it cannot be applied to any other text than the test set.

The correct way to deal with OOV words is either:

  • To simply ignore them completely, i.e. filter them out before applying the model.
  • To account for this possibility in the model from the start. Typically this can be done by having a special token UNKNOWN. This option is often combined with filtering out rare words from the training set: every occurrence of rare words can be replaced by the UNKNOWN token.

The cause of the higher performance when including these features is unclear. I suspect that the inclusion of these features causes the model to be different in a way that happens by chance to have a positive effect on the test set. It's quite unlikely in general, I'm not sure.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.