I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, whether makes sense to use my trained model to make classification on top of the new dataset. The problem is the new dataset has no assigned target value yet, so I can't calculate accuracy or another metric to check the model's performance. I assume that the model will perform well if the distribution of the new data is similar to the old one. E.g. If I train my model on Wikipedia articles, I shouldn't expect my model to work well on tweets, which are way shorter (max 160 characters) and contaminated by a big number of non-obvious and rare abbreviations.

In the case of tabular data, the task is pretty simple. I can visualize each of the features, perform a z-test, etc. What about the text data?

My idea is to leverage such quantities as:

  1. Distribution of the nb of tokens;
  2. Distribution of the length of the input strings;
  3. Number of the common words;
  4. Distribution of the percentage of OOV (out of vocabulary words);

Is there any other, more robust method?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.