1
$\begingroup$

I am working with a corpus that has 5 datasets in product reviews (A, B, C, D and E), mine is a text classification problem and I need to find the best 5 top models in terms of classification performance (F1).

I started with collection A: the mp3 reviews, Because it has the largest numbers of documents (900: yes, 750: No).

I trained the data using 10-fCv using different algorithms and pre-processing tasks, got the weighted results for all experiments.

I chose the top 5 models and I want to apply them to the rest of the corpus: B, C, D and E (other products' reviews).

My plan is to run 10-fCv and get the results for all the collections and compute the Micro-average for precision, recall and F1.

Is this the right way to choose a model for a large collection?

$\endgroup$
  • $\begingroup$ Please can anyone help with this :) $\endgroup$ – user43166 Dec 12 '17 at 2:08
1
$\begingroup$

This is an interesting question.

In general the split of data is about the underlying distribution. That means you split a dataset into train-test sets in a way that random split of train-test does not dramatically affect the distribution. But Splitting based on topics is not random!

Specially in your case you are talking about text in which the distribution is super sensitive to the domain i.e. if you collect the commentary of 1000 football games and the narrations of 1000 documentary movies about wild life, you will see that they are literally two different things. The conceptual difference between products most likely affects the distribution of words/terms/phrases therefore the model trained on reviews of mp3s MUST NOT be validated on reviews of football shoes!

In your case, I would say the train-test split (CV folds) should be done on whole data together so that you maintain the original topology of word distribution (topology here is not a Math term but I simply mean the shape of distribution).

In this case if you do Topic Modeling on the whole training data you simply see 5 different product topics. Or if you use word2vec or doc2vec you hopefully see 5 different clusters. Then you can run your models in this setting for evaluation.


NOTE:

If the size of classes are very different you need to come up with some solution for small classes. If it was the case, just drop me a line in comments and we can discuss solutions.

Good Luck :)

$\endgroup$
  • $\begingroup$ My classification problem is about predicting the helpfulness of a review. It is a binary classification yes: helpful and no: unhelpful. $\endgroup$ – user43166 Dec 12 '17 at 21:14
  • $\begingroup$ The distribution of the classes are ok: mp3 (55%: yes 45%: no ) DVD player (51%: yes 49: no) cam1 (56%: yes 44% : no) printer (59%: yes 41% : no) and cell (54%: yes 46%: no). So I think there is no skew problem here. $\endgroup$ – user43166 Dec 12 '17 at 21:36
  • 1
    $\begingroup$ My original plan was to use the best top models I trained on the mp3 collection (it is the biggest collection in the corpus) and train the rest of the collections only using those top models. after getting all the results for each collection I will compute the micro-average of all the 5 collections of reviews. After reading your comments. However, after reading your comment I feel that I have to find the top models for each collection (individually) because I noticed that the initial results are not consistent. $\endgroup$ – user43166 Dec 12 '17 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.