1
$\begingroup$

Let's suppose that I have a dataset of 1000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

I am not thinking about using any word embeddings for now so I will probably use a TF or TF-IDF model (even though this may be a bit off topic for current question).

Let's suppose that in my case I split (in a stratified way) my dataset into a training set of 800 observations and into a test set of 200 observations.

My question is the following: Is it better to have 800 separate documents in my training set or to merge these documents based on its categories/labels and create 3 very big documents?

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

My question stems from the fact that in the latter case if for example I do TF-IDF then this will be applied based on different categories/labels since each document will be about a category/label.

On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

$\endgroup$
1
$\begingroup$

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

The first thing you need to think about in any ML problem is: what is an instance for the problem? In other words, what is going to be the input for which you want a prediction at the end of the process?

Imagine you train your model with 3 big documents, one of each label. Then the input for such a model is a big set of documents with the same label. So it can only predict a label for a set of documents sharing the same label. This means that somehow you need to have the labels before applying your model... difficult isn't it? :)

This is why in this case an instance must be a single document. It's the job of the learning algorithm to learn to discover the label based on the instances, and for that it needs many instances of each possible label.

On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

This is where there is a confusion: the TF-IDF weights are not supposed to encode the label in any way, they represent the importance of a particular word in a document. The learning algorithm will use this information for all the words, that is it's going to learn the difference between when the word delicious has a high TF-IDF and when the word disgusting has a high TF-IDF (for instance).

Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

That would be true as well, but the main issue is the one I mentioned above: you won't be able to provide the same kind of input when you apply your model on your unlabeled data.

$\endgroup$
  • $\begingroup$ So in a concise way what you are saying is that it is much better to train your model on observations of the same "form" (e.g. of the same number of words approximately) as your test data will be? And except for that you do not see anyother principal problem etc? You can also see and possibly answer a similar question of mine (datascience.stackexchange.com/questions/52929/…) which I posted a bit ago. $\endgroup$ – Poete Maudit May 30 at 17:59
  • $\begingroup$ It's not only better, it's the only way which makes sense. As an experiment you can try both, you will see that one of the models is just useless. $\endgroup$ – Erwan May 30 at 18:04
  • $\begingroup$ I am not saying no this (I think this too at the end) but I am asking why is this exactly)? (By the way, upvoted your answer) $\endgroup$ – Poete Maudit May 30 at 18:05
  • $\begingroup$ So as you said the difference was massive (with using tf-df and random forest at least). In the case where I split the documents in multiple smaller documents then in my classification project the accuracy was 94% whereas in the case where I did not split the text then it was 0%. $\endgroup$ – Poete Maudit Jun 3 at 13:05
  • $\begingroup$ However, to be honest I would like a theoretically well-founded explanation of why this happens. My intuition, as yours, was that something like this would happen but I just want to a good explanation of that. $\endgroup$ – Poete Maudit Jun 3 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.