Fewer observations & larger documents vs More observations & smaller documents

Question

Let's suppose that I have a dataset of 1000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

I am not thinking about using any word embeddings for now so I will probably use a TF or TF-IDF model (even though this may be a bit off topic for current question).

Let's suppose that in my case I split (in a stratified way) my dataset into a training set of 800 observations and into a test set of 200 observations.

My question is the following: Is it better to have 800 separate documents in my training set or to merge these documents based on its categories/labels and create 3 very big documents?

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

My question stems from the fact that in the latter case if for example I do TF-IDF then this will be applied based on different categories/labels since each document will be about a category/label.

On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

Answer 1

There 800 separate documents of any of the 3 labels or 3 big documents of each of the labels is the best way to go and why?

The first thing you need to think about in any ML problem is: what is an instance for the problem? In other words, what is going to be the input for which you want a prediction at the end of the process?

Imagine you train your model with 3 big documents, one of each label. Then the input for such a model is a big set of documents with the same label. So it can only predict a label for a set of documents sharing the same label. This means that somehow you need to have the labels before applying your model... difficult isn't it? :)

This is why in this case an instance must be a single document. It's the job of the learning algorithm to learn to discover the label based on the instances, and for that it needs many instances of each possible label.

On the other hand, if I do (as we usually do actually) like in the former case then the TF-IDF will be categories/labels-agnostic and I do not know this helps things.

This is where there is a confusion: the TF-IDF weights are not supposed to encode the label in any way, they represent the importance of a particular word in a document. The learning algorithm will use this information for all the words, that is it's going to learn the difference between when the word delicious has a high TF-IDF and when the word disgusting has a high TF-IDF (for instance).

Is the answer simply that this an interesting but pretty bad idea because in this way you simply massively decrease the number of the observations with which the model/algorithm is trained and so you make much harder for him to figure out how to successfully classify things?

That would be true as well, but the main issue is the one I mentioned above: you won't be able to provide the same kind of input when you apply your model on your unlabeled data.