4
$\begingroup$

I have a dataset with 3075 unique classes (they represent rubric IDs that can be assigned to an order to categorize it into this or that rubric).

Some rubrics were only assigned once meaning that there is only one row with this given rubric ID in the dataset, and because of that I could not do a stratified split. So i have duplicated ONLY those rows that had unique rubric IDs.

After that I ran:

df_X_train, df_X_test, y_train, y_test = train_test_split(df, df['rubric_id'], test_size=.2, random_state=42, stratify=df['rubric_id'])

After that running:

df_X_train['rubric_id'].value_counts()

produces a Series of correct length 3075, confirming that I have all rubrics IDs in the train split (even though some of them appear there only once, which is OK since I'm oversampling later).

But running:

df_X_test['rubric_id'].value_counts()

produces a Series of length 2560 and I do not understand this number.

Why were not all rubrics included into the test split?

$\endgroup$

1 Answer 1

3
$\begingroup$

Answer to your exact question: I assume that you duplicated these unique-class instances only once, right? If so, the full dataset contains 2 instances for each of these classes. Since I see that the test set is made of 20% of the instances, then it's normal that there's not always one such instances in the test set: even stratified, 20% of 2 instances is only 0.4 instances in average in the test set, so it can be zero.

More importantly, I think that your design is flawed: these single-class instances are irrelevant and should be removed before training the model. The training data is supposed to contain a representative sample for every class, and one instance is never a statistically representative sample. Moreover, the performance is completely biased for these cases since the test set contains a single instance which is a duplicate of the single training instance. This can in turn bias the overall evaluation, especially if macro-average of any kind is used.

Normally classes which are rare (for instance $N\leq 5$) should be removed, they can only cause overfitting in the model. The goal of a model is to recognize patterns which are sufficiently frequent, not extremely rare events. it's already hard enough for the model to distinguish between many classes: keep in mind that a baseline model would only reach 0.001 accuracy with 1000 possible classes.

$\endgroup$
5
  • 1
    $\begingroup$ Erwan, thank you for a detailed answer! Yes, I did duplicate those unique rows only once for each such row. I'm actually on the fence myself regarding these unique rows and share your concern of overfitting the model. So I'm planning to train and test the model in two scenarios: with say N<5 and without N<5 and then compare the metrics. $\endgroup$
    – ruslaniv
    Jul 14, 2022 at 8:03
  • $\begingroup$ Another thing is that it seems my understanding of train_test_split is incorrect. I thought that if I stratify the train test split and some classes are represented only in two instances then the stratify parameter will make sure that one instance in present in train set and another one in test set. I kind of deduced that from the fact that stratified split will fail if there is only one instance of a class in the dataset. But it looks like it is not the case. $\endgroup$
    – ruslaniv
    Jul 15, 2022 at 7:49
  • $\begingroup$ Removing rare classes can make the model misunderstand the remaining data. E.g. if you have 6 examples where a>100, and it gives class 1 once, 5 three times, and 8 twice, but you remove the data for 5 and 8 because their N<5, then the model now thinks a>100 is a 100% certainty that it will be class 1. If a is an organ function measurement, class 1 is "common cold - no treatment needed", and classes 5 and 8 are rare diseases... well, you get my point. $\endgroup$ Jul 21, 2022 at 9:56
  • $\begingroup$ @ruslaniv sorry, I forgot to answer. In theory stratified sampling just means that the sampling is done for every value of the variable independently. For example with 3 instances for value x and a split 0.5/0.5, then it's guaranteed to have at least one instance in each set. But if the split is 0.8/0.2, then the sampling has only probability 0.2*3=0.6 to assign one of the instances in the test set. $\endgroup$
    – Erwan
    Jul 21, 2022 at 10:24
  • $\begingroup$ @DarrenCook good point. In this idea, it's better to assign a default class to these instances which don't have enough instances. $\endgroup$
    – Erwan
    Jul 21, 2022 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.