-1
$\begingroup$

I have a dataset where I dont have label for 70% of data. So, my target is to train a model on 30% data that has label and then apply this model to get the label for the rest 70% of data. But, it is highly imbalanced and no single model performs well on this. So, I took the following approach :

Method 1

missing_df = df[df.label.isna()]
nomissing_df = df[~df.label.isna()]


#split the labeled dataset into 80-20 parts for train and validation
...... = train_test_split(nomissing_df)


while new samples are added : 
    #train a 5 Fold Random Forest on the train set
   clf = RandomForest(....)
   # code for training

   #evaluate the missing data 
   label = clf.predict_proba(missing_df)

   if label > 0.75 for all fold :
       add to positive training samples
   if label < 0.25 for all fold : 
       add to negative training samples

Method 2

missing_df = df[df.label.isna()]
nomissing_df = df[~df.label.isna()]


#NOOOOOOOOOOOOOOOOOOOOOOOO SPLIT HERE


while new samples are added : 
    #train a 5 Fold Random Forest on the train set
   clf = RandomForest(....)
   # code for training

   #evaluate the missing data 
   label = clf.predict_proba(missing_df)

   if label > 0.75 for all fold :
       add to positive training samples
   if label < 0.25 for all fold : 
       add to negative training samples

#split the full label dataset ( original + by the above process)
...... = train_test_split(final_train_df)

#train final model 
clf = RandomForest().train(....)

#evaluate on validation set

Result

As our dataset is highly skewed, I used precision as my evaluation metric. For method 1, training precision is around 80% but when evaluated at the held-out validation set (at the beginning), the score is 33%.

But for method 2, the precision score for both train and validation is 80%.

My question is why is a discrepancy in validation accuracy? Is method 2 is wrong or any chance of data leakage here? Or any suggestion to improve the precision of method 1 as I think method 1 more correct way to do this? Finally, is this approach of labeling data is correct or there is any other way to do this better?

$\endgroup$

1 Answer 1

2
$\begingroup$

Method 1

  • This looks like the correct way to go assuming you're not using the predicted labels of the missing data in your model validation set. The train,test and validation sets should come from nomissing_df only.
  • If your labels are highly imbalanced it might be good to stratify the train_test_split using your target labels. You can also try downsampling the more frequent label. There's tonnes of good resources online to help you navigate imbalanced data. Which one is best for you depends on the data and the use case of the model.

Method 2

  • You can't do it like this. To train the model you need to ensure you have a target variable for every sample (at least for supervised learning). You can use the ouput of a model as a feature for another, but you can't treat the output of a model as a true value, which is what you're doing here by training on the output of a model.

As a final comment on the train and test scores. If you're dataset is highly imbalanced, say 90% are labeled 1 and 10% are labeled 0. Then a precision score of 90% isn't very impressive since the model could get this score by simply predicting 1 for every sample in the dataset which is not a very good model. It's probably better to evaluate your model using recall instead of precision, but again this depends on your use case.

$\endgroup$
2
  • 1
    $\begingroup$ I think now I understand why in method 2 my validation score is high as i am using predicted labels in my validation set. Thank you for the insight. $\endgroup$
    – SrJ
    Commented Sep 23, 2022 at 13:13
  • $\begingroup$ I have applied different oversampling/undersampling methods but none gives a very good score. Note : All the variables are categorical $\endgroup$
    – SrJ
    Commented Sep 23, 2022 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.