# Cross Validation of Random Forest results in two different score based on when the test set is created

I have a dataset where I dont have label for 70% of data. So, my target is to train a model on 30% data that has label and then apply this model to get the label for the rest 70% of data. But, it is highly imbalanced and no single model performs well on this. So, I took the following approach :

Method 1

missing_df = df[df.label.isna()]
nomissing_df = df[~df.label.isna()]

#split the labeled dataset into 80-20 parts for train and validation
...... = train_test_split(nomissing_df)

while new samples are added :
#train a 5 Fold Random Forest on the train set
clf = RandomForest(....)
# code for training

#evaluate the missing data
label = clf.predict_proba(missing_df)

if label > 0.75 for all fold :
if label < 0.25 for all fold :



Method 2

missing_df = df[df.label.isna()]
nomissing_df = df[~df.label.isna()]

#NOOOOOOOOOOOOOOOOOOOOOOOO SPLIT HERE

while new samples are added :
#train a 5 Fold Random Forest on the train set
clf = RandomForest(....)
# code for training

#evaluate the missing data
label = clf.predict_proba(missing_df)

if label > 0.75 for all fold :
if label < 0.25 for all fold :

#split the full label dataset ( original + by the above process)
...... = train_test_split(final_train_df)

#train final model
clf = RandomForest().train(....)

#evaluate on validation set


Result

As our dataset is highly skewed, I used precision as my evaluation metric. For method 1, training precision is around 80% but when evaluated at the held-out validation set (at the beginning), the score is 33%.

But for method 2, the precision score for both train and validation is 80%.

My question is why is a discrepancy in validation accuracy? Is method 2 is wrong or any chance of data leakage here? Or any suggestion to improve the precision of method 1 as I think method 1 more correct way to do this? Finally, is this approach of labeling data is correct or there is any other way to do this better?

Method 1

• This looks like the correct way to go assuming you're not using the predicted labels of the missing data in your model validation set. The train,test and validation sets should come from nomissing_df only.
• If your labels are highly imbalanced it might be good to stratify the train_test_split using your target labels. You can also try downsampling the more frequent label. There's tonnes of good resources online to help you navigate imbalanced data. Which one is best for you depends on the data and the use case of the model.

Method 2

• You can't do it like this. To train the model you need to ensure you have a target variable for every sample (at least for supervised learning). You can use the ouput of a model as a feature for another, but you can't treat the output of a model as a true value, which is what you're doing here by training on the output of a model.

As a final comment on the train and test scores. If you're dataset is highly imbalanced, say 90% are labeled 1 and 10% are labeled 0. Then a precision score of 90% isn't very impressive since the model could get this score by simply predicting 1 for every sample in the dataset which is not a very good model. It's probably better to evaluate your model using recall instead of precision, but again this depends on your use case.

• I think now I understand why in method 2 my validation score is high as i am using predicted labels in my validation set. Thank you for the insight.
– SrJ
Commented Sep 23, 2022 at 13:13
• I have applied different oversampling/undersampling methods but none gives a very good score. Note : All the variables are categorical
– SrJ
Commented Sep 23, 2022 at 13:19