Test Binary Classifier on a Test-set that includes only one class

Question

I'm working on a disease binary classification problem.

0 = healthy , 1 = not healthy

The disease is a movement disorder that appears on the patient while moving a specific movement. I applied leave-one-out cross-validation to train on all patients except one to test on and so on.

The problem is, some patients don't show class not healthy at all in their datasets. Thus, the evaluation metrics f1-score, precision & recall went down sharply because of those patients.

I tried to use SMOTE oversampling, but it didn't work because it will generate new samples from the data of other patients. Additionally, I tried class-weights, but it didn't work as well, because there is not a minor class in the test set to give it a higher priority.

How can I solve this problem?

Answer 1

You can try pooling: "pool" together all the predictions from each round of testing, and compute the metrics once over these pooled predictions

Basically you could just keep a list of the predictions and true values for each test set. Then after testing for each patient, combine all sets of predictions and true values together and produce one confusion matrix at the end containing the combined predictions and true values from all patients.

Then you'll be able to calculate accuracy, recall, and precision

Answer 2

Give cross-validation or bootstrapping a shot! Also, for metrics, look at per class performance and macro averaged.

Answer 3

Try using stratify to see if the problem is solved. When splitting add the stratify parameter as train_test_split(x, y, test_size = 0.2, random_state = 69, startify = y).