# Test Binary Classifier on a Test-set that includes only one class

I'm working on a disease binary classification problem.

0 = healthy , 1 = not healthy

The disease is a movement disorder that appears on the patient while moving a specific movement. I applied leave-one-out cross-validation to train on all patients except one to test on and so on.

The problem is, some patients don't show class not healthy at all in their datasets. Thus, the evaluation metrics f1-score, precision & recall went down sharply because of those patients.

I tried to use SMOTE oversampling, but it didn't work because it will generate new samples from the data of other patients. Additionally, I tried class-weights, but it didn't work as well, because there is not a minor class in the test set to give it a higher priority.

How can I solve this problem?

Try using stratify to see if the problem is solved. When splitting add the stratify parameter as train_test_split(x, y, test_size = 0.2, random_state = 69, startify = y).