# Different validation sets give very different results. What can be the reason?

I have ~78k microscopy images of single cells, where the task is to classify for cancer (binary classifier). The images are labeled according to which patient the data came from. I do the train-val split, making sure no patient has images in both train and validation. I noticed that depending on which patients I put in the validation set (one malignant patient, one benign patient, always perserving 20% validation size and about the same class distribution) I get wildly different validation accuracies.

Below is a plot of a test I did, where I tried all permutations of validation set for each patient with cancer. The dashed lines marks where a new patient with cancer is replaced in the validation set. It seems that it is which patient with cancer I put in the validation set that influences the validation accuracy heavily.

My question is, what does this tell me and are there any popular methods for dealing with similar situations? My thinking is that I should train the model using the split in the dashed group number 3 in the plot, since it has the highest validation accuracy without lowering training accuracy, but then again maybe those results are due to some unknown leak.

EDIT: It should be noted that the images are labeled according to if they came from a patient with cancer or not, not whether the cell actually is cancerous. Below is an example of what the pictures look like, with very little difference between all images as far as what I can see with my eyes.

• From your description, I did not get what exactly the C0 to C23 in your plot are. Can you please explain, what "all permutations of validation set for each patient with cancer" means? May 26, 2023 at 21:17
• I use C as short for combination. If I have 4 patients, A, B, C and D. A and B has cancer and C and D has no cancer and I always want one patient with and without cancer in the validation set. I train the model on all combinations of (A,B) and (C,D) meaning: ---------------------- Validation set 1 (C1): A,C Validation set 2 (C2): A,D ---------------------- Validation set 3 (C3): B,C Validation set 4 (C4): B,D ---------------------- The dashed lines here are the same markers as in the plot, meaning for each group within, they all have the same malignant patient in the validation set. May 27, 2023 at 10:52

## 1 Answer

Different validation splits will give different results because the data points will vary. How severe can the change of results be depends on how different the data points are.

One way to reduce this impact is to use CrossValidation while training your model. Since you have a case of Binary Classification, you should go for StratifiedCV. This helps your model to capture the majority of the diverseness of the dataset.

Also since you mention that the majority of the images are similar (as far as you can tell), you should use image augmentation techniques. Keras has a helpful library which you can use. This will help your model to become more robust to any diverseness it might encounter when deployed.

These 2 methods will definitely solve your issue!

Cheers!