I have a problem where I am to construct a classification variable Yes/No based on another feature's value. We are interested in the Yes class in this case. I am told to use 10-fold cross validation.
The data set contains 384 observations with only 6 Yes classifications. Obviously it is going to be nearly impossible to get any sort of usable model to predict Yes class without leaking the variable the class is based on into the model, but I don't want to do that.
I am using 100% of the dataset as I will import a different dataset to predict on.
When I try to run models in Orange, my evaluation results don't show an AUC for either yes or no, just the average, I have a warning saying some scores can't be computed? Because they are too low/high, number of folds? How I am to interpret this?
My ultimate question is, would this ever be practical in the real world? Would you even try to use this data for prediction of the Yes class?
I have added screenshots below of my test widget as well as my lift curves