I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do exhibit activity (actives) and another set of molecules that do not exhibit activity (inactives).
In total, I have a dataset of approximately 100 molecules, where the number of actives and inactives is the same, making the dataset balanced.
I am performing cross-validation, where I use 80 molecules for validation and 20 molecules for testing. I am using 5 folds in the cross-validation process.
I am using SVC (Support Vector Classifier) as the algorithm. The problem is that during cross-validation, I am obtaining average values of accuracy, specificity, recall, and ROC AUC close to 1. Then, during testing, I am also obtaining values close to 1.
Subsequently, I tested the models against all the actives and inactives in my dataset separately. I found that the model only detects 50% of the actives correctly and 60% of the inactives correctly. This result contradicts the metrics I obtained earlier. I'm not sure if this could be a problem of overfitting or something else.
I would be very grateful for any help and insights into this issue.