Challenges in Predicting Molecule Activity

Question

I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do exhibit activity (actives) and another set of molecules that do not exhibit activity (inactives).

In total, I have a dataset of approximately 100 molecules, where the number of actives and inactives is the same, making the dataset balanced.

I am performing cross-validation, where I use 80 molecules for validation and 20 molecules for testing. I am using 5 folds in the cross-validation process.

I am using SVC (Support Vector Classifier) as the algorithm. The problem is that during cross-validation, I am obtaining average values of accuracy, specificity, recall, and ROC AUC close to 1. Then, during testing, I am also obtaining values close to 1.

Subsequently, I tested the models against all the actives and inactives in my dataset separately. I found that the model only detects 50% of the actives correctly and 60% of the inactives correctly. This result contradicts the metrics I obtained earlier. I'm not sure if this could be a problem of overfitting or something else.

I would be very grateful for any help and insights into this issue.

Answer 1

Activity prediction is notoriously difficult (https://pubs.acs.org/doi/abs/10.1021/acs.jmedchem.2c00487)

You also probably have data leakage (common scaffolds, etc)

That said, if your training process says you have high metrics, but predicting on the same items later with the same model gives bad results, there is likely a flaw somewhere in your pipeline.