I want to share a concern I have. I want to obtain a machine learning model that can predict whether a molecule exhibits biological activity. For this purpose, I have a set of molecules that do exhibit activity (actives) and another set of molecules that do not exhibit activity (inactives).

In total, I have a dataset of approximately 100 molecules, where the number of actives and inactives is the same, making the dataset balanced.

I am performing cross-validation, where I use 80 molecules for validation and 20 molecules for testing. I am using 5 folds in the cross-validation process.

I am using SVC (Support Vector Classifier) as the algorithm. The problem is that during cross-validation, I am obtaining average values of accuracy, specificity, recall, and ROC AUC close to 1. Then, during testing, I am also obtaining values close to 1.

Subsequently, I tested the models against all the actives and inactives in my dataset separately. I found that the model only detects 50% of the actives correctly and 60% of the inactives correctly. This result contradicts the metrics I obtained earlier. I'm not sure if this could be a problem of overfitting or something else.

I would be very grateful for any help and insights into this issue.

  • $\begingroup$ Before everything, getting perfect result is always suspicious. Check for data leakage and coding error first. $\endgroup$
    – lpounng
    Aug 2 at 3:51

1 Answer 1


Activity prediction is notoriously difficult (https://pubs.acs.org/doi/abs/10.1021/acs.jmedchem.2c00487)

You also probably have data leakage (common scaffolds, etc)

That said, if your training process says you have high metrics, but predicting on the same items later with the same model gives bad results, there is likely a flaw somewhere in your pipeline.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.