0
$\begingroup$

The original dataset is of ~17K compound structures almost equally divided with labels indicating yes or no, after heavy use of mol2vec and rdkit I have created ~300 datapoints

Using the boosted trees method on the same shuffled train and test dataset gives 98% train accuracy and 89% test accuracy, but a simple neural network gives 100% train and test accuracy

I have checked the code again to ensure target leakage is not occurring, I have also coded it from the scratch twice to ensure I haven't made any mistake, yet I do not believe I should be getting 100% accuracy on both train and test

Does this mean that the model is actually accurate due to so many data points?

Training log

$\endgroup$
  • $\begingroup$ Can you clarify one more thing: was feature selection done using only a training subset of the data? $\endgroup$ – C8H10N4O2 Sep 7 at 17:04
  • $\begingroup$ @C8H10N4O2 No, it was done on the entire dataset. After this it was split into train and test $\endgroup$ – Navan Chauhan Sep 7 at 17:14
  • 1
    $\begingroup$ That was a mistake. Until it’s time to test, treat the test data as if they don’t exist. $\endgroup$ – Dave Sep 7 at 17:19
0
$\begingroup$

Yes - Getting 100% accuracy is possible for neural networks compared to tree-based models. Neural networks can learn non-linear relationships through the activation function. Tree-based models are restricted to piece-wise linear relationships.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Doing feature selection on the full dataset can lead to just this scenario, especially when so many features are available. This tread from the Stats stack exchange has some more information on how this happens, but the upshot is to make sure to do feature selection on some subset ("training set").

https://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation/

Please give this a try and speak up if you're still running into issues--just update the question with the relevant additional detail. It's certainly possible to hit 100%, but you would want to make sure there was no information leakage.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.