12
$\begingroup$

I have just completed the machine learning for R course on cognitiveclass.ai and have begun experimenting with randomforests.

I have made a model by using the "randomForest" library in R. The model classifies by two classes, good, and bad.

I know that when a model is overfit, it performs well on data from its own trainingset but badly on out-of-sample data.

To train and test my model I have shuffled and split the complete dataset into 70% for training and 30% for testing.

My question: I am getting a 100% accuracy out of the prediction done on the testing set. Is this bad? It seems too good to be true.

The objective is waveform recognition on four on each other depending waveforms. The features of the dataset are the cost results of Dynamic Time Warping analysis of waveforms with their target waveform.

Improve this question
$\endgroup$
13
  • $\begingroup$ welcome to the site! Did you try predicting on some noise data? $\endgroup$
    – Toros91
    Commented Feb 8, 2018 at 9:15
  • $\begingroup$ Every time you reshuffle, train and test, the accuracy is 100%? $\endgroup$
    – Alex
    Commented Feb 8, 2018 at 9:19
  • $\begingroup$ @Alex Not exactly but it stays very high like 98,55% $\endgroup$
    – Milan van Dijck
    Commented Feb 8, 2018 at 9:24
  • 1
    $\begingroup$ @Alex 11.35% "ok" and 88.65% "bad" $\endgroup$
    – Milan van Dijck
    Commented Feb 8, 2018 at 9:47
  • 1
    $\begingroup$ That's quite imbalanced. Try using resampling (repeated sampling) to tip the balance in the training set towards OK class (make it 30% for example) and keep the 11/89 ratio in the test/validation sets. What do you get? $\endgroup$
    – Alex
    Commented Feb 8, 2018 at 13:16

2 Answers 2

Reset to default
29
$\begingroup$

High validation scores like accuracy generally mean that you are not overfitting, however it should lead to caution and may indicate something went wrong. It could also mean that the problem is not too difficult and that your model truly performs well. Two things that could go wrong:

  • You didn't split the data properly and the validation data also occured in your training data, meaning it does indicate overfitting because you are not measuring generalization anymore
  • You use some feature engineering to create additional features and you might have introduced some target leakage, where your rows are using information from it's current target, not just from others in your training set
Improve this answer
$\endgroup$
1
  • 11
    $\begingroup$ 100% accuracy always screams "target leakage." $\endgroup$
    – Paul
    Commented Feb 8, 2018 at 14:47
1
$\begingroup$

Investigate to see what your most predictive features are. Sometimes you accidentally included your target (or something that is equivalent to your target) amongst your features.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.