11
$\begingroup$

I have just completed the machine learning for R course on cognitiveclass.ai and have begun experimenting with randomforests.

I have made a model by using the "randomForest" library in R. The model classifies by two classes, good, and bad.

I know that when a model is overfit, it performs well on data from its own trainingset but badly on out-of-sample data.

To train and test my model I have shuffled and split the complete dataset into 70% for training and 30% for testing.

My question: I am getting a 100% accuracy out of the prediction done on the testing set. Is this bad? It seems too good to be true.

The objective is waveform recognition on four on each other depending waveforms. The features of the dataset are the cost results of Dynamic Time Warping analysis of waveforms with their target waveform.

$\endgroup$
  • $\begingroup$ welcome to the site! Did you try predicting on some noise data? $\endgroup$ – Toros91 Feb 8 '18 at 9:15
  • $\begingroup$ Every time you reshuffle, train and test, the accuracy is 100%? $\endgroup$ – Alex Feb 8 '18 at 9:19
  • $\begingroup$ @Alex Not exactly but it stays very high like 98,55% $\endgroup$ – Milan van Dijck Feb 8 '18 at 9:24
  • 1
    $\begingroup$ @Alex 11.35% "ok" and 88.65% "bad" $\endgroup$ – Milan van Dijck Feb 8 '18 at 9:47
  • 1
    $\begingroup$ That's quite imbalanced. Try using resampling (repeated sampling) to tip the balance in the training set towards OK class (make it 30% for example) and keep the 11/89 ratio in the test/validation sets. What do you get? $\endgroup$ – Alex Feb 8 '18 at 13:16
29
$\begingroup$

High validation scores like accuracy generally mean that you are not overfitting, however it should lead to caution and may indicate something went wrong. It could also mean that the problem is not too difficult and that your model truly performs well. Two things that could go wrong:

  • You didn't split the data properly and the validation data also occured in your training data, meaning it does indicate overfitting because you are not measuring generalization anymore
  • You use some feature engineering to create additional features and you might have introduced some target leakage, where your rows are using information from it's current target, not just from others in your training set
$\endgroup$
  • 11
    $\begingroup$ 100% accuracy always screams "target leakage." $\endgroup$ – Paul Feb 8 '18 at 14:47
0
$\begingroup$

Investigate to see what your most predictive features are. Sometimes you accidentally included your target (or something that is equivalent to your target) amongst your features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.