0
$\begingroup$

I have collected dataset with two class labels and used the SVM Method to classify the dataset, and this is the results. Does this appear suspicious or not?

scikit-learn classifiers with SVM SVC train on 114859 instances, test on 49227 instances Excution (Training) Time: 9.82799983025

Excution (Testing) Time: 3.75

accuracy: 0.999837487558

Precision-Recall AUC: 1.00

ROC AUC: 1.00

pos precision: 0.999822253822

pos recall: 1.0

pos F-measure: 0.999911119012

neg precision: 1.0

neg recall: 0.998107404779

neg F-measure: 0.999052806062

$\endgroup$
1
$\begingroup$

The statistics here are obviously very good, in fact too good for any practical data set. Your model is almost perfect... Unfortunately, it's practically useless and I'll explain.

In machine learning, if you see something like this you know you are in trouble. That can happen if there are problems with your data workflow. For example, you might have removed all outliers that you shouldn't, or you actually used a subset of your training data for the test set.

It's fine if you're just toying SVM, but you'll never encounter something like this in real life.

$\endgroup$
  • $\begingroup$ Also thank you. What do you think about my problem if you know that my dataset was assembly instructions equivalent to nops, and the other category was the assembly of real instructions .. do you think now it is suspicious as the features I have selected made the problem more specific! $\endgroup$ – FADY R S AL KHATEEB Sep 6 '16 at 16:56
0
$\begingroup$

In the dark I would say that it is suspicious. However this really depends on the problem you are solving. Certain datasets will have a lot of clear examples, a very easy task to solve etcetera.

What could have happened is that during the preprocessing somehow your targets leaked into your features. Some more advanced preprocessing techniques like aggregating numerical features and the target per category and appending those to your features allow for such mistakes and will lead to extremely high accuracy. Any information that is not available during new predictions should not be used in your training set.

$\endgroup$
  • $\begingroup$ Thank you for your reply. What do you think about my problem if you know that my dataset was assembly instructions equivalent to CPU NOP's, and the other category was the assembly of real instructions .. do you think now it is suspicious as the features I have selected made the problem more concrete! $\endgroup$ – FADY R S AL KHATEEB Sep 6 '16 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.