I am trying to make a malicious software detector using machine learning.I have obtained a dataset from internet which containes md5 hash and other features of pe files.I fed this databse to weka to find out the most important features,which came out to be only one(static_prio).I tried running the K-NN algorithm on the data using only one feature(static_prio) and I got 99.82% accuracy but I am a little confused about my approach here.I am naive with machine learning so please help by putting me on the right track.Thank you.
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
It is not clear in your post whether or not you are using training/test sets so I will answer in this way:
The first thing to do if you want to validate your results is to cut your set into a training set and a validation set. This way you train the K-NN method on your training set, and you use the trained classifier on the validation set. Then you can monitor the validation error, and the training error.
In the best case scenario, it should look like this : 
The aim is to obtain a model that generalizes well, which means being able to accurately predict the class a sample belongs to, when it has never seen the sample before.
-
$\begingroup$ I have used training and test set and I calculated confusion matrix using y_pred and y_test.using confusion matrix values I calculated the accuracy(by cross adding opposite cells of matrix and dividing total wrong predictions by total predictions).I dont know much about confusion matrix. $\endgroup$ Commented Jun 27, 2018 at 14:04
-
1$\begingroup$ Well you can always plot ROC curve, precision and recall with scikit learn packages. But like Sahar Milis said, if you train your classifier on 99% "0" and 1% "1" it will have 99% accuracy on a test set having the same proportions. But you are interested in the remaining 1%. Over/under sampling might be a way to overcome this unbalanced data problem. $\endgroup$– NufeuCommented Jun 27, 2018 at 15:15
$\begingroup$
$\endgroup$
When trying to classify fraud/malware/etc,
don't score your model based only on Accuracy ... and always check for imbalanced data.
The reason is the Accuracy paradox - wikipedia | Towards Data Science
Check out this Github code - it's very much suitable for begginers...
The problem with your model is that it's almost always classifying the software as "not malware", and due to the very small number of "malware" it gets high accuracy.
Good Luck :)
