I have a dataset with a binary classification, on which I train a kNN algorithm and a SVM algorithm.

On the tests set I get approximately 75% error when using kNN, but only approximately a 20% error using an SVM.

What does this tell me about the dataset?


Given that we have not applied any preprocessing steps then the first thing that comes to mind is that SVM works better in higher dimensionalities while kNN suffer from the curse of dimensionality and therefore our dataset should have a large dimensionality especially in comparison to the number of instances. Therefore we have a very sparse space where the points can be separated by a hyperplane while the k-nearest neighbors give little information of the actual class.

Note that we should take under consideration that we have no knowledge if the test size follows the rule of thumb (10%-20%) and we have not repeat the experiment with multiple train-test splits to get a range of possible errors. We might have a very unlucky test set..

Finally we should take into account that we have little knowledge if the classes are imbalanced and how much. A highly imbalanced class could cause most of classifications to belong to the majority class while SVM would be worse because it would try, in vain, to find a hyperplane that would separate the classes and thus have a larger error. However here we have the opposite scenario. Therefore we should assume the case where the classes are more or less balanced but they are separable only in one (or a few) dimensions while pretty much mixed in the rest of the dimensions. Since knn does not learn any weights for each dimension, it assumes that the ones closest are the correct points while svm has taken a clear cut decision that anything beyond this line should be classified as such. enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.