I've recently started learning to work with
sklearn and have just come across this peculiar result.
I used the
digits dataset available in
sklearn to try different models and estimation methods.
When I tested a Support Vector Machine model on the data, I found out there are two different classes in
sklearn for SVM classification:
LinearSVC, where the former uses one-against-one approach and the other uses one-against-rest approach.
I didn't know what effect that could have on the results, so I tried both. I did a Monte Carlo-style estimation where I ran both models 500 times, each time splitting the sample randomly into 60% training and 40% test and calculating the error of the prediction on the test set.
What could account for such a stark difference? Why does the linear model have such higher accuracy most of the time?
And, relatedly, what could be causing the stark polarization in the results? Either an accuracy close to 1 or an accuracy close to 0, nothing in between.
For comparison, a decision tree classification produced a much more normally distributed error rate with an accuracy of around .85.