SVM with polynomial kernel: strange results with high degree?

Question

Currently I'm working in WEKA, using the SMO classifier (an implementation of SVM). For an assignment I am requested to use a polynomial kernel, and report the results for degrees varying from 1 up to 50. I need to comment on overfitting, and my understanding is that higher degrees lead up to a higher risk of overfitting. (And hence, the percentage of correctly classified instances, using 10-fold cross validation should increase as the degree increases).

My intuition is right for the 'first' degrees, resulting in a 99.6% rate at degree=10. However, when I set the degree to 50, WEKA reports a 9% rate.

How is it possible that this rate drops (and by so much)?

Answer 1

Setting the polynomial kernel degree to 50 is likely causing the SVM to severely overfit to the data, which would explain the 9% you are seeing. Increasing the degree helps the SVM make an appropriate generalization, but when you start to see the validation/test accuracy decrease, then the SVM is starting to overfit.