4
$\begingroup$

I have a classification problem with 60 data points in a 2-dimensional feature space. The data originally is divided into 2 classes. Earlier I was using Statistics Toolbox of Matlab so it was giving me fairly good results.It was giving 1 false negative and no false positive.

I used the following code:

SVMstruct = svmtrain(point(1:60,:),T(1:60),'Kernel_Function','polynomial','polyorder',11,'Showplot',true);

I am using a polynomial kernel and with polynomial order of 11.

Boundries Generated after training data using svmTrain from Matlab

But when I use same kernel configuration in scikit-learn SVC it does not gives the same result rather it gives very undesirable result with classifying all of them to single class.

Boundries Generated after training data using SVC from Scikit-learn in python

I am using it as

svc = svm.SVC(kernel='poly', degree=11, C=10)

I have used with many values of C too. No major difference.

Why there is so much difference in results ? How can I get same result as I got using Matlab ? For me it is compulsory to do with python-scikit.

$\endgroup$
  • $\begingroup$ Slightly off-topic, but how did you decide to use an 11th-degree polynomial? That's a very complex function for what appears to be a small amount of data! $\endgroup$ – Kyle. Jan 7 '16 at 19:05
  • $\begingroup$ @kyle in Matlab I tried with many polynomial degrees and then finally found that degree 11 gives desirable results so I used it. $\endgroup$ – Shaleen Jain Jan 8 '16 at 3:52
3
$\begingroup$

You have to be sure that the algorithm is the same and the kernel functions are really the same. If you look at this python documentation page for kernels in scikit learn you will see there a description of poly kernel.

Notice that you have a gamma and a degree. Gamma is by default 'auto' which is evaluated at 1/n_samples. For the same kernel you have 'coef0' (a great name for a parameter), which is used in poly as the free term. I do not know how matlab put this values as defaults, but the usual formula for poly kernel in literature I found it to be $poly(x_1, x_2) = (<x_1,x_2> + 1)^{d}$. So no gamma and the free term is $1$. I think matlab uses that. (Anyway I found the 'improvements' in scikit learn to have a not so good smell).

Also in this SVC documentation page they state that there is a parameter called shrinking. I really do not know it's effect, but its auto, which means is enabled. Might be an issue.

Later edit:

I found this documentation page for svm in matlab which describes the kernel in the way I stated (no degree, free term with $1$). Also it states that 'SMO' is used by default, make it sure you use 'SMO' in python also.

On the other hand you have to understand that these kind of algorithms are solved by optimization methods which are usually iterative and to save some memory, or cycles their implementations can be different in small details, which will almost produce different results. I agree however that the results should be similar.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks alot @rapaio I also found the same while reading it. But at least if using same configuration they must give almost similar result. And moreover also found that If I scale my points rather from [0,1] to [0,100] the in scikit-learn SVM changes, it was not so in Matlab, The SVM scales it accordingly. $\endgroup$ – Shaleen Jain Jan 12 '16 at 4:11
3
$\begingroup$

SVC Parameters:

What prompted you to use a polynomial kernel? There are cases where this might make sense, but it certainly wouldn't be my first choice. A radial basis function is likely a better fit, which is why it is the default in SKLearn. Step back a minute and take in the big picture perspective. Does your human mind want to create a decision surface using an 11th order polynomial given the points that you showed? It looks like there are a couple islands of white points in your second image that could easily be wrapped with two Gaussian functions aka radial basis functions.

What does the SVC with only default parameters do? I've seen plenty of cases where a regularization parameter as high as 10 or 100 is needed, but you haven't spoken to the cross validation procedure you used to arrive at $C=10$, so I wonder how you got there.

I suggest running SVC with default parameters and seeing what this returns. Then maybe try tuning gamma (this is effectively the buffer zone between the decision surface and points around it) and C (this is the regularization term that controls overfitting aka high variance).

Hope this helps!

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have tried many of the options for C and hence arrived for the value of it. Moreover its like my human mind see a polynomial running on the surface which separates the two regions. Lately I found the Solution using RBF with low gamma and high C value $\endgroup$ – Shaleen Jain Jan 13 '16 at 5:53
  • $\begingroup$ My guess is that you've reached some sort of limit in SKL treatment of polynomial basis functions at order 11. All I'm saying is back off and retune your model from the default parameters. Maybe try finding the best SKL model, then try porting those params over to Matlab. Take a look at what RBFs look like by comparison (actually visualize the decision surface) and I bet you'll use those by default next time! $\endgroup$ – AN6U5 Jan 13 '16 at 6:04
0
$\begingroup$

I am clueless about the difference coming both the implementation yet, but found how to find a decision boundary for the problem.

At First step I scaled the data to a range [0,100] which earlier was [0,1] for the input. (Still do not know why scaling creates a difference)

And then used a RBF kernal with low gamma and high C value.

svc = svm.SVC(kernel='rbf', gamma=.004, C=1000).

This gives me the following result.

Separation boundary created using above mentioned kernal

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.