1
$\begingroup$

I have a Dataset with 580 samples and 7 features. I compared the time between three kernels: Linear, Quadratic and Gaussian and using RandomizedSearchCV as the following:

from sklearn.model_selection import RandomizedSearchCV
from sklearn.preprocessing import StandardScaler

paramSVMLinear = {
        'kernel': ['linear'],
        'C': [ii for ii in np.linspace(1,1e7,50000)]
        }

scal = StandardScaler()
X = scal.fit_transform(X)

RandomizedSearchCV(SVC(), paramSVMLinear, n_iter=1, cv=2, scoring='accuracy', verbose=4).fit(X_train,Y_train)

SVC(kernel = 'linear').fit(X_train,Y_train)
SVC(kernel = 'poly',degree=2).fit(X_train,Y_train)
SVC(kernel = 'rbf').fit(X_train,Y_train)

The elapsed time in each fit is 187.64 sec, 0.001672 sec, 0.00187 sec, 0.001586 sec. Why the randomized search take so long for 1 iteration and 2 CV? Why the linear kernel take a longer time than RBF? Thanks!

$\endgroup$
0
$\begingroup$

A plausible reason that linear kernel takes longer is that data are not linearly separable, thus it takes longer to fit than a non-linear kernel like RBF.

The kernel trick avoids the explicit mapping that is needed to get linear learning algorithms to learn a nonlinear function or decision boundary. [..] Some algorithms that depend on arbitrary relationships in the native space $X$ would, in fact, have a linear interpretation in a different setting: the range space of $\varphi$. The linear interpretation gives us insight about the algorithm. Furthermore, there is often no need to compute $\varphi$ directly during computation, as is the case with support-vector machines. Some cite this running time shortcut as the primary benefit.

(emphasis mine)

References:

  1. Kernel trick
$\endgroup$
5
  • $\begingroup$ I see, thanks for the comment! Why it takes a lot longer on RandomizedSearchCV (10000x) than just one fit? Theoretically, split the training data in 2, make 2-fold cross-validation and does the fit. $\endgroup$
    – Felipe
    Jun 11 at 20:16
  • $\begingroup$ This has to do with the implementation of RandomizedSearchCV, and how CV and fitting is done. See documentation. Is this time consistent or might be different if ran again? $\endgroup$
    – Nikos M.
    Jun 12 at 9:10
  • $\begingroup$ Its time consistent. Seems like a bug, I cant see an explanation for this behavior $\endgroup$
    – Felipe
    Jun 14 at 13:55
  • $\begingroup$ If the time taken is consistent, then dive into the implementation of RandomizedSearchCV to see what operations it does. Else if time was not consistent, then it might depend on which random value for C was chosen. $\endgroup$
    – Nikos M.
    Jun 14 at 14:30
  • $\begingroup$ You can set the verbose option to True to see what is going on. And here is the source code $\endgroup$
    – Nikos M.
    Jun 14 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.