2
$\begingroup$

I'm testing hyperparameters for an SVM, however, when I resort to Gridsearch or RandomizedSearchCV, I haven't been able to get a resolution, because the processing time is exceeding hours.

My dataset is relatively small: 4303 rows and 67 attributes, with four classes (classification problem)

Here are the tested parameters:

params =[{'C': [0.1,1, 10, 100], 
         'kernel': ['poly','sigmoid','linear','rbf'],
         'gamma': [1,0.1,0.01,0.001]}
]
sv = SVC()

clf = RandomizedSearchCV(estimator=sv,
                   cv = 3, 
                   param_distributions=params,
                   scoring='accuracy', 
                   n_jobs = -1,
                   verbose=1)
clf.fit(X, y)
print("Best parameters:", clf.best_params_)
print("better accuracy: ", (clf.best_score_)**(1/2.0))

I've already reduced the number of parameters and the number of cvs, but I still can't get a result that doesn't take hours of processing.

Is it possible to optimize this process? Am I making a mistake regarding gridsearch or SVM?

$\endgroup$

2 Answers 2

2
$\begingroup$

It looks like your current approach is taking a long time because you are trying to search a large space of hyperparameters. One way to make the hyperparameter search more efficient is to use a smaller number of values for each hyperparameter, as this will reduce the total number of combinations that need to be tried.

There are several ways to optimize the hyperparameter tuning process for an SVM, including the following:

  1. Use a smaller sample of the dataset for hyperparameter tuning, as the processing time will be proportional to the size of the dataset.

  2. Use a more efficient algorithm for hyperparameter tuning, such as Bayesian optimization or genetic algorithms, which can find the optimal hyperparameters in a more efficient manner.

  3. Use a more efficient implementation of the SVM algorithm, such as the LibSVM library, which can be faster than the default SVM implementation in scikit-learn.

  4. Try different combinations of hyperparameters manually, rather than using grid search or randomized search, which can be computationally intensive.

  5. Use a more efficient kernel, such as the linear kernel, which can be faster to train than more complex kernels such as the polynomial or RBF kernels.

  6. Use a smaller number of hyperparameters, as the processing time will be proportional to the number of hyperparameters being tuned.

  7. Use a coarser grid for hyperparameter tuning, such as increasing the stepsize for the values of the hyperparameters, as this can reduce the number of combinations to be tested.

Overall, it is important to carefully select and optimize the hyperparameters for an SVM to improve its performance and reduce the processing time.

Also, check - SVC classifier taking too much time for training

$\endgroup$
2
  • 1
    $\begingroup$ Thank you very much for the excellent contributions! $\endgroup$ Commented Dec 9, 2022 at 10:14
  • $\begingroup$ This is wrong for RandomizedSearchCV, because you specify the number of searches to perform via n_iter and the default is 10. $\endgroup$
    – Princy
    Commented May 7 at 1:40
1
$\begingroup$

Note that the only other answer is incorrect. They suggest the reason it takes long is because you have too many hyperparameters, and it suggests to reduce the number of parameters you are searching.

The thing is that you are not using grid-search, you are using randomized-search, meaning that the search time is independent of the size of the hyperparameter space you are searching, but is controlled by 2 variables: CV and n_iter

CV is 3, and n_iter is not defined, which means it is its default value of 10

This means your search will only test your model 10*3 = 30 times, independently from your hyperparameter space

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.