Scaling does not speed up the SVM model

Question

I tried to standardize the training data with samples of 629,145 rows and 24 features:

from sklearn import datasets
import pandas as pd
from sklearn import svm
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import StandardScaler

df = pd.read_csv('mydata.csv', dtype='object')

#manually choosing 24 features
X=df.loc[:, ['Bwd Pkt Len Min','Subflow Fwd Byts','TotLen Fwd Pkts','TotLen Fwd Pkts','Bwd Pkt Len Std','Flow IAT Min',
             'Fwd IAT Min','Flow IAT Mean','Flow Duration','Flow IAT Std','Active Min','Active Mean','Fwd IAT Min',
             'Bwd IAT Mean','Fwd IAT Mean','Init Fwd Win Byts','ACK Flag Cnt','Fwd PSH Flags','SYN Flag Cnt','Fwd Pkts/s',
             'Bwd Pkts/s','Init Bwd Win Byts','PSH Flag Cnt','Pkt Size Avg']]
Y= df['Label'] 

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4,random_state=42) # 60% training and 40% test
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

#Create a svm Classifier
clf = svm.SVC(kernel='rbf') # not linear Kernel
clf.fit(X_train, y_train)

Already 6 hours and SVM does not converge. Same data is converged with other algorithms very fast as RF and I know it is quite normal as SVM considered computationally high algorithm compared to KNN and RF.. I read quite a bit of questions/answers and articles.

1. I wonder how can I track and analyze the problem visually (probably plotting some graphs as cost function or ?).
2. Can parameter tuning (C parameter) be of the help and speed up this?
3. What could be your advice?

Thanks very much

Answer 1

I do not program on python. Nevertheless, I would say the key relies in the number of samples (629,145). They are many. The SVM has to test them to pick which are good support vectors for data partition/regression and given the size of the dataset there are a lot of alternatives. That issue plus the number of different C, gamma/sigma and, perhaps, epsilon (not sure you are classifying or regressing) tested during the optimisation of the SVM hinders convergence. There are people who used clusters instead of the original dataset to train the SVM. For instance: Barros de Almeida, M., de Padua Braga, A., Braga, J.P., 2000. SVM-KM: speeding SVMs learning with a priori cluster selection and k-means, in: Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks. Rio de Janeiro, (Brazil), pp. 162–167. can be a good starting reference but, I am aware there a others. There is at least one R package based on that idea (LinearizedSVR). In addition, there are analytical methods to infer the values of the parameters C, gamma/sigma and epsilon (for regression) based on the characteristics of the training dataset. That means no optimisation is necessary, although there is a friend that still tunes gamma/sigma but fixes C fallowing these approaches. I think there is available code (at least for R) somewhere in the net. The refs are: Cherkassky, V., Ma, Y., 2004. Practical selection of SVM parameters and noise estimation for SVM regression. Neural Networks 17 (1), 113–126. 10.1016/S0893-6080(03)00169-2 for regression and, Keerthi, S.S., Lin, C.-J., 2003. Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel. Neural Comput. 15 (7), 1667–1689. 10.1162/089976603321891855 for classification SVMs.

Answer 2

Here are some suggestions for optimizing the training time for an SVM with a large dataset:

1. Use a more efficient implementation of the SVM algorithm, such as the LibSVM library, which can be faster than the default SVM implementation in scikit-learn.

2. Use a linear kernel for the SVM, as this can be faster to train than more complex kernels such as the polynomial or RBF kernels.

3. Use the n_jobs parameter to specify the number of CPU cores to use for training the SVM, which can speed up the training process.

4. Use a smaller sample of the dataset for training the SVM, as the processing time will be proportional to the size of the dataset.

5. Use dimensionality reduction techniques, such as PCA or LDA, to reduce the number of features in the dataset, which can also speed up the training process.

6. Use a coarser grid for hyperparameter tuning, such as increasing the stepsize for the values of the hyperparameters, as this can reduce the number of combinations to be tested.

7. Use a more efficient algorithm for hyperparameter tuning, such as Bayesian optimization or genetic algorithms, which can find the optimal hyperparameters in a more efficient manner.

8. Use regularization techniques, such as L1 or L2 regularization, to prevent overfitting and improve the generalization performance of the SVM.

• Grid_search (RandomizedSearchCV) extremely slow with SVM (SVC)