# Skewed multi-class data

I have a dataset which contains ~100,000 samples of 50 classes. I have been using SVM with an RBF kernel to train and predict new data. The problem though is the dataset is skewed towards different classes.

For example, Class 1 - 30 (~3% each), Class 31 - 45 (~0.6% each), Class 46 - 50 (~0.2% each)

I see that the model tends to very rarely predict the classes which occur less frequent in the training set, even though the test set has the same class distribution as the training set.

I am aware that there are technique such as 'undersampling' where the majority class is scaled down to the minor class. However, is this applicable here where there are so many different classes? Are there other methods to help handle this case?

I would suggest you to use libsvm, which already has adjustable class weights implemented in it. Rather than replicating the training samples, one modifies the C parameter for different classes in the SVM optimization. For example if your data has 2 classes, and the first class is only 10% of the data, you would choose class weights to be 10 and 1 for class 1 and 2 respectively. Therefore, margin violations of the first class would cost 10 times more than the margin violations for second class, and per-class accuracies would be more balanced.

• thanks for the advice, do you know if libsvm automatically does this or do I need to manually pass in the class weights? Jul 14 '14 at 19:24
• You have to manually pass in the class weights. The way to do that is different based on the interface you are using (python, java, matlab, c). It is well documented in the read me files if you download the tool from csie.ntu.edu.tw/~cjlin/libsvm. Also your data size seems to be large and the default multi-class implementation of libsvm will use one-vs-one classification which may take too long to run. You can try training 50 one-vs-all binary classifiers specifying the weights appropriately. Jul 14 '14 at 19:57

I am not an export in using SVMs, but usually (if you are using a machine learning library like Python's scikit-learn or R's libsvm, there is the class_weight parameter, or class.weights, respectively.

Or if you'd use a Bayes classifier, you would take this "skew" into account via the "prior (class) probabilities" P(ωj)

Regarding the approach, SVM with an RBF kernel does a good job, but SVMs can be slowed down by large object sizes, unless you are employing CV with e.g. one tenth of the data randomly assigned to each fold. However, did you ask yourself why you are employing SVMs in the first place?

Have you tried multivariate linear regression, $\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}$, where each record of $\mathbf{Y}$ is coded $y_{ij}=+1$ if the $i$th object is in class $j$, and $y_{ij}=-1$ otherwise? If the classification accuracy is appreciably high using linear regression, then your data are linearly separable, and more complex methods such as SVMs and ANNs aren't needed. Step 2 would be to show that k-nearest neighbor, naive Bayes, linear (Fisher) discriminant analysis, polytomous logistic regression, etc., break down and fail.

For terminology, you might couch the issue of having more class weights in the context of "lower proportions of objects in certain classes," or "near-zero class size." Skew tends to be used for describing the distribution of a feature's values, as in skewness, fat tails, etc.

How many features do you have? Did you try unsupervised clustering (class discovery) on the 100,000 objects before trying supervised classification (class prediction) with SVM? Maybe the 100,000 objects can be grouped into fewer classes than 50, for which the new class membership could be used as the target class during classification analysis. This may alleviate the problem of having near-zero class size.

I have faced this problem many times while using SVM with Rbf kernel. Using Linear kernel instead of Rbf kernel solved my problem, but I dealt with lesser number of classes. The results were less skewed and more accurate with the linear kernel. Hope this solves your problem.

Edit: While I wrote original answer I was naive enough to not consider weighting the classes as one of them correctly answered. Also, while using rbf kernel its important to make sure that the penalty parameter or the 'C' value as per sklearn's svm module is too generic. I find that the default value of C=1 is too generic most of the time and I typically end up with a value of C=10000. Hope this helps others who get skewed results with svm(rbf) despite of having good distribution of classes in data.

• Thanks for the reply! I have actually tried numerous other algorithms/kernels and still have the same type of problem. So I am looking for more of an approach like undersampling or some way to even out the classes. Jul 14 '14 at 16:40
• Ok, you might also want to try replicating rows for classes containing sparse data, although its useful only if the features of the sparse data are really good. Jul 14 '14 at 17:25