Skewed multi-class data

Question

I have a dataset which contains ~100,000 samples of 50 classes. I have been using SVM with an RBF kernel to train and predict new data. The problem though is the dataset is skewed towards different classes.

For example, Class 1 - 30 (~3% each), Class 31 - 45 (~0.6% each), Class 46 - 50 (~0.2% each)

I see that the model tends to very rarely predict the classes which occur less frequent in the training set, even though the test set has the same class distribution as the training set.

I am aware that there are technique such as 'undersampling' where the majority class is scaled down to the minor class. However, is this applicable here where there are so many different classes? Are there other methods to help handle this case?

Answer 1

I would suggest you to use libsvm, which already has adjustable class weights implemented in it. Rather than replicating the training samples, one modifies the C parameter for different classes in the SVM optimization. For example if your data has 2 classes, and the first class is only 10% of the data, you would choose class weights to be 10 and 1 for class 1 and 2 respectively. Therefore, margin violations of the first class would cost 10 times more than the margin violations for second class, and per-class accuracies would be more balanced.

Answer 2

I am not an export in using SVMs, but usually (if you are using a machine learning library like Python's scikit-learn or R's libsvm, there is the class_weight parameter, or class.weights, respectively.

Or if you'd use a Bayes classifier, you would take this "skew" into account via the "prior (class) probabilities" P(ω_j)

Answer 3

Regarding the approach, SVM with an RBF kernel does a good job, but SVMs can be slowed down by large object sizes, unless you are employing CV with e.g. one tenth of the data randomly assigned to each fold. However, did you ask yourself why you are employing SVMs in the first place?

Have you tried multivariate linear regression, $\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}$, where each record of $\mathbf{Y}$ is coded $y_{ij}=+1$ if the $i$th object is in class $j$, and $y_{ij}=-1$ otherwise? If the classification accuracy is appreciably high using linear regression, then your data are linearly separable, and more complex methods such as SVMs and ANNs aren't needed. Step 2 would be to show that k-nearest neighbor, naive Bayes, linear (Fisher) discriminant analysis, polytomous logistic regression, etc., break down and fail.

For terminology, you might couch the issue of having more class weights in the context of "lower proportions of objects in certain classes," or "near-zero class size." Skew tends to be used for describing the distribution of a feature's values, as in skewness, fat tails, etc.

How many features do you have? Did you try unsupervised clustering (class discovery) on the 100,000 objects before trying supervised classification (class prediction) with SVM? Maybe the 100,000 objects can be grouped into fewer classes than 50, for which the new class membership could be used as the target class during classification analysis. This may alleviate the problem of having near-zero class size.

Answer 4

I have faced this problem many times while using SVM with Rbf kernel. Using Linear kernel instead of Rbf kernel solved my problem, but I dealt with lesser number of classes. The results were less skewed and more accurate with the linear kernel. Hope this solves your problem.

Edit: While I wrote original answer I was naive enough to not consider weighting the classes as one of them correctly answered. Also, while using rbf kernel its important to make sure that the penalty parameter or the 'C' value as per sklearn's svm module is too generic. I find that the default value of C=1 is too generic most of the time and I typically end up with a value of C=10000. Hope this helps others who get skewed results with svm(rbf) despite of having good distribution of classes in data.