# How to deal with a skewed data-set having all the samples almost similar?

I have a very large skewed training set where every feature's data-points are very similar ?

For example, following is some part of the training data :

93.65034,94.50283,94.6677,94.20174,94.93986,95.21071,1
94.13783,94.61797,94.50526,95.66091,95.99478,95.12608,1
94.0238,93.95445,94.77115,94.65469,95.08566,94.97906,1
94.36343,94.32839,95.33167,95.24738,94.57213,95.05634,1
94.65813,94.65246,94.64984,95.29596,95.14167,95.39941,1
95.50876,94.45346,95.23837,95.26877,94.84924,94.8021,0
94.5774,93.92291,94.96261,95.40926,95.97659,95.17691,0
93.76617,94.27253,94.38002,94.28448,94.19957,94.98924,0


where the last column is the class-label - only 0 and 1.

This only a part of the dataset, but the actual dataset contains about 95% of samples with class-label being 1, and the rest with class-label being 0, despite the fact that more or less all the samples are very much similar.

Please suggest an appropriate classifier (using scikit-learn) as well.

• you also might want to rescale your features... maybe differences in the samples become more visible that way
– oW_
Dec 13, 2016 at 23:18

This is the so-called "Class Imbalance Problem". Fortunately, there are a number of possible approaches.

Probably easiest is going to be to use the class_weight option present in a number of scikit-learn's classifiers. For example, LogisticRegression.

You can either set this option to "balanced" (or "auto" if you're using an older version of sklearn), in which case it automatically adjusts weights inversely proportional to class frequency (i.e. it tries to compensate for the class imbalance); or you can manually set the weights for each class label using a dict, e.g. {0:1, 1: 20}.

For a worked example using an SVM, see here.

• Can you share any reference for how to implement any of these approaches in code ? (preferably using scikit-learn's classifiers in python) @PriceHardman Sep 14, 2016 at 17:42
• Yep, updated the answer. Sep 14, 2016 at 20:21
• I tried your method, but still its not helping, can you have a look at the complete .csv file, how can I contact you ? @PriceHardman Sep 15, 2016 at 2:04

I find the proposed answer very good, using a cost sensitive approach would be the first step. Another solution to your issue would be to use some sort of Sampling to balance the two classes, like SMOTE. Have a look at imbalance-learn module. Moreover, you may try to use other Classifiers, depends to what extend you want to go. I would give a try with Naive Bayes via Kernel estimation, which is simple and quick. You will have to check. In the end if no approach seems to predict the minority class you could degrade the majority class via under sampling.

The question mentions three training data characteristics.

• Very large
• Skewed
• Similarity in values for all but the last dimension
• Binary final dimension (~95% 1)

The large N is actually a good thing in terms of classification accuracy, but the volume of data may necessitate distributed computing and/or large memory, disk, network, and CPU provisions.

The narrow range of the real dimensions and the skew of the binary dimensions do not provide any information necessary to decide what classifier from scikit-learn is best. If you plot any two dimensions and look at the pictures from the visual comparison page you may be able to make a few best guesses yourself and try those first.

The key to getting the best performance from these classifiers is to and normalize what you can in three main ways.

• Skew
• Range
• Offset

One cannot de-skew the binary dimension since there are no intermediate values and changing a 1 to a 0 makes no sense statistically. However, if the desired range to use in the classifier is -1 to 1, the data in that 7th dimension can be translated using x7' = 2 x7 - 1.

If one of the other dimensions has a skewed distribution and the classifier is known to work best with a normal distribution, it is possible that the data is exponentially distributed, in which case, if it is the 2nd dimension, the general translation might be x2' = ln (x2) / k1 - k0 where the constants are chosen to translate the distribution data to within the desired ideal range of the classifier.

Various non-linear functions that compensate for skew can be used to improve classification significantly and the resulting learned behavior (model) can then be re-expressed using the inverse of these non-linear functions

Scatter plots will become clearer indicators of which classifiers are likely to work best after distributions are normalized.