0
$\begingroup$

I have a very large skewed training set where every feature's data-points are very similar ?

For example, following is some part of the training data :

93.65034,94.50283,94.6677,94.20174,94.93986,95.21071,1
94.13783,94.61797,94.50526,95.66091,95.99478,95.12608,1
94.0238,93.95445,94.77115,94.65469,95.08566,94.97906,1
94.36343,94.32839,95.33167,95.24738,94.57213,95.05634,1
94.65813,94.65246,94.64984,95.29596,95.14167,95.39941,1
95.50876,94.45346,95.23837,95.26877,94.84924,94.8021,0
94.5774,93.92291,94.96261,95.40926,95.97659,95.17691,0
93.76617,94.27253,94.38002,94.28448,94.19957,94.98924,0

where the last column is the class-label - only 0 and 1.

This only a part of the dataset, but the actual dataset contains about 95% of samples with class-label being 1, and the rest with class-label being 0, despite the fact that more or less all the samples are very much similar.

Please suggest an appropriate classifier (using scikit-learn) as well.

Improve this question
$\endgroup$
1
  • $\begingroup$ you also might want to rescale your features... maybe differences in the samples become more visible that way $\endgroup$
    – oW_
    Commented Dec 13, 2016 at 23:18

3 Answers 3

Reset to default
2
$\begingroup$

This is the so-called "Class Imbalance Problem". Fortunately, there are a number of possible approaches.

Probably easiest is going to be to use the class_weight option present in a number of scikit-learn's classifiers. For example, LogisticRegression.

You can either set this option to "balanced" (or "auto" if you're using an older version of sklearn), in which case it automatically adjusts weights inversely proportional to class frequency (i.e. it tries to compensate for the class imbalance); or you can manually set the weights for each class label using a dict, e.g. {0:1, 1: 20}.

For a worked example using an SVM, see here.

See here for more information: https://stats.stackexchange.com/questions/131255/class-imbalance-in-supervised-machine-learning

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can you share any reference for how to implement any of these approaches in code ? (preferably using scikit-learn's classifiers in python) @PriceHardman $\endgroup$
    – Jarvis
    Commented Sep 14, 2016 at 17:42
  • $\begingroup$ Yep, updated the answer. $\endgroup$
    – PriceHardman
    Commented Sep 14, 2016 at 20:21
  • $\begingroup$ I tried your method, but still its not helping, can you have a look at the complete .csv file, how can I contact you ? @PriceHardman $\endgroup$
    – Jarvis
    Commented Sep 15, 2016 at 2:04
2
$\begingroup$

I find the proposed answer very good, using a cost sensitive approach would be the first step. Another solution to your issue would be to use some sort of Sampling to balance the two classes, like SMOTE. Have a look at imbalance-learn module. Moreover, you may try to use other Classifiers, depends to what extend you want to go. I would give a try with Naive Bayes via Kernel estimation, which is simple and quick. You will have to check. In the end if no approach seems to predict the minority class you could degrade the majority class via under sampling.

Improve this answer
$\endgroup$
0
1
$\begingroup$

The question mentions three training data characteristics.

  • Very large
  • Skewed
  • Similarity in values for all but the last dimension
  • Binary final dimension (~95% 1)

The large N is actually a good thing in terms of classification accuracy, but the volume of data may necessitate distributed computing and/or large memory, disk, network, and CPU provisions.

The narrow range of the real dimensions and the skew of the binary dimensions do not provide any information necessary to decide what classifier from scikit-learn is best. If you plot any two dimensions and look at the pictures from the visual comparison page you may be able to make a few best guesses yourself and try those first.

The key to getting the best performance from these classifiers is to and normalize what you can in three main ways.

  • Skew
  • Range
  • Offset

One cannot de-skew the binary dimension since there are no intermediate values and changing a 1 to a 0 makes no sense statistically. However, if the desired range to use in the classifier is -1 to 1, the data in that 7th dimension can be translated using x7' = 2 x7 - 1.

If one of the other dimensions has a skewed distribution and the classifier is known to work best with a normal distribution, it is possible that the data is exponentially distributed, in which case, if it is the 2nd dimension, the general translation might be x2' = ln (x2) / k1 - k0 where the constants are chosen to translate the distribution data to within the desired ideal range of the classifier.

Various non-linear functions that compensate for skew can be used to improve classification significantly and the resulting learned behavior (model) can then be re-expressed using the inverse of these non-linear functions

Scatter plots will become clearer indicators of which classifiers are likely to work best after distributions are normalized.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.