Take the 2-minute tour ×
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It's 100% free, no registration required.

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should go with the similar kind of data for training my models as well. But then I did come across a research paper or two (in my area of work) which have used a balanced data to train models, implying equal number of instances of benign and malicious traffic.

In general, if I am building ML models, should I go for a dataset which is representative of the real world problem, or is a balanced dataset better suited for building the models (since certain classifiers do not behave well with class imbalance, or due to other reasons not known to me)?

Can someone shed more light on the pros and cons of both the choices, and how to decide which one to go for?

share|improve this question

4 Answers 4

up vote 13 down vote accepted

I would say the answer depends on your use case. Based on my experience:

  • If you're trying to build a representative model -- one that describes the data rather than necessarily predicts -- then I would suggest using a representative sample of your data.
  • If you want to build a predictive model, particularly one that performs well by measure of AUC or rank-order and plan to use a basic ML framework (i.e. Decision Tree, SVM, Naive Bayes, etc), then I would suggest you feed the framework a balanced dataset. Much of the literature on class imbalance finds that random undersampling (down sampling the majority class to the size of the minority class) can drive performance gains.
  • If you're building a predictive model, but are using a more advanced framework (i.e. something that determines sampling parameters via wrapper or a modification of a bagging framework that samples to class equivalence), then I would suggest again feeding the representative sample and letting the algorithm take care of balancing the data for training.
share|improve this answer
1  
If you're trying to build a representative model -- one that describes the data rather than necessarily predicts ... who builds a model which doesn't predcit?? Didn't get you there... –  pnp Jul 22 at 15:23
3  
Unsupervised learning would be an example where you build a model that isn't necessarily geared to predict. In some instances you might want to explore or summarize your data. –  DSea Jul 22 at 18:07
1  
I'd say it's safer to balance your sample, but also collect sampling weights so you can later re-weight your data for representativeness if you need to. @pnp plenty of social scientists build non-predictive models, e.g. for confirming theories. –  ssdecontrol Aug 4 at 23:56
    
How would a balanced model compare to a representative model using weighted observations to mimic a balanced model? –  Andy Blankertz Aug 22 at 22:25

There always is the solution to try both approaches and keep the one that maximizes the expected performances.

In your case, I would assume you prefer minimizing false negatives at the cost of some false positive, so you want to bias your classifier against the strong negative prior, and address the imbalance by reducing the number of negative examples in your training set.

Then compute the precision/recall, or sensitivity/specificity, or whatever criterion suits you on the full, imbalanced, dataset to make sure you haven't ignored a significant pattern present in the real data while building the model on the reduced data.

share|improve this answer

I think it always depends on the scenario. Using a representative data set is not always the solution. Assume that your training set with 1000 negative examples and 20 positive examples. Without any modeification of the classifier, your algorithm will tend to classify all new examples as negative. In some scenarios this is O.K. But in many cases the costs of missing postive examples is high so you have to find a solution for it.

In such cases you can use a cost sensitive machine learning algorithm. For example in the case of medical diagnosis data analysis.

In summary: Classification erros do not have the same cost!

share|improve this answer

I think there are two separate issues to consider: Training time, and prediction accuracy.

Take a simple example : consider you have two classes, that have a multivariate normal distribution. Basically you need to estimate the respective class means and class covariances. Now the first thing you care about is your estimate of the difference in the class means: but your performance is limited by the accuracy of the worst estimated mean: its no good estimating one mean to the 100th decimal place - if the other mean is only estimated to 1 decimal place. So its a waste of computing resources to use all the data - you can instead undersample the more common class AND reweight the classes appropriately. ( those computing resources can then be used exploring different input variables etc)

Now the second issue is predictive accuracy: different algorithms use different error metrics, which may or may not agree with your own objectives. eg logistic regression will penalise overall probability error, so if most of your data is from one class, then it will tend to try to improve accurate probability estimates ( eg 90 vs 95% probability) of that one class rather than trying to identify the rare class. in that case you would definitely want to try to reweight to emphasize the rare class ( and subsequently adjust the estimate [by adjusting the bias term] to get the probability estimates realigned)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.