0
$\begingroup$

I am quite new machine learning methods, so I may not write proper technical formulas.

My question is about the optimal proportion between sample size in Class = 1 and Class = 0 in a binary classification problem. (I might use different models, but I will start with decision tree).

I am trying to develop a risk model for a population of 300.000 by using some risk factors, and each risk factor has a risk-weight. So basically it is a accumulative model

Risk_amount = Risk_factor_1 * weight_1 + Risk_factor_2 * weight_2 + Risk_factor_3 * weight_3 ...

My big question is if a Risk_amount really belongs to Class = 1 or not.

I want my ML-model to find the most optimal weights to those risk factors.

I already know that 1000 out of 300.000 are risky in real. So the amount of target data with Class = 1 is 1000.

So the total amount of target data with non-risky population Class = 0 is 299.000

The question is what is the optimal proportion between the amount of Class = 1 and the amount of Class = 0?

1000 Class = 1 vs. 1000 Class = 0

or

1000 Class = 1 vs. the whole Class = 0 as 299.000?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The balance between two classes in a classification is very important, as you do not want your model to overfit for a particular class.

This is where you use metrics apart from accuracy to really evaluate how good your model truly is. In case you are not able to balance the dataset, there are multiple ways to work with imbalanced data. They are as follows:

  1. You could use certain techniques like SMOTE to generate more samples of the undersampled class
  2. You must split your dataset in test and train with stratification so that you have a balance in your evaluation.
  3. You can sub-sample the large class and balance the two classes, and this multiple times by taking random sub samples of the larger class.

Thorough analysis of the result is very important to understand how to proceed towards the solution to the problem. Please look at f1 score, precision and recall apart from accuracy. Also read about micro/macro averaging of these metrics.

There are a lot of conversations on datascience stackexchange and stackoverflow on how to work with imbalanced data for classification. Here is a link : https://stackoverflow.com/questions/40568254/machine-learning-classification-on-imbalanced-data

Have fun with machine learning :)

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ So as I understand it is better to choose equal number of samles for both Class-1 and Class-0. I used an option in my tool (SAS) as "Simple Random Sampling" which gives equal chance to each member to be chosen. I guess it should be enough for my case. $\endgroup$
    – user3714330
    Jan 8, 2019 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.