Why is it important to have sufficient number of instances in your dataset for each stratum?

As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \\$60,000.

I didn't understand the explanation behind why housing['median_income'] has to be divide by 1.5

 housing['income_cat'] = np.ceil(housing['median_income'] / 1.5)


Explanation - It is important to have a sufficient number of instances in your dataset for each stratum, or else the estimate of the stratum's importance may be biased.

Can someone help me to understand the explanation and also why is it only 1.5 ?. As per the explanation, why the estimate of stratum's importance will be biased when there is no adequate instances for each category ?

• Is stratum equivalent to epoch by chance? If yes then it's done to reduce the overall bias of the network by introducing the diversity among the training samples as much as possible, because let's say you are using a nn, nn are's power enough to evenearn the batch order, so to avoid that people do shuffling... Jun 3 '18 at 1:50

Lets take an example that you are trying to evaluate average income of a company. Assume company has 100 employees in total and they belong to two stratum (70 management and 30 technical executives) and management executives on average are paid higher compared to technical executives.

You are the surveyor: Due to resource constraints, you have access to survey of any 20 individuals chosen at random.

Case 1: you did a random sampling from 100 employees and end up with 10 management and 10 technical executives. you will get an estimate for average income that is biased towards technical executives. In other words, you are giving more importance/ representation to income of technical executives than actual representation in the population.

Similarly, in other attempt: Case 2: you did a random sampling from 100 employees and end up with 17 management and 3 technical executives. you will get an estimate for average income that is biased towards management executives.

Both the above sampled cases have bias and are not representative of the population. Therefore, when we have the stratum identified before the sampling, stratified sampling should be done. For eg. random sample for 14 out of 70 management executives and 6 out of 30 technical executives.

In your given example of the housing and income data, income scale has been shrunk by a factor of 1.5, so as to create less no of income categories for the same bin width. Any other factor > 1 could also be used to ensure each stratum has considerable number of instances.

• Thanks for answering. I googled on stratified sampling and now I am curious about the advantages of stratified sampling ?. Is it only about giving equal importance to each stratum?. Jun 3 '18 at 12:02
• Not equal but the right importance. it is about obtaining a sample that is representative of the population under consideration. Jun 4 '18 at 1:34