1
$\begingroup$

The census data set I'm using: https://archive.ics.uci.edu/ml/datasets/Adult

So, I'm currently using this census data to make observations and predict whether someone is married or not. However, when I plot a categorical graph for the race, it's overwhelmingly white and I was going to just not take race into account at all. Before I did that, though, I realized that I didn't take into account the "final weights" in the data set, which shows the number of people the census believes that the entry represents.

So, I have a few questions:

Should I use these weights when making observations? Am I interpreting this correctly? Or should I just ignore them?

If I were to use these weights, how exactly would I go about using these weights? I want to be able to predict whether someone is married or not based on a few of these factors given, but I'm having trouble figuring out how I should incorporate these weights.

Also, would it be dumb to divide all the final weights by 100 or something similar? All of these weights are in the hundreds of thousands, and I was thinking that if we divide the weights by some constant, then we can just add that number of rows (for each row based on its weight) to our data set.

I'm new to ML in general so forgive me if this is a newbie question.

$\endgroup$
0
$\begingroup$

Firstly, if a variable is imbalanced (Race in your case), you shouldn't worry about. As its a categorical variable, it will be converted to two or more dummy variables.

Secondly, if the weight represents the number of people for each entry, as the following example, you should include the weights.

hypothetical data

Race       Education    Marital    weight
------------------------------------------
White       tertiary    Married      10   
White       tertiary    Single        6
Black       tertiary    Married       8
White       secondary   Single        4
...
...

For this example, I would oversample each entry by its weight. Because your real database is:

          X               y
---------------------  --------
Race        Education   Marital
-----------------------------------
White       tertiary    Married   
White       tertiary    Married
White       tertiary    Married   10 records
...
White       tertiary    Married

White       tertiary    Single 
White       tertiary    Single     6 records
...
White       tertiary    Single
Black       tertiary    Married
Black       tertiary    Married    8 records
...
Black       tertiary    Married
White       secondary   Single
White       secondary   Single
White       secondary   Single
White       secondary   Single

However, I should see few records of your dataset to help you better.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks for answering! We were thinking about doing that, but the weights have an average of 200k, and the dataset already has 46k rows. If we were to oversample each entry by its weight then I think we would have too many rows in our dataframe. Is there any other way we could incorporate the weights? $\endgroup$ – scouts9 Dec 12 '19 at 17:26
  • $\begingroup$ As you said, you can divide all weights by a constant. If the minimum weight is 1K for instance, divide all the weights by 1000. Then the 200k weight becomes 200 $\endgroup$ – Mehdi Dec 13 '19 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.