1
$\begingroup$

I have a dataset in which one column is as given in the picture. What will be the best way to handle such a column?

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

This is a categorical variable. There are multiple ways to handle categorical variables in the literature -

Dummification (mostly for nominal variables)

The variable is split and converted into multiple binary variables (the number of variables should be 1-(number of classes) ).

Python example -

import pandas as pd
series = pd.Series(['<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', 
                    '>50k', '>50k', '>50k', '>50k', '>50k', '<=50k'])

print(pd.get_dummies(series).iloc[:, 1:])

Output

>50k
0      0
1      0
2      0
3      0
4      0
5      0
6      0
7      1
8      1
9      1
10     1
11     1
12     0

Notice how the information that was string has been now encoded in numbers. The reason was dropping a column is so that there is no perfect multicollinearity.

Ranking(in case of ordinal variables)

These are the cases where you can clearly rank the variables. In your case it would make more sense to rank them as we know then >50k is actually more than <50k.

Use the ranking method in you salary case and the dummification method for nominal cases (like location variables - where it can't be ordered)

Python example -

import pandas as pd
series = pd.Series(['<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', 
                    '>50k', '>50k', '>50k', '>50k', '>50k', '<=50k'])
uniq = series.unique()
ranks = { uniq[i]:i+1 for i in range(len(uniq)) }
print(ranks)
series_ranked = series.apply(lambda x: ranks[x])
print(series_ranked)

Output

0     1
1     1
2     1
3     1
4     1
5     1
6     1
7     2
8     2
9     2
10    2
11    2
12    1
dtype: int64

Again, in no way is this an exhaustive answer, but rest assured these are the most popular methods used to deal with categorical variables. Hope that helps. Cheers!

$\endgroup$
1
  • $\begingroup$ Thanks. Actually I am new to this and more familiar with R. It will take some time understand this. $\endgroup$
    – Aki
    Commented Feb 25, 2018 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.