0
$\begingroup$
Hello, I have a dataFrame and one of features is categorical and I want to convert that to ordinal category(including a category for missing values)

enter image description here

enter image description here

enter image description here

enter image description here

but in the last cell as you see it assumes all of my categories as NaN (-1) and even I use X.fillna('missing') assumes all of those as 'missing'
I don't no what's the problem and I couldn't find any helpful documentation.

Thank you

$\endgroup$
2
  • $\begingroup$ Have a look at what pandas.Categorical returns and saved in the cat variable, they are all NaN values. This is because you are setting the categories to a list of strings, whereas the values in the edu_level column are values, meaning that the string values do not occur in the column and the values are therefore set to missing. $\endgroup$
    – Oxbowerce
    Commented Mar 23, 2022 at 13:39
  • $\begingroup$ Thanks a lot... $\endgroup$
    – H4KN
    Commented Mar 31, 2022 at 13:59

1 Answer 1

1
$\begingroup$

You mentioned two issues in the code :

  1. issue 1 : Our missing value is encoded as a seperate class

It is no longer a missing value , as you put np.NAN there and nan will considered as a seperate class in the Ordinal Encoder

  1. order is not respected becuse the categories are not specified, so it will take as auto and the most occured value will treated as first .

As @Oxbowerce commented , your pandas.Categorical will return -1 for all since the values are present instead of string .

in your code , I am not sure why you are substituing np.NAN() .

X = pd.DataFrame ( np.array ( [ 'M', 'O-', 'medium',
                                 'M', 'O-', 'high', 
                               'F', 'O+', 'high', 
                               'F', 'AB', 'low', 
                               'F', 'B+', np.NAN]).reshape((5,3)))
X.columns = ['sex', 'blood_type', 'edu_level']

at this point , if you know this is missing value, it is better to substitute here itself rather than complicating

or after your processing , if you want to convert back to categories including your missing value, you can simply do by

edu_dict = {2.0: 'Low', 0.0: 'High', 1.0: 'Medium',  3.0: 'Missing', }
X['edu_level'] = X['edu_level'].map(edu_dict)

output :

  sex blood_type edu_level
0   M         O-       Low
1   M         O-      High
2   F         O+      High
3   F         AB    Medium
4   F         B+   Missing
$\endgroup$
1
  • $\begingroup$ Thank you very much sir , I get it $\endgroup$
    – H4KN
    Commented Mar 31, 2022 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.