2
$\begingroup$

I am trying to build a classification model. One of the variables called specialty has 200 values. Based on a previous post I saw, I decided I wanted to include the values that have the highest mean. I am thinking greater than 0.5. How would I filter the specialty to have only values greater than 0.5 for the mean? I am trying to get my final dataset ready for machine learning. Any advice is appreciated.

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

So if I understand you correctly you want to "one-hot-encode" or dummy-encode your variable "specialty" so that it goes from an interval scaled variable to a binary variable where 1 == >.5 and 0 == <=.5 correct?

So seeing as you are in python the following code would create a new variable that does what you want:

import pandas as pd
import numpy as np

df2['specialty_binned'] = np.digitize(df2['specialty'],bins=[0.5], right = True)

This would create a new variable in your data frame called 'specialty_binned' that is only 1s and 0s with 1 being values above 0.5 in the old variable.

$\endgroup$
2
  • $\begingroup$ Right now, the specialty category has 200 different specialties. It is not an interval scaled variable. It is categorical variable right now with text. I want to associate each category with a number so I can build a classification model. I was thinking of using a mean encoder. However, I think it might be difficult to interpret the model then. $\endgroup$
    – bulldog23
    Commented Apr 18, 2022 at 15:04
  • $\begingroup$ This is a completely different task. In this case you have a factor variable that cannot be simply represented by numbers. Especially if those 200 specialities don't follow a strict ordinal value like e.g. typical survey responses (disagree to agree). You will have to one-hot-encode this variable which results in 200 new variables that are 1/0 scaled. Because that's a lot of variables it's best to cluster those 200 specialties before you do that so that you might have only 50 or so new variables. $\endgroup$
    – Fnguyen
    Commented Apr 18, 2022 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.