Dealing with categorical variables

Question

I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions.

1. Should i use a dummy for all of them? For example, only type variable has 33 values itself, or i can use clustering and reduce them? Or any other way if you know)

2. Is there a difference in terms of behaviour between categorical variables which have a rank or not? For example type is "A","B",..."S" so no rank between A and B but quality is "A1","A2","A3" which A1 means highest quality.

I don't know why, I can't find enough information about variable selections and making data ready. So now i have lots of variable and I think I should choose between them and also reduse number of dummies.

Answer 1

You should convert the categorical variables to dummies. For each individual variable in general you want to have equal number of elements of each class, or at least the numbers should be close. If not, you can cluster smaller classes to form a larger one. For example, let's assume you have a categorical variable with 5 different categories. You want each class to be approximately %20 of the data. If it is not, you can define a new class which combines smaller classes to make each class approximately equal.

For the second part, if you can actually quantify how much A1 is better than A2, or able to assign a relative value to them based on some heuristics; you can convert them to numerical variables.

You can find an example of this in this notebook (section titled "Aggregating categorical variables"). It is from the course "Principles of Machine Learning: R Edition" on edX. You can watch the videos on audit mode for free; and the notebooks are on github.

Answer 2

1. For your problem having 33 variables in the dataset perform a value_counts for those variables.

2. If you feel the values as you go down are not a lot and hence should not be given a category, you can give them a category of 'OTHER'. Revert to the image below to see how that happens.

3. You can tweak the parameter of nlargest from 4 to whichever value you find suitable.

4. If you want to find out the difference between categorical variable and target variable, perform EDA using a library like seaborn

   import seaborn as sns sns.factorplot('categorical_variable_column_name','target_column_name', data=dataframe)

5. If you don't have seaborn library or just want fast results use aggregate function like :

   dataframe.groupby(['categorcategorical_variable_column_name'])['target_column_name'].mean()

6. If you find some variables have higher values than other variables like A1>A2>A3 etc. you can assign numeric values in that order itself

   dataframe['categorical_column_name'].str.replace(['A1','A2','A3'],[30,20,10])

7. You can create dummies for rest of the variables involved, but I would still advice you to use replace using a loop if variables are high in number like A1,A2 ... A33 instead of get_dummies since using get dummies, you'll get very sparse columns through which your model may not learn much from.