Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I have a panel data set. My dependent variable is total costs, and almost all of my independent variables are categorical variables. For instance, age is "old","new". Now i have some questions.
Should i use a dummy for all of them? For example, only type variable has 33 values itself, or i can use clustering and reduce them? Or any other way if you know)
Is there a difference in terms of behaviour between categorical variables which have a rank or not? For example type is "A","B",..."S" so no rank between A and B but quality is "A1","A2","A3" which A1 means highest quality.
I don't know why, I can't find enough information about variable selections and making data ready. So now i have lots of variable and I think I should choose between them and also reduse number of dummies.
You should convert the categorical variables to dummies. For each individual variable in general you want to have equal number of elements of each class, or at least the numbers should be close. If not, you can cluster smaller classes to form a larger one. For example, let's assume you have a categorical variable with 5 different categories. You want each class to be approximately %20 of the data. If it is not, you can define a new class which combines smaller classes to make each class approximately equal.
For the second part, if you can actually quantify how much A1 is better than A2, or able to assign a relative value to them based on some heuristics; you can convert them to numerical variables.
You can create dummies for rest of the variables involved, but I would still advice you to use replace using a loop if variables are high in number like A1,A2 ... A33 instead of get_dummies since using get dummies, you'll get very sparse columns through which your model may not learn much from.