0
$\begingroup$

I am running the Kaggle Video Games sales dataset through an XGboost algo.

I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5

when I use: data = pd.get_dummies(data=data, columns=['Game_Rating])

pandas adds a column to my dataframe for each unique observation in Game_Rating

Rating_E  Rating_E10+  Rating_EC  Rating_K-A  Rating_M  Rating_RP  Rating_T
1            0          0           0         0          0         0
0            0          0           0         0          0         0
1            0          0           0         0          0         0
1            0          0           0         0          0         0
0            0          0           0         0          0         0

My question is two-fold: Does Pandas have the ability to replace categorical values of a dataframe inline, preserving the original column structure

if not:

I have a (slower) method that does the encoding inline and preserves the column structure of the dataframe. I am trying to keep the dataframe as similiar to the original dataset as possible for future processing. What are the risks I may not be seeing of using my own (inline) encoding method?

Improve this question
$\endgroup$
  • $\begingroup$ What does your own encoding do? It's hard to give recommendations without knowing anything about it $\endgroup$ – marco_gorelli Oct 5 '18 at 15:45
  • $\begingroup$ my encoding method does the following: 1.) creates a list of unique (categorical) values in a column. 2.) creates an index of 0 to 'x' to represent each unqiue (categorical) value 3.) uses pandas.iloc to replace the unique categorical value with the numerical value. I could summarize by saying it does the same thing as pd.get_dummies - but replaces values inline, leaving column structures in their original state $\endgroup$ – learningtofly Oct 5 '18 at 15:49
  • $\begingroup$ If I've understood correctly, that's the same as numeric encoding (see scikit-learn.org/stable/modules/generated/…) $\endgroup$ – marco_gorelli Oct 5 '18 at 15:56
  • $\begingroup$ You can get the same effect very efficiently in pandas by using df['your column name'].astype('category').cat.codes. Be careful when doing this, because it implies an ordering on the columns. Your learning algorithm might overfit to spurious relationships that appear in the training data $\endgroup$ – timleathart Oct 6 '18 at 7:58
0
$\begingroup$

credit for this answer goes to @timleathart in the comments above:

You can get the same effect very efficiently in pandas by using df['your column name'].astype('category').cat.codes. Be careful when doing this, because it implies an ordering on the columns. Your learning algorithm might overfit to spurious relationships that appear in the training data

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.