I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. df has 5000 rows and 30 columns.

df = pd.DataFrame({'col1':list('mno'),

   col1     col2              col3        col4
0   m    ['a', 'b']            1        [1, 3, 4]
1   n    ['d']                 2        [5, 6, 7]
2   o    ['a', 'b', 'c', 'd']  3        [8, 3]

I am hoping to use k-prototypes to segment customers into distinct groupings since there is no ground truth on class labels and I have mixed data. However, I am unsure how to effectively analyze the columns that have lists of multiple values, e.g. col2 and col4. I looked into one-hot encoding to break up those columns into multiple dummy columns, e.g. pd.get_dummies, but I'm concerned that creating a sparse matrix may negatively affect my results. Can anyone suggest a better way of handling this problem?

  • $\begingroup$ No, creating a sparse matrices won't be a negative for your problem, it's that's why they are made for, to save memory when you have lot.of zeros $\endgroup$
    – Aditya
    Apr 28 '18 at 0:50

Are you concerned that a column might have too many values and that one hot encoding will produce features with less predictive power? In that case you could select a handful of the features for more prevalent class values and keep them as is, while combining the other features for rarer class values into one feature.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.