Segmenting pandas dataframe with lists as elements

Question

I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. df has 5000 rows and 30 columns.

df = pd.DataFrame({'col1':list('mno'),
                    'col2':[['a','b'],['d'],['a','b','c','d']],
                    'col3':[1,2,3],
                    'col4':[[1,3,4],[5,6,7],[8,3]]})
df

   col1     col2              col3        col4
0   m    ['a', 'b']            1        [1, 3, 4]
1   n    ['d']                 2        [5, 6, 7]
2   o    ['a', 'b', 'c', 'd']  3        [8, 3]

I am hoping to use k-prototypes to segment customers into distinct groupings since there is no ground truth on class labels and I have mixed data. However, I am unsure how to effectively analyze the columns that have lists of multiple values, e.g. col2 and col4. I looked into one-hot encoding to break up those columns into multiple dummy columns, e.g. pd.get_dummies, but I'm concerned that creating a sparse matrix may negatively affect my results. Can anyone suggest a better way of handling this problem?

Answer 1

Are you concerned that a column might have too many values and that one hot encoding will produce features with less predictive power? In that case you could select a handful of the features for more prevalent class values and keep them as is, while combining the other features for rarer class values into one feature.