I am working with a pandas DataFrame that contains a combination of numerical and categorical data on customer demographics and preferences. Each row contains information about an individual customer. df
has 5000 rows and 30 columns.
df = pd.DataFrame({'col1':list('mno'),
'col2':[['a','b'],['d'],['a','b','c','d']],
'col3':[1,2,3],
'col4':[[1,3,4],[5,6,7],[8,3]]})
df
col1 col2 col3 col4
0 m ['a', 'b'] 1 [1, 3, 4]
1 n ['d'] 2 [5, 6, 7]
2 o ['a', 'b', 'c', 'd'] 3 [8, 3]
I am hoping to use k-prototypes to segment customers into distinct groupings since there is no ground truth on class labels and I have mixed data. However, I am unsure how to effectively analyze the columns that have lists of multiple values, e.g. col2
and col4
. I looked into one-hot encoding to break up those columns into multiple dummy columns, e.g. pd.get_dummies
, but I'm concerned that creating a sparse matrix may negatively affect my results. Can anyone suggest a better way of handling this problem?