Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Suppose I have a movies dataframe in pandas. One of the features is Genre.
It has a list of genre names. for example:
Movie_ID Genre
1 [Action, Thriller, Drama]
2 [Romance, Comedy]
3 [Action, Romance]
How do I use this column for a clustering problem, say K-means clustering?
You can easily use get_dummy function in Pandas to convert them to numerical vectors.
The idea is that categorical variables do not have a numerical intuition e.g. when it comes to the definition of Distance. But just imagine you have one feature Genre with 3 values Comedy, Romance and Crime. Then you can model them in a 3-dimensional space by saying Comedy = (1,0,0), Crime = (0,1,0) and Romance = (0,0,1). It replaces 1 feature with three but intuitively works well.
I just understood your question after editing it! It was a bit fuzzy previously. But I keep my initial answer and add an update.
In this case use the values of the feature Genre (unique values of union of all genre sets in that column) as new features and determine their presence with 1 and 0 otherwise. Should work.
Movie_ID Action Thriller Drama Romance Comedy
1 1 1 1 0 0
2 0 0 0 1 1
3 1 0 0 1 0
K-means will work really bad on such data, because the method is designed to process continuous values, where squared errors need to be optimized.
Rather than trying to find a hammer that matches your "nail", you first need to understand your "nail" as it might be a screw! So what is your objective, what is an answer result, and when is a result good? Only then you can find an algorithm to optimize this problem. If you simply try random algorithms, forcing your data into some unnatural form that doesn't preserve the relevant properties, this is a waste of time. They will literally be solving a different problem.
