1
$\begingroup$

Suppose I have a movies dataframe in pandas. One of the features is Genre.

It has a list of genre names. for example:

Movie_ID    Genre

1        [Action, Thriller, Drama]
2        [Romance, Comedy]
3        [Action, Romance]

How do I use this column for a clustering problem, say K-means clustering?

$\endgroup$
2
$\begingroup$

You can easily use get_dummy function in Pandas to convert them to numerical vectors.

The idea is that categorical variables do not have a numerical intuition e.g. when it comes to the definition of Distance. But just imagine you have one feature Genre with 3 values Comedy, Romance and Crime. Then you can model them in a 3-dimensional space by saying Comedy = (1,0,0), Crime = (0,1,0) and Romance = (0,0,1). It replaces 1 feature with three but intuitively works well.

Update

I just understood your question after editing it! It was a bit fuzzy previously. But I keep my initial answer and add an update.

In this case use the values of the feature Genre (unique values of union of all genre sets in that column) as new features and determine their presence with 1 and 0 otherwise. Should work.

Movie_ID    Action  Thriller  Drama  Romance  Comedy

1             1       1         1       0       0
2             0       0         0       1       1
3             1       0         0       1       0
| improve this answer | |
$\endgroup$
0
$\begingroup$

K-means will work really bad on such data, because the method is designed to process continuous values, where squared errors need to be optimized.

Rather than trying to find a hammer that matches your "nail", you first need to understand your "nail" as it might be a screw! So what is your objective, what is an answer result, and when is a result good? Only then you can find an algorithm to optimize this problem. If you simply try random algorithms, forcing your data into some unnatural form that doesn't preserve the relevant properties, this is a waste of time. They will literally be solving a different problem.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.