Handling a feature containing multiple values

Question

I have a dataset in following format:

---

Movie ID | Actors | Director | language | ReleaseYear | Genre

1 | Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran | Vidhu Vinod Chopra | hin | 1994 | Drama;Romance;Patriotic

---

As you can see the columns Actors and Genre has multiple values. I need to perform cluster analysis on this dataset. I don't know what would the best way to deal with such data. I am thinking of two possible solution(don't know if it is a correct approach for this type of problem)--

1. Split the column, say, 'Actors' into multiple columns e.g. Actors 1, Actor2,..., and then perform the cluster anylysis.
2. Row split i.e. convert a single row of data into multiple rows each row for one value of columns like 'Actors' and 'Genre'.

Please suggest me the best way to deal with such data for cluster analysis.

Answer 1

You need to rethink your approach. Rather than "what can I code to make things work", you need to ask "what is the right thing to do, and how can I implement this".

Clustering is hard (easy to run, hard to get good results).

Clustering non-continuous data is even harder.

The reason why people create a lot of false results here is because you have infinitely many way to weight different values and attributes, and essentially you can get almost any result you want just by playing around with parameters. Don't let yourself be lured by the common hack of "one hot encoding" everything; it just shows that people don't (want to) know what they are doing.

To get a reliable result, you need to be very clear on your assumptions, such as "I assume all actors are equally representative, and the overlap as measured by Jaccard is a good indicator of similarity" (I would disagree here, actors should be weighted). Then you need to do the same thing with genres. Here it is even more questionable hos to compute similarity of genres. And after that, you need to combine all these different similarities into one. That is probably the hardest step, and will involve even more weighting parameters.

All in all, I'd say: whatever you do, the clustering is not statistically sound. It's too many parameters chosen without even a good reason (yet a proof) that this way is better than another way of choosing them. In particular, you will never know if there is an even better clustering with different parameters.

Answer 2

You should create multiple new columns - one column per actor and then set it to 1 if respective actor is present in the movie, 0 otherwise. Same for genres. It is called one hot encoding and ensures that data is treated as categorical instead of continuos.

Answer 3

Going with one hot encoding will work but it will increase your dimensions. What I suggest is that you should replace the label by its frequency. And if you want to use one hot encoding then you first find the labels that are rare in your data and combine all of them into one label and then apply one hot encoding, this will help you to control your dimensionality.

Answer 4

You can try One hot encoding (Dummy variables). If you are on Python, Pandas library has get_dummies() function.