Kmeans clustering with multiple columns containing strings

Question

I have the following dataset: https://www.kaggle.com/carolzhangdc/imdb-5000-movie-dataset

What I want to find is clusters based on imdb score per genre per country. I have created a pandas data frame that contains per country for every unique genre the average imdb rating.

The dataframe looks like this:

country       object
genre         object
avgRating    float64
dtype: object

Since the columns country and genre contain strings, I can't use Kmeans for this.

Is there anyway I can achieve what I want?

Answer 1

After some more research we found this library: https://github.com/nicodv/kmodes.

The library k-modes is used for clustering categorical variables. It defines clusters based on the number of matching categories between data points. (This is in contrast to the more well-known k-means algorithm, which clusters numerical data based on Euclidean distance.) The k-prototypes algorithm combines k-modes and k-means and is able to cluster mixed numerical / categorical data.

Because the dataframe contains categorical data we can't visualize it in a scatterplot. So I added the number representing the cluster the row was assigned to, for every row to get some form of visualization.

Normally you can only cluster ordinal data, because clustering happens based on distance. So I don't know to what extent this is reliable.

Answer 2

You need to represent your categorical data as numerical data. There are different ways to do that(e.g LabelEncoder, OneHotEncoder, replacing the values manually...)

Since the algorithm that is used is KMeans which uses euclidean distance as distance metric, we need numerical values in order to calculate it. Simply put, if you have ['Red','Blue','Green'] in your column you may convert them as [0, 1, 2] for instance...

You may check out this detailed guidance. --> https://pbpython.com/categorical-encoding.html