Clustering for mixed numeric and nominal discrete data

Question

My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individual level.

Data is of shape (n=7219, p=105).

Couple things:

• I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data. There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.

• I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.

Answer 1

Taking a stab:

I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.

Gower Distance is a useful distance metric when the data contains both continuous and categorical variables.

There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.

I wasn't able to find an implementation of Gower Distance in Python when I searched for it about 4-5 months back. So I came up with my own implementation.

import pandas as pd
import numpy as np
from sklearn.neighbors import DistanceMetric


def gower_distance(X):
    """
    This function expects a pandas dataframe as input
    The data frame is to contain the features along the columns. Based on these features a
    distance matrix will be returned which will contain the pairwise gower distance between the rows
    All variables of object type will be treated as nominal variables and the others will be treated as 
    numeric variables.
    Distance metrics used for:
    Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)
    Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)
    """
    individual_variable_distances = []

    for i in range(X.shape[1]):
        feature = X.iloc[:,[i]]
        if feature.dtypes[0] == np.object:
            feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))
        else:
            feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / np.ptp(feature.values)

        individual_variable_distances.append(feature_dist)

    return np.array(individual_variable_distances).mean(0)

The link to the same piece of code: https://github.com/matchado/Misc/blob/master/gower_dist.py

With regards to the clustering technique, I haven't used the ones you've mentioned. But I've used hierarchical clustering in R along with gower distance with success in the past.

Looking into the clustering techniques available in scikit learn, Agglomerative Clustering seems to fit the bill. http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering

I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.

Once you've assigned cluster labels to each row of your data, for each cluster look into the distribution of the features (summary stats for continuous variables & frequency distributions for categorical variables). This is easier to analyze visually if your number of features are manageable (<20 maybe?).

But since you have 100+ features, I suggest a more organized approach. Create a matrix with cluster labels in the columns and the summary stat of the features in the rows (I suggest using median for continuous variable and percentage occurrence of most frequent value in cluster for categorical variable)

It might look something like this.

╔═══════════════════════╦═══════════╦═══════════╦════╦═══════════╗
║        Feature        ║ Cluster 1 ║ Cluster 2 ║ …  ║ Cluster N ║
╠═══════════════════════╬═══════════╬═══════════╬════╬═══════════╣
║ Numeric feature 1     ║ 15        ║ 37        ║ .. ║ 1         ║
║ Numeric feature 2     ║ 34        ║ 56        ║ …  ║ 56        ║
║ Categorical feature 1 ║ 47%       ║ 87%       ║ …  ║ 25%       ║
║ …                     ║ …         ║ …         ║ …  ║ …         ║
║ Categorical feature N ║ 25%       ║ 91%       ║ …  ║ 11%       ║
║ Numeric feature N     ║ 0.2       ║ 0.7       ║ …  ║ 0.5       ║
╚═══════════════════════╩═══════════╩═══════════╩════╩═══════════╝

Answer 2

I appended my answer to this question below - you guys essentially asked the same thing.

---

This question seems really about representation, and not so much about clustering.

Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called "color" that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm will think that red (1) is actually closer to blue (2) than it is to yellow (3). We need to use a representation that lets the computer understand that these things are all actually equally different.

One simple way is to use what's called a one-hot representation, and it's exactly what you thought you should do. Rather than having one variable like "color" that can take on three values, we separate it into three variables. These would be "color-red," "color-blue," and "color-yellow," which all can only take on the value 1 or 0.

This increases the dimensionality of the space, but now you could use any clustering algorithm you like. It does sometimes make sense to zscore or whiten the data after doing this process, but the your idea is definitely reasonable.

Answer 3

The distance metric implemented by @gregorymatchado has one bug. For Numeric attributes, range will give NaN for same values throughout. For that we need a change use max(np.ptp(feature.values),1) instead of np.ptp(feature.values). Complete code below :

import pandas as pd
import numpy as np
from sklearn.neighbors import DistanceMetric


def gower_distance(X):
"""
This function expects a pandas dataframe as input
The data frame is to contain the features along the columns. Based on these features a
distance matrix will be returned which will contain the pairwise gower distance between the rows
All variables of object type will be treated as nominal variables and the others will be treated as 
numeric variables.
Distance metrics used for:
Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)
Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)
"""
individual_variable_distances = []

for i in range(X.shape[1]):
    feature = X.iloc[:,[i]]
    if feature.dtypes[0] == np.object:
        feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))
    else:
        feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / max(np.ptp(feature.values),1)

    individual_variable_distances.append(feature_dist)

return np.array(individual_variable_distances).mean(0)

Answer 4

I think you have bug, too. If feature vector has very small scale. then your distance is useless. So, I would convert like following:

epsilon = 10**(-8)
... / max(np.ptp(feature.values), epsilon)