8
$\begingroup$

My data includes survey responses that are binary (numeric) and nominal / categorical. All responses are discrete and at individuals level. Snapshot of data

Data is of shape (n=7219, p=105).

Couple things:

  • I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data. There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.

  • I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.

$\endgroup$
  • $\begingroup$ I don't think any clustering will return meaningful results on such data. Make sure to validate your findings. Also consider implementing an algorithm yourself, and contributing it to sklearn. But you can try to use e.g. DBSCAN with dice coefficient or another distance function for binary/categorial data. $\endgroup$ – Anony-Mousse Nov 2 '15 at 7:08
  • 1
    $\begingroup$ It is common to convert categorical to numeric in these cases. See here scikit-learn.org/stable/modules/generated/…. Doing this you will now only have binary values in your data, so will not have scaling issues with clustering. You can now try a simple k-means. $\endgroup$ – user13684 Nov 2 '15 at 14:16
  • $\begingroup$ Perhaps this approach would be useful: zeszyty-naukowe.wwsi.edu.pl/zeszyty/zeszyt12/… $\endgroup$ – user14742 Dec 14 '15 at 19:29
  • $\begingroup$ You should start from the simplest solution, by trying the convert the categorical to one-hot-encoding representations as noted above. $\endgroup$ – geompalik Sep 10 '16 at 18:07
  • $\begingroup$ This is the subject of my doctoral thesis prepared in 1986 at the IBM France Scientific Center and the Pierre et Marie Currie University (Paris 6) entitled new techniques of coding and association in automatic classification. In this thesis I proposed data coding techniques called Triordonnance to classify a set described by numerical, qualitative and ordinal variables. $\endgroup$ – Said Chah slaoui Nov 16 '16 at 21:21
6
$\begingroup$

Taking a stab:

I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.

Gower Distance is a useful distance metric when the data contains both continuous and categorical variables.

There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.

I wasn't able to find an implementation of Gower Distance in Python when I searched for it about 4-5 months back. So I came up with my own implementation.

import pandas as pd
import numpy as np
from sklearn.neighbors import DistanceMetric


def gower_distance(X):
    """
    This function expects a pandas dataframe as input
    The data frame is to contain the features along the columns. Based on these features a
    distance matrix will be returned which will contain the pairwise gower distance between the rows
    All variables of object type will be treated as nominal variables and the others will be treated as 
    numeric variables.
    Distance metrics used for:
    Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)
    Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)
    """
    individual_variable_distances = []

    for i in range(X.shape[1]):
        feature = X.iloc[:,[i]]
        if feature.dtypes[0] == np.object:
            feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))
        else:
            feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / np.ptp(feature.values)

        individual_variable_distances.append(feature_dist)

    return np.array(individual_variable_distances).mean(0)

The link to the same piece of code: https://github.com/matchado/Misc/blob/master/gower_dist.py

With regards to the clustering technique, I haven't used the ones you've mentioned. But I've used hierarchical clustering in R along with gower distance with success in the past.

Looking into the clustering techniques available in scikit learn, Agglomerative Clustering seems to fit the bill. http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering

I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.

Once you've assigned cluster labels to each row of your data, for each cluster look into the distribution of the features (summary stats for continuous variables & frequency distributions for categorical variables). This is easier to analyze visually if your number of features are manageable (<20 maybe?).

But since you have 100+ features, I suggest a more organized approach. Create a matrix with cluster labels in the columns and the summary stat of the features in the rows (I suggest using median for continuous variable and percentage occurrence of most frequent value in cluster for categorical variable)

It might look something like this.

╔═══════════════════════╦═══════════╦═══════════╦════╦═══════════╗
║        Feature        ║ Cluster 1 ║ Cluster 2 ║ …  ║ Cluster N ║
╠═══════════════════════╬═══════════╬═══════════╬════╬═══════════╣
║ Numeric feature 1     ║ 15        ║ 37        ║ .. ║ 1         ║
║ Numeric feature 2     ║ 34        ║ 56        ║ …  ║ 56        ║
║ Categorical feature 1 ║ 47%       ║ 87%       ║ …  ║ 25%       ║
║ …                     ║ …         ║ …         ║ …  ║ …         ║
║ Categorical feature N ║ 25%       ║ 91%       ║ …  ║ 11%       ║
║ Numeric feature N     ║ 0.2       ║ 0.7       ║ …  ║ 0.5       ║
╚═══════════════════════╩═══════════╩═══════════╩════╩═══════════╝
$\endgroup$
  • $\begingroup$ Solid answer, nicely done. $\endgroup$ – Astrid Mar 19 '18 at 16:30
2
$\begingroup$

I appended my answer to this question below - you guys essentially asked the same thing.


This question seems really about representation, and not so much about clustering.

Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called "color" that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm will think that red (1) is actually closer to blue (2) than it is to yellow (3). We need to use a representation that lets the computer understand that these things are all actually equally different.

One simple way is to use what's called a one-hot representation, and it's exactly what you thought you should do. Rather than having one variable like "color" that can take on three values, we separate it into three variables. These would be "color-red," "color-blue," and "color-yellow," which all can only take on the value 1 or 0.

This increases the dimensionality of the space, but now you could use any clustering algorithm you like. It does sometimes make sense to zscore or whiten the data after doing this process, but the your idea is definitely reasonable.

$\endgroup$
1
$\begingroup$

The distance metric implemented by @gregorymatchado has one bug. For Numeric attributes, range will give NaN for same values throughout. For that we need a change use max(np.ptp(feature.values),1) instead of np.ptp(feature.values). Complete code below :

import pandas as pd
import numpy as np
from sklearn.neighbors import DistanceMetric


def gower_distance(X):
"""
This function expects a pandas dataframe as input
The data frame is to contain the features along the columns. Based on these features a
distance matrix will be returned which will contain the pairwise gower distance between the rows
All variables of object type will be treated as nominal variables and the others will be treated as 
numeric variables.
Distance metrics used for:
Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)
Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)
"""
individual_variable_distances = []

for i in range(X.shape[1]):
    feature = X.iloc[:,[i]]
    if feature.dtypes[0] == np.object:
        feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))
    else:
        feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / max(np.ptp(feature.values),1)

    individual_variable_distances.append(feature_dist)

return np.array(individual_variable_distances).mean(0)
$\endgroup$
0
$\begingroup$

I think you have bug, too. If feature vector has very small scale. then your distance is useless. So, I would convert like following:

epsilon = 10**(-8)
... / max(np.ptp(feature.values), epsilon)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.