Taking a stab:
I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.
Gower Distance is a useful distance metric when the data contains both continuous and categorical variables.
There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.
I wasn't able to find an implementation of Gower Distance in Python when I searched for it about 4-5 months back. So I came up with my own implementation.
import pandas as pd
import numpy as np
from sklearn.neighbors import DistanceMetric
def gower_distance(X):
"""
This function expects a pandas dataframe as input
The data frame is to contain the features along the columns. Based on these features a
distance matrix will be returned which will contain the pairwise gower distance between the rows
All variables of object type will be treated as nominal variables and the others will be treated as
numeric variables.
Distance metrics used for:
Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)
Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)
"""
individual_variable_distances = []
for i in range(X.shape[1]):
feature = X.iloc[:,[i]]
if feature.dtypes[0] == np.object:
feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))
else:
feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / np.ptp(feature.values)
individual_variable_distances.append(feature_dist)
return np.array(individual_variable_distances).mean(0)
The link to the same piece of code: https://github.com/matchado/Misc/blob/master/gower_dist.py
With regards to the clustering technique, I haven't used the ones you've mentioned. But I've used hierarchical clustering in R along with gower distance with success in the past.
Looking into the clustering techniques available in scikit learn, Agglomerative Clustering seems to fit the bill. http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering
I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.
Once you've assigned cluster labels to each row of your data, for each cluster look into the distribution of the features (summary stats for continuous variables & frequency distributions for categorical variables). This is easier to analyze visually if your number of features are manageable (<20 maybe?).
But since you have 100+ features, I suggest a more organized approach. Create a matrix with cluster labels in the columns and the summary stat of the features in the rows (I suggest using median for continuous variable and percentage occurrence of most frequent value in cluster for categorical variable)
It might look something like this.
╔═══════════════════════╦═══════════╦═══════════╦════╦═══════════╗
║ Feature ║ Cluster 1 ║ Cluster 2 ║ … ║ Cluster N ║
╠═══════════════════════╬═══════════╬═══════════╬════╬═══════════╣
║ Numeric feature 1 ║ 15 ║ 37 ║ .. ║ 1 ║
║ Numeric feature 2 ║ 34 ║ 56 ║ … ║ 56 ║
║ Categorical feature 1 ║ 47% ║ 87% ║ … ║ 25% ║
║ … ║ … ║ … ║ … ║ … ║
║ Categorical feature N ║ 25% ║ 91% ║ … ║ 11% ║
║ Numeric feature N ║ 0.2 ║ 0.7 ║ … ║ 0.5 ║
╚═══════════════════════╩═══════════╩═══════════╩════╩═══════════╝