I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects.

Now to build a model, I don't like to train the usual way (where we build a generalised model).

Instead I would like to group similar patients together like group1, group2, group3, group4 etc.

And when I new subject to predict, I would like to know which group does he belong to and find out the important predictors for each group?

I know this sounds like K-means clustering but am I right to understand that?

Can anybody provide me the list of steps (pseudocode type) on how to do this? Sorry, am new to ML and exploring various ways to do a classification problem


Clustering would indeed give you something like group1, group2, etc., i.e. it would assign every instance to a group meant to represent similar instances together. With the model you could also assign any new instance to one of the groups in the model, i.e. find the group that this instance is the most similar to. In general clustering doesn't give you which predictors are the most important for a group (there might be ways to calculate this from the model, but afaik it's not standard).

Note that with clustering alone you can't predict the target variable for a new subject. However you could obtain the proportion of positive/negative for every group, i.e. the chances of being positive/negative knowing that the instance belongs to a particular group. You could also do the opposite: chances of belonging to a particular group knowing whether the instance is positive/negative.

An important question is whether you want to include the target variable in the features used for clustering:

  • If it's included, the model uses it like any other feature to group instances by their similarity.
  • If it's not included, the model itself doesn't know it so the proportion of positive/negative in a cluster doesn't depend on it. This might be more insightful, depending on what you want to obtain. Another advantage of not including it is that you can still use the outcome of the clustering for training a binary classification model afterwards, typically by adding the cluster an instance belongs to as a feature.

As you can see, there are a few important design decisions to make depending on the goal.

[edit: answer to comment]

In order to use the clustering as a basis for classifying any new instance, here are a few general options:

  • Basic option: predict the cluster the instance belongs to, then just output the probability of positive of the cluster.
  • Clustering model + classification model: train a classifier with all the training instances, with one additional categorical feature corresponding to the cluster the instance belongs to. The classification might ge improved compared to only the original features.
    • Variant: add one feature for every possible cluster, where the value represents how similar the instance is to each cluster (most clustering methods can calculate a distance or similarity against any cluster).
  • Multiple independent classification models: based on the clustering, train a classification model for every cluster independently, using only the instances from this cluster. Advantage: this allows the classification model to take into account more fine-grained patterns within this cluster. Disadvantage: the model has less instances to be trained from, especially for small clusters.
  • $\begingroup$ Thanks. Upvoted. One quick question. Let's say I don't include the target variable. Now the subjects might be grouped based on their similarity (of all features) except the target variable. Later I will find out the proportion of positive and negative in each group... Now my question. Let's say I have a new instance fed to the model...I want this subject's prediction to be based on other instances which are similar to him (in his group).. Not across groups... In short, this is what am referring to sciencedirect.com/science/article/pii/S1532046418301072 $\endgroup$ – The Great Jan 9 at 1:28
  • $\begingroup$ @TheGreat Yes, that would be a good approach I think. I'm not going to read the full paper but I can outline a few strategies, I'll edit my answer. $\endgroup$ – Erwan Jan 9 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.