Classification based on a Clustering Result

Question

Here is the Sample data:

Values    Attribute1    Attribute2    Attribute3    Attribute4
  2.5       1980            A            1.5           C
  1.8       2000            A            0.4           B
  12.4      2017            S            18.5          D
  0.4       1987            R            10            P
......

In my real data, I have more than 60 attributes. What I need to do is:

1. Category Values into different groups
2. classify these groups using Attribute1 - Attribute4.

Some of the difficulties are:

1. Values mean something in my application. For example, Values>10 will be put into one group. So, it might be not ideal to use a clustering algorithm such as density based to divide Values into groups.

2. When I use the groups based on Values and all the attributes to run a classification, I might need to choose some important attributes.

What I want is to give Values(meaningful data) to a clustering algorithm. So, that I can get best results/Insights from that Clustering.

So, here are my Questions:

1. How to cluster?
2. How to choose attributes?

Answer 1

I think you need to do some Feature Engineering, i.e., as you explained in the question, those values mean something to your application.

For example : 1-3 : Bad, 4-6 : Average, 7-10 : Good

V1 new_V1
3   Bad
5   Average
7   Good

Something like this, so that clustering algorithm can make sense out of it.

Assuming that you don't have any constraint in using R. If you have both Numeric and Categorical variable(as above) then you can usethis package : ClustMixType, it can understand and do the clustering based on the data fed. As traditional K Means algorithm is not applicable here as it works only for Numeric Data.

If there is any discrepancy in data you can convert them explicitly by new_V1 <- as.factor(new_V1); if it is a numeric variable V2 <- as.numeric(V2) before feeding the data to Clustering algorithm.

Once you get it, then you can use this package: Bruota, use this package to get Predictor Importance(what all variables are important). This can be done only when you know the Target Variable, Most likely your target variable would be the Clustering outcome.

Implementation is over here, please follow this Link

Most likely you would end up getting your desired result, Do let me know if you have any additional questions.

Answer 2

1. About clustering of Values

Since Values has a specific meaning in that problem, you might want to select the categories arbitrarily setting ranges yourself. (using the same example as the above answer, this would look like: 1-3 : Bad, 4-6 : Average, 7-10 : Good). In this case, if the range of answers is not pre-defined (so it's not like high-school grades, but more like people's weight), make sure to use ranges of same size.

Another approach would be to use a distance based algorithm for this purpose (e.g. kmeans), so that the ranges of the clusters won't be stable but automatically selected to minimize distances. It is similar as the t-shirts example in this video (starting 1.35) but with one variable.

In both cases though you will have to set the number of clusters yourself. In the second case this can be done easier by visualizing the data and see if the clusters make sense. Since this is your output variable (and this procedure will be performed only once) I think you should not do this important step automatically but use your judgement before selecting the final grouping.

2. About feature selection: There are plenty feature selection algorithms you can choose from. This is a review paper that can give you some ideas. SVM-RFE (Support Vector Machines with Recursive Feature Elimination) is a very strong combination of feature selection technique with classification that usually works well. You can deal with categorical attributes by turning them into dummy variables.

A Decision Tree Classifier could also do the job. They do the classification based on automatically generated rules, based to some criterion (e.g. entropy). You can see the importance of each attribute for the classification (GINI importance)

There are very good implementation for both suggested approaches in python's scikit learn package:

• SVM and RFE
• Decision Tree Classifier

Food for thought: I don't know what is the exact problem you are trying to solve, but have you thought of using a Decision Tree Regressor directly to the data? You can use Values as the target-variable and the rest of attributes as your dataset. The end-leaves of the created Decision Tree will be the "Values clusters" and each one of the samples will be classified to one of them. (Also available in scikit learn toolbox here)