# Classification based on a Clustering Result

Here is the Sample data:

Values    Attribute1    Attribute2    Attribute3    Attribute4
2.5       1980            A            1.5           C
1.8       2000            A            0.4           B
12.4      2017            S            18.5          D
0.4       1987            R            10            P
......


In my real data, I have more than 60 attributes. What I need to do is:

1. Category Values into different groups
2. classify these groups using Attribute1 - Attribute4.

Some of the difficulties are:

1. Values mean something in my application. For example, Values>10 will be put into one group. So, it might be not ideal to use a clustering algorithm such as density based to divide Values into groups.

2. When I use the groups based on Values and all the attributes to run a classification, I might need to choose some important attributes.

What I want is to give Values(meaningful data) to a clustering algorithm. So, that I can get best results/Insights from that Clustering.

So, here are my Questions:

1. How to cluster?
2. How to choose attributes?

I think you need to do some Feature Engineering, i.e., as you explained in the question, those values mean something to your application.

For example : 1-3 : Bad, 4-6 : Average, 7-10 : Good

V1 new_V1
5   Average
7   Good


Something like this, so that clustering algorithm can make sense out of it.

Assuming that you don't have any constraint in using R. If you have both Numeric and Categorical variable(as above) then you can usethis package : ClustMixType, it can understand and do the clustering based on the data fed. As traditional K Means algorithm is not applicable here as it works only for Numeric Data.

If there is any discrepancy in data you can convert them explicitly by new_V1 <- as.factor(new_V1); if it is a numeric variable V2 <- as.numeric(V2) before feeding the data to Clustering algorithm.

Once you get it, then you can use this package: Bruota, use this package to get Predictor Importance(what all variables are important). This can be done only when you know the Target Variable, Most likely your target variable would be the Clustering outcome.

Most likely you would end up getting your desired result, Do let me know if you have any additional questions.

• Thanks a lot for your answer. This really helps. Just one more thing that I need to take into consideration. Now I am not sure how to group Values. For example, 1-3 : Bad, 4-6 : Average, 7-10 : Good in your example is one way to group. 1-5:Bad, 6-10:Good is another possible way. So, different grouping will obviously impact the result of classification. So, how to design a model so that: 1. automatically grouping values; 2. for every grouping, having a classification and performance measure; and 3. compare the performance of every grouping and pick up the best one. – Feng Chen Nov 16 '17 at 4:14
• But it is very subjective question, if you feel that making them into Binary is more sensible is better than multi-nominal, grouping should be done manually for the first iteration, later you can come up with some better solution. That is good process to do but it very tedious. If you do that you will get good quality variables. but doing that for all 60 features is not a feasible solution. – Toros91 Nov 16 '17 at 4:20
• You are right, Probably just try it first to see what it is going on. – Feng Chen Nov 16 '17 at 4:24
• Cool! Do let me know if you have any questions! – Toros91 Nov 16 '17 at 4:29

Since Values has a specific meaning in that problem, you might want to select the categories arbitrarily setting ranges yourself. (using the same example as the above answer, this would look like: 1-3 : Bad, 4-6 : Average, 7-10 : Good). In this case, if the range of answers is not pre-defined (so it's not like high-school grades, but more like people's weight), make sure to use ranges of same size.

Another approach would be to use a distance based algorithm for this purpose (e.g. kmeans), so that the ranges of the clusters won't be stable but automatically selected to minimize distances. It is similar as the t-shirts example in this video (starting 1.35) but with one variable.

In both cases though you will have to set the number of clusters yourself. In the second case this can be done easier by visualizing the data and see if the clusters make sense. Since this is your output variable (and this procedure will be performed only once) I think you should not do this important step automatically but use your judgement before selecting the final grouping.

2. About feature selection: There are plenty feature selection algorithms you can choose from. This is a review paper that can give you some ideas. SVM-RFE (Support Vector Machines with Recursive Feature Elimination) is a very strong combination of feature selection technique with classification that usually works well. You can deal with categorical attributes by turning them into dummy variables.

A Decision Tree Classifier could also do the job. They do the classification based on automatically generated rules, based to some criterion (e.g. entropy). You can see the importance of each attribute for the classification (GINI importance)

There are very good implementation for both suggested approaches in python's scikit learn package:

Food for thought: I don't know what is the exact problem you are trying to solve, but have you thought of using a Decision Tree Regressor directly to the data? You can use Values as the target-variable and the rest of attributes as your dataset. The end-leaves of the created Decision Tree will be the "Values clusters" and each one of the samples will be classified to one of them. (Also available in scikit learn toolbox here)