multivariate clustering, dimensionality reduction and data scalling for regression

Question

I have a dataset with approximately 20000 observations consisting of 40 independent and 1 dependent variable. My initial objective is to develop a model that will predict the dependent variable. I have tried several models and applied linear regression and other algorithms such as Random Forests, of course by splitting the dataset into training and testing sets.

Unfortunatelly I cannot get any meaningful results; I have very large errors. I believe there is something "messy" with the dataset, so I have decided to do some clustering first and then apply regression within each cluster. Considering that my dependent variable may exhibit a lot of variation I believe I should do clustering with all variables (dependent and independent), so as each cluster will have similar values of my dependent variable. I have tried to apply Kmeans and I faced several problems. First of all, it seems I cannot identify the right number of clusters. The "elbow" method gives an unclear number and when I use it with less data (about 2000 observations) I get something like this:

I also had similar problems with hierarchical clustering. I have already tried to apply regressions within the clusters identified, but the results are still very poor.

Right now I believe I should possibly use some kind of "weight" to my data, in order to put more weight on the dependent variable when I do clustering, since I believe that this is the problem. Hence, my questions here are:

1. is there a way/algorithm where I adjust weights in the variables to be clustered?

Moreover I am confused with two more issues:

2. data scaling: is it necessary to scale the data before clustering? does this guarantee more accurate results? when do we scale the data?

3. dimensionality reduction: I have read a lot about principal component analysis and dimensionality reduction, but I am still confused. Again; is this necessary? how many variables are too many to consider applying PCA? are 10 variables too many? or maybe 20? or 50? when should we apply dimensionality reduction? a problem with PCA is that I would still need my original variables to extract the coefficients after regression, while to my understanding with PCA I cannot do that.

This question is more about discussion in order to understand some particular concepts and find a solution to my problem and does not refer to coding issues. Any example and/or references would be appreciated though. I am coding in R.

Answer 1

Great question, I will try to answer the aspects related to dimensionality reduction mentioned above. $Dimensionality\: Reduction:$ The number of dimensions which you want to keep after doing PCA is an experimental value you can experiment with the number of dimensions and check you results. Although you have mentioned all 40 features are independent i would still ask you to do a correlation analysis of the variables.

Pca removes these correlated features and gives you a set of features which amount to explain most of your data. One advantage of dimensionality reduction is in regression. Correlated features often are a cause of multicollinearity. Doing a dimensionality reduction helps us get rid of this problem. Also once we have a reduced set of features we can apply the cluster analysis. The reason is K-means calculates the l2- distance between data points. in very high dimensions the concept of euclidean distance becomes less useful because of the curse of dimensionality. (probably the reason of the problems with your elbow curve.)

Hence bringing down the number of features using pca and clustering later will give a better idea of the groupings of the data. The problem which you mention about PCA can be solved in fact it is not an issue at all. You can do regression with the features obtained by the data and only take the coefficients of the reduced feature set.Some other techniques which you might want to look at are regularization in regression. For example L1-regularization helps in feature selection and helps with correlated variables.

Answer 2

I will try to answer to your questions one by one. Unfortunately I am not familiar with R, but I will provide some python links whenever relevant - even if you won't use the code you will be able to find there more details on the suggested methods.

1. I suggest you use a Decision Tree Regressor for the clustering. It gives you the ability to set what the target variable is and you can directly select as optimization criterion the minimization of Mean Square Error. You can select as stopping criterion a maximum number of levels according to the number of clusters/MSE that you find acceptable.

   The following picture summarizes the logic behind tree-based hierarchical clustering and also shows that you usually have to do some pruning (some samples might be too different to be assigned to any cluster). The picture comes from this link, where hierarchical clustering is explained; the optimisation criterion there is distance as it fits better with the type of data they are using, but of course you can use the one that fits to your own data.

   

   In python, this can been done using scikit-learn.DecisionTreeRegressor and you can even plot the tree using export_graphviz function (you can read some interesting information about the advantages and limitations of Decision Trees here). For better understanding, the tree below shows an example of clustering results using a Decision Tree Regressor. I have highlighted the end-clusters with red.

2. Yes, it is advisable to do some kind of data scaling before proceeding with the clustering. This way, you ensure that the clustering results are not biased by the scale of the variables (e.g. a variable that takes large values might influence certain clustering algorithm results more that one with smaller values). Keep in mind though that many clustering algorithms that come from analysis toolkits anyway scale data automatically before clustering (at least in python). There are 2 alternatives for scaling: standardisation and normalisation; you need to choose which one of those fits better to the current problem and clustering algorithm.

   • Standardization transforms the data in order to have zero mean and unit variance.
   • Normalisation rescales the data into the range [0,1].

   You can read more about standardisation and normalisation here, while here you can find some very good examples about their use.

3. As far as I know there is no trustworthy "rule of thumb" regarding the number of features for each problem, this is something that depends a lot on the nature of the problem and most importantly on the nature of the features. It is always important though to make sure that the input variables are uncorrelated with each other. PCA is a good way to ensure that - it actually applies some orthogonal transformation to your set of features in order to produce a set of linearly non-correlated variables based on the initial set of features - the new features are ordered based on the amount of information they contain. @Shubham summarised very well the logic behind pca, so I don't have much to add.