I have a dataset with approximately 20000 observations consisting of 40 independent and 1 dependent variable. My initial objective is to develop a model that will predict the dependent variable. I have tried several models and applied linear regression and other algorithms such as Random Forests, of course by splitting the dataset into training and testing sets.
Unfortunatelly I cannot get any meaningful results; I have very large errors. I believe there is something "messy" with the dataset, so I have decided to do some clustering first and then apply regression within each cluster. Considering that my dependent variable may exhibit a lot of variation I believe I should do clustering with all variables (dependent and independent), so as each cluster will have similar values of my dependent variable. I have tried to apply Kmeans and I faced several problems. First of all, it seems I cannot identify the right number of clusters. The "elbow" method gives an unclear number and when I use it with less data (about 2000 observations) I get something like this:
I also had similar problems with hierarchical clustering. I have already tried to apply regressions within the clusters identified, but the results are still very poor.
Right now I believe I should possibly use some kind of "weight" to my data, in order to put more weight on the dependent variable when I do clustering, since I believe that this is the problem. Hence, my questions here are:
- is there a way/algorithm where I adjust weights in the variables to be clustered?
Moreover I am confused with two more issues:
data scaling: is it necessary to scale the data before clustering? does this guarantee more accurate results? when do we scale the data?
dimensionality reduction: I have read a lot about principal component analysis and dimensionality reduction, but I am still confused. Again; is this necessary? how many variables are too many to consider applying PCA? are 10 variables too many? or maybe 20? or 50? when should we apply dimensionality reduction? a problem with PCA is that I would still need my original variables to extract the coefficients after regression, while to my understanding with PCA I cannot do that.
This question is more about discussion in order to understand some particular concepts and find a solution to my problem and does not refer to coding issues. Any example and/or references would be appreciated though. I am coding in R.