# How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)

I came through this questions and I failed to find the right answer for it.

How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)?

a- Creating different models for different cluster groups.
b- Creating an input feature for cluster ids as an ordinal variable.
c- Creating an input feature for cluster centroids as a continuous variable.
d- Creating an input feature for cluster size as a continuous variable.


Just an idea:

You may be able to cluster a continuous variable and add the clusters as indicator in a linear regression jointly with an interaction term.

Say your linear model is income (inc) explained by years of job experience (exp)...

inc = b0 + b1*exp + u.

Now say you have two groups (low/high skilled workers). If you cluster inc in two groups, and add this as a dummy with interaction term, you may be able to increase the fit of the model.

inc = b0 + b1*exp + b2*indicator + b3*exp*indicator + u.

The idea is that low/high skilled workers will increase income contingent on experience in a different way (different slope). However, this approach requires that you can identify clusters well.

Edit: same strategy may work with clustering the X.

• But how would you find the indicator? It seems to require the income that you’re trying to predict?
– Paul
May 30 '19 at 20:36
• you use unsupervised on the y of your supervised dataset to try to get more information out of it May 30 '19 at 20:41
• Exactly - but when you’re going to use your regression you only have a new example of x, so how do you know what value to put for the indicator in your formula?
– Paul
May 30 '19 at 20:42
• True. I understood you in a way that you only want to improve fit. However, same strategy may work with clustering X. Just make sure you use indicators and interaction terms. May 30 '19 at 20:45
• How is the indicator defined? If you have for example 4 clusters?
– Paul
May 30 '19 at 20:51

I would propose to cluster the examples based on the features that you know (so that excludes the variable you are trying to predict) and then apply linear regression to each cluster independently. So you are essentially permitting different linear relations between the input features and output variable, depending on the cluster.

• When you do separate regressions you actually throw away information. Adding indicators and interaction terms is able to differentiate the clusters by a separate intercept and slope, but uses all information in one regression. May 30 '19 at 20:48

Think about it as if the data set is mix of more then one source (in term of regression formula.. where each could regress differently). Clustering will help you separate them back. This approach is tricky, where you need to focus on how cluster-able are they? Please consider the maximise outer distance and minimise inner distance approach.

You can check this post https://stackoverflow.com/questions/19197715/scikit-learn-k-means-elbow-criterion