I used clustering on my dataset. Now when I'm trying to use a LASSO with cv to predict a response, one of the variables it takes into consideration is which cluster a new point is classified into.(I included the cluster variable as a predictor to see if being in a particular group affects the response) Since the information on all variables is already captured by the cluster variable,using it again in the Lasso model with some other variables,does it become redundant/biased?


Since the information on all variables is already captured by the cluster variable, using it again in the Lasso model with some other variables, does it become redundant/biased?

It does not become biased, at least not in the statistical sense. Depending on the clustering mechanism, the information may be redundant.

If you're using regression with the RMSE as the loss function and the squared-error as your loss metric for your clustering algorithm, then the information will work out to be redundant.

Regardless, it's harmless to throw into the model and test this. The LASSO should learn that this information is redundant.

Here's a simulation in R.


n <- 1e5
nclusters <- 5
rmse <- function(y, yhat){
  return (sqrt( sum( (y-yhat)**2 )))
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.-1,data=ls)


# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- runif(dim(xs)[2])

# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)

# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.-1, data = data.frame(cluster = factor(clusterpred[,1])))
# Concatenating the original features and the assigned clusters
totalxs <- cbind2(xs, cxs)


# Setting alpha = 1 implies LASSO for the GLMNET Package
model <- cv.glmnet(y=ys, x=xs, alpha=1)
cmodel <- cv.glmnet(y=ys, x=cxs, alpha=1)

# Running on the clusters and the original features
totalmodel <- cv.glmnet(y=ys, x=totalxs, alpha=1)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
yhatt <- predict(totalmodel, totalxs)

# Looking at the difference RMSEs 
print(rmse(ys, yhat))
print(rmse(ys, yhatc))
print(rmse(ys, yhatt))

# It seems to select most of the features and *one* cluster
print(coef(totalmodel, s='lambda.min'))

In this simulation, we see that the performance of the model slightly worse, but fairly similar, from including the clusters.


I think that by doing so, you have expanded the feature space, and I don't think that this additional variable is redundant.

The cluster categorical variable is a linear combination of the other variables. When you apply Lasso on the dataset, the categorical variable undergoes one-hot encoding and Lasso would pick a subset of values given a level of regularisation. Suppose in the best performing regularisation parameter, Lasso picked clusters 1,3,7 out of clusters 1 - 10; These pockets of clusters might not be picked up using just Lasso on the other variables without the cluster variable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.