Group lasso and feature selection

Question

I have a dataset with a lot of categorical variables and a binary target variable and I want to put it to an svm. I converted the categorical variables to dummy variables and since my observations are a lot less than the variables I want to perform feature selection. Since I have categorical variables converted to dummy variables I understand that I cant use simple lasso since it will drop part of the dummy variables.

I'm searching to find a package to implement group lasso on python with a binary target but I cant find any.

I found Adaptive Sparse Group Lasso (asgl) but as far as I understand it doesn't support binary target variables.

I also found group lasso which does sparse group lasso and supports logistic regression. As far as I understand sparse lasso does a combination between group lasso and lasso. I tried using it with parameters values to group_reg=0, or l1_reg=0 hopping that it would just do group lasso but it keeps droping part of the dummy variables in both cases.

My question is. How do you do group lasso with a binary target variable in python.

Thanks in advance.

Answer 1

Presumably, you need a sparse group logistic regression model to perform feature selection while considering the binary response.

skglm is a new modular, scikit-learn conform, python package that provides implementations of sparse generalized linear models. You can make a feature request, on its GitHub repository, through the issue section. The team will reach out to you asap. Thanks to the modularity of the package, it's a matter of a few hours to implement your desired model.

Meanwhile, you can use celer. It offers an efficient implementation of Group Lasso that aligns with the scikit-learn API. Here is also a helpful medium article on celer group Lasso and how it can be used to perform feature selection of continuous and categorical variables.

Answer 2

Lasso regularization is a type of cost function regularization that only works with certain types of machine learning algorithms. For example, lasso regularization does not work with support vector machines (SVMs).

A more common way to frame your problem is feature selection for categorical features in binary classification. A couple of techniques are the chi-squared statistic and the mutual information statistic.