# Custom regularisation for logistics regression

My understanding of l2 regularisation:

Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function.

My problem statement:

I am making a reasonable assumption(based on domain knowledge) that my features are independent which means I can use the weights of the features to infer the importance of features in influencing Y.

From domain knowledge, I want to assume priors about the ratio of weights. Eg:

W1/W2 is a guassian distribution with mean 0.7

W2/W5 is a guassian distribution with mean 2.4

And then MAP estimate over data will give me a cost function, with extra penalty added.

Is my thought process correct? My intent is weights should be close to priors computed from some heuristic. This will also help me handle sparse data.

Implementation Details:

Is there any library(in any language) where it is easy to do this? Or do I have to compute the cost function and it's gradient myself and implement gradient descent over it?

Help with Math:

Also I have started learning stats(and maths in general) more rigorously, but I have lots of ground yet to cover. Assuming all Wi/Wj are given, can someone please give me the new cost function. I will write a program to solve for weights based on that cost function.

EDIT: As I know Wi/Wj, can I assume sum(Wi) = 100 and fit a bayesian logistics regression? Also bayesian logistic regression fit method api requires a hessian matrix. http://bayes-logistic.readthedocs.io/en/latest/usage.html?highlight=fit_bayes_logistic calls hessian matrix covariance matrix of fitted MAP parameters. Assuming that features are independent and variance of a feature to be 1, can I assume hessian matrix to be an identity matrix?

• this is exactly what I wanted nbviewer.jupyter.org/github/MaxPoint/bayes_logistic/blob/master/…. Guys let me know if there is any flaw in my thought process. Commented Feb 26, 2018 at 7:42
• Only problem here is I know the mean value of Wi/Wj but I don't know the mean value of Wi by itself which is what is required by bayesian logistic regression. Commented Feb 26, 2018 at 8:02
• What are you trying to accomplish? Regression? Classification? Commented Feb 26, 2018 at 11:26
• the target is binary so classification.... I plan to interpret the weights of logistics regression to get a sense of importance of features in influencing the target Commented Feb 26, 2018 at 14:13