1
$\begingroup$

My understanding of l2 regularisation:

Weights of the model are assumed to have a prior guassian distribution centered around 0. Then MAP estimate over data adds an extra penalty in cost function.

My problem statement:

I am making a reasonable assumption(based on domain knowledge) that my features are independent which means I can use the weights of the features to infer the importance of features in influencing Y.

From domain knowledge, I want to assume priors about the ratio of weights. Eg:

W1/W2 is a guassian distribution with mean 0.7

W2/W5 is a guassian distribution with mean 2.4

And then MAP estimate over data will give me a cost function, with extra penalty added.

Is my thought process correct? My intent is weights should be close to priors computed from some heuristic. This will also help me handle sparse data.

Implementation Details:

Is there any library(in any language) where it is easy to do this? Or do I have to compute the cost function and it's gradient myself and implement gradient descent over it?

Help with Math:

Also I have started learning stats(and maths in general) more rigorously, but I have lots of ground yet to cover. Assuming all Wi/Wj are given, can someone please give me the new cost function. I will write a program to solve for weights based on that cost function.

EDIT: As I know Wi/Wj, can I assume sum(Wi) = 100 and fit a bayesian logistics regression? Also bayesian logistic regression fit method api requires a hessian matrix. http://bayes-logistic.readthedocs.io/en/latest/usage.html?highlight=fit_bayes_logistic calls hessian matrix covariance matrix of fitted MAP parameters. Assuming that features are independent and variance of a feature to be 1, can I assume hessian matrix to be an identity matrix?

Improve this question
$\endgroup$
4
  • $\begingroup$ this is exactly what I wanted nbviewer.jupyter.org/github/MaxPoint/bayes_logistic/blob/master/…. Guys let me know if there is any flaw in my thought process. $\endgroup$
    – claudius
    Feb 26, 2018 at 7:42
  • $\begingroup$ Only problem here is I know the mean value of Wi/Wj but I don't know the mean value of Wi by itself which is what is required by bayesian logistic regression. $\endgroup$
    – claudius
    Feb 26, 2018 at 8:02
  • $\begingroup$ What are you trying to accomplish? Regression? Classification? $\endgroup$
    – Elias Strehle
    Feb 26, 2018 at 11:26
  • $\begingroup$ the target is binary so classification.... I plan to interpret the weights of logistics regression to get a sense of importance of features in influencing the target $\endgroup$
    – claudius
    Feb 26, 2018 at 14:13

1 Answer 1

Reset to default
0
$\begingroup$

You are describing Bayesian Logistic Regression which allows priors and distributional estimates.

Python's PyMC3 supports Bayesian Logistic Regression with its glm module.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.