A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship:

$y_{i}=\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+\epsilon_{i}, i=1,...,n$

I understand the typical objective to learn the $\beta$ paramters is least-squares, which means to minimize the sum of the sqaure of $\epsilon_{i}$. Now I want other kinds of objective, for example to maximize the Shannon entropy of the sequences of $\epsilon$ (or other self-specified objective). I googled towards this direction but no luck. I am wondering if there is any problem (and tool to solve it if possible) I can look into to do that?

Thank you for your help.

  • $\begingroup$ Does anyone have a clue? $\endgroup$ – Kimi Jun 29 '15 at 15:01
  • $\begingroup$ This is too vague...are you interested in the Shannon Entropy specifically? $\endgroup$ – user9424 Jun 29 '15 at 16:34
  • $\begingroup$ @Bey Yes, let's say Shannon Entropy. $\endgroup$ – Kimi Jun 29 '15 at 20:56
  • $\begingroup$ Ok, well unlike least squares, the Shannon Entropy requires that you be able to assign a probability or density value to each error. Do you want to assume a gaussian error with mean 0 and a pre-specified stdev? Also, do you really want to maximize entropy or minimize entropy? $\endgroup$ – user9424 Jun 30 '15 at 3:23
  • $\begingroup$ @Bey You are right a pdf is needed. But since we get a sequence of error, i.e. {epsilon_i}, can we use emperical distribution? I think it is the simplist and roughest way but please tell me if I am wrong (like using a Gaussian actually help). And I want to maximize entropy. $\endgroup$ – Kimi Jun 30 '15 at 13:28

This question seems really old, it may be helpful to new visitors.

Yes, we can define any objective function and use gradient descent to minimize it. For that the objective function need to have some properties like

  • It should be convex function else we'll stuck at local minima or may never find the minimum value
  • It must be differentiable on every point and must have non zero derivative. Else gradient descent will not make progress and get stuck

If cost function have these properties then we can use tensorflow (or other library) to create graph to calculate the cost and then to calculate its gradient using auto-diff algorithm and use gradient descent to minimize it.

Auto-diff basically breaks calculation of cost function into elementary arithmetic computation like addition, multiplication, division etc and use chain rule to find derivative.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.