1
$\begingroup$

A multiple linear regression is to use several predictor variables to predict the outcome of a response variable, like the following relationship:

$y_{i}=\beta_{1}x_{i1}+...+\beta_{p}x_{ip}+\epsilon_{i}, i=1,...,n$

I understand the typical objective to learn the $\beta$ paramters is least-squares, which means to minimize the sum of the sqaure of $\epsilon_{i}$. Now I want other kinds of objective, for example to maximize the Shannon entropy of the sequences of $\epsilon$ (or other self-specified objective). I googled towards this direction but no luck. I am wondering if there is any problem (and tool to solve it if possible) I can look into to do that?

Thank you for your help.

Improve this question
$\endgroup$
10
  • $\begingroup$ Does anyone have a clue? $\endgroup$
    – Kimi
    Jun 29, 2015 at 15:01
  • $\begingroup$ This is too vague...are you interested in the Shannon Entropy specifically? $\endgroup$
    – user9424
    Jun 29, 2015 at 16:34
  • $\begingroup$ @Bey Yes, let's say Shannon Entropy. $\endgroup$
    – Kimi
    Jun 29, 2015 at 20:56
  • $\begingroup$ Ok, well unlike least squares, the Shannon Entropy requires that you be able to assign a probability or density value to each error. Do you want to assume a gaussian error with mean 0 and a pre-specified stdev? Also, do you really want to maximize entropy or minimize entropy? $\endgroup$
    – user9424
    Jun 30, 2015 at 3:23
  • $\begingroup$ @Bey You are right a pdf is needed. But since we get a sequence of error, i.e. {epsilon_i}, can we use emperical distribution? I think it is the simplist and roughest way but please tell me if I am wrong (like using a Gaussian actually help). And I want to maximize entropy. $\endgroup$
    – Kimi
    Jun 30, 2015 at 13:28

1 Answer 1

Reset to default
1
$\begingroup$

This question seems really old, it may be helpful to new visitors.

Yes, we can define any objective function and use gradient descent to minimize it. For that the objective function need to have some properties like

  • It should be convex function else we'll stuck at local minima or may never find the minimum value
  • It must be differentiable on every point and must have non zero derivative. Else gradient descent will not make progress and get stuck

If cost function have these properties then we can use tensorflow (or other library) to create graph to calculate the cost and then to calculate its gradient using auto-diff algorithm and use gradient descent to minimize it.

Auto-diff basically breaks calculation of cost function into elementary arithmetic computation like addition, multiplication, division etc and use chain rule to find derivative.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.