2
$\begingroup$

I have a data-frame which has looks similar to this:

A   B   C
1   2   2
2   4   3
4   8   5
9   16  7
16  32  11
22  43  14
28  55  17
34  67  20
40  79  23

A,B and C can be assumed to be the features in machine learning literature. I have read maximum likelihood estimation for 1 variable assuming Gaussian distribution.

The equation is something like, where xi's are each data-point:

enter image description here

Where x1,x2....xn are n data points each having dimension 3. If we assume p(x) to be gaussian, then we can use the Gaussian Normal distribution equation as:

enter image description here

This is well understandable if we have only 1 feature.

How can I generalise the above normal distribution equation when we have more than 1 features, here we have 3 features? Can someone help me to write the maximum likelihood for the above data-frame?

Do we learn mu and sigma for each features A,B and C that is total of 6 learnable parameters?

If we have 3 different distribution say Normal, exponential and so on for columns A,B and C then how does the MLE equation looks like over entire-data-frame?

If we do argmax of equation 1, we don't require the ground truth for it. right? We are just maximising the equation?

$\endgroup$
1
  • 1
    $\begingroup$ You also need a functional form of the predictors that relate to the mean. For example, in a traditional Bayesian linear regression you set mu = intercept + b1x1 + b2x2 + b3x3 + .... Then in the likelihood mu simply gets substituted for this expression. Sigma is just held constant and given say, a half cauchy or flat prior. $\endgroup$
    – aranglol
    Oct 11, 2020 at 17:05

1 Answer 1

0
$\begingroup$

First, I'd like to clarify. The maximum likelihood function you gave there is NOT with respect to "one feature". Or at the very least, it's not meaningful to think of this as a "feature" because in your example, you would actually evaluate the loglikelihood function at observed values of y not x. You aren't using your variables in x to explain anything about y. The resulting maximum likelihood estimate you would get for $\mu$ would be $\bar x$, the sample mean of your target variable y.

Now, suppose you actually do want to use x (your features) to predict y as in the case of supervised learning. Then, as I alluded to in your comments you need to specify two things - the predictor or model function, denoted typically as $\hat f(X)$ and the "link function", denoted as $g$.

$\hat f(X)$ is a function of your predictor variables such that:

$$g(E[Y|X]) = \hat f(X)$$

In your case, $E[Y | X] = \mu $ since you have a Normal distribution. Hence,

$$g(\mu) = \hat f(X) \rightarrow \mu = g^{-1}(\hat f(X))$$

Now, in terms of choosing $\hat f(X) $, this depends on your goals and how complicated you wish to go. Regardless, this is a function that can take on any real number. In the standard case (say in linear regression) you set $\hat f(X) = B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}.$ There are other examples where writing out $\hat f(X)$ is impossible or tiresome, for example in the case of gradient boosting trees or deep neural networks. Other algorithms may set $\hat f(X) = B_{0} + h_1(x_{1}) + h_2(x_{2}) + h_3(x_{3})$ where $h_{i}$ are smooth functions. It really depends on again how complicated you wish to get and how interpretable you need your models to be.

With respect to g, the "link function", this is almost always chosen depending on the range of the response variable or the range of the parameter that you are linking with ($\mu$ in your case). In your case, since the normal distribution can take on any real number, most of the time $g$ is chosen as the identity function since $\hat f(X)$ naturally can take on any real number already. This leads to:

$$\mu = B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}.$$

Finally, when dealing with your likelihood function:

$$p(Y) = constant * \prod_{i = 1}^{n} e^{\frac{-1}{2}\frac{(y_{i} - (B_{0} + B_{1}X_{1} + B_{2}X_{2} + B_{3}X_{3}))^2}{\sigma^2}} $$

Solving this equation will provide you with the ordinary least squares estimates which I am sure you have seen before. Of course, choosing a different $\hat f(X)$ or a different $g$ will likely change everything and often you may not even get nice-looking closed-form solutions that come out with ordinary least squares. This motivates different numerical optimization methods. However, the "ingredients" are the same.

I hope this helps.

$\endgroup$
4
  • $\begingroup$ Ok. But I see many blogs where they take one variable say x and try to fit some distribution on it using MLE. So what does equation 1 means that I have written? $\endgroup$
    – MAC
    Oct 12, 2020 at 3:58
  • $\begingroup$ Second question is, If I wish to find Normal distribution for all the columns A,B,C can I transalte gaussian equation for multi-dimensional settings? $\endgroup$
    – MAC
    Oct 12, 2020 at 4:00
  • 2
    $\begingroup$ Okay, I see what you mean now. You aren't trying to predict a target variable y you are just trying to fit a normal distribution to some data. In this case, what you are looking for is the multivariate generalization of the normal distribution known as the multivariate normal. en.wikipedia.org/wiki/Multivariate_normal_distribution $\endgroup$
    – aranglol
    Oct 12, 2020 at 4:59
  • 1
    $\begingroup$ In other words, you assume your likelihood is multivariate normal with mean vector (u1, u2, u3) and a 3x3 covariance matrix. You can then use maximum likelihood in the exact same way you described above. In total, you fit 9 parameters - 3 means in the mean vector, 3 variance terms on the diagonal of the covariance matrix, and 3 off-diagonal covariance terms (the covariance matrix is symmetric). $\endgroup$
    – aranglol
    Oct 12, 2020 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.