1
$\begingroup$

I am currently studying the Elements of Statistical Learning book. The following equation is in page 120.

It calculates the Hessian matrix for the log-likelihood function as follows

\begin{equation} \dfrac{\partial^2 \ell(\beta)}{\partial\beta\partial\beta^T} = -\sum_{i=1}^{N}{x_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))} \end{equation}

But is the following calculation it is only calculating $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i^2}$ terms. But Hessian matrix should also contain $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i\partial\beta_j}$ where $i\neq j$.

Please explain the reason for missing out these terms.

$\endgroup$

1 Answer 1

1
$\begingroup$

Beta is a vector of parameters, therefore:

$ \frac{\delta l(\beta)}{\delta\beta}= [\frac{\delta l(\beta)}{\delta\beta_1}\quad\frac{\delta l(\beta)}{\delta\beta_2}\quad\frac{\delta l(\beta)}{\delta\beta_3}\quad...\quad\frac{\delta l(\beta)}{\delta\beta_n}]$ and so

$ \frac{\delta(\frac{\delta l(\beta)}{\delta\beta})}{\delta\beta^{T}}= \begin{bmatrix} \frac{\delta l^2(\beta)}{\delta\beta_1^2} & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_n} \\ \frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_2^2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_n^2} \end{bmatrix}$, which is your Hessian.

The term on the right side of your equation is also a matrix, because there is a multiplication of vectors in it: $x_i \cdot x_i^T$, which gives a $n \times n$ matrix.

$\endgroup$
1
  • $\begingroup$ No actually. x[i] will be a column matrix as x is an nXn matrix. So x[i]*x[i]^T will be a single value. $\endgroup$ Apr 25, 2020 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.