I am currently studying the Elements of Statistical Learning book. The following equation is in page 120.

It calculates the Hessian matrix for the log-likelihood function as follows

\begin{equation} \dfrac{\partial^2 \ell(\beta)}{\partial\beta\partial\beta^T} = -\sum_{i=1}^{N}{x_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))} \end{equation}

But is the following calculation it is only calculating $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i^2}$ terms. But Hessian matrix should also contain $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i\partial\beta_j}$ where $i\neq j$.

Please explain the reason for missing out these terms.


Beta is a vector of parameters, therefore:

$ \frac{\delta l(\beta)}{\delta\beta}= [\frac{\delta l(\beta)}{\delta\beta_1}\quad\frac{\delta l(\beta)}{\delta\beta_2}\quad\frac{\delta l(\beta)}{\delta\beta_3}\quad...\quad\frac{\delta l(\beta)}{\delta\beta_n}]$ and so

$ \frac{\delta(\frac{\delta l(\beta)}{\delta\beta})}{\delta\beta^{T}}= \begin{bmatrix} \frac{\delta l^2(\beta)}{\delta\beta_1^2} & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_n} \\ \frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_2^2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_n} \\ \vdots & \vdots & \ddots & \vdots \\ \frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_n^2} \end{bmatrix}$, which is your Hessian.

The term on the right side of your equation is also a matrix, because there is a multiplication of vectors in it: $x_i \cdot x_i^T$, which gives a $n \times n$ matrix.

  • $\begingroup$ No actually. x[i] will be a column matrix as x is an nXn matrix. So x[i]*x[i]^T will be a single value. $\endgroup$ Apr 25 '20 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.