I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation.

On the logistic regression code available online, I am trying to understand this line of code:

-T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])

It is basically saying perform this average: $\begin{align} J(\theta) = - \left[ \sum_{i=1}^{b} \sum_{k=1}^{K} 1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right] \end{align}$ where $b$ is the batch size.

So is it related to theano code, or it is just a python notation ? I am interested exactly on this piece of code:

T.log(self.p_y_given_x)[T.arange(y.shape[0]), y]

If you could please give me some explanation.


I agree the code may be hard to read at the first glance, but I find the comment pretty clear:

  • y.shape[0] is (symbolically) the number of rows in y, i.e., number of examples (call it n) in the minibatch
  • T.arange(y.shape[0]) is a symbolic vector which will contain [0,1,2,... n-1].
  • T.log(self.p_y_given_x) is a matrix of Log-Probabilities (call it LP) with one row per example and one column per class
  • LP[T.arange(y.shape[0]),y] is a vector v containing [LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ..., LP[n-1,y[n-1]]]
  • T.mean(LP[T.arange(y.shape[0]),y]) is the mean (across minibatch examples) of the elements in v, i.e., the mean log-likelihood across the minibatch.

Everything is Python notation (but sometimes using Theano functions).

  • $\begingroup$ Thank you for you answer, in fact it is 2D-array indexing using Numpy, for more details have a look at the link bellow (section: Indexing Multi-dimensional arrays): docs.scipy.org/doc/numpy-1.10.1/user/basics.indexing.html $\endgroup$ – Mohamed Lakhal Dec 11 '15 at 9:05
  • $\begingroup$ I am struggling with the T.arange(y.shape[0]) part, as I understand this is a list of indices from 0 to n-1 (n being the number of examples in the mini batch), where is this used in the code and for what? $\endgroup$ – Jan van der Vegt Jun 15 '16 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.