Theano logistic regression understanding code

Question

I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation.

On the logistic regression code available online, I am trying to understand this line of code:

-T.mean(T.log(self.p_y_given_x)[T.arange(y.shape[0]), y])

It is basically saying perform this average: $\begin{align} J(\theta) = - \left[ \sum_{i=1}^{b} \sum_{k=1}^{K} 1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right] \end{align}$ where $b$ is the batch size.

So is it related to theano code, or it is just a python notation ? I am interested exactly on this piece of code:

T.log(self.p_y_given_x)[T.arange(y.shape[0]), y]

If you could please give me some explanation.

Answer 1

I agree the code may be hard to read at the first glance, but I find the comment pretty clear:

• y.shape[0] is (symbolically) the number of rows in y, i.e., number of examples (call it n) in the minibatch
• T.arange(y.shape[0]) is a symbolic vector which will contain [0,1,2,... n-1].
• T.log(self.p_y_given_x) is a matrix of Log-Probabilities (call it LP) with one row per example and one column per class
• LP[T.arange(y.shape[0]),y] is a vector v containing [LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ..., LP[n-1,y[n-1]]]
• T.mean(LP[T.arange(y.shape[0]),y]) is the mean (across minibatch examples) of the elements in v, i.e., the mean log-likelihood across the minibatch.

Everything is Python notation (but sometimes using Theano functions).