# Loss function not working (RNN)

I am building an RNN by following Siraj Raval's video implementation. I've adapted it to use my dataset rather than importing one from a file. When the program gets to the loss function, it reports loss += -np.log(ps[t][y[t], 0]) IndexError: index 7 is out of bounds for axis 0 with size 4

What does this mean and how can i fix this? Also, in Siraj's version, he calculates the loss as:

loss += -np.log(ps[t][y[t], 0])


loss += -np.log(ps[t]) * [y[t], 0]


as the cross entropy loss is L = -yln(yhat)?

My code:

import numpy as np

# Data Processing
x = np.array([
# t/no. of inputs
[1, 2, 3, 4, 5, 6],
[7, 8, 9, 10, 11, 12],     # Samples
[13, 14, 15, 16, 17, 18],
[19, 20, 21, 22, 23, 24]]).T

# Model Parameters
numInputs = x.shape # Yields 4
numNeurons = 8 # Yields 8
numEntries = x.shape # Yields 6

u = np.random.random((numNeurons, numInputs))
v = np.random.random((numInputs, numNeurons))
w = np.random.random((numNeurons, numNeurons))
bh = np.zeros((numNeurons, 1))
bo = np.zeros((numInputs, 1))
atimeline = [] # Contains 6 timestep's woth of a's
yhattimeline = [] # Contains 6 timestep's worth of yhat's
hprev = np.zeros((8, 1)) # Contains previous a state

# Training
def loss(x, y, hprev):
xs, hs, ys, ps = {}, {}, {}, {}
hs[-1] = np.copy(hprev) # Copies hprev so hprev can still be used
# Adds key: -1, value: hprev, to the dict
loss = 0
xs = np.zeros((numInputs, 1)) # Sets t = 0 to 0's
for t in xrange(numEntries):
xs[t + 1] = x[t] # Adds in data from x to the dict
hs[t] = np.tanh(np.dot(u, xs[t]) + np.dot(w, hs[t - 1]) + bh)
ys[t] = np.dot(v, hs[t]) + bo
ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # Softmax
loss += -np.log(ps[t][y[t], 0])
du, dv, dw = np.zeros_like(u), np.zeros_like(v), np.zeros_like(w)
dbh, dbo = np.zeros_like(bh), np.zeros_like(bo)
dhnext = np.zeros_like(hs)
for t in reversed(xrange(numEntries)):
dy = np.copy(ps[t])
dy[targets[t]] -= 1 # Derived from dL/dyhat
dv += np.dot(dy, hs[t].T)
dbo += dy
dh = np.dot(v.T, dy) + dhnext
dhraw = (1 - hs[t] * hs[t]) * dh # tanh
dbh += dhraw
du += np.dot(dhraw, xs[t].T)
dw += np.dot(dhraw, xs[t - 1].T)
dhnext = np.dot(w.T, dhraw)
return du, dv, dw, dbo, dbh, hs[numEntries - 1]

u, v, w, dbo, dbh, hprev = loss(x, x, hprev)


What happened was that

xs = np.zeros((numInputs, 1))


Generates an array of ((4, 1)), and

xs[t + 1] = x[t]


Generates an array of ((1, 4)). To solve this, I transposed x[t] so now it is the same shape as above - ((4, 1)):

xs[t + 1] = np.array([x[t]]).T


Annoying bug and hard to solve, but a fairly easy solution.