2
$\begingroup$

I am building an RNN by following Siraj Raval's video implementation. I've adapted it to use my dataset rather than importing one from a file. When the program gets to the loss function, it reports loss += -np.log(ps[t][y[t], 0]) IndexError: index 7 is out of bounds for axis 0 with size 4

What does this mean and how can i fix this? Also, in Siraj's version, he calculates the loss as:

loss += -np.log(ps[t][y[t], 0])

Shouldn't it be this instead

loss += -np.log(ps[t]) * [y[t], 0]

as the cross entropy loss is L = -yln(yhat)?

My code:

import numpy as np

# Data Processing
x = np.array([
    # t/no. of inputs         
    [1, 2, 3, 4, 5, 6],
    [7, 8, 9, 10, 11, 12],     # Samples
    [13, 14, 15, 16, 17, 18],
    [19, 20, 21, 22, 23, 24]]).T

# Model Parameters
numInputs = x.shape[1] # Yields 4
numNeurons = 8 # Yields 8
numEntries = x.shape[0] # Yields 6

u = np.random.random((numNeurons, numInputs))
v = np.random.random((numInputs, numNeurons))
w = np.random.random((numNeurons, numNeurons))
bh = np.zeros((numNeurons, 1))
bo = np.zeros((numInputs, 1))
atimeline = [] # Contains 6 timestep's woth of a's
yhattimeline = [] # Contains 6 timestep's worth of yhat's
hprev = np.zeros((8, 1)) # Contains previous a state

# Training
def loss(x, y, hprev):
    xs, hs, ys, ps = {}, {}, {}, {}
    hs[-1] = np.copy(hprev) # Copies hprev so hprev can still be used
                            # Adds key: -1, value: hprev, to the dict
    loss = 0
    xs[0] = np.zeros((numInputs, 1)) # Sets t = 0 to 0's
    for t in xrange(numEntries):
        xs[t + 1] = x[t] # Adds in data from x to the dict
        hs[t] = np.tanh(np.dot(u, xs[t]) + np.dot(w, hs[t - 1]) + bh)
        ys[t] = np.dot(v, hs[t]) + bo
        ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # Softmax
        loss += -np.log(ps[t][y[t], 0])
    du, dv, dw = np.zeros_like(u), np.zeros_like(v), np.zeros_like(w)
    dbh, dbo = np.zeros_like(bh), np.zeros_like(bo)
    dhnext = np.zeros_like(hs[0])
    for t in reversed(xrange(numEntries)):
        dy = np.copy(ps[t])
        dy[targets[t]] -= 1 # Derived from dL/dyhat
        dv += np.dot(dy, hs[t].T)
        dbo += dy
        dh = np.dot(v.T, dy) + dhnext
        dhraw = (1 - hs[t] * hs[t]) * dh # tanh
        dbh += dhraw
        du += np.dot(dhraw, xs[t].T)
        dw += np.dot(dhraw, xs[t - 1].T)
        dhnext = np.dot(w.T, dhraw)
        return du, dv, dw, dbo, dbh, hs[numEntries - 1]

u, v, w, dbo, dbh, hprev = loss(x, x, hprev)
$\endgroup$
1
$\begingroup$

What happened was that

xs[0] = np.zeros((numInputs, 1))

Generates an array of ((4, 1)), and

xs[t + 1] = x[t]

Generates an array of ((1, 4)). To solve this, I transposed x[t] so now it is the same shape as above - ((4, 1)):

xs[t + 1] = np.array([x[t]]).T

Annoying bug and hard to solve, but a fairly easy solution.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.