4
$\begingroup$

I have been attempting to train a RNN on a set of time series data. The goal is to predict one of six categorical outputs. The input is given as 5 time steps of 14 inputs, six of which at one-hot attributes for the output. There is an output at each time step, but the goal is to use previous recorded time spots and their human-assigned outputs to assign an output to the most recent event.

Confusingly the RNN is unable to learn that one of the inputs is in fact the output of the classification. This is simply a sanity check for me, but it seems that it may indicate a larger underlying problem.

The data is heavily imbalanced, 91%, 4%, 2%, 1%,<1%,<1%, but a cost function is being use to weight mis-classification inversely to it's make-up in the data set. Could the imbalance cause this issue? I'm using 60,000 training examples right now, is this not enough?

I'm working off of this dynamic RNN model: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py. Note the additional matrix multiplication for a hidden layer. Is this done correctly?

def dynamicRNN(x, seqlen, weights, biases):

# Prepare data shape to match `rnn` function requirements
# Current data input shape: (batch_size, n_steps, n_input)
# Required shape: 'n_steps' tensors list of shape (batch_size, n_input)

# Permuting batch_size and n_steps
x = tf.transpose(x, [1, 0, 2])
# Reshaping to (n_steps*batch_size, n_input)
x = tf.reshape(x, [-1,n_input])
x  = tf.matmul(x, weights['hidden'])+ biases['hidden']
# Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)
x = tf.split(0, n_steps, x)

# Define a lstm cell with tensorflow
lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)

# Get lstm cell output, providing 'sequence_length' will perform dynamic
# calculation.
outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32,
                            sequence_length=seqlen)

# When performing dynamic calculation, we must retrieve the last
# dynamically computed output, i.e, if a sequence length is 10, we need
# to retrieve the 10th output.
# However TensorFlow doesn't support advanced indexing yet, so we build
# a custom op that for each sample in batch size, get its length and
# get the corresponding relevant output.

# 'outputs' is a list of output at every timestep, we pack them in a Tensor
# and change back dimension to [batch_size, n_step, n_input]
outputs = tf.pack(outputs)
outputs = tf.transpose(outputs, [1, 0, 2])

# Hack to build the indexing and retrieve the right output.
batch_size = tf.shape(outputs)[0]
# Start indices for each sample
index = tf.range(0, batch_size) * n_steps + (seqlen - 1)
# Indexing
outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index)

# Linear activation, using outputs computed above
return tf.matmul(outputs, weights['out']) + biases['out']
$\endgroup$
2
  • $\begingroup$ Are you using a library that implements RNNs on top of TensorFlow? If not, what reference to RNN in TensorFlow are you using? $\endgroup$ Jul 20, 2016 at 6:37
  • $\begingroup$ My data has inconsistent numbers of time steps, so 5 is actually the maximum. I'm trying to work off of this example of a RNN for dynamic length time series: github.com/aymericdamien/TensorFlow-Examples/blob/master/… $\endgroup$
    – Cole
    Jul 20, 2016 at 16:33

1 Answer 1

1
$\begingroup$

Strangely, once the columns of each block of 5 data entries were normalized, with null rows included, the accuracy jumped to >99%. Again the outcome is included in the data set but this seemed to correct the issue with this sanity check.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.