I have been attempting to train a RNN on a set of time series data. The goal is to predict one of six categorical outputs. The input is given as 5 time steps of 14 inputs, six of which at one-hot attributes for the output. There is an output at each time step, but the goal is to use previous recorded time spots and their human-assigned outputs to assign an output to the most recent event.
Confusingly the RNN is unable to learn that one of the inputs is in fact the output of the classification. This is simply a sanity check for me, but it seems that it may indicate a larger underlying problem.
The data is heavily imbalanced, 91%, 4%, 2%, 1%,<1%,<1%, but a cost function is being use to weight mis-classification inversely to it's make-up in the data set. Could the imbalance cause this issue? I'm using 60,000 training examples right now, is this not enough?
I'm working off of this dynamic RNN model: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/dynamic_rnn.py. Note the additional matrix multiplication for a hidden layer. Is this done correctly?
def dynamicRNN(x, seqlen, weights, biases): # Prepare data shape to match `rnn` function requirements # Current data input shape: (batch_size, n_steps, n_input) # Required shape: 'n_steps' tensors list of shape (batch_size, n_input) # Permuting batch_size and n_steps x = tf.transpose(x, [1, 0, 2]) # Reshaping to (n_steps*batch_size, n_input) x = tf.reshape(x, [-1,n_input]) x = tf.matmul(x, weights['hidden'])+ biases['hidden'] # Split to get a list of 'n_steps' tensors of shape (batch_size, n_input) x = tf.split(0, n_steps, x) # Define a lstm cell with tensorflow lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0) # Get lstm cell output, providing 'sequence_length' will perform dynamic # calculation. outputs, states = tf.nn.rnn(lstm_cell, x, dtype=tf.float32, sequence_length=seqlen) # When performing dynamic calculation, we must retrieve the last # dynamically computed output, i.e, if a sequence length is 10, we need # to retrieve the 10th output. # However TensorFlow doesn't support advanced indexing yet, so we build # a custom op that for each sample in batch size, get its length and # get the corresponding relevant output. # 'outputs' is a list of output at every timestep, we pack them in a Tensor # and change back dimension to [batch_size, n_step, n_input] outputs = tf.pack(outputs) outputs = tf.transpose(outputs, [1, 0, 2]) # Hack to build the indexing and retrieve the right output. batch_size = tf.shape(outputs) # Start indices for each sample index = tf.range(0, batch_size) * n_steps + (seqlen - 1) # Indexing outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index) # Linear activation, using outputs computed above return tf.matmul(outputs, weights['out']) + biases['out']