Correct output labelling for RNNs/ANNs with sequential data and single label

Question

I have a labelled time-series dataset; for example, a sequential input length nn from kk sensors corresponds to class '1' (see below for n=50,k=2n=50,k=2). How can I correctly classify it?

If I use RNN with infinite input length, the input layer will have 2 units (2 channels), but what about the output? Label '1' applies to the full sequence, not just a single input.

Therefore, there are two related questions: 1) how do I design an output unit? It has to reflect the input of the full sequence, 2) how do I compute an error? It also needs to account for the error of the whole sequence.

If I use ANN with a sliding window, the problem remains virtually the same - different classes can have comparable values in the window, so the output must correspond to the full sequence.

In other words, how do I match a time series to just 1 number?

EDIT:

For example, there is an $n \times k$ array labelled '1' (this can be for example readings off $n$ sensors, which match 1 class, like a gesture: 'snap'), and another $m \times j$ array labelled '0', where $n$ is the number of features/input neurons and $k \neq j$ is the length/size of the input data. If I use an RNN to feedforward the first array I do it $k$ times, obviously. But how do I label it? Do I use a unity vector length $k$, get an error each time and them average out to get the sample error and backprop? I don't think so, but not too sure.

Answer 1

Recurrent Neural Networks can learn very complex mappings between sequences (unlike Feed Forward Nets which are capable of transforming only fixed-sized vectors). You really only need a RNN which takes the sequence of the form $(a_t, b_t)$ as the input (assuming $t \in \{1...T\}$ you have seq. $a$ and $b$ both with $T$ samples and you have to classify these signals as one of $k$ classes). Then you can run the following procedure (pseudocode) for the sampling (given already trained model):

for t in (1..T)           //loop over an input sequence
    rnn.input[0] = a[t]   //assign t-th point of the blue seq.
    rnn.input[1] = b[t]   //assign t-th point of the green seq.
    rnn.forward()         //make forward pass
rnn.output.normalize()    //renormalize output probabilities
class_prob = rnn.output.max()
class      = rnn.output.argmax()

For example we have the following sequence with length 4:
signal A: {1, 3, 4, 6}
signal B: {4, 3, 0, 1}
Then in the 1st timestep we assign vector $(1, 4)$ to the input layer, then $(3, 3)$ and so on. At each timestep we propagate the input forward the network updating its hidden state which is responsible for remembering past datapoints. After every forward pass our neural network generates a prediction at the output layer which is temporarily uninmportant. This fluctuating prediction reflects the rnn's belief about the part of our dataset- it have seen only $t$ first input vectors. After we propagated entire signals we can do the classification.
Your RNN's output layer should be $k$-dimensional and its $i$-th component is the pseudo-probability of the signal beeing in class $i$. For example you can do argmax (take the index with greatest prediction) or sample with a softmax distribution. For example if you have class $a$ and class $b$ and your output vector is $(5, 15)$ then your signal is classified as $b$. Or if you do softmax then $softmax(5, 15) = (0.000045, 0.999955)$ so you assign the class $b$ with nearly 100% probability.
Training RNN is much more triky. I would suggest you taking some existing toolkit which have RNN such as LSTM already implemented. To train Recurrent NN we can apply the above procedure but at the end (after iterating over the sequence) we calculate Gradient of the loss function and do a backward pass. However this technique is called Backpropagation through time (it is really similar to ordinary FFN backprop, but we must "unfold" out RNN in time to adjust recurrent connections). Also notice that if you choose to have softmax output units your loss function should be a Cross-Entropy loss function.
But what is the expected output vector to compute gradient loss from? If your dataset contains $N$ (green-blue) sequences each of them is in $k_n$ class then their expected output vector is simply one-hot vector $(0...,1,...0)$ where $1$ is at $k_n$-th component only. For example if you have a sequence classified as $b$, then its output vector target is $(0,1)$. @EDIT: Suppose that as in a previous example your sequence have class $b$ so you expect to see the vector $(0,1)$. You take now the prediction of the net (say $(0.34, 0.57)$) and calculate the error function as normal f.e. squared error or cross-entropy.

More information you can learn from karpathy and hinton