# Correct output labelling for RNNs/ANNs with sequential data and single label

I have a labelled time-series dataset; for example, a sequential input length nn from kk sensors corresponds to class '1' (see below for n=50,k=2n=50,k=2). How can I correctly classify it?

If I use RNN with infinite input length, the input layer will have 2 units (2 channels), but what about the output? Label '1' applies to the full sequence, not just a single input.

Therefore, there are two related questions: 1) how do I design an output unit? It has to reflect the input of the full sequence, 2) how do I compute an error? It also needs to account for the error of the whole sequence.

If I use ANN with a sliding window, the problem remains virtually the same - different classes can have comparable values in the window, so the output must correspond to the full sequence.

In other words, how do I match a time series to just 1 number?

EDIT:

For example, there is an $n \times k$ array labelled '1' (this can be for example readings off $n$ sensors, which match 1 class, like a gesture: 'snap'), and another $m \times j$ array labelled '0', where $n$ is the number of features/input neurons and $k \neq j$ is the length/size of the input data. If I use an RNN to feedforward the first array I do it $k$ times, obviously. But how do I label it? Do I use a unity vector length $k$, get an error each time and them average out to get the sample error and backprop? I don't think so, but not too sure.

Recurrent Neural Networks can learn very complex mappings between sequences (unlike Feed Forward Nets which are capable of transforming only fixed-sized vectors). You really only need a RNN which takes the sequence of the form $(a_t, b_t)$ as the input (assuming $t \in \{1...T\}$ you have seq. $a$ and $b$ both with $T$ samples and you have to classify these signals as one of $k$ classes). Then you can run the following procedure (pseudocode) for the sampling (given already trained model):

for t in (1..T)           //loop over an input sequence
rnn.input[0] = a[t]   //assign t-th point of the blue seq.
rnn.input[1] = b[t]   //assign t-th point of the green seq.
rnn.forward()         //make forward pass
rnn.output.normalize()    //renormalize output probabilities
class_prob = rnn.output.max()
class      = rnn.output.argmax()


For example we have the following sequence with length 4:
signal A: {1, 3, 4, 6}
signal B: {4, 3, 0, 1}
Then in the 1st timestep we assign vector $(1, 4)$ to the input layer, then $(3, 3)$ and so on. At each timestep we propagate the input forward the network updating its hidden state which is responsible for remembering past datapoints. After every forward pass our neural network generates a prediction at the output layer which is temporarily uninmportant. This fluctuating prediction reflects the rnn's belief about the part of our dataset- it have seen only $t$ first input vectors. After we propagated entire signals we can do the classification.
Your RNN's output layer should be $k$-dimensional and its $i$-th component is the pseudo-probability of the signal beeing in class $i$. For example you can do argmax (take the index with greatest prediction) or sample with a softmax distribution. For example if you have class $a$ and class $b$ and your output vector is $(5, 15)$ then your signal is classified as $b$. Or if you do softmax then $softmax(5, 15) = (0.000045, 0.999955)$ so you assign the class $b$ with nearly 100% probability.
Training RNN is much more triky. I would suggest you taking some existing toolkit which have RNN such as LSTM already implemented. To train Recurrent NN we can apply the above procedure but at the end (after iterating over the sequence) we calculate Gradient of the loss function and do a backward pass. However this technique is called Backpropagation through time (it is really similar to ordinary FFN backprop, but we must "unfold" out RNN in time to adjust recurrent connections). Also notice that if you choose to have softmax output units your loss function should be a Cross-Entropy loss function.
But what is the expected output vector to compute gradient loss from? If your dataset contains $N$ (green-blue) sequences each of them is in $k_n$ class then their expected output vector is simply one-hot vector $(0...,1,...0)$ where $1$ is at $k_n$-th component only. For example if you have a sequence classified as $b$, then its output vector target is $(0,1)$. @EDIT: Suppose that as in a previous example your sequence have class $b$ so you expect to see the vector $(0,1)$. You take now the prediction of the net (say $(0.34, 0.57)$) and calculate the error function as normal f.e. squared error or cross-entropy.

• OK, so during both training and testing we have a label for each input, and we obtain an error for every input, is that right? If, in your example, input 1 wil be (1,4), it has a label (e.g. 1), then input 2 (3,3) with label 0 and so on. At the end of the sequence we somehow average out the errors to get the argmax class of the input, is this right?
– Alex
Jan 26, 2016 at 10:57
• Not quite, it's simpler than that. In test time these predictions are completely irrelevant for us- we just sample it after forwarding the full sequence. During training we are forwarding whole sequence and only at the end we calculate loss function between the final prediction and the label for the whole sequence. And we backpropagate the error through time but then we are not samling predictions. Jan 26, 2016 at 19:01
• Of course this applies only to "Many to one" model (I'm refering to karpathy). You said "..Label '1' applies to the full sequence, not just a single input." so that's your answer. You literally cannot compute error after each datapoint becouse your labels describe whole sequences. But BPTT propagates that final error backwards in time so that the error is "evenly distributed" between your consequtives datapoints. Jan 26, 2016 at 19:36
• 'only at the end we calculate loss function between the final prediction and the label for the whole sequence' but how? Look, we've fed forward a data point, ok. How do we get an error out of it? Do we have the same label for all data points? For example, in gesture classification it will be '1' for 'snap', so every data point will be labelled '1'. Do I miss something?
– Alex
Jan 26, 2016 at 21:27
• If we backprop the error signal after every data point we are connecting these single data points with a label. Suppose that we have a dataset of phrases and we want to assing some class to them. Our input vector will encode one single character. Does 'c' or 'd' belong to a specified class? No. But RNN allows us to model very complex and even longterm dependencies between our letters so we can assign labels to sequences. We can calculate the error only once because RNN have the memory cells which remember the history of a sequence. How do we calculate that? I will update my answer in a moment. Jan 26, 2016 at 22:09