My question is the following:

It is known that a LSTM can remember sequences of one-hot encodings which represent integers (i.e. output $x_1, ... x_n$ after receiving $x_1, ... x_n$ as inputs, $x_k \in \{0,1\}^m$, where $m$ is the number of distinct integers).

Is it theoretically possible for the LSTM to learn to remember sequences of real numbers instead (that can be expressed in a finite number of bits), i.e. if $x_t \in \mathbb{R}$ instead.

The task I'm concerned with is much simpler - I just want to output the first input $x_1$ after reading the entire sequence $x_1, ... x_n$. I have done some small experiments with $x_t \in \mathbb{R}$, using square loss. There seems to be some level of success, however the results aren't very interpretable (when I look at the weights). Can anyone shed some light on this, specifically:

  1. Does such a configuration of weights exist? (the questions following this quesetion suggests that it does exist)
  2. If so, what are they and if not, why not?

The LSTM model is specified by:

The input, forget and outputs gates:

$$f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$$ $$i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$$ $$o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$$

And the internal state $c_t$ and hidden state $h_t$:

$$c_t = f_t * c_{t-1} + i_t * \text{tanh}(W_c[h_{t-1}, x_t] + b_c) $$ $$h_t = o_t * \text{tanh}(c_t)$$

As requested, this is the assignment question:

Memory Task Description

Consider the following task: Given an input sequence of $n$ numbers, we would like a system which, after reading this sequence will return the first number in the sequence. That is given an input sequence: $(x_1, x_2, \cdots x_n)$, $x_i \in \mathbb{R}$ the system has to return, at time $t=n$ after 'reading' the last input $x_n$, the first input $x_1$.

  1. Given the task above, consider the above recurrent models (RNNs/LSTMs/GRUs). Which of these arhitectures can (theoretically) perfom the task above? In answering this questions, please consider a simple one-layer model of RNNs/GRU/LSTM with a one-dimensional input $x_t$, a $32$-dim hidden and output layer, followed by a transformation to a one-dimensional final output which should predict $x_0$. Whenever the answer is positive, give the gates' activations and weigths that will produce the desired behaviour. Whenever the answer is no, prove that there exists no such parameters that an arbitrary input sequence can be transformed to produce the first symbol read.

Welcome to the site! If you're referring to a series of numbers, like what you would get during tokenization/NLP then, yes, LSTM can certainly handle that without many issues. If you are talking about a much larger range than that, then you might want to consider a scenario where you scale your inputs instead.

  • 1
    $\begingroup$ Thanks for your answer! but could you perhaps explain how it manages to retain just the first input in its memory whilst not "corrupting" the memory with the inputs further down the road? $\endgroup$
    – Sean Lee
    Mar 5 '19 at 21:09
  • $\begingroup$ As you've defined your goal, x=sequence, y=first element. You would set up your inputs accordingly, so that the y associated with each input sequence sample is the first element. I'd imagine after training, all the weights associated with the other time-steps would converge to 0. This function should be easily learned by an lstm. $\endgroup$
    – kylec123
    Mar 5 '19 at 22:09
  • 1
    $\begingroup$ @kylec123 Sorry, I don't quite understand the statement "all the weights associated with the other time-steps would converge to 0" - could you give a particular configuration of the weights where this is the case? $\endgroup$
    – Sean Lee
    Mar 6 '19 at 4:18
  • $\begingroup$ @SeanLee Help me understand why you are concerned about this "corruption". I just haven't seen that be a concern across various implementation or models. Is there something unique about the way you're going about this that makes that a concern? Is this something you've run into before? $\endgroup$ Mar 6 '19 at 16:38
  • $\begingroup$ @I_Play_With_Data because after looking at the equations for quite a bit, I've come to the following understanding - it's not possible to read $x_1$ into the cell state without also reading the rest ($x_2, ..., x_n$) of the observations. Trying to reconstruct $x_1$ from the final cell state using some transformation seems to not admit any particularly analytic solutions (for both the transformations and the weights/biases). For what it's worth, this is an assignment question which I've been stuck at for quite a while. $\endgroup$
    – Sean Lee
    Mar 6 '19 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.