0
$\begingroup$

When training my LSTM, is there any incentive to pre-pass its inputs through an auto-encoder, or should I always supply as raw data as possible?

I have is a small amount of training data, and it contains only a few very rare significant events (spikes, spaced out in time)

$\endgroup$
2
$\begingroup$

The purpose of the autencoder would be similar to doing an embedding of text to pass latent space features of the input to the LSTM, as opposed to raw inputs. In the case of text, this is usually done to reduce the very large and sparse dictionary (word subword, whatever) space to a space which is less sparse, significantly smaller, and therefore more computationally efficient.

In your case, it looks like the data is not high dimensional to start with (I am assuming this because of the significant events over time), so you would not get the benefit of mapping the input to a latent space. If your trend over time is high dimensional, then it may be worth it, as the embedding/autencoding may pick up and reduce correlations between trends in a few dimensions.

$\endgroup$
2
  • $\begingroup$ Thanks! Could you please also give a real-world example where the state would be high dimensional (except for text)? $\endgroup$
    – Kari
    May 6 '18 at 21:54
  • 1
    $\begingroup$ Real world example is pretty much all text problems with a real vocabulary (which can be 100k words) for example predicting the next word in a sentence, or the likely hood of a word (or synonym) given some text... For high-D sequential trend prediction I really have not seen this done, but I would imagine if you wanted to predict energy usage spikes over time, and as your input you had geo-spatial data of all humans in the area, it could be formulated that way ... or if you wanted to do stock prediction with 1000's of individual stocks ... I am really stretching. $\endgroup$ May 6 '18 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.