15

I prefer not to add drop out in LSTM cells for one specific and clear reason. LSTMs are good for long terms but an important thing about them is that they are not very well at memorising multiple things simultaneously. The logic of drop out is for adding noise to the neurons in order not to be dependent on any specific neuron. By adding drop out for LSTM ...


12

Simple explanation with images We know that an activation is required between matrix multiplications to afford a neural network the ability to model non-linear processes. A classical LSTM cell already contains quite a few non-linearities: three sigmoid functions and one hyperbolic tangent (tanh) function, here shown in a sequential chain of repeating (...


7

There is not a consensus that can be proved across all model types. Thinking of dropout as a form of regularisation, how much of it to apply (and where), will inherently depend on the type and size of the dataset, as well as on the complexity of your built model (how big it is).


4

Assuming that the LSTM is going to be used for sequence generation (e.g. in a language model or the decoder part of an encoder-decoder NMT architecture), we have the following: In supervised learning setups: In a language model, the LSTM's $h_{-1}$ and $c_{-1}$ are initialized to zeros, for all the layers. If the LSTM is the decoder part of an encoder-...


4

You are correct that "stacking LSTMs" means to put layers on top of one-another as in your second image. The first picture is a "bi-directional LSTM" (BiLSTM), whereby we can analyse a point in a series (e.g. a word in a sentence) from both sides. We care about the context of that point. The most common example I know of is within NLP. Here we want to know ...


3

As you can see here, derivatives will be propagated by the chain rule although they are stacked. Actually, there will be two main paths. The first one will be backpropagation through time and the next one will be the backpropagation from the output of each unrolled cell which can directly be connected to the output or can be connected to the stacked unrolled ...


2

What you are getting as the output is the internal LSTM state. In order to get value comparable to your labels, add a dense layer on top of it. Output dimension of dense layer would be the number of labels you want result. If its 0 and 1, only 1 output neuron can work along with sigmoid If there are 5 label classes, then output dimension of dense layer ...


1

There are two well known algorithms called Isolation Forest and One-Class SVM for outlier detection. You will find implementations of these in Sckikit learn. Doing a search for "Anomaly Detection" on github, there seem to be entries to the NAB competition available publicly eg. nareshkumar66675/Numenta. This one has a Jupyter notebook which mainly uses ...


Only top voted, non community-wiki answers of a minimum length are eligible