11

You are right that LSTMs work very well for some problems, but some of the drawbacks are: LSTMs take longer to train LSTMs require more memory to train LSTMs are easy to overfit Dropout is much harder to implement in LSTMs LSTMs are sensitive to different random weight initializations These are in comparison to a simpler model like a 1D conv net, for ...


11

The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing. Dealing with such a Model: Data Preprocessing: Standardizing and Normalizing the data. Model compelxity: Check if the model is too complex. Add dropout, reduce number of layers or number of neurons in each layer. Learning Rate and Decay ...


9

Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in turn it is vastly more performant, while a statefull LSTM you can have varying timesteps, but at a performance penalty. The stateless LSTM does have state, it's ...


8

To begin with, you can think of the batch size as a way to control the smoothness of the learning curve. With a huge batch size, you are taking the average of many errors for each update, and this average loss (on average), doesn't have great variance. Using a batch size of 1, your cost on each iteration is solely dependent on the single sample that you fed ...


7

Yes, there have been many attempts, but perhaps the most noteable one is the approach described in the paper of Andrej Karpathy and Li Fei-Fei where they connect a CNN and RNN in series (CNN over image region + bidirectional RNN + Multimodal RNN) and use this for labeling a scene with a whole sentence. Though, this one is more than just object detection as ...


7

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the ...


6

From what I've seen on Github while looking for open source projects is that people usually do both. You can have a section where one loads the models and runs the inference, and another section where you let the user train the models from scratch using your code. I recommend doing this since some people do not want to retrain the model, especially if it'...


5

Don't remove a feature to find out its importance, but instead randomize or shuffle it. Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way. Here's the theory behind my suggestion: feature importance


4

If I understand your question correctly, the reason you think cannot concatenate your time series into a one dataset is because of their different length. Depending on your problem, you can handle this issue in multiple manners in preprocessing. But the more common way is to use sequence padding. Preprocessing methods are natively implemented in keras: https:...


4

just re-use model.fit() on the fresh datasets using the already trained model, simple as that :) ! (given that you do it in Keras)


4

You are correct that "stacking LSTMs" means to put layers on top of one-another as in your second image. The first picture is a "bi-directional LSTM" (BiLSTM), whereby we can analyse a point in a series (e.g. a word in a sentence) from both sides. We care about the context of that point. The most common example I know of is within NLP. Here we want to know ...


4

Yes this is an overfitting problem since your curve shows point of inflection. This is a sign of very large number of epochs. In this case, model could be stopped at point of inflection or the number of training examples could be increased. Also, Overfitting is also caused by a deep model over training data. In that case, you'll observe divergence in loss ...


4

You are right in a sense that it is better to be called log of unnormalized probability. This way, the quantity could be positive or negative. For example, $\text{log}(0.5) < 0$ and $\text{log}(12) > 0$ are both valid log of unnormalized probabilities. Here, in more detail: Probability: $P(i) = e^{o_i}/\sum_{k=1}^{K}e^{o_k}$ (using softmax as ...


4

Don't get hung up on the word "govern" here. $W_{ax}$, $W_{ay}$ and $W_{aa}$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $W_{aa}$): $W_{ax}$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks) $W_{ay}$ ...


4

BERT is a transformer. A transformer is made of several similar layers, stacked on top of each others. Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n. The hidden state you mention is simply the output of each layer. You might want to quickly look into this explanation of the Transformer architecture : ...


3

Currently your model is not restricted from guessing any number. In this case, you may want to use something like a ReLU activation function to restrict your output domain. The ReLU activation needs to be added to the last layer of your model. If you add it to an intermediate layer but not the last layer then your model can still output negative numbers, e....


3

The authors or the publisher may have corrected the inconsistent notation. The current version online looks like below: Here the $\mathbf{1}_{\text{condition}}$ is the indicator function, as you mentioned. Regarding the two version of softmax, use the first one for calculating gradient. The second version is for improving numerical stability, and ...


3

Actually, I do not think it should be a good way of using RNN only to do object detection work, because there is no "Receptive Field" conception in RNN compared with CNN, which I think should be a key point in doing vision related task.


3

Google translate itself uses Deep learning to translate sentences which can be seen here. You can translate sentences across languages for which you need two things : A large dataset which has pairs of translations ( like English-French ). You can find such a dataset from here. A sequence-to-sequence RNN model. They have Encoder-Decoder architecture which ...


2

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks. Since your question is asking about hidden state initialization: Hidden states on the other hand can be initialized in a variety of ways, initializing to ...


2

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\...


2

Recurrent Neural Networks (RNN) are the state of the art algorithm for sequential data and Long Short-Term Memory (LSTM) networks are an extension for RNN. This method can be used on object detection in case detect object in video or moving images, etc. You can try this https://github.com/tensorflow/models/tree/master/research/lstm_object_detection. It ...


2

Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often. In practice, both approaches are actually used in production and hence are equivalent. It's just a matter of preference. Tensorflow For example, Tensorflow is often optimized for performance. Here's how basic RNN cell is ...


2

Abstractly, if you've already considered decision trees as decomposable into directed acyclic graphs, then one example of you're looking for is, straightforwardly, a Markov Chain. Markov chains can, indeed, model sequences of arbitrary length. Additionally, markov chains containing cycles are possible-- usually referred to as hamiltonian-embedded markov ...


2

That's an interesting proposition ... What would you hope to gain from adding a cyclic component? Various quick answers of my own to this question have lead to existing methods like boosting (ex: xgboost) and ensembling (ex: random forest) which help address bias, variance, smoothing, etc. Maybe you could use a cyclical component to create a ...


2

The goal of decision trees is to partition the feature space into successively smaller regions where each region is best characterized by a single label or value. Adding cyclical components would not help accomplish the goal, it would unnecessarily repeat the modeling fitting for a given region.


2

I would say that option 1 will not work out too well: in my experience, the model will either only be good for the first or last model you train, depending on how much freedom you give the algorithm to change weights as time goes on (e.g. with the learning rate). You really need to decide what you are going to be predicting. Is it the pollution level for a ...


2

Like you said, you can proceed your temporal data with a recurrent network like LSTM. I suggest you simply concatenate your other features to the output of the LSTM (with return_sequence = False). Then add some dense layers (1 could be enough) : this will output the probability of belonging to your classes.


2

I am too impatient to ask the question when I haven't read the chapters before RNN because I have thought it simple. Soon I found it more complicated than I had thought. Then I read the underlying chapter again and I think I can answer my own question now. The expression of (10.18) is as the answer by @user12075: the authors have corrected the notion $\...


2

It's correct. The reason it sounds so weird is that a 1-layer-NN without activation function is simply a linear map, so it's equivalent to any linear model, the only difference being the inputs having some interpretation. This even holds true for any NN, no matter the number of layers, without activation functions. The reason: A k-layer-NN is just k matrix ...


Only top voted, non community-wiki answers of a minimum length are eligible