Skip to main content
11 votes
Accepted

what is the first input to the decoder in a transformer model?

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value (...
noe's user avatar
  • 27k
5 votes

Sequence data vs time series data

Sequential Data is any kind of data where the order matters as you said. So we can assume that time series is a kind of sequential data, because the order matters. A time series is a sequence taken ...
Christos Karatsalos's user avatar
5 votes

How to evaluate sequence to sequence models?

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of ...
mauvilsa's user avatar
  • 151
5 votes
Accepted

In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence?

The answer to your needs is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding. In tensorflow, you can do it with ...
noe's user avatar
  • 27k
4 votes
Accepted

Find average sequence from a set of sequences

One way would be not to approach this as a calculation per session. Most data science solutions like to end up with a number, probability or classification. I suggest you structure your data ...
Snympi's user avatar
  • 242
4 votes
Accepted

Loss on whole sequences in Causal Language Model

The figure and the blog post are simply incorrect. Doing a reverse image search, I see that the image you posted comes from a blog post on Towards Data Science. That image is so wrong. Just think that ...
noe's user avatar
  • 27k
3 votes

Query on unstable loss curves for RNN

There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule? A schedule might be that ...
n1k31t4's user avatar
  • 14.9k
3 votes

Principle behind seq2seq model's example in keras?

The original Seq2Seq paper uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed teacher forcing. There exists a simplified architecture in ...
kamalbanga's user avatar
3 votes
Accepted

LSTM to multivariate sequence classification

Since you are using LSTMs for classification using the multivariate time series data, you need to model your time-series data into a supervised learning problem and specify the previous time steps you ...
vignesh_md's user avatar
3 votes

Recommender Model for Human Action in Income Protection

It looks like the inverse reinforcement learning problem defined by Stuart Russell as Given measurements of an agent’s behaviour over time, in a variety of circumstances. measurements of the sensory ...
Valentas's user avatar
  • 1,239
2 votes
Accepted

Sequence Batching in RNNs

It makes no sense to re-order inputs in the general case because the order might matter. In your example it does not; you can shuffle the columns as long as the corresponding outputs remain the same. ...
Emre's user avatar
  • 10.5k
2 votes

How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

The point of using any recurrent layer is to have the output be a result of not only a single item independent of other items, but rather a sequence of items, such that the output of the layer's ...
mevoki's user avatar
  • 31
2 votes
Accepted

Are there cyclic decision trees?

Abstractly, if you've already considered decision trees as decomposable into directed acyclic graphs, then one example of you're looking for is, straightforwardly, a Markov Chain. Markov chains can, ...
Thomas Cleberg's user avatar
2 votes

Are there cyclic decision trees?

That's an interesting proposition ... What would you hope to gain from adding a cyclic component? Various quick answers of my own to this question have lead to existing methods like boosting (ex: ...
Eduard Gelman's user avatar
2 votes

Are there cyclic decision trees?

The goal of decision trees is to partition the feature space into successively smaller regions where each region is best characterized by a single label or value. Adding cyclical components would not ...
Brian Spiering's user avatar
2 votes

Very long sequence in neural networks

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a ...
Adrien D's user avatar
  • 1,113
2 votes
Accepted

Why do position embeddings work?

The token embeddings are not fixed, they are learned. Therefore, during training, the value learned for the token embeddings is intrinsically one that is useful after adding it up with the positional ...
noe's user avatar
  • 27k
2 votes
Accepted

Machine learning Classification model for binary input and output data

Your problem is one of "sequence classification" for which Recurrent Neural Networks (RNN) e.g. Long short-term memory (LSTM) are generally used. See here ...
rnso's user avatar
  • 1,578
2 votes
Accepted

Multiple merging multiple convolutions

What you describe sounds a lot like the inception module. You can use Keras to concatenate convolutions with different filter sizes acting on the same input, such as: This will increase the compute ...
serali's user avatar
  • 1,252
2 votes
Accepted

Can OLS regression be used to predict from a complete sequence of data?

Ordinary least squares (OLS) is an optimization method to find the best parameter estimates for linear regression, gradient descent is another. Regardless of the specific optimization method, linear ...
Brian Spiering's user avatar
2 votes
Accepted

Can bidirectional RNN use variable sequence length?

The short answer is no, a bidirectional architecture will still take in a variable sequence length. To understand why, you should understand how padding works. For example, let's say you are ...
Derek O's user avatar
  • 354
2 votes

HMM and its competitive alternatives

Transformer based architectures are some of the most popular in NLP right now. You can check this blog post for more information: https://ai.googleblog.com/2017/08/transformer-novel-neural-network....
Anshul G.'s user avatar
  • 535
2 votes

Predict status of upcoming project milestones with intermediate activities

Given the relative complexity of the data to the amount of data, machine learning might not be useful. Here are several different machine learning options: If you ignore the sequence information (aka,...
Brian Spiering's user avatar
2 votes

In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence?

Found a solution, which is to pass a custom batch generator of type keras.utils.Sequence to the model.fit function (where one can write any logic to construct batches and to modify/augment training ...
Tonnz's user avatar
  • 63
2 votes

Sequence Embedding using embedding layer: how does the network architecture influence it?

My understanding is you want to classify protein sequences. For example, the sequence HWLQMRDSMNTYNNMVNRCFATCIRSFQEKKVNAEEMDCTKRCVTKFVGYSQRVALRFAE belongs to a dog. ...
Brian Spiering's user avatar
2 votes

Can MLP model sequential data?

Yes, it can be learned. In fact many time series problems are solved with non-RNN/LSTM/1D-CNN architectures due to their simplicity. The reason the more exotic architectures are used are to increase ...
GooJ's user avatar
  • 435
1 vote

Statistical methods for Sequence learning

it sounds like a stochastic process problem. Have you looked into estimating transition matrices for markov chains?
Kane Chua's user avatar
  • 206
1 vote
Accepted

bert-as-service maximum sequence length

The default setting for max_seq_len is 25 as seen here under heading Server API: bert-as-service readme There is an open issue regarding this on the Github repo ...
mLstudent33's user avatar
1 vote

Classification of keystrokes

Looks like you have a classification problem. A simple way to solve this is with a linear regression model. Here is how I would do that with the data you've provided: 1) Determine a "unit" of time, ...
bstrain's user avatar
  • 421
1 vote
Accepted

What clustering algorithm is appropriate for clustering paths?

Like Ricardo mentioned in his comment on your question, the main step here is finding a distance metric between paths. Then you can experiment with different clustering algorithms and see what works. ...
Eric A. Bunch's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible