9

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value ($V_{endec}$) for the encoder-decoder attention blocks. the target tokens decoded up to the current decoding step: for the first step, the matrix contains in ...


8

Please see my comment above and this is my answer according to what I understood from your question: As you correctly stated you do not need Clustering but Segmentation. Indeed you are looking for Change Points in your time series. The answer really depends on the complexity of your data. If the data is as simple as above example you can use the difference ...


5

Sequential Data is any kind of data where the order matters as you said. So we can assume that time series is a kind of sequential data, because the order matters. A time series is a sequence taken at successive equally spaced points in time and it is not the only case of sequential data. In the latter the order is defined by the dimension of time. There ...


5

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values correlate with a good or bad performance of the ...


4

One way would be not to approach this as a calculation per session. Most data science solutions like to end up with a number, probability or classification. I suggest you structure your data differently so that you try to answer the question - what next action is likely given the last action. In order to do this you would have to restructure your session ...


4

The answer to your needs is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding. In tensorflow, you can do it with tf.data.experimental.bucket_by_sequence_length. Take into account that previously it was in a different python package (tf.contrib.data.bucket_by_sequence_length), so the ...


3

There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule? A schedule might be that the LR decreases by 50%, every time the validation loss of 5 epochs in a row does not decrease. This will help get closer and closer to a minimum of the loss. ...


3

The original Seq2Seq paper uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed teacher forcing. There exists a simplified architecture in which fixed length encoded input vector is passed to each time step in decoder (analogy-wise, we can say, decoder peeks the encoded input at each time step). ...


3

One way would be to create 20 features (each feature representing a codon). In this way, you would have a dataset with 3190 instances and 20 categorical features. There is no need to treat the sequence as a Markov chain. Once the dataset has been featurized as suggested above, any supervised classifier can work well. I would suggest using a gradient ...


3

As you can't disclose much detail, I'm forced to be a bit generic in my answer. I hope it will be helpful nevertheless. First of all, I would only consider reducing the sequences before classification (be it by using the dot product or something else) if you can make sure that you don't lose information you need for classification afterwards. So this ...


3

Since you are using LSTMs for classification using the multivariate time series data, you need to model your time-series data into a supervised learning problem and specify the previous time steps you need to look before by specifying the time-lag count You need to look into the to_supervised function and specify the number of outputs your model has. In ...


3

It looks like the inverse reinforcement learning problem defined by Stuart Russell as Given measurements of an agent’s behaviour over time, in a variety of circumstances. measurements of the sensory inputs to that agent; a model of the physical environment (including the agent’s body). Determine the reward function that the agent is optimizing. It is ...


2

It makes no sense to re-order inputs in the general case because the order might matter. In your example it does not; you can shuffle the columns as long as the corresponding outputs remain the same. I've seen the input reversed, which is a less arbitrary transformation than the one you cite, to improve prediction in sequence-to-sequence models, though that'...


2

Abstractly, if you've already considered decision trees as decomposable into directed acyclic graphs, then one example of you're looking for is, straightforwardly, a Markov Chain. Markov chains can, indeed, model sequences of arbitrary length. Additionally, markov chains containing cycles are possible-- usually referred to as hamiltonian-embedded markov ...


2

That's an interesting proposition ... What would you hope to gain from adding a cyclic component? Various quick answers of my own to this question have lead to existing methods like boosting (ex: xgboost) and ensembling (ex: random forest) which help address bias, variance, smoothing, etc. Maybe you could use a cyclical component to create a ...


2

The goal of decision trees is to partition the feature space into successively smaller regions where each region is best characterized by a single label or value. Adding cyclical components would not help accomplish the goal, it would unnecessarily repeat the modeling fitting for a given region.


2

You can indeed use the ability of recurrent network like LSTM to handle the varying length problem. But unfortunately if you use keras or Tensorflow, all the Tensor must have the same length in a batch. What you can do : Pad all the sequences with an unused value (typically 0) so that all the sequences have the same length. Use a mask layer just after the ...


2

The point of using any recurrent layer is to have the output be a result of not only a single item independent of other items, but rather a sequence of items, such that the output of the layer's operation on one item in the sequence is the result of both that item and any item before it in the sequence. The number of timesteps defines how long such a ...


2

What you describe sounds a lot like the inception module. You can use Keras to concatenate convolutions with different filter sizes acting on the same input, such as: This will increase the compute time, but might give you what you want. Above image is taken from here, where you can also find the code for building such modules using Keras in the section ...


2

Ordinary least squares (OLS) is an optimization method to find the best parameter estimates for linear regression, gradient descent is another. Regardless of the specific optimization method, linear regression is not appropriate for predicting sequence data. Generally, time series methods are used to predict sequence data.


2

Transformer based architectures are some of the most popular in NLP right now. You can check this blog post for more information: https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html Other than performance, one major advantage of transformers is that operations can be parallelized, making it much faster than RNNs/LSTMs.


1

it sounds like a stochastic process problem. Have you looked into estimating transition matrices for markov chains?


1

The default setting for max_seq_len is 25 as seen here under heading Server API: bert-as-service readme There is an open issue regarding this on the Github repo here and the creator seems to be implementing a feature: bert-as-service issues


1

Looks like you have a classification problem. A simple way to solve this is with a linear regression model. Here is how I would do that with the data you've provided: 1) Determine a "unit" of time, for example 1 minute of keystrokes. Once this process is complete you can tweak the unit of time to see if different intervals give you better results (5 minutes,...


1

Like Ricardo mentioned in his comment on your question, the main step here is finding a distance metric between paths. Then you can experiment with different clustering algorithms and see what works. What comes to mind is dynamic time warping (DTW). DTW gives you a way to find a measure of "distance" (it is actually not strictly a distance metric, but it ...


1

You can try padding the inputs with zero that is of lesser in length.


1

A model is a simplified representation of a complex real world object/phenomena. As a simple example, I see a mountain and because I cannot study the exact shape of the mountain I represent it by a model that it is a triangle. Representing the mountain by a triangle is a model. Someone else comes and makes a better model by saying it is a cone. Every model ...


1

Each input sequence should be padded to the same length. The most common method is to find the longest sequence and then add zeros to all shorter sequences. Most Deep Learning frameworks will have a built-in function to do this. Consistently sized-input data allows common neural networks models (e.g., Convolutional Neural Network and Recurrent Neural ...


1

You might choose to demand predictions only after N steps of your sequence have elapsed. Then, predictions are trust-worthy. You've got to give your LSTM something to begin from, some context so to speak. Usually you sum the errors your network produced across all timesteps, but in such a case you ignore its outputs until the N'th timestep onwards. As ...


1

Is there libraries to analyze sequence with python You can take a look at here. You can also use TensorFlow if your task is sequence classification, but based on comments you have referred that your task is unsupervised. Actually, LSTMs can be used for unsupervised tasks too depending on what you want. Take a look at here. And is it right way to use ...


Only top voted, non community-wiki answers of a minimum length are eligible