Transformers can be used for time series forecasting. See the following articles:
Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al.
Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case, by Neo Wu, Bradley Green, Xue Ben, & Shawn O'Banion
The Time Series Transformer, by Theodoras Ntakouris
The size of the input image is usually decided by computational resources you have. If input image has larger dimension you need to increase filter size, then model will have more parameters, hence you need more computational resources and longer training time.
Usually, I resize the images to min(heights of all images), min(widths of all images).
If there ...
That part of the code will select the samples that belong to a specific batch. The for loop first loops over the data in train_X in steps of BATCH_SIZE, which means that the variable i holds the first index for each batch in the training dataset. The rest of the samples for the batch are then the ones after that index up to the sample which completes the ...
Here it's a mistake to one-hot-encode the class, because it turns the task into multi-label classification instead of regular multi-class classification. In your task an instance can only have a single class, so the class should be encoded as an int (for example with LabelEncoder).
This is why the predicted probabilities don't sum to 1, because in multi-...
torch.argmax has an extra argument dim which you can specify such that the maximum value is taken over a specific dimension. If you specify the dimension which represents the number of images it will return an array of indices where each value is for one image. For example:
# 3 images with 5 classes
t = torch.randn(3, 5)
# tensor([[-1.2917, 1....
Installed a dualboot a week ago on a similar setup. I didn't use Pytorch or TF yet, but I didn't ran into much struggle for CUDA and CudNN installation. Here is what I've done and the few problems I've run into :
Downloaded Ubuntu 20.04 LTS, created a USB bootable with Rufus.
Created a 30 GB partition on my disk (only 10GB are required, but why not), ...
Commonly you divide the domain up into different accuracy,stability regimes and apply a different approximation within each.
For $|x|<1$ you can use a polynomial approximation, like Chebyshev. For $1<|x|\leq E$, $E$ depending on implementation of the formula, you can use a related formula to the one you described.
For $|x|>E$ you can use $\tanh(x) \...
Got the very same error recently.
Your network is usually defined as a class (here class EfficientNet(nn.Module). It seems when we load a model, it needs the class to be defined so it can instantiate it.
In my case, the class was defined in the training .py file. So what I did to fix that error was just copy-paste (it seems importing it didn't work for me, ...
The loop for the validation data would look very similar to your training loop, but for your validation data you only have to calculate the loss and not backpropagate the error. It would look something like this:
BATCH_SIZE = 64
EPOCHS = 10
for epoch in range(EPOCHS):
# training loop
The problem seems to come from your learning rate and the non-normalization of your data. Here your network is clearly unstable and thus gets to sky high values (10^20) which lead to NaN values. A typical learning rate for SGD is 0.001, but this is for normalized datas (inputs-outpus between 0 and 1). Here your inputs and ouputs have high values, that are ...
Nice question! I'm looking at the PyTorch documentation: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html
If i get that right, lstm_out gives you the output features of the LSTM's last layer, for all the tokens in the sequence. This might mean that if your LSTM has two layers and 10 words, assuming batch size of 1, you'll get an output tensor of (...
For that, you should repeat b 200 times in the appropriate dimension this way:
c = torch.cat([a, torch.unsqueeze(b, 1).repeat(1, 200, 1)], dim=2)
As desired, the shape of the result is torch.Size([500, 200, 15])
It is the learning rate that you set on the optimizer for your problem. It requires a fair bit of experimentation to figure out what learning rate works best for your problem. It is a more of an art than a science, though there exists heuristics on an optimal range of values to select from.
For most problems, the default learning rate should be a good ...
CNNs work because of those zeros (the zeros create the boundary on which the change is values is the highest i.e. what networks learn). They are not the problem. Look at your regularizing your network (use dropout, reduce filters if overfitting or increase if underfitting).
The reason the tensor takes up so much memory is because by default the tensor will store the values with the type torch.float32. This data type will use 4kb for each value in the tensor (check using .element_size()), which will give a total of ~48GB after multiplying with the number of zero values in your tensor (4 * 2000 * 2000 * 3200 = 47.68GB). What you ...
About the need for tgt_key_padding_mask
While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this.
For left padding to be handled correctly, you must mask the padding tokens, because the ...
How to obtain the same behavior as v3.x in v4.x
In order to obtain the same behavior as version v3.x, you should install sentencepiece additionally:
In version v3.x:
pip install transformers
to obtain the same in version v4.x:
pip install transformers[sentencepiece]
pip install transformers sentencepiece
When they are talking about aggregating their outputs, they mean the final embeddings (just before the classification layer) not the output of the network itself.
You take the embeddings from both of the networks and concatenate them vertically. This concatenated embedding you use to predict the final output you wanted.
Torch has a function torch.cat for ...
I don't have any answer as to why this is. The hand-wavy answer I once received was that PyTorch doesn't effectively utilize large number of CPU cores. But as to your second question, I have experienced the same issue using the python framework and had success using the torch.set_num_threads(n) function to artificially limit cores on machines with more CPUs ...
I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else.
With time series you aren't changing something to something else, you're trying to find ...
Disclaimer: I'm not at all expert in predicting stock market.
The error between the actual and the true evolution might simply be caused by the fact that the model is reproducing a pattern it observed in the past data: if in the past a small plateau was more often followed by a progressive decrease, then it makes sense to predict a decrease. This might ...
CNN-LSTM are used in text recognition. How is text structured? Characters are stacking in a horizontal manner. Do the same with your MNIST images and for the ground truth make proper labels (should be of 4 size).