62
votes
Accepted
What is the use of torch.no_grad in pytorch?
The wrapper with torch.no_grad() temporarily sets all of the requires_grad flags to false. An example is from the official ...
- 1,123
39
votes
Accepted
What loss function to use for imbalanced classes (using PyTorch)?
What kind of loss function would I use here?
Cross-entropy is the go-to loss function for classification tasks, either balanced or imbalanced. It is the first choice when no preference is built from ...
- 9,442
31
votes
Accepted
Strange behavior with Adam optimizer when training for too long
This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.
One thing Adam does is ...
- 29.2k
16
votes
Loading own train data and labels in dataloader using pytorch?
Assuming both of x_data and labels are lists or numpy arrays,
...
- 260
16
votes
Accepted
model.cuda() in pytorch
model.cuda() by default will send your model to the "current device", which can be set with torch.cuda.set_device(device).
An ...
- 27.5k
15
votes
What is the use of torch.no_grad in pytorch?
Torch.no_grad() deactivates autograd engine. Eventually it will reduce the memory usage and speed up computations.
Use of ...
- 191
13
votes
Accepted
How does the forward method get called in this pyTorch conv net?
If you look at the Module implementation of pyTorch, you'll see that forward is a method called in the special method __call__ :
...
- 1,091
12
votes
Accepted
Determining size of FC layer after Conv layer in PyTorch
Hello and welcome to Stack Exchange!
The answer to your question is quite simple: you did not use the correct formula.
The formula you used is (assuming we are working with square inputs)
$$
W'=\...
- 236
12
votes
How to get sentence embedding using BERT?
Which vector represents the sentence embedding here? Is it hidden_reps or cls_head?
If we look in the ...
- 2,772
11
votes
Accepted
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
- 27.5k
10
votes
What is the use of torch.no_grad in pytorch?
with torch.no_grad()
will make all the operations in the block have no gradients.
In pytorch, you can't do inplacement changing of w1 and w2, which are two ...
- 101
10
votes
Accepted
What is the difference between Pytorch's DataParallel and DistributedDataParallel?
As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's ...
- 8,416
10
votes
Accepted
PyTorch vs. Tensorflow Fold
There are a couple of good threads on Reddit right now (here and here).
I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic ...
8
votes
Accepted
An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs
The answer may depend on the significance of the length of the input vector or how it originates.
However, the simplest solution is usually to know the largest size input and use that as number of ...
- 226
8
votes
Accepted
How to convert my tensorflow model to pytorch model?
You can build the same model in pytorch. Then extract weights from tensorflow and assign them manually to each layer in pytorch. Depending on the amount of layers it could be time consuming. Building ...
- 1,299
8
votes
Is time series forecasting possible with a transformer?
Transformers can be used for time series forecasting. See the following articles:
Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al.
Deep Transformer Models for Time ...
- 286
7
votes
Accepted
7
votes
Loading own train data and labels in dataloader using pytorch?
I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
- 171
7
votes
Accepted
What is the difference between FC and MLP in as used in PointNet?
Well, I have found a lot of confusing materials on the Internet and even in papers. MLP and FC layers are kind of similar, but not the same actually.
1) Short answer:
MLP - mini neural network with 3 ...
- 121
7
votes
Accepted
Implementation of BERT using Tensorflow vs PyTorch
There are not only 2, but many implementations of BERT. Most are basically equivalent.
The implementations that you mentioned are:
The original code by Google, in Tensorflow. https://github.com/...
- 27.5k
6
votes
How to install pytorch in windows?
As of August 14, 2017, you can install Pytorch from peterjc123's fork as follows. Currently, python 3.5 and 3.6 are supported.
...
- 565
6
votes
What is the difference between Pytorch's DataParallel and DistributedDataParallel?
DataParallel is easier to debug, because your training script is contained in one process. DataParallel may also cause poor GPU-...
- 61
6
votes
Differences between gradient calculated by different reduction methods in PyTorch
Let's start by just recalling what each of these means. Reduction 'none' means compute batch_size gradient updates independently for the loss with respect to each ...
- 171
6
votes
Autoencoder: using cosine distance as loss function
Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being ...
- 161
6
votes
Accepted
How does the Transformer predict n steps into the future?
The Transformer is a seq2seq model.
At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the ...
- 27.5k
6
votes
Accepted
Changing output size from a model
In general, when you want to use an existing net for a new task that has different output requirements, you would usually remove the output layer and replace it with a new one.
Freeze all of the ...
- 806
5
votes
How/What to initialize the hidden states in RNN sequence-to-sequence models?
It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural ...
- 95
5
votes
Determining size of FC layer after Conv layer in PyTorch
If you are willing to give additional input parameters to the CNN, you can calculate it automatically.
Input dim for MNIST is input_dim=(1,28,28). So that, I can ...
- 111
5
votes
How to force pytorch model to predict only positive values
If you know that your output are positive, I think it makes more sense to enforce the positivity in your neural network by applying relu function or softplus $\ln(1. + \exp(x))$. You could also have a ...
- 529
Only top scored, non community-wiki answers of a minimum length are eligible