Skip to main content
61 votes
Accepted

What is the use of torch.no_grad in pytorch?

The wrapper with torch.no_grad() temporarily sets all of the requires_grad flags to false. An example is from the official ...
Adrien D's user avatar
  • 1,113
39 votes
Accepted

What loss function to use for imbalanced classes (using PyTorch)?

What kind of loss function would I use here? Cross-entropy is the go-to loss function for classification tasks, either balanced or imbalanced. It is the first choice when no preference is built from ...
Esmailian's user avatar
  • 9,352
31 votes
Accepted

Strange behavior with Adam optimizer when training for too long

This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them. One thing Adam does is ...
Neil Slater's user avatar
16 votes

Loading own train data and labels in dataloader using pytorch?

Assuming both of x_data and labels are lists or numpy arrays, ...
ASHu2's user avatar
  • 260
16 votes
Accepted

model.cuda() in pytorch

model.cuda() by default will send your model to the "current device", which can be set with torch.cuda.set_device(device). An ...
noe's user avatar
  • 27k
15 votes

What is the use of torch.no_grad in pytorch?

Torch.no_grad() deactivates autograd engine. Eventually it will reduce the memory usage and speed up computations. Use of ...
Rohan Shetty's user avatar
13 votes
Accepted

How does the forward method get called in this pyTorch conv net?

If you look at the Module implementation of pyTorch, you'll see that forward is a method called in the special method __call__ : ...
Elliot's user avatar
  • 1,081
12 votes
Accepted

Determining size of FC layer after Conv layer in PyTorch

Hello and welcome to Stack Exchange! The answer to your question is quite simple: you did not use the correct formula. The formula you used is (assuming we are working with square inputs) $$ W'=\...
RaptorDotCpp's user avatar
12 votes

How to get sentence embedding using BERT?

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the ...
zachdj's user avatar
  • 2,734
11 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
noe's user avatar
  • 27k
10 votes

What is the use of torch.no_grad in pytorch?

with torch.no_grad() will make all the operations in the block have no gradients. In pytorch, you can't do inplacement changing of w1 and w2, which are two ...
Jianing Lu's user avatar
10 votes
Accepted

PyTorch vs. Tensorflow Fold

There are a couple of good threads on Reddit right now (here and here). I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic ...
christopherlovell's user avatar
9 votes
Accepted

What is the difference between Pytorch's DataParallel and DistributedDataParallel?

As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's ...
Dawny33's user avatar
  • 8,296
8 votes
Accepted

An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

The answer may depend on the significance of the length of the input vector or how it originates. However, the simplest solution is usually to know the largest size input and use that as number of ...
Dipan Mehta's user avatar
8 votes
Accepted

How to convert my tensorflow model to pytorch model?

You can build the same model in pytorch. Then extract weights from tensorflow and assign them manually to each layer in pytorch. Depending on the amount of layers it could be time consuming. Building ...
keiv.fly's user avatar
  • 1,269
8 votes

Is time series forecasting possible with a transformer?

Transformers can be used for time series forecasting. See the following articles: Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al. Deep Transformer Models for Time ...
Brian O'Donnell's user avatar
7 votes
Accepted

Does torch.cat work with backpropagation?

Yes, torch.cat works with backward operation.
Louis T's user avatar
  • 1,148
7 votes

Loading own train data and labels in dataloader using pytorch?

I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
Johannes's user avatar
  • 171
7 votes
Accepted

Implementation of BERT using Tensorflow vs PyTorch

There are not only 2, but many implementations of BERT. Most are basically equivalent. The implementations that you mentioned are: The original code by Google, in Tensorflow. https://github.com/...
noe's user avatar
  • 27k
6 votes

How to install pytorch in windows?

As of August 14, 2017, you can install Pytorch from peterjc123's fork as follows. Currently, python 3.5 and 3.6 are supported. ...
Hossein's user avatar
  • 565
6 votes

What is the difference between Pytorch's DataParallel and DistributedDataParallel?

DataParallel is easier to debug, because your training script is contained in one process. DataParallel may also cause poor GPU-...
Alex Yin's user avatar
6 votes

Differences between gradient calculated by different reduction methods in PyTorch

Let's start by just recalling what each of these means. Reduction 'none' means compute batch_size gradient updates independently for the loss with respect to each ...
an1lam's user avatar
  • 171
6 votes

Autoencoder: using cosine distance as loss function

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being ...
Nasheed Yasin's user avatar
6 votes
Accepted

What is the difference between FC and MLP in as used in PointNet?

Well, I have found a lot of confusing materials on the Internet and even in papers. MLP and FC layers are kind of similar, but not the same actually. 1) Short answer: MLP - mini neural network with 3 ...
Dmitry Demidov's user avatar
6 votes
Accepted

Changing output size from a model

In general, when you want to use an existing net for a new task that has different output requirements, you would usually remove the output layer and replace it with a new one. Freeze all of the ...
MuhammedYunus's user avatar
5 votes

How/What to initialize the hidden states in RNN sequence-to-sequence models?

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural ...
Mati K's user avatar
  • 95
5 votes

Determining size of FC layer after Conv layer in PyTorch

If you are willing to give additional input parameters to the CNN, you can calculate it automatically. Input dim for MNIST is input_dim=(1,28,28). So that, I can ...
komunistbakkal's user avatar
5 votes

How to force pytorch model to predict only positive values

If you know that your output are positive, I think it makes more sense to enforce the positivity in your neural network by applying relu function or softplus $\ln(1. + \exp(x))$. You could also have a ...
Robin Nicole's user avatar
5 votes
Accepted

Tensorflow (or Keras) vs. Pytorch vs. some other ML library for implementing a CNN

If you are looking for something easy to use and to read, definitely go for Keras. Example of CNN in Keras : ...
Astariul's user avatar
  • 1,004

Only top scored, non community-wiki answers of a minimum length are eligible