Hot answers tagged

52 votes
Accepted

What is the use of torch.no_grad in pytorch?

The wrapper with torch.no_grad() temporarily sets all of the requires_grad flags to false. An example is from the official ...
user avatar
  • 993
31 votes
Accepted

What loss function to use for imbalanced classes (using PyTorch)?

What kind of loss function would I use here? Cross-entropy is the go-to loss function for classification tasks, either balanced or imbalanced. It is the first choice when no preference is built from ...
user avatar
  • 8,479
27 votes
Accepted

Strange behavior with Adam optimizer when training for too long

This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them. One thing Adam does is ...
user avatar
  • 27.2k
22 votes

How to get sentence embedding using BERT?

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. They also have a github repo which is easy to work with.
user avatar
15 votes

Loading own train data and labels in dataloader using pytorch?

Assuming both of x_data and labels are lists or numpy arrays, ...
user avatar
  • 250
14 votes
Accepted

model.cuda() in pytorch

model.cuda() by default will send your model to the "current device", which can be set with torch.cuda.set_device(device). An ...
user avatar
  • 15k
12 votes

What is the use of torch.no_grad in pytorch?

Torch.no_grad() deactivates autograd engine. Eventually it will reduce the memory usage and speed up computations. Use of ...
user avatar
12 votes
Accepted

How does the forward method get called in this pyTorch conv net?

If you look at the Module implementation of pyTorch, you'll see that forward is a method called in the special method __call__ : ...
user avatar
  • 1,071
11 votes
Accepted

Determining size of FC layer after Conv layer in PyTorch

Hello and welcome to Stack Exchange! The answer to your question is quite simple: you did not use the correct formula. The formula you used is (assuming we are working with square inputs) $$ W'=\...
user avatar
10 votes

What is the use of torch.no_grad in pytorch?

with torch.no_grad() will make all the operations in the block have no gradients. In pytorch, you can't do inplacement changing of w1 and w2, which are two ...
user avatar
10 votes
Accepted

PyTorch vs. Tensorflow Fold

There are a couple of good threads on Reddit right now (here and here). I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic ...
user avatar
9 votes
Accepted

What is the difference between Pytorch's DataParallel and DistributedDataParallel?

As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's ...
user avatar
  • 7,996
9 votes

How to get sentence embedding using BERT?

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the ...
user avatar
  • 2,261
8 votes
Accepted

An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

The answer may depend on the significance of the length of the input vector or how it originates. However, the simplest solution is usually to know the largest size input and use that as number of ...
user avatar
7 votes

Loading own train data and labels in dataloader using pytorch?

I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
user avatar
  • 171
7 votes
Accepted

Implementation of BERT using Tensorflow vs PyTorch

There are not only 2, but many implementations of BERT. Most are basically equivalent. The implementations that you mentioned are: The original code by Google, in Tensorflow. https://github.com/...
user avatar
  • 15k
6 votes
Accepted

Does torch.cat work with backpropagation?

Yes, torch.cat works with backward operation.
user avatar
  • 1,108
6 votes

How to install pytorch in windows?

As of August 14, 2017, you can install Pytorch from peterjc123's fork as follows. Currently, python 3.5 and 3.6 are supported. ...
user avatar
  • 375
6 votes
Accepted

How to convert my tensorflow model to pytorch model?

You can build the same model in pytorch. Then extract weights from tensorflow and assign them manually to each layer in pytorch. Depending on the amount of layers it could be time consuming. Building ...
user avatar
  • 1,169
6 votes

Autoencoder: using cosine distance as loss function

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being ...
user avatar
5 votes

What is the difference between Pytorch's DataParallel and DistributedDataParallel?

DataParallel is easier to debug, because your training script is contained in one process. DataParallel may also cause poor GPU-...
user avatar
5 votes

Differences between gradient calculated by different reduction methods in PyTorch

Let's start by just recalling what each of these means. Reduction 'none' means compute batch_size gradient updates independently for the loss with respect to each ...
user avatar
  • 161
5 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
user avatar
  • 15k
5 votes

Is time series forecasting possible with a transformer?

Transformers can be used for time series forecasting. See the following articles: Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al. Deep Transformer Models for Time ...
user avatar
4 votes

An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

I think you might have misunderstood the fixed number of inputs for the RNN. This is the number of inputs per timestep. All your examples have a fixed number of inputs per timestep: 1! You feed them ...
user avatar
4 votes

How/What to initialize the hidden states in RNN sequence-to-sequence models?

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural ...
user avatar
  • 85
4 votes

Any pytorch tools to monitor neural network's training?

I am using tensorboardX. It supports most of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. Haven't tried the rest like audio and graph. But the repo ...
user avatar
4 votes

Determining size of FC layer after Conv layer in PyTorch

If you are willing to give additional input parameters to the CNN, you can calculate it automatically. Input dim for MNIST is input_dim=(1,28,28). So that, I can ...
user avatar
4 votes
Accepted

Tensorflow (or Keras) vs. Pytorch vs. some other ML library for implementing a CNN

If you are looking for something easy to use and to read, definitely go for Keras. Example of CNN in Keras : ...
user avatar
  • 799

Only top scored, non community-wiki answers of a minimum length are eligible