Hot answers tagged

41

The wrapper with torch.no_grad() temporarily sets all of the requires_grad flags to false. An example is from the official PyTorch tutorial. x = torch.randn(3, requires_grad=True) print(x.requires_grad) print((x ** 2).requires_grad) with torch.no_grad(): print((x ** 2).requires_grad) Output: True True False I recommend you to read all the tutorials ...


23

This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them. One thing Adam does is maintain a rolling geometric mean of recent gradients and squares of the gradients. The squares of the gradients are used to divide (another rolling mean of) the ...


19

What kind of loss function would I use here? Cross-entropy is the go-to loss function for classification tasks, either balanced or imbalanced. It is the first choice when no preference is built from domain knowledge yet. This would need to be weighted I suppose? How does that work in practice? Yes. Weight of class $c$ is the size of largest class ...


16

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. They also have a github repo which is easy to work with.


10

There are a couple of good threads on Reddit right now (here and here). I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic graphs in PyTorch is a 'top down design principle', whereas TensorFlow Fold is bolted on to the original Tensorflow framework, so if you're doing anything reasonably ...


10

Assuming both of x_data and labels are lists or numpy arrays, train_data = [] for i in range(len(x_data)): train_data.append([x_data[i], labels[i]]) trainloader = torch.utils.data.DataLoader(train_data, shuffle=True, batch_size=100) i1, l1 = next(iter(trainloader)) print(i1.shape)


9

with torch.no_grad() will make all the operations in the block have no gradients. In pytorch, you can't do inplacement changing of w1 and w2, which are two variables with require_grad = True. I think that avoiding the inplacement changing of w1 and w2 is because it will cause error in back propagation calculation. Since inplacement change will totally ...


8

Torch.no_grad() deactivates autograd engine. Eventually it will reduce the memory usage and speed up computations. Use of Torch.no_grad(): To perform inference without Gradient Calculation. To make sure there's no leak test data into the model. It's generally used to perform Validation. Reason in this case one can use validation batch of large size.


8

As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's docstrings for figuring out the difference. So, the docstring of the DistributedDataParallel module is as follows: Implements distributed data parallelism at ...


7

Hello and welcome to Stack Exchange! The answer to your question is quite simple: you did not use the correct formula. The formula you used is (assuming we are working with square inputs) $$ W'=\frac{W-F+2P}{S} $$ but the correct formula is $$ W'=\frac{W-F+2P}{S}+1 $$ Now if we redo your calculations starting with $(1 \times 28 \times 28)$ inputs: $$ ...


7

model.cuda() by default will send your model to the "current device", which can be set with torch.cuda.set_device(device). An alternative way to send the model to a specific device is model.to(torch.device('cuda:0')). This, of course, is subject to the device visibility specified in the environment variable CUDA_VISIBLE_DEVICES. You can check GPU usage ...


6

As of August 14, 2017, you can install Pytorch from peterjc123's fork as follows. Currently, python 3.5 and 3.6 are supported. # If your main Python version is not 3.5 or 3.6 conda create -n test python=3.6 numpy pyyaml mkl # for CPU only packages conda install -c peterjc123 pytorch # for Windows 10 and Windows Server 2016, CUDA 8 conda install -c ...


6

The answer may depend on the significance of the length of the input vector or how it originates. However, the simplest solution is usually to know the largest size input and use that as number of vectors. If the given input has lesser length, you can do the padding with zeros or appropriate symbols. So instead of having a vector $[1, 2, 3]$ and $[1, 2, 2, 3]...


6

You can build the same model in pytorch. Then extract weights from tensorflow and assign them manually to each layer in pytorch. Depending on the amount of layers it could be time consuming. Building the model depends on the model and I think not everything is possible in pytorch that is possible in tensorflow. Examples how to assign weights in pytorch and ...


6

If you look at the Module implementation of pyTorch, you'll see that forward is a method called in the special method __call__ : class Module(object): ... def __call__(self, *input, **kwargs): ... result = self.forward(*input, **kwargs) As you construct a Net class by inheriting from the Module class and you override the default behavior ...


6

There are not only 2, but many implementations of BERT. Most are basically equivalent. The implementations that you mentioned are: The original code by Google, in Tensorflow. https://github.com/google-research/bert Implementation by Huggingface, in Pytorch and Tensorflow, that reproduces the same results as the original implementation and uses the same ...


5

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the forward() method of the BERT model, we see the following lines explaining the return types: outputs = (sequence_output, pooled_output,) + encoder_outputs[1:] # add hidden_states and attentions if they are here return outputs # sequence_output, ...


4

I am using tensorboardX. It supports most of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. Haven't tried the rest like audio and graph. But the repo also contains examples for those usecases. Installation can be done easily with pip. It's all explained in the readme. There are also other software which ...


4

DataParallel is easier to debug, because your training script is contained in one process. DataParallel may also cause poor GPU-utilization, because one master GPU must hold the model, combined loss, and combined gradients of all GPUs. For a more detailed explanation, see here.


4

Yes, torch.cat works with backward operation.


4

If you are looking for something easy to use and to read, definitely go for Keras. Example of CNN in Keras : model = Sequential() model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape)) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(...


4

You can implement categorical cross entropy pretty easily yourself. It is calculated as $$ \text{cross-entropy} = -\frac{1}{n} \sum_{i=0}^{n} \sum_{j=0}^m \mathbf{y}_{ij} \log \hat{\mathbf{y}}_{ij} $$ where $n$ is the number of samples in your batch, $m$ is the number of classes, $\mathbf{y}_i$ is the one-hot target for example $i$, $\mathbf{\hat{y}}_i$ ...


4

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being cosine proximity = -1*(Cosine Similarity) of the two vectors. This is done to keep in line with loss functions being minimized in Gradient Descent. To elaborate, Higher the angle between x_pred and x_true. lower is the cosine value. ...


3

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks. Since your question is asking about hidden state initialization: Hidden states on the other hand can be initialized in a variety of ways, initializing to ...


3

You have (at least) two options. You can either: Use a bottleneck architecture where you use pooling layers to reduce the dimensions of your data and then upcast it again using deconvolutional/upsampling layers to go back to the original dimensions. The advantage of this approach is that you use a bigger receptive field for the final output, which means it ...


3

I think you might have misunderstood the fixed number of inputs for the RNN. This is the number of inputs per timestep. All your examples have a fixed number of inputs per timestep: 1! You feed them one at a time to your neural network, finishing with a special "end" token (you could always have a second input for this). Teach it to give no output until it ...


3

There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule? A schedule might be that the LR decreases by 50%, every time the validation loss of 5 epochs in a row does not decrease. This will help get closer and closer to a minimum of the loss. ...


3

In Pytorch you can use cross-entropy loss for a binary classification task. You need to make sure to have two neurons in the final layer of the model. Make sure that you do not add a softmax function. Use the below for resources: https://discuss.pytorch.org/t/do-i-need-to-use-softmax-before-nn-crossentropyloss/16739 https://discuss.pytorch.org/t/why-does-...


3

If you know that your output are positive, I think it makes more sense to enforce the positivity in your neural network by applying relu function or softplus $\ln(1. + \exp(x))$. You could also have a look at Generalized models which extend linear regresssion to cases where the variable to predict is only positive (Gamma regression) or between 0 and 1 (...


Only top voted, non community-wiki answers of a minimum length are eligible