Hot answers tagged

266

Deconvolution layer is a very unfortunate name and should rather be called a transposed convolutional layer. Visually, for a transposed convolution with stride one and no padding, we just pad the original input (blue entries) with zeroes (white entries) (Figure 1). In case of stride two and padding, the transposed convolution would look like this (Figure 2)...


57

I think one way to get a really basic level intuition behind convolution is that you are sliding K filters, which you can think of as K stencils, over the input image and produce K activations - each one representing a degree of match with a particular stencil. The inverse operation of that would be to take K activations and expand them into a preimage of ...


48

Step by step math explaining how transpose convolution does 2x upsampling with 3x3 filter and stride of 2: The simplest TensorFlow snippet to validate the math: import tensorflow as tf import numpy as np def test_conv2d_transpose(): # input batch shape = (1, 2, 2, 1) -> (batch_size, height, width, channels) - 2x2x1 image in batch of 1 x = tf....


34

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features. The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from: Classifying plankton with deep neural networks. ...


28

The notes that accompany Stanford CS class CS231n: Convolutional Neural Networks for Visual Recognition, by Andrej Karpathy, do an excellent job of explaining convolutional neural networks. Reading this paper should give you a rough idea about: Deconvolutional Networks Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor and Rob Fergus Dept. of Computer ...


14

Just found a great article from the theaon website on this topic [1]: The need for transposed convolutions generally arises from the desire to use a transformation going in the opposite direction of a normal convolution, [...] to project feature maps to a higher-dimensional space. [...] i.e., map from a 4-dimensional space to a 16-dimensional space, while ...


11

I would not apply convolutional neural networks to your problem (at least from what I can gather from the description). Convolutional nets' strengths and weaknesses are related to a core assumption in the model class: Translating patterns of features in a regular way either has a minor impact on the outcome, or has a specific useful meaning. So a pattern 1 ...


11

We could use PCA for analogy. When using conv, the forward pass is to extract the coefficients of principle components from the input image, and the backward pass (that updates the input) is to use (the gradient of) the coefficients to reconstruct a new input image, so that the new input image has PC coefficients that better match the desired coefficients. ...


8

I have just struggled with this same question for a few hours. Thought I'd share the insite that helped me understand it. The answer is that the filters for the second convolutional layer do not have the same dimensionality as the filters for the first layer. In general, the filter has to have the same number of dimensions as its inputs. So in the first ...


8

I am not sure about the alternatives described above, but the commonly used methodology is: Before the application of the non-linearity, each filter output depends linearly on all of the feature maps before within the patch, so you end up with $k_2$ filters after the second layers. The overall number of parameters is $3 \dot{} 3\dot{}k_1 + k_1\dot{} 5 \dot{}...


8

Convolutions from a DSP perspective I'm a bit late to this but still would like to share my perspective and insights. My background is theoretical physics and digital signal processing. In particular I studied wavelets and convolutions are almost in my backbone ;) The way people in the deep learning community talk about convolutions was also confusing to ...


6

I had a lot of trouble understanding what exactly happened in the paper until I came across this blog post: http://warmspringwinds.github.io/tensorflow/tf-slim/2016/11/22/upsampling-and-image-segmentation-with-tensorflow-and-tf-slim/ Here is a summary of how I understand what is happening in a 2x upsampling: Information from paper What is upsampling? "...


6

In addition to David Dao's answer: It is also possible to think the other way around. Instead of focusing on which (low resolution) input pixels are used to produce a single output pixel, you can also focus on which individual input pixels contribute to which region of output pixels. This is done in this distill publication, including a series of very ...


4

No it is not possible to learn those meta parameters from a data set using a learning algorithm. There is no way to calculate a gradient of any objective function with respect to pool_size and stride params. Even if there were, they are typically discrete values and relatively small integers, so cannot be updated in the same way as e.g. the weights are. ...


3

Two points: Dropout is also usually compared with neural networks ensembles. It seems it has some of the performance benefits of training and averaging several neural networks. Dropout is easier to calibrate than regularization. There is only one hyperparameter which is the dropout rate and people widely use 0.5 while training (and then 1.0 on evaluation of ...


3

Although I agree with Neil Slater's response, you should keep a couple of things in mind. 1) "you never know!" In data exploration, you never know what you may find. If you have a ton of data, perhaps playing around with a 20x20 conv net will give you some decent results. Of course, it would be helpful if there are more than just a few features for it to ...


2

Check this lecture and this visualization Usually it is used type 2.1 convolution. In the input you have $NxMx1$ image, then after first convolution you will obtain $N_1xM_1xk_1$, so your image after first convolution will have $k_1$ channels. The new dimension $N_1$ and $M_1$ will depend on your stride $S$ and padding $P: N_1 = (N - 3 + 2P)/S + 1$, you ...


2

i wrote up some simple examples to illustrate how convolution and transpose convolution is done, and as implemented by software libraries like PyTorch https://makeyourownneuralnetwork.blogspot.com/2020/02/calculating-output-size-of-convolutions.html an example of the visual explanations:


2

You might be interested in this paper that explores a few of the questions you are asking: http://arxiv.org/pdf/1312.6184.pdf. It is aptly titled: "Do Deep Networks Really Need to Be Deep?" The crux of the matter is that deep networks allow for a LOT of non-linearity in the data that is being described. For CIFAR-10, I suspect that something similar to what ...


1

The first layer consists of $k_1$ kernels with size $3 \cdot 3 \cdot 1$ to give $k_1$ feature maps which are stacked depth-wise. The second layer consists of $k_2$ kernels with size $5 \cdot 5 \cdot k_1$ to give $k_2$ feature maps which are stacked depth-wise. That is, the kernels in a convolutional layer span the depth of the output of the previous layer. ...


Only top voted, non community-wiki answers of a minimum length are eligible