Hot answers tagged

259

Deconvolution layer is a very unfortunate name and should rather be called a transposed convolutional layer. Visually, for a transposed convolution with stride one and no padding, we just pad the original input (blue entries) with zeroes (white entries) (Figure 1). In case of stride two and padding, the transposed convolution would look like this (Figure 2)...


56

I think one way to get a really basic level intuition behind convolution is that you are sliding K filters, which you can think of as K stencils, over the input image and produce K activations - each one representing a degree of match with a particular stencil. The inverse operation of that would be to take K activations and expand them into a preimage of ...


46

Step by step math explaining how transpose convolution does 2x upsampling with 3x3 filter and stride of 2: The simplest TensorFlow snippet to validate the math: import tensorflow as tf import numpy as np def test_conv2d_transpose(): # input batch shape = (1, 2, 2, 1) -> (batch_size, height, width, channels) - 2x2x1 image in batch of 1 x = tf....


36

Your Example In your example we have 3 input and 2 output units. To apply convolutions, think of those units having shape: [1,1,3] and [1,1,2], respectively. In CNN terms, we have 3 input and 2 output feature maps, each having spatial dimensions 1 x 1. Applying an n x n convolution to a layer with k feature maps, requires you to have a kernel of shape [n,...


34

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features. The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from: Classifying plankton with deep neural networks. ...


29

In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements: It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...] large-batch methods tend to converge to sharp minimizers of ...


28

The notes that accompany Stanford CS class CS231n: Convolutional Neural Networks for Visual Recognition, by Andrej Karpathy, do an excellent job of explaining convolutional neural networks. Reading this paper should give you a rough idea about: Deconvolutional Networks Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor and Rob Fergus Dept. of Computer ...


25

Actually I guess the question is a bit broad! Anyway. Understanding Convolution Nets What is learned in ConvNets tries to minimize the cost function to categorize the inputs correctly in classification tasks. All parameter changing and learned filters are in order to achieve the mentioned goal. Learned Features in Different Layers They try to reduce ...


22

In the paper Batch Normalization,Sergey et al,2015. proposed Inception-v1 architecture which is a variant of the GoogleNet in the paper Going deeper with convolutions, and in the meanwhile they introduced Batch Normalization to Inception(BN-Inception). The main difference to the network described in (Szegedy et al.,2014) is that the 5x5 convolutional ...


22

By default, the filters $W$ are initialised randomly using the glorot_uniform method, which draws values from a uniform distribution with positive and negative bounds described as so: $$W \sim \mathcal{U}\left(\frac{6}{n_{in} + n_{out}}, \frac{-6}{n_{in} + n_{out}}\right),$$ where $n_{in}$ is the number of units that feed into this unit, and $n_{out}$ is ...


20

In the context of image processing, upsampling is a technique for increasing the size of an image. For example, say you have an image with a height and width of $64$ pixels each (totaling $64 \times 64 = 4096$ pixels). You want to resize this image to a height and width of 256 pixels (totaling $256 \times 256 = 65536$ pixels). In the new, larger image you ...


19

Ok, so after a lot of experimentation I have managed to get some results/insights. In the first place, everything being equal, smaller batches in the training set help a lot in order to increase the general performance of the network, as a negative side, the training process is muuuuuch slower. Second point, data is important, nothing new here but as I ...


19

The convolution operation, simply put, is combination of element-wise product of two matrices. So long as these two matrices agree in dimensions, there shouldn't be a problem, and so I can understand the motivation behind your query. A.1. However, the intent of convolution is to encode source data matrix (entire image) in terms of a filter or kernel. More ...


16

ConvNets work because they exploit feature locality. They do it at different granularities, therefore being able to model hierarchically higher level features. They are translation invariant thanks to pooling units. They are not rotation-invariant per se, but they usually converge to filters that are rotated versions of the same filters, hence supporting ...


15

First of all: There is no way to determine a good network topology just from the number of inputs and outputs. It depends critically on the number of training examples and the complexity of the classification you are trying to learn.[1] and Yoshua Bengio has proposed a very simple rule: Just keep adding layers until the test error does not ...


15

Do Read - When Transfer Learning is Disadvantageous ?(thanks to @media) (looks very informative to me, so added here to make this answer a complete one...) Answer to your Question..(starts here) Transfer Learning is What you are Looking for.. When we are given a Deep Learning task, say, one that involves training a Convolutional Neural Network (Covnet) on ...


14

Just found a great article from the theaon website on this topic [1]: The need for transposed convolutions generally arises from the desire to use a transformation going in the opposite direction of a normal convolution, [...] to project feature maps to a higher-dimensional space. [...] i.e., map from a 4-dimensional space to a 16-dimensional space, while ...


13

What you have are predicted class probabilities. Since you are doing binary classification, each output is the probability of the first class for that test example. To convert these to class labels you can take a threshold: import numpy as np probas = np.array([[0.4],[0.7],[0.2]]) labels = (probas < 0.5).astype(np.int) print(labels) [[1] [0] [1]] ...


12

With border mode "valid" you get an output that is smaller than the input because the convolution is only computed where the input and the filter fully overlap. With border mode "same" you get an output that is the "same" size as the input. That means that the filter has to go outside the bounds of the input by "filter size / 2" - the area outside of the ...


12

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source. Forward pass As you observed the forward pass of the convolutional layer can be expressed as $x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$ ...


12

Yes, it may. In machine-learning there is an approach called early stop. In that approach you plot the error rate on training and validation data. The horizontal axis is the number of epochs and the vertical axis is the error rate. You should stop training when the error rate of validation data is minimum. Consequently if you increase the number of epochs, ...


11

I would not apply convolutional neural networks to your problem (at least from what I can gather from the description). Convolutional nets' strengths and weaknesses are related to a core assumption in the model class: Translating patterns of features in a regular way either has a minor impact on the outcome, or has a specific useful meaning. So a pattern 1 ...


11

We could use PCA for analogy. When using conv, the forward pass is to extract the coefficients of principle components from the input image, and the backward pass (that updates the input) is to use (the gradient of) the coefficients to reconstruct a new input image, so that the new input image has PC coefficients that better match the desired coefficients. ...


11

Although many solutions in production systems still use a sliding window as described below in this answer, the field of computer vision is moving quickly. Recent advances in this field include R-CNN and YOLO. Detecting object matches in an image, when you already have an object classifier trained, is usually a matter of brute-force scanning through image ...


11

The reason that sigmoid functions are being replaced by rectified linear units, is because of the properties of their derivatives. Let's take a quick look at the sigmoid function $\sigma$ which is defined as $\frac{1}{1+e^{-x}}$. The derivative of the sigmoid function is $$\sigma '(x) = \sigma(x)*(1-\sigma(x))$$ The range of the $\sigma$ function is between ...


11

Here is the correct formula for computing the size of the output with tf.layers.conv2d_transpose(): # Padding==Same: H = H1 * stride # Padding==Valid H = (H1-1) * stride + HF where, H = output size, H1 = input size, HF = height of filter e.g., if `H1` = 7, Stride = 3, and Kernel size = 4, With padding=="same", output size = 21, with padding=="valid", ...


10

I'd want to add @Neil_Slater's answer by sharing my application. In my application, I want to train a model that can automatically load a chess position from a chess book like this: Before I did anything, I made sure I had a model that can accurately detect a chess piece. It was not a hard-problem because it was like training the MINST digits. I collected ...


10

I guess it has been a mistake. Take a look at here. The other author's were Ilya Sutskever and Geoffrey Hinton. So, AlexNet input starts with 227 by 227 by 3 images. And if you read the paper, the paper refers to 224 by 224 by 3 images. But if you look at the numbers, I think that the numbers make sense only of actually 227 by 227.


8

I am not sure about the alternatives described above, but the commonly used methodology is: Before the application of the non-linearity, each filter output depends linearly on all of the feature maps before within the patch, so you end up with $k_2$ filters after the second layers. The overall number of parameters is $3 \dot{} 3\dot{}k_1 + k_1\dot{} 5 \dot{}...


8

Convolutions from a DSP perspective I'm a bit late to this but still would like to share my perspective and insights. My background is theoretical physics and digital signal processing. In particular I studied wavelets and convolutions are almost in my backbone ;) The way people in the deep learning community talk about convolutions was also confusing to ...


Only top voted, non community-wiki answers of a minimum length are eligible