Hot answers tagged

4

It is common in applied machine learning to have the model with the lowest generalization error, as measured by score on validation data, also have the biggest delta from the score on the training data. There is nothing inherently wrong with overfitting, it depends on the goal of the project. The typical goal of applied machine learning is high predictive ...


4

Your question is, what model is better between one that seems more overfitted (larger difference between train and eval set) but it has also higher scores or one that has less variance between train and eval set but at the same time it has worst results. Everything assuming that you have done a correct train test split and there is no data leakage and ...


4

Note that a CNN is a feed-forward neural network. Thus, if you understand how to perform backpropagation in feed-forward neural networks, you have it for CNNs. A convolution layer can be understood as a fully connected layer, with the constraints that several edge weights are identical and many edge weights are set to 0. You can also build a pooling layer in ...


3

This is a job for dimensionality reduction techniques. PCA is a simple and intuitive method that you should probably try first. If you find it is insufficient or ineffective, you could try using an autoencoder. As you said, using a CNN would imply a locational relationship between variables. If such a relationship does not exist, the CNN could have ...


3

As @fuwiak mentioned, transfer learning may not work if pre-trained model has been fitted on a "very different" dataset. Typically if the pre-trained network extract information that is not relevant for your problem. Moreover, in the paper License Plate Recognition System Based on Transfer Learning (that you shared with me), they have tried to ...


3

Receptive field refers to the number of input pixels that a convolutional filter will operate on. There's a nice distill article about how to calculate receptive field size for your filters (with a nice visualization of receptive field size) and an interactive calculator here if you're only curious about how receptive field size grows with changes to depth ...


2

In convolution layers, weights are the filter values. Weights of a CNN network is determined by the number and size of filters. Weights doesn't depend on shape of input image. Filters of pre-trained network are capable of detecting different features in an image. Consider sobel filter which detects edges in an image. A 3x3 sobel filter can be used to detect ...


2

In principle, it is to be expected that several feature maps result in zero output all over the image. What the convolutional layers are doing, essentially, is to move a stencil across the image and give a high value where this stencil shape is found in the image, and zero everywhere else. Each of the feature maps, i.e. each of the channels in the ...


2

I glanced over the paper, it seems very interesting. It would be truly fascinating to see if this phase transition from pattern recognition too interpolation of the data actually holds true as the authors claim, though I am a bit sceptical. However, I think you are far away from interpolating your data. Your models are not very complex yet. So for your case ...


2

Setting stride to 0 is not necessary, torch will simply compute with respect to the input tensor sizes, so you can set stride to (1,1). For x of size (batch_size, 3, max_dim_0, max_dim_0) (square image) the tensor output will be of size (batch_size, 32, 1, max_dim_0).


2

i wrote up some simple examples to illustrate how convolution and transpose convolution is done, and as implemented by software libraries like PyTorch https://makeyourownneuralnetwork.blogspot.com/2020/02/calculating-output-size-of-convolutions.html an example of the visual explanations:


2

With GlobalAveragePooling2D, only one Feature per Feature map is selected by averaging every elements of the Feature Map. e.g. if your global average pooling layer input is 220 x 220 x 30 you will find 1x1x30 output. It means that you are finding a global representative feature from every slice. That is Global Average Pooling. No, this isn't specific to ...


1

I think there are three questions here: How to incorporate non-spatial information into the network? When combining different information modalities, a typical approach is to do it at the internal representation level, that is: the point where you lose the spatial information (normally with a flatten operation) after the convolutions. You can have your extra ...


1

As you quoted The padding argument effectively adds dilation * (kernel_size - 1) - padding, so you see the padding value is subtracted, the resulting shape becomes lower. It's a reverse (in some sense) operation to Conv2d, which means the arguments work the opposite way here. And I think this behavior is introduced to make it easier to design neural nets ...


1

Although the first answer has explained the difference, I will add a few other points. If the model is very deep(i.e. a lot of Pooling) then the map size will become very small e.g. from 300x300 to 5x5. Then it is more likely that the information is dispersed across different Feature maps and the different elements of one feature map don't hold much ...


1

Bot the name and input_shape come from the Layer class which Conv2D inherited. In the doc you provide, they are implicitly in **kwargs


1

At least two issues: Negative transfer Transfer learning working if the initial and our problem are similar. Unfortunately, we think that there are similar enough, but its just illusion. Data greedy Often model start working well, if we provide much more data.


1

The backpropagation algorithm attributes a penalty per weight in the network. To get the associated gradient for each weight we need to backpropagate the error back to its layer using the derivative chain rule. Flattening layer The derivative of a layer depends on the function that is being applied. In the case of the flattening layer it is simply reshaping (...


1

Andrew Ng is making this point in comparison to a simple Neural network. Let's say you have a 10x10 image, In a dense neural network, - We will connect every 100 neurons to the 100 in the next layer.(Dense) - Over that, each all will have a distinct weight (No sharing) So, total parm = 10K In a Convolution Neural Network, the approach is as shown in this ...


1

In fact these involve different aspects of parameters in a CNN. Parameter sharing, means one parameter may be shared by more than one input/connection. So this reduces total amount of independent parameters. Parameters shared are non-zero. Sparsity of connections means that some parameters are simply missing (ie are zero), nothing to do with sharing same ...


1

They are defined that way. Have a look a the paper cited in the docstring. The sse is defined in eq. 3, and the scSE is defined in the text in 2.3.


1

Those are the activation maps of the learned features. In the case of this specific model, the filters learn that the "fishy" parts of the image: Head-ish and dorsal fin-ish sections that define convex hulls Scale-ish textures insides the convex hulls and surrounded by water-ish textures outside Orange-ish and gray-ish colors insides the convex hulls and ...


1

You can get the output of your models with model.output or get_layer and combine them with tf.keras.layers.concatenate


1

In many cases in deep learning it works well to start off with a model which has a very high capacity and potentially overfits. From thereon you can reduce the model capacity to narrow the gap between train and validation error. In this chapter of the Deep Learning Book by Goodwell you find a good description of manual hyperparameter selection and how they ...


1

Following up on my comment since I think it will be useful to anyone coming here. "a" can be trainable weight in tf.keras class WeightedSum(layers.Layer): """A custom keras layer to learn a weighted sum of tensors""" def __init__(self, **kwargs): super(WeightedSum, self).__init__(**kwargs) def ...


1

The only difference is when you need to carry out certain operations across the channel axis such as BatchNormalisation. If you look at https://keras.io/layers/normalization/, one of the parameters for batch normalisation is as such: axis: Integer, the axis that should be normalized (typically the features axis). For instance, after a Conv2D layer with ...


1

When we say the filter size if 5x5 it is for an image with 1 input channel, for three the filter size is 5x5x3 (but at a lot of places this additional info is skipped to make things easy to understand). When you apply a kernel of 5x5x3 to an image the output is just one channel. To get an output of 8 channels you need 8 such kernels. In that case, the number ...


1

The interpretation that the 1d convolution given in the OP can be duplicated with four separate fully-connected layers is correct (see diagram). Also, in at least some implementations, kernel weights used during a 1x1 convolution can be made trainable the same way weights in a fully-connected layer can be made trainable. These points made, every fully-...


1

Not sure that still matters for your project but it is important: the Dense layer does not flatten the entry first! It takes the last dimension of the entry tensor and connects it to the neurons of your dense layer. To be sure there is a simple thing to do: count the number of parameters of your layer. In your case, it is: 4096 (number of neurons in dense) ...


1

One possible solution when you do not have enough data is to use Transfer learning. This helps you to improve the performance of your model on the test data set. So, you can easily use one of the available pre-trained models in technical literature and update its weights based on your data. Take a look at this video. It is very helpful and you get a lot of ...


Only top voted, non community-wiki answers of a minimum length are eligible