Hot answers tagged

16

A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source. Forward pass As you observed the forward pass of the convolutional layer can be expressed as $x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$ ...


8

Even though I am more familiar with the use of RBF kernel with Gaussian Processes, I think your intuition is correct since, generally speaking, a larger lengthscale means that the learnt function varies less in that direction, which is another way of saying that that feature is irrelevant for the learnt function. So if you have to choose which feature is ...


3

filters are the numbers of kernels or feature detectors that we choose for the convolutional layer to learn. in the end, the number of feature maps that we get equals to this number of filters that we have declared here. Now, what are these feature detectors? these are matrixes usually smaller (much smaller) than the input images. It's occasionally $3by3$, $...


3

Both formulations lead to the same solution if you correctly choose $C$ for both cost functions and if $C>0$. If we have the regularized loss $$J_1=\dfrac{1}{2}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}C\sum_{k=0}^pw_k^2$$ we will have strong regularization for larger $C$ and small regularization for small positive $C$. If we devide the loss $J_1$ by the positive ...


3

I'd say there is no direct relation between the kernel size and the accuracy. If you start using larger kernel you may start loosing details in some smaller features (where 3x3 would detect them better) and in other cases, where your dataset has larger features the 5x5 may start detect features that 3x3 misses. So, I'd say "no". Anyway, if you add a ...


3

Odd numbers, usually, but not necessarily. This can depend on the problem at hand. The odd length of the side of the kernel is used in order to emphasize the center of the kernel. For a more detailed discussion of the geometry of kernels and convnets have a look at this thesis.


2

Without knowing the geometry of your data, determining a kernel is generally achieved through trial and error. The radial basis function (RBF) kernel is a good starting choice because most data are not linearly separable. Fortunately training an SVM is fast, so brute-forcing the kernel search is not a terrible method. For selecting the C and gamma values, ...


2

There is vast literature for bandwidths. Different bandwidth perform good in different situations. Its not necessary to use Rule of Thumb bandwidth with normal data. For instance, see Silverman 1986. "Density Estimation".


2

Mercer's Theorem and infinite-dimensional spaces aren't used directly. It justifies use of things like the Gaussian kernel in SVMs. Mercer's theorem says this kernel is just an inner product in some other space, but we need not figure out what that space is, or a mapping to it. The fact that it exists is essential to proving that the SVM still works.


2

Does your CNN contain pooling layers? They are used for used for handling invariances as explained in https://stats.stackexchange.com/a/239079


2

Learning for a CNN depends on the width and depth of a network. Wider and deeper networks can learn more complex data structures, including data augmented images. Increasing the width and depth of a network increase the capacity of the model to learn features. Width is generally associated with the number of features. The wider a network is the more ...


2

There is no direct relationship between these two concepts. However we can find some indirect ones. According to Merriam Webster, kernel means a central or essential part which hints why they are called "kernel". Specifically, deciding "how to measure point-point similarity (a.k.a. kernel function)" is the central part of kernel methods, and deciding "...


2

The kernel matrix $K$ or Gram Matrix is a positive-semidefinite symmetric matrix that has the form: $$ K(x_1,x_2,\dots,x_n) = \begin{vmatrix} \langle x_1,x_1 \rangle & \langle x_1,x_2 \rangle & \dots & \langle x_1,x_n \rangle \\ \langle x_2,x_1 \rangle & \langle x_2,x_2 \rangle & \dots & \langle x_2,x_n \rangle \\ \vdots & \vdots ...


2

Normally a convolutional neural network will get flattened into a single column vector after the convolutions and then maybe be processed by dense layer. In this model, the convolution $1\times1$ is used as an output layer. It will have $C$ channels like every other layer, but it is not dilatated. Hence, you can use this layer as the input to other ...


2

First, please note that spectral clustering is very sensitive to the affinity kernel. With the standard RBF kernel, my experience is that spectral clustering often isolates outliers (in the spectral space), leaving clusters with numerous observations which can be separated by great distances. This is a major difference with direct k-means: there is no notion ...


2

I think you can interpret autoencoders as essentially performing kernel PCA, but with an extremely complex kernel. Like, really, stupidly complex. The kinds of things you can accomplish with autoencoders are spectacular. For example, consider BicycleGAN. If kernel PCA is a step above PCA, autoencoders are miles away. If you have an application where PCA ...


2

Yep, this is the correct interpretation. The kernels make a difficult classification problem into a much simpler one by making the data linearly separable by transforming it into a higher dimension. I think this image does a good job of illustrating that.


2

I am using both almond and toree. For executing only scala code, I am using almond. If you want to learn new features in scala (ex: 2.13), then use almond(Add Scala 2.13.4 support in almond v0.11.0). Almond might start supporting scala 3.0 once it is released. For spark, I am using toree. Toree works fine for me. I am using toree 0.5.0 with spark 3.0.0. ...


2

PCA is the best (in the mean-squared error sense) linear decomposition method. PCA is defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest ...


1

The differences are indeed not too large. There is a paper called Kernel k-means, Spectral Clustering and Normalized Cuts by Inderjit S. Dhillon, Yuqiang Guan, Brian Kulis from KDD 2004 that is discussing that relationship. The authors show that spectral clustering of normalized cuts is a special case of weighted kernel k-means. The reason is that the "...


1

Spectral clustering does not make sense on low dimensional data. In fact, it will first project your data in a 4 dimensional embedding and then run k-means there. That is likely where this is behavior comes from. I'd rather use the actual positions and a distance tolerance threshold.


1

It doesn't mean that $\langle x, y \rangle = \langle \phi(x), \phi(y)\rangle$. Kernel method in general means that for an algorithm that involve $\langle x, y\rangle$, we can replace it with a function $K(x,y)=\langle \phi(x), \phi(y)\rangle$ where computing $K(x,y)$ is easy. It is known that for $K$ that satisfies Mercer's theorem, there is a corresponding ...


1

Let us say $K$ is an $m \times n$ matrix. You want to find a vector $a$ that maximizes $\frac{||Ka||}{||a||}$. Now maximizing $\frac{||Ka||}{||a||}$ implies maximizing its square: $$\begin{eqnarray*} \frac{||Ka||^2}{||a||^2} \\ = \frac{(Ka)^TKa}{a^Ta} \\ = \frac{a^TK^T Ka}{a^Ta} \\ = \frac{a^T(K^TK) a}{a^Ta} \\ = \frac{a^TSa}{a^Ta} \end{eqnarray*} $$ where $...


1

The technique you have mentioned is called "Depth-wise Separable Convolutions" and it relies on the idea that spatial and depth-wise information can be decoupled. Its main advantage is that compared with standard convolution, since it doesn't need to perform convolution across all channels, it has considerably less connections, (so less parameters) which ...


1

I want to identify the main important relationship among the features and the prices. I don't need a predictor. For example, at the end I will discover that the price depends principally by #rooms/perimeter and #neighbours^2. If it depends principally by #neighbours^2, it depends by #neighbours to the same extent. The same hold for the other combinations. ...


1

As far as I know, kernel methods cannot deal with categorical variables (don't now whether it is the case). In addition, you will have to use indirect methods to evaluate the variable importance. This could work, although I havent tested it yet: Giam, X., Olden, J.D., 2015. A new R2-based metric to shed greater insight on variable importance in artificial ...


1

A 3x3 convolution can do things a 1x3 followed by a 3x1 cannot do, for instance diagonal filtering. As you said, it's all about the number of parameters, the first convolution has far more capabilities than the second solution.


1

Lots of the dimensionality problems are trial and errors at first. Only very few datasets are linear "manifolds" that can be described with PCA. But it's a good start to see if there are disjoint sets, or if you can figure out some structure in 2D or 3D. KPCA is a good technique, but there are many different kernels you can try. You may want to start with ...


1

I think the Rbf (Radial basis function) kernel (Wikipedia for rbf) is the most suited for this problem, as its contour plots are circles. For instance, this blog solves very similar problems using the same kernel.


1

Equation (4) simply gives us an identity (without proof): $$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$ Let's check that it does indeed hold. The left hand side equals \begin{align*} (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\ &= \left(RB^{-T}(P^{-1} + B^TR^{-1}...


Only top voted, non community-wiki answers of a minimum length are eligible