1
$\begingroup$

Gaussian kernel

$k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$

has a hyperparameter $\sigma$.

I know grid search cross validation, but this would require a lot of computation since computational cost of kernel method scales with the number of samples to the power of 2.

$\endgroup$
  • $\begingroup$ Perhaps sisualize the kernel values for different bandwidth values. Compare to your dataset $\endgroup$ – Carl Rynegardh May 17 '18 at 1:45
1
$\begingroup$

As shown in wikipedia for KDE, a rule-of-thumb bandwidth estimator can be given if the underlying density for your data is Gaussian. This estimator is given by: $h = (\frac{4\hat{\sigma}}{3n})^{1/5}$, where $h$ is the bandwidth of your KDE estimation, $n$ the number of data and $\hat{\sigma}$ the estimation of the standard deviation of your sample.

If the underlying distribution for your data is not gaussian, you can still try with this bandwidth, but it might smooth everything too much. In that case, you should go for cross-validation with smaller bandwidths.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I think your answer is not applicable here. The Gaussian kernel in KDE is equivalent to the standard normal distribution. The stated kernel function in the question however is not even a density. In contrast, it is the radial basis function which is also sometimes called Gaussian kernel in the context of e.g. SVMs. Then, you would rather use grid search, random search or other more advance hyperparameter optimization procedures. $\endgroup$ – stats_guy Feb 9 '19 at 14:05
0
$\begingroup$

After searching a few, I find this problem is really simple actually in some circumestances.

In Gausiann process regression, one can simply refer to Automatic relevance determination (ARD), which is to optimize the hyperparameters in kernels and also the observation noise variance, by maximize the marginal likelihood, using gradient descent!

This has been pretty standard and mature.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.