In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x) \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\right] + \sigma_\epsilon^2 \\ = Bias(x)^2 + Variance(x) + Irreducible, $$ the first two terms depend on the argument $x$. So, in principle, if we choose a set of hyperparameters that yields the best tradeoff between the global bias and variance (somehow approximated, depending on our cross-validation strategy), these hyperparameters may lead to regions of $x$-es that have sub-optimal tradeoffs between their local biases and variances.

My questions:

  1. In which scenarios may it be an actual problem? I mean: when are we better off training two classifiers on two separate sets of observations, with hyperparameters tweaked separately?

  2. How could we identify such regions? I was expecting that a Boltzmann machine might be adapted for discovering highly volatile regions of $x$-es, but I didn't find anything of that sorts.

  • 1
    $\begingroup$ "...all of the terms depend on the argument x". This isn't true; the expectation is computed over the training set, so you're essentially averaging over all values of $x$ (and $y$), weighted according to their distribution. $\endgroup$ – vbox Jan 1 '17 at 23:58
  • $\begingroup$ Actually, the average is over data sets (see: www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf, page 34).. However, I might have overdone the dependence of the irreducible error (i.e. I didn't want to assume heteroscedasticity), so I dropped that dependence. $\endgroup$ – ponadto Jan 2 '17 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.