Take the following case of a hyperparameter and prediction error:
Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- too high and you're underfit and too low and you're overfit.
I keep getting nonconvex plots like the one above when doing cross-validation.
I guess this just points to a lot of noise during training -- I've got a lot of variables for a modest sample size, and I need to regularize heavily to get a good model. But still I'm a little bit unsure whether this sort of thing might point to a bug in some aspect of my implementation.
Has anyone come across this sort of thing before? And did you just shrug off the nonconvexity and go with the model that minimized the prediction error?
If so, that begs the question: why not just compute a prediction error at each update during training, saving any set of weights that minimizes prediction error -- even if the model is nowhere near converged. Basically letting the noise work in your favor. This seems appealing, because sometimes I get really low prediction errors early on, only to have them evaporate as the loss function declines.
This seems horribly unprincipled, but I ask myself "why should I care if it is"? And "is it unprincipled anyway?"