In the case of a neural net with a relatively small training data set, doing simple classification with categorical cross entropy (log loss), it is very easy for the results of the network to be "overconfident" because that improves log loss on the training data.

What I mean by overconfident (this is partly intuition and partly something that can be derived from Bayesian stats): given 1000 training examples in 2 equal classes, it is unreasonable for any classifier to give a probability of a new input as being in one class or the other which is substantially higher than (1-1/1000), even if the classifier is perfect on the training data. That is because if the training set was increased by just one more example, the 1001st example could break whatever rule seems to have been followed in the data set up to that point, and that would force all the predicted values/probability estimates to be updated accordingly.

However, in practice trained neural nets with log loss function will output values which are arbitrarily close to 0 or 1 given a long enough training run, because that continues improving their performance on the training data way past the point that it makes sense to do so based on the size of the training data.

The question: how can the usual log loss function be modified so that making predictions extremely close to 0 or 1 does not hugely improve log loss compared to predictions which are only somewhat close?


You could add a penalty by adding a small "distance" to every loss calculation. They you would have a "floor" below which no comparison could be below. The tricky decision is how large to make the penalty. There is a large literature on how to choose the "right" penalty that keep your overconfidence low but maximizes the predictive ability.

If you have a copy of "The Elements of Statistical Learning", then look at page 61 and the 10 + pages following that. There are also other items in the index covering penalized methods. The Wikipedia article on the Lasso methods is also a possible starting point.

When I type this into an R console on a new Linux box I'm outfitting:


I get:

Help pages:

brglm::profileObjectives        Objectives to be profiled
mda::gen.ridge      Penalized Regression
mgcv::bam       Generalized additive models for very large datasets
mgcv::bam.update        Update a strictly additive bam model for new data.
mgcv::gam       Generalized additive models with integrated smoothness estimation
mgcv::jagam     Just Another Gibbs Additive Modeller: JAGS support for mgcv.
mgcv::null.space.dimension      The basis of the space of un-penalized functions for a TPRS
mgcv::pcls      Penalized Constrained Least Squares Fitting
mgcv::smooth.construct.bs.smooth.spec       Penalized B-splines in GAMs
mgcv::smooth.construct.cr.smooth.spec       Penalized Cubic regression splines in GAMs
mgcv::smooth.construct.tp.smooth.spec       Penalized thin plate regression splines in GAMs
quantreg::rq.fit.lasso      Lasso Penalized Quantile Regression
quantreg::rq.fit.scad       SCADPenalized Quantile Regression
rms::pentrace       Trace AIC and BIC vs. Penalty
VGAM::sm.ps     Defining Penalized Spline Smooths in VGAM Formulas
VGAM::summarypvgam      Summarizing Penalized Vector Generalized Additive Model Fits

I would have gotten more if I were using my older machine ( a macpro that died recently) but I havent' installed an lasso-type packages yet. There are quite a few:

> rownames(available.packages()[  grep("lasso", available.packages()[,1]) , ])
 [1] "biglasso"      "cglasso"       "CVglasso"      "DIFlasso"      "dpglasso"      "elasso"       
 [7] "extlasso"      "gglasso"       "glamlasso"     "glasso"        "glassoFast"    "GPCMlasso"    
[13] "grplasso"      "hglasso"       "iilasso"       "ipflasso"      "lasso2"        "lassopv"      
[19] "lassoscore"    "lassoshooting" "lmmlasso"      "nnlasso"       "palasso"       "ppmlasso"     
[25] "prioritylasso" "PUlasso"       "sealasso"      "sglasso"       "Tlasso"       
  • $\begingroup$ Thanks! Could you give a few literature examples on how to choose the "right" penalty? I think it may need to be related to the size - and noisiness - of the training set... $\endgroup$ – Alex I Dec 7 '18 at 19:08
  • 1
    $\begingroup$ There is a huge literature on this topic. See above. The penalty to apply to a particular feature depends on a host of factors. These come immediately to mind: the magnitude of the measurements (typically measurements are normalized to get them all on comparable scales), the degree of cross correlation in the signals, the dispersion and skewness of deviations from predictions, and the degree of non-linearity in the functional relationships between predictors and outcomes. $\endgroup$ – 42- Dec 7 '18 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.