I am training a neural network for classification using Matlab, and I don't understand if I can use the trainbr training function (Bayesian Regularization Backpropagation). It uses the MSE performance measure, but I want to use the crossentropy. If I set crossentropy as the performance function, the algorithm sets it back to MSE.

On the other way, I can't use a validation set with this training and I don't find how to change it.

The code is:

x = A';
t = y';

% Choose a Training Function
% For a list of all training functions type: help nntrain
% 'trainlm' is usually fastest.
% 'trainbr' takes longer but may be better for challenging problems.
% 'trainscg' uses less memory. Suitable in low memory situations.
trainFcn = 'trainbr';  % Scaled conjugate gradient backpropagation.

% Create a Pattern Recognition Network
net = patternnet(hiddenLayerSize,trainFcn);

% Choose Input and Output Pre/Post-Processing Functions
% For a list of all processing functions type: help nnprocess
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};

% Setup Division of Data for Training, Validation, Testing
% For a list of all data division functions type: help nndivide
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;

% Choose a Performance Function
% For a list of all performance functions type: help nnperformance
net.performFcn = 'crossentropy';  % Cross-Entropy
net.trainParam.epochs = 5000;

% Choose Plot Functions
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotconfusion', 'plotroc'};

% Train the Network
[net,tr] = train(net,x,t);

% Test the Network
y = net(x);
e = gsubtract(t,y);
net.performParam.regularization = 0; 
performance = perform(net,t,y); 

tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);

% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y); 
valPerformance = perform(net,valTargets,y);
testPerformance = perform(net,testTargets,y);



The trainbr mode uses the Bayesian regularization backpropagation. This method was presented in 1, which presents a regression problem with the loss function $$ E_D = \sum_{i=1}^n (t_i - a_i)^2 $$ where $t_i$ is the target and $a_i$ is the network's response. The paper proposes to add a regularization term, leading to a loss function $F$ of the form $$ F = \beta E_D + \alpha E_W $$ where $E_W$ is the square of the sums of all network weights, i.e. $E_W = \sum_{i,j} \| w_{ij} \|^2$. The two parameters $\alpha$ and $\beta$ control the weighing of the two parts $E_D$ and $E_W$: For $\alpha \ll \beta$, the network will minimize the loss, without really trying to keep weights low. For $\alpha \gg \beta$, the network will minimize the weights, allowing for some more error. In reality, this means a large $\alpha$ will stop the network from overfitting, which leads to a better generalization at the cost of a larger training error.

The key to find a train a model which generalizes well, but still has a low error rate, is the right setting of $\alpha$ and $\beta$. This is achieved by treating them as random variables and finding an optimal setting, using the Bayesian methods presented in 2. (I won't talk about the details on that here, you can find that in the two linked papers.) Finally, the paper presents an algorithm, which calculates the optimal $\alpha$ and $\beta$ in each training iteration. This makes this algorithm generalize really well, especially in the presence of noisy input signals.

However, as described, the loss function is a weighted sum between the MSE ($E_D$) and the regularization term ($E_W$). So, in short: you can only use it with the MSE, and not with cross-entropy. You'll need a different training algorithm, you can find a list in the MATLAB documentation, here.


1 F. D. Foresee and M. T. Hagan: "Gauss-Newton Approximation to Bayesian Learning", in Proceedings of the 1997 International Joint Conference on Neural Networks, June 1997. DOI: 10.1109/ICNN.1997.614194. [Link to PDF].

2 D. J. C. McKay: "Bayesian Interpolation", Neural Computation, May 1992, Vol. 4, No. 3, pp. 415-447. DOI: 10.1162/neco.1992.4.3.415. [Link to PDF].

  • $\begingroup$ Thanks, but it's still correct to use this method to classification problems? $\endgroup$ – juan9793 Nov 21 '16 at 8:15
  • $\begingroup$ Yes sure, but only with MSE or SSE - not CE $\endgroup$ – hbaderts Nov 21 '16 at 16:49

Setting a big number of validation checks (Higher than the maximum number of iterations), Matlab computes the validation error, but it doesn't affects the convergence. I still don't know how to compute the crossentropy instead of the MSE.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.