Using the trainbr function for classification in Matlab

Question

I am training a neural network for classification using Matlab, and I don't understand if I can use the trainbr training function (Bayesian Regularization Backpropagation). It uses the MSE performance measure, but I want to use the crossentropy. If I set crossentropy as the performance function, the algorithm sets it back to MSE.

On the other way, I can't use a validation set with this training and I don't find how to change it.

The code is:

x = A';
t = y';

% Choose a Training Function
% For a list of all training functions type: help nntrain
% 'trainlm' is usually fastest.
% 'trainbr' takes longer but may be better for challenging problems.
% 'trainscg' uses less memory. Suitable in low memory situations.
trainFcn = 'trainbr';  % Scaled conjugate gradient backpropagation.

% Create a Pattern Recognition Network
net = patternnet(hiddenLayerSize,trainFcn);

% Choose Input and Output Pre/Post-Processing Functions
% For a list of all processing functions type: help nnprocess
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};

% Setup Division of Data for Training, Validation, Testing
% For a list of all data division functions type: help nndivide
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;

% Choose a Performance Function
% For a list of all performance functions type: help nnperformance
net.performFcn = 'crossentropy';  % Cross-Entropy
net.trainParam.epochs = 5000;

% Choose Plot Functions
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
    'plotconfusion', 'plotroc'};

% Train the Network
[net,tr] = train(net,x,t);

% Test the Network
y = net(x);
e = gsubtract(t,y);
net.performParam.regularization = 0; 
performance = perform(net,t,y); 

tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);

% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y); 
valPerformance = perform(net,valTargets,y);
testPerformance = perform(net,testTargets,y);

Thanks

Answer 1

The trainbr mode uses the Bayesian regularization backpropagation. This method was presented in ¹, which presents a regression problem with the loss function $$ E_D = \sum_{i=1}^n (t_i - a_i)^2 $$ where $t_i$ is the target and $a_i$ is the network's response. The paper proposes to add a regularization term, leading to a loss function $F$ of the form $$ F = \beta E_D + \alpha E_W $$ where $E_W$ is the square of the sums of all network weights, i.e. $E_W = \sum_{i,j} \| w_{ij} \|^2$. The two parameters $\alpha$ and $\beta$ control the weighing of the two parts $E_D$ and $E_W$: For $\alpha \ll \beta$, the network will minimize the loss, without really trying to keep weights low. For $\alpha \gg \beta$, the network will minimize the weights, allowing for some more error. In reality, this means a large $\alpha$ will stop the network from overfitting, which leads to a better generalization at the cost of a larger training error.

The key to find a train a model which generalizes well, but still has a low error rate, is the right setting of $\alpha$ and $\beta$. This is achieved by treating them as random variables and finding an optimal setting, using the Bayesian methods presented in ². (I won't talk about the details on that here, you can find that in the two linked papers.) Finally, the paper presents an algorithm, which calculates the optimal $\alpha$ and $\beta$ in each training iteration. This makes this algorithm generalize really well, especially in the presence of noisy input signals.

However, as described, the loss function is a weighted sum between the MSE ($E_D$) and the regularization term ($E_W$). So, in short: you can only use it with the MSE, and not with cross-entropy. You'll need a different training algorithm, you can find a list in the MATLAB documentation, here.

References:

¹ F. D. Foresee and M. T. Hagan: "Gauss-Newton Approximation to Bayesian Learning", in Proceedings of the 1997 International Joint Conference on Neural Networks, June 1997. DOI: 10.1109/ICNN.1997.614194. [Link to PDF].

² D. J. C. McKay: "Bayesian Interpolation", Neural Computation, May 1992, Vol. 4, No. 3, pp. 415-447. DOI: 10.1162/neco.1992.4.3.415. [Link to PDF].

Answer 2

Setting a big number of validation checks (Higher than the maximum number of iterations), Matlab computes the validation error, but it doesn't affects the convergence. I still don't know how to compute the crossentropy instead of the MSE.