1
$\begingroup$

I am using the neural network toolbox of Matlab to train a network. Now my code is as follows:

x = xdata.';
t = target1';
% Create a Pattern Recognition Network
hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize);
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
net.layers{2}.transferFcn = 'softmax';
net.divideFcn = 'dividerand';  % Divide data randomly
net.divideMode = 'sample';  % Divide up every sample
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;
net.trainFcn = 'trainscg';  % Scaled conjugate gradient
net.performFcn = 'mse';
net.performParam.regularization = 0.5;
%net.performParam.normalization = 0.01;
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
  'plotregression', 'plotfit', 'plotconfusion'};

% Train the Network
[net,tr] = train(net,x,t);
% Test the Network
y = net(x);
e = gsubtract(t,y);
tind = vec2ind(t);
yind = vec2ind(y);
percentErrors = sum(tind ~= yind)/numel(tind);
performance = perform(net,t,y)
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t  .* tr.valMask{1};


Now I am supposed to get different accuracies with different run of data since the sampling (division of dataset into train test and validation set) is random. But I am getting a constant accuracy (89.7%). The variable 'xdata' contains only those features selected by a feature selection algorithms. Is there any reason why my accuracy value is constant?

I have trained an SVM too with the same dataset. There too I am getting the a stale accuracy even with multiple run (94%)

The output y contains 2 values. What do those values signify?

$\endgroup$
  • $\begingroup$ Haven't you fixed the random number generator seed somewhere with the 'rng' function? $\endgroup$ – rcpinto Aug 8 '16 at 14:35
  • $\begingroup$ @rcpinto But i am getting a stable result? Still do i need ti fix rng? $\endgroup$ – girl101 Aug 9 '16 at 3:09
0
$\begingroup$

I see that you have set a Random Number Generator(rng) seed in the following line in your code: rng(1).

So, this splits the data in the same way no matter whichever run. So, that is the reason why you are getting the same error values.

Try removing the line. Then, the data shall be slit randomly (as there is no seed now). You shall get different error then, depending on how the split is done (which is, randomly).

A model which generalizes well should be robust to the choice of seed

| improve this answer | |
$\endgroup$
  • $\begingroup$ yes I know. But if I get different accuracy which accuracy should I take? Or should I average the accuracies ? $\endgroup$ – girl101 Aug 9 '16 at 5:56
  • $\begingroup$ if I am not using the rng, then I get a constant accuracy. Does this mean that the model can be used for further classification and is stable? $\endgroup$ – girl101 Aug 9 '16 at 5:59
  • $\begingroup$ @Rishika As I have already mentioned under one of your qns before, plot validation curves and take the dataset, which has the best balance of bias and variance (Also, refer to this answer). Yes, this means that your model is stable and has fitted the data nicely. $\endgroup$ – Dawny33 Aug 9 '16 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.