Could you tell me, are there any techniques for building neural networks with non-negative weights?
$\begingroup$ if your software supports expressions for weights (as opposed to only allow parameters as weights not depending on anything else) you can use weight expressions which are squares or exponentials of parameters. Lasagne for example supports this (look for 'Positive weights' in this article). $\endgroup$– Andre HolznerMar 10, 2017 at 20:00
One possible method to approach neural networks with non-negative weights is using Feedforward neural network. We can construct the neural network using optimization techniques with the constraint of non-negative weights, as opposed to the normal back propagation method.
A modified Matlab example taken for here is as follows:
load iris_dataset % Number of neurons n = 4; % Number of attributes and number of classifications [n_attr, ~] = size(irisInputs); [n_class, ~] = size(irisTargets); % Initialize neural network net = feedforwardnet(n); % Configure the neural network for this dataset net = configure(net, irisInputs, irisTargets); %view(net); fun = @(w) mse_test(w, net, irisInputs, irisTargets); % Add 'Display' option to display result of iterations ps_opts = psoptimset ( 'CompletePoll', 'off', 'Display', 'iter', 'MaxIter', 100); %, 'TimeLimit', 120 ); % There is n_attr attributes in dataset, and there are n neurons so there % are total of n_attr*n input weights (uniform weight) initial_il_weights = ones(1, n_attr*n)/(n_attr*n); % There are n bias values, one for each neuron (random) initial_il_bias = rand(1, n); % There is n_class output, so there are total of n_class*n output weights % (uniform weight) initial_ol_weights = ones(1, n_class*n)/(n_class*n); % There are n_class bias values, one for each output neuron (random) initial_ol_bias = rand(1, n_class); % starting values starting_values = [initial_il_weights, initial_il_bias, ... initial_ol_weights, initial_ol_bias]; % alter the patternsearch function with the appropriate constraints, in your case we would change it so that the lower bounds of the weights are zero [x, fval, flag, output] = patternsearch(fun, starting_values, , ,,, zeros(size(starting_values)), 1e10, ps_opts);
mse_test.m function is as follows:
function mse_calc = mse_test(x, net, inputs, targets) net = setwb(net, x'); y = net(inputs); [row col] = size(y); mse_calc = sum(sum((y - targets).^2))/(row * col); end
You can easily take an existing Multilayer Perceptron implementation and modify the backpropagation algorithm to prevent weights from becoming negative. Of course, you would also need to make sure you initialize weights with only non-negative values.
But you need to be aware that you are limiting the decision surfaces that the neural network can learn. So you should consider whether a neural network with only non-negative weights can provide an acceptable solution for your given data set.
$\begingroup$ Thanks a lot. But are there any examples of this modification? $\endgroup$– MaassaMay 6, 2015 at 21:16
$\begingroup$ I'm not aware of examples of this modification for the reasons I mentioned above - it limits the solution space. Plus, I'm not aware of any benefit to having only non-negative weights. But as I stated, you could always modify the weight update rule of an existing implementation to check if the new value is less than zero and clip it to zero if it is. $\endgroup$– bogatronMay 7, 2015 at 3:57