0
$\begingroup$

I wrote a simple neural network that functions similarly to many of the C# examples I've seen online. It uses weights and biases and can be trained using backpropagation. It works well for predicting XOR. But I want to know if it can look at a series of inputs and always return the last input as the output. For example, I want it to predict like this:

Input: [   1,    2, 0] -> Output: [0]
Input: [1000, 2000, 0] -> Output: [0]
Input: [   1,    2, 1] -> Output: [1]
Input: [1000, 2000, 1] -> Output: [1]

(My actual data is much more randomized. Notice that the last input becomes the output.)

I just wanted to see if I could teach the neural network to ignore all inputs except the last one. Ideally, all weights except those attached to the last input would be 0. At least, that was my hope. (I think this is obvious, but I don't want to set the weights manually. I want the network to "learn" the weight.)

Can someone recommend how something like this could be accomplished? I'm currently using 3 input neurons, and one ReLU hidden layer with 3 neurons. I'm using sigmoid in my output layer with 1 neuron. I'm also generating weights in the range of -.5 to .5. I've experimented with different combinations of functions and randomized weights without success. Thank you.

$\endgroup$
1
  • $\begingroup$ How about the linear activation function (if you can make it fit into the whole structure)? $\endgroup$ Jan 19 at 12:59

1 Answer 1

0
$\begingroup$

You should use L1 Regularization. What it does is it adds another term to the error function in the training. This change allows for the weights of the less relevant features to be lowered very close to zero.

$\endgroup$
1
  • $\begingroup$ I wish I could say whether or not this strategy works, but I haven't found a clear answer about how to implement L1 regularization in C#. I'll have to keep looking. If I can figure out how to do it and if it does allow the intended weights to go to zero, I intend to accept your answer. Thank you. $\endgroup$ Jan 20 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.