I'm not gonna lie, I'm very new to neural networks but am also so interested in them and learning the way they work and what can be made from them. So in my endeavors for learning, I stumbled upon Siraj Ravel's youtube channel and furthermore, his Github where he posted this 1 neuron neural network that predicts the output given inputs.* I looked at it and his accompanying video and got the basic gist of how it works. After running it a few times making minor adjustments to say the inputs, I decided why not try to add numbers other than 1 and 0. I made a pattern with the output being the inputs added, however, I ran into an error where the output of its test was '1.'. I assumed this was because of the sigmoid function forcing it to be in between 1 and 0.

Summary: How can I change that program so I can give it an input of patterns with different numbers than 1 and 0 and it will output correctly (examples below)

*In his program inputs were like [1, 0, 1] = 1, [0, 0, 1] = 0 and [1, 0, 0] = 1 (pattern is first column of matrix is answer) and he asked the program to give output for [1, 1, 0] and it correctly outputted 1.

*My goal for it was so given "[2, 2, 2] = 6, [1, 2, 0] = 3, and [1, 1, 2] = 4 (more training included if necassary)" as input could it get 4 as output for "[2, 2, 0]?

Code for program

  • $\begingroup$ What error are you getting? $\endgroup$
    – gokul_uf
    Jul 28 '17 at 23:53
  • $\begingroup$ Without any edits, the code works, as to say it can correctly predict either the 1 or 0, I'm trying to find how I can edit it to predict other numbers, but the answer below is a very simple script for doing such, so many thanks to Andrey, @gokul_uf $\endgroup$
    – Leo C
    Jul 29 '17 at 13:10

Well, the main problem is that the existing code and the desired codes solve different problems.

The existing code solves a classification problem, when you predict one of certain classes. And you want to solve a regression problem, so that neural net will predict an arbitrary number.

Sigmoid function is necessary when you predict classes, you need to remove it for regression. Or in other words you need not Logistic Regression, but Linear Regression. Here is a great article with explanations on this: http://peterroelants.github.io/posts/neural_network_implementation_part01/

I have written an example of implementation: https://gist.github.com/Erlemar/6a5cfcca423ef3b5f6e890c6bef6d5ed

  • $\begingroup$ Hey, @AndreyLukyanenko , the response got me thinking (thanks for it by the way) how exactly is the first program a classification problem? $\endgroup$
    – Leo C
    Jul 28 '17 at 9:11
  • $\begingroup$ Well, strictly speaking it isn't classification indeed, it is more like finding a pattern. But using sigmoid activation makes it very similar to classification. $\endgroup$ Jul 28 '17 at 9:19
  • $\begingroup$ Do you have an idea on how I can edit the program (possibly getting rid of the sigmoid) to make it predict numbers other than 1 or 0? @AndreyLukyanenko $\endgroup$
    – Leo C
    Jul 28 '17 at 9:34
  • $\begingroup$ The key idea here is that saturating non-linearities such as the sigmoid function are good for classification, as networks will tend to learn the outcome towards two finite limits (often 0 and 1). For any other kind of regression, you can either apply another non-linearity that respects the intended output distribution, or simply have no non-linearity at the end. $\endgroup$ Jul 28 '17 at 10:26
  • $\begingroup$ @LeoC I have written a very simple example without functions, simply one loop: gist.github.com/Erlemar/6a5cfcca423ef3b5f6e890c6bef6d5ed The prediction accuracy heavily depends on the randomness of weight initialization. You can improve accuracy by adding more training examples, adding more layers, changing weights initialization etc. $\endgroup$ Jul 28 '17 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.