5
$\begingroup$

This might be a weird question but I'm trying to have a deep understanding of how neural networks work theoretically.

I was doing some tests with my perceptron and I decided to test it on a single input single output dataset. What I was looking for was 100% accuracy since what I was testing on was a trivial separable dataset with binary output. However, the accuracy was below 50% (it wasn't balanced). I realized that this is due to the fact that there is a single weight that when trained is either going to be =>0 or <0. So after going through the activation function (e.g. hardlim), unless the input is scaled to be [-1,1] range, it's always going to give the same output.

Does my explanation make sense? Are there any theoretical aspects I'm missing?

Also, are neural networks useful at all for single input problems? They seems useless to me since we're basically classifying by putting a point or points (for multiclass problems) on a single line to separate outputs, which is a pretty simple problem that doesn't need the intricacy of neural networks.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Does your network have one or more hidden layers? What are their sizes? Does the output neuron in your network have bias as well as weight? A graph of input vs output for your training data might help (I am imagining it looks like samples from a step function, but not 100% sure) $\endgroup$
    – Neil Slater
    Apr 14, 2018 at 7:41
  • 1
    $\begingroup$ Give your neural networks a bit more power and then see what they can/cannot do, as Neil said, add more hidden layers, bias, Also Use a single input neuron and single output neuron without any hidden layers will definately make neural networks useless... $\endgroup$
    – Aditya
    Apr 14, 2018 at 11:35

1 Answer 1

Reset to default
1
$\begingroup$

You have saturation of the weights which results in constant predictions.

There are several ways to fix that:

  • Replace hardlim with a standard activation function like ReLU or sigmoid
  • Pick a different random initialization
  • Normalize inputs
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.