I am trying to create a neural network from scratch using numpy. I have created a network that can classify iris data base to a high degree of accuracy.

I am facing the problem that sometimes instead of predicting a class given input, the network just trains to return the probability distribution of the training sample irrespective of the input.

So, after training the network on 150 samples (50 sample from each class), then on testing on a sample I would get (0.333 0.333 0.333) for each of the three classes, where the expected output was supposed to be, say, (1,0,0).

I had solved that problem by tweaking the Hyperparameters, but I am facing a similar problem with wine data set, for which I cannot do the same.

Has anybody faced this problem before? How did you solve it?

  • $\begingroup$ If you are using softmax, make sure the layer feeding into it has linear activation function. $\endgroup$
    – Kari
    Jun 19 '19 at 19:55
  • $\begingroup$ I am using a sigmoid activation with quadratic or cross entropy loss(both converge to the same point in wine data). $\endgroup$
    – user276806
    Jun 20 '19 at 3:20

In case of any neural network if you are trying to solve the classification problem then it should generate probabilities.

  1. If you are trying to solve 2 class classification problem then one can opt for sigmoid or softmax. In the case of sigmoid, your neural network only contains 1 output neuron which generates value probability and if probability greater than equal to 50 then belongs to one class and less than 50 then belongs to another class. e.g.(0.333 0.333 0.333) = (0.80) that means belongs class 1 and (0.666 0.333 0.999) = (0.40) that means belongs class 0
  2. In the case of softmax, the neural network should have an equal number of output neurons to the number of categories. e.g.(0.333 0.333 0.333) = (0.80, 0.12, 0.08) that means belongs class 1 and e.g.(0.333 0.333 0.333) = (0.08, 0.20, 0.72) that means belongs class 3
  • $\begingroup$ I think I might have worded my problem wrong, I am getting the ratio of training data of the classes. Suppose there are 3 classes in a 100 training data samples: 20 of class 1, 44 of class 2, 36 of class 3. Then after training in my output there(having 3 nodes) I would get (0.20 .044 0.36) irrespective of the input. $\endgroup$
    – user276806
    Jun 20 '19 at 19:08
  • $\begingroup$ Ideally it should have given a value close to 1 for the most probable class, and a value close to 0 for the other classes. Also, I am doing a 3 class classification here, but I am not using softmax. Every node in the network is sigmoid, including output layer. $\endgroup$
    – user276806
    Jun 20 '19 at 19:16
  • $\begingroup$ How many neurons are present in the output layer? $\endgroup$ Jun 24 '19 at 6:08
  • $\begingroup$ Three, each with sigmoid activation. Represents independently how close the input is to one of the three classes. However I've solved the problem in this particular case, the input values were all over the place, needed normalization. $\endgroup$
    – user276806
    Jun 25 '19 at 8:13

In this particular problem, the data needed normalization. Some input values were in the order of magnitude of 1000, some 10, some 1.

After normalizing the data, everything worked perfectly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.