I'm building a neural net for classifying characters in pictures. The input can be any character a-z, lowercase and uppercase. I only care about classifying the characters, and not the case, so the neural net has an output vector of length 26; one for each character. It makes sense, intuitively, to have a hidden layer of size 26*2 just upstream of the output layer. It also makes intuitive sense for this layer not to be fully connected to the output layer, but instead having two and two hidden nodes connect to each output node.

I have some questions:

a) Does this make sense? I'm getting about 75 % success rate on a pretty hard data set with just one hidden layer, but I'm not certain on how to improve from there.

b) If so, What activation function should I use from the hidden layer with 26*2 nodes to the output layer? Maybe I should use an OR function for this, since both the lowercase and the uppercase version of a character should output for a single character.

c) Would it be wiser to have 26*2 output nodes instead, and just combine lowercase and uppercase outputs after the neural net?

  • $\begingroup$ Which dataset do you use? $\endgroup$ Apr 19, 2016 at 21:18
  • $\begingroup$ It's an 8-bit grayscale version of the English-only chars74k data-set (ee.surrey.ac.uk/CVSSP/demos/chars74k). I don't know if it's hard compared to other data sets, or what success rate to expect, but I sure can't make out some of the characters using my own eyes. $\endgroup$
    – tsorn
    Apr 19, 2016 at 21:32
  • $\begingroup$ How are you measuring success rate? A quick review of published papers would put 75% accuracy at better than state-of-the-art, which seems unlikely (I might be misunderstanding some context though). The official measure is to use 15 exemplars from each class for training and the other 15 for testing. Are you using test/train split in that way to assess your model? $\endgroup$ Apr 19, 2016 at 21:48
  • $\begingroup$ I don't use the official way to measure. I train on 90 % of the data and test on the remaining 10 %. Are they only training with 15*26 examples? That sounds low to me. If they are only training with ~400 examples and I'm using ~67k, that might explain the difference. $\endgroup$
    – tsorn
    Apr 19, 2016 at 22:26

1 Answer 1


Your design makes some sense, but there is no need to limit connections even if you expect to represent probabilities of upper/lower case separately, because they will interact usefully. E.g if the character could most likely be one of o, O, Q, G then this might be useful information to choose the correct one.

If you went ahead, you would need to train this network without the final layer (so that it learns the representations you expect, not some other group of 52 features), then add the final layer later, with no need for special connection rules, just use existing ones. Initially you would training the new layer separately from the full output of the 52-class net i.e. probability values, not selected class. Then you would combine with the existing net and fine-tune the result by running a few more epochs with a low learning rate on the final model.

That all seems quite complex, and IMO unlikely to gain you much accuracy (although I am guessing, it could be great - so if you have time to explore ideas, you could still try). Personally I would not take your hidden layer idea further. The full 52-class version with simple logic to combine results is I think simpler. This is also not necessary, the neural net can learn to have two different-looking images be in the same class quite easily, provided you supply examples of them in training. However, it may give you useful insights into categorisation failures in training or testing.

It is not clear from the question, but if you are not already using convolutional neural network for lower layers, then you should do so. This will make the largest impact on your results by far.

  • $\begingroup$ Thanks. And so a ReLU activation function between the 52-node hidden layer and the 26-node output layer would be optimal? I don't use convolution. I don't know what CNN's are or how they work, but I'll have to look into that. For which layers, then, should I use convolution? Between all layers excluding between the two last (52->26)? $\endgroup$
    – tsorn
    Apr 19, 2016 at 16:52
  • $\begingroup$ @tsorn: A fully-connected layer with ReLU activation should be fine. For a CNN, you want lowest layers to be convolutional (typically Convolve2D with ReLU activation followed optionally by MaxPool and Dropout), and then at least one hidden layer between the last convolving layer and your output layer. The 52/26 idea of yours is kind of 2 stacked output layers, so you'd still want something fully-connected before them. $\endgroup$ Apr 19, 2016 at 16:58
  • $\begingroup$ One more thing; do you think bias nodes are necessary in such a network? $\endgroup$
    – tsorn
    Apr 19, 2016 at 20:23
  • $\begingroup$ Yes, I expect bias (and training of bias weights) to be critical. $\endgroup$ Apr 19, 2016 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.