I'm trying to develop a captcha solver using a simple fully-connected neural network in TensorFlow. All captchas have 5 digits/letters, each character can be a number 0-9 or a letter A-Z. They all look like this:

captcha1 captcha2 captcha3 captcha4 captcha5

Each captcha has a label like this for example: "AB4D1". I take these labels, split them ['A', 'B', '4', 'D', '1'], convert each of the characters to a number between 0 and 35 (26 letters and 10 digits) and convert each of those to one-hot representations [0, 0, 0, 1, 0,..]. Then I concatenate the one-hot arrays for all the letters in a captcha so I end up with a single 5*36-length array for each image. (This array has 5 ones and the rest zeros)

Now I'm wondering what loss function to use. I tried mean-squared-error but that didn't work at all for some reason. Now I'm trying to use cross-entropy (with softmax first) but as far as I understand it, I would need to apply it to each letter separately and then add up the loss for each letter since softmax requires exclusive categories.

I tried splitting the model output and labels:

split_logits = tf.split(logits, num_or_size_splits=5, axis=1)
split_train_labels = tf.split(tf_train_labels, num_or_size_splits=5, axis=1)

This works afaik. Now I just add the cross-entropy losses for each letter together.

loss = 
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[0], logits=split_logits[0])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[1], logits=split_logits[1])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[2], logits=split_logits[2])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[3], logits=split_logits[3])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[4], logits=split_logits[4])) 

But this is where I have a problem. As long as I only have the line for the first (0th) digit/letter in there it works really well (But obviously only learns recognizing the first letter). As soon as I add the term for another letter, or or just have a single other letter instead of the first one, the network immediately plateaus after training starts and never improves to above-random performance.

I'm not sure how to fix this. I'm thinking the problem is either in my loss function or in my network structure.

Is the problem that all the letters except the first one are varying a lot in their position due to the different font sizes/widths? I understand that convolutional networks would be much better for this but I really want to try it with fully-connected networks first.

  • $\begingroup$ I ain't an expert in metrics and not sure whether it makes sense or not but can't we use something equivalent to like distance between the two syllables for each one as we know their bounds also $\endgroup$
    – Aditya
    May 29, 2018 at 19:55

1 Answer 1


Another way to frame the problem is single character prediction. If you can split the image into separate characters, predict each character separately.

If you frame the problem that way the most appropriate loss function is softmax. The network would learn to predict the relative likelihood of each 36 characters.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.