I'm trying to develop a captcha solver using a simple fully-connected neural network in TensorFlow. All captchas have 5 digits/letters, each character can be a number 0-9 or a letter A-Z. They all look like this:
Each captcha has a label like this for example: "AB4D1". I take these labels, split them ['A', 'B', '4', 'D', '1'], convert each of the characters to a number between 0 and 35 (26 letters and 10 digits) and convert each of those to one-hot representations [0, 0, 0, 1, 0,..]. Then I concatenate the one-hot arrays for all the letters in a captcha so I end up with a single 5*36-length array for each image. (This array has 5 ones and the rest zeros)
Now I'm wondering what loss function to use. I tried mean-squared-error but that didn't work at all for some reason. Now I'm trying to use cross-entropy (with softmax first) but as far as I understand it, I would need to apply it to each letter separately and then add up the loss for each letter since softmax requires exclusive categories.
I tried splitting the model output and labels:
split_logits = tf.split(logits, num_or_size_splits=5, axis=1)
split_train_labels = tf.split(tf_train_labels, num_or_size_splits=5, axis=1)
This works afaik. Now I just add the cross-entropy losses for each letter together.
loss =
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[0], logits=split_logits[0])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[1], logits=split_logits[1])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[2], logits=split_logits[2])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[3], logits=split_logits[3])) +
tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(labels=split_train_labels[4], logits=split_logits[4]))
But this is where I have a problem. As long as I only have the line for the first (0th) digit/letter in there it works really well (But obviously only learns recognizing the first letter). As soon as I add the term for another letter, or or just have a single other letter instead of the first one, the network immediately plateaus after training starts and never improves to above-random performance.
I'm not sure how to fix this. I'm thinking the problem is either in my loss function or in my network structure.
Is the problem that all the letters except the first one are varying a lot in their position due to the different font sizes/widths? I understand that convolutional networks would be much better for this but I really want to try it with fully-connected networks first.
