I was recently trying to train a convolutional neural network to classify people as Hispanic or white (for learning purposes). I couldn't find a good dataset of just those two races, so I had to manually scrape images from the web. I ended up with 48 total images, 24 of them Hispanic and 24 white. When I trained my network, I tried many different architectures and hyperparameters, but the accuracy stayed at exactly 50%. I learned this was because the network was outputting 1 every time, classifying every image as Hispanic because it found a local optima for the cost function. I believe this is because of my 50/50 data split, although I may be wrong. If if is because of the split, how does this not happen in the real world? For example, if I had 10 million images, half of them white and the other half Hispanic, how does the network avoid falling prey to the same trap?
First of all your dataset is very small to train a deep learning model you would need at least 10k images to train the model and produce good results. Secondly, if you still think that your cost function gets stuck in local minima then you might consider using Nesterov momentum.
In general, the cost function of a neural network is non-convex, so it's mathematically okay that you're converging at a local minima. (To convince you that it's non convex, take any permutation of the set of weights and it would also yield you to the same point in space). As a suggestion, try using a linear classifier, or a multilayer perceptron and compare your results.