Binary Classifier making only one prediction

Question

tl;dr I'm building a binary classifier that always eventually predicts all "0" or all "1" after some number of epochs and I'm looking for possible reasons why/how to proceed.

Below is all just more details on my approach and thoughts:

What I'm doing: I am building a binary classification basic feed-forward neural network for signal processing with tensorflow.

I pre-process my data by separating it into "windows" of length window_size points (say like 50), and that window/example can be considered either a "1" (positive) or a "0" (negative). The official, manually annotated labels are located at specific points (e.g. 403, 875, 1450, etc.) and are relatively evenly spaced. I decided that my example/window is positive if the window contains a label, negative otherwise. Multiple examples are generated from moving the window by some stride_length (say 1, 2, or 10 points, etc.), until my moving window reaches the end of the signal.

For the training data, since most of my examples are negative, I've tried to normalize the ratio of neg to pos examples so it's close to 1:1. I do this by deleting a calculated ratio of negative examples. For test data I skip this last step, but everything else is the same.

I run it through a 3 layer NN with RELU's on hidden layers and sigmoid on the sole output, with <0.5 predicting a negative and >0.5 predicting a positive. Cost is sum of (label - prediction)^2, using AdamOptimizer at default LR (0.001).

My issue: No matter what parameters I adjust, my NN seems to predict everything as either negative or positive after a certain number of epochs (sometimes even immediately at the first epoch). It's pretty random too. I'll run the same thing multiple times and I'll get something different every time, about half of the predicting all positive, and the other half all negative.

Things I have tried/thought about:

• I shuffled my training examples before training my model, with not much improvement.
• I cut down on my negative examples as I stated above, since I had this problem before I cut down (was predicting basically all negative immediately). But now that I have close to a 1:1 ratio of pos/neg examples, it's flip-flopping between predicting all negative and all positive, so it's extremely sensitive to this ratio.
• I originally had a softmax_cross_entropy_with_logits cost function with one-hot encoding, so 0 1 was positive and 1 0 was negative. (I thought maybe the cost function was rewarding extreme predictions, so I switched to sum of squares which is harsher)
• I'm thinking of trying batch normalization and dropout, but I don't expect those to fix my problem.
• I think the problem might be in how I am pre-processing the data. My thought process is this: if the ratio of pos/neg examples is so important that it is making my model predict all positive or all negative, then maybe there is no real trend in my data (which is wrong, since anyone with eyes can see the difference). My thought was that because I have a moving window, sometimes the label is near the left edge of my moving window and sometimes the right edge, which means the actual region of interest is getting input into different neurons each time, which confuses the neurons so it feels like there is no real trend.
• However, I took a look at the weights and biases and they seem to be stabilizing after a few epochs. Shouldn't they still be jumping around wildly if there really is no trend in the data? Maybe that is wrong.
• I was thinking about testing my hypothesis out by running the left-leaning label examples and right-leaning label examples through different neural networks and seeing if I get better results, but maybe it's not worth the effort it will take.

I've been trying this for a while and can't seem to think of a good explanation or approach. I would really appreciate any help I can get on this, and I'm happy to provide more information. Thank you!

Answer 1

I suspect a couple issues:

1. The cost function is unusual for classification. You would typically use something like cross entropy which you mentioned. Make sure not to threshold the outputs when you input to the cross entropy loss. Also, I wouldn't characterize L2 loss as being more "harsh". If your predicted probability for a particular data point is 0, but its label is 1, L2 loss would be 1. But for cross entropy, the loss would be very large.
2. It predicts all one class on the validation set or just any particular batch?
   • If it is just a particular batch, maybe how you batch the data is off and all of one class is in that batch. Look at metrics like accuracy and loss.
   • If it is the validation set, I would look into if the training loss is consistently going down. Maybe the learning rate is too high. You are using Adam which is based off of momentum so it does not necessarily need as high of a learning rate as standard SGD.
3. If the weights are stabilized, it means that the network is saturated or you have 0 loss and your optimization problem is solved (probably not the latter). If the network did saturate too fast, batch norm would definitely help.

Answer 2

Rather than sampling your negative values to achieve a 1:1 ratio, you should try weighting your classes. You can simply use the existing positive:negative ratio you have already calculated and pass this to the class_weights parameter in skflow (assuming you are using skflow, if not, there are almost always equivalent ways of doing this in any ML package/language).

Example:

class_weight = tf.constant([0.9, 0.1]))

skflow.models.logistic_regression(X, y, class_weight=class_weight)

The reason I suggest this is that your random all positive or all negative outputs may be down to the way you're down-sampling the negative examples. If you're doing this at random, you may be removing some key examples in the dataset that the classifier would have used to learn to strongly distinguish between the classes (for example: stock data, where two candles show a drop, but one shows much more of a drop). By weighting you can train on all of your data and remove this potential problem, whilst still 'learning' both classes equally.