tl;dr I'm building a binary classifier that always eventually predicts all "0" or all "1" after some number of epochs and I'm looking for possible reasons why/how to proceed.
Below is all just more details on my approach and thoughts:
What I'm doing: I am building a binary classification basic feed-forward neural network for signal processing with tensorflow.
I pre-process my data by separating it into "windows" of length window_size points (say like 50), and that window/example can be considered either a "1" (positive) or a "0" (negative). The official, manually annotated labels are located at specific points (e.g. 403, 875, 1450, etc.) and are relatively evenly spaced. I decided that my example/window is positive if the window contains a label, negative otherwise. Multiple examples are generated from moving the window by some stride_length (say 1, 2, or 10 points, etc.), until my moving window reaches the end of the signal.
For the training data, since most of my examples are negative, I've tried to normalize the ratio of neg to pos examples so it's close to 1:1. I do this by deleting a calculated ratio of negative examples. For test data I skip this last step, but everything else is the same.
I run it through a 3 layer NN with RELU's on hidden layers and sigmoid on the sole output, with <0.5 predicting a negative and >0.5 predicting a positive. Cost is sum of (label - prediction)^2, using AdamOptimizer at default LR (0.001).
My issue: No matter what parameters I adjust, my NN seems to predict everything as either negative or positive after a certain number of epochs (sometimes even immediately at the first epoch). It's pretty random too. I'll run the same thing multiple times and I'll get something different every time, about half of the predicting all positive, and the other half all negative.
Things I have tried/thought about:
- I shuffled my training examples before training my model, with not much improvement.
- I cut down on my negative examples as I stated above, since I had this problem before I cut down (was predicting basically all negative immediately). But now that I have close to a 1:1 ratio of pos/neg examples, it's flip-flopping between predicting all negative and all positive, so it's extremely sensitive to this ratio.
- I originally had a softmax_cross_entropy_with_logits cost function with one-hot encoding, so 0 1 was positive and 1 0 was negative. (I thought maybe the cost function was rewarding extreme predictions, so I switched to sum of squares which is harsher)
- I'm thinking of trying batch normalization and dropout, but I don't expect those to fix my problem.
- I think the problem might be in how I am pre-processing the data. My thought process is this: if the ratio of pos/neg examples is so important that it is making my model predict all positive or all negative, then maybe there is no real trend in my data (which is wrong, since anyone with eyes can see the difference). My thought was that because I have a moving window, sometimes the label is near the left edge of my moving window and sometimes the right edge, which means the actual region of interest is getting input into different neurons each time, which confuses the neurons so it feels like there is no real trend.
- However, I took a look at the weights and biases and they seem to be stabilizing after a few epochs. Shouldn't they still be jumping around wildly if there really is no trend in the data? Maybe that is wrong.
- I was thinking about testing my hypothesis out by running the left-leaning label examples and right-leaning label examples through different neural networks and seeing if I get better results, but maybe it's not worth the effort it will take.
I've been trying this for a while and can't seem to think of a good explanation or approach. I would really appreciate any help I can get on this, and I'm happy to provide more information. Thank you!