I am training an RNN to classify some high-frequency financial data. A very good performance on this data would be an accuracy of >52% or so.

I have around 650K training examples and 150K dev set examples (per cross-fold). Each training example has 50 time samples and a single binary output for the entire sequence. I am using BCE as the loss and the AdamW optimizer.

There are 17 features per time sample. The input tensor therefore is (512 x 50 x 17) -- that is batch size x samples x features

This is the network architecture:

  (lstm): LSTM(17, 32, batch_first=True, dropout=0.8)
  (fcl1): Linear(in_features=32, out_features=32, bias=True)
  (relu): ReLU()
  (drop): Dropout(p=0.8, inplace=False)
  (fcl2): Linear(in_features=32, out_features=1, bias=True)
  (sigmoid): Sigmoid()
Model params:

Initially I had some difficulty with overfitting. I was using a learning rate of 0.001, weight decay of 0.1 and dropout of 70%. I ran about 20 epochs. The overfitting is shown here.

enter image description here

I decided to reduce the learning rate to 0.0001, increase the weight decay to 0.5 and increase the dropout to 80%. I needed to run more epochs but ultimately this worked well to reduce overfitting. It reduced the dev set loss, but didn't really improve the final dev set accuracy.

enter image description here

So it would appear to me that I have a bias problem rather than a variance problem and I should find ways to improve the network's learning of the training data. My concern is that a more complicated network would lead to greater overfitting and I think I am out of options for further regularization without really hurting the network's ability to learn.

Looking to get some guidance as to what I might try next.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.