1
$\begingroup$

Problem statement: E2E classifier

Input: [7x3600] time series of physiological parameters recorded from a medical device.

Output: I am trying to learn a binary classifier to determine if the device is in use (with biological material) or experiencing a maintenance test.

Data: ~500 labeled examples. ~30:70 imbalance

Loss metric: 3x positive class weighted Binary Cross Entropy. Baseline: Randomly assigning positive case 3/5 of time => ~1.1

Problems

Non ANN approaches: Random forest & ROCKET did not give compelling results.

Tried using standard MLP, 1d-CNN and a RNN(S4) approach. All methods overfit after 1-2 epochs and validation loss is never better than optimally randomly assigning class. On one single run (1d-CNN 6 layers, with some pooling & layernorm no residual connections) validation loss decreased to 0.6 but without adjusting hyperparams or model structure this never happened again.

Tried leveraging pretrained image classifiers by making images of the graphs with fixed axes. Also no better results.

In each case: Upping L2 regularisation does not help. Dropout (<0.2) stops any convergence. Dropping learning rate does not help.

I have not tried transformers, so may try this or a VAE based classifier.

Questions

  • Am I being greedy seeking a loss of <0.6.
  • I want to maximise the test-AUROC ideally >0.9
  • Do I need more data?
$\endgroup$

1 Answer 1

2
$\begingroup$

It's not necessarily greedy to seek a loss of less than 0.6, as that could still represent a significant improvement over the baseline of 1.1. However, it's possible that the dataset simply doesn't contain enough information to achieve a low loss.

More data could help, especially if the imbalance between the two classes is high. A larger dataset would give the model more examples of each class and may provide more information for the model to learn from. You could also try data augmentation techniques to artificially increase the size of your dataset.

It might also be worth exploring different architectures or fine-tuning existing models that have been pre-trained on similar tasks. You mentioned that you tried a few different models, but perhaps there are other models that you haven't tried yet that could work better for this problem. You could also consider ensembling multiple models to see if combining their predictions can improve the overall performance of the classifier.

If you're still having trouble getting good results, it might also be worth trying to engineer more features or transformations of the data that could better capture the underlying patterns in the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.