I have a industrial dataset containing labeled machine data for fault classification(3 classes: 1 ok, 2 for faults). The problem is that i have less (~16) different machines, thus iam currently having instance shift problems: The accuracies on the training set is perfect but validation on holdout instances fails. As background information, the machine data is time series, where i extracted statistical (domain specific) features from (14 in total). This features are my dataset for classification. I tried different model types, like SVM and MLP.

My question to you is: I tried to reduce the generalisation error with methods like Dropout and L1L2 regularisation - But this does not work well as it causes the training accuracy to stay low. It would be very helpful if you could get me some hints and tips would i could try. Thanks in advice.

  • $\begingroup$ can you collect more data, or if not is it possible to generate synthetic data? so overfitting might happen $\endgroup$ – Nikos M. Jan 10 at 9:28
  • $\begingroup$ The number of samples is quit large but the samples coming from a low number of different instances and i cannot get more currently. Synthetic data generation with gans perhaps? $\endgroup$ – deniz Jan 10 at 9:37
  • $\begingroup$ you can also train the model longer, if you dont do that already. There are many algorithms to generate synthetic data, you can try that which fits best your problem domain $\endgroup$ – Nikos M. Jan 10 at 18:47
  • $\begingroup$ This is a good question, however since you say regularization "causes the training accuracy to stay low" I wonder if you are testing properly, maybe your methods just overfit and they don't learn anything interesting at all, even for a machine in the training set as tested on data in an unseen time interval? $\endgroup$ – Valentas Jan 14 at 11:54
  • $\begingroup$ @Valentas, do you know how i could check that or how i can make shure that the Problems are not caused by this? $\endgroup$ – deniz Feb 25 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.