I ask you for help on a classification problem (classes are represented by the numbers 0,1 and 2). All features are extracted from time series data (fundamental is sinus shape).
I have a source dataset with features, which do not follow a gaussian distribution:
I can train classifiers with high accuracies with less effort. BUT: My validation data (target domain) has features with a distribution which is also not gaussian but which differ from the distribution the classifier was trained on.
E.g., we compare the distribution of the same feature for class 0 in source and target domain:
For classification models I used several different types of models which do not assume the features to be gaussian distributed (Trees, KnearestN,...) and I also tried a neural network (MLP with 3 hidden dense layers).
What my problem is
The problem I am facing is the same: The model trained on the source data (to ~98% valid. acc) fails on the target data with null-accuracy like (~30%). To validate that the problem is not only caused by the feature set (some features shown above) I tried a neural network approach. I am having the same problem if I train a neural network on the raw time series data, train/test is okay but performance on unseen instances is about null-accuracy only.What I currently tried to solve it
- Transforming the features to a gaussian like distribution using e.g. quantile and box-cox transformation -> The problem with that is, that I cant fit a well performing classifier on that transformed data since the models do not converge
- Different optimal transport strategies to transform the source distributions to the target shapes and train a supervised model with the transformed source data and then evaluate on the target data. Acc. also in range of null-accuracy
- Domain Adaptation models using non-parametric distance metrics for loss functions, like MMD (maximum mean discrepancy) and CORAL (correlation alignment). The problem is, I can only get the target accuracy to ~50% until the source accuracy drops
