0
$\begingroup$

I am trying to use different ML classifiers for binary classification (SVM, logistic regression,DNN). The dataset used for training contains 333 columns and about 2000 rows. The classes being slightly unbalanced, I am using SMOTE to account for that. Even though the roc curves of all the classifiers are showing an AUC of more than 90%, once I provide a validation dataset, the model can barely predict correctly 30 % of the objects classified. The model is trained and tested while generating the model based on 3/4 of the training dataset. I then use 1/4 together with 2 other datasets that are not labeled but one of which is expected to have similar properties with the class 1 and the other one similar properties with the class 0. Other than expected, the classification is not accurate (at least for the dataset for which we know the labels). I did 10-fold cross-validation during the training and I guess the code is correct. I tried both to include or not pca prior to training. Without pca the train and test performance as shown by the loss function, seem to diverge. Also the pca and k-means clustering show that the dataset in two principal components is so densely pact with insignificant tendency of separation between the two classes. Also, as shown by the pearson correlation, from the total of 333 columns 320 are highly correlated (corrcoeff>0.8). As compared to the label column, the feature columns seem to be mildly correlated (0.3<corrcoeff<0.5). Here is a screenshot of how the data looks like:enter image description here

What I am intending to do next is to train the model on subsets of columns of the initial training set and try to find the particular subset that will be able to enhance the prediction power on the given validation dataset. Nevertheless, I am not sure if this makes sense. The ultimate goal being to generate a model that will be as generalizable as possible, I am wondering if what I am intending to do is just tailoring the model to a particular validation dataset. Once another validation dataset is provided, there must not be a warranty that the model will predict accurately if it was previously tailored to a given validation dataset.

I will appreciate any comment and suggestion what should I do to make some sense of the model I would like to create. Many thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Just a few comments:

  • You probably don't need SMOTE if your data is only slightly imbalanced. It can be a source of bias.
  • 30% accuracy is extremely low: there's probably a mistake somewhere, your classifier does worse than a random baseline. Also you need to make sure that this validation dataset follows the same or similar distribution than the training set.
  • There is certainly some overfitting happening, so I agree that you should try to use feature selection to make things easier for the model
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks. In regard to standardizing the data, I first standardized separately the training and validation datasets, thus both datasets have for each feature column a mean of 0 and variance of 1. I then switched into using the same scalar for both the training and validation dataset. This time I get for the training dataset feature mean 0 for each column and variance 1, but for the validation dataset I get a mean value around -0.5 for the feature columns and variance around 1. What is the right way to do in my case, i.e. standardizing separately both datasets or rather using the same scalar ? $\endgroup$
    – user249018
    Commented May 11, 2022 at 7:27
  • $\begingroup$ @user249018 The parameters for the standardization should be calculated only on the training data, then these parameters should applied to both. $\endgroup$
    – Erwan
    Commented May 11, 2022 at 12:58
  • $\begingroup$ As for the pca, which should be the correct scenario in using pca on the training dataset aiming at fitting the model on the lets say 10 first principal components ? It is clear that the validation dataset will undergo the same treatment. In my case, when I use pca for the purpose of classification, the loss function on both the train and test datasets are steadily decreasing and follow closely each other. Without pca, they seem to diverge and one of them also increases after a certain number of epochs. At the other hand, without pca the predictions seem to be more accurate than with pca. $\endgroup$
    – user249018
    Commented May 11, 2022 at 15:13
  • $\begingroup$ @user249018 pca applies feature reduction, so it kind of "simplifies" the data a bit to emphasize the really important patterns. It's expected that this might reduce accuracy a bit, but normally it also reduces the risk of overfitting and makes the model more stable. $\endgroup$
    – Erwan
    Commented May 11, 2022 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.