I am trying to use different ML classifiers for binary classification (SVM, logistic regression,DNN). The dataset used for training contains 333 columns and about 2000 rows. The classes being slightly unbalanced, I am using SMOTE to account for that. Even though the roc curves of all the classifiers are showing an AUC of more than 90%, once I provide a validation dataset, the model can barely predict correctly 30 % of the objects classified. The model is trained and tested while generating the model based on 3/4 of the training dataset. I then use 1/4 together with 2 other datasets that are not labeled but one of which is expected to have similar properties with the class 1 and the other one similar properties with the class 0. Other than expected, the classification is not accurate (at least for the dataset for which we know the labels). I did 10-fold cross-validation during the training and I guess the code is correct. I tried both to include or not pca prior to training. Without pca the train and test performance as shown by the loss function, seem to diverge. Also the pca and k-means clustering show that the dataset in two principal components is so densely pact with insignificant tendency of separation between the two classes. Also, as shown by the pearson correlation, from the total of 333 columns 320 are highly correlated (corrcoeff>0.8). As compared to the label column, the feature columns seem to be mildly correlated (0.3<corrcoeff<0.5). Here is a screenshot of how the data looks like:
What I am intending to do next is to train the model on subsets of columns of the initial training set and try to find the particular subset that will be able to enhance the prediction power on the given validation dataset. Nevertheless, I am not sure if this makes sense. The ultimate goal being to generate a model that will be as generalizable as possible, I am wondering if what I am intending to do is just tailoring the model to a particular validation dataset. Once another validation dataset is provided, there must not be a warranty that the model will predict accurately if it was previously tailored to a given validation dataset.
I will appreciate any comment and suggestion what should I do to make some sense of the model I would like to create. Many thanks.