I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are microarray expression values from two different microarray platforms performed by two different research groups and so their value ranges are very different. I am new to SVM and R, and so at first, I did a Z-score standardization on my Training set and then I did a Z-score standardization on my Test set separately. i.e.
(x_Train - mean_Train) / (stdev_Train)
(x_Test - mean_Test) / (stdev_Test)
As I explored more forum posts on this topic, I see that it is suggested to use the parameters from Training set on my Test set
(x_Test - mean_Train) / (stdev_Train)
So that my SVM model is more generalizable and can be applied to new samples.
Here is the problem,
When I standardized the sets separately, my best model (after 10-fold cross validation using the tune function) had around 75% accuracy classifying the test set. However, after standardizing as suggested with my Training parameters the accuracy of my model to classify the Test set dropped drastically to 25% using the same features.
I tried using different combinations of features to see if it makes a difference and I saw that no matter what I had changed, all samples from the Test set were classified into one class (all in normal, and zero in tumor), while my previous model with separate standardization classified samples into both classes.
Here is an example of one of the features
Training set (raw) Feature 1 Sample 1 25.82977 Sample 2 42.62437 Sample 3 28.91158 Sample 4 53.5708 Sample 5 30.92296 Sample 6 99.16994 Sample 7 40.75973 Test set (raw) Feature 1 Sample 1 2.885865028 Sample 2 2.572860413 Sample 3 2.809136715 Sample 4 2.259630716 Sample 5 2.797155715 Sample 6 2.439700763 Sample 7 2.197087754 Training set (standardized) Feature 1 Sample 1 -0.795137358 Sample 2 -0.132081654 Sample 3 -0.673466601 Sample 4 0.300086594 Sample 5 -0.594056732 Sample 6 2.100353934 Sample 7 -0.205698184 Test set (standardized) Feature 1 Sample 1 -1.700969415 Sample 2 -1.713326928 Sample 3 -1.703998671 Sample 4 -1.725693328 Sample 5 -1.704471684 Sample 6 -1.71858411 Sample 7 -1.728162542
I know SVM uses a threshold to classify samples into either normal or tumor. From what I showed above, their ranges are completely different, is that what is affecting its decision?
What is the problem here and how can I tackle it, thank you in advance. I am so lost and need every bit of advice!
*Edit: As you can see the values of Training set have a completely different range from the values of the Test set, can I still use the mean and standard deviation from my Training set to scale the corresponding features in the Test set?