1
$\begingroup$

I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are microarray expression values from two different microarray platforms performed by two different research groups and so their value ranges are very different. I am new to SVM and R, and so at first, I did a Z-score standardization on my Training set and then I did a Z-score standardization on my Test set separately. i.e.

(x_Train - mean_Train) / (stdev_Train)

(x_Test - mean_Test) / (stdev_Test)

As I explored more forum posts on this topic, I see that it is suggested to use the parameters from Training set on my Test set

(x_Test - mean_Train) / (stdev_Train)

So that my SVM model is more generalizable and can be applied to new samples.

Here is the problem,

When I standardized the sets separately, my best model (after 10-fold cross validation using the tune function) had around 75% accuracy classifying the test set. However, after standardizing as suggested with my Training parameters the accuracy of my model to classify the Test set dropped drastically to 25% using the same features.

I tried using different combinations of features to see if it makes a difference and I saw that no matter what I had changed, all samples from the Test set were classified into one class (all in normal, and zero in tumor), while my previous model with separate standardization classified samples into both classes.

Here is an example of one of the features

Training set (raw)
         Feature 1     
Sample 1 25.82977   
Sample 2 42.62437   
Sample 3 28.91158   
Sample 4 53.5708    
Sample 5 30.92296
Sample 6 99.16994   
Sample 7 40.75973

Test set (raw)
         Feature 1
Sample 1 2.885865028    
Sample 2 2.572860413    
Sample 3 2.809136715    
Sample 4 2.259630716    
Sample 5 2.797155715    
Sample 6 2.439700763    
Sample 7 2.197087754

Training set (standardized)
         Feature 1
Sample 1 -0.795137358
Sample 2 -0.132081654
Sample 3 -0.673466601
Sample 4  0.300086594
Sample 5 -0.594056732
Sample 6  2.100353934
Sample 7 -0.205698184

Test set (standardized) 
         Feature 1 
Sample 1 -1.700969415
Sample 2 -1.713326928
Sample 3 -1.703998671
Sample 4 -1.725693328
Sample 5 -1.704471684
Sample 6 -1.71858411
Sample 7 -1.728162542

I know SVM uses a threshold to classify samples into either normal or tumor. From what I showed above, their ranges are completely different, is that what is affecting its decision?

What is the problem here and how can I tackle it, thank you in advance. I am so lost and need every bit of advice!

*Edit: As you can see the values of Training set have a completely different range from the values of the Test set, can I still use the mean and standard deviation from my Training set to scale the corresponding features in the Test set?

$\endgroup$
  • 1
    $\begingroup$ Have you adjusted the hyperparameters in the SVM model? $\endgroup$ – from keras import michael Nov 15 '18 at 21:55
  • $\begingroup$ Yes, I have tried but it still classifies one class only. $\endgroup$ – Jane Nov 16 '18 at 3:57
1
$\begingroup$

I can't add a comment so I'll post this here.

My main question is why are your feature values so wildly different different between you your test and training sets?

For example the raw values of feature 1.

Training set: All values > 20, looks like they average about 35-40.
Test set: All values around 2, looks like they all fall between 2-3.

What is the reason for this. Also have you shuffled you data before performing a train test split? The you should plot a distribution of the feature values for both your training and test sets sets, and make sure they are somewhat similar.

It looks like you you have a bad covariate shift between training and test sets. Don't worry about this yet as I think it's a data preprocessing problem. First answer the question if the distribution of values of each feature are similar between training and test sets.

$\endgroup$
  • $\begingroup$ I'm not sure if this answers your question correctly, but my Training set is from one study and my Test set is from another separate study and separate group. Each set uses a different microarray platform and so their measured intensities are ranged differently. $\endgroup$ – Jane Nov 16 '18 at 3:02
  • $\begingroup$ The distribution of values is very different between the Training and Test sets $\endgroup$ – Jane Nov 16 '18 at 8:29
  • 1
    $\begingroup$ I just saw your edit. the first thing to test is if when you normalize both sets, are the distribution means and variances similar? Basically do they now look like they come from the same or very similar distributions? The reason I ask this is because if you think about training an SVM as trying to model a probability distribution if you train on one distrb and try to test on another different distrib, The values do not match up with what the SVM expects, So it fails. $\endgroup$ – ASS466uiuc Nov 16 '18 at 19:50
  • $\begingroup$ I'm not familiar with microarrays, but is there a way to transform the measurements from one set into another? Like is there a way that makes sense in the context of your problem? Also another big issue I see is that you have more features than sample. I think before going any further into training any model you need to try to do some dimensionality reduction, and more so collect more samples. I might be wrong about the sample, but I don't know of any simple techniques to deal with your ratio of features to samples. for Dim reduction look into PCA. Again though you need more data over all else $\endgroup$ – ASS466uiuc Nov 16 '18 at 19:56
  • $\begingroup$ It looks to me like your training set and test set do not conform to the usual understanding of the terms: your training set and test set do not appear to come from the same distribution. The test set is meant to be a holdout set. $\endgroup$ – from keras import michael Nov 16 '18 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.