3
$\begingroup$

I know that you're supposed to scale your test data using the parameters (mean and stdev) from your training data. This is relatively simple; but what if the number of samples is limited in one training data set (e.g. Set A = 5 samples) so I want to combine two data sets (i.e. Set A + Set B = 10 samples) to have enough samples for training, what can I do so that I can scale/normalize the two sets into one and then use those parameters on my test set? If I scale them individually I will have 2 means and 2 stdev.

The context is I'm trying to combine two microarray expression from two different microarray platform so their expression ranges are different.

Thank you for your help in advance

$\endgroup$
  • 1
    $\begingroup$ Are you doing a classification task or a regression task? $\endgroup$ – atmarges Oct 31 '18 at 13:14
  • $\begingroup$ It is unclear what is the goal of the procedure $\endgroup$ – keiv.fly Oct 31 '18 at 14:01
  • $\begingroup$ @Wargream I'm trying to do a classification task $\endgroup$ – Jane Oct 31 '18 at 20:24
0
$\begingroup$

From a proper methodological standpoint, you should do the scaling you're proposing after the two sets are merged.

Either way, the model is not going to be able to differentiate between, say, an anomalous reading from one generating process that falls into the range generated by the other generating process unless you also make a variable indicating the source system that the observation came from (assuming all else is equal).

You need to make sure that both sets of observations actually represent the same population of possible observations in order to make this modeling decision.

$\endgroup$
0
$\begingroup$

I think that what you need is some preprocessing technique such as quantile normalization. You can check this document by Jeff Leek on quantile normalization. In the tutorial he uses R code for normalizing two studies from different populations but on same genes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.