4
$\begingroup$

I know that you're supposed to scale your test data using the parameters (mean and stdev) from your training data. This is relatively simple; but what if the number of samples is limited in one training data set (e.g. Set A = 5 samples) so I want to combine two data sets (i.e. Set A + Set B = 10 samples) to have enough samples for training, what can I do so that I can scale/normalize the two sets into one and then use those parameters on my test set? If I scale them individually I will have 2 means and 2 stdev.

The context is I'm trying to combine two microarray expression from two different microarray platform so their expression ranges are different.

Thank you for your help in advance

$\endgroup$
4
  • 1
    $\begingroup$ Are you doing a classification task or a regression task? $\endgroup$
    – atmarges
    Oct 31, 2018 at 13:14
  • $\begingroup$ It is unclear what is the goal of the procedure $\endgroup$
    – keiv.fly
    Oct 31, 2018 at 14:01
  • $\begingroup$ @Wargream I'm trying to do a classification task $\endgroup$
    – Jane
    Oct 31, 2018 at 20:24
  • $\begingroup$ If data from set A and set B are on different scales, then you should not be combining them into a single dataset. For example, imagine you are predicting some output based on measurements. Set A is in the metric system and set B is in the imperial system. You need to work out the relative scale of each feature and convert one of the sets so that both sets are on the same scale for each feature. Then you can combine them and do your normization scaling. $\endgroup$ Jan 12, 2022 at 2:23

2 Answers 2

1
$\begingroup$

From a proper methodological standpoint, you should do the scaling you're proposing after the two sets are merged.

Either way, the model is not going to be able to differentiate between, say, an anomalous reading from one generating process that falls into the range generated by the other generating process unless you also make a variable indicating the source system that the observation came from (assuming all else is equal).

You need to make sure that both sets of observations actually represent the same population of possible observations in order to make this modeling decision.

$\endgroup$
1
$\begingroup$

I think that what you need is some preprocessing technique such as quantile normalization. You can check this document by Jeff Leek on quantile normalization. In the tutorial he uses R code for normalizing two studies from different populations but on same genes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.