I am newbie to data science and I do not understand the difference between fit and fit_transform methods in scikit learn.
I have seen similar questions but I did not get intuition from answers. Can anybody simply explain why we might need to transform data?
What does it mean fitting model on training data and transforming to test data.Does it mean for example converting categorical variables into numbers in train and transform new feature set to test data?
Any help would be really appreciated. Please do not refer to scikit learn documentations. Try to explain as simple as possible on your own words :)

up vote 72 down vote accepted

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation.

$$x' = \frac{x-\mu}{\sigma}$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal objects state. Afterwards, you can call its transform() method to apply the transformation to a particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, but it also returns a transformed $x'$. Internally, it just calls first fit() and then transform() on the same data.

  • 1
    Thanks a lot for your answer.Just one thing.By parameters in model it does not mean for exmple slope and intercept for regression? when you fit let's say a linear regression for example which parameters are fitted in fit method? Normalization parameters or model parameters like slope and intercept? – Kaggle Jun 23 '16 at 7:29
  • 1
    I mean parameters internal to the transforms ($\mu$ and $\sigma$ in case of StandardScaler). Whatever transform's get_params() method returns. See this chapter on imputation, for example: scikit-learn.org/stable/modules/… – K3---rnc Jun 23 '16 at 14:40
  • Thx a lot for clarification! – Kaggle Jun 24 '16 at 8:04
  • 2
    My previous comment is actually wrong. In case of linear regression, the fitted parameters are the coef_ (i.e. slope and intercept), not the ones returned by get_params() (which, instead, returns the set of model constructor arguments with their associated values). – K3---rnc Jan 25 '17 at 17:23
  • Great answer! I came across your post while searching on this topic, but I need to clarify. Does that mean that if suppose we want to transform each set of subsequent examples, we should never call fit_transform() as it would not allow us to access the internal objects state, to transform subsequent examples with the same parameters that were obtained using fit() on the initial dataset? This arises for example when, you have a test dataset and want to transform the test set to pass it to your trained classifier. – AKKA Jun 1 at 13:56

The following explanation is based on fit_transform of Imputer class, but the idea is the same for fit_transform of other scikit_learn classes like MinMaxScaler.

transform replaces the missing values with a number. By default this number is the means of columns of some data that you choose. Consider the following example:

imp = Imputer()
# calculating the means
imp.fit([[1, 3], [np.nan, 2], [8, 5.5]])

Now the imputer have learned to use a mean (1+8)/2 = 4.5 for the first column and mean (2+3+5.5)/3 = 3.5 for the second column when it gets applied to a two-column data:

X = [[np.nan, 11], 
     [4,      np.nan], 
     [8,      2],
     [np.nan, 1]]

we get

[[4.5, 11], 
 [4, 3.5],
 [8, 2],
 [4.5, 1]]

So by fit the imputer calculates the means of columns from some data, and by transform it applies those means to some data (which is just replacing missing values with the means). If both these data are the same (i.e. the data for calculating the means and the data that means are applied to) you can use fit_transform which is basically a fit followed by a transform.

Now your questions:

Why we might need to transform data?

"For various reasons, many real world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. Such datasets however are incompatible with scikit-learn estimators which assume that all values in an array are numerical" (source)

What does it mean fitting model on training data and transforming to test data?

The fit of an imputer has nothing to do with fit used in model fitting. So using imputer's fit on training data just calculates means of each column of training data. Using transform on test data then replaces missing values of test data with means that were calculated from training data.

In layman's terms, fit_transform means to do some calculation and then do transformation (say calculating the means of columns from some data and then replacing the missing values). So for training set, you need to both calculate and do transformation.

But for testing set, Machine learning applies prediction based on what was learned during the training set and so it doesn't need to calculate, it just performs the transformation.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.