I am newbie to data science and I do not understand the difference between fit and fit_transform methods in scikit learn.
I have seen similar questions but I did not get intuition from answers. Can anybody simply explain why we might need to transform data?
What does it mean fitting model on training data and transforming to test data.Does it mean for example converting categorical variables into numbers in train and transform new feature set to test data?
Any help would be really appreciated. Please do not refer to scikit learn documentations. Try to explain as simple as possible on your own words :)


To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation.

$$x' = \frac{x-\mu}{\sigma}$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal objects state. Afterwards, you can call its transform() method to apply the transformation to a particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, but it also returns a transformed $x'$. Internally, it just calls first fit() and then transform() on the same data.

  • 1
    $\begingroup$ Thanks a lot for your answer.Just one thing.By parameters in model it does not mean for exmple slope and intercept for regression? when you fit let's say a linear regression for example which parameters are fitted in fit method? Normalization parameters or model parameters like slope and intercept? $\endgroup$ – Kaggle Jun 23 '16 at 7:29
  • 1
    $\begingroup$ I mean parameters internal to the transforms ($\mu$ and $\sigma$ in case of StandardScaler). Whatever transform's get_params() method returns. See this chapter on imputation, for example: scikit-learn.org/stable/modules/… $\endgroup$ – K3---rnc Jun 23 '16 at 14:40
  • $\begingroup$ Thx a lot for clarification! $\endgroup$ – Kaggle Jun 24 '16 at 8:04
  • 2
    $\begingroup$ My previous comment is actually wrong. In case of linear regression, the fitted parameters are the coef_ (i.e. slope and intercept), not the ones returned by get_params() (which, instead, returns the set of model constructor arguments with their associated values). $\endgroup$ – K3---rnc Jan 25 '17 at 17:23
  • $\begingroup$ Great answer! I came across your post while searching on this topic, but I need to clarify. Does that mean that if suppose we want to transform each set of subsequent examples, we should never call fit_transform() as it would not allow us to access the internal objects state, to transform subsequent examples with the same parameters that were obtained using fit() on the initial dataset? This arises for example when, you have a test dataset and want to transform the test set to pass it to your trained classifier. $\endgroup$ – AKKA Jun 1 '18 at 13:56

The following explanation is based on fit_transform of Imputer class, but the idea is the same for fit_transform of other scikit_learn classes like MinMaxScaler.

transform replaces the missing values with a number. By default this number is the means of columns of some data that you choose. Consider the following example:

imp = Imputer()
# calculating the means
imp.fit([[1, 3], [np.nan, 2], [8, 5.5]])

Now the imputer have learned to use a mean (1+8)/2 = 4.5 for the first column and mean (2+3+5.5)/3 = 3.5 for the second column when it gets applied to a two-column data:

X = [[np.nan, 11], 
     [4,      np.nan], 
     [8,      2],
     [np.nan, 1]]

we get

[[4.5, 11], 
 [4, 3.5],
 [8, 2],
 [4.5, 1]]

So by fit the imputer calculates the means of columns from some data, and by transform it applies those means to some data (which is just replacing missing values with the means). If both these data are the same (i.e. the data for calculating the means and the data that means are applied to) you can use fit_transform which is basically a fit followed by a transform.

Now your questions:

Why we might need to transform data?

"For various reasons, many real world datasets contain missing values, often encoded as blanks, NaNs or other placeholders. Such datasets however are incompatible with scikit-learn estimators which assume that all values in an array are numerical" (source)

What does it mean fitting model on training data and transforming to test data?

The fit of an imputer has nothing to do with fit used in model fitting. So using imputer's fit on training data just calculates means of each column of training data. Using transform on test data then replaces missing values of test data with means that were calculated from training data.


In layman's terms, fit_transform means to do some calculation and then do transformation (say calculating the means of columns from some data and then replacing the missing values). So for training set, you need to both calculate and do transformation.

But for testing set, Machine learning applies prediction based on what was learned during the training set and so it doesn't need to calculate, it just performs the transformation.


These methods are used for dataset transformations in scikit-learn:

Let us take an example for Scaling values in a dataset:

Here the fit method, when applied to the training dataset,learns the model parameters (for example, mean and standard deviation). We then need to apply the transform method on the training dataset to get the transformed (scaled) training dataset. We could also perform both of this steps in one step by applying fit_transform on the training dataset.

Then why do we need 2 separate methods - fit and transform ?

In practice we need to have a separate training and testing dataset and that is where having a separate fit and transform method helps. We apply fit on the training dataset and use the transform method on both - the training dataset and the test dataset. Thus the training as well as the test dataset are then transformed(scaled) using the model parameters that were learnt on applying the fit method the training dataset.

Example Code:

scaler = preprocessing.StandardScaler().fit(X_train)

By Applying the Transformations you are trying to make your data to behave normally for example if you have two variables $V_1$ and $V_2$ both measures the distances but $V_1$ has the units as centimeters and $V_2$ has the units in Kilometers so in order to compare these two you have to convert them to same units... just like that Transforming is making similar behavior or making to behave like normal distribution

Coming to other question you first build the model in training set that is (the model learns the patterns or Behavior of your data from the training set) and when you run the same model in the test set it tries to identify the similar patterns or behaviors once it identifies it makes its conclusions and gives results accordingly training data


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.