I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there other methods?

My search results came up with a solution


But this works only with panda dataframes (they have column names). Is there a way to implement column headers in the matrix?

import pandas as pd
import numpy as np
import random
import sklearn.preprocessing import StandardScaler

data = pd.DataFrame({'sepal_length': [3.4, 4.5, 3.5], 
                     'sepal_width': [1.2, 1, 2],
                'petal_length': [5.5, 4.5, 4.7],
                'petal_width': [1.2, 1, 3],
                    'species': ['setosa', 'verginica', 'setosa']})

#Shuffle the data and reset the index
from sklearn.utils import shuffle
data = shuffle(data).reset_index(drop = True)  

#Create Independent and dependent matrices
X = data.iloc[:, [0, 1, 2, 3]].values 
y = data.iloc[:, 4].values

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1, random_state = 0)

#Impute missing values at random
prop = int(X_train.size * 0.5) #Set the % of values to be replaced
prop1 = int(X_test.size * 0.5)

a = [random.choice(range(X_train.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
b = [random.choice(range(X_train.shape[1])) for _ in range(prop)]
c = [random.choice(range(X_test.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
d = [random.choice(range(X_test.shape[1])) for _ in range(prop)]
X_train[a, b] = np.NaN
X_test[c, d] = np.NaN

This is where I get the error :Input contains NaN, infinity or a value too large for dtype('float64').

from sklearn.preprocessing import StandardScaler #importing the library that does feature scaling

sc_X = StandardScaler() # created an object with the scaling class

X_train = sc_X.fit_transform(X_train)  # Here we fit and transform the X_train matrix
X_test = sc_X.transform(X_test)

4 Answers 4


This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform:
https://scikit-learn.org/stable/whats_new/v0.20.html#id37 (fourth bullet)


Standardizing (subtracting mean and dividing by standard deviation for each column), can be done using numpy:

Xz = (X - np.nanmean(X, axis=0))/np.nanstd(X, axis=0) 

where X is a matrix (containing NaNs), and Xz is the standardized version of X. Hope this helps.


For a test/training scenario, the mean and std could be stored in respective variables:

m         = np.nanmean(X_train, axis=0)
s         = np.nanstd(X_train, axis=0)
X_train_z = (X_train - m)/s 
X_test_z  = (X_test - m)/s
  • 1
    $\begingroup$ at test data time, what does one do? $\endgroup$
    – Anuj Gupta
    Commented Jul 30, 2019 at 4:11

You can use sklearn.preprocessing.Imputer.


from sklearn import datasets as ds
from sklearn.model_selection import train_test_split

# load Iris data set    
data = ds.load_iris()

X = data.data
y = data.target

# artificially set 33% of [X] data set to NaN's    
X.ravel()[np.random.choice(X.size, int(X.shape[0]*.33), replace=False)] = np.nan


In [137]: X
array([[5.1, 3.5, nan, nan],
       [nan, 3. , 1.4, 0.2],
       [4.7, nan, 1.3, 0.2],
       [6.5, 3. , nan, 2. ],
       [6.2, 3.4, 5.4, 2.3],
       [5.9, 3. , nan, 1.8]])

now we can impute and standardize it:

imp = Imputer(strategy="mean", axis=0)
scale = StandardScaler()

In [139]: X_new = scale.fit_transform(imp.fit_transform(X))


In [160]: X_new
array([[-1.03733263e+00,  1.22587069e+00, -1.37398311e-15, -3.17837019e-16],
       [ 1.18191646e-15, -5.32987255e-02, -1.43399195e+00, -1.35522269e+00],
       [-1.56962048e+00,  2.27226133e-15, -1.49587065e+00, -1.35522269e+00],
       [ 8.25674859e-01, -5.32987255e-02, -1.37398311e-15,  1.22131653e+00],
       [ 4.26458969e-01,  9.70036804e-01,  1.04115598e+00,  1.65073974e+00],
       [ 2.72430791e-02, -5.32987255e-02, -1.37398311e-15,  9.35034396e-01]])

Demo2, using Pipeline:

from sklearn.preprocessing import Imputer, StandardScaler
from sklearn.pipeline import Pipeline


estimator = Pipeline([("impute", Imputer(strategy="mean", axis=0)),
                      ("scale", StandardScaler()),
                      ("forest", RandomForestRegressor(random_state=0,

estimator.fit(X_train, y_train)
  • $\begingroup$ The code runs, but it there a way to perform fit_transform for this? $\endgroup$
    – uharsha33
    Commented Apr 16, 2018 at 14:12
  • $\begingroup$ @uharsha33, sure you can. Just don't stack it in a pipeline and use Imputer(...).fit_transform() separately... $\endgroup$ Commented Apr 16, 2018 at 14:15
  • $\begingroup$ My coding skills are pretty limited, would be helpful if you could show me a demo. Also, my y_train wouldn't need imputation as it doesn't have any missing values. And when i tried estimator.fit(X_train), it threw up an error about singleton array $\endgroup$
    – uharsha33
    Commented Apr 16, 2018 at 14:21
  • $\begingroup$ Sorry about that, I've edited the question as you've suggested. $\endgroup$
    – uharsha33
    Commented Apr 16, 2018 at 14:41
  • $\begingroup$ @uharsha33, please test you script, i'm getting: IndexError: single positional indexer is out-of-bounds $\endgroup$ Commented Apr 16, 2018 at 14:52

Working with NaNs is always a bit difficult. Maybe it would be useful if you try to enrich NaN values. For example, by averaging the considered feature for groups like an age class. If only a few records have NaN values, you might simply drop these (pandas dropna).

  • $\begingroup$ dropping rows is out of question. My dataset has 50% missing values. $\endgroup$
    – uharsha33
    Commented Apr 16, 2018 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.