How do we standardize arrays with NaN?

I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there other methods?

My search results came up with a solution

df['col']=(df['col']-df['col'].min())/(df['col'].max()-df['col'].min())


But this works only with panda dataframes (they have column names). Is there a way to implement column headers in the matrix?

import pandas as pd
import numpy as np
import random
import sklearn.preprocessing import StandardScaler

data = pd.DataFrame({'sepal_length': [3.4, 4.5, 3.5],
'sepal_width': [1.2, 1, 2],
'petal_length': [5.5, 4.5, 4.7],
'petal_width': [1.2, 1, 3],
'species': ['setosa', 'verginica', 'setosa']})

#Shuffle the data and reset the index
from sklearn.utils import shuffle
data = shuffle(data).reset_index(drop = True)

#Create Independent and dependent matrices
X = data.iloc[:, [0, 1, 2, 3]].values
y = data.iloc[:, 4].values

#train_test_split
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1, random_state = 0)

#Impute missing values at random
prop = int(X_train.size * 0.5) #Set the % of values to be replaced
prop1 = int(X_test.size * 0.5)

a = [random.choice(range(X_train.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
b = [random.choice(range(X_train.shape[1])) for _ in range(prop)]
c = [random.choice(range(X_test.shape[0])) for _ in range(prop)] #Randomly choose indices of the numpy array
d = [random.choice(range(X_test.shape[1])) for _ in range(prop)]
X_train[a, b] = np.NaN
X_test[c, d] = np.NaN


This is where I get the error :Input contains NaN, infinity or a value too large for dtype('float64').

from sklearn.preprocessing import StandardScaler #importing the library that does feature scaling

sc_X = StandardScaler() # created an object with the scaling class

X_train = sc_X.fit_transform(X_train)  # Here we fit and transform the X_train matrix
X_test = sc_X.transform(X_test)


Standardizing (subtracting mean and dividing by standard deviation for each column), can be done using numpy:

Xz = (X - np.nanmean(X, axis=0))/np.nanstd(X, axis=0)


where X is a matrix (containing NaNs), and Xz is the standardized version of X. Hope this helps.

EDITED:

For a test/training scenario, the mean and std could be stored in respective variables:

m         = np.nanmean(X_train, axis=0)
s         = np.nanstd(X_train, axis=0)
X_train_z = (X_train - m)/s
X_test_z  = (X_test - m)/s

• at test data time, what does one do? – Anuj Gupta Jul 30 '19 at 4:11

You can use sklearn.preprocessing.Imputer.

Demo:

from sklearn import datasets as ds
from sklearn.model_selection import train_test_split

X = data.data
y = data.target

# artificially set 33% of [X] data set to NaN's
X.ravel()[np.random.choice(X.size, int(X.shape[0]*.33), replace=False)] = np.nan


yields:

In [137]: X
Out[137]:
array([[5.1, 3.5, nan, nan],
[nan, 3. , 1.4, 0.2],
[4.7, nan, 1.3, 0.2],
...,
[6.5, 3. , nan, 2. ],
[6.2, 3.4, 5.4, 2.3],
[5.9, 3. , nan, 1.8]])


now we can impute and standardize it:

imp = Imputer(strategy="mean", axis=0)
scale = StandardScaler()

In [139]: X_new = scale.fit_transform(imp.fit_transform(X))


result:

In [160]: X_new
Out[160]:
array([[-1.03733263e+00,  1.22587069e+00, -1.37398311e-15, -3.17837019e-16],
[ 1.18191646e-15, -5.32987255e-02, -1.43399195e+00, -1.35522269e+00],
[-1.56962048e+00,  2.27226133e-15, -1.49587065e+00, -1.35522269e+00],
...,
[ 8.25674859e-01, -5.32987255e-02, -1.37398311e-15,  1.22131653e+00],
[ 4.26458969e-01,  9.70036804e-01,  1.04115598e+00,  1.65073974e+00],
[ 2.72430791e-02, -5.32987255e-02, -1.37398311e-15,  9.35034396e-01]])


Demo2, using Pipeline:

from sklearn.preprocessing import Imputer, StandardScaler
from sklearn.pipeline import Pipeline

#...

estimator = Pipeline([("impute", Imputer(strategy="mean", axis=0)),
("scale", StandardScaler()),
("forest", RandomForestRegressor(random_state=0,
n_estimators=100))])

estimator.fit(X_train, y_train)
#...

• The code runs, but it there a way to perform fit_transform for this? – uharsha33 Apr 16 '18 at 14:12
• @uharsha33, sure you can. Just don't stack it in a pipeline and use Imputer(...).fit_transform() separately... – MaxU Apr 16 '18 at 14:15
• My coding skills are pretty limited, would be helpful if you could show me a demo. Also, my y_train wouldn't need imputation as it doesn't have any missing values. And when i tried estimator.fit(X_train), it threw up an error about singleton array – uharsha33 Apr 16 '18 at 14:21
• Sorry about that, I've edited the question as you've suggested. – uharsha33 Apr 16 '18 at 14:41
• @uharsha33, please test you script, i'm getting: IndexError: single positional indexer is out-of-bounds – MaxU Apr 16 '18 at 14:52

This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform:
https://scikit-learn.org/stable/whats_new.html#id82 (fourth bullet)
https://github.com/scikit-learn/scikit-learn/issues/10404

Working with NaNs is always a bit difficult. Maybe it would be useful if you try to enrich NaN values. For example, by averaging the considered feature for groups like an age class. If only a few records have NaN values, you might simply drop these (pandas dropna).

• dropping rows is out of question. My dataset has 50% missing values. – uharsha33 Apr 16 '18 at 15:06