Reshaping big dataset with MinMaxScaler giving error

Question

My data set is of shape (1249, 228). Most of the entries are zero and other are integers like 1,2,5,10,20 etc. I want to transform this set for the input into LSTM. But when I am applying MinMaxScaler. It is giving the following error:

load the dataset:

dataset1 = pd.read_csv('g:/hello.csv', engine='python')
dataset1= dataset1.drop('packages', axis=1)
dataset1 = dataset1.astype('float32')

normalizing the set

scaler = MinMaxScaler(feature_range=(0, 1))
dataset1 = scaler.fit_transform(dataset1)

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

How can I transform this data set according to the input in LSTM.

Answer 1

I assume you checked for NaN and Inf values manually.

According to the solutions posted here: https://stackoverflow.com/a/44869902/6204860 and https://datascience.stackexchange.com/a/11933/52089

if dataset1 is a Pandas DataFrame try converting it to a matrix by running this:

dataset1 = dataset1.as_matrix().astype(np.float)
np.nan_to_num(dataset1)

and then run dataset1 = scaler.fit_transform(dataset1).

Let me know if this works :)