# Why doesn't loss go down during Neural Net training?

I am working on a Kaggle competition and have tried 2 different code approaches and have the same issue: the loss is large (18247478709991652.0000) and does not go down or is nan.

I'm not sure if there is something wrong with the code or with the data. I tried both scaled and non-scaled data and got the same results. I tried it with the full data set (3,000 examples) and an abbreviated data set.

Here is the abbreviated data.

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

dataframe.drop('id', axis=1, inplace=True)

Y = dataframe['revenue'].values
dataframe.drop(columns=['revenue'], inplace=True)
X = dataframe.values

def baseline_model():
model = Sequential()
return model

seed = 7
numpy.random.seed(seed)

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Result: %.2f (%.2f) MSE" % (results.mean(), results.std()))


Your loss does go down, but not significantly. This is because your target values are very large ~10e7 and the default learning rate is scaled for smaller values. The easiest way to fix this is to normalize Y.

If your intention is for your code scale Y, then the problem is that Pipeline does not apply StandardScaler (or any transformations) to Y. You have to use sklearn.compose.TransformedTargetRegressor, or apply the transforms to Y outside of Pipeline.

Pick 1:

Outside the pipeline:
Y = dataframe['revenue'].values
Y = StandardScaler().fit_transform(dataframe['revenue'].values.reshape(-1,1))

Inside the pipeline:
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))
from sklearn.compose import TransformedTargetRegressor
estimators.append(('mlp',TransformedTargetRegressor(
regressor=KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1),
transformer=StandardScaler())))

• I have done that in another code attempt, and forgot to include that in the answer. I'll give this a try. Thanks! – B Seven Feb 21 '19 at 14:33
• I updated my code with the above and got the same symptom: 2700/2700 [==============================] - 1s 406us/step - loss: nan – B Seven Feb 21 '19 at 14:50
• Also, why do you use TransformedTargetRegressor? Why would you use different code for scaled output? – B Seven Feb 21 '19 at 15:16
• TransformedTargetRegressor just applies StandardScalar().fit_transform to Y prior to regressor. Essentially, TransformedTargetRegressor is just another way of doing the same thing as the first snippet. You need to use different code because Pipeline only applies StandardScalar to X. Does the fix work with the abbreviated data? – Aaron Elliot Feb 21 '19 at 16:44
• Oh, I see. That's why you said "Pick 1". The fix does not work on the abbreviated data, with either solution. I thought the issue was with my Tensorflow, so I reinstalled it and it has the same symptom. – B Seven Feb 21 '19 at 17:58