Why doesn't loss go down during Neural Net training?

Question

I am working on a Kaggle competition and have tried 2 different code approaches and have the same issue: the loss is large (18247478709991652.0000) and does not go down or is nan.

I'm not sure if there is something wrong with the code or with the data. I tried both scaled and non-scaled data and got the same results. I tried it with the full data set (3,000 examples) and an abbreviated data set.

Here is the abbreviated data.

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

dataframe = pandas.read_csv('data/tmdb/train_processed.csv')
dataframe.drop('id', axis=1, inplace=True)

Y = dataframe['revenue'].values
dataframe.drop(columns=['revenue'], inplace=True)
X = dataframe.values

def baseline_model():
  model = Sequential()
  model.add(Dense(13, input_dim=3, kernel_initializer='normal', activation='relu'))
  model.add(Dense(1, kernel_initializer='normal'))
  model.compile(loss='mean_squared_error', optimizer='adam')
  return model

seed = 7
numpy.random.seed(seed)

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Result: %.2f (%.2f) MSE" % (results.mean(), results.std()))

Answer 1

Your loss does go down, but not significantly. This is because your target values are very large ~10e7 and the default learning rate is scaled for smaller values. The easiest way to fix this is to normalize Y.

If your intention is for your code scale Y, then the problem is that Pipeline does not apply StandardScaler (or any transformations) to Y. You have to use sklearn.compose.TransformedTargetRegressor, or apply the transforms to Y outside of Pipeline.

Pick 1:

Outside the pipeline:
Y = dataframe['revenue'].values
Y = StandardScaler().fit_transform(dataframe['revenue'].values.reshape(-1,1))

Inside the pipeline:
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1)))
from sklearn.compose import TransformedTargetRegressor
estimators.append(('mlp',TransformedTargetRegressor(
regressor=KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=1),
transformer=StandardScaler())))