My Neural network in Tensorflow does a bad job in comparison to the same Neural network in Keras

Question

I am trying to predict "sales" from this dataset:

https://www.kaggle.com/c/rossmann-store-sales

There are>1,000,000 rows, I use 10 features from the dataset to predict sales

I merged two datasets into one in advance. I created a code in Keras to predict "sales". Firstly I created some new variables, threw away some unneeded data. Then I applied one hot encoding on categorical variables, split the dataset into train and test parts, scaled variables of X_train and X_test with StandardScaler. After that, I created a Keras model that looks like this:

model = Sequential()    
model.add(Dense(units = 64, kernel_initializer = 'uniform', activation = 'relu', input_dim = 31))
model.add(Dropout(p = 0.1))
model.add(Dense(units = 64, kernel_initializer = 'uniform', activation = 'relu'))
model.add(Dropout(p = 0.1))
model.add(Dense(units = 64, kernel_initializer = 'uniform', activation = 'relu'))
model.add(Dropout(p = 0.1))
model.add(Dense(units = 64, kernel_initializer = 'uniform', activation = 'relu'))
model.add(Dropout(p = 0.1))
model.add(Dense(units = 1, kernel_initializer = 'uniform', activation='linear'))
model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae', 'mape'])
history = model.fit(X_train, y_train, batch_size = 10000, epochs = 15)

It is a pretty basic model: 4 layers, each has 64 neurons, small dropout to prevent overfitting, relu as an activator, mean squared error as loss function, adam as an optimizer, 15 epochs.

The results of this model:

• R-squared: 0.86
• MSE: 20841
• MAE: 103

I suppose it is doing a good job, this is a comparison of real and predicted values

        y_test  final_preds
0       1495.0  1737.393188
1        970.0   763.265747
2        660.0   696.281006
3        695.0   884.019226
4        802.0   620.464294
5        437.0   413.590912
6        599.0   564.844177
7        426.0   507.872650
8       1163.0   934.790405
9        563.0   591.833313
10       798.0   729.736572
11       507.0   422.795746
12       447.0   546.338440
13       437.0   437.536194
14       599.0   643.752441
15       607.0   667.271423
16       836.0   793.968872
17       568.0   599.968262
18       522.0   508.874084
19       350.0   395.198883
20      1160.0  1277.464111

I tried to "mimic" the same structure of neural network with the same configurations in Tensorflow by using DNNRegressor. The results were not even close to what Keras achieved. My code for TF is:

• Creating feature columns

  DayOfWeek_vocab = [4, 3, 1, 5, 6, 2, 7]
  DayOfWeek_column = tf.feature_column.categorical_column_with_vocabulary_list(
    key="DayOfWeek", vocabulary_list=DayOfWeek_vocab 
  
  Open_vocab = [1]
  Open_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="Open", vocabulary_list=Open_vocab)
  
  Promo_vocab = [1,0]
  Promo_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="Promo", vocabulary_list=Promo_vocab)
  
  StateHoliday_vocab = ['0', 'b', 'a', 'c']
  StateHoliday_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="StateHoliday", vocabulary_list=StateHoliday_vocab)
  
  SchoolHoliday_vocab = [1, 0]
  SchoolHoliday_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="SchoolHoliday", vocabulary_list=SchoolHoliday_vocab)
  
  StoreType_vocab = ['a', 'd', 'c', 'b']
  StoreType_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="StoreType", vocabulary_list=StoreType_vocab)
  
  Assortment_vocab = ['a', 'c', 'b']
  Assortment_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="Assortment", vocabulary_list=Assortment_vocab)
  
  month_vocab = [10,  3,  4,  2,  9,  6,  5,  7,  1,  8, 12, 11]
  month_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="month", vocabulary_list=month_vocab)
  
  Season_vocab = ['Autumn', 'Spring', 'Winter', 'Summer']
  Season_column = tf.feature_column.categorical_column_with_vocabulary_list(
        key="Season", vocabulary_list=Season_vocab)
  
  feature_columns = [
  tf.feature_column.indicator_column(DayOfWeek_column),
  tf.feature_column.indicator_column(Open_column),
  tf.feature_column.indicator_column(Promo_column),
  tf.feature_column.indicator_column(StateHoliday_column),
  tf.feature_column.indicator_column(SchoolHoliday_column),
  tf.feature_column.indicator_column(StoreType_column),
  tf.feature_column.indicator_column(Assortment_column),
  tf.feature_column.numeric_column('CompetitionDistance'),
  tf.feature_column.indicator_column(month_column),
  tf.feature_column.indicator_column(Season_column),
    ]
  

• The model itself

  input_func = tf.estimator.inputs.pandas_input_fn(x=X_train,y=y_train ,batch_size=10000,num_epochs=15,
                                      shuffle=True)
  model = tf.estimator.DNNRegressor(hidden_units=[64,64,64,64],feature_columns=feature_columns,                                
  optimizer=tf.train.AdamOptimizer(learning_rate=0.0001),
                            activation_fn = tf.nn.relu)
  model.train(input_fn=input_func,steps=1000000)
  

The structure is the same as in Keras, 4 layers, 64 neurons. relu, adam and mse as cost (it is a default for DNNRegressor), but tf does not work as good as Keras

Results are a mess, MSE is 44303762026251.3, MAE is 3809120.3086946052, R-squared is even negative, -4598900.028032559

What did I do wrong here? Did I forget something in Tensorflow? Keras is using TF, so I suppose that results should be similar if the model is tuned in the same way.

I randomly put numbers in layers, neurons, learning rate, epochs, but it does not work as well Thank you in advance!

edit1

Thanks for your comments! I tried to apply what you recommended. I totally abanded DNNRegressor and tried to "manually" create everything with tf.layers.dense. I, again, copied the structure of keras (changed to glorot in keras as well). Thats how it looks now:

import tensorflow as tf
import numpy as np
import uuid

x = tf.placeholder(shape=[None, 30], dtype=tf.float32)
y = tf.placeholder(shape=[None, 1], dtype=tf.float32)


    dense = tf.layers.dense(x, 30, activation = tf.nn.relu,
                            bias_initializer = tf.zeros_initializer(),
                            kernel_initializer = tf.glorot_uniform_initializer())
    dropout = tf.layers.dropout(inputs = dense, rate = 0.1)
    dense = tf.layers.dense(dropout, 64, activation = tf.nn.relu,
                            bias_initializer = tf.zeros_initializer(),
                            kernel_initializer = tf.glorot_uniform_initializer())
    dropout = tf.layers.dropout(inputs = dense, rate = 0.1)
    dense = tf.layers.dense(dropout, 64, activation = tf.nn.relu,
                            bias_initializer = tf.zeros_initializer(),
                            kernel_initializer = tf.glorot_uniform_initializer())
    dropout = tf.layers.dropout(inputs = dense, rate = 0.1)
    dense = tf.layers.dense(dropout, 64, activation = tf.nn.relu,
                            bias_initializer = tf.zeros_initializer(),
                            kernel_initializer = tf.glorot_uniform_initializer())
    dropout = tf.layers.dropout(inputs = dense, rate = 0.1)
    dense = tf.layers.dense(dropout, 64, activation = tf.nn.relu,
                            bias_initializer = tf.zeros_initializer(),
                            kernel_initializer = tf.glorot_uniform_initializer())
    dropout = tf.layers.dropout(inputs = dense, rate = 0.1)
    output = tf.layers.dense(dropout, 1, activation = tf.nn.sigmoid)

    cost = tf.losses.absolute_difference(y, output) #mae
    optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(cost)
    init = tf.global_variables_initializer()

    tf.summary.scalar("cost", cost)
    merged_summary_op = tf.summary.merge_all()

    with tf.Session() as sess:
        sess.run(init)
        uniq_id = "/tmp/tensorboard-layers-api/" + uuid.uuid1().__str__()[:6]
        summary_writer = tf.summary.FileWriter(uniq_id, graph=tf.get_default_graph())
        x_vals = X_train
        y_vals = y_train
        #for step in range(673764):
        for step in range(673764):
            _, val, summary = sess.run([optimizer, cost, merged_summary_op],
                                       feed_dict={x: x_vals, y: y_vals})
            if step % 20 == 0:
                print("step: {}, value: {}".format(step, val))
                summary_writer.add_summary(summary, step)

TF model is slower, so I cannot check precisely the output, but first steps of TF are close to results of a first epoch of keras:

Epoch 1/15
673764/673764 [==============================] - 13s 19us/step - loss: 57019592.1866 - mean_squared_error: 57019592.1866 - mean_absolute_error: 6883.4074 - mean_absolute_percentage_error: 2668499.3291

TF:

step: 0, value: 6957.24365234375
step: 20, value: 6957.2373046875
step: 40, value: 6957.23583984375
step: 60, value: 6957.22998046875

So MAE of both models are close, around 6900. I suppose that the issue is solved now.

I just have one question left, how to apply batches in this type of tensorflow? It is the first time I ever built tf like this and I haven't found an obvious solution online. Thanks!

Answer 1

You can try these:

a) In the tensorflow implementation you have used learning rate of 0.0001 while in keras the default value of learning rate is 0.001. You have set different learning rates in the two implementations. Try the same learning rate and the results you will get should be close enough.

b) The initializations you have used in keras is 'uniform' while in tensorflow the default initialization is glorot uniform. Use the same in both implementation.

c) Try same dropout. In keras implementation you have used dropout after every hidden layer. Try the same in tensorflow implementation.

Tip:-

Try same random initializations for both the implementation.

Edit 1:

I have never implemented the procedure you have asked. A bit of googling has led me to some useful pointers.

You can have a look at this link on SO for training the network in batches.

You have to iterate and train the network over the samples in the last part of your code. For each iteration you have to train the model over a number of data points which is basically the batch size. creating an object that will feed in the number of data points at every iteration to the optimizer should do the trick, I guess.