1
$\begingroup$

I am very perplexed by the lower loss but higher RMSE:

Here's a newer model with better loss scores on the same dataset and many predictors:

Fold 3 of 3

Epoch 1/10
170362/170362 [==============================] - 640s 4ms/step - loss: 4.7891e-04
Epoch 2/10
170362/170362 [==============================] - 636s 4ms/step - loss: 1.0931e-04
Epoch 3/10
170362/170362 [==============================] - 639s 4ms/step - loss: 8.6029e-05
Epoch 4/10
170362/170362 [==============================] - 641s 4ms/step - loss: 7.6854e-05
Epoch 5/10
170362/170362 [==============================] - 637s 4ms/step - loss: 6.7049e-05
Epoch 6/10
170362/170362 [==============================] - 637s 4ms/step - loss: 6.3263e-05
Epoch 7/10
170362/170362 [==============================] - 638s 4ms/step - loss: 5.8497e-05
Epoch 8/10
170362/170362 [==============================] - 639s 4ms/step - loss: 6.2477e-05
Epoch 9/10
170362/170362 [==============================] - 638s 4ms/step - loss: 5.3870e-05
Epoch 10/10
170362/170362 [==============================] - 639s 4ms/step - loss: 5.4414e-05
170362/170362 [==============================] - 273s 2ms/step
85181/85181 [==============================] - 138s 2ms/step
Average Train Score: 1732.52 RMSE
Average Test Score: 1732.63 RMSE

Older model with worse loss scores on the same dataset, but with fewer predictors:

Fold 3 of 3


Epoch 1/5
164325/164325 [==============================] - 423s 3ms/step - loss: 2.1169e-04
Epoch 2/5
164325/164325 [==============================] - 419s 3ms/step - loss: 1.4752e-04
Epoch 3/5
164325/164325 [==============================] - 421s 3ms/step - loss: 1.3906e-04
Epoch 4/5
164325/164325 [==============================] - 421s 3ms/step - loss: 1.3506e-04
Epoch 5/5
164325/164325 [==============================] - 421s 3ms/step - loss: 1.3155e-04
41082/41082 [==============================] - 46s 1ms/step
20541/20541 [==============================] - 23s 1ms/step
Average Train Score: 1655.46 RMSE
Average Test Score: 1655.44 RMSE

The 5th epoch of the older model has a loss: 1.3155e-04 with RMSE:

Average Train Score: 1655.46 RMSE
Average Test Score: 1655.44 RMSE

My newer model with more data and a lower epoch loss of: loss: 5.4414e-05 has a higher RMSE:

Average Train Score: 1732.52 RMSE
Average Test Score: 1732.63 RMSE

How can this happen? My code seems pretty straightforward:

scaler_price = MinMaxScaler(feature_range=(0, 1))
scaler_features = MinMaxScaler(feature_range=(0, 1))
scaled_price = scaler_price.fit_transform(stack[['target']])
# I put all of the features here so I can iterate on what subsets to use and see which ones perform best
scaled_features = scaler_features.fit_transform(stack[[
    # 128 features here, not including the target
]])
scaled_data = np.concatenate([scaled_price, scaled_features], axis=1)

from keras.utils import Sequence
import numpy as np
import keras
from sklearn.model_selection import KFold
from keras.models import Sequential
from keras.layers import LSTM, Dropout, Dense
from keras.optimizers import Adam
import math
from sklearn.metrics import mean_squared_error

# define custom data generator
class TimeSeriesGenerator(Sequence):
    def __init__(self, data, targets, length, batch_size):
        self.data, self.targets = data, targets
        self.length = length
        self.batch_size = batch_size

    def __len__(self):
        return int(np.ceil(len(self.data) / float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x = self.data[idx * self.batch_size:(idx + 1) * self.batch_size]
        batch_y = self.targets[idx * self.batch_size:(idx + 1) * self.batch_size]
        return np.array(batch_x), np.array(batch_y)

# initial setup
n_epochs = 10
n_batch_size = 8
look_back = 10
X, y = create_dataset(scaled_data, look_back)
k = 3
random_seed = 13
kf = KFold(n_splits=k, shuffle=True, random_state=random_seed)

train_scores = []
test_scores = []

for i, (train_index, test_index) in enumerate(kf.split(X)):
    print(f"\nFold {i + 1} of {kf.get_n_splits()}\n")

    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    # create data generators
    train_generator = TimeSeriesGenerator(X_train, y_train, look_back, n_batch_size)
    test_generator = TimeSeriesGenerator(X_test, y_test, look_back, n_batch_size)

    # define model
    model = Sequential([
        LSTM(250, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences=True),
        Dropout(0.2),
        LSTM(100),
        Dense(1)
    ])
    model.compile(optimizer=Adam(), loss='mean_squared_error')
    
    # experimental code start
    log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
    # Start TensorBoard through the command line or within a notebook experience. The two interfaces are generally the same. In notebooks, use the %tensorboard line magic. On the command line, run the same command without "%".
    # %tensorboard --logdir logs/fit
    # experimental code end

    # fit model using generator
    model.fit(train_generator, epochs=n_epochs, verbose=1)

    # predict and evaluate
    train_predict = model.predict(train_generator)
    test_predict = model.predict(test_generator)
    train_predict = scaler_price.inverse_transform(train_predict)
    test_predict = scaler_price.inverse_transform(test_predict)
    train_score = math.sqrt(mean_squared_error(y_train[:len(train_predict)], train_predict))
    test_score = math.sqrt(mean_squared_error(y_test[:len(test_predict)], test_predict))
    train_scores.append(train_score)
    test_scores.append(test_score)

# calculate average scores
avg_train_score = np.mean(train_scores)
avg_test_score = np.mean(test_scores)

print(f'Average Train Score: {avg_train_score:.2f} RMSE')
print(f'Average Test Score: {avg_test_score:.2f} RMSE')
$\endgroup$

1 Answer 1

0
$\begingroup$

In this case, seeing high RMSE values indicates large errors still happening even though your average MSE is smaller. Remember, RMSE heavily penalizes large errors, so in your case, the frequency of errors is less; however, the errors are more significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.