There isnt a lot of wisdom here but on a regression data set, I can get a slightly lower RSME with adding in a learning rate schedule for exponential decay. The curve on the plot is also slightly smoother too. Its sort of hard to see but this first plot is with adding the exponential decay.

Exponential decay, Final RSME is 6.5

enter image description here

No exponential decay, final RSME is 7.5 enter image description here

Would anyone have any other tips to try for deep learning tuning/better performance? This is my script with tf.keras below. The model depth is something I already experimented with on the layers & amount of epochs...

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import math
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from scipy.stats import t
from tensorflow.keras import backend
import os

# This function keeps the learning rate at 0.001
# and decreases it exponentially after that.
def scheduler(epoch):
  if epoch < 2:
    return 0.001
    return 0.001 * tf.math.exp(0.1 * (1 - epoch))

callback = tf.keras.callbacks.LearningRateScheduler(scheduler)

#function to calculate RSME
def rmse(y_true, y_pred):
    return backend.sqrt(backend.mean(backend.square(y_pred - y_true), axis=-1))

# load training/baseline dataset
#df = pd.read_csv('School District Test Data/hourly_summary.csv', index_col='Date', parse_dates=True)
df = pd.read_csv('df_ols_noHvac_all.csv', index_col='Date', parse_dates=True)

dfTrain = df.copy()

# split into input (X) and output (Y) variables
X = dfTrain.drop(['Demand'],1)
Y = dfTrain['Demand']

#define training & testing data set
offset = int(X.shape[0] * 0.7)
X_train, Y_train = X[:offset], Y[:offset]
X_test, Y_test = X[offset:], Y[offset:]

#define model
model = Sequential()
model.add(Dense(60, input_dim=42, kernel_initializer='normal', activation='relu'))
model.add(Dense(55, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, kernel_initializer='normal', activation='relu'))
model.add(Dense(45, kernel_initializer='normal', activation='relu'))
model.add(Dense(30, kernel_initializer='normal', activation='relu'))
model.add(Dense(20, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
model.compile(loss='mse', optimizer='adam', metrics=[rmse])

# train model, test callback option for rlrp or no rlrp
history = model.fit(X_train, Y_train, epochs=85, batch_size=1, verbose=2, callbacks=[callback])
#history = model.fit(X_train, Y_train, epochs=85, batch_size=1, verbose=2)

# plot metrics
plt.title('kW RSME Vs Epoch')

print('[INFO] Saved model to disk')

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.