I have a problem that I have great difficulties understanding the concept that leads to these results. I use a keras dense layer to map 13 input features to 3 output labels. During the training, the loss (RMSE) for the training data constantly decreases even after 2000 epochs. However, the loss for the validation data does not decrease any more after some 200 epochs as you can see in the figure.
Now the thing that I don't understand at all is how you can explain this outcome given that the training and validation data have (almost) exactly the same distribution for all of the 13 input features and the 3 output labels. They are drawn from the very same distribution. To illustrate that I have plotted the histograms for some input features and output labels (and the kernel density functions) as you can see here:
If you want to see the histograms for all input features and output labels you can find them here (in all the distribution of the trai, valid and test dataset look quite similar): https://filetransfer.io/data-package/nHxgDfvF#link
Also I calculated the correlation between each of the input features and the output labels for the training, validation and test dataset and the values are almost the same across the datasets for all combinations. If wou want to the see values you can find them here: https://filetransfer.io/data-package/iVruYbLx#link. This just further highlights, that the training and validation data have the same distribution.
Here you can find my code:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_percentage_error
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import BatchNormalization, Dense, Flatten
from matplotlib import pyplot as plt
#Read data from csv files
ANN_input_data_features = pd.read_csv("C:/Users/User1/Desktop/TestDataANN_InputFeatures.csv", sep=';')
ANN_input_data_labels = pd.read_csv("C:/Users/User1/Desktop/TestDataANN_OutputLabels.csv", sep=';')
ANN_input_data_features = ANN_input_data_features.values
ANN_input_data_labels = ANN_input_data_labels.values
# standardize input features X and output labels Y
scaler_standardized_X = StandardScaler()
ANN_input_data_features = scaler_standardized_X.fit_transform(ANN_input_data_features)
scaler_standardized_Y = StandardScaler()
ANN_input_data_labels = scaler_standardized_Y.fit_transform(ANN_input_data_labels)
#Split dataset into train, validation, an test
index_X_Train_End = int(0.7 * len(ANN_input_data_features))
index_X_Validation_End = int(0.9 * len(ANN_input_data_features))
X_train = ANN_input_data_features [0: index_X_Train_End]
X_valid = ANN_input_data_features [index_X_Train_End: index_X_Validation_End]
X_test = ANN_input_data_features [index_X_Validation_End:]
Y_train = ANN_input_data_labels [0: index_X_Train_End]
Y_valid = ANN_input_data_labels [index_X_Train_End: index_X_Validation_End]
Y_test = ANN_input_data_labels [index_X_Validation_End:]
#Train the model
optimizer_adam = tf.keras.optimizers.Adam(learning_rate= 0.001)
numberOfInputFeatures = len(ANN_input_data_features[0])
numberOfOutputNeurons = len(ANN_input_data_labels[0])
model = keras.Sequential([
Flatten(input_shape=(numberOfInputFeatures,)),
Dense(30, activation='relu'),
#BatchNormalization(axis = 1),
Dense(50, activation='relu'),
#BatchNormalization(axis = 1),
Dense(50, activation='relu'),
#BatchNormalization(axis = 1),
Dense(30, activation='relu'),
keras.layers.Dense(numberOfOutputNeurons)])
entireFolderNameForTheResultsOfTheRun = "C:/Users/User1/Desktop/Training/"
pathOfTheFileForBestModel = entireFolderNameForTheResultsOfTheRun + "bestModelSingleTimeSlotTest.keras"
callbacks = [ keras.callbacks.ModelCheckpoint(pathOfTheFileForBestModel, save_best_only=True) ]
model.compile(loss="mean_squared_error", optimizer=optimizer_adam, metrics=['mean_absolute_percentage_error'])
history = model.fit(X_train, Y_train, epochs=2000, batch_size=10, validation_data=(X_valid, Y_valid), callbacks=callbacks)
# Predict the values from the test dataset
model = keras.models.load_model(pathOfTheFileForBestModel)
Y_pred = model.predict(X_test)
# Rescale the results of the predictions in the test dataset
Y_test_traInv = scaler_standardized_Y.inverse_transform(Y_test)
Y_pred_traInv = scaler_standardized_Y.inverse_transform(Y_pred)
# Calculate the error in the test dataset
rms = mean_squared_error(Y_test_traInv, Y_pred_traInv, squared=True)
mape = mean_absolute_percentage_error(Y_test_traInv, Y_pred_traInv)
print("Evaluation with the test data")
print("Root Mean Squarred Error: ", rms)
print("Mean Absolute Percentage Error:", mape)
#Plot training results
plt.plot(history.history['mean_absolute_percentage_error'])
plt.plot(history.history['val_mean_absolute_percentage_error'])
plt.title('Mean absolute percentage errror')
plt.ylabel('Mean absolute percentage errror')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Loss function')
plt.ylabel('Mean squared Error')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()
Here is the data that I use (input features and output labels): https://filetransfer.io/data-package/ldU9KENV#link
I also tried it with batch normalization but this led to overall worse results. And I also played around with the number of layer, neurons, batchsize etc., but the problem remains the same or the results become way worse.
Now my question is how can this be explained. As far as I understand artifical neural networks, the networks adjusts its weight during the training such that it can explain the training dataset as well as possible. But if the training and validation dataset have the same distribution and the same correlation between input and outputs, as in my case, a model capable of explaining the traning dataset should also be able to explain the validation dataset which is not the case here.
Reminder: As I have still not received a satisfying answer, I would like to remind you on this question. I'll highly appreciate every further answer on this issue.
