I want to use a Multi-Layer-Perceptron in Keras (Dense layer) to map 6 inputs to 1 output. For doing this I use the following code:

from sklearn.preprocessing import StandardScaler
import tensorflow as tf
from tensorflow import keras
from matplotlib import pyplot as plt
import numpy as np

# Define the file paths
MLSupvervised_input_data_file_path = r"C:\Users\User1\Desktop\MLSupvervised_input_data.csv"
MLSupervised_output_data_file_path = r"C:\Users\User1\Desktop\MLSupervised_output_data.csv"

# Read MLSupvervised_input_data and MLSupvervised_input_data from a CSV file as a NumPy array
MLSupvervised_input_data = np.loadtxt(MLSupvervised_input_data_file_path, delimiter=";", dtype=np.float64)
MLSupervised_output_data = np.loadtxt(MLSupervised_output_data_file_path, delimiter=";", dtype=np.float64)

#Standardize the data
scaler_standardized_X = StandardScaler()
MLSupvervised_input_data = scaler_standardized_X.fit_transform(MLSupvervised_input_data)

scaler_standardized_Y = StandardScaler()
MLSupervised_output_data = scaler_standardized_Y.fit_transform(MLSupervised_output_data.reshape(-1, 1))

#Split data
index_X_Train_End = int(0.7 * len(MLSupvervised_input_data))
index_X_Validation_End = int(1.0 * len(MLSupvervised_input_data))

X_train = MLSupvervised_input_data[0: index_X_Train_End]
X_valid = MLSupvervised_input_data[index_X_Train_End: index_X_Validation_End]
Y_train = MLSupervised_output_data[0: index_X_Train_End]
Y_valid = MLSupervised_output_data[index_X_Train_End: index_X_Validation_End]

# Set the random seed
seed_value = 3

#Define optimizer
optimizer_adam = tf.keras.optimizers.Adam(learning_rate= 0.001)

#Define model
model = keras.Sequential([
    keras.layers.Dense(50, activation='relu'),
    keras.layers.Dense(50, activation='relu'),
    keras.layers.Dense(30, activation='relu'),
    keras.layers.Dense(80, activation='relu'),
    keras.layers.Dense(1, activation='relu')])

#Run the model
model.compile(loss="mean_squared_error", optimizer=optimizer_adam, metrics=['mean_absolute_percentage_error'])
history = model.fit(X_train, Y_train, epochs=20, batch_size=16, validation_data=(X_valid, Y_valid))

# Plot training & validation loss values
plt.title('Mean Sqaurred Error')
plt.legend(['Train', 'Validation'], loc='upper right')

I also have uploaded the input data here: https://filetransfer.io/data-package/d53U4e5A#link

The problem is that when using the code multiple times with the same data and the same random_seed, sometimes the training results look okay and sometimes the training and validation loss remain constant and nothing happens.

Here you see the plots from the training with okay results: enter image description here

And here with bad results (constant training and validation losses): enter image description here

Do you have any idea as to why this is happening? The strange aspect is that I use the same random_seed and the same data and still the training performance differs extremely strongly and for me this behaviour occurs kind of "randomly", meaning I have no clue why sometimes the training is okay and sometimes not.


1 Answer 1


To answer your question:

  • The thumb rule for standardization (not a rule, but it's a good practice) that you should first separate the X_train and X_test and then use standardization or normalization techniques over them.
  • Also try using Linear activation function in last layer as you are interested in getting outputs and not the positive outputs only. Because that's what the relu is doing. Also, your loss function and metric are not matching.
  • Finally, about the randomness, if you fix random_state while splitting the data or any random value that you are using then you will get same score every time. But randomness could be good sometimes. Also, if you make sure that you are getting the same rows every time you split the data, then you won't be getting different results.

Hope I answered your questions.

  • $\begingroup$ @Harshard: Thanks Harshad for your answer. To your points 1) When I standardize the data after plitting the standardization will only consider a fraction of the data. So we have 2 different scalars then for the input data x and 2 for y. Why should this be beneficial. 2) Actually the final output should only be positive and relu is a linear function as far as I understand. 3) The most important remark: My question is not primarily about the randomness itself in traning which is not a bad thing. It is about the reason why in several cases "nothing" happens when training (no decrease of loss) $\endgroup$
    – PeterBe
    Jun 29, 2023 at 8:51
  • $\begingroup$ About RELU, it will only give you output when it's positive, and it will zero everything else. That's why I suggested linear function there. About standardization, why we are only fitting on X_train is because we want our test/validation data to resemble the train set. By the way, we will only have 1 scalar not 2. Because we are fitting our StandardScalar on X_train and then transforming other datasets (X_valid and X_test). Finally why thisis the case, because linear models have certain requirements to fulfill to make them work better. If you still get confused, then let me know. $\endgroup$ Jun 29, 2023 at 9:14
  • $\begingroup$ Thanks Harshad for your answer and effort. I really appreciate it. To be totally honest, I am very confused about all of my 3 points. 1) Why shall I not use Relu in the output layer when my output labels are all positive (see linked dataset)? 2) When only having 1 StandardScalar for X_train and 'X_valid ` (any maybe X_test), what difference does it make to scale before or after the splitting? 3) How can these 2 points explain why in some cases nothing happens during the training and the error function remains constant $\endgroup$
    – PeterBe
    Jun 29, 2023 at 12:19
  • $\begingroup$ If all your outputs arr positive, then it's fine. But I was suggesting it because it's a good practice. StandardScaling converts your data into a gaussian format. StandardScaler removes the mean and scales each feature/variable to unit variance. This operation is performed feature-wise in an independent way. StandardScaler can be influenced by outliers. $\endgroup$ Jun 29, 2023 at 13:03
  • $\begingroup$ Neural networks often benefit from scaling the data for several reasons: 1. Most neural network optimization algorithms, such as gradient descent, are sensitive to the scale of the input features. Features with larger scales can dominate the learning process, making it difficult for the network to converge. Scaling the data helps mitigate this issue by bringing all features to a similar scale. $\endgroup$ Jun 29, 2023 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.