0
$\begingroup$

I have a simple toy model that I'm using to learn from (identity function). The dataset is every increment of 0.01 from [0, 1], for both $x$ and $y$. So if $x_i$ is 0.01, $y_i$ is also 0.01. If $x_i$ is 0.71, $y_i$ is also 0.71. etc.

When I train the exact same model with no changes at all, sometimes it works and other times it doesn't.

Success:

enter image description here

Fail:

enter image description here

I'm using Jupyter in VSCode. I've tried restarting the runtime between trainings. I've tried exiting VSCode and restarting. But I can't figure out how to get the model to consistently train succesfully. Why does this happen?

       X     Y
0   0.01  0.01
1   0.02  0.02
2   0.03  0.03
3   0.04  0.04
4   0.05  0.05
..   ...   ...
95  0.96  0.96
96  0.97  0.97
97  0.98  0.98
98  0.99  0.99
99  1.00  1.00
import numpy as np
import pandas as pd
from math import floor
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Input, Dense


data = {
    "X": [(i+1)*0.01 for i in range(100)],
    "Y": [(i+1)*0.01 for i in range(100)]
}
df = pd.DataFrame(data)

X = df[['X']] # double brackets so dataframe can be directly input
Y = df[['Y']]

# seperate into train / test split
train_split = 0.7

train_size = floor(len(X) * train_split)
test_size = len(X) - train_size

X_train = X[0:train_size]
X_test = X[train_size:]

Y_train = Y[0:train_size]
Y_test = Y[train_size:]

# model
model = Sequential()

model.add(Input(shape=1))

model.add(Dense(units=1,
                activation='relu'))

model.compile(loss='mse',
              optimizer='adam',
              metrics=['mse'])

model.summary()

# train
history = model.fit(x=X_train,
                    y=Y_train,
                    epochs=1000,
                    validation_split=0.1,
                    shuffle=True,
                    verbose=0)

# metrics
plt.plot(history.history['loss'], label='Training loss')
plt.plot(history.history['val_loss'], label='Validation loss')
plt.legend()

# make prediction
input_sample = X_test[1:1 + 1]
print(input_sample)
prediction = model.predict(input_sample, verbose=0)
print(prediction)
$\endgroup$
3
  • $\begingroup$ Have you tried a different learning rate? It might be caused by the fact that in some cases the model get stuck because of the loss landscape as a result of a bad weight initialization. $\endgroup$
    – Oxbowerce
    Aug 6, 2023 at 18:17
  • $\begingroup$ No I have not I will experiment with that now thank you. The default learning rate for adam is learning_rate=0.001, I'll experiment but do you think it should be higher or lower? $\endgroup$
    – mathbike
    Aug 6, 2023 at 18:21
  • $\begingroup$ I'm experimenting and it seems a bit smaller is working, then too small doesn't work. $\endgroup$
    – mathbike
    Aug 6, 2023 at 18:31

1 Answer 1

1
$\begingroup$

Because the models are randomly initialized, thus they start from different points in the loss landscape, which being a non-convex one most likely, means that they will take different trajectories

Absolute no learning usually happens with activations that have 0-gradient areas such as ReLU, if they are initialized badly

If you need a strictly positive value as output, you can use exp or selu as activation functions, which have saturating areas but never 0-gradient areas, which are less susceptible from the initialization ()

$\endgroup$
3
  • $\begingroup$ I'm starting to understand what you're explaining, thanks for the information. This problem has led to learning about hyperparameter tuning, scheduling the learning rate and momentum, and using callbacks. So overall this was a good project. Thanks for your help. $\endgroup$
    – mathbike
    Aug 6, 2023 at 21:20
  • $\begingroup$ @mathbike I would also encourage you to consider giving a look at the activation functions cited, or at least to initialization... if you have normalized features and strictly positive weights, you will see that relu somewhat works $\endgroup$ Aug 6, 2023 at 22:14
  • $\begingroup$ For my notes I'm working on derivations of all common activation functions including selu, and I do intend to learn more about when relu initialization fails because I have heard this before. I do also need to learn more about initialization. I'm learning KerasTuner right now though because it's solves exactly the problem I've been thinking a lot about recently, what parameters should your model have. Also going through the Andrew Ng coursera course again because I forgot a lot of stuff I learned the first time. Anyway $\endgroup$
    – mathbike
    Aug 6, 2023 at 23:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.